I'll prefix the following words by admitting upfront I'm a little (lot!) rusty on Dirac notation, but let me add my interpretation...
<a|b> is the :
[some alternative "phrasings"]
1. 'projection' of |b> onto |a>,
2. the inner product of states |a> & |b>
3. the probability of |b> being in state |a>
In this sense, both the definitions you mention are correct. The inner product is akin to the dot product - an operation that returns a complex coefficient indicating the "degree of overlap" between two vectors. If these vectors represent two states [can this be generalised to different basis? Apparently..(?)], then the inner product could be interpreted as a "transformation coefficient".
The definition in the above picture is also OK. \psi is the wavefunction, indicating the complex probability amplitude of the particle at position (x,y). The probability amplitude is the "practical interpretation" of the quantity returned via the inner product (which can be converted into a physically observable probability density by taking |\psi|^2), as seen by 'alternative phrasing #3' above.
Hope that helps!