There was a similar thread on this some time back
https://www.edaboard.com/threads/360476/. Let me try and summarize this here.
First I feel two things should be made clear. 1) A system is stable as long as the closed loop response has poles in the left half plane. In other words, the system is asymptotically stable. 2) Even though the close loop poles are in the left half plane, for good step response behavior, one needs to tweak the open loop response such that it is away from the critical 0 dB and -180 phase. This is the classical Barkhausen criteria. But it's only a necessary condition for instability.
Assume the open loop response is L. For unity feedback, the error signal (difference of input and feedback input) follows the equation -e*L=e. This equation is satisfied for e≠0, only when L=-1(the critical point). Only when L=-1, the system can sustain oscillations.
If L is anything else, e has to be 0. The loop cannot sustain oscillations.. But having L=-1, doesn't say anything whether the closed loop poles are in right half plane or not.
Now on bode plot, one only looks for point (2) above. That's where phase margin and gain margin come into picture. Looking at only a point or a portion of frequency response where the gain is > 0 dB and phase equal to -180 or >-180, one cannot conclude that the poles of the closed loop are in the right half plane. This is completely erroneous. To ascertain right half plane poles of the closed loop, the only way out is to look at the complete response for frequencies from -∞ to +∞. Using Cauchy theorem from complex analysis, Nyquist plot does it very efficiently. On bode plot one only looks for the response for positive frequencies or sometimes just around the unity gain crossover frequency(when designing opamps).
For monotonic phase response(which always decreases with frequency), a special case arises. In these systems, the boundary between stable and unstable is very clearly defined. As the gain keeps increasing, at some point the unity gain crossover moves so far towards the right that it becomes unstable. For these systems, one can be sure that the closed loop poles will be in right half plane as the gain increases. But the moment phase response shows non-montonic behaviour the system becomes conditionally stable. It is stable only for certain ranges of gain. These are also called non-minimum phase systems.
One can do the same excercise on bode plot, by plotting the response from -∞ to +∞, and come up with an equivalent encirclement criteria similar to Nyquist plot. I think it should be counting how many times the phase has +ve or -ve slopes. But I guess bode plot will be to messy to look at for such a large range of frequencies. That's why Nyquist plots are more popular when one wants the complete response for frequencies from -∞ to +∞ and determine stability.
In summary, stability has two parts 1) Are there right half poles? Is the system asymptotically stable? 2) If it is asymptotically stable, how far is it from the critical L=-1 point. Bode plot doesn't care about (1), one just checks for (2). On the other hand, Nyquist plot looks for both (1) and (2), and is the full proof method to check stability.