pll peaking versus phase margin]
PLL operates pretty much like a classical control loop.
The closed loop gain of a loop with a forward gain of G(s) and a feedback gain of H(s) is
Gain = G(s)/(1+GH(s))
GH(s) is the gain when the feedback loop is "opened up", or commonly known as the "open loop gain".
So, the closed loop gain behaves well EXCEPT when it goes to infinity. It can go to infinity whenever the quantity GH(S)= -1. In this case, the closed loop gain is
Gain = G(S)/(1-1) = infinity, or the loop oscillates.
So lets say instead that GH(S) = -.98
The closed loop gain does not go to infinity, but gets really high! So any perturbation to the loop has an exagerated response (ringing in the time domain, or peaking in the frequency domain).
So, a good control loop engineer designs his control loop filter to not exhibit this abnormal high gain condition. He realizes that there are two criteria he has to simultaneously meet:
1) When the magnitude of GH(s) is around 1 (or 0 dB), then the open loop phase angle of GH(s) had better be far away from 180 degrees. This is known as phase margin.
2) When the phase angle of GH(s) is around 180 degrees, then the open loop gain had better be far away from 0 dB. This is called gain margin.
It is often helpful to plot the open loop gain and phase angle on a "Bode plot" to visualize this.
IF the above two criteria are not met, then the control engineer adjusts the loop parameters accordingly to do a better job. When he achieves 60 degrees of phase margin, he notes that the "peaking" has completely gone away.
He "adjusts" the loop parameters by changing the gain, dominant pole frequency, or dominant zero frequency.
So, in summary, you get peaking because in your control loop filter, you are making the system "divide by zero" at some frequency near your control loop bandwidth.
Fixing the ringing is not that intuitively obvious. In a standard type of pll, where there is an integrating op amp in the loop filter, there input resistor R1 connected to the inverting input, and the op amp output is fed back to the inverting input via a series RC pair, R2 and C. Then the product R1 and C determine the pole frequency and "gain" (gain is a function of frequency). The product R2 and C determine the zero frequency. In a crude way, one can then vary the R2 value around to try to move the zero frequency closer but always above to the pole frequency, which gives more phase margin and less ringing.