smps loop compensation
OK, I got it.
The ringing you are seeing on the primary is actually a normal occurrence for discontinuous mode flybacks. It comes about because at the end of the flyback time, or diode conduction time, the MOSFET's D-S capacitance is charged to a voltage higher than the input voltage and it now has to discharge to the input voltage. The voltage will not simply decrease to Vin, because Coss forms an LC resonant tank with the transformer inductance. To that you need to add the reflected capacitance of the secondary diode (which has just turned off, so it looks like a variable capacitor), reflected by the square of the turns ratio. (Even if Coss were zero, the diode's capacitance, reflected to the primary would still cause ringing, resonating with the transformer's inductance.) Anyway, that is why it happens: you have a charged capacitance (part of an LC tank) which needs to discharge to a lower voltage. The discharging process will cause the LC tank to resonate, creating the ringing you see.
The waveform will decay exponentially, but if there are low losses in the circuit, then the exponential time constant will be fairly long.
You can connect a snubber across the secondary diode (say a 10Ω/2W in series with a 2.2~10nF cap). That will reduce the frequency of the ringing but it will also make it decay faster, due to the resistor, which is a lossy component.
The ringing is usually harmless, but sometimes can cause EMI issues. The cure is usually provided by the snubber across the secondary diode. The ringing does not reduce the duty-cycle, since the ringing can only occur AFTER the diode has turned off and the diode turns off only AFTER the MOSFET has turned off. The gate drive waveform is clearly in sync with the primary waveform.
(The primary waveform is normally viewed across the transistor, because that allows you co connect the scope's ground to the primary ground, not Vcc. Just keep in mind for future measurements.)
I do not see any evidence of oscillation in your waveforms. That would manifest itself as a "jitter" on the drain waveform, which I do not see (perhaps it is there, but it's hard to tell from only two cycles; the second one might seem shorter, although to my eyes, the both look OK).
To make sure, you should measure the voltage at pin 1 of the chip, with respect to primary ground. The voltage should be a DC voltage that changes with the load, but if you keep the load constant it shoulbe pretty much DC. If you see something that looks like a sinewave (a few kHz) superimposed on it, then you know the loop oscillates.
Note that in SMPS you do not measure voltages with the probe tips and ground leads attached, as they will pick up lots of noise. Instead, you remove the probe's tip and attach a short ground adaptor. If you do not have one, just use some bus wire that you wind (4-5 turns) on the ground sleeve of the probe, close to the probe's inner tip (pin). Leave only about 0.5" of bus wire attached, which you bend parallel with the probe's tip. Touch the ground wire to GND and the probe's pin to the point you need to measure. When you measure the error amp output (pin 1) you touch the ground wire to pin 5 and the tip to pin 1. That way you have the minimum loop area to pick up noise. But be very careful, a slip of the hand and... the short is sure to happen (I hope you are using some sort of isolating transformer while you are working on this P/S). You can solder several of these ground "coils" at key points and only insert the probe when you need to measure.