You have to add the following resistors (at least R3 and R4) to avoid opamp output oscillations:
Those resistors actually change the IN+ voltage depending on opamp output.
By example, to trigger high the opamp output you have to apply an OVERCURRENT_SENSE voltage of VREF + 1V but to trigger it low you have to apply an OVERCURRENT_SENSE voltage of VREF - 1V.
With your current circuit, the opamp triggers its output for the same IN+ voltage (VREF) so the oscillations begin.
The R3/R4 ratio set the voltage difference between positive and negative triggering thresholds.
On current implementation, the capacitor is discharging through the SG3525 pin 10, too (not through R5 only). The pin 10 could sink up to 1mA (reading the datasheet) so it's discharging the capacitor much faster than the R5.
The shutdown voltage is also decreasing during capacitor discharging process.
For a more accurate and steady shutdown signal, I STRONGLY RECOMMEND YOU to put another opamp in a similar configuration (hysteresis comparator) after the current one. This way, you could supply a stable and constant shutdown voltage and a controlled shutdown reset delay.
If you want, I'll post a complete schematic of this circuit.
- - - Updated - - -
That's the complete schematic: