As Klaus noted, the op amp no longer can maintain zero relative voltages between the op amp positive and negative input when it reaches the rails but the capacitor keeps charging through R3. That's called "integrator windup". This excess charge must then be removed when the input changes, leading to the delay in the output noted.
The solution is to keep the integrator from saturating, either by detecting the voltage and stopping the integration as Klaus stated, or possibly by placing zeners across the capacitor. The problem with low voltage zeners is that they have a high leakage current and will not limit well with the high integration impedance you have.
Note that the capacitor must be non-polarized (not an electrolytic).