Okay, I'm surprised that the ESR of the battery is high enough to cause a voltage change of more than a few mV. But even so, you should never observe the output of the comparator/op amp being anything other than 0V or 5V (same for the gate of the PFET), except for the very brief switching times. Even in the case where the current feeds back to the error amplifier. What should happen is the circuit will form a relaxation oscillator, whose behavior will be related to various circuit parameters (ESR of the battery, battery charge, output current, comparator hysteresis, time constant of the feedback, etc). This is not necessarily a bad thing. The PFET is still acting like a good switch, but it may be toggled very fast, meaning the average output current will drop. But even it's still a switch.
How are you observing this behavior? An oscilloscope, or a DMM?