The scale of the "problem" depends on a lot of things.
If you pump charge, by some rectifying effect (and there
are many opportunities) then the result is a sampling
pedestal that looks like an input offset voltage to the
next thing downstream. Differential circuits well matched
might first-order reject this.
If your injected perturbation settles quicker than the
next stage acts upon it, there might be no impact. But
more likely this mechanism accounts for some of the
rolloff of of ENOB with sampling frequency, as you move
closer to the perturbed region the error un-settled is
greater.
Generally it's a LSB-scale problem, an accuracy
detractor. Whether this is "serious", is contextual and
often a negotiation.