Probably charge injection, the Cgs/Cgd and gate
voltage transition impose a unidirectional charge
impulse when you go to hold (some in sample, too,
but it just scoots back out the now-on channel to the
driving source).
You probably want to size the NMOS and PMOS as
small as possible (so the charge is less, making less
pedestal voltage on the hold cap) and tweak sizes
until you get a first-order cancellation of net charge.
Too small a switch increases sample-mode settling
time, which is the other design bind.
However this may only be good near VCM, otherwise
dV(N) and dV(P) may differ (voltage swing prior to
turnoff, doesn't count but voltage swing after, does;
so at high common-mode offset the N
cancellation
may degrade a lot). Limiting the switch gate "off"
voltage swing to a small overdrive beyond
min(VCM,VIN) for N, max(VCM,VIN) for P, will also
minimize the charge injection at the cost of some
elaborateness in the gate driver circuitry.