You need some ohmic or clocked means to ensure the front
end "knows what's zero". If this is part of a sampled data
lineup then maybe consider turning this into a T'H amp that
during "hold" is also applying a shorting switch to the core
inputs and puts the inboard side of the caps to a resting
reference (Vdiff=0, Vcm=fortunate) and then on return to
"track", reconnect as shown without this.
The inputs as is, could drift anywhere short of breakdowns
and you should be looking at core input common mode vs
common mode "works right" range.