If (Vout,p + Vout,n)/2 != Vcm_out,desired ==> you have a common mode feedback problem
If Vout,p!=Vout,n when Vinp,p == Vinp,n ==> you have offset. This offset has nothing to do with common mode feedback. Your common mode feedback circuit can only change the value of {(Vout,p + Vout,n)/2}
P.S. Its not trivial to go for a very high gain of your common mode feedback amplifier as you will run into stability issues for sure (poles in your main amplifier + poles due to CMFB amplifier).