can u connect a new trace to charge the capacitor not throw the P mos instead using a diode .. in other words make the charge way different form the discharge way.
and why are u using super capacitor ?? 22000uF is not very high espically at low voltage , check if u want nichicon caps the size for 22000uF 6.3v capacitor measures 18X35.5
if u are concerned about the size you can use a "watch battery" they are used in computer motherboard for keeping time and other settings stored.