pay attention to the current loading of your regulator, sometimes, if current loading is not stable during power up, say, too small current, then the output voltage may fluctuate heavily, if your regulator is the traditional sample and feedback type.
Added after 1 minutes:
and to continue on this topic, when we design regulator, sometimes, we put a constant current dumping or controlled current dumping circuit under it, so that the dynamic of current supply range can be reduced, to help the stability of output voltage.