Hi,
In detail the datasheet will give answers.
There may be pins that are "floating" after power up, there may be pins that are pulled to a known level (dedicated I2C pins may be internally pulled up).
To be sure you need to read the datasheets - like all of us.
So at most microcontrollers I've seen the port pins go "high impedance" and not "HIGH" as you assumed.
The usual method to get a known state at power up is to use external pull up or pull down resistors. Whether an additional capacitor is useful depends on your requirements.
Klaus
Note: at most of my microcontroller projects there are some pins externally pulled HIGH or LOW. Like /SS, /CS, /RD, /WR...
This avoids short circuit that happens when multiple bus partners actively drive data lines at the same time.