Hi
Hot swapping requires you to ensure correct "powering up" your card.
For example: if only 3.3v supply is in used: connect a P-channel fet - Source to the 3.3v pin, resistor between S and G, and connect the G to the GND. the circuit will be powered by the D of the fet.
What have you got? - if the 3.3v is connected without the GND - FET will not open - card will not get VCC without GND... You can extrapolate for extra connection and powering sequence.
Also smart to note: hot swapping many time ends with misconnections (user insert part of the card with miss-alignment) and result with power pins connected to the wrong inputs - diode are used for protection at the appropriate places.
Shutter_man