What is the value of your load and smoothing capacitor?
Even a 320k ohm load, drawing 1 mA, will require several hundred mA from your 5V supply. Can your supply provide that?
Is the resistance low enough through the power loop to permit this?
At startup it will draw a few amperes (if possible). If resistance is too high then the converter may not reach your desired output (320 V).
Also in this development stage, I recommend that you try putting a full diode bridge on the secondary. In running simulations I have observed that current will flow during the switch-On cycle at startup, but then it flows during the switch-Off cycle once the capacitor is charged (normal running).
Here is my simulation, with scope traces.
The load has reached 316V. The switch is Off, and the secondary loop is conducting 7 mA. (See rightmost scope traces.)
However it was the opposite at startup. The secondary conducted during switch-On. (See leftmost scope traces.)