I would also go for the resistive divider with about 1 k parallel with the input of the oscilloscope (so don't use a 1:10 probe). As mtwieg suggests, a series circuit of resistors would be OK. Compensation is very likely not requried.
When your want 400 mV output with 4 kV input, you need a division ratio of 10k. So the series resistance should be 10 Mohms (using 1k parallel to the input of your oscilloscope).
About 21, 470k resistors (just use leaded types) will do the job. If you have a pulse generator available with relatively large output, you could have just enough output voltage to measure the flatness of your home made probe (you may need averaging function on the oscilloscope to reduce noise).
If your application needs more then 10 MOhm, you may reduce the output voltage to say 40 mV. Make sure that your (plastic) support for the resistors is clean before assembling/using your probe.