First problem I think is to use a DVM on fluctuating voltage. Use an oscilloscope, and use a variable attenuator before your detector! Typical microwave sweep oscillators deliver +13...+15 dBm output, which can kill your detector in seconds. It starts by heating it- this looks to you as a variation. Sweep oscillators also need adjustments and do not generate too stable power over time.
First you can test your detector when the signal generator is set to CW at some frequency. I would advise to start with a power meter, and adjust generator settings for a stable power. Apply no more than -10 dBm to the detector, otherwise you damage it.
Then if the power output is stable, replace the power meter with the attenuator and detector, with a power to the detector < -10 dBm. Then you can use the swept frequency, and on the oscilloscope you can see output voltage detected across the swept band. Then connect the sweep horizontal output from the sweep generator to oscilloscope X channel, and you can see how the sweeper and detector behave together over the swept frequency band.