Firstly, check what division ratio you are using. The MB501L can divide by 64, 65, 128 or 129 depending on the voltage at the SW and MC pins.
The manufacturer only specifies it's performance down to 10MHz but I suspect it would work lower as long as your signal generator has a fairly fast rise and fall time. Keep the input level at around 200mV for best results.
Assuming the device is working, you might be seeing instability due to slow rising and falling edges on your signal generator waveform. To check properly you need far better test equipment I'm afraid. Try temprarily linking pins 1 and 8 to ground, both at the same time and see if the output changes. It won't be a conclusive test because you can't predict what state the output flip-flop will stop at but if the 3.7V changes at all it will give an indication that some signal was passing through it and therefore the chances it is working are good.
Brian.