Most dividers output insufficent power to drive the LO of say a diode based mixer. many frequency dividers will output say 1 Vpp, while a burly mixer may require anywhere from 7-20 dBm of LO drive. The reverse isolation can keep your IF and RF signals from getting back into your system as well.
As stated a broadband match on the LO port of the mixer (and RF, and IF) really helps to keep the mixer "happy". A common diode based mixer creats a lot of odd harmonics (3rd, 5th, etc). If these reflect back into the mixer it can result in changes to the conversion gain. Similarly the match at the image frequency at the RF port can affect the conversion gain. So especially in broadband applications (i.e. swept LO) it is of paramount importance to either use pads on all ports, provide termination at the image frequency, or use isolators if you can't afford the loss of a pad. Doing so will greatly reduce the amplitude of ripples in both conversion gain, and ripples in your IP3.