You may know that the actual noise figure of an active device depends on the source impedance. When parasitic capacitance/inductance can be ignored, optimum source impedance depends on the noise current and voltage (all transfered to the input of the active device). So changing the load impedance may change gain, but effect on F can be small.
Mostly Zs(optimum noise) <> input impedance of active device. So matching a device for maximum power gain, does mostly not result in lowest noise figure (F). By using feedback, one can modify the input impedance of the amplifier (=active device + surrounding components) to get the desired input impedance.
When Rf is not very large with respect to Zs, it will contribute some noise to the input (hence reducing the noise figure). When you change ZL, the voltage gain (not insertion voltage gain) of the amplfiier changes, hence the current through Rf and the amplifier's input impedance.
When Zs=50 Ohms gives the desired noise figure and due to the feedback the input impedance of the amplifier = 50 Ohms, you don't need matching.