Have anyone thought about the filter outband supression?

Outband supression requirement is very important when we design a RF system.
We often can easily calculate how much we need on a frequency point according Radio budget allcation.But it often under 50Ohm system.

My question is does it needed for a requirement with 50Ohm system? Because sometime there is not 50 Ohm load in RF system on outband frequency.For example, if filter output load is a mixer, it can working in some frequency(narrow).
Maybe in N*LO-RF frequency, the input impedence is not 50Ohm, filter will be difficult to get seem performance we design by network analyzer.

So why we need a requirement for filter? Or how to make a requirement?

You should measure the load the filter is working into. It is generally best to put the filter early on in the system. Even having two filters. One at the antenna input and one after the RF amplifier. Once you specify the source and load impedances and the in band ripple and out of band attenuation you can get the filter element values.

It may be necessary to use a simulator with an optimizing function to vary the filter element values to get the response you want in the real system with varying loads and sources.

As far as the mixer goes, it is imperative that all three ports be terminated in the designed for load (usually 50 ohms) over all frequencies if you want to have the IP3 values be high. This means never put a filter next to a mixer in the signal flow path. You can put a diplexer inbetween with the out of band port terminated with a resistor. For lower frequencies you can use a bridged T constant resistance bandpass network.

Back before the earth cooled, we used to overdesign everything. We would put a 3 dB pad between the filter and mixer, or an isolator, for example. And the electrons were happy.

Nowadays, where they flail you alive for adding an extra $2 to a circuit, such throwing away of gain or adding cost is not allowed. You have to learn to live with the mismatch. If you really have a problem, design in some sort of self terminating feature on that side of the filter, like a highpass to a resitive load in parallel with the mixer, so that out of band you at least have a partial match no matter what the mixer does.

Added after 3 minutes:

BTW, this is why you can not take two filters with 40 dB of rejection each, put them in series, and expect to get 80 dB. Because one of the filters is terminated in an open or short out of band, it no longer works well! You need a pad or isolator between them now.