The maximally flat response is obtained by the Butterworth topology. You can estimate the required order to meet your requirements independently from the implementation (lumped or distribuited components) starting from the theory. Of course actual implementation has to be choosen to minimize unwanted behaviour due to non-idealities.
You forgot to indicate the maximum attenuation can be tolerated in band. I can suppose here Amax(dB)=0.2 dB that in linear is Amax(lin)=10^(0.2/20)=1.0233.
The minimum attenuation at stop band ws=2•Π•10 GHz is Amin(dB)=20 dB --> Amin(lin)=10
The transfer function of a lowpass Butterworth filter is:
H(ws)=Ho/sqrt[1+岕(ws/wp)^(2•N)]
where Ho is the gain in DC, ws the stopband frequency, wp the pasbband frequency (in you case wp=2•Π•5.6 GHz), H(ws) is equal to 1/Amin(lin), N is the order and ε can be calculated from:
Amax(lin)=sqrt(1+ε²) --> ε²=[Amax(lin)²-1] --> ε²=(1.0233²-1)=0.0471
sing now the transfer function:
1/10=1/sqrt{1+0.0471•[(2•Π•10)/(2•Π•5.6)]^(2•N)}
that is;
10=sqrt{1+0.0471•(1.786)^(2•N)}
then:
100 = 1+0.0471•(1.786)^(2•N)
Solving with respect to N, I obtained 7 (the higher interger close to the calculated value).