You don't need a switch. Speed wants simplicity. But to get simple you must express the electrical operation neatly.
Like, source or sink? To gnd, virtual gnd or lean and consistent over some common mode range?
Might be a tad slow but I'd look at "NPN and PNP matched pairs" as simplest current mirror, couple of resistors to set the levels and a 50-ohm sig gen to toggle between.
Such specialty chips I have seen in old Intersil UHF-1X (10V, 8/4Ghz) and THAT Corp (36V, 500/300Mhz fT). May be others.