This is a difficult challenge. I have discovered "some" IR laser mice ( with red indicators) can sense hand waving at 3 meters, indicated by going out of sleep mode to active mode or slow pulse to fast pulse and placing mouse inverted on the floor. But to detect direction requires a differential and sum measurement of at least 2 laser or narrow beam 10deg proximity sensors used in mice
Since Mice focus is just at/below the surface, many mice will not work for long range, but some do,. It all depends on the optical splitter and lens. Then you would just need to use a dual beam emitter/detector with fixed location to detect with detector is stronger for motion direction feedback and sum of the two (or logical OR) for motion detection. The cleverness is in the optics.
In your case with 3 narrow emitters and one broad receiver can be sequenced for 1st reflections but you will get 2ndary reflections off walls and other things that will make it unreliable. Whereas independent Emitter/detectors sequenced or mux'd in TDM fashion can be synchronized with detection to determine the result.. COnsider a 6 stage Joihnson counter with interleaved off-states. Very narrow beam laser emitters with some beam spreading lens is required for Class 1 lasers . even 150mA narrow beam IR diodes can be hazardous.
In any case. reverse engineer Laser Mice for more ideas.