Even if you know the precise spectral output of your light source, imagine this loose example:
Wavelength A gives a reflected power of 3mW per unit area, and the photodiode is sensitive to this wavelength at 2V per mW.
Wavelength B gives a reflected power of 2mW per unit area, and the photodiode is sensitive to this wavelength at 3V per mW.
The output is 6V. Which wavelength has caused the output? There is no way to know, you cannot work backwards from a single scalar value to multiple input variables. And make no mistake: you have input variables that you cannot eliminate from this system.
In a laboratory setup, with carefully measured equipment and specific, pre-determined colour samples, you might be able to differentiate between them in this manner. But in the real world, as soon as a new variation is encountered, the system will fail.