Every person has a different capability to hear clear tones and distortion. Many papers on physiology of hearing were written but nobody understands the problem.
This is why the technicians designed distortion meters that quantify the harmonics of a clean tone which define the distortion.
On the other hand, some people can perceive other distortion not related to the harmonics.
Generally, for frequencies below ~200-400 Hz, very few listeners can perceive distortion, and only at a high volume. Most listeners can perceive distortion for >400 Hz as harmonics. People with "absolute" hearing can perceive as a distortion also tones off the common scale, like in Arabic and Hindu scales.
Even if you buy the best available speakers, the effect of other objects, room reverberation and presence or absence of persons will affect your perception.This is why good headphones are often preferred as only instrumental distortion is present and can be technically reduced.