There are three methods to measure the distortions:
1. The one was mentioned above, using a narrow band reject filter (or a notch filter, or a steep high-pass filter) with at least 80dB rejection. The measurement is done comparing the signal before the filter, and after the filter, with a reference signal.
2. Second method is using a selective voltmeter (or a selective receiver). This method can find harmonic distortions and also the intermodulation distortions, measuring the spectral amplitudes.
The setup compares a reference signal to the measured fundamental signal and also to its harmonics. Known equations can calculate the distortions of the signal, function of the amplitude of the harmonics.
3. The 3rd method is using a Spectrum Analyzer, which the same as a selective voltmeter is measuring in frequency domain, but instead gives the spectral density of the signal (which depends also by the distortions), practically measuring the average power in a specific bandwidth.