If you send sine signals with frequencies f1 and f2 to an ideal 4-quadrant multiplier, you get the output signals f1+2 and f1-f2 (respectively f2-f1, which is just the same signal), but no "fundamental" wave at all. So how would you calculate THD?
In case of a non-ideal multiplier, you get all kinds of modulation products N*f1+M*f2, with N and M being integer numbers. You'll notice N,M = +/-1 stand for the first order products which are also present with an ideal multiplier, the zero order products represent offsets and carrier feedthrough and |N|,|M| > 1 higher order intermodulations products. Their presence can be used as a performance criterion similar to THD for a linear circuit, but it's a different calculation.
- - - Updated - - -
I'm not an expert in this regard, but as far as I see from the manual, Spectre RF can measure mixer intermodulation effects with QPSS analysis.