You remove the signal power (include +/- a couple of bins is the signal does not fall in one bin) from the output spectrum, then summarize the remaining powers (should be noise + distorton power) inside your band of interests, then the signal power divided by the the latter noise+distortion power will be the SNDR.
To decide the noise floor, you need to do a sweep of the input (equivalently, output) amplitude, then extract the noise floor from the output SNR vs amplitude curve. The output amplitude at which the SNR falls to zero is the noise floor.