"Touching ground" doesn't talk about residual voltage in a mV range. You have to refer to datasheet detail specifications instead of reading a specification into the first datasheet page.
I must confess, that I don't exactly remember what's the typical minimal output voltage without a load is. If you look at the transistor level schematic, you'll see that the saturation voltage is determined by a 50 µA current mirror (drawn as a current source). Because it's bipolar technology, you can expect a few mV saturation voltage.
A few mV output voltage is considered as "zero" in single-supply OP terminology, but it probably isn't zero enough for a mV buffer amplifier.