I believe that Ethernet is a balanced system, i.e. on transmission one line goes +, its pair -. The receiver should be looking between these two lines and ignoring any other signals. What are you using as a receiver for your test? It sounds as though the line has got one side earthed somewhere. So to maximise the TX level you have to provide a low impedance path from the lines to earth.
Frank
SMSC PHYs are known to generate some common mode EMI. The optional capacitors in the linked refernce circuit are e.g. provided for this reason, the manufacturers application note also suggest a certain PHY to ethernet magnetics trace length to attenuate the interferences. Part of the interferences is coupled over the transformer to the external side.
Shorting cable shield to circuit ground (via Y-capacitor) can (partly) conduct the EMI away. Additional common mode chokes respectively a ethernet transformer with better isolation are the regular way to get rid of it.