From HFSS help file:
Modal
For calculating the mode-based S-parameters of passive, high-frequency structures such as microstrips, waveguides, and transmission lines which are "driven" by a source, and for computing incident plane wave scattering. Network Analysis is the default and functions as before. Composite Excitation provides a method for solving fields in a large frequency domain problem.
Terminal
For calculating the terminal-based S-parameters of passive, high-frequency structures with multi-conductor transmission line ports which are "driven" by a source.
This solution type results in a terminal-based description in terms of voltages and currents. Some modal data is also available. Network Analysis is the default and functions as before.
Composite Excitation provides a method for solving fields in a large frequency domain problem.
I use modal for single signal models and terminal if there's more than one signal line, such as differetial pairs.