The quarter wavelength condition comes from the formulation of the radiation boundaries. These boundaries assume that the incident field is uniform, and doesn't have much phase variation across the incidence plane - like a plane wave. The quarter wavelength is the usual rule of thumb for the distance after which this condition is *close enough*. If you need the ABCs to be closer, use a PML instead.
In HFSS, radiation boundaries are programmed to work with the default material parameters. If you leave the default material as a vacuum, and put the radiation boundaries along a dielectric, they will not work properly, and reflect much of the incident power. You can modify this material in the HFSS -> Boundaries -> Edit Global Material Environment (HFSS 13).