As you increase the frequency, the number of mesh cells increases. That might be why you are able to solve at 6 GHz, but not at 8.7 GHz.
1. You can change the Maximum Delta S that is acceptable. Sometimes changing it from 0.02 (default) to 0.05 might lead to lesser number of adaptive passes before convergence (and generally a coarser mesh).
2. If you want to, you can import your own mesh in HFSS using the 'Import Mesh' option under the 'Advanced' tab in the solution setup window.
3. If the memory error occurs only in your fast sweep, but not in the initial adaptive solution at 8.7 GHz, you can change your sweep to a discrete sweep. Sometimes, the fast sweep needs more memory than a discrete sweep when generating the krylov vectors.