The cutoff frequency for this waveguide is ~6.55GHz. Also, for this analysis, dont plot S12. It is best to plot Real and Imaginary Gamma and observe when Im Gamma becomes greater than zero. You also only need to do a ports only solution for this...much faster. I would check your math one more time to ensure you did not make a mistake. Here is a quick reference: **broken link removed**
Also, to increase accuracy of HFSS for these simulations, go to HFSS->Mesh Operations->Initial Mesh Settings and change the default Surface Normal Deviation from 22.5degrees to, say, 5degrees. When I solve this I get a cutoff that corresponds closely to the theoretical as per the website.
have Fun
Later