There is a common error when discussing PCB laminates. FR4 is NOT one material with one set of specifications. It is a CLASS of materials based on its flamibility. FR4 is an abbreviation for FLAME RETARDANT (FR) class 4 material.
There are FR4 laminates with properties that are entirely suitable for use in the frequency range that you are planning to use. At our company, we normally use a high Tg FR4 with Er=3.5 for boards with fast risetime digital, and RF into the 8GHz range.
I suggest you talk with your board fab, and possilby some of the laminate manufacturers to see what FR4 materials are available to you.
Remember, FR4 is not one material - it is an entire class of materials with different properties depending on the type and amount of glass and resin.