FR4 is not one kind of PCB material - it is an entire class of flame retardant glass and epoxy type board materials. Depending on the resin type and content, an FR4 board can perform quite nicely to 4 or 5 GHz or even higher (there are even 9 GHz backplanes described on the web that were done with FR4). Dielectric constants can vary between 3.7 and 4.7 depending on the manufacturer, type of glass fiber, and resin content. In general, high Tg FR4 has a lower Er and lower loss tangent. But you should look at the data sheet for the specific FR4 you will use.
You should contact the fabricator who will make your board and ask them what FR4 materials they have available. They should be able to provide a copy of the data sheets for the types of FR4 they can get to make your board.
Bottom line - not all FR4 is the same. There are high performance FR4 materials available. You need to ask the contractor who will make your board for data.