The 5880 substrate is PTFE with a thin glass matrix. It is soft, and to have grounding vias, requires a special sodium etch before plating.
The loss tangent is 0.0009 at 10 GHz. There are many other RT Duroids that use part ceramic loading, still have low loss, are dimensionally stable, and can be etched and plated with conventional PCB techniques. There is also Taconic RF-35A. The strip losses for filters are often more affected by the suface roughness under the strip than simply going for the lowest value tan(delta) loss. Rolled versions are better.
For an edge coupled micristrip filter for which there are handy examples, the gaps can get very thin. Choose a substrate thickness that is more forgiving for this parameter. The very thinnest are not suitable for filters. Lines get very thin.
It is hard to tell why the simulated filter does not agree well with the first speculated design, but it will have something to do with differences in assumptions about the substrate properties. Taking an example, and scaling it for frequency may be a better approach. Keep in mind this means scale the resonant lengths. To retain the line impedances, you have to leave the widths, and the substrate alone, unless you know what you want, and alter those deliberately.