Hi amicloud,
you didn't mention if you used circuit simulator only or you used EM simulation as well. Having in mind the size of a filter it might be a chalenge for EM.
Anyway, your filter seems to be mismatched meaning that positions (frequencies) of transmission poles and zeros are not where they should be. Filter has a lot of Ts and opens, and is electrically very long. There is an relatively large error in modeling Ts in cicuit simulators (2-3%). This make a small difference between stub reactance value expected in the simulator and a value in real circuit at every T. The other possible source of error is the open end of the stub. Error is smaller here, but it exsists.
When the small difference in a T is transformed over a large length of transmission lines (towards the input and towards the output), and when you take all Ts and add opens in consideration you will have large difference in overall filter response as a result. This is my explanation.
All I can think on to make your life easier is to use EM simulator. If the structure is to large to simulate in whole, you can at least encounter for every T and open in detail and than tweak a length of stubs and lines between the stubs to reachieve specifications. So, simulate Ts and opens in EM simulator and transfer them in ciruit simulator for tuning/optimization. I think this should work better than your first result.
flyhigh