Hi,
This FIR synthesis uses the Parks-McClellan procedure, that is based in an iterative method (the Remez exchange algorithm). This algorithm not always converges to the specification.
Situations when convergence can fail are, for instance, too high order, specifications with extremely high attenuations in stopbands or too low ripples in passbands, very narrow bands or sharp transitions (relative to sampling frequency), etc. This happens in the present case.
The "Density Factor" is a paramenter that can help, but I'm not sure how much.
Another issue is that the filter order estimated with the option "minimum order" could be insufficient.
In order to met your specification, you can design the filter as the cascade ot two or more filters (e.g. two identical filters with Apass=0.5dB, Astop=100dB, or 4 cascaded with Apass=0.25dB, Astop=50dB). Although the total order would be higher than if designed as a single filter, it would be very reasonable.
Even with a nice design, in a practical implementation finite-precision arithmetic and numerical errors can be the dominant limitation.
Regards
Z