Well, you problem is indeed strange. One theory that I like says: If a high performance system generate the same results (right or wrong, no matter) running with with the same input variables, regardless of how often you process those variables, it means that the software inside it must be somehow ok. If the results are good or bad, that's programmer's fault. But if the system behave strange and generate chaotic results at consecutive processes, than check your PCB. The procedure described by you seem ok, but lemme ask you som' :
Such a system like yours definetely is a mixed analog-digital design. Two causes may exist.
1. check some details of your pcb: separate analog and digital power planes, do not route analog signals over digital planes and vice-versa, decouple well every digital IC with capacitors with self-resonant frequency much higher than knee frequency, route gnd and vcc pins of ic directly to power planes, not through long and thin traces, do not use vias on sensitive signals like sampling clock.
2. check routing inside FPGA. if your problem dissapears when lowering frequency, then you may have a tight condition inside FPGA algorythm, somewhere you may miss a setup/hold time for an input signal.
Also, from your description, I understood that you insert a sin signal obtained from a PWM. For this I think you use a converter from PWM to analog. How is this circuit powered? From digital power or from analog? I hope you did not connect digital ground and analog ground respectively digital vcc and analog vcc in more than one point.
Well, I hope this helps...lemme know.