You can save a lot of space by using the fact that sine waves are symmetric in two ways. In the range from 180 to 360 degrees, you can use the values from 180 to 0 degrees and negate them. Also the range from 180 to 90 degrees is the same as from 0 to 90 degrees.
Therefore you can have a table that is only the first quadrant of the circle. When going from 0 to 90 degrees you step forward through the table. When going from 90 to 180 you step back through the table. From 180 to 270 you again step forward but negate the value, and from 270 to 360 you step back and negate.
As for debugging, you probably need to break the problem down into manageable chunks to start with. I assume you know how to use the debug capabilities of the IDE. I really don't know your problem domain but I could imagine that you should start with (say) the encoder to make sure that you are reading the input pulses correctly - even when you are slowly turning the rotor by hand; then that you can get the counter to work; then look at the table lookup part using the counter. Next start with getting the PWM signal to be at 50% and then vary that correctly - forgetting about trying to do this very quickly. You can then connect the encoder part up to the PWM (still turning it by hand) to make sure that the PWM signal is being updated over the full turn of the rotor. If all that is working at slow speed then you can try to speed things up until it is working at the required speed.
What you are really doing is making sure that each step is based on a working foundation. If you add something and it stops the whole thing working then they last thing you changed will tell you where to start looking for the problem.
Susan