If your purpose is just generating VHDL code, MATLAB HDL Coder is suitable. However, it generates lots of files, and most of time result is not suitable for FPGA bitstream generation.
SystemGenerator is nice tool, it allows you to design your system with IPCores like designing in Simulink.
If you knew IP CoreGen, SystemGenerator allows you to connect IPCores like Simulink blocks.
There are lots of IPCore blocks,such as filter, fft, multiplier, dds, cordic, divider, etc..., Also you can write your own mini MATLAB code such as blocks and connect this block with ready-to-use IPCores.
But, Matlab codes should be as simple as possible to get fine results. Synthesizer tries to solve your code and if it is complex, your design becomes slower and uses high resources.
SystemGenerator convert your design into HDL, NGC or bitstream as you want. Find out tutorials of SystemGenerator, read it.
MATLAB HDL coder isn't a good solution for big and aimed-to-synthesized designs but, SystemGenerator is created for this purpose.
If you can't convert your solution into IPCores, try to make your design simpler or try to understand the blocks. Anyway, you can get some results with SystemGenerator.