Hi,
There is a Verilog (in the context it may be referred as Verilog-D) to model digital processes, there is Verilog-A to model analog processes. Verilog-AMS allows modeling both analog and digital processes in the same module. You can consider that Verilog-D and Verilog-A are subsets of Verilog-AMS.
I am inclined to think ModelSim won't simulate Verilog-AMS code. Search their website.
Usually, there should be two simulators involved: digital and analog, like VCS and HSIM.
To use Verilog-AMS language I would suggest first reading a textbook on Verilog-A/AMS (The Designer's Guide to Verilog-AMS is quite good), walk through examples. Next, you should invest some time to understand tools setups for cosimulation, like selecting analog simulator, setting interfaces between digital and analog, handling ports mismatches, etc.
If you know SPICE and Verilog, Verilog-AMS is easy to learn.