A GSM modem would be classified by most people as a DCE. Your LPC1768's serial port (assuming that you are using RS232 signal/pin connections) would be configured as it were a DTE. The specifications sheet for the modem should show you the signals on its pins and their definition/direction. Whenever you have a DTE <-> DCE connection, the RxD, Txd, CTS, RTS, etc. pins all have defined functions. If you want to make two of the same device types (example DTE <-> DTE) communicate that is when one must do the "null modem" cross connecting of RxD, TxD, RTS, CTS, etc.