NMOS high speed than PMOS. But in the usual p-substrate process, NMOS source follower will have harmonics distortion caused by body effect since NMOS's substrate cannot be tied to the source.
The body effect mentionned by terryssw will also make the gain less than 1, due to gmb. PMOS could be closer to 1, but still lower.
Other main differences I can think of:
- PMOS will create a positive shift in your DC point whereas it will be negative for NMOS (about Vth).
- Noise performance migth be slightly better with PMOS