The simple answer is that the phase noise is additive. That is your phase noise is the, phase noise of your synthesizer added to the residual phase noise of your buffer amp.
The phase noise of your buffer amp probably will degrade a little if driven hard, expecially if it is a FET amplifier. The phase noise of the amplifier will generally be too low to worry about. THis is usually true, unless you care about very close to the carrier noise (where the buffer amp might add in some 1/f noise), or if you are driving the buffer amp with a switching power supply and incur some sidebands.
In one respect, driving the amplifier hard will push it into a region where it does not have much am to pm conversion, so little of the synthesizer's AM noise will get converted to PM noise.