The input and output impedance of a two port network are related to S-parameter as:
Zin=Zo[(1+S11)/(1-S11)]
Zout=Zo[(1+S22)/(1-S2)]
Be careful not to confuse Zin and Zout (input and output impedance) with Z11 and Z22 that are elements of the impedance matrix. They are different concept.
These two Zin and Zout are measured from the VNA and don't vary changing the characteristic impedance of the system, so if you just want to calculate the return loss you can just use them referred to the Zo impedance you want (f.i. 1 ohm).
Then if you have measured Zin and Zout in a 50 ohm system (or any other impedance: doesen't matter) and you want calculate the return loss in any Zo system (1 ohm or what you prefer), simply apply:
RLin=20*log(|Zin-Zo|/|Zin+Zo|)
RLout=20*log(|Zout-Zo|/|Zout+Zo|)
If you want, instead, recalculate the S-parameter (that depends from the value of Zo) you have to apply a little math.
Starting from the two previous equation (omitting the subscript numbers) if we perform the measurement under two different system characteristic impedance, let say Zoa and Zob we can write:
Z=Zoa[(1+Sa)/(1-Sa)]
Z=Zob[(1+Sb)/(1-Sb)]
where Z will be Zin or Zout depending from which you want to convert, Sa and Sb will be accordingly S11 or S22 measured in a Zoa and Zob systems. Then
Zoa[(1+Sa)/(1-Sa)]=Zob[(1+Sb)/(1-Sb)]
Let's suppose you performed the measurement in a system with Zoa and you want to convert to Zob, then Sa is known and Sb is unknown.
Solving then by Sb We have:
Sb=[Zoa-Zob+Sa*(Zoa+Zob)]/[Zoa+Zob+Sa(Zoa-Zob)]
Sa and Sb are complex numbers.