If P2 and P3 are terminated in Open, and VNA measurements are performed between P1 and P4, what is the relation between the initial 4-port S-matrix (describing the 4-port) and the measured 2-port S-matrix (obtained from VNA)
Interesting question here. firstly we should consider that the 4-port s-parameters for the device probably assume proper 50 ohm terminations at all ports. so I don't see how you can compare them. The coupler won't work properly without the right terminations.
mkonca is right S-Parameters are measured assuming termination on all other ports. See the theory of S-parameter. I am afraid you are on wrong direction.