First of all you did not specify x. It should be quantized value of the signal y. You must first quantize y with n bits resolution and then form symbols with values 0,1,2 or 3 using consecutive 2 bits of quantized signal (for M=4) . Store the resulting symbols in x and then execute