Hi
I think total BER is total bits in error divided by total number of bits. So, if you are using 16-QAM and 32-QAM and for example you transmit 20 symbols from two transmitters. So, total number of bits transmitted are 4*20 + 5*20 = 9*20. Now if BER for 16-QAM is say 1/40 and that of 32-QAM is 1/25, then total bits in error are 4*20/40 + 5*20/25 = 2+4=6 and total BER is 6/(9*20)=1/30.