Basically, if the underlying fading channels are Rayleigh, then SNR follows exponential distribution. That is y0, y1, and y2 are all exponentials, where y0, y1, and y2 are the instantaneous SNR of the direct path, the first hop of the relay path, and the second hop of the relay path, respectively. After generating them, calculate the equivalent SNR for AF as: y_eq=(y0+(y1y2/y1+y2+1)), then substitute this equivalent SNR in Q(sqrt(2*y_eq)) if you are using BPSK, which is usually assumed for simplicity. Repeat this several times and at the end average the BERs you got.