is 't' your trellis? and what 'tb' are you using? and since you're essentially doing a bpsk demod with pskdemod(b2,2), try doing a slice to implement hard decision bpsk demod (demodout = (b2>0) ), or use b3 = real(b3) in your code.. post the answers you find, and we'll discuss further.