Probably a RMW problem which is well documented. The PIC16 series doesn't simply set individual bits when you write to a port, it reads the whole port, sets the bit then writes it all back again. If bit 4 is externally floating low or pulled low, the write (OR operation) to bit 5 reads bit 4 as zero so it writes it back as zero.
The PIC18 series uses LAT registers to overcome the problem but the simple solution to your issue is to keep a variable for the bit operations and write it to PORTB whenever you change its contents.
Brian.