this is normal
pin a.4 is an init pin for severl special function registers
so i dont think the config and functions register
has pull ups portb only i think on 84 {not sure}
look in the data sheet
just need too set up the code or better is use a 220R res ad a 10k to + too give pull up