sky_123 is absolutely right. You problem is compunded by the regulator itself which needs at least 8V at it's input to maintain 5V at it's output. Below 8V it becomes somewhat unpredictable and can even oscillate it's output voltage up and down.
Commercial LED displays usually use a system called multiplexing where they use a trick of eyesight to reduce wiring and current consumption. Our eyes can't see very fast flashes so rather than drive all the LEDs simultaeously like you do, each digit is switched on and off again in sequence, so fast that they all appear to be on at once. Electrically, because only one has power to it at a time, the current consumption is much less. Unfortunately, your driver circuit doesn't lend itself to multiplexing easily although with care you could sequence the 4511s so all but one is disabled at any time.
Brian.