To clarify, here's a segment of your code:
case 2:
PORTD = 0; // I added FvM's first suggestion here, which I assume you've already done...
PORTB = 0x02;
// ...because without it, the previous digit #1 will be output to #2 and cause flicker for as long as it takes to perform...
PORTD = CA[sec/10]; // ...this, of which the division in particular may be relatively slow
break;
Each digit needs to be lit at least 100x per second, or more, to appear constant to the human viewing it.
However, the actual digits displayed don't change 100x per second. Thus FvM's 2nd suggestion to move the math, like divisions, outside the ISR; and compute them at a slower rate, or only as frequently as needed. Let's say you defined a second array CB, and did this in your main code loop, or perhaps using a timer:
CB[2]=CA[sec/10]; // precompute digit #2
Now the ISR can set that digit by doing only a simple lookup:
PORTD = CB[2]; // instead of CA[sec/10]
And since each digit is now handled the same from the ISR's point of view, the switch structure can also be eliminated, saving even more time.
Now while you may not actually need to minimize time spent in your ISR for this particular project, in others you will - so doing stuff like this is good practice.