That is very suspicious math!
Basically, the LED current must NEVER exceed the manufacturers limit, no matter for how long.
The perceived brightness is proportional to the average current.
The average is taken over time, at least the duration of one full on and off period. It is the current you have decided multiplied by the ratio of on to off time. For example, if you set the current to 10mA and it was on for 25% of the time, off for 75%, the average current would be 2.5mA (25% of the total).
If you want to make the LED brighter, you can increase the current but it must stay within manufacturers limit.
To reduce the brightness, you can either leave a longer 'off' period before next LED is powered so the average is lower or, you can 'chop' the on periods into smaller on/off periods using PWM or, you can use analog circuits to reduce the LED current.
You should also consider the time taken for human perception of light and persistence of vision. You want the matrix to appear as though all the intended LEDs are on at the same time. This imposes other limits, if you 'scan' through the matrix too quickly the LED drivers will not have time to turn the current on and off, this will result in the display losing brightness and may show smearing if you animate the LED shape. If you scan too slowly, you start to see flicker and moving shapes may appear to have steps in them, for example horizontal scrolling text may look like italics. The 1mS you mention is probably bordering on too short, you might like to consider 2 - 10mS as being more appropriate but you need to experiment to find the optimum speed for your application.
Brian.