There are a number of possible reasons.
1. To stabilize the current, the input voltage must be at least the LM317 overhead voltage + Vref + combined Vf of all the LEDs and must of course be able to maintain that at the load current.
2. The LM317 is not a precision reference device, it will slightly change it's reference and regulation current as it warms up.
3. The current is decided by the voltage drop across the resistor, is it's value staying the same as it warms up?
4. Depending on wiring lengths, a single 0.1uF capacitor at the input may not be sufficient to ensure stability. Try a capacitor of 10uF or more in parallel with it.
Brian.