ICs can be expected to get damaged if exceeding the absolute maximum ratings. To achieve the specified peformance, you are required to keep the recommended operation conditions, e.g. Vcc = 5V +/- 5% for classical TTL and LSTTL devices.
TTL logic isn't designed to operate over a wider voltage range, HCMOS (74HC familiy) in contrast is. It has voltage dependent speed according to variation of MOSFET Rdson over gate voltage.
It's unlikely to damage logic ICs by a too low supply voltage.