If you are just experimenting, go for it. If you have a critical application and need the part to not fail....all things have a shelf life! Semiconductors are subject to damage if exposed to the elements, and a plastic package does not protect them forever. Eventually air or moisture will leak in, or the plastic itself will outgas and contaminate the semiconductor. I would probably not try to use semiconductors more than 10 years old for that reason.
Aluminum electrolytics can be "rejuvenated", but yes they do die after a while. Older tantalum capacitors can fail rather spectacularly. Old circuit boards are hard to solder too unless it is just a thin layer of surface corrosion which can be cleaned up with a wire brush.
Ceramic capacitors are basically rocks, and will last ~forever. Old solder will have the flux dry out in the middle, and will not work too well.
But with the price of components being so high today, the only way an experimenter can get by is to buy used components. Some of the better deals are kits of components, like a kit of every common leaded resistor value in a set of drawers, a partial reel of 0.1 uf chip capacitors or 4.7 uf tantalum capacitors, a small box of cut reel pieces of chip resistors (50 of each value). Stuff like that just keeps on giving day after day!
Don't get tempted to buy any dangerous stuff! You do not want a big kit full of cadmium plated screws for instance, as the cadmium dust is toxic! Too much lead plated stuff around the house is a bad idea. Asbestos is a bad thing to breath in. Common solvents from days gone by are now considered carcinogenic. etc etc.
Used tools can be a great deal too. Pick up a couple used weller controlled temp soldering irons for $15 each, but some new tips and sponges, and you are good to go for a $400 savings! Used test equipment can be a good deal too if it is in working condition. For a few hundred bucks you can get a quality tektronix scope, a couple power supplies, and a function generator.