If you think of the drivers as mechanical devices with defined excursion limits, overdriving them can and will damage them mechanically at least. This can happen either due to poor signal choices from the user and/or due to poor enclosure design. A great way to do this is to take a speaker, select a very good amplifier with good power output, and feed it the actual canon shots recorded in the Telarc version of the 1812 Overture at full gain. OTOH to picture what clipping does to a driver look at a driver not only as a mechanical device, but as an energy converter. When the signal is clipped the driver's efficiency at generating heat increases. Some people explain this by asking to visualize the cone not moving when the signal is clipped (at the plateaus). The energy goes into heating the voice coil(s) instead. This is a backwards way of looking at a driver but it makes sense when the end result is smoke.
The reason I said that in practice RMS power ratings don't matter is that in practice they do no exist, at least not with any definition I can understand given the little physics I can follow. RMS describes current or potential difference -- the alternating current equivalent of a direct current that yields a given amount of heat for a load (for a sine wave this can be calculated as the peak multiplied by the inverse of the square root of 2). I see no way of describing power in terms of RMS values -- only potential difference or current (granted, they yield power, but unless someone can clarify then I'll stick to my thinking).
In any case I question devices rated as X power, when X is enough to run a saw mill, the device also states that it only draws a fraction of the claimed output power to run itself, plus generate heat due to inefficiency, and all this depends on a stiffening cap, a simple alternator, and a car battery (to be fair I question similar claims when the devices uses the mains voltage -- you know, when a 110 or 220 V device in the home claims to produce 0.5 kW but uses... 50 W from the mains to do so).