A speaker designer should take into account the various values of inductance and so forth that occur as a driver's motor warms with use. These values will sum with the crossover eventually altering the effect of the crossover's components slightly and can move the actual LP and HP frequency. However these are not absolutely static numbers but rather values that change with temperature and the amount of change to the overall values of the crossover should be rather small in a better speaker design. I've never heard anyone speak of warming up their speakers in the manner you describe. I would be interested in how you know it is the speakers themself that are "warming up" in this case and not just the amplifier.

I could see that the capacitors (and their dielectric) in the speaker's crossover might benefit from signal passing through, especially if the system is going from being completely off to on.

I tend to think it's the wire in the voice coils.

Your problem is a new one for me. Never encountered it over more than forty years and a number of different speakers. What speakers are we talking about?

All new speakers, as I suspect you know, benefit from "break-in". Some take longer than others, but all I've ever encountered improve over time. Once the break-in period is over, however, unlike amps., they "wake up" ready to go when you give them some music. If the distortion you hear is a severe as you say and if it goes away after a while, my guess is that the source is somewhere in the electronics and not in the speakers. Temperature change in the voice coils and cross-over might produce some subtle change, but nothing so severe as what you've described.