Look on the back of your speakers and determine their Ohm rating. If they are 8 Ohm speakers and you aren't tripping the amp's thermal protection circuit causing it to shut down from time to time, you're probably fine.

If the speakers are 4 Ohm and you are running them at medium to loud levels and your amp is tripping the thermal protection circuit then you are asking for trouble.

If you have the amp manual or can see the back of it where the speakers are connected, it should say what Ohm rating the speakers should be to run both pairs at the same time.

What kind of speakers and what kind of amp do you have?

will help one make a determination.