Hsu Research is selling a system that may be what you're looking for. You might want to check their website. They started by making sub woofers that are well regarded.

You aren't the first person to be underwhelmed by Bose. That being said, are you sure it has been set up correctly? The reason I ask is that I've heard Bose systems that sounded pretty decent (not amazing, not audiophile, but decent) and others that sounded horrible. I assume that most of that has to do with the setup. The other reason I ask is that for $600, you may have a hard time doing much better than a Bose system. For a pair of decent speakers (a two channel system) there are plenty options out there to try for $600. But for a complete home theater setup, $600 doesn't go so far.
So, I would recommend that you first make sure everything is set up ok on your current Bose system. No use in throwing money at the problem if all you need is to set it up better. If you're sure it is set up correctly, but bose just doesn't do the job for you (and rest assured, you are not alone--not even close) I would think about upping the spending a bit.
I'm partial to Paradigm speakers myself.
I've listened to a lot of them and I feel like on the "best-bang-for-the-buck" scale, they rank pretty high.
They have a home theater in a box (HTIB) for around $800 (the Cinema CT series) that I've heard in store and thought it sounded great. I think they may have one HTIB that is even cheaper, but I haven't heard it, so I can't recommend it. But as I said, Paradigm has a good reputation so it may be what you are looking for.
Alas, all smaller speakers eventually come up against the boundaries of physics and you can only get so much quality sound from a small speaker. That being said, if the room it's going in is small to moderate size, a smaller system like the Paradigm or possibly even the Bose may do the job.
Good luck, I hope this helped some

Sub

Center

Front

Rear

I don't see how you can avoid going the used route and these speakers should keep you in budget and give you incredible sound.

Four Infinity Primus 150 monitors - $150-200
One Infinity Primus C25 center channel - $85-125
One Infinity PS-10 subwoofer - $200-250
Total - $435-575

These will make your Yamaha HTR-5850 sing!

http://www.stereophile.com/standloudspeakers/404infinity/

I bought last years NHT SB-2 speakers from listen-up here in colorado I think I paid 235 a pair and the center was $150. That would fit close to your budget, I think they still have them on their website and I got free shipping. Of course this does not include a SUb though; bummer.