Start with the loudspeakers. I'd suggest Vandersteen 2Ce Signatures. $2000 Best sounding speakers I've found for $2000 and under.
The Outlaw RR2150 Stereo Receiver has ample clean power and a USB port for streaming audio from a computer. $650
Turntable for your vinyl? The Rega P1 is very good with an upgrade to a Ortofon 30 stylus. $520
A good CD player is the Cambridge Audio Azur 640C. $600

I can't recommend any particular components because I'm rather new to all this too, but I will say from recent experience that finding a good audio store where you can listen to different setups in your price range will help you narrow down your choices. Not everything in the same approximate price range sounds the same.

Then when you do that, you might decide to defer the purchase of certain components of your system in order to buy higher quality in the ones that matter most to you.

For example, if you have a large collection of MP3's and that is mainly what you are listening too, then defer the purchase of the CD player for now and put that money into better speakers. If you are going to just rip your vinyl but don't intend to use the TT for playback and listening, consider buying a used but higher quality TT, rip your vinyl with it, then sell it when you're done. Maybe check audio stores to see if you can get a good price on a demo unit.

Be forewarned. If you go to an audio store (a store that specializes in stereo audio, not a generic consumer electronics store) and listen to different amps and speaker combos, you may find yourself preferring to concentrate most of your initial budget on speakers.

Well, first of all, don't bother with MP3s. They're really too low-fidelity to justify good equipment. Don't get me wrong-I have a lot of MP3s too, but they're not why I invest so much money in this hobby (And, believe me, I'm a college student and have to live with a tight budget and long saving periods. I don't have a pricey system, nor do I believe that it is difficult to assemble a very satisfying one for $3-4k).

Also, if you have a ton of vinyl, listen to it!
Unless you'll be ripping purely for backup, I think you'd get a lot more enjoyment out of a good turntable than a bunch of noisy digital files.

If you plan to import more music to your computer, be sure to tell your program to copy the data in an uncompressed or losslessly compresses format.

I'll build you a hypothetical system that falls in your price range, has tons of flexibility, and sounds awesome. If you write a little bit about your musical tastes, listening room, and desired/essential features, I (and others) can hone in on gear that would suit you perhaps even better.

-Outlaw Audio RR2150 "Retro Receiver" ($650, sold direct at outlawaudio.com)

This baby has too many features to list, but you should definitely go to the site and check it out. serious boat-floating potential Note: I have not auditioned the RR2150, and recommend it based on reviews, "word on the street", and the extensive features list.

-Rega P3-24 turntable with Elys 2 cartridge ($1045)

This 'table is so good for the money, it's scary. It walks all over anything below it, and can frighten much more expensive rigs, as well. I know you didn't mention that you wanted a turntable, but take some records to your nearest Rega dealer, and you might change your mind.

-Oppo DV-980H universal DVD player ($170)

This box will play pretty much any disk you stick in it, and do so astonishingly well. For the somewhat-less-relevant-but-still-around CDs that you have, it's a solid choice.

-Logitech (formerly Slim Devices) Squeezebox streaming audio player ($300)

Connects to your home network and allows you to use your computer to beam just about any audio file to your stereo. It also features internet radio.

-Paradigm Reference Studio/60 v.4 ($1900)

Whatever you listen to, these are fantastic speakers. The rest of the Paradigm lineup is also excellent.

Total price: $4065

This is just one suggestion. Have fun and start listening to gear and figuring out what's important to you! If you do it right, it's hard to mess up in this price range.

As far as what you're paying for, that's primarily (a) better parts and (b) careful design. You're also paying a premium because "economies of scale" tends to be a relative term in the industry. None of these companies are moving product like Sony, Yamaha, etc.

Buy stuff used on Audiogon. Your 4k will go much further.