MP3 vs AAC vs FLAC vs CD Page 2

For reference, fig.3 shows the spectrum of the signal on the CD. Other than the well-defined green vertical lines representing the tones and the uniform background noise, the spectrum is clean. Important points to note with this graph are that 1) all musical fundamentals lie to the left of the 4000Hz (4kHz) mark; 2) the region between the next three divisions, 4kHz, 8kHz and 16kHz, is where musical harmonics and the "air" on a recording reside; and 3) the region above 16kHz—more than a quarter of the horizontal scale—will be inaudible to most adults.

Fig.3 Spectrum of 500Hz-spaced multitone signal at –10dBFS, 16-bit linear PCM encoding (linear frequency scale, 10dB/vertical div.).

Fig.4 shows the spectrum of this demanding signal as preserved by lossless coding, in this case the popular FLAC codec (at its slowest "8" setting). To all intents and purposes, it is identical to the spectrum of the original CD. The lossless coding is indeed lossless, which I confirmed by turning the FLAC file back to WAV (LPCM) and doing a bit-for-bit comparison with the signal used to generate fig.3. The bits were the same—the music will also be the same!

Fig.4 Spectrum of 500Hz-spaced multitone signal at –10dBFS, FLAC encoding (linear frequency scale, 10dB/vertical div.).

How did the MP3 codec running at 128kbps cope with the multitone signal? The result is shown in fig.5. The dark red vertical lines represent the tones, and none are missing; the codec has preserved them all, even those at the top of the spectrum that will be inaudible to almost every listener. But the background noise components, which on the CD all lay at around –132dB, have all risen to the –85dB level. With its limited bit budget, the codec can't encode the tones without reducing to almost half the 16 bits of CD resolution. Even with the masking of this noise in the presence of the tones implied by psychoacoustic theory, this degradation most certainly will be audible on music. Yes, this kind of signal is very much a worst case, but this result is not "CD quality."

Fig.5 Spectrum of 500Hz-spaced multitone signal at –10dBFS, MP3 encoding at 128kbps (linear frequency scale, 10dB/vertical div.).

How about other lossy codecs? I looked at how the iTunes AAC codec (a version of MPEG 4, a later development than MP3) performed on this test, running at the same 128kbps. The result is shown in fig.6. At first it looks very similar, to fig.5, but there are significant differences. Note that almost all the tones above 18kHz are missing and that those above 16kHz are increasingly rolled off. The designers of the codec obviously decided not to waste the limited bit budget by encoding information that would most probably not be heard even from the CD. Instead, they devoted those resources to a more accurate depiction of the musically significant regions at lower frequencies. You can see in this graph that, below 4kHz, the noise level is 10–20dB lower than with the MP3 codec (though perhaps more "granular"). In effect, in the frequency region that is musically most important, an AAC file with this test signal has 2–3 bits more resolution than an MP3 file with the same bit rate. The AAC noise floor is higher than the MP3 noise floor between 8kHz and 18kHz, but given the physics of human hearing, this is insignificant.

Fig.6 Spectrum of 500Hz-spaced multitone signal at –10dBFS, AAC encoding at 128kbps (linear frequency scale, 10dB/vertical div.).

The degradation is dependent on bit rate—the higher the bit rate, the bigger the bit budget the codec has to play with and the fewer data must be discarded. I therefore repeated these tests with both lossy codecs set to 320kbps. The file size is three times that at 128kbps, though still significantly smaller than a lossless version, but are we any closer to "CD quality"?

Fig.7 shows the spectrum produced by the MP3 encoder running at 320kbps. (This is the format used by Deutsche Grammophon for its classical downloads.) Again, all the tones are reproduced correctly, and the noise has dropped by around 6dB or so at higher frequencies and up to 15dB at lower frequencies. But it is still not quite as low as AAC at 128kbps below 1kHz or so.

Fig.7 Spectrum of 500Hz-spaced multitone signal at –10dBFS, MP3 encoding at 320kbps (linear frequency scale, 10dB/vertical div.).

AAC at 320kbps now encodes all the tones, even the inaudible ones at the top of the audioband (fig.8). The noise floor is quite high above 18kHz, but—and it's a big "but"—the noise-floor components have dropped to below –110dB below 16kHz, and to below –120dB for the lower frequencies. Though some spectral spreading can be seen at the bases of the vertical lines representing the tones, it is relatively mild. Given the bigger bit budget at 320kbps, the AAC codec produces a result that may well be indistinguishable from CD for some listeners some of the time with some music. But the spectrum in fig.8 is still not as pristinely clean as that of the original CD in fig.3.

Fig.8 Spectrum of 500Hz-spaced multitone signal at –10dBFS, AAC encoding at 320kbps (linear frequency scale, 10dB/vertical div.).

For my final series of tests, I used Test CD 3's track 26, which replaces some of the tones in track 25 with silence. The spectrum of the CD original is shown in fig.9. You can see clean vertical lines representing the tones, with silence in between. You can see the random background noise below –130dB, as expected. Also as expected, encoding with FLAC gave the identical spectrum, so I haven't shown it.

Fig.9 Spectrum of multitone signal with frequency gaps at –10dBFS, 16-bit linear PCM encoding (linear frequency scale, 10dB/vertical div.).

MP3 at 320kbps gave the spectrum shown in fig.10. All the tones are present, but if you look closely, you can see some extra ones, at low levels. The noise also leaks into the spaces between the groups of tones. AAC at 320kbps gave the spectrum in fig.11. Again, there is much more noise and less resolution above 18kHz, where it doesn't really matter. Again, the noise around the groups of tones is lower than with MP3 at the same bit rate. Some low-level spurious tones can be seen in the spaces between the groups of tones; though there are more than with MP3, these are all lower in level. The noise floor between the groups is also higher in level than with MP3, but is still low in absolute terms.

Fig.10 Spectrum of multitone signal with frequency gaps at –10dBFS, MP3 encoding at 320kbps (linear frequency scale, 10dB/vertical div.).

Fig.11 Spectrum of multitone signal with frequency gaps at –10dBFS, AAC encoding at 320kbps (linear frequency scale, 10dB/vertical div.).

What does all this mean?
Basically, if you want true CD quality from the files on your iPod or music server, you must use WAV or AIF encoding or FLAC, ALC, or WMA Lossless. Both MP3 and AAC introduce fairly large changes in the measured spectra, even at the highest rate of 320kbps. There seems little point in spending large sums of money on superbly specified audio equipment if you are going to play sonically compromised, lossy-compressed music on it.

It is true that there are better-performing MP3 codecs than the basic Fraunhöfer—many audiophiles recommend the LAME encoder—but the AAC codec used by iTunes has better resolution than MP3 at the same bit rate (if a little noisier at the top of the audioband). If you want the maximum number of files on your iPod, therefore, you take less of a quality hit if you use AAC encoding than if you use MP3. But "CD quality"? Yeah, right!

ARTICLE CONTENTS
Share | |
COMMENTS
da5id's picture

I would expect to find reputable audiophile discussions to be well-thought out and current with the technology at issue. This article fails to meet those expectations. Its conclusions are stated succinctly:

"If you want the maximum number of files on your iPod, therefore, you take less of a quality hit if you use AAC encoding than if you use MP3. But "CD quality"? Yeah, right! "

The hard drive based Classic IPod is still sold. The Classic IPod is, not unexpectedly, smaller and technologically superior to the original, including a proprietary Apple lossless format. It "holds" 260 GB. Your strawman has 4 GB. Almost no one makes a decent MP3 player with only 4 GB. In any event, all of the solid-state ones manufactured reputable companies go up to 64 GB. All of these players support FLAC, either natively or with an app.

Now let us examine the author's numbers applied to existing technology. We'll use the worst-case scenario for FLAC, 60%, on an 800 MB CD. That equals 480 MB per CD and we'll round that up to 500 MB for easy calculations going forward. Those numbers would yield 128 CDs on a run-of-the-mill 64 GB flash drive based MP3 player and 520 CDs for the Classic iPod. And they can be swapped out every night, all ready for the next morning, with whatever you want. I'm sure there is an app for that.

I read an interesting article in Slate by an audiophile not bemoaning the easy acceptance of inferior MP3s, but rather, the general lack of appreciation of live music and the failure of, even fetish, of audiophiles over things like $1600 phono cartridges at the expense of that music.

– Gene Girard

PS: You can find all the technical detail you want, far exceeding your own, in the Wikipedia articles on the topics of FLAC and Ogg Vorbis. FLAC is not the only, let alone optimal, bit-for-bit lossless compression technology.

 

fergusof's picture

Thanks for the great article. It explains a lot about various audio formats that I didn't know.

Apparently, someone named da5id 'expects' to find 'reputable' audiophile discussions to be 'well-thought out' and 'current.' Perhaps da5id should note that the article is only current to 2008. Even so, perhaps da5id could give credit where credit is due and thank the author for spending much time and effort on this article instead of doing a a 'drive-by' flaming. Da5id, it seems, doesn't understand basic politeness and seems to feel that his 'expert' knowledge allows him to insult anyone whose knowledge, to him at least, doesn't compare to his. If you disagree with anything in Mr. Atkinson's article, then, by all means, initiate a polite discourse. But don't act like a know-it-all loudmouth. A loud bronx cheer to Mr. da5id.

Archimago's picture

Anyone interested in participating in an online survey of high bitrate MP3 vs. the original CD audio, come check out:

http://archimago.blogspot.com

Instructions and samples to download and listen to!

Take it to the next level beyond reading about it and put your ears, brain, equipment to the test... Taking survey submissions until the end of January 2013.

Archimago's picture

Happy New Year everyone!

A reminder with plenty of time to go before closing the data collection.

--- Originally posted on AudioAsylum ---
Since opening the high bitrate MP3 vs. CD test on Dec. 11th, I have received 41 responses so far to the detailed survey. I will not analyze the data until the end of January, but just eyeballing the spread of results (Set A, B, "same") is already quite interesting. The respondents have come from 4 continents so far and reading some of the comments, I really appreciate the time people have put into this!

Furthermore, it's great to see a nice spread of equipment used from inexpensive (but good) headphone gear all the way to megabuck $50K+ systems.

Although we "shoot the breeze" around here and have great discussions around the hardware (sometimes inflaming arguments), it is infrequent that we actually do something like this where we stand up and be counted based on the actual experience of listening. I know that this isn't strictly "scientific" and many variables cannot be controlled in an open test like this, but for us "non-pro's", this could be the closest we get to participating in something which I hope is educational and (hopefully) fun as a hobbyist beyond theoretical discussions.

If you haven't given this a try, I encourage you to take some time and give it a shot. Be involved in a simple "blind test" (perhaps the only time in one's life) knowing you've tried something like this and contributed to the data set (whether one believes it's significant or not).

Thanks again AbeC. for hosting the fast link! Much appreciated, bro.

PS: One request - could some of you who participate in audiophile discussions in Asia (India, China, Japan, Korea, Malaysia, Singapore, Indonesia, Philippines, etc...) spread the test around. Would love to get some data from those folks!

Get the test here:

http://archimago.blogspot.com

Archimago's picture

Test complete with 151 respondents! Results up on the blog...  I think many would find the results surprising.

KUppiano's picture

This is an interesting article, and the tests are worthy of consideration, but we have to consider why someone would use MP3 or AAC over FLAC or raw PCM to store their music. Certainly, many users have space constraints, but they still want to listen to their tunes on their desktop, laptop or portable machine. 30 years ago, they would have used a cassette machine, such as a Sony Walkman. 

Fast forward to MP3. Is there a perceptible difference from the original sound? Sometimes, if the bit rate is low enough, and/or if the listening environment or equipment is good enough. But once you get above 128 Kbps, those differences become quite insignificant. Not for all program material, not for all listeners, and all circumstances of course. But in general. I found cassettes to be virtually unlistenable, but at reasonable bit rates, I can usually enjoy listening to MP3 and AAC.

I would suggest that you run the same tests that you made with MP3s on cassette tape, and compare those results to MP3 or AAC. I think the digital formats even with all their faults, would win hands down. Wow, flutter, and frequency response -- as well as noise and distortion, would be much worse on cassette tape. Perspective, perspective, perspective.

You might also consider that, although the charts in your article look dramatically different, showing obvious disturbances in the force, the perceptual coders are just that: perceptual. You should expect to see differences when information is discarded. That's a given, and the charts will reflect that. The researchers who developed the algorithms worked very hard to minimize the perceptual trade-off. They did pretty well with MP3, and got much better with AAC. 

Ultimately, the question is, does it matter perceptually? And the answer is, "it depends". It depends on the bit rate, it depends on the perceptual model, it depends on the algorithm, it depends on the source material, it depends on the listening environment and the equipment, and most importantly perhaps, it depends on the listener - his tolerance for whatever the distortion is, and his skill at recognizing it.

I made a slightly different study, that I published on my blog. I subtracted the MP3 from the original sound in WAV format. The result was the difference between the original sound and the bit-rate-reduced version. It was quite interesting, and maybe a different way to look at this issue. 

Finally, with all types of storage prices dropping rapidly, I wonder how much longer perceptual coders such as MP3 or AAC will even matter (they might for downloading or streaming over the Internet, but even there, bandwidth is increasing too). We may continue to use perceptual coders, but if we can run them at 256 Kbps or even 512 Kbps or greater, will the losses even matter?

garyding2003's picture

I did an interesting test. The same song in 3 formats of ape, flac and mp3 of 320bit are played on my computer (gigabyte motherboard / realtek 889 onboard audio), itouch 4 and Sony Dej011 portable player. My headphone is Creative Aurvana Live.

The CD player has the best sound quality, warm, very clear and full of details; The second is Itouch 4, clear and full details but dry; The last one is my computer, dry, lack of details and a little distortion of music even when playing ape or flac format.

supra-mp3's picture

What I do is I remaster the original or not too bad files before I put it into a 320Kbps file that sound excellent in an Ipod or my car or even in my home sytem with great results . In some cases its better than it was before.I believe that format is the way to go . It gives excellent quality and the files are not too large.

Liam McMahon's picture

What's galling and upsetting about iTunes (and to a less extent Amazon) having created the market standard audio formats (mp4 and mp3 respectively) is that for, I would suggest, a majority of today's (younger) music buyers and listeners, they do not know the difference between the respective sound qualities of a CD and an mp3/mp4 (AAC).  And iTunes does very little if anything to educate people.  It is almost a pervasive urban myth that iTunes files are simply automatically CD-quality...and this is far from the case!

The galling and upsetting and dangerous things this does is:

1. People become used to inferior sound recordings, and do not even know about the concept of "hi-fi".  Surely this must over time impact on the quality of music produced, as what is heard is inferior, what gets produced must presumably also suffer? 

2. For the prices that iTunes charges, one would SURELY expect CD quality music! If an album is, say, $US10, how in the world can they sell vastly inferior quality files?  A physical CD is the same price or not much more (around $13.50 on Amazon for a new CD).  How can iTunes sell a tenth of the file information for virtually the same price?

Finally of all, why in the world can iTunes not offer the option of lossless files? Even if you end up paying MORE than you would for a CD (which is patently an absurd situation) - heck, even if you pay a lot more - there is absolutely no doubt that professional DJs, discerning hi-fi listeners, and audiophiles in general would gobble up those lossless files, even for a premium.

 

The whole entire situation is appalling. The fact is that for online music retailing - which has long since far outstripped its physical counterpart - there is quite simply NO MERCHANT OFFERING A WIDE RANGE OF CD QUALITY MUSIC!  Only at specialist dance music/DJ-orientated online retailers do you tend to have the option of WAV or other lossless (as far as I know - if there is a mass market retailer akin to iTunes who offer lossless files, somebody please tell me about them!) This is a sickening, dreadul, unthinkable situation, utterly perplexing, bizarre and crazy.

cd.vs.mp3's picture

In case you are interested, I have devised an online blind test CD vs AAC 256k. It is really a humbling experience for those who think compressed formats are not good enough. http://cdvsmp3.wordpress.com. Try it. I get good feedback from people who take it! There are also some interesting posts on HD audio using null testing.

X
Enter your Stereophile.com username.
Enter the password that accompanies your username.
Loading