A New Angle to an Old Discussion

If you're new to hi-fi, you might not be aware of the fiercely controversial and seemingly endless argument regarding the importance of blind listening in determining a component's worth. Essentially, there are those who believe that if differences in sound cannot be perceived while listening blindfolded, or under similarly exhausting, stressful, and inhumane conditions, then differences do not exist, and we're all just full of shit when we tell you that the $30,000 Musical Fidelity Titan sounds nothing at all like the $159 Sonic Impact Super T, that, in fact, they're basically the same amp, so you might as well buy the cheaper one.

Of course, I'm exaggerating. You see where I stand on the subject, and I hope you get the point. Proponents for blind listening will argue that the only way to determine whether sonic differences between two components exist is to escape our many biases, is to use a carefully controlled listening environment in which the listener does not know what he or she is listening to. Imagined differences should then vanish, leaving only the "real" differences, or those that can be correlated with objective measurements.

Fascinating, right?

But, if I learned anything in school, I learned how to take a test. And I learned that some tests are just impossible to pass.

Back in 1989, when I was getting straight As in the 7th grade, John Atkinson and Will Hammond conducted a blind listening test at Stereophile's old High End Hi-Fi Show. The result?

In short, the data from the Stereophile blind listening test are most likely to be due to biased guessing rather than audible differences between amplifiers.

Aww, bummer!

Anyhoo, this came in today:

A New Angle to an Old Discussion: AB and ABX Tests

Dear International Colleagues,

I believe you all are aware of the long standing dispute between the subjectivists and objectivists in Audio. I believe nothing new has been learned in these discussions for the last 30 years or so and they basically come down to the inability of people who hear sonic differences to prove this in an ABX-Test which is then often considered to be proof of the non-existence of sonic differences.

We are putting a new spin onto this by resorting to a different field. People who work with graphics have often seen colour charts that show minimal differences that are only apparent in comparison to the surrounding colours. If one was to identify an individual colour without the surrounding ones for comparison that would be mostly guess work.

In other words: Even though there is a subjective difference to the observer's senses and an objective difference as the colours really are mixed differently that does mean not it will always show in an ABX-Test. Thereby the claim that not passing an ABX-Test means that there is no real difference is void.

We have integrated this concept into a small flash program you can access at www.sieveking-sound.de/abx which will do a little test with you where you are asked to first make sure that you do see the difference between two colours and then have to identify them later on. I yet have to score perfect and more often than not my results would let some people claim that this was proof that there is no difference in the colours. Alas—at the end of the test we actually show people the colours again and also the colour-channel difference between them.

Please try this out and tell other people about it. I hope this has the ability to help heal the gap that has separated the different schools of hi-fi enthusiasts and reunite them in the enjoyment of great sounding music.

With best regards from Germany,
Jan Sieveking

Nice, right? I am all for bonding with my brothers in the enjoyment of great sound, so I took the test. It was fun! But let me tell you: I was pretty much clueless, feeling around in the dark. I could barely see the difference between the two colors in the first place. In fact, I'm not sure that there even was a difference. I was just playing along. At one point, I clicked entirely by accident when the banana I was eating fell on my mouse. I wonder if my banana got that one right. I would have to guess that my banana had about as good a chance as I had. I scored a 9 out of 20, which the computer tells me is worse than what I would have scored had I simply guessed.

Well, there you go. Give it a shot, and post your score, if you dare!

Al Marcy's picture

The reason old arguments never get resolved is people are assholes."Broken Hearts Are For Asshole" - FrkZpaGet over it ;)

Jan Sieveking's picture

Dear Stephen,did you realize that you can actually increase the colour difference on the starting page to make sure that you really see a difference? You can go up to a colourchannel difference of 10 after which you get a new pair of colours which is generated randomly as you might not see the difference because you are red/green or blue/yellow colour blind.The bigger the difference the more likely you are to score better but then you are actually supposed to start the test once you are sure you do see a difference.Have fun,Jan

Stephen Mejias's picture

Dear Stephen, did you realize that you can actually increase the colour difference on the starting page to make sure that you really see a difference?Hi, Jan. Yes, I increased the heck out of that color difference! My trouble may have more to do with my screen, or my apparent blindness. Thanks for the fun test!

Butch Blawd's picture

That was painfull! Got 13 and thought that was good training and had another go. Got 12 the second time. Guess I will just go and injoy the music.

Jon Iverson's picture

The only thing blind listening testing tests is the listener and their ability.You can rigorously train for a blind test and beat someone who does not. So what does that prove about two amps? Nothing. This is because audiophiles mistake blind tests used in medicine, where the body cannot develop a skill to skew the results, to those in audio where skill can trump other attributes.http://www.stereophile.com/asweseeit/805awsi/index.html

Rob's picture

Much as I enjoy Stereophile, there are two sorts of people who have the sort of attitude towards double blind testing as you "golden ears"You, and so called psychics.

Larry's picture

Stephen,My concern has always been with the inflated hyperbole that some writers use/used in subjective reviews. If a component really is "night and day", "immediately obvious", "vastly superior", "my wife heard differences while cooking in the kitchen", etc...then it seems to me it should be clearly different, on immediate hearing, under blind testing conditions.It appears that writers have toned down over the years some of this rhetoric precisely because this usually is not the case. If we want to talk about subtle shadings, amp-speaker interactions, and so on, I agree that there are differences. But as this test shows, that you can make such a tiny difference in 2 things, does not in my mind "invalidate" double blind testing.

Stephen Mejias's picture

Okay, I tried again, this time on my Mac, using the gray-blue squares and maximizing the color difference. I scored a 17/20.

Tom B.'s picture

Anyone who believes that this says anything at all about the validity of the assumptions underlying blind or double-blind testing, has been duped. This is a cheap parlor trick, and nothing more. For this to make any sense, the premise being tested by the experiment would need to be that visual memory has specific limits. It is manifest that to conduct this experiment, it would make no sense for you to be told which of the two images you are looking at, and then be asked which one you are looking at. The inability to determine which of the two original images that you are seeing, under blind conditions, does not imply a problem with blind testing, and it is preposterous to suggest otherwise. This is one of the stupidest things I have ever seen. I feel sorry for anyone who thinks that this demonstrates anything at all about the validity of blind testing.I have read any number of arguments against blind testing over the past several decades, and without exception, they amount to bogus reasoning.

James Kontol's picture

If sonics, and sonics only is to be evaluated, the equipments must not be seen. JK.

David Robinson's picture

So listening without knowing the equipment's identity is "exhausting, stressful, and inhumane"?Or is this the case only when you're asked to _review_ the equipment?It seems to me that it will reproduce music equally well whether you know what it is, or not.I suppose the stress comes from the possibility that your golden ears may have tarnished.

stephan's picture

The flash script is not an ABX test. In an ABX test you are allowed to evaluate the known A and B against an unknown X. What the script tests is just your visual memory.

DVDdoug's picture

"... there are those who believe that if differences in sound cannot be perceived while listening blindfolded, or under similarly exhausting, stressful, and inhumane conditions, then differences do not exist.." As I understand it, that's NOT what ABX proponents claim. They claim that you have either "FAILED TO PROVE" there IS an audible difference, or "PROVEN THAT THERE IS" a difference. "..that are only apparent in comparison to the surrounding colours. If one was to identify an individual colour without the surrounding ones for comparison that would be mostly guess work." The surrounding colors make it very much like an ABX test... You are allowed to compare X (unknown) to A & B (known)."...when we tell you that the $30,000 Musical Fidelity Titan sounds nothing at all like the $159 Sonic Impact Super T."Well... if you can't hear a difference in a blind test, I'd say you're wasting money by buying the Titan. And, if

Skeptic's picture

Joe Consumer reads about some wonderful new speaker cables. He buys some to replace the wire he got from the hardware store. He doesn't hear any difference. Later, he finds out the reviewer could ONLY hear a difference when he KNEW which cable he was listening to...

AlexO's picture

Even if we were to assume that the test is valid and that most people can't reliably and consistently see a difference between the two colors and that this somehow relates to audio, then the question would be: Does it matter? Let's say there is a difference between the 30k amp and the 200 dollar amp, but the difference, if there is one, is so subtle that most people can't reliably and consistently hear it, then that difference doesn't matter and certainly doesn't justify the price delta.

BC's picture

ABX tests are intresting events. There are those audiophiles who are cured and won't ever overspend again on ultimately stupid things like expensive interconnects. Not ever, not once, ever, has even a single person been able to hear a difference in interconnects, except when they KNOW it's there. In fact, if you believe you can, you should drop Randi a line. Then there are those really die hard audiophiles who'll go out of their way to disprove the validity of ABX testing. This test can't be right, I can hear the differences easily! Yeah, right, easily, except when you can't see the freaking thing. Emperor's new clothes anyone?

hifisound's picture

If I am not wrong, this test points out that one can't compare A after B or B after A but needs to compare them together. In audio world this would translate to : one can't compare an audio components by listening to it one after another (though I can't think of hearing and comparing them together ).But this problem exists even when you are seeing/knowing the components. So blind testing is just to just eliminate any bias which may occur by seeing the component's brands, size etc . So even when you are hearing one after another, it should not be so that you hear differences when you see/know the components and don't hear them when you don't see/know the components.Though if no differences are heard from blind testing the result be concluded as inconclusive instead of saying both components sound same, which itself could be good enough reason to buy the cheaper of the two :) (not considering other factors like build quality, reliability, warranty, etc here)

Chris's picture

Its not worth the time to argue if the subjective vs. objective listening test debate will ever be answered because someone can always create a reason why the tests are flawed. The solution is to do both--first the subjective to formulate an opinion on the performance of the source then do the objective to see if the opinion holds. A week or so later do the opposite--objective first, subjective second--and print the results. I have a simple test to determine if interconnects and all sorts of additional paraphernalia are worth the money--if your favorite add-on is not in the box with your favorite amp, preamp, CD player, or turntable then you don't need it. The logic is simple--the manufacturer is suppose to be making equipment that strives to be the best it can be in sound reproduction so this manufacturer should care about everything that will make this happen. Spending $15k on a set of amps and then buying $30 worth of wooden blocks doesn't make a lot of sense.

sweetsounds's picture

I took the test 3 times with color delta of 10 and scored 18/20 (green), 19/20 (grey), 17/20 (pink). I guess, I can identify differences.Still, a delta of 10 is only a nuance. Transferring to the audio world, this nuance corresponds to a difference - say - of a B&W 801 to a B&W 800. Hearable, but without concentration difficult to find.In no way this corresponds to the large difference of an electrostatic speaker to a horn.I have yet to experience myself dramatic differences of amplifiers and cables. I already fell into the trap, when I raved about the sound of some new speaker cables in a home audio hearing, and found out, that my friend had not even connected them :-)My conclusion: spend all you can on good speakers, then on a good CD player. Rest is less important.