The Highs & Lows of Double-Blind Testing Page 5
The double-blind method may have its own problems, however, ranging from listener impatience with large-trial studies to difficulties in obtaining adequate switching equipment. I would guess that the double-blind method will require a good deal of tinkering and refinement before it flowers into the blessing many believe it to be. And, perhaps, practical problems with the method may be too great to overcome. But I believe the method is worth a concerted, open-minded try.—Les Leventhal
The blind-testing establishment responded in the form of a letter from one of the inventors of the ABX box, David Clark, which appeared in Stereophile , Vol.9 No.5, August 1986, accompanied by responses by Les Leventhal, and John Atkinson.
The Double-Blind & the Not-So-Blind
Ladies and Gentlemen! Step right up! See the Academic take on the might of the Engineering Establishment and jump through hoops of real fire! See the Engineering Establishment use "real science" to support its entrenched position!
In Round One—"The Highs & Lows of Double-Blind Testing", (Stereophile, Vol.9 No.2)—Les Leventhal, of the University of Manitoba's Psychology Department, Winnipeg, Canada, put forward a hypothesis concerning the double-blind listening test widely used by "objective" equipment reviewers. He indicated that the statistics commonly used to analyze data from these tests had a built-in bias to produce null results (ie, real differences between amplifiers, say, would not be detected), unless the number of trials were increased significantly above that normally used.
As a reader of hi-fi magazines, you will undoubtedly be aware that such null results are widely used to promote the idea that "subjective" reviewers, who have found that amplifiers, for example, sound different to one another, are, at best, incompetent or, at worst, dishonest in their judgments. Who can forget the Stereo Review speaker cable test? Or the tests in England that showed that one Quad amplifier sounds pretty much like any other?
Well, Ladies and Gentlemen, here we present Round Two—and the first combatant to enter the arena is David Clark, of DLC Design, a widely respected engineer who played a major role in the development of the ABX comparator, a most useful piece of equipment for those investigating the subjective/objective frontiers.—John Atkinson
David Clark opens with an attack on Leventhal's ideas
Les Leventhal's critique of the statistical analysis commonly used in blind subjective testing is misleading, erroneous, and borders on the incompetent. His letter is written in a style that prompts the casual reader to think "Someone has finally figured out what's wrong with all those blind tests where they don't hear anything." Not only has Leventhal failed to prove his case; he has demonstrated his own lack of understanding of how the audiophile benefits from double-blind testing.
Leventhal's first ploy is to state that we blind and double-blind testers are attempting to prove something which we, in fact, are not. He says we may erroneously conclude that no difference is audible in a particular test. He is mistaken. We never make that error (he calls it "Type 2 error"), because we never formally conclude that any difference is inaudible. We may make some informal statements of our opinions or we may issue a challenge to others to prove that they can hear the difference, but this is a far cry from making Leventhal's "Type 2 error."