Blind Tests & Bus Stops

On mornings when I can get up early enough after a late-night listening session, I take the last express bus from my Brooklyn suburb to Stereophile's Manhattan office. An inveterate people watcher, I notice that while my fellow travelers and I don't form a traditional queue at the bus stop, preferring instead to mill around in something that resembles a jelly donut, we still enter the bus in the order in which we arrived at the stop. The balance between individualism and social necessity is thus preserved.

At the other end of the line that separates social sophistication from mob behavior are the free-for-alls that develop on the Internet newsgroups and Web forums. Freed from the need for personal responsibility and the usual social rules of personal interaction, many individuals write anything and everything to attack those they perceive as the "enemy."

Stereophile appears to have become a lightning rod for these disaffected souls, one of whom, Arnold (Arny) B. Krueger, of audio-review website, has been criticizing this magazine weekly, if not daily, for almost eight years. Feeling it was high time Mr. Krueger came out from behind his PC to confront in person those he criticized, I invited him to debate me at Home Entertainment 2005, held at the end of April at the Manhattan Hilton.

A report on the debate and an MP3 recording of it can be found on our website. As you can hear, Mr. Krueger's opening argument ended with his stating that he had three major criticisms of this magazine: 1) "Stereophile willfully ignores much that is known about reliably evaluating audio products"; 2) "Stereophile frequently reaches conclusions and makes recommendations that are improbable if not just completely wrong"; and 3) "Stereophile does not take enough pains to ensure that it is publishing correct information."

However, as you can also hear, these assertions were not supported or fleshed out. I assumed, therefore, that Mr. Krueger was basing them on the one criticism he has repeatedly made over the years that is correct: to wit, that Stereophile's reviewers do not perform their listening evaluations under blind conditions. Instead, they know what it is they are listening to. This, of course, is as J. Gordon Holt envisaged it when he founded this magazine 43 years ago: that the optimal way to judge a component's performance is to use it for its intended purpose—to listen to it.

As I explained at the debate, I didn't always hold this view. In fact, when I first joined English magazine Hi-Fi News in 1976, I was as hard-line an "objectivist" as Arny Krueger, due both to the arrogance of youth and to the fact that I had trained as a scientist, working for some years in government research labs in the late 1960s and early 1970s. I believed—no, I knew that amplifiers operated short of clipping did not sound different from one another. An editorial I wrote in the April 1979 HFN illustrates the man I was back then: "The result of a subjective test can only be regarded as valid if all potentially misleading variables have been removed, but this is very often not the case," I thundered.

In the summer of 1978 I took part in a blind listening test organized by Martin Colloms, in which the panel tried to distinguish by ear between two solid-state power amplifiers—a Quad 405 and a Naim NAP250—and a tube amp, a Michaelson & Austin TVA-1. The results of the test were inconclusive, the listeners apparently not being able to distinguish between the amplifiers (see HFN, November 1978). Having been involved in the tests, having seen how carefully Martin had organized them, and having experienced nothing that conflicted with my beliefs, I concluded that the null results proved that the amplifiers didn't sound different from one another. I bought a Quad 405.

However, over time I began to realize that even though the sound of my system with the Quad was the same as it ever had been, the magic was gone. Listening to records began to play a smaller role in my life—until I replaced the 405 with an M&A tube amplifier two years later.

The lesson was duly learned. Whether or not they can be told apart under blind conditions, amplifiers can have a major effect on a system's sound quality. And more important, normal listening had revealed what the blind test had missed. I told this anecdote at the debate to make two specific points. First, it demonstrates that my following the then-as-now "objectivist" mantra—that audiophiles should buy the cheapest amplifier that offers the power and features they need—had let me down. Second, it pits against one another two core beliefs of the believers in "scientific" testing: 1) that a blind test, merely by being blind, reveals the reality of audible amplifier differences; and 2) that sighted listening is dominated by nonaudio factors, the so-called "Placebo Effect."

To explain my quarter-century-old Damascene experience, you have to accept that either the blind test was flawed—in which case all the reports that cited that 1978 test as "proving" the amplifiers sounded the same were wrong—or that the nonaudio factors were irrelevant, in which case the criticisms of sighted listening based on that factor must be wrong.

Remember, the nonaudio factors were all working in favor of my not hearing any problem with the amplifier: the Quad was inexpensive; it was small for a 100Wpc design (it appealed to my intellectual nature by being no bigger than it had to be); it ran cool; it was nice-looking; and Peter Walker of Quad was a hero of mine. If, as the "objectivists" repeatedly claim, these factors were going to influence my listening, I would have been satisfied with the amplifier. However, my increasing dissatisfaction with the 405 was real. I was having to work harder to appreciate my music through the amplifier, and it was this cognitive dissonance that triggered the tipping point at which I changed from a hard-line objectivist into someone who recognized the value of listening.

I have never said that listeners can't fool themselves—read Jim Austin's essay in the May Stereophile (p.5) concerning the fuss over the so-called Intelligent Chip, as well as his further thoughts in this issue's "Letters" (pp.11–12)—or that sighted listening is not without its own set of pitfalls. But if a listener is true to what his ears are telling him, it is unlikely that that listener will end up with a system that disappoints. And that, surely, is the point of all of that we do: to put together an audio system that makes us happy.

Texas Instruments and Samsung sponsored a digital screening of The Hitchhiker's Guide to the Galaxy at HE2005, in which planetary architect Slartibartfast expresses the sentiment "Science has achieved some wonderful things, of course, but I'd far rather be happy than right any day." I suspect that the believers in "scientism" who uncritically promote blind testing would rather be right than happy, at least when it comes to choosing amplifiers.