The Great Debate...and Then Some
[Editor's Note: Click here to download the recording of the debate. It is a 30MB MP3 file.]
The Great Debate proved reason enough to attend the Show. Although the hour, devoid of either moderator or speaker time limits, hardly resembled a standard debate, the formality with which John Atkinson and Arny Krueger addressed each other offered immediate evidence of the degree of disagreement and rancor between the opposing camps.
Arny Krueger approached the dais with a seven-year history of criticizing the high-end industry, and Stereophile in particular. An audiophile since the BS (before stereo) era, the frequent poster to the rec.audio.pro and rec.audio.opinion Internet newsgroups constructed his first tube and solid-state amplifiers in 1954, and built his first ABX comparator for conducting blind tests in 1977. Proprietor of the pcavtech.com and pcabx.com websites, Krueger distributes an ABX Comparator program that enables you to conduct what he considers foolproof ABX tests in the privacy of your own home.
John Atkinson brought his own history, including former editorship of Hi-Fi News & Record Review and 19 years as editor of Stereophile. Recounting one of his first experiences with blind testing, he described pitting solid-state amplification against one of the early tube revival amplifiers, the Michaelson & Austin TVA-1, back in 1978. When carefully conducted single-blind comparisons revealed no statistically significant differences between the two, John decided to save money and buy one of the subjects of the test, a solid-state Quad 405. He subsequently found himself suffering through listening sessions until he finally acknowledged that the ABX protocol had failed him. He switched to a tube amplifier similar to the one that had "sounded the same" as the solid-state amp and spent the next two years in audio heaven (until he discovered another amp that played "Nearer My God to Thee" with greater fidelity).
Arny arrived armed with absolute certainty (ABX for short). Subjectivism, he declared, is a philosophy that states that we cannot know anything for sure. Objectivism, on the other hand, postulates reason as an absolute, and declares that the scientific approach applies to all areas of knowledge. Stereophile lets down its readers because it willfully ignores incontrovertible evidence that ABX is the only adequate method for evaluating equipment.
John, on the other hand, has participated in a large number of blind tests over the past 28 years, and has discovered that it is "extraordinarily hard to produce anything but a statistically null result." In the end, he has concluded that if so-called objective testing lets you down, it is best to follow Stereophile founder J. Gordon Holt's strategy and judge equipment by how it sounds playing real music in real time.
Once positions were established and barracks fortified, it got a little wild. John noted that Arny's PCABX protocol requires that one digitize the output. This introduces an unverified mode of testing and adds variables to the mix, first in the nature of the digitizer, which conceivably (if not indubitably) alters the original sonic data, and second in the unknown nature of the amplifier over which the digital files are played back. He further asserted that while most audiophiles and reviewers audition equipment by listening to familiar music for extended periods of time on one unit, ABX involves listening to short musical snippets of often unfamiliar music on ever-changing units.
Arny strongly emphasized that sighted listening as practiced by magazine and webzine reviewers fundamentally changes the listener's mental state and is therefore unreliable as a means of assessing sound quality.
John in turn underscored that because ABX fundamentally changes the nature of listening experience, it also alters one’s mental state. To detect small but real differences between amplifiers in an ABX test, he noted, requires very careful test design and a very large number of time-consuming, often impractical trials.
When John cited highly regarded psychoacoustic research that demonstrates that 40 minutes is as long as one can reliably listen in an ABX test, Arny claimed that he can "knock off" 16 foolproof trials in a few minutes by playing specific predetermined sounds rather than music. When John Marks asked about the need to maintain absolute polarity while recording and reproducing sound, Arny declared it irrelevant.
By the end of the hour, if you had been a recent arrival from another planet and had oriented yourself to life in the USA by watching action flicks and video games, you would have been certain that Arnold B. Krueger was God and John Atkinson was a pathetic girly man.
Time for a reality check.
I've had my own sorry experience with ABX testing. In November 2004, I was both instigator and participant in a blind ABX test that attempted to verify sonic differences between generic power cables and those from Nordost Valhalla. Because I knew zilch about testing procedure, I relied on a sound engineer—one who has participated in any number of blind tests and designed breakthrough speaker technology—to create what he considered a valid ABX protocol.
We did everything within our means to enable our 15 or so participants (split into two groups, one of which included me) to hear the major differences that I so clearly and consistently hear between these power cables. Yet the ABX test, consisting of 10 different 60-second musical excerpts that took close to 2.5 hours to get through, failed to reveal statistically significant differences. Leave it to our engineer to gleefully declare the results strongly supportive of his previously unvoiced belief that aftermarket power cables cannot possibly make a difference.
A few days after the test, several trial participants visited the home of a Bay Area Audiophile Society member to try various power cables and interconnects on his new Halcro amp and Mark Levinson preamp. Our small group reached conclusions so unanimously in favor of Nordost Valhalla power cables that the man whose system was on "trial" promptly determined to purchase at least one of them.
Yet when questions about the validity of the ABX protocol were raised on the website (hometheaterhifi.com) for which the trial was conducted, all objections were dismissed with the same degree of absolute certainty as voiced by Arny Krueger. Claims of subjective delusion, glamour, status (Nordost power cables are visibly arresting and cost $2500 each), and wholesale seduction by the "cable mongers" were hurled with a fury usually reserved for debates on abortion, same-sex marriage, and evolution. Blue states/red states, ashes/cables, all fall down.
We sorry subjectivists are in august company. Back in 1979, fabled designer John Curl, with engineering credentials up the wazoo, certainly heard differences between the line-stage sections of the Mark Levinson JC-2 preamp that he designed and a Dynaco PAS-3X. Yet he failed to hear those differences in an ABX test. When he wrote about his experiences in Audio Amateur (the precursor to AudioXPress), he was soundly trashed by ABX absolutists. A quarter of a century later, John Atkinson receives the same treatment.
After reading the classic book Drawing on the Right Side of the Brain, Curl speculated that there's something about the stress of ABX testing that causes the right brains of most participants to shut down or at least skip a synapse or two. It's a reasonable hypothesis that deserves further scrutiny.
The crazy thing about this ongoing debate is that listening to music is not a rational, scientific phenomenon. Who knows where the music that a composer puts on paper actually comes from, let alone the source of the inspiration with which musicians bring those notes to life? Can you explain why one interpretation of a classic song moves you to tears and another leaves you cold? Can you scientifically explain what makes your heart skip a beat?
Of course not. And since audio components and cables are nothing more than electrical conduits for that which lies beyond words and reason, how can we expect charts, graphs, and short bursts of preselected tones to tell us everything about how a component will affect us, let alone how it will interact with other components in a given listening environment?
One thing is certain: Music is music, and protocols are protocols. ABX may be a valid testing protocol for laboratory equipment, but when you bring auditory sensation, emotional reaction, and other, at best, only partially understood mechanisms of brain response into the equation, you are pretending that you can objectively test that which you cannot adequately explain. And please, please don't try to convince me that because you can't fully explain something or detect it with scientific instruments, it doesn't exist. Better to sell me the Brooklyn Bridge.
How best to evaluate equipment? The night before HE2005’s Great Debate, I attended a concert by the New York Philharmonic. The sound of live music was in my head and the effects of live music were in my heart as I went from room to room auditioning systems. I had a strong sense of what to listen for.
We need to do what all audiophile reviewers of worth have learned to do: trust our ears.
Don’t get me wrong. Tests are important. The more accurate the tests, the better they can provide a sense of what is going on mechanically and electronically. But tests can neither adequately describe nor convey everything we hear, sense and feel. Nothing can replace the experience of comparing the sound of equipment with live performance. Whatever your musical tastes may be, you'd be wise to thoroughly indulge in the real thing before placing your bets on equipment that may test better than it sounds.
Music critic and audio reviewer Jason Victor Serinus (jasonserinus.com) recently won Third Place Grand Champion at the International Whistler's Convention. The photographs were taken by Robert Deutsch