As We See It

Obfuscation?
Editor: Robert Harley's 26-page obfuscation entitled "The Listeners' Manifesto" [January 1992, Vol.15 No.1] best illustrates the subjectivists' refusal to address the objectivists' singular objection: the lack of blindness in listening evaluations.

Rather than addressing this primary issue, Harley concocts a litany of straw-man arguments: blast instantaneous A/B comparisons, chuck the purportedly "nontransparent" ABX comparator, eschew a "test" protocol. Instead, sit in the comfort of your home before your own familiar playback system, request a friend to install the audio component in question, use whatever cabling the manufacturer suggests, relax, take a deep breath, and give a good long listen for as long as you want—months, even. Just one simple request: do not peek at the identity of the component.

Is that too much to ask? With respect to the sound of the component, that knowledge is totally irrelevant. Yet everyone—subjectivists and objectivists alike—would have to admit that at the very least, the possibility exists, even among self-styled "professional listeners," that such information could influence and thereby compromise the listening evaluation. And that possibility—some would say probability (but let us not argue the point)—is intolerable to a credible attempt to be objective in one's subjective assessment of an audio component's sonic performance.

Until the audiophile community adopts a listening protocol which is as blind as Harley's arguments to the contrary, its judgments can only be regarded as suspect.—Jeff Silberman (an objective subjectivist), Baltimore, MD

Not afraid of a blind test
Editor: I have been following the double-blind testing discussion for some time, and I don't understand what Stereophile is so afraid of. I think all of your metaphorical tales of the mysteries of perception and criticisms of ABX are a crock (you supply the contents).

How can you expect to be treated with any credibility when you shy away from the only type of testing that removes preconception and bias? Please don't give me any more of the peering-through-an-eyepiece or taking-a-test-makes-me-nervous stuff. I am not asking for any specific methodology like ABX; just make the test double-blind. Put two amps behind a screen and have someone else do the switching. If you can't reliably identify a Krell from a Peavey without reading the nameplate, how can you ethically recommend one over the other?

You don't need to convince me that a Levinson amp sounds better than a "Best Buy" Pioneer receiver, but the vast majority of electronic design engineers do need convincing. Until credible evidence appears to the contrary, these engineers will continue putting electrolytic capacitors in the signal path and using cheap op-amps.

Worthless dealer demonstrations and five-digit prices make accurate reviews essential to high-end consumers, but how can we weigh your opinions? What protection do we have that you have not been fooled? I have some expensive choices to make (like a new DAC), and your dragging your feet against double-blind testing isn't helping at all. Pick some "easy" comparisons, be scrupulously double-blind, and put an end to this "they all sound alike" mentality.—Michael G. Ford, Orange, CA

Steps toward the truth?
Editor: I attended the loudspeaker cable workshop at the 1991 AES Convention in New York and came away with impressions very different from Robert Harley's ("Audio McCarthyism," January 1992).

The loudspeaker cable workshop was an opportunity to discuss and learn about the technical, aesthetic, and legal issues surrounding this controversial subject. It was not intended to be the last word on the subject, and it certainly was not the conspiratorial persecution of "critical listeners" suggested by Harley. As I recall, Dan Dugan introduced the workshop as a mere step toward the truth. Never did I hear him claim to know the "truth" regarding cables, and certainly no one suggested that "if they couldn't measure it, it didn't exist." That statement is more often used to discredit anyone looking for physical or statistical evidence.

The listening test was probably meaningless, but who suggested it was conclusive? It was done in a practical manner which, if far from perfect, at least was honest and out in the open. For some, perhaps it was a waste of time. I enjoyed the chance to hear 30-gauge wire and learn about ABX testing.

The presentations given by the panelists were well within their respective areas of expertise. I found each to be intelligent and articulate. The psychologist and lawyer were especially interesting because they were not audio experts and had no vested interest. Psychologist Jeff Corey spoke authoritatively (and accurately) on scientific methods, but pointedly did not draw conclusions about speaker cables.

Lawyer Wilfredo Lopez presented consumer protection law in general terms. Most of what I heard sounded like good, common sense. Deceptive advertising is illegal, and making unsubstantiated claims is deceptive. As I understand it, one cannot imply improved performance by simply using technical-sounding jargon. Technical terms must have widely accepted definitions. The Department of Consumer Affairs is asking for a well-reasoned consensus from the audio community on the merits of sophisticated cable designs, and practical terms which describe those merits. Harley's Doomsday account of future legal consequences is a frightening piece of disinformation.

Surely, the majority of AES members recognize the limitations of conventional signal measurements and are open-minded about advancements in cable design. However, the combination of extraordinarily expensive cables and questionable performance gains has roused a few skeptics—hardly the "Spanish Inquisition." Harley needs to show some token concern for innocent, even ignorant consumers...His characterization of this workshop as "Audio McCarthyism" and his portrayal of Dan Dugan as another Senator McCarthy is a pathetic attempt to bury the debate.—Walter Sargent, Stony Brook, NY (Electrical Engineer, member AES, IEEE)

They know what's best for us
Editor: I am a pre–Electrical Engineering student here in Houston, and I just want to make clear the fact that not all engineers are in agreement with the AES and its sympathizers. Even though all differences in audio equipment cannot be shown on testing devices, most engineers realize that differences do exist.

This high-end bashing by some members of the AES therefore has me worried, to say the least. I am afraid someday the government will tell me that I cannot buy the $6000 Tookie-Flookie speaker cables because they are too expensive and have been proven by laboratory equipment to sound no better than less expensive cables. It does not matter that I am an adult, free to make my own decisions, that I can hear a difference, and that I know what I like.

With all that has been said, the government might come in and say, "Certain people have found that this or that piece of audio equipment does not represent a good value to the general public. Because of this we will now take away your right to buy certain things. We do this because there are those who know what is best for you!!"

Not to get overly dramatic, but I think that George Orwell might get a little upset if he was here to see what is happening. If the general public wants to buy this or that cable, either on the advice of a dealer, manufacturer, or just because they want to say they have the most expensive cables available, they certainly have that right. If a customer decides to buy a cable without listening to it first to see if it meets his or her own needs—then it is that individual's choice to do so. These people should not be misled by unscrupulous people, but they should also be able to buy what they want.

And just what are we supposed to use as speaker cable—lamp cord?!? Yeah, right!!!—Clayton Dreyer, Houston, TX

Comic-book reporting
Editor: In the January 1992 Stereophile, Robert Harley's coverage of the October 1991 AES Convention continued the great comic-book reporting that we have all come to love. Stereophile has become a wonderful amalgam of Amazing Comics and The National Enquirer.

Naturally, I disagree with most of John Atkinson's and Robert Harley's coverage, but more specifically I have a question about something he said on p.73. Here, Mr. Harley stated that my research ("Can You Trust Your Ears?," AES Preprint 3177), which shows that listeners are strongly biased to report differences when given two identical alternatives, "...calls into question the whole idea of A/B testing. Because music has meaning, we interpret it differently each time we hear it. These different interpretations often obscure the audible effect one is trying to disprove exists." (sic)

However, Mr. Harley uses music to evaluate the Linn Karik in the same issue. He notes that the Linn made him "feel right," which was "the most valid indicator of quality, and that listening for specific performance attributes could preclude this perception of feeling right." How, then, could he be sure that it wasn't just his differing interpretations of the music each time he heard it that made him feel right?

Doesn't the use of music call into question your evaluation techniques as well? Even if you overlearn the program material, as Bob Stuart recommends (a good idea), how can you be sure that the musical meaning won't influence your evaluation? Stuart suggests that other research disciplines use stimulus without content. So how about it: test signals, anyone?

Oh, and yes, Mr. Harley "put the Linn CD player up against some very tough competition" in his single-presentation evaluations, and later "compared" it "with other digital processors," apparently using music for both kinds of evaluation. Aren't these just A/B tests? The single-presentation method has more time and distance between comparisons, but they both include some kind of A/B comparison.

"Can You Trust Your Ears?" helps quantify listener bias to help consumers make better buying decisions. It validated phenomena that appeared in earlier research: eg, even trained, experienced listeners are biased to report differences when presented with two identical alternatives. Therefore, bias control is perhaps the most important element of good-quality subjective evaluation.

Aren't ABX and other double-blind evaluations just A/B comparisons, similar to your own, with listener bias control added? Blind comparisons aren't limited to length or fast switching intervals. In fact, one of the first amplifier tests published took three weeks to get 40 trials. Others have employed single listeners with personal reference systems. Another very early one used highly regarded listeners. In these tests, when bias controls are added to comparative evaluations of electronic signal-path devices, the results rather strongly support the conclusion that important sonic differences between such devices are nil. Stereophile's own tests verify this conclusion. In other words, when bias controls have been added, your own research fails to support your hypotheses.

You might be well advised to start listening closely to the content of your own research. It shows that amplifiers and CD tweaks stop making you feel better when the faceplates and name tags are removed. It's not bad news...be thankful N-Ray machines are no longer available.

In summary, if "Can You Trust Your Ears?" calls A/B testing into question, it calls Stereophile's equipment evaluations into question as well. I read your magazine with interest and good cheer. Many of my friends, including me, want to think of Stereophile as the product of well-meaning, if occasionally misguided, good guys in white hats.

J. Gordon Holt, in my opinion, fit that description, but your defensively biased reporting, persistent rejection of contradictory information (for example, any experiment with null results, even your own), endorsement of ill-fated ideas like Armor All, and mean-spirited comments, make it increasingly hard to judge the color of your headgear. Chill out, Dude.—Tom Nousaine, Cary, IL

John Atkinson responds: Stereophile has stated many times in print its opposition to double-blind, forced-choice, ABX-presentation test procedures. There is hardly the space here to go over the complete reasons for that opposition yet again, so I refer readers to RH's article in Vol.15 No.1, p.111, for a full exposition, as well as back to Mr. Hodge's letter above. Briefly, however, despite Mr. Nousaine's statement that blind tests strongly suggest that "important sonic differences...are nil," having myself taken part in many such tests, I am forced to the conclusion that obtaining null results must have been the motive behind the methodology adopted in nearly all of them.

In addition, as in any scientific procedure, the variables in a listening test must be reduced to just the one to be examined by that test. The methodology advocated by Mr. Nousaine, Mr. Ford, Mr. Silberman, et al, actually introduces additional variables that no one has yet explained to our satisfaction why they don't matter. Sighted listening as practiced by Stereophile, of course, has a different set of drawbacks, but at least they are ones that are predictable and can be allowed for by a professional reviewer/experienced listener, as suggested by Mr. Hodge. (Pace, Mr. Silberman—but why do you use the words "self-styled professional" for people who do earn their livings from what they do?)

Let me illustrate this first point with an example taken from my own experience. Amid Mr. Nousaine's obfuscatory bluster over A/B tests—RH was talking about blind A/B tests as practiced by Mr. Nousaine, of course (footnote 1)—he asks the question—sarcastically, I assume—"Test tones, anyone?"

Having taken part in blind tests that have used both tones and music, I have found tones (which, by definition, do not change with time) to be a far more efficacious signal source for use in blind tests than music. Using an asymmetrical signal in a blind test with the ABX box back in 1983, for example, I consistently scored 10 out of 10 detecting acoustic polarity reversal, whereas with music, my identification dropped to that due to chance. Does this mean that absolute polarity is inaudible with music? Not necessarily. An equally strong hypothesis can be made that the changing nature of music with time renders the ABX methodology invalid. The fact that the overall published evidence supports the audibility of absolute polarity with music suggests that this latter hypothesis is the correct one. Doesn't it?

There is also a practical point here. If our review methodology is suspect, then our descriptions of sound quality and our value judgments will also be suspect, bearing only a random relationship with reality. We publish as much supporting information in our reviews as possible to enable readers to test our conclusions for themselves. Every month, therefore, we give everyone the opportunity to test the validity of what we say. If our descriptions and conclusions are at variance with reality, then Stereophile has no business staying in business. If they do correspond with reality, then all the philosophical objections don't amount to a hill of beans.

Regarding the "Wire Roast" at the October 1991 AES Convention, I am happy that, despite the protestations of Mr. Sargent and Mr. Nousaine, Robert Harley accurately reported what occurred and was said. As the entire affair was taped, I encourage readers to order the cassette tape of the session, therefore. (Contact Conference Copy, Inc., 2222 Avenue X, Brooklyn, NY 11235. Charge-card orders are accepted at (718) 934-2890.) And to those who seem to feel that Stereophile somehow "has it in" for the Audio Engineering Society, please note that both Bob Harley and I have been AES members for many years and regularly attend the conventions. We have nothing but respect for the vast majority of the Society's membership; the criticisms made in our January issue referred to the actions and statements of a tiny number of very vocal activists with an agenda I feel to be at odds with the interests of the Society as a whole.

Please do not assume that our opposition to the legal implications of Dan Dugan's "Wire Roast" means that we support deceptive advertising or the dissemination of disinformation, as has been suggested elsewhere. The fact is, however, that, as Mr. Dreyer points out, for a small number of politically motivated militants to try to set up an a priori set of restrictions on trade is Orwellian in its implications. Under current law, if a consumer feels that he or she has been duped, remedies are easily available. (I also note with interest that retailers often allow customers to try before they buy with cable.) To set up an alternate system where only those manufacturers who can satisfy criteria established by a small faction of the AES membership are allowed to bring their products to market is an anti-democratic action that I cannot support. As Michael T. Saliba said in our March "Letters" column, "We must be constantly wary of those who purport to use government to promote the well-being of those other than themselves."

And who, if not the consumer, is qualified to decide what is and what is not deceptive advertising? One of the advertisements used by Mr. Dugan in his presentation to illustrate his thesis (all taken from Stereophile, Mr. Sargent) concerned the Duotech Cable Enhancer, which "burns in" cables and interconnects. I must admit to having been skeptical myself when I first learned of this product. Yet if you turn to this issue's "Industry Update," you will read that Peter Mitchell (one of Stereophile's more conservative writers) took part in a listening test where he concluded that this box does appear to make a difference to the sounds of cables! There is nothing as strange or as complex as reality, something that the Nousaines and Dugans of this world would apparently like not to be true.—John Atkinson

---

Footnote 1: There is the factor that, in a blind test, you can never go back to check any particular aspect of performance. Under normal sighted conditions, let's say you notice what might be a narrow-band coloration on female voice. You can play the same track again or reach for different recordings to explore, confirm, or deny the observation. The overall conditions and ever-changing musical selection in the blind test (necessary to make it practical) work against the process of diagnosis, leaving the listener's perception to some extent lagging events. While blind panel testing has its place, it is too blunt a tool to become the be-all and end-all of review procedure, in my humble opinion.

Before all you blind-test buffs rush for your pens and papers to write angry letters to the editor accusing me of striking a political stance, note that I know that of which I speak. As well as carrying out dozens of normal reviews in the last 10 years for Hi-Fi News & Record Review and Stereophile, I have both taken part in and organized many tens of blind tests since the Spring of 1977, when I was part of a panel for a blind test on loudspeakers organized by the late Jimmy Moir.—John Atkinson