Hopefully you trust me/us enough to know I love Stereophile and would not snipe just to try and hurt your feelings, but I disagree on a couple things you say.
Understood, Buddha. And my apologies for the tardy response. I wanted to think some more about this subject.
Imagine how cumbersome making a recording would be if for every one of those decisions you had to organize a formal blind test to prove that the problem you thought you were hearing and trying to solve was real or not!...
JA, you are making a sweeping statement to discredit calls for some blind reviewing. Nobody wants you to have to reinvent the wheel every time you make a recording. Your experience is a great labor and time saving device for getting to the realm of hi-fi recording quickly and efficiently.
So why, when there is so much at stake in a recording session, is it okay to rely on my experience and presumably hearing abilities, but not so when a reviewer is preparing a review? The two situations are, to me at least, analogous.
Yet with a commercial recording, there is a large amount of money hanging on the outcome of those subjective decisions...
Exactly. However, is this the case when you receive a box with an audio component in it? Is there a coterie of people sitting there waiting on you to quickly unpack it, make a decision, and you are spending money even as you look at it? Reviewing is far less time and money sensitive, the decisions you make are not required to be done "on the fly" during a critical review.
Good point. But the fact remains that reviews are done under time constraints. It is my responsibility as the magazine's editor to ensure that those constraints don't negatively affect the reliability of the published review conclusions. Over the years I have listened to my reviewer's systems, I have listened to them describing the differences they hear, I have compared those descriptions with my own experiences of the products, and as I wrote a while back, when we have done formal blind listening tests, I have used the results of those tests to examine the reliability and consistency of the reviewers' hearing abilities.
Yes, there is an enormous flaw in this argument, which is that it shifts the responsibility for not misleading readers from the individual reviewers to the editor, but it doesn't guarantee that our reviews are not misleading. All I can say is that Stereophile is pretty much alone in doing this, in taking this step toward accountability.
I bet you don't call Wes and say, "Dude, there's a large amount of money hanging on your review of the Thiels - you've had them ALL DAY, and the whole magazine is paralyzed while you try to ponder if Dengue Fever sounds better on them than on the B&W's. Come on, Wes, trust your experience and review on the fly!"
No, but consider the alternative approach using blind testing, not of loudspeakers, where I _know_ that Wes's opinion on the Thiels is going to be reliable and transportable, but of amplifiers or disc players. After the reviewer had done some slighted listening, I insist that he undertake a single-blind test where I (or an operator) switches between the test amplifier and another. My experience has been that in the simple kind of test that is so often performed, with a minimal number of trials, the reviewer's test results will be null.
Audio magazine used to do this in the early 1980s: Larry Greenhill (who now writes for Stereophile) used to do an amplifier review, then David Clark would do a simple blind test to show that Larry couldn't distinguish the review amplifier from another. Audio would publish the review at this point. But that is pointless, I suggest, because there are _two_ possible reasons for the null result: 1) the amplifiers sound identical to one another; 2) the amplifiers sound different but the test was inadequate to detect that difference.
This was the point of my April editorial. I wasn't making "a sweeping statement to discredit calls for some blind reviewing." I was pointing out that the test conditions _themselves_ were an interfering variable (see later) and that if a blind test is to produce meaningful results, it needs to be designed to minimize the effect of that variable.
JA, blind listening is fun. Really. I bet you do it when we're not looking.
I have taken part in an enormous number of blind tests, Buddha, over 100 since my first, of loudspeakers, in 1977. It is that experience that leads me to question their validity _as generally performed_. Take the hypothetical single-blind amplifier comparison I mentioned above. Here's why it produces null results:
The listener listens to Amplifier A on Music Example A. The test operator then either changes the amplifier to Amplifier B or keeps Amplifier A. The listener listens again to Music Example A but now, instead of listening to the music as _music_, he is trying to compare aspects of the music against what he remembers of those aspects from the first presentation. The same thing happens with the third presentation: Is the third amplifier the same as the second or the first? Or was the second the same as the first and the third different?
Okay, try quicker switching where Music Example A is left playing and the operator switches between the amplifiers (or not) on the fly. Now the listener has a third level of complexity imposed in that he has auditioned Passage A from Music Example A on one amplifier and is mentally comparing his memory of that experience with Passage B of Musical Example A on what might either be the same amplifier or a different one.
Dealing with such questions is a lot of work for the listener to be doing. More importantly, it displaces what he _should_ be doing, which is _listening_. The possibility for the listener to become confused, for his scoring of the test to become randomized, _even when a real audible difference exists_, is very high and the test results therefore become meaningless.
Unless, of course, the operator of the test wants to "prove" that no differences exist, which has been the case with a high proportion of the articles on blind testing that have been published in other magazines.
Don't complain to me about "practicality."
My point of pointing out that the most vociferous proponents of an exclusively blind test regime for reviewers don't practice it for their own listening was to put forward the notion that such people are actually playing the game of "blame the reviewer."
"...you trust your hearing and your experience and hope that you are not fooling yourself. Which is basically what audio reviewers do.
That's fine, but if you trust your hearing, why not a little blind listening here and there to check yourself out?
As I said, I have done much blind listening. As a result of that experience, I feel that for such tests to produce anything other than null results _when there are small but real audible differences between products_ is enormously consuming of resources. To the extent that it is not practical for a review magazine to practice a blind test regime unless it drastically reinvents itself. And as I also said, there is no evidence that doing so will result in the magazine gaining readership. And if it doesn