Quote:
Hopefully you trust me/us enough to know I love Stereophile and would not snipe just to try and hurt your feelings, but I disagree on a couple things you say.

Understood, Buddha. And my apologies for the tardy response. I wanted to think some more about this subject.

Quote:

Quote:
Imagine how cumbersome making a recording would be if for every one of those decisions you had to organize a formal blind test to prove that the problem you thought you were hearing and trying to solve was real or not!...

JA, you are making a sweeping statement to discredit calls for some blind reviewing. Nobody wants you to have to reinvent the wheel every time you make a recording. Your experience is a great labor and time saving device for getting to the realm of hi-fi recording quickly and efficiently.

So why, when there is so much at stake in a recording session, is it okay to rely on my experience and presumably hearing abilities, but not so when a reviewer is preparing a review? The two situations are, to me at least, analogous.

Quote:

Quote:
Yet with a commercial recording, there is a large amount of money hanging on the outcome of those subjective decisions...

Exactly. However, is this the case when you receive a box with an audio component in it? Is there a coterie of people sitting there waiting on you to quickly unpack it, make a decision, and you are spending money even as you look at it? Reviewing is far less time and money sensitive, the decisions you make are not required to be done "on the fly" during a critical review.

Good point. But the fact remains that reviews are done under time constraints. It is my responsibility as the magazine's editor to ensure that those constraints don't negatively affect the reliability of the published review conclusions. Over the years I have listened to my reviewer's systems, I have listened to them describing the differences they hear, I have compared those descriptions with my own experiences of the products, and as I wrote a while back, when we have done formal blind listening tests, I have used the results of those tests to examine the reliability and consistency of the reviewers' hearing abilities.

Yes, there is an enormous flaw in this argument, which is that it shifts the responsibility for not misleading readers from the individual reviewers to the editor, but it doesn't guarantee that our reviews are not misleading. All I can say is that Stereophile is pretty much alone in doing this, in taking this step toward accountability.

Quote:
I bet you don't call Wes and say, "Dude, there's a large amount of money hanging on your review of the Thiels - you've had them ALL DAY, and the whole magazine is paralyzed while you try to ponder if Dengue Fever sounds better on them than on the B&W's. Come on, Wes, trust your experience and review on the fly!"

No, but consider the alternative approach using blind testing, not of loudspeakers, where I _know_ that Wes's opinion on the Thiels is going to be reliable and transportable, but of amplifiers or disc players. After the reviewer had done some slighted listening, I insist that he undertake a single-blind test where I (or an operator) switches between the test amplifier and another. My experience has been that in the simple kind of test that is so often performed, with a minimal number of trials, the reviewer's test results will be null.

Audio magazine used to do this in the early 1980s: Larry Greenhill (who now writes for Stereophile) used to do an amplifier review, then David Clark would do a simple blind test to show that Larry couldn't distinguish the review amplifier from another. Audio would publish the review at this point. But that is pointless, I suggest, because there are _two_ possible reasons for the null result: 1) the amplifiers sound identical to one another; 2) the amplifiers sound different but the test was inadequate to detect that difference.

This was the point of my April editorial. I wasn't making "a sweeping statement to discredit calls for some blind reviewing." I was pointing out that the test conditions _themselves_ were an interfering variable (see later) and that if a blind test is to produce meaningful results, it needs to be designed to minimize the effect of that variable.

Quote:
JA, blind listening is fun. Really. I bet you do it when we're not looking.

I have taken part in an enormous number of blind tests, Buddha, over 100 since my first, of loudspeakers, in 1977. It is that experience that leads me to question their validity _as generally performed_. Take the hypothetical single-blind amplifier comparison I mentioned above. Here's why it produces null results:

The listener listens to Amplifier A on Music Example A. The test operator then either changes the amplifier to Amplifier B or keeps Amplifier A. The listener listens again to Music Example A but now, instead of listening to the music as _music_, he is trying to compare aspects of the music against what he remembers of those aspects from the first presentation. The same thing happens with the third presentation: Is the third amplifier the same as the second or the first? Or was the second the same as the first and the third different?

Okay, try quicker switching where Music Example A is left playing and the operator switches between the amplifiers (or not) on the fly. Now the listener has a third level of complexity imposed in that he has auditioned Passage A from Music Example A on one amplifier and is mentally comparing his memory of that experience with Passage B of Musical Example A on what might either be the same amplifier or a different one.

Dealing with such questions is a lot of work for the listener to be doing. More importantly, it displaces what he _should_ be doing, which is _listening_. The possibility for the listener to become confused, for his scoring of the test to become randomized, _even when a real audible difference exists_, is very high and the test results therefore become meaningless.

Unless, of course, the operator of the test wants to "prove" that no differences exist, which has been the case with a high proportion of the articles on blind testing that have been published in other magazines.

Quote:
Don't complain to me about "practicality."

My point of pointing out that the most vociferous proponents of an exclusively blind test regime for reviewers don't practice it for their own listening was to put forward the notion that such people are actually playing the game of "blame the reviewer."

Quote:

Quote:
"...you trust your hearing and your experience and hope that you are not fooling yourself. Which is basically what audio reviewers do.

That's fine, but if you trust your hearing, why not a little blind listening here and there to check yourself out?

As I said, I have done much blind listening. As a result of that experience, I feel that for such tests to produce anything other than null results _when there are small but real audible differences between products_ is enormously consuming of resources. To the extent that it is not practical for a review magazine to practice a blind test regime unless it drastically reinvents itself. And as I also said, there is no evidence that doing so will result in the magazine gaining readership. And if it doesn

The DBT issue is more about hearing more or determining what is "different", but whether or not it is "better" is another debate all together. Your comments about this being painful, possible, or impossible; your arguments are well taken. There are just too many variables.

I think my issue with DBT is just this: I think it can be an entertaining exercise, possibly even exercising your audio accumen. I think this issue is DBT proponents want control, they want everything the same, but this external manipulation is hard to achieve as there is always some "difference" that will exist.

I firmly believe that with the level of EQ, clarity, resolution from the software we listen to all over the map, what do we control really? Is there one system that really does it all? Are we willing to sacrifice "enjoyment" by just having a totally analytical system that reveals every recording flaw? Is this DBT test only accurate for this software and can we choose one amp over another and say it was best on this, but so much other music was not auditioned.

Maybe what DBT proponents really want is validation that what ever they own that is not the most expensive is just as good as that Class A $25,000.00 amp and that because we see and it, know who makes it, and how much it costs we loose the ability to be honest with ourselves about what we hear.

It is possible for two Ayre C5-XEs to sound different enough that some DBT proponent could hear the "possible" miniscule differences? I doubt it. How many components are inside with 1% or 5% tolerances. If the 1%rs were high and low, now we are at 2%. Does this matter...not to me it doesn't. I know for sure I could not hear it. Is this the kind of difference DBT proponents think they can hear? How much audiophile paranoia can exist?

Even if we could, how many great recording engineers have bought matched pair mics, expensive ones at $5-10K each, and even the matched ones are at what... .5-1db in variation at different freqs. I can tell you that equipment manufacturers are not sweating that detail at all. Well, maybe some are.

In going back to Robert Harley's "Phile" article on CD manufacturing and looking at varable pit sturcture from different manufacturers makes we wonder what most of the DBT fuss is all about. All of these issues out of the listeners control other than to turn it off or listen to something else.

I have begun transferring some LPs into digital files to see if moving all of my lps into a "music server" makes sense. In another conversation here we talked about the EQ being variable disc to disc. Are disc manufacturers within 3db of the RIAA curve to be honest?

These differences are way more prominent that what might be "slight" variations in A DBT of 2 quality amps or CD players. I am using 2 turntable setups to do this LP transfer; one I know is more accurate, the other is warmer in the upper bass and lower midrange. The thinner sounding discs get imported on this turntable only because it sounds better to me. At the age of near 59 I am looking for fun as once I do this I am not doing it again. It is way too time consuming. I even questiong continuing this whole process. It will be of more value in my will I am afraid.

I wish more DBT proponents were recordist, John. I stopped complaining about this stuff when I started just recording local talent and realized just how difficult it is to even begin to capture it all. I can't because of money and gear constraints. Once you leave the recording venue there is some things you can do, but you can't put more "there"...there. How you agonize over mic placement is more than most agonize over speaker placement or room EQ in their own systems I firmly believe. When you use the quality of gear you use, inches can make a big difference. With a quiet noise floor of your gear even the laptop fan is a devil to deal with.