Hello PM and welcome to the forum. You sure didn't waste any time with a few nice, simple and noncontroversial posts now did you? Instead you just jumped right in with a post about the topic that draws the most animated posts and responses, the subject of blind testing (considered by many to be the objective method) versus subjective testing. Here's a quote from a fellow forum member, DBZ, that first appeared in a thread about the Sonos Zone Player:

Quote:
By the way, I'm sure you'll be vilified for doing blind testing, regardless of the results. I would hope we can all agree that blind testing is a useful tool, without necessarily placing it on a pedestal above other useful tools, such as subjective "sighted" listening and measurements. If a blind test shows that the listeners can consistently detect a difference, that's awfully good evidence that there really is an audible difference. If a particular blind test doesn't turn up a difference, that certainly doesn't rule out the possibility that the difference might be audible under different circumstances, by different people, with different associated gear, etc. But the negative finding is still a very useful piece of data. Certainly as useful as reading a post from a person you've never met who insists that some change in equipment DID make a huge difference to his ears during a sighted comparison.

I think that pretty much sums up what's good and bad with blind testing.

I wanna see Mf tell me he can hear teh difference between his $100K TT and a good VPI, which are still pricey, but not insane. Without knowing which one is playing , Just like so many of the next breakthrough CD player, that seems to coem each month. And knowing how much it costs, sure do make one expect to hear an improvment. Oh, MF did say he didn't hear a difference between his CHEAP $6500 CD player and some $28K absurdity. How bout' between a mortal under $1000 player, and some absurdity like his Zander (en) multi boxed nonsense. Just think how these CD makers are now going into separate boxes with power supplies etc., on teh claims of better sound. 20 years ago, they wanted to compress products make em smaller, geee, wonder how teh IC got invented, marketing, now teh marketing trend for absurditys is multi boxed CD players. There is zero engineering advantages to doing it, other than marketing. Like 3 box pre amps etc. Absolutly nuts. The LP is sold in multi packs, it used to be called 45's. Didn't sound better did it? When is some mfg gonna sell it's own generator to power up teh CD or amp? Claiming their AC power is specially designed for music!!! Furutech, Music AC generators, not for refrigirators, not for tv's, only CD players. White papers will prove teh advantages of course. Why doesn't teh $100K TT come with it's own generator, designed for TT's? I'm sure they could market some advanatge they found. Cept' the gas fumes will certainly l;ower it's customer base for the future

Quote:
I wanna see Mf tell me he can hear teh difference between his $100K TT and a good VPI, which are still pricey, but not insane.

The last couple of shows, Michael Fremer has been letting people listen to a CD-R on which he had recorded the output of the Continuum player and a less expensive turntable/tonearm, both fitted with the same phono cartridge. Differences were readily audible, and listener's reactions were consistent in preferring the Continuum, though it was not identified as such for them until after the listening.

John Atkinson
Editor, Stereophile

Mr. Atkinson:

I'm delighted to see that you've joined this discussion. I'm the one quoted above pointing out why blind tests can provide useful data, even though a negative result is never definitive. If you carry out the thread from which my quote was taken, you'll see this: "Maybe I'll send my comments to John Atkinson. It really bugs me that he's taken such a hard line against blind tests. He's usually the voice of reason in the audiophile world." But it looks like the mountain came to Mohammed.

So now that you're here with us, maybe we can convince you to tone down your stance a bit. Your example about MF

I have no problem with double-blind testing as long as its suggestion is not laced with the implication that the differences can't be heard, or aren't there at all. For every audiophile trying to make his buddies hear what he does, there's a skeptic with an agenda.
We're covering the auditory system this week in Neurobiology. I never fail to be impressed by the sheer complexity of the cochlea, the inferior colliculus, and all the other amazing structures, working together, that allow us to perceive sound. It's illogical to think that there are not individual differences in hearing, and it's arrogant to imply that sonic differences are all in a person's head. It is in my head; It's called the auditory cortex. Okay, enough ranting from me. I think double-blind testing has a place in audio reviewing, but not any greater than that of subjective methods. After all, if someone is seriously considering buying a five-figure CD player, then there is as much emotion as rationality involved in that decision. I subscribe to the diminishing returns principle. When I can afford the expensive stuff, I'll jump off the curve as soon as I stop hearing differences. However, being satisfied myself does not justify trying to rain one someone else's parade who claims to hear qualities that I can't. I'll not deny that knowing the price of a piece of equipment can be biasing, but that doesn't mean that the extra cost didn't go into making the thing sound really, really good. When I'm making enough bank to be trolling in Mark Levinson, Simaudio, and Sonus Faber territory, you can bet I'll be giving that stuff a listen. I think part of the fun of auditioning is perceiving the differences in character that set component A apart from component B. Maybe DBT would be a more scientific approach, but it sure wouldn't be nearly as enjoyable.

Quote:
I'm wondering whether the reason you've taken such a hard line against blind testing is because many people take such an unreasonably hard line in favor of it.

No, it's because I have done so much blind testing over the past 30 years -- the first such test I took part in, of loudspeakers for Hi-Fi News, was in the spring of 1977 -- that I am well aware of how difficult it is to design and perform a test that produces worthwhile results when the sonic differences are small but real. (See www.stereophile.com/asweseeit/705awsi.)

There is also the fact that no-one's mind is changed by the results of such tests. Back in 1984, I took part in a test using an ABX box where I found that, after a training period, I could identify a change in absolute polarity 10 times out of 10 trials. Yet 20 years later, skeptics were still arguing that these test results didn't prove that absolute polarity was audible.

Similarly, Tom Nousaine, currently with Sound & Vision and The Audio Critic, has been performing blind tests for many years that he admits are intended to show that no differences exists between almost everything other than loudspeakers, yet audiophiles ignore _his_ test results. (Rightly so, in my opinion, given Mr. Nousaine's own agenda.)

So if performing properly designed, suitably sensitive blind tests is difficult and very consuming of the magazines' resources, and no-one will be convinced by the results in any case, parsimony dictates that I not adopt a blind test regime. Our resources are better devoted elsewhere.

Sure, occasionally, one of my reviewers will be wrong, but I please remember that I do listen with my reviewers as much as possible, seeing if I can hear what they describe, and that the components being tested mostly find their way to my own listening room during measurement.

Even without blind testing, I think that my review team accurately characterize the sounds of the equipment they review. I don't think further validation of their abilities is necessary.

John Atkinson
Editor, Stereophile

You must admit, that after 30 years of being dissapointed with blind testing, you finally seem to have hit paydirt when you endorsed Mr. Fremer's method!

Let the parade begin!

In all seriousness, why not, now that you know the proper method, prepare a Stereophile DBT test disc and sell us all a copy for 10 bucks each.

We can all listen and form opinions, then after about 3 months or so, you could publish what gear was on the disc and we can all see for ourselves if we can hear those things as well as you and MF and all those critics he impressed!

I think this is win-win here.

Per JA's input, I am convinced that the real reason is the cost/difficulty of setting up blind tests:

"performing properly designed, suitably sensitive blind tests is difficult and very consuming of the magazines' resources..."

Some blind testing is easy. MF's turntable test was easy and clearly illustrated that blind testing works. Any test involving switching digital sources is fairly easy. I've done a CD player comparison and made the distinction between two volume matched CDP's 19/20 times. The 20th time I wasn't sure. That seems pretty darn convincing to me.

But swapping out a preamp, or amplifiers, is very time consuming. In fact, it seems like you would want duplicates of the downstream equipment, or some sophisticated switching devices. I really believe THAT is the reason blind testing isn't performed.

Quote:

Quote:
I wanna see Mf tell me he can hear teh difference between his $100K TT and a good VPI, which are still pricey, but not insane.

The last couple of shows, Michael Fremer has been letting people listen to a CD-R on which he had recorded the output of the Continuum player and a less expensive turntable/tonearm, both fitted with the same phono cartridge. Differences were readily audible, and listener's reactions were consistent in preferring the Continuum, though it was not identified as such for them until after the listening.

John Atkinson
Editor, Stereophile

If one can tell a difference between a 100k and a less pricey TT from a CD-R, doesn't that in fact demonstrate that CD's have greater resolution than analog equipment?

Quote:
Your post points to a 1978 blind test that left you convinced that no difference in two amplifiers could be heard. Later you purchased a tube amp and were happier. Well thats fine. Why did you decide that the blind test was then worthless?

Because, despite it apparently being well-designed from my perspective as one of the listeners, it had not detected differences between the amplifiers that longer-term, relaxed, sighted listening did. Yes. It was therefore worthless and its results meaningless.

Quote:
I would then be curious as to why I liked the tube amp. Maybe another test to match the tube amp against the other two?

My primary goal, as with any audiophile, was the choice of an amplifier for my own music listening. Further tests were not really necessary once I had achieved that goal. I am not under any dictate to have to provide further proof in order to satisfy my own needs, any more than you are.

Quote:
Could the tube amp have been producing a pleasing distortion? Did the glow make you happy? If indeed all amps, unless otherwise designed, sound the same, then we hi-fi consumers could save LOTS of money.

Sure. That was what I believed at the time to be the case as a result of the blind test, which is why I bought the smaller, cheaper, more powerful solid-state amplifier. But despite the blind test results and despite the placebo effect working in the amplifier's favor in sighted listening, it didn't satisfy me.

Quote:

Quote:
"I am well aware of how difficult it is to design and perform a test that produces worthwhile results when the sonic differences are small but real"

Are you saying a blind test, like the one you participated in and could not discern differences, is invalid because there were real differences?

Of course I am. Just because a test is blind doesn't _by itself_ confer on it the resolving power to detect a small but real audible difference. Take the 1987 Stereo Review listening tests on amplifiers, where the results indicated that under the circumstances of that test, there were no audible differences between a cheap Pioneer receiver and a multi-kilobuck OTL tube amplifier. Yet with the test loudspeaker used, there was the usual response modification due to the interaction between the tube amp's source impedance and the speaker impedance that _was_ large enough to be audible. The Stereo Review test -- which had been loudly proclaimed by that magazine's editor as proving the point you make above -- was therefore too poorly designed to detect the _real_ audible differences that existed between the amplifiers.

Quote:
You mention that Tom Nousaine has been performing blind ABX tests for years. Since he conducted all of these tests, I do wonder how hard or expensive it really is.

As Mr. Nousaine's tests show, it is neither difficult nor expensive to perform blind tests that produce null results, if that is the result you wish for. The results of the many published Nousaine tests are meaningless, however.

Quote:
My preamp has a polarity switch and I have never heard a difference either way. Could you please publish your method of how to listen for and determine correct polarity?

No mystery. Use of an ABX box without time constraints but with control of the switch. The test signal was a test tone rather than music. yes, it is very much harder to detect a inversion of absolute polarity with music, especially on some systems, but my 1984 experiment investigated whether it could be audible _at all_. People also differ in their sensitivity to the effect, please note, and many commercial multitrack recordings do not preserve absolute polarity on individual tracks, making them insensitive to polarity inversion.

Quote:
As a subscriber, please start regularly conducting blind testing and publishing the results in your magazine. I cannot understand why more audiophiles do not demand this. Perhaps they are mesmerized by marketing like me.

Why should they be the ones "mesmerized," and not yourself? How many blind listening tests have you been involved in to be so sure of their efficacy? I took part in my first formal blind listening test in the spring of 1977 and have been involved in well over 100 since then. My statement that it is difficult to design and perform a test that produces worthwhile results when the sonic differences are small but real are based on that considerable experience. Yet your position seems to be that the blind test, merely by the fact that is blind, confers legitimacy on its results. If that is the case, I am afraid that you are wrong.

John Atkinson
Editor, Stereophile

Quote:
If one can tell a difference between a 100k and a less pricey TT from a CD-R, doesn't that in fact demonstrate that CD's have greater resolution than analog equipment?

Certainly it appears that way, and is undoubtedly the case at low frequencies and in the top audio octave. But I have always felt that it is in the midrange that a good LP playback system scores over 16-bit digital and objectively a case can be made for that being the case.

John Atkinson
Editor, Stereophile

Well, I hope you

http://www.theaudiocritic.com/cwo/Our_Philosophy/

DBZ - you make good points. I reiterate my point that blind tests probably aren't performed because they are inconveniant, not because they are meaningless. In my opinion, any other perspective is an excuse, and it is painfully obvious that it's an excuse that one hides behind. Let's be honest about it!

For those who would like to try our simple comparison as noted in Bob Cordell's page, we will be at HE2007 in NY this May 11-13. You can try it out for yourself. Thanks to JA and Ray Kimber for letting us do this experiment. We are in room 1627.

We will be running a series of clinics and workshops, I have the old schedule at my site www.helarc.com and will post the HE shedule tonight along with the abstact. I also have a write up on the previous show at my site.

Yes, we decided to avoid highly debated science in this clinic because we found in early versions of this comparison clinic that no matter how many hoops you jump through, people always find a reason to invalidate the science. So we went the other way and developed an extremely simple amplifier comparison. You switch between a SS and tube amp all you want and let us know what you hear. We encourage you to bring your best test tracks. How hard can this be?

We will be running 6 clinics including amp testing and speaker testing using JA's graphs and explianing what they mean. We will also be doing a speaker comparison with the switch, plus a peak power demonstration.

Workshop #2
Amplifier Listening Comparison

Few audiophiles have ever been able to participate in a real-time, level-matched comparison of two power amplifiers driving the same speaker system in the identical acoustic environment. Amplifiers will be auditioned in real-time listening comparisons using a line-level matching scheme that does not degrade audio quality or introduce impedance-dependent colorations (e.g., as opposed to the use of L-pads that kill the damping factor). This is an interactive presentation, as component comparisons are a hot topic on most of the Internet boards. Some people will come away from this presentation amazed at how hard it is to hear variances in obviously different components, while others will relish the ease with which they can perceive subtle audible differences. The highlight of this presentation will be a listening comparison of a 35 wpc vacuum tube amplifier against a solid state amplifier rated at more than 200 wpc.

I would suggest to those of you fortunate enough to be involved in this experiment to bring along flawed recordings for the blind test.

My thinking is that high quality amplifiers, be they valve or solid state, will likely sound more similar than different with excellent program material. However, when asked to reproduce obviously compromised program material, the likelihood of one of the amps handling the colorations differently would increase.

Plus, it's easier to recognize flaws than it is to recognize virtues when you are limited by time and environmental differences.

I posted the schedule and abstract at my site, below.

P

As Peter mentioned, we will be presenting a series of six interactive workshops at HE2007 titled "Listening & Measurement". The workshops are as follows:

Thanks, Bob. It sounds like a great program.

Let us know what you learn.

Quote:
Thanks, Bob. It sounds like a great program.

Let us know what you learn.

Thanks. You can see the schedule details and abstracts for each of the planned HE2007 Workshops on my web site at www.cordellaudio.com. Also, there are extended descriptions of the results of similar workshops that we presented last Fall at RMAF.

Bob

There is a difference between discerning changes in a system and deciding whether those changes are for the better. I am not from the "Peter Belt" school but can discern differences in the overall sound of my system with changes in hardware isolation, power cables and the like. That does not mean I subscribe to some of the pseudo-science that often accompanies such 'tweaks.' Even the simplest of these systems is complex, and the difficulty of isolating a single component as "correct" or "accurate" when it has to depend on other components from different suppliers, with different design philosophies and goals, makes it impossible to ignore the interactions between components and what connects them. Why do we need to dismiss out of hand one school of thought or the other? THe subjectivist v. objective debate has been around longer than I have, and I started in this hobby in the late '60s.

We focus 90% on the "can you hear a difference" question and the rest is discussion. The idea is that with tubes versus SS, we read a lot about how much difference there is.

P

Quote:
There is a difference between discerning changes in a system and deciding whether those changes are for the better. I am not from the "Peter Belt" school but can discern differences in the overall sound of my system with changes in hardware isolation, power cables and the like. That does not mean I subscribe to some of the pseudo-science that often accompanies such 'tweaks.' Even the simplest of these systems is complex, and the difficulty of isolating a single component as "correct" or "accurate" when it has to depend on other components from different suppliers, with different design philosophies and goals, makes it impossible to ignore the interactions between components and what connects them. Why do we need to dismiss out of hand one school of thought or the other? THe subjectivist v. objective debate has been around longer than I have, and I started in this hobby in the late '60s.

You are exactly right. That debate will probably be with us forever, and each camp has valuable things to say. I think that the healthy tension that exists between the two sides is a positive thing. The polarizing extremes at both ends are what sometimes make the whole thing tedious.

Bob