You are here

Log in or register to post comments
Xenophanes
Xenophanes's picture
Offline
Last seen: 3 years 7 months ago
Joined: Sep 7 2005 - 2:48pm
Re: Why is Blind Testing Such a Hot Button?


Quote:
>>
> The obvious fallacy here is Atkinson's constant implicit
> assertion that all DBTs are equally trustworthy. OK, so
> Atkinson did one tiny little DBT 20 years ago that mislead > him.

I refer you to the recording of our debate, Mr. Krueger, in which I clearly listed the very large number of blind tests in which I have particpated over the past 28 years. This recording is available at http://www.stereophile.com/news/050905debate. I mentioned the 1978 test as being representative, not because it was the only such test in which I have participated.

> We really know nothing about Atkinson's 1978 Quad
>amplifier tests.

They were reported in exhaustive detail in the November 1978 issue of Hi-Fi News. My point is that at that time, I had no reason to doubt the results of the tests. And as I reported in the July Stereophile, I told this anecdote at the debate to make two specific points. First, it demonstrates that my following the then-as-now "objectivist" mantra -- that audiophiles should buy the cheapest amplifier that offers the power and features they need -- had let me down. Second, it pits against one another two core beliefs of the believers in "scientific" testing: 1) that a blind test, merely by being blind, reveals the reality of audible amplifier differences; and 2) that sighted listening is dominated by nonaudio factors, the so-called "Placebo Effect."

To explain my quarter-century-old Damascene experience, you have to accept that either the blind test was flawed -- in which case all the reports that cited that 1978 test as "proving" the amplifiers sounded the same were wrong -- or that the nonaudio factors were irrelevant, in which case the criticisms of sighted listening based on that factor must be wrong.

John Atkinson
Editor, Stereophile

We still don't know much about the blind test in 1978. I certainly don't have a copy of the article. What was the output impedance of the tube amp? What was the impedance vs. frequency curve of the speakers used? Also, of your own speakers? Or, more simply, what was the frequency response of the amps into the speaker loads?

It also seems that you have set up a false dichotomy, which Mr. Krueger's response has already shown. And of course, to talk of a blind test proving that two amplifiers sound the same is an overstatement, as statistics don't give absolute proof. As to the dichotomy you have set up, we are not simply left with a blind test in which all non-audio factors were eliminated which nonetheless failed, or a declaration that the non-audio factors were irrelevant, thus justifying sighted auditioning.

How do you know that the failure of the DBT to detect significant differences was due to non-audio factors? Mr. Krueger already suggested that it might be that the program material selected was not as good as it might be, that listeners might better detect differences with something else. We don't have the measurements of the FR of both amps into the speaker load so as to see what the differences were.

By what reasoning do you arrive at the conclusion that the failure of a particular DBT says anything about how reliable sighted listening is? I can't make any sense out of your argument here.

jimreiter
jimreiter's picture
Offline
Last seen: Never ago
Joined: Sep 12 2005 - 10:08am
Re: Why is Blind Testing Such a Hot Button?

My wife has one working ear that is terribly sensitive. She cares not a whit about equipment but loves music. She heard the difference when I upgraded from my Cary 300 to my 303-300. She even asked what i had done when I switched from solid state to tube output on the 303-300. Liked it a lot better. Earlier CD players, before the other Cary , bothered her. There is a difference in the good stuff.

philip
philip's picture
Offline
Last seen: 1 year 7 months ago
Joined: Sep 4 2005 - 1:48am
Re: Why is Blind Testing Such a Hot Button?

To answer the topic directly, it is my opinion that a lot of it is a matter of pride. No one wants to look like silly/irrational for purchasing a hugely expensive component while not being able to tell a difference between a cheaper one, let alone be able to establish that the newer component is better, not just different.

Perhaps the equillibrium point for a lot of people is between money and the sonic differences. Again, it's pride/ rationality: A less-well-to-do person purchasing a $2000 source which he can't hear improvements on, will feel a lot worse than a much richer one.

Also, lets remember that a lot of people purchase things 'blind' and or 'deaf' under conditions where they can't compare a source and its synergy with thier entire audio chain, not to mention room acoustics.

Even I myself as a headphone listener for most of the year find it hard comparing a new source with my own -remembering that I bring my own amplifier and headphone to the audition.

arnyk
arnyk's picture
Offline
Last seen: Never ago
Joined: Sep 1 2005 - 9:36am
Re: Why is Blind Testing Such a Hot Button?

>Also, not to cause any affront to any cult of DBT, but why all this comparator, level matching stuff?

You left out time-synchronization. ;-)

The procedures and requirements are based on trying a number of alternatives, and picking the ones that worked best. "Worked best" was based primarily on trying for the best listener comfort and the most positive results while avoiding false positives.

For example, level and timing mismatches obscure real differences between audio gear. If you mismatch levels in a comparison by more than about 0.5 dB, a component will sound different from even its own sweet self. If you don't synchronize the music playing from component "A" from that playing from component "B" within a fraction of a second, component "A" will always sound different from component "B", even if they are identical in every other way.

The idea of the comparator was to allow the listener to control as much of the test by himself as is possible. He doesn't have to line up a friend to help him. He can test whenever his spirit moves, for as long as he likes with whatever music or other sounds as he likes.

>Since I don't listen to components by jumping from one to the other instantaneously, why should a reviewer?

Instantaneous switching can easily be shown to maximize listener sensitivity. Putting in switching delays can turn sensitive listeners into insensitive listeners. Differences that you can hear with a quick switch are masked by and made inaudible by slow switching.

The PCABX Comparator you can download for free from www.pcabx.com has a switchover delay adjustement. I advise people who are skeptical about this to try to get their best results in some of the tougher tests posted on the PCABX Training Room page, and then crank in a lot of switching delay and see what happens to their accuracy and sensitivity as listeners.

Never fails!

Buddha
Buddha's picture
Offline
Last seen: 2 years 9 months ago
Joined: Sep 8 2005 - 10:24am
Re: Why is Blind Testing Such a Hot Button?

Quote: "Instantaneous switching can easily be shown to maximize listener sensitivity. Putting in switching delays can turn sensitive listeners into insensitive listeners. Differences that you can hear with a quick switch are masked by and made inaudible by slow switching."
____________________________________________________
____________________________________________________

Aye, there's the rub.

If a difference can only be discerned with instantaneous switching and not with longer delays in changes, then is it really of any concern? Can it even be said to exist if the only way you can identify it only in "artificial circumstances?"

I am flabbergasted at your take, not in mean way, but in a listener's way.

Do you sit at home and listen to a piece of equipment and take any satisfaction from telling yourself that although you can't hear a difference in two choices of equipment sitting their drinking a fine Burgundy, at least you could when you had every switched up and were doing instantaneous DBT?

Is that even an outcome we should strive for? Is that DBT result even meaningful?

For me, the consumer, if a difference in sound cannot be perceived outside a situation that involves instantaneous switching and extra items in my listening chain, please tell me how it has any bearing on my equipment choice.

The whole basis for our systems is long term listening.

I think maybe I look at this in a fundamentally different way than you do...

If DBT can so easily be ruined by artifacts of time and volume, then I would venture the best way to avoid those problems is to not install artifact generating materials into the audio chain in the first place.

Again, I say give a reviewer a set up and have him listen critically. For months, days, I don't care. Then change the item in question (or don't) with no hints as to brand name or cost, even if it has been "burnt in" or not, and repeat the listening process. If this yields a repeatable finding, great. If not, then we have just learned something about either the product or the listener, eh?

All of this is happily free of extra switches and goo that are not part of what should be expected to be part of the consumer's experience of the same piece of gear.

For me, it's far easier to have someone change or not change an item in the listening chain and leave me to it that it is to construct what you want me to do.

Too bad I missed that debate in NY! Nothing more fiery to do thatn argue about our audio "articles of faith!"

To steal from Robert Frost..."We sit on our sides of the debate and propose, while the truth sits in the middle and knows."

Editor
Editor's picture
Offline
Last seen: 3 years 8 months ago
Joined: Sep 1 2005 - 8:56am
Re: Why is Blind Testing Such a Hot Button?


Quote:
To steal from Robert Frost..."We sit on our sides of the debate and propose, while the truth sits in the middle and knows."

Excellent!

John Atkinson
Editor, Stereophile

Xenophanes
Xenophanes's picture
Offline
Last seen: 3 years 7 months ago
Joined: Sep 7 2005 - 2:48pm
Re: Why is Blind Testing Such a Hot Button?


Quote:
Quote: "Instantaneous switching can easily be shown to maximize listener sensitivity. Putting in switching delays can turn sensitive listeners into insensitive listeners. Differences that you can hear with a quick switch are masked by and made inaudible by slow switching."
____________________________________________________
____________________________________________________

Aye, there's the rub.

If a difference can only be discerned with instantaneous switching and not with longer delays in changes, then is it really of any concern? Can it even be said to exist if the only way you can identify it only in "artificial circumstances?"

I am flabbergasted at your take, not in mean way, but in a listener's way.

Do you sit at home and listen to a piece of equipment and take any satisfaction from telling yourself that although you can't hear a difference in two choices of equipment sitting their drinking a fine Burgundy, at least you could when you had every switched up and were doing instantaneous DBT?

Is that even an outcome we should strive for? Is that DBT result even meaningful?

For me, the consumer, if a difference in sound cannot be perceived outside a situation that involves instantaneous switching and extra items in my listening chain, please tell me how it has any bearing on my equipment choice.

The whole basis for our systems is long term listening.

I think maybe I look at this in a fundamentally different way than you do...

If DBT can so easily be ruined by artifacts of time and volume, then I would venture the best way to avoid those problems is to not install artifact generating materials into the audio chain in the first place.

Again, I say give a reviewer a set up and have him listen critically. For months, days, I don't care. Then change the item in question (or don't) with no hints as to brand name or cost, even if it has been "burnt in" or not, and repeat the listening process. If this yields a repeatable finding, great. If not, then we have just learned something about either the product or the listener, eh?

All of this is happily free of extra switches and goo that are not part of what should be expected to be part of the consumer's experience of the same piece of gear.

For me, it's far easier to have someone change or not change an item in the listening chain and leave me to it that it is to construct what you want me to do.

Too bad I missed that debate in NY! Nothing more fiery to do thatn argue about our audio "articles of faith!"

To steal from Robert Frost..."We sit on our sides of the debate and propose, while the truth sits in the middle and knows."

I'll make a couple of comments.

I have to agree that if something doesn't bother me in the long term even though it will show up in a DBT with quick switching, then it is not very significant. The problem is when reviewers wax poetic about the particular sound qualities of different components that will not show up in a DBT under the same circumstances. When reviewers start telling me about the wonderful qualities of some expensive interconnects and speaker cables (here, expensive means more than at Walmart or the hardware store!), I am sceptical--highly sceptical. Same with most solid state amplifiers, decent CDPS, as they usually measure quite well, and so on. If someone wants to sell me or recommend such components on the basis that they sound better than most, well, I say prove you can at least tell the difference. Otherwise, I'm not interested in what they say. There may, of course, be other reasons to prefer one piece of equipment over another besides sound quality.

Time and volume. OK let's suppose you want to compare two CDPs. Well, it is simply highly unlikely that both of them will have the same output level. As well, if you don't synchronize the two players pretty closely, then you can tell which is which simply because one is slightly ahead of the other on the same CD. Or, if you can't switch quickly, within a fraction of a second, your aural memory declines. Now, when a reviewer compares equipment, how many of them bother with little trivialities as level matching? And how do we know he/she can actually tell them apart based solely on the sound?

arnyk
arnyk's picture
Offline
Last seen: Never ago
Joined: Sep 1 2005 - 9:36am
Re: Why is Blind Testing Such a Hot Button?

>If a difference can only be discerned with instantaneous switching and not with longer delays in changes, then is it really of any concern? Can it even be said to exist if the only way you can identify it only in "artificial circumstances?"

If you're are complaining that bias-controlled tests are too sensitive to small differences to suit you, than that is a criticism that I am happy to accept and live with.

Monty
Monty's picture
Offline
Last seen: Never ago
Joined: Sep 16 2005 - 6:55pm
Re: Why is Blind Testing Such a Hot Button?

If I were a professional reviewer I wouldn't be fond of blind testing equipement. Experience is worth something and most manufacturers tend to have a certain reliable sound quality that they have developed over the years.

I would think that most reviewers familiar with the various products and manufacturers would much rather concentrate on the more subtle differences of high-end equipment. When some sonic characteristic immediately strikes you as being unusual for a particular brand it naturally peaks curiosity.

British gear comes to mind. A reviewer putting an NAD amp through its paces would automatically assume a rather polite and laid back tonal balance and would only find it interesting if the amp were to exhibit a tonal balance more in line with say a Bryston amp.

If time were of no concern then I would find it hard to defend the practice of not doing it, but since time is a concern for people who make their living testing equipment, you gotta consider these guys are getting paid in large part because of their experience. That's worth something.

Buddha
Buddha's picture
Offline
Last seen: 2 years 9 months ago
Joined: Sep 8 2005 - 10:24am
Re: Why is Blind Testing Such a Hot Button?


Quote:
>If a difference can only be discerned with instantaneous switching and not with longer delays in changes, then is it really of any concern? Can it even be said to exist if the only way you can identify it only in "artificial circumstances?"

If you're are complaining that bias-controlled tests are too sensitive to small differences to suit you, than that is a criticism that I am happy to accept and live with.

Interesting.

You point out that there are changes that are not discernable in long term listening that you find can be identified with instantaneous comparisons, and you point that out as though it matters?

Seriously, if something cannot be discerned over longer listening periods, how can that in any way point to importance of DBT to the hobby?

I suppose you'd be a joy doing sports car comparisons, requiring instantaneous changes between cars while driving the test track.

(Disclaimer: Meant as a good humored rib about DBT, no personal criticism implied.)

arnyk
arnyk's picture
Offline
Last seen: Never ago
Joined: Sep 1 2005 - 9:36am
Re: Why is Blind Testing Such a Hot Button?

>You point out that there are changes that are not discernable in long term listening that you find can be identified with instantaneous comparisons, and you point that out as though it matters?

No, I never said that. I think that if you can hear something in the short term, eventually you'll probably hear it in the long term. And vice-versa.

The scenario I'm familiar with relating to the use of ABX as a diagnostic is that I think I hear something wrong in general use, and I use ABX to confirm or deny that casual perception.

Norm Strong
Norm Strong's picture
Offline
Last seen: Never ago
Joined: Sep 2 2005 - 11:06am
Re: Why is Blind Testing Such a Hot Button?


Quote:
As the HE2005 Debate thread reveals, blind testing stirs strong passions, pro and con. Why _is_ this?

Why indeed? I believe you can get better information about the performance of audio equipment when you evaluate it blind, but I certainly don't get "passionate" about it. If you want to spend big bucks on a component that you can't tell from a much cheaper one without seeing it, go ahead. It is, after all, your money. No law requires you to get full value when you purchase something--and most people don't!

a3dtot
a3dtot's picture
Offline
Last seen: Never ago
Joined: Oct 25 2005 - 11:13am
Re: Why is Blind Testing Such a Hot Button?

Blind testing, the debate or argument continues. In my 30 plus years as an audiophile, I have had from very inexpensive to a good high end system. I took a sound design cheap system added a couple of speakers and completely changed the sound. My current system is a combination of a tubed pre-amp and a solid state amp with old bozac speakers, home made wires a cd transport with an older theta dac. The sound for me is incredible. I have heard many of the new speakers, wilson, jmlab, electrostatic etc. I prefer the sound of the bozacs. Yes there are many problems with sound stageing, my treble range is around 19k. I support my bass with a diy 18" subwoofer and even with all the aknowledged faults of my system, it still sounds better than any other system I have heard so far. This is why blind testing does not work in the audio field. Each listener is different. I like having 8 tweeters in a vertical array. I like the sound of vinyl better than digital. I like what a tube pre-amp does. Therefore I am automatically biased before I could ever begin a blind test. Can I tell the difference between two different interconnects, maybe. In what context is the experiment. When I changed the cables in my system I could hear the difference. When I sheilded my speaker wire I could hear the difference. Are these changes better or different. Some are better some are just different. But better is my own choice of what I like. Blind testing is meaningless to me, just as much as knowing the specs of components. If it sounds good I like it, if it doesn't I don't.

Lighthouse
Lighthouse's picture
Offline
Last seen: Never ago
Joined: Nov 3 2005 - 2:07pm
Re: Why is Blind Testing Such a Hot Button?

This DBT Objective vs Subjective debate is the most boring arguement. Why do some people have to insist on telling others what they can or cannot hear.

I work as a Mastering Engineer, and I hear things that others often do not, At the recent London hi-fi show, I listened to a

Buddha
Buddha's picture
Offline
Last seen: 2 years 9 months ago
Joined: Sep 8 2005 - 10:24am
Re: Why is Blind Testing Such a Hot Button?


Quote:

..What I observe in myself, is that my ability to hear will vary. Some days it is good, and other days it is not so good. Also, I observe that my hearing can focus on certain qualities of sound in such a way, that differences can be magnified to appear quite large. I may remark that a difference is obvious and very large, only to find later, when listening again, that the difference is actually extremely subtle. Perhaps this is the case with the subjective reviews in magazines like Stereophile.

A standard practice of mine whilst mastering, is to engage and disengage an equalization change whilst not knowing when it is in or out. This is my way of testing whether what I am doing is "really" improving the music.

I have subjected myself to ABX testing, and have found that it renders me deaf to even rather gross changes. I find the act of installing the ABX equipment makes everything sound the same..."

Dave

Those are awesome points!

I agree, instantaneous DBT doesn't thrill me. As you mentioned, it tends to obscure subtlte differences and doesn't always match one's skill set of the day.

However, I am still enamored of the occasional attempt at long term (and convenient) "blind" reviewing (especially of interconnects) so as to accomodate those "off" and "on" days we have.

As demonstrated with the fad tweaks, there can be a large amount of placebo effect that occurs when setting one's expectations - like knowing price, company name, etc...

I bet comparing different pressings of different CD's or LP's "blindly" over time would be fun, as well.

Especially for something like this "CD chip" thing that people are supposed to place on their CD players during playback to "improve" discs could be addressed in this manner.

Perhaps JA could send every reviewer a "blinded set" of treated and untreated discs of the same title to allow the reviewers to relax and compare over time. They could compare and contrast the two and respond, not knowing which disc is treated or not.

That would be great fun!

Lighthouse
Lighthouse's picture
Offline
Last seen: Never ago
Joined: Nov 3 2005 - 2:07pm
Re: Why is Blind Testing Such a Hot Button?


Quote:

I bet comparing different pressings of different CD's or LP's "blindly" over time would be fun, as well.

Hi Buddha,

This is another of my pet things. I have many different versions of some of my favourite discs. For example I have 6 copies of Joni Mitchell's Blue on CD.

1 is the first UK release from the 80's,
2 more are later UK pressings (non remastered),
another is an australian pressing,
then there is the current UK HDCD remastered version,
and finally Steve Hoffman's Gold DCC remastered version.

Obviously the remasters sound different from the original, but what is shocking, is that the 2nd pressings (non remastered) sound nothing like the first release, and the Australian pressing sounds like a different recording.

My order of preference is:

1. 1st UK pressing
2. DCC Gold remaster
3. HDCD remaster
4. 2nd UK pressing
5. Australian pressing

I can pick out the 1st pressing in a blind test pretty easily.

I have a few other CD's, where early pressings sound quite different to later ones.

Dave

Kal Rubinson
Kal Rubinson's picture
Offline
Last seen: 6 hours 48 min ago
Joined: Sep 1 2005 - 9:34am
Re: Why is Blind Testing Such a Hot Button?


Quote:
I agree, instantaneous DBT doesn't thrill me. As you mentioned, it tends to obscure subtlte differences and doesn't always match one's skill set of the day.

However, I am still enamored of the occasional attempt at long term (and convenient) "blind" reviewing (especially of interconnects) so as to accomodate those "off" and "on" days we have.

Blind testing, without other constraints has value but it is very difficult to achieve in practical terms.


Quote:
Especially for something like this "CD chip" thing that people are supposed to place on their CD players during playback to "improve" discs could be addressed in this manner.

Perhaps JA could send every reviewer a "blinded set" of treated and untreated discs of the same title to allow the reviewers to relax and compare over time. They could compare and contrast the two and respond, not knowing which disc is treated or not.

That would be great fun!

Not for me. I tried this with the "CD chip" thingie as I had two copies of the MTT Mahler 6 SACDs. Repetitive listening was numbing and I quickly lost attention and interest.

Kal

Buddha
Buddha's picture
Offline
Last seen: 2 years 9 months ago
Joined: Sep 8 2005 - 10:24am
Re: Why is Blind Testing Such a Hot Button?

Quote: "Not for me. I tried this with the "CD chip" thingie as I had two copies of the MTT Mahler 6 SACDs. Repetitive listening was numbing and I quickly lost attention and interest.

Kal"

___________________________
___________________________

Not to sound flip, but that's why they call it "work," right?

What I mean is that this sort of thing is the very thing that is in the realm of the reviewer to suss out. We, the comsumers are faced with this device and its claims, and all we get is "too tedious to figure out" from a reviewer?

I didn't mean that to sound so sarcastic, but if you (reviewers) can't tackle this issue and these claims, then who will?

It can't all be trips abroad, red carpets, paparazzi, hot reviewer groupies, and seven course meals, eh?

A friend and I tried this over a period of months to no detectable result. I tried to describe it in a thread about a "triumph of double blind testing" but I think even thinking about this device is wearying!

Kal Rubinson
Kal Rubinson's picture
Offline
Last seen: 6 hours 48 min ago
Joined: Sep 1 2005 - 9:34am
Re: Why is Blind Testing Such a Hot Button?


Quote:

Not to sound flip, but that's why they call it "work," right?


Yeah, right, but I can generally choose my work. Testing the "chip" is not one I choose. As with other components, I am willing to spend time only with products which, for one reason or another, interest me. I have participated in some tests at the request of JA. For example, there is a report in press about CD signal reprocessing which I participated in despite its dreariness.


Quote:
A friend and I tried this over a period of months to no detectable result. I tried to describe it in a thread about a "triumph of double blind testing" but I think even thinking about this device is wearying!


I understand.

300Binary
300Binary's picture
Offline
Last seen: 12 months 1 day ago
Joined: Sep 1 2005 - 10:47am
Re: Why is Blind Testing Such a Hot Button?

Fully synced, matched and balanced DBT is an Olympic level sport - like, maybe, individual synchronized swimming. Only the true fans don't think is a stupid waste of time. It is vital Art! Not all pearls are in front of the swine ...

Tyll
Tyll's picture
Offline
Last seen: Never ago
Joined: Oct 15 2005 - 9:38am
Re: Why is Blind Testing Such a Hot Button?

*Throws hat in ring*

I think the reason this issue is such a hot button is because audiophiles tend to be "gadget" centric. This "object" orientation is similar to that of "motorheads" or ham radio fanatics as I see it. But the difference is that the result of audiophilial activities does not have a measured outcome but an experiential one. Motorheads shoot for low times in the quarter mile, and ham radio operators shoot for long distances in their communications, but audiophiles have no similar objective measure of performance as a direct end result. My assumption here is that audio gear is designed to give a listening experience, as opposed to a measured performance target, as its primary end result.

In an effort to find an objective measure to hang their hat on, objectivist audiophiles invent a scale of good, better, best that they claim can be evaluated by a human in DBT. I think this is an erroneous view.

There are, in fact, objective measures that could be used: standard (and maybe the invention of not-so-standard) audio electronic measurements. Sadly, for objectivists, this would require significant additional expenditures in audio measurement gear (like a $20,000 Audio Precision System Two) and tackling the difficult educational task of learning how to effectively and accurately test audio gear and interpret the data. The problem here is that it is very difficult at best, and impossible at worst, to draw meaningful relationships between measured performance and the listening experience. It seems to me that humans do not have a simple axis upon which audio equipment could be placed in a good, better, best ordering. In fact, personal differences that effect our satisfaction with audio gear exist in MANY domains including: music tastes, audio tastes, hearing physiology, perceptual psychology, listening acuity and experience. This multidimensional array of factors in listening make it impossible for any one set of objective measures that point toward broad listener satisfaction.

Before I go any further I would like to say that I do use objective measurements all the time in evaluating product performance, and I do think that you can sort audio gear into poorly and well engineered piles using these measures. (For those that don

Tyll
Tyll's picture
Offline
Last seen: Never ago
Joined: Oct 15 2005 - 9:38am
Re: Why is Blind Testing Such a Hot Button?

Well, that settles that ... I guess!?

Sorry, I didn't mean to kill the thread. I'm getting the last word a little too easily, seems to me. I thought for sure someone would want to keep the dialog going.

(Maybe I'm just used to Head-Fi where you can get a five page thread overnight.)

dcrowe
dcrowe's picture
Offline
Last seen: Never ago
Joined: Sep 1 2005 - 9:39am
Re: Why is Blind Testing Such a Hot Button?


Quote:
Well, that settles that ... I guess!?

Sorry, I didn't mean to kill the thread. I'm getting the last word a little too easily, seems to me. I thought for sure someone would want to keep the dialog going.

(Maybe I'm just used to Head-Fi where you can get a five page thread overnight.)

This thread started on September 2. Endurance to repeat it is waning....

Tyll
Tyll's picture
Offline
Last seen: Never ago
Joined: Oct 15 2005 - 9:38am
Re: Why is Blind Testing Such a Hot Button?

No worries. I'll hang around and wait for another topic to come up. And this one will surely rear it's contentious head again sooner or later.

ohfourohnine
ohfourohnine's picture
Offline
Last seen: Never ago
Joined: Sep 1 2005 - 7:41pm
Re: Why is Blind Testing Such a Hot Button?

Sorry you missed the slugfest between JA and Arny Krueger, but I suspect you've heard Arny's stuff elsewhere. Added to the fatigue most of us who came in at the beginning have experienced you should remember that we don't have to listen critically to design or review a product, we just listen because we like to.

misterc
misterc's picture
Offline
Last seen: 2 months 1 week ago
Joined: Sep 1 2005 - 11:45pm
Re: Why is Blind Testing Such a Hot Button?

On the other hand, "Single Blind Testing" can be fun, casual, and even informative! Read how the folks at UHF (Ultra High Fidelity) Magazine, a Canadian publication, in their last two editions, have chosen a few speaker cables and interconnects using SBT with some surprising results. I don't think Stereophile has to follow suit; what they do is fine with me and we don't want all magazines to look alike, do we?

Smitty
Smitty's picture
Offline
Last seen: Never ago
Joined: Jan 18 2006 - 6:49am
Re: Why is Blind Testing Such a Hot Button?

All I know is that I wish this issue would be resolved once and for all.

Twenty years ago I was an audiophile on a limited budget. I'd go to these high-end boutiques to listen to their low-end equipment (NAD, Hafler, etc.). Invariably they would have me listen to their high-end stuff, I suppose to walk me up the price ladder. Now at the time I could be spot on with a phono cartridge's frequency slope, but I couldn't tell the difference from one amp to the next even with the salesman standing there telling me how much sweeter this $4,000 tube amp sounded. I had the same experience at my audiophile friends' houses. Night and day difference between speakers and phono cartridges, but when they switched electronics (shrug). McIntosh, Quad, Hafler, NAD, even Technics receivers ... all sounded the same to me.

So imagine my horror when I recently started planning an entertainment room containing an audiophile-caliber system. Nothing's been resolved! And now I'm not on such a limited budget. So now I'm going to have to go back out to the high-end boutiques and spend my time sitting there listening to salesmen tell me how sweet this $8,000 (inflation) amp sounds while I probably still won't hear the difference.

I know it's not in the best interests of the audiophile mags to find out there is no real difference (advertising revenues), but what surprises me is the weight that is put on what is claimed by even the subjectivists to be subtle differences. So I guess the bottom-line is: if you don't hear it, don't spend the money. I just wish I wouldn't have to spend the time just to make sure.

Buddha
Buddha's picture
Offline
Last seen: 2 years 9 months ago
Joined: Sep 8 2005 - 10:24am
Re: Why is Blind Testing Such a Hot Button?

I think people get confused.

It's not your job to answer that question for me or anyone esle, just you. If you can't hear the difference, then for you, that's the answer!

Just be glad you got off cheaper and enjoy! Emphasis on enjoy.

On a related note, I'm the happiest bastard in town when I go to a wine tasting and prefer a cheaper wine. Enthusiasts should rejoice when a less expensive product pleases them as much or more the a pricier one.

It seems we get insecure when not hearing what the alpha males tell us to hear, but if you can get past that, let the party begin!

From what you posted, it has been resolved...and you won! Let us know what you decide to put together, with pics!

Monty
Monty's picture
Offline
Last seen: Never ago
Joined: Sep 16 2005 - 6:55pm
Re: Why is Blind Testing Such a Hot Button?

Buddha, is of course, right.

What fascinates me about this is the rather large differences in resolution that is common in many widely praised products. At the very least, I suspect that this would be apparent even to the most suspcicious of consumers.

Naturally, this is a priority to me and my enjoyment of reproduction and something that I place very high in my pecking order. I certainly understand that many others do not share my view, especially when I consider how many amps I have listened to that do not resolve very well.

Smitty
Smitty's picture
Offline
Last seen: Never ago
Joined: Jan 18 2006 - 6:49am
Re: Why is Blind Testing Such a Hot Button?


Quote:
If you can't hear the difference, then for you, that's the answer!

I did know the answer twenty years ago. But how do I know I won't hear the difference now? I'd say chances are pretty good that I won't. But there's still that nagging little doubt. After all, 90% of the amp makers I'm reading about today didn't even exist back then. It would have been so much nicer if everyone on all these audiophile forums said, "Nah, there's no difference". I wouldn't have to waste my time getting rid of that nagging little doubt by spending a Saturday at the local (well, not so local, but that's another problem) boutique. Instead I could waste it pondering the gazillion additional speaker choices I have now compared to twenty years ago. I think I've narrowed the speaker choices down to about 100 now .

I haven't read about CD players yet. I'm sure there's some kind of debate there too. The debate was just starting back when I bought my first one. I couldn't hear the difference in CD players either. Of course there were only a few to choose from at that time.

I'll probably be kicked off the forum when I come back and say I've bought something like a JVC receiver and a Panasonic DVD/CD player attached by $15 cables from my local Best Buy , but I will have them connected to a nice pair of speakers. At least I can guarantee that.

Buddha
Buddha's picture
Offline
Last seen: 2 years 9 months ago
Joined: Sep 8 2005 - 10:24am
Re: Why is Blind Testing Such a Hot Button?


Quote:

But how do I know I won't hear the difference now?

The other caveat is that nobody can do your listening for you. Ya gotta put in the miles to know what you hear.

If you want validation that everything sounds the same, go buy a Consumers Report and page through their philosophy of hi-fi...it takes less time than the average crap, so at least they're succinct.

Seriously, though, if it ain't fun, don't do it. Then again, why waste time here? I'm starting to get an approach/avoidance vibe...are you just flirting with us until Circuit City opens?

This site is obviously skewed toward people who enjoy the listening/comparison process. If you don't, then that's fine, but please, don't tease!

Smitty
Smitty's picture
Offline
Last seen: Never ago
Joined: Jan 18 2006 - 6:49am
Re: Why is Blind Testing Such a Hot Button?

No tease. I wouldn't waste my time researching this stuff if I didn't have at least a nagging interest.

I just found it quite funny that the same argument can last for 20+ years without resolution. I guess Julian Hirsch didn't have the last say. I mean, I figured by now someone would have set up some digital fake ear that could measure all the waves across the spectrum and at least say, "By the time it hits your ear they are/aren't the same." This is the forum to "rant" about such things, isn't it?

I also found it interesting how audiophiles have shifted. A hard-core audiophile twenty+ years ago wouldn't have been caught dead with a solid-state amp. So what happened?


Quote:
This site is obviously skewed toward people who enjoy the listening/comparison process.

As do I, but I suspect I'll be spending all my comparison time in the Loudspeaker forum. But first I'll at least sit down and listen to a few amps before I go to the people who still think they can hear a difference

Buddha
Buddha's picture
Offline
Last seen: 2 years 9 months ago
Joined: Sep 8 2005 - 10:24am
Re: Why is Blind Testing Such a Hot Button?

Hi, Smitty!

You raise a point that's been puttering around my brain. I think it would be cool to see just what those measurement mics could pick up from a given pair of speakers driven by different amps.

You know how they publish the frequency response curves for speakers under review? I'd like the same thing, but with different power amps in the chain. Same speakers, same measuring situation, just plug in some different power amps.

You are right, if we hear differences, then the mic print out may show those differences.

That would really stir things up with the DBT people, too!

I don't know if they'd differ significantly or not. I can't recall this ever having been brought up in print.

Wouldn't it be cool to compare? Especially if we started seeing differences!

__________________
__________________

For your search, I'd suggest picking your speakers and then seeing if any amps sound different as you listen to your favorite reference recordings. Let us know what happens!

dcrowe
dcrowe's picture
Offline
Last seen: Never ago
Joined: Sep 1 2005 - 9:39am
Re: Why is Blind Testing Such a Hot Button?


Quote:

You raise a point that's been puttering around my brain. I think it would be cool to see just what those measurement mics could pick up from a given pair of speakers driven by different amps.

I don't know if they'd differ significantly or not. I can't recall this ever having been brought up in print.

Wouldn't it be cool to compare? Especially if we started seeing differences!

I have suggested on this forum a few times that it would be interesting to see the data that quantifies differences between systems, to see what the magnitude and nature of the differences are whose audibility is argued about. So far, no one with the equipment to make such measurements has expressed interest. In imaging systems, for example, mean squared error between compressed and uncompressed images has been used to judge the worth of an algorithm. I would be interested to know the relative magnitude of mean squared error (mean squared differences, actually) between systems with a single change in speakers, amplifiers, source, or cables. Would we get the 60% fraction for speakers and less for everything else that was qualitatively speculated upon earlier?

jazzfan
jazzfan's picture
Offline
Last seen: 2 months 5 days ago
Joined: Sep 8 2005 - 8:55am
Re: Why is Blind Testing Such a Hot Button?


Quote:
You raise a point that's been puttering around my brain. I think it would be cool to see just what those measurement mics could pick up from a given pair of speakers driven by different amps.

Buddha, I do believe that you hit the nail on the head - would the mics be able to pick up the same subtle differences that the human ear does? Things like bass control, midrange clarity and all those other wacky audiophile terms we bandy about. I would think not since so much of the final stages of audio design is done by "voicing" the equipment, as is listening to just how it sounds in the real world.

Buddha
Buddha's picture
Offline
Last seen: 2 years 9 months ago
Joined: Sep 8 2005 - 10:24am
Re: Why is Blind Testing Such a Hot Button?


Quote:

Quote:
You raise a point that's been puttering around my brain. I think it would be cool to see just what those measurement mics could pick up from a given pair of speakers driven by different amps.

Buddha, I do believe that you hit the nail on the head - would the mics be able to pick up the same subtle differences that the human ear does? Things like bass control, midrange clarity and all those other wacky audiophile terms we bandy about. I would think not since so much of the final stages of audio design is done by "voicing" the equipment, as is listening to just how it sounds in the real world.

Hi, Jazzfan!

Well, I think this is getting more interesting.

Now we're talking dynamic curve construction.

Perhaps better "bass control" would show up as a faster return to baseline that a system with less bass control. Perhaps we could graph damping factor!

All sorts of permutations come to mind. A dummy head with mics in the ears may show different tracings when "listening" to one amp vs. another.

I may step in doo-doo here, but I can't recall speakers going through the same "impulse" tests that amps do. maybe showing how an amp/speaker combination handels impulse signals would be cool when comparing amps and using the same speaker.

misterc
misterc's picture
Offline
Last seen: 2 months 1 week ago
Joined: Sep 1 2005 - 11:45pm
Re: Why is Blind Testing Such a Hot Button?

Bottom Line - Do you like what you hear? DBT, mics, measurements and statistics won't tell you that...

An aside (since this thread has headed off on a tangent)- Did anyone see the move "Equilibrium" with Christian Bale? It's about a law-enforcement officer (Bale) living in a society that has banned anything that gives people pleasure or causes one to feel an emotion, like art, music, and literature. I can think of some "music" systems that would fit into a place like that.

Buddha
Buddha's picture
Offline
Last seen: 2 years 9 months ago
Joined: Sep 8 2005 - 10:24am
Re: Why is Blind Testing Such a Hot Button?

Hi, Misterb,

I don't want the mics and measurements to tell me what I hear.

Since hi fi is of such interest to me, I think it's fun to think about just what those things can hear and see how it relates to my experience.

For me, the root of hi fi is how fun it is and the enjoyment I can get from something as simple as a great tune. The equipment is a toy, and it seems fun to figure out as much about how the toys do what they do as I can.

It's also fun to challenge myself and try to learn about what I hear and how it relates to what the toys do.

If I like something, it makes me want to play with it, learn about it, try new ways of looking at it, monkey with it...

I agree with you, it's all about what I like, but part of that is seeing what makes it (and me) tick!

Monty
Monty's picture
Offline
Last seen: Never ago
Joined: Sep 16 2005 - 6:55pm
Re: Why is Blind Testing Such a Hot Button?

Hey, Misterb. Stephan eluded to this very thing in his blog earlier today. You might find this interesting.

misterc
misterc's picture
Offline
Last seen: 2 months 1 week ago
Joined: Sep 1 2005 - 11:45pm
Re: Why is Blind Testing Such a Hot Button?

Ah interesting, Buddha. I guess I don't relate to the toys in themselves as much, but if someone writes a paper on why spinning DCC's vinyl remaster of Joni Mitchell's Blue on my LP12 brings tears to my eyes, I might read it...

misterc
misterc's picture
Offline
Last seen: 2 months 1 week ago
Joined: Sep 1 2005 - 11:45pm
Re: Why is Blind Testing Such a Hot Button?

Thanks Monty, that was good! Kinda like Spider Robinson meets Stereophile. And where is Corey Greenberg these days anyways?

Buddha
Buddha's picture
Offline
Last seen: 2 years 9 months ago
Joined: Sep 8 2005 - 10:24am
Re: Why is Blind Testing Such a Hot Button?

That was a great post.

Well done.

If we're ever in the same bar at the same time I'd be happy to buy the first round!

Actually, I'd buy two and see if you could tell them apart!

The more I ponder this whole topic, the more I drift to your way of thinking.

Buddha
Buddha's picture
Offline
Last seen: 2 years 9 months ago
Joined: Sep 8 2005 - 10:24am
Re: Why is Blind Testing Such a Hot Button?

I've got a question I need help with from the DBT people.

How do you shop for or buy equipment?

I know the importance of "proper" DBT testing, so how is you came by the equipment you own?

Also, after you're done shopping, why hang out and discuss hi fi? Your work is done.

A DBT zealot would seem to have less ongoing interest or new opinion in the sound of components than he would hanging out at the golf ball forums talking about ball to ball variation and dimple patterns. At least those can vary in DBT tests.

If all the gear sounds the same, what are you doing here? It kind of feels like an atheist hanging out outside the cathedral trying to change minds.

What do you guys do at the DBT forums?

I can only think of two topics:

1) Fred: Hey! How's that new Sansootchie amp sound?

Karl: Same as all the others.

Fred: Thanks!

2) Fred: Hey, my new Sansootchie amp sounds better than the Hibachi amp!

Karl: Then you're either a deaf idiot or your DBT set-up is flawed. Adjust your DBT aparatus until they sound the same and then you'll have it right.

Fred: Thanks, that's better. What was I thinking?

_________________________
_________________________

I don't mean that meanly, I just figure that a DBT person would shop once and then move on to a hobby where he could tell things apart.

Buddha
Buddha's picture
Offline
Last seen: 2 years 9 months ago
Joined: Sep 8 2005 - 10:24am
Re: Why is Blind Testing Such a Hot Button?

The Bad Science blog got me thinking about this again.

If a manufacturer were to hook up with a DBT expert, they could design an amplifier or preamp and make it with incrementally cheaper parts until, at last, they got a positive finding on DBT and then going back one step and building that product.

In doing that, they would be doing a service to audiophiles everywhere by number one, demonstrating a positive finding, and two, by making a piece of gear that would be optimally priced.

That would revolutionize the industry.

Monty
Monty's picture
Offline
Last seen: Never ago
Joined: Sep 16 2005 - 6:55pm
Re: Why is Blind Testing Such a Hot Button?

bwuhahahahahahahaha!

misterc
misterc's picture
Offline
Last seen: 2 months 1 week ago
Joined: Sep 1 2005 - 11:45pm
Re: Why is Blind Testing Such a Hot Button?

Hi Anonymous,
Your two cents was well worth throwing out there....I don't think most people understand the logic of the science or the science of the logic...that is, how to set up a DBT that be would be "meaningful" and "statistically significant". The main problem I see with DBT of audio products is that the procedure does not answer or even ask the question "Do you even like the sound of products A,B,C, etc?"

misterc
misterc's picture
Offline
Last seen: 2 months 1 week ago
Joined: Sep 1 2005 - 11:45pm
Re: Why is Blind Testing Such a Hot Button?

And then there are "smart" people who use Science and Statistics (in much the same way as Religion) to promote a particular political agenda and lend credibility ("correctness") to their point of view....

Brucest
Brucest's picture
Offline
Last seen: 3 years 2 months ago
Joined: Feb 11 2006 - 7:31am
Re: Why is Blind Testing Such a Hot Button?

Two Simple Questions

With a brief prologue to each.

I read the blind testing discussion both in this forum and in Stereophile with a great deal of interest. And, to offer my bias I have to say I tend to think in terms of accurate reproduction as the gold standard, and as distortion measurements of equipment start having increasing number of zeros to the right of the decimal point, I become more and more skeptical of alleged differences. (Obviously I'm speaking of electronics and wires.) You will note I am a subscriber, not continuous, of Stereophile for many years. Your reviews of equipment, and those of others, seem to often refer to stark differences among components, wires and electronics included. Don't press me but I seem to even recollect the phrase "the differences aren't subtle,) many times.

Q1. How can one reconcile such dramatic statements of differences with the results of DBT? As I read these results, we seem to be rooting around in the limits of statistical significance (as arbitrarily defined by statisticians) to find whether such differences can reliably be perceived in a DBT environment. Based on what I read, I'd expect the degree of discrimination to be 80 or 90%, way above statistical significance, instead of hovering around 50%. How can these alleged stark differences escape ready notice?

So called subjectivists are surely right in suggesting there is no absolute measure of accuracy in the sense that there are a variety of "mistakes"; that any equipment makes, and that these mistakes may differ as to the unpleasantness they cause, and of course some of the mistakes may even be found to be pleasant by many. Objectively, I want to think of this as a weighting function for distortion or mistakes. How much unpleasantness is HD compared to IMD, quantitatively? Such a weighting, if generally agreed to yields a clear set of design objectives for equipment.

Q2. Instead of just passively listening to equipment, why haven't scientific studies been done to examine such weightings among equipment distortions? If such studies reveal a high degree of commonality among listeners, then indeed we have a foundation for design criteria. If in fact differences are all over the map then we have to admit it's mostly a matter of personal preference. Why is not the human ear systematically used as a scientific instrument to establish clear standards for the purpose of designing equipment as suggested above?

Finally, to get back to the issue of accuracy, there is a practical reason beyond engineering principle in its favor. If relatively accurate equipment can be manufactured for relatively low prices, and some folks don't like the accuracy per se, I don't have a quarrel with that. But then, as a cost matter, it seems to me it would be much less costly to "add in" preferred distortions to accurate equipment rather than try to gild inherently flawed stuff. Why spend tens of thousands for esoteric inaccurate tubes, when accurate solid state can be tweaked to achieve that distortion for thousands, or even hundreds?

Monty
Monty's picture
Offline
Last seen: Never ago
Joined: Sep 16 2005 - 6:55pm
Re: Why is Blind Testing Such a Hot Button?

You make some very valid observations, particulary the notion that some people actually prefer a certain amount of distortion. This is precisely why the review of equipment is somewhat subjective in nature.

I think most people approach the design of a product from the perspective that absolute accuracy in reproduction is not possible. From there, they inject their subjective opinion on what is and isn't important (not to mention possible) at whatever price point they intend to offer the product for sale. Some might design the product based on the likelihood that the potential customer would be partnering the component with similarly priced products and make the trade-offs based on that philosophy. Someone else might take the approach of designing a product that requires very fine partnering components to achieve the desired sound.

I really can't think of a practical way to blind test components that react very differently depending on the supporting components, the type of music, optimum placement and so many other variables that a music system encompasses.

This isn't to say that many components couldn't be quickly pigeon-holed, some certainly can be. However, the whole debate on the validity of blind testing centers on a very impractical methodology that would be required to be meaningful.

First, the testing would take a long time between components and assistance from other people in setting up the gear under various conditions dictated by the person doing the testing. This alone makes doing it impractical from the perspective of a magazine and anyone else that values time and money.

Second, this would require conditions intimately familiar to the person doing the reviewing. If you want to blind test mulitiple people, you are going to have to lug a lot of gear around to many different locations. One guy may have a 12x16 listening room and another may have a 22x28 room. They would both be very familiar with the room accoustics and the way it interacts with sound, but not if you put them in a strange environment. If you were to put the first guy in the second guy's room, he might think a particular amp is lacking power, but not if it was in his room.

Finally, you would be asking people to devote their time to proving that the Earth is round. How fun is that?

Experienced audiophiles and particularly reviewers don't have a burning desire to prove anything to anybody, they just listen and enjoy all that the hobby offers. If there is to be a comprehensive blind test organized and debated, I think most audiophiles are more than happy to let the skeptical technophiles have at it.

misterc
misterc's picture
Offline
Last seen: 2 months 1 week ago
Joined: Sep 1 2005 - 11:45pm
Re: Why is Blind Testing Such a Hot Button?

To do DBT for audio equipment "properly" would require so much time, energy and money and the data derived from such tests would be of such limited use and value that the idea of DBT for consumer level audio components is not even worth considering. To put what Monty said in his post in another way, "Why do some people insist on trying to prove that the Earth is Flat?"
On the other hand, using "Single Blind Testing", even in an informal way, can reveal surprising results. See what the folks at UHF Magazine (a Canadian publication) have done in their last couple issues when comparing several speaker cables and interconnects using SBT. Don't get me wrong, I think the writers of Stereophile do an excellent job, too...

Pages

  • X
    Enter your Stereophile.com username.
    Enter the password that accompanies your username.
    Loading