| Columns Retired Columns & Blogs |
October 14, 2008 - 11:11pm
#1
Double blind test are inherently flawed.
| Loudspeakers Amplification Digital Sources | |
Analog Sources Accessories Featured | |
Music Columns Retired Columns | |
Show Reports | |
Features Latest News Community | |
Resources Subscriptions |
© 2024 StereophileAVTech Media Americas Inc., USA
All rights reserved
F. Alton Everest needs to get to a university neurophysiology course and get some education on what we know in the 21st. century.
Please propose a better method than DBTs, or show how ad hoc listening and reviewing is superior.
Everyone (virtually) knows that DBTs aren't perfect. But they are used in extremely scientific and important fields (e.g. pharmaceuticals) because they are pretty good, and there isn't anything that is better.
That's why for a test to be useful you need a large number of people. And a large number of trials.
Maybe, maybe not. In the recent DBT test written up in the AES journal proving that hi-res audio is indistinguishable from 44/16, many of the test subjects were audio pros or audio students. But even my grandmother (RIP) could tell good audio from bad audio. I'd say anyone who has a keen interest in hi-fi is a usable subject.
--Ethan
Yeah, DBT's are flawed.
Is that the best we can do when faced with the dilemma of how flawed sighted listening is?
If I say, "Dude, you love the 15K wire 'cause you are a suck up for 15K wire," the best retort is, "Well, DBT is flawed?"
Can't argue with that.
Sure. If he was still alive, he could attend mine.
Kal
"Dude, you love the 15K wire 'cause you are a suck up for 15K wire,"
Are you saying that's really every one's issue- how it looks and how much it costs skews the eval process? I've had plenty of opportunities (for a time as a sales guy in college at a high-end shop) where price and appearance did NOTHING to convince me X product was great. All the time you can hear stuff that looks fantastic but leaves you cold in the listening dept. I've even spent good money on stuff that I later had to admit wasn't so great and, although painful on the wallet, was put away or sold at a loss. I just don't get the argument it's ALL about presentation. That just doesn't apply to almost anything else. Sure, appearance can add to the ownership aspect, but like a swanky car, it doesn't add to the real world performance or reliability. A beautiful and expensive meal can look great, but when it hits the tongue it's all over if it doesn't make those taste buds smile. Same for audio. You go to audio shows that feature all sorts of fancy gear that often don't live it to the display value, right?
DBTs are a subject many around here seem to advocate, but I bet it's not how they bought their own system.
No, not saying that. I was picking the ridiculous arguments at either end of this spectrum.
In the DBT debate, I happen to think both extremes inhabit untenable positions.
OK- my mistake. Well- I stand by my points anyway- as the "appearance trumps listening" argument gets hauled out quite frequenctly around here... Cheers.
I wish I could.
I love this stuff.
The fact that different people have different inherent abilities, as well as carried experience, is an issue. For example, when the difference between components, such as wire, is unknown one is testing both whether there actually is a difference as well as the participant's ability to hear any such difference.
A partial solution is large numbers of participants as well as multiple trials, but the problem remains.
As I understand psychoacoustic testing, the difference between samples is typically known. The only issue is whether humans, as a group, can detect the difference.
This criticism does not invalidate DBT, but does place some limits upon what can be accomplished.
I don't need no steenkin' DBT - I buy based on specs alone and that has never failed me once in the 40+ years I've been buying audio gear. The main reason DBT is needed is to convince people they were wrong when they thought they heard "something." Like convincing people who believe hi-res audio beats 44/16, or who think which competent wires you use matters, or who think power conditioners change the basic character of the music, etc.
--Ethan
Buying purely on spec.
Since all the new mass produced gear measures similarly, why even bother with that?
Just head down to Best Buy and grab a random product!
Seriously. The mass produced Pioneers, Yamahas,& Denons all measure the same, so why bother to look at specs?
Well, specs can't be everything, right? I probably agree that if the specs are better for one product vs. another, I'll go with the better specs.
But, what do you do when some specs are better and some are lower? For example, a TT in your price range has slight speed issues, but another has less isolation? And with speakers, it has to get even more complicated. In other words, you sometimes need some sort of tests, especially when technical specs are mixed.
Damn, all this crazy talk.
Specs specs specs...it gets so complicated a guy might turn to something ridiculous, like listening, to make decisions.
Yet those wires do sound different and do make a difference.
That was my point - although I don't know whether you intended to say otherwise.
I'm sorry, I meant to tease Ethan. I hope I didn't seem to be being disagreeable towards your point!
Wow. I skimmed the article and am flummoxed. I'm on the skeptical side for some tweaks and I don't worry much about the more "out there" stuff. But I clearly hear that my SACD's are superior to CD's and I know (I think I know) that that's not imaginary.
What about this excerpt from the article? I don't understand how the testers couldn't hear the difference, yet they heard superiority thus:
"Despite the fact that no one could hear the difference in playback systems, they reported that
Billb, I know this has been brought up before, and I admit that I haven't looked this test. But, it seems that it would be hard to compare formats, because the player may be responsible for the difference. If it is a CD and SACD player, however, it is a little unclear. Could it be that the SACD function of the player is superior to the CD player? It seems that it is possible, but I'm not familiar with the equipment to know.
Good question. However, in this particular series of tests, that seemed to be controlled for. Doggone it.
I figured, thus my caveat of not having read the study. Thanks.
Very good article. I'm with the author:
If fact I agree with most of what you have to say. But I like to keep my mind (and ears) open to other possibilities.
Sometimes I can hear a difference (when supposedly I shouldn't) sometimes I can't(when supposedly I should). So I'm not convinced by the arguments that attempt to explain differences in a universal way.
While were on the topic of listening, I found this article particularly helpful to me:
http://www.enjoythemusic.com/magazine/viewpoint/0601/audiohell.htm
It gives a different approach to comparing different components and I have found that it works surprisingly well (for me). Here is a small excerpt:
And this:
EDIT: corrected link.
You guys argue about the stupidest shit.
Alex, please.
This is a perfectly respectful discussion. If you are not interested, simply skip ahead to the next thread.
But join in if you ahve something to say.
It's a horrible discussion. All you guys are doing is just splitting hairs.
Fair enough.
Skip it and move on.
Agreed fully. Specs were more important 20+ years ago. Today even most cheap stuff is audibly transparent.
Agreed again. When my aging Yamaha receiver died for the second time about 8 years ago I went to Costco and grabbed the cheapest Pioneer they had. It was $170 and had more features than the Yamaha. When one of the digital inputs stopped working after four years I went back to Costco and got another Pioneer receiver with even more features and more power, and this one was only $150. Ain't modern technology grand!
--Ethan
Of course they are. What else do you think there is? Now, it's true that many published specs are incomplete. For example, when was the last time you saw distortion listed for a loudspeaker? But that doesn't mean distortion is not an important spec! It just means the speaker maker doesn't want you to know.
That's a great point and there's no good answer. Me, I have no use for a turntable so that sort of decision is not needed. But your question could be applied to a power amp that has lower distortion but less output than another. In that case you have to assess the importance of the spec in question. If the more powerful amp's distortion is 0.01 percent and the better amp is only 0.007, I'd go with the more powerful amp because audibly there's no difference between 0.01 and 0.007 percent. In both cases the distortion artifacts are too soft to hear.
Agreed fully. Speakers are much more complex to assess. Again, this doesn't mean that specs don't tell the whole story, because they do. But there are so many variables it's simpler to just try a few models in your home and - gasp! - do as Buddha suggests: listen
--Ethan
Exactly. The approach these guys took is scientific because they removed those variables and down-res'd the same program material, rather than trying to compare different material which is pointless.
--Ethan
Me too. I may have strong opinions, but I'm willing to change in a heartbeat when presented compelling evidence. Versus some random guy piping up to say merely "wires do too sound different" which is not evidence.
I could not disagree more. All an accurate system can do is faithfully reproduce what is fed into it. Contrast between instruments, and clarity, and sound stage, ad nauseum, is entirely a function of the processing decisions made by the mix engineer. That, and the clarity of the source as captured by the microphones. For example, even a purist recoding done in a hall with two microphones and no further processing may have clarity issues due to comb filtering, microphone placement, and of course the hall's acoustic quality. A good mix engineer knows how to find those frequencies that harm clarity, or make the sound tubby or shrill, and improve them using an equalizer.
--Ethan
Blech! There are the specs, and there are our own listening impressions. And that duality is one of the many things that make our hobby beautiful.
And then there is something else, which is unmeasureable, as far as I know, and more important than anything else: the influence a component has over its owner to discover and explore new music. I've said this before. If a component doesn't compel you to listen to more music and discover more music, then it sucks -- no matter what the specs are and no matter how it sounds. If it doesn't inspire discovery of new music, it's not worth your time or your money.
Stephen, I think you're off on a tangent more related to Jan and KBK's ideas about "perception." Versus what something actually sounds like. I'm talking only about concrete sound issues. If something sounds different to you tomorrow and the only difference is your mood or how many beers you had, that's something else entirely and unrelated to what I'm discussing. Also, a good sounding system - which BTW will also spec well - is its own reward and can lead one to discover new depths in music etc.
--Ethan
No, my response was in direct reference to your idea that nothing matters but the specs, that you can actually make a wise decision based simply upon a product's specs. I disagree completely. And I think I always will. I am happy, however, that you have this ability; I imagine it must simplify things.
That said, I do think it's time for a beer and some good music.
Man, it's five o'clock somewhere, eh?
I hope your beer specs out to your liking.
I think that is the point. If everything sounds the same then, so the reasoning goes, it is likely the equipment has reduced everything to some common denominator. As you point out the engineer, the mics, all the other gear and how they're used will tend to create individual character. It is therefor likely that the more accurate a system is the more contrasts it will reveal.
The whole idea of "it is live or is it memorex" is mute in my opinion. I think that the best that can be achieved is a faithful representation of what the engineer heard not what actually took place.
If it doesn't sound great without alcohol, it's not a good system. Will you respect it in the morning, a good sytem will sound good without beer, or wine, or anything else. I think anyone who needs to be under the influence to listen is not a good voucher for how good a system sounds, Buzzed listening IS DRUNK listening, to steal a current marketing theme. So I therefore contend, you are listening to sub par stuff, that probably doesn't measure well either, measurements always come out the same, unlike a person's mood or level of inhalants or intoxicants. If you need alcohol to take the "edge" off, then you are using the wrong stuff. Buzzed listening is dRUNK listening, friends don't let friends listen to crap.
"mute"? Maybe MOOT?
I buy only on specs, and everything so far has won. Specs is it, ears change, moods change, specs are forever. If you like how something sounds and it has 3% THD then you are not interested in accuracy, you are interested in totaly colored, revised information. Can you still call it colored? Or is it, pigmentaly enhanced audio signal.
You caught me. Yes I mean moot.
(although it could be argued that something that is moot leads one to be mute about it.
)
Man, when you have Dup correct your grammar, it's time to take a break and get loaded!
No cus' just indicating that it is MOOT, means you where not quiet or mute about it, doesn't fly.
See what I mean Scott? I'm rolling a bomber for both of us!
I find Budweiser, with it's freshness dating to be the best spec'ed beer. Far more accurate than those iffy, all-over-the-place micro brews. What a crap shoot those are, especially if you've had a beer or two!
Yes, it's hard to beat the security and feeling of self-satisfaction that comes with knowing when your beer will expire. I'll bet that's what Mr. Whiner drinks.
Dave
You got it pal! This is exactly what's so wrong-headed about seeking components and systems that add "color" to the sound. With that kind of gear the same color is added to everything. So that makes things more bland than accurate gear. Hey, if you want some color buy a parametric equalizer and twiddle to your heart's content.
--Ethan
I'll be right over, please make that fattie fat.
--Ethan
I agree Stephen. Specs should be the excellent, but listening is the final test.
If one gets a chance, compare a solen polypropylene to Auricap polypropylene capacitors. Both same value, both with nearly identical DA and ESR, less than .03%. the component measures the same under static conditions, yet the sound between caps is quite different under musical conditions.
As far as sitting position and possible perceived room interactions, one can test by simply sitting and then move one's head 3-5 inches in all directions and notice what sonic changes, if any, occur.
As mentioned above, specs are important and the closer the better, but listening is the final test.
Dr. Edgar, Audio Research, Conrad Johnson, Atmasphere, and many other companies use "selected" capacitors, not the most expensive or inexpensive, but what performs best in their designs. And some companies started this before the internet was prevalent, decades ago.
By the way, if doing dbt tests, be careful of "habituation to stimuli" and cochlea fatigue problems. This means limit ABs to 3-4 max per session (also limits time involved per session) and volume levels.
Take care.
Steve
I told you previously, my COCHLEA is summer's eve FRESH, it doesn't get unfresh with time, thus Summer's eve. If your hearing changes after only a few minutes of listening, your COCHLEA is messed up. So if you are at alive event, you only listen for 3-4 minutes and leave? Don't think so, but then why would you only listen to a system for such a short period of time...your concepts of listening are weird. Is that part of your specially developed methods? Makes no sense. You worry too much about COCHlea not about sound, if your COCHlea is fatiguing so soon, perhaps you need meds? ED? EAR DYSFUNCTION?
The room is gonna have a much bigger effect on how stuff sounds than a capacitor type, as long as a cap is the correct type, spec for it's ckt function. Temp stability, etc. Your concepts are bizzare indeed.
It is quite normal for "habituation to stimuli" and "cochlea fatigue" to occur between 65 and 80db spl. Do some research, read articles from years past. I recently contacted several hositals and was referred to a professor who teaches at a famous east coast university for information.
So AB testing must be limited to 3-4 back and forths for each session to minimize these effects. It is science my friend.
Nothing weird.
Take care ok.
Pages