You are here

Log in or register to post comments
tbng
tbng's picture
Offline
Last seen: Never ago
Joined: Jan 13 2006 - 10:00am
John Atkinson on subjective testing

John Atkinson, in several posts on the topic of subjective testing in another post, has presented arguments so void of logic that I must respond. In this comparison of medical, wine, and other blind testing to audio, he wrote, "[The item] being tested is the direct effect of the stimulus. With audio, you must test the stimulus indirectly, through its effect on music." He then goes on to talk about test signals and other issues that can only lead to a reasonable response of,

Monty
Monty's picture
Offline
Last seen: Never ago
Joined: Sep 16 2005 - 6:55pm
Re: John Atkinson on subjective testing

Surely, you are not suggesting that my idea of placing a few fresh tangerines under my CD player is pure nonsense?

BTW, does anyone have an environmentally friendly way to rid your listening room of fruit loving insects? Any advice would be most appreciated.

Jim Tavegia
Jim Tavegia's picture
Offline
Last seen: Never ago
Joined: Sep 1 2005 - 4:27pm
Re: John Atkinson on subjective testing

Mr. Brandt:
(He is biased because he is human, not because he is a bad person.)
Mr. Atkinson may be "flawed" as a person, as we ALL (me especially) are, but Bias, for the most part, is learned. We again are led back to some "adjenda" about the likes and dislikes of the Phile staff. We are not talking about the differences between chocolate, vanilla, or strawberry, we are talking about "gnat farts at 50 feet different". All to often it is not that diffficult.

Have you ever compared CD players from $300 to $3000 and heard any difference? Aren't the bits the same? I would doubt you are listening on a swell $39 DVD player. I have done just that and it was no contest. I will tell you that the most expensive was not deemed to differ "enough" to make laying down $1500 more, but you could not get there with a $300 CD player. Once you get to a certain performance point, going further takes much more engineering and expense.

I would suggest that if you cannot hear the difference between an Adcom preamp and the Mark Levinson preamp you SHOULD buy the Adcom and save your self tons of cash. If they do not test the same (they do not) and may certainly differ due to impedence input and output issues, certainly the quality of components within the Levinson chassis and the overall design are much different (better?), including their sonic signatures.

It is clear you fall into the Bob Carver camp of let me tweek my Phase Linear 400 to have more distortion and I can make it sound like a tube amp. I can make it sound like any amp I want (voicing). You may be able to make it sound different, but better???? Well I can do that if I spend more money on better components. I probably can rest assured you did not buy a GSIC disc.

Mr. Brandt quote:
Spend hours, days, weeks, or months doing this at your leisure making comparison with any source material you wish. If, without knowing to which preamp you

Buddha
Buddha's picture
Offline
Last seen: 2 years 10 months ago
Joined: Sep 8 2005 - 10:24am
Re: John Atkinson on subjective testing

Before making a decision on this debate, I need a timeline.

Mr. Brandt, would this position of yours hold true between my 1974 Pioneer preamp and a new Adcom or Levinson preamp?

If not, then is your assertion time limited? If so, to what duration?

I need to know what year all preamps became identical.

Is a ten year old Adcom preamp allowed to sound different than a 2006 model?

The possibilities are mind boggling.

If these all sound the same, then all I need from you is the date after which this phenomenon took over and I can shop accordingly.

I would also like the dates of identicality for power amps, phono preamps, and CD players.

I remember Stereo Review telling me they all sounded the same, but I didn't save the date on that issue.

Editor
Editor's picture
Offline
Last seen: 3 years 9 months ago
Joined: Sep 1 2005 - 8:56am
Re: John Atkinson on subjective testing


Quote:
Mr. Atkinson alleges himself and his staff to be experts and charges subscribers hard-earned cash to read his supposedly enlightened opinions.

Hmm. I read your comments with interest, Mr. Brandt, particularly when it comes to the single-blind tests I performed at hi-fi shows. These tsts were both fun for all concerned but also educational, I believe, because for many it was the first time they had been involved in formalized critical listening.

However, I did stumble over the above statement of yours. I don't believe I have ever alleged that I and my writers are "experts." Others may regard us as experts, of course, but when I have written about this subject, I have repeatedly said that we are no different in substance from any other audiophile in that all we do is to offer our opinions for others to take or leave as they wish. I have described those opinions as "informed" because, with the many years of experience the Stereophile team brings to what they do, I believe that to be a true description. But the word "enlightened" is, with respect, your own projection.

And as for our subscribers having to part with their hard-earned cash to read those opinions, enlightened or otherwise, I am not sure what you are suggesting. As I said, my readers are free to take or leave what we say. That is how the free market operates and I am a big believer in the efficient operation of a free market.

Thanks for contributing to our forum.

John Atkinson
Editor, Stereophile

Monty
Monty's picture
Offline
Last seen: Never ago
Joined: Sep 16 2005 - 6:55pm
Re: John Atkinson on subjective testing


Quote:
I remember Stereo Review telling me they all sounded the same, but I didn't save the date on that issue.

I don't think any particular date would make much difference. Of course, I would hate to catagorize all issues of Stereo Review as sounding the same...

Jim Tavegia
Jim Tavegia's picture
Offline
Last seen: Never ago
Joined: Sep 1 2005 - 4:27pm
Re: John Atkinson on subjective testing

Exactly.

Now just where did I put that PAT 5 I built that had a channel imbalance of nearly 2db. I am so thankful that Mark Levinson builds in the same imbalance flaw into their new $10K preamp so it would be the same as my old PAT 5. Their new slogan: "Built to a Standard". Really! Just really low.

Now to all of you who own Levinson gear, please step back off the roof and do not jump because you wasted all your money because "everything sounds the same". It has all been just a trick. I wish I was hearing what you heard and was "feeling your pain".

It's also nice to know that the PAT 5 phono section, or the Adcom, sounded the same as the Manley Steel Head. If it even sounded as good as a Monolithic Sound phono stage that would be something. I feel so much better about being poor. Perfect sound forever has been achieved. All for a couple of hundred bucks.

Monty
Monty's picture
Offline
Last seen: Never ago
Joined: Sep 16 2005 - 6:55pm
Re: John Atkinson on subjective testing

Jim is secretly lobbying Apple to introduce a tube based ipod and has vowed never to rest until everyone in the world has heard at least one matched set of Russian Military tubes.

Editor
Editor's picture
Offline
Last seen: 3 years 9 months ago
Joined: Sep 1 2005 - 8:56am
Re: John Atkinson on subjective testing


Quote:
At the show the following year in New York, Stereophile was going through its
Jim Tavegia
Jim Tavegia's picture
Offline
Last seen: Never ago
Joined: Sep 1 2005 - 4:27pm
Tubed IPod

I saw Carrott Top with one on Jay Leno last night. Sorry, it was the RedNeck/turntable IPod. I'll get back to work.

If Gallahger saw it he would just put a water-mellon on it and hammer the heck out of it.

Kal Rubinson
Kal Rubinson's picture
Offline
Last seen: 3 hours 45 min ago
Joined: Sep 1 2005 - 9:34am
Re: John Atkinson on subjective testing


Quote:
Jim is secretly lobbying Apple to introduce a tube based ipod and has vowed never to rest until everyone in the world has heard at least one matched set of Russian Military tubes.

A tube-based iPod dock/amp was shown at CES.

Kal

jazzfan
jazzfan's picture
Offline
Last seen: 2 months 4 weeks ago
Joined: Sep 8 2005 - 8:55am
Re: John Atkinson on subjective testing

Once again Buddha and Monty come through and leave me LOL.

So just what it is with some people? Why do they feel this compelling need to prove that if they can't hear, taste, see, feel, touch, taste or smell the difference between two things than no one else possibly can. And if science can be used to prove that there is no difference between those two things, even better! Bits are bits. Watts are watts. Wire is wire. Etc is etc.

One of my long time friends and fellow audiophiles who can out spend me by several orders of magnitude has managed to put together one heck of a killer stereo system. Am I jealous? Yes, a little, but he's my friend and I'm happy that he's happy. Does his system sound better than my system? Yes. Would I have the same system? No, because there are a few things I would do differently. And that's because I've learned that every little thing can make a big difference in the final sound of a system.

But perhaps more importantly, I've learned that just because I don't "get it", whether the "it" is about audio or music (well maybe not music ), food, wine, cars, art, etc. doesn't necessarily mean that the "it" isn't real but rather that I may not either care enough or know enough. Or it may be snake oil. And if I do care and still don't get it and can't prove that it's snake oil, then I just say "Oh, well maybe I'll get it someday and maybe I won't."

Jim Tavegia
Jim Tavegia's picture
Offline
Last seen: Never ago
Joined: Sep 1 2005 - 4:27pm
Kal: Tube Based IPod

Life IS good!!!!! Now how about the IPod 5.1 dock that will circle my dinner plate and a NXT transducer that will turn my plate into a sub. I need music everywhere.

tbng
tbng's picture
Offline
Last seen: Never ago
Joined: Jan 13 2006 - 10:00am
Re: John Atkinson on subjective testing

Nothing like a little logic to bring out the spleen venting. Some comments on the reactions to my post:

Jim Tavegia demonstrated that he is unfamiliar with scientific method by taking personally my comment on John Atkinson (among others) being biased because he is human, not a bad person. ALL participants in a scientific test are defined as biased. The greatest scientist in the world will immediately end any test in which the double-blind is broken because his or the subject's bias can flaw the test results. Mr. Tavegia asked, "When ST was auditioning a passive preamp vs. active line stages and he heard differences he was biased?" If he didn't take steps to perform comparison testing according to scientific method, of course he was biased. How could that Mark Levinson NOT sound better than a Pioneer? It just HAS to. (However, in this particular example, the higher output impedance of a passive output can interact with the cables attached to it and create a unintended low pass filter. Just science, folks, that a blind test would reveal. Through two different active outputs? I doubt he hears a difference.)

Mr. Tavegia also asked, "Have you ever compared CD players from $300 to $3000 and heard any difference?" The point is, can anyone hear those differences under controlled testing conditions? Telling me you can hear the difference between the $300 and $3000 unit is meaningless unless you consistently heard those differences under controlled conditions. Otherwise, the only clear conclusion is that all differences are likely only in the tester's mind.

John Atkinson wrote, "I don't believe I have ever alleged that I and my writers are 'experts.'" Come now, that's lame. You have the power to substantially affect the success or failure of product lines, the pull to obtain huge amounts of equipment for review, and the clout to hold large conventions that advertise millions of dollars in high-end equipment. You can't feign ignorance while doing that. This sounds like Charles Barkley claiming he wasn't a role model at the same time a million kids wore his jersey and imitated his outrageous behavior.

To the individual who wanted a timeline: Of course aging can affect sound, but Stereophile's claims are always for new equipment.

Lastly, note that none of my critics pointed to a single documented example where their perceived differences of relative equipment sound was proven by a blind test. Note also that many of their responses frequently dipped into sarcasm. Translation: I can't argue with you, so I'll try to belittle you.

I once asked the father of subjective testing, J. Gordon Holt, why he did not like blind testing. Mr. Holt replied with his inimitable candor, "Because I can never pass the damn things!" That in a nutshell is why the "golden ear" crowd fights blind testing so vehemently. It proves the "differences" they hear, often proclaimed with outrageous hyperbole ("blows away the competition" etc.), most likely do not exist.

I would point to the ABX comparator testing executed by Peter Azcel for two decades. I also suggest the following websites: Hydrogenaudio, ABC/Hidden Reference Audio Comparison Tool, Columbia University Department of Electrical Engineering, and the Boston Audio Society ABX Testing Article. There are more. Google yourself silly. The articles don't claim that blind testing is not without issues, but note that the experts, which Mr. Atkinson firmly asserts he is not, almost unanimously vote for blind testing as the only way to discern audible differences. I believe I will be listening to the experts.

Buddha
Buddha's picture
Offline
Last seen: 2 years 10 months ago
Joined: Sep 8 2005 - 10:24am
Re: John Atkinson on subjective testing

"To the individual who wanted a timeline: Of course aging can affect sound, but Stereophile's claims are always for new equipment."

Mr. Brandt,

If aging can affect the sound, why not different capacitors, resistors, transistors, or tubes?

What I really wanted to know was at what point all preamps starting sounding the same. Then I'd know how to do my used preamp shopping.

If they all became identical in 1983, then I'd buy accordingly and save cash.

Also, if they sound different as they age, does that mean I need a fresh one every so often? How often, in your opinion?

Is my 10 year old preamp OK?

Will my 10 year old preamp sound identical to other 10 year old preamps?

You claim they all sound the same, but at some point a crappy 10 dollar job may become sufficiently crappy to sound different. Thus, at what price point am I guaranteed identicality?

Can I go get a 29 dollar preamp at Radio Shack and it will sound the same as the Mark Levinson?

At what price point in the preamp market do they all become identical?

Come on, you made the claim, now quantify it. The line of identicality begins where?

I want to make sure I'm on your side of the line!

Details, man, details!

jazzfan
jazzfan's picture
Offline
Last seen: 2 months 4 weeks ago
Joined: Sep 8 2005 - 8:55am
Re: John Atkinson on subjective testing

Okay Dennis,

Here's what I hope will be a reasonable response, with no name calling or sarcasm.

First of all I basically agree with what you say about John Atkinson's feigned naivet

Monty
Monty's picture
Offline
Last seen: Never ago
Joined: Sep 16 2005 - 6:55pm
Re: John Atkinson on subjective testing

I'm sure you are well versed in the science of electronics and have a firm foundation for your conclusions. Science attempts to remove as much subjectivity from the experimentation as is possible. This is a good thing and totally understandable.

The problem that I see is in trying to approach something from a purely scientific standpoint assumes that science fully understands something that involves human emotions and by default dismisses that which science does not fully understand.This is pretty evident in the substantial study by science in the area of human emotion and the development of drugs to alter the chemical balances to produce a certain affect.

Audiophiles are far less concerned with cause than we are with effect. Our conclusions are based on how something sounds and how that effects us emotionally.

Rather than assume that audiophiles are fooling ourselves, perhaps a scientist would do what most all good scientists do, which is to ask...why? You seem to have reached the conclusion that you understand the why without even approaching the question in a scientific manner, which is to say with a passionate curiousity, "why?"

Malcom Forbes used to begin each month's issue with the phrase, "with all thy getting, get understanding." I don't think you understand what we are hearing because you either don't hear it, don't appreciate it like we do, or have no interest in trying to genuinely hear it for yourself.

As for blind testing, I don't care one way or another. I want to know what impressions John Atkinson or any of his fellow contributors are left with after an extended time listening to a component under conditions that their experiences have shown to work well for them. If they want to do this with their eyes superglued shut, that's ok with me so long as their copy comes with a picture attached.

Buddha
Buddha's picture
Offline
Last seen: 2 years 10 months ago
Joined: Sep 8 2005 - 10:24am
Re: John Atkinson on subjective testing

I like this part: "Note also that many of their responses frequently dipped into sarcasm. Translation: I can't argue with you, so I'll try to belittle you."

It's always a tickle to see a self-proclaimed bearer of absolute audio truth stroll into a room, call everyone deaf and stupid, and then take umbrage if he's not bowed to.

Mr. Brandt, I did not intend derogatory sarcasm. I'm honestly interested in your claims and want more data to help me.

You make a claim, say Mr. Atkinson doesn't walk the walk, and then dismiss him. If I am to trust you, I need the details of your claim - it sounds rather incredible, actually.

If you can answer my questions in a straightforward manner, I will be able to consider your opinion with appropriate credibility.

misterc
misterc's picture
Offline
Last seen: 3 months 2 days ago
Joined: Sep 1 2005 - 11:45pm
Re: John Atkinson on subjective testing

Hi Dennis,
Please enlighten me.... How is the test results of one person doing a DBT on any audio components any more valid than the subjective testing described in Stereophile? In most fields of scientific research, the test results of one test from a sample of "one" does not make a "scientific conclusion".

Editor
Editor's picture
Offline
Last seen: 3 years 9 months ago
Joined: Sep 1 2005 - 8:56am
Re: John Atkinson on subjective testing


Quote:
I basically agree with what you say about John Atkinson's feigned naivet
Editor
Editor's picture
Offline
Last seen: 3 years 9 months ago
Joined: Sep 1 2005 - 8:56am
Re: John Atkinson on subjective testing


Quote:
John Atkinson wrote, "I don't believe I have ever alleged that I and my writers are 'experts.'" Come now, that's lame. You have the power to substantially affect the success or failure of product lines, the pull to obtain huge amounts of equipment for review, and the clout to hold large conventions that advertise millions of dollars in high-end equipment. You can't feign ignorance while doing that.

I addressed this point in another posting. All I am saying is that, whether or not we are regarded as experts on audio, whether or not our opinions carry such currency, contrary to your assertion we do not make that claim. The characterization is a projection on your part.


Quote:
the experts, which Mr. Atkinson firmly asserts he is not...

Why do you feel the need to project in this way, Mr. Brandt? I have not expressed an opinion on my status as an expert either way. As I wrote, I feel it inappropriate for me to make such a claim; that is for others to decide.


Quote:
...almost unanimously vote for blind testing as the only way to discern audible differences. I believe I will be listening to the experts.

Exactly my point, Mr. Brandt. Let such matters be decided by the free market. But the question is begged: If that is your decision, then why would you frequent this forum at all? It would seem that you have chosen your course. Why not allow others the same freedom?

John Atkinson
Editor, Stereophile

jazzfan
jazzfan's picture
Offline
Last seen: 2 months 4 weeks ago
Joined: Sep 8 2005 - 8:55am
Re: John Atkinson on subjective testing

To John and your fine Stereophile staff,

I'm sorry if I may have offended you with my remarks, that wasn't my intention. I realise that Stereophile does often walk a very fine line within the world of high end audio. Your "findings" on the equipment under review are all too often held up as the holy scripture of godly appointed experts, when in fact all you demand is for them to be taken for what they actually are, i.e. the observations of people dedicated to the pursuit of good quality sound and audio equipment.

I am well aware of the difficulties of reviewing as I've done my share in the past. My reviewing was limited to music, mostly jazz (no surprize there), and I often wondered whether I was qualified to pass judgement on so many musicians when I can't even play an instrument. Yes, I understand that a reviewers job is no picnic.

One thing I find funny about this whole thing is that I am engineer, which seems to mean that I should be in the DBT camp, however, I'm also an avid listener and that has landed me squarely on the subjective testing side.

Jim Tavegia
Jim Tavegia's picture
Offline
Last seen: Never ago
Joined: Sep 1 2005 - 4:27pm
Re: John Atkinson on subjective testing

Mr. Brandt,

You still don't seem to get what most of us are saying. You seem to think that is not possible for honesty to over take some Bias you believe is totally life controlling in everybody. I do not.

If I can HEAR a difference in gear, but is doesn't quite pass your mustard as "Pure Science" I am sorry. If something is obvious to me why do I have to put it behind some curtain just to pacify you?

When I asked my wife to listen to CD players and the same manufacturer of preamps you like and SHE hears a difference, with no audiophile axe to grind, are you still going to force us to put it behind YOUR curtain? My wife could give a flip about who made what and why.

If the passive preamp manufacturer chooses to make it output impedance at a certain point, that is his "scientific" point of view. As a scientist did he make a mistake? He decided it, not me or Sam Tellig. Stereophile has made very clear that system synergy is important. His choice may have rendered it less resolving in many systems customers already own. That was his design choice. I do not believe that ST said the piece was not worth owning. Someone else might be very happy with the performace of THAT passive line stage in their system, DBT or not. If someone is left to their own bias and chooses less resolution that is up to them.

There may be "ties" in the game of audio resolution/performance where some equipment sounds so similar that a decision cannot be reached as to subjectively determine which amp, preamp etc is the winner (better?). That is fine.

I am glad Adcom can make preamps as well as they do as they have many customers. I will guarantee you they bought them because they were at a price/performance point that solved their economic/listening criteria. I would not doubt that if $10K for a preamp were a financial issue, most would go home with the Adcom, but if they have the green to freely spend might opt for a Mac, Cary, Levinson, BAT, Krell, or other great piece.

If it is easy for me to dicern the difference in sound from one piece over another I do not "need to go behind the curtain" to satisfy some zealot of the scientific community. If you do, then put your choices behind the curtain and test away. What ever helps you choose a better audio system that is what you should do.

And finally, Mr. Brandt, I will tell you what I think about some in the "scientific" community I have delt with as a Broadcast Engineer, men trained in some science just like me.

I have watched educated men defy the laws of science and push cells sites further apart so satisfy some criteria. They know by proper science, topography and system loading (number of subscribers) where they need to be positioned. They (scientists) chose to ignor the facts. When one bi-directional amplifer is a significantly better performer, obvious scientific testing proves it is, but they chose something else, science lost and so did their subscribers.

I have seen educated men of engineering chose the wrong lightning protection, ignoring the science put before them. After I wore them out with seminars and documented proof they relented and bought our product. What moved them to action was a lightning strike that took one of their sites down with their product of choice. My company sifted through numerous manufactuers to find, what we beieved to be, the best lightning protector for cell sites, bar none. Or for your home for that matter.

The next time I hear a scientist tell me all of you walk on water because you believe people are dishonest unless all is DBT or "put behind the curtain", I will speak up. People who claim to be scientists, but ignore the "laws of science", should be chastised. Being a scientist or engineer carries no weight with me. I have seen too much ignorance. If you can't hear the difference between a $300 cd player and a $1,000 cd player I feel sorry for you, DBT or not.

The ghost of Marley said to Scrooge: "Ebeneezer, why do you not trust your senses"?

No blot of mustard or a slighly under-done potato has ever gotten the best of me in my audio life. I do not lie to myself.

tbng
tbng's picture
Offline
Last seen: Never ago
Joined: Jan 13 2006 - 10:00am
Re: John Atkinson on subjective testing

Some responses:

From Buddha: "If aging can affect the sound, why not different capacitors, resistors, transistors, or tubes?"

If you can show by controlled listening tests that the components you listed when new and operating within designed electrical parameters "sound" differently, a blind test will prove it. Longevity of operation may well be a factor in component selection and certainly relevant to a decision to purchase, but that is not what the "golden ears" say they speak of the latest super capacitor. Stating that a certain kind of resistor "sounds" better is a statement that made inevitably without proof. It would, however, be interesting to compare similarly aged components to see which one held up better over time. That

tbng
tbng's picture
Offline
Last seen: Never ago
Joined: Jan 13 2006 - 10:00am
Re: Mr. Tavegia

Now, that's a rant. I was going to all an end to this. I

Jim Tavegia
Jim Tavegia's picture
Offline
Last seen: Never ago
Joined: Sep 1 2005 - 4:27pm
Re: John Atkinson on subjective testing

You, just like Mr. Brandt wants to talk "science" (the be all-end all of independent thought) and yet we are suppose to take YOUR "belief" that we "want" to see "more expensive" gear as better. The adgenda isssue rears its ugly head.

What you believe about me carries no weight. As I said before, I do not lie to myself. Can you "prove" to me that I am lying to my self about my audio choices? I am paying no attention to the "man behind the curtain". Maybe the dealer is "Pushing" a certain brand harder on my visit, but my "hearing" resists his sales efforts.

Scientists have an adgenda as well. I have met one who has a job designing product components to have "less" reliability. Talk about honesty and integrity in the Scientific Community! Working hard to make expensive "white goods" less reliable so your company can make more money doing repairs and selling parts says much to me about any scientist/engineer who would make THIS their life work.

You choose your components any way you like, DBT or not, and I will do the same. I know what I hear. I have never bought a component because of a name plate or a review. Ever!!!

Editor
Editor's picture
Offline
Last seen: 3 years 9 months ago
Joined: Sep 1 2005 - 8:56am
Re: John Atkinson on subjective testing


Quote:
Mr. Atkinson, can you possibly call it
Editor
Editor's picture
Offline
Last seen: 3 years 9 months ago
Joined: Sep 1 2005 - 8:56am
Re: John Atkinson on subjective testing


Quote:
I'm sorry if I may have offended you with my remarks, that wasn't my intention.

No problem, jazzfan. I don't think I worded my original statement clearly enough. :-(

John Atkinson
Editor, Stereophile

Buddha
Buddha's picture
Offline
Last seen: 2 years 10 months ago
Joined: Sep 8 2005 - 10:24am
Re: John Atkinson on subjective testing

Dennis, don't go, I am enjoying your replies. They became much more conversational than your first post portrayed you.

I do blind but not instantaneous "testing" all the time.

I'll have a buddy change something in my system...polarity on a speaker, a different cable...and let me listen as long as I like in order to figure out what I hear.

I'm good at some things, lousy at others.

I will do "pseudo" science and compare other components.

For preamps, after each switch by an accomplice, not looking at which was put in the system, I'd start right on top of the speaker, listening for grain or noise. It was easy!

Then I'd keep moving back until I made it to the final listening position.

Starting out with my ear on the damn speaker was a silly test, but I needed it to hear differences. Not to judge quality prematurely, but to try and sensitize me what to listen for.

I've yet to meet a preamp I can't do that with. You are welcome, now that I know you better, to come over and take the test with me.

The really scary issue is that I've done it frequently for supposedly "identical "models! Especially in the tube world. No tubes in my preamp, too much variability! Imagine how bummed out I get if I can tell preamps of the same model apart! I'll leave that for a whole 'nother discussion.

There are some things about tossing up a DBT that are deeply flawed, no offense to you.

1) Test subjects should be trained somewhat in what to be trying to listen for.

A random crowd off the street should not be expected to be able to differentiate subtle differences that a skilled listener can easily discern.

Differences would have to be too great before such an unprepared audience would note any differences.

Not to be elitist, but just as a novice may not be able to identify a wine that an expert could easily and repeatably identify, same with unprepared listeners.

2) We should be able to exclude even a trained listener with hearing defects. I actually think the old joke about requiring critics to publish hearing exam results is funny but valid at some level.

With how "bright" some hi end speakers are, it makes me wonder about the age and acuity of some reviewers' hearing.

If we are going to use people as listening instruments, we should be allowed to calibrate the test properly.

3) For some things, instantaneous comparisons aren't sufficient to make very subtle distinctions.

Certain preamps may alter sounds in a way that would require certain situations to hear - poor bass response, small phase shifts, etc. I would think that open ended listening times would be acceptable.

Honest story: I knew two girls who were identical twins. I grew up with them, and even after 15 years of knowing them, they could fool me on an instantaneous comparison. The illusion would fail after a few moments, but even as adults, they could bluff me for a few moments.

It doesn't mean that they were proven identical because I failed an instantaneous blind challenge, it just took me longer to tell them apart than it did for other people.

That's my biggest problem with DBT...the difference may not stand out at the moment of the switch.

Same with comparing cars...I would need time to take each around the track before choosing better or different.

Your point about watch quality is well taken, but I would add that in an instantaneous comparison they would each appear to keep time equally well, it's only over time that they would be able to be told apart.

4) Many people perform very poorly on testing "under duress." The stress/performance curve can turn bright individuals with a command of a subject when they are in their own environment into apparent fools when put on the spot. DBT creates a defensiveness that I think a more leisurely blind test could minimize.

5) I've always wondered this, and maybe you can answer. Has anybody ever done a DBT with microphones as the "listeners?"

I'd be curious to see if a studio or analytical microphone in the listeners seat could tell different components apart.

It would be way cool to get printouts from high quality microphones in DBT's of components and inspect them for differences, it may be very telling!

This is not meant as a criticism of you, it just seems like an obvious thing to try. If the mic can "hear" the difference, that would be a potentially interesting and repeatable test, and comparisons could be made to correlate with what listeners report!

We test speakers with mics, why not toss this test into the mix and see if we can tell the differences between components? Especially when claims regarding bass or treble response are made!

6) It is never freakin' quiet enough for good DBT or good listening with too many people in a room. I would expect every "auditorium" DBT to be flawed with multiple people on and off axis, the drone of a ventilation system hiding in the vents...please don't use those results to make any claims, pro or con.

DBT should be done in a well known (to the listener) space that is frequently used for critical listening. I'd rather take your test in my listening room than anywhere else. If I just get dragged into your audio club's group room and you start hitting switches, nothing useful will be gleaned.

You have to do DBT'ing on someone's home turf.

_____________________________

_____________________________

So, nothing against DBT, but big problems with some of the biases that can be built into a test and making an improper conclusion.

Cheers. Stick around!

misterc
misterc's picture
Offline
Last seen: 3 months 2 days ago
Joined: Sep 1 2005 - 11:45pm
Re: John Atkinson on subjective testing

Hi Dennis,
Again, why should I believe the test results of one person doing a DBT on any audio component? In all fields of scientific research, one test result is insignificant. Please explain.
Thanks.

jazzfan
jazzfan's picture
Offline
Last seen: 2 months 4 weeks ago
Joined: Sep 8 2005 - 8:55am
Re: John Atkinson on subjective testing

Mr. Brandt,

I refer you back to my first post of this thread. And I believe those basic sentiments have been echoed in the posts for several others here. If there was really no difference between a $50K Krell and $500 Sony amp other than the snob factor and judging by how little snob appeal high end actually has, I fear that Krell would be in business for any length of time. But instead the exact opposite is true - Krell is a triving company and their equipment commands high prices on the new and used markets. That sir, is a fact, scientific or otherwise.

Perhaps what troubles so many people like yourself (if I may be so bold as to make an assumption) is the fact the world of high end audio is filled with many people trying to make a fast buck by using some half baked crazy scheme or idea and passing it off as hard science. Just because I believe that a $3K CD player sounds better than a $60 portable CD player doesn't mean I believe that baking my interconnects in a pizza will make them sound better.

Another thing I stated earlier is why I prefer real world listening tests to DBTs. Only through extended listening can one truly understand the capabilities of a given piece of equipment. How does it handle everything one choses to play through it. Rock, jazz, opera, vinyl, CD, etc.

And finally, as stated earlier different systems require different types of equipment. One may need the power and control supplied by the likes of a Krell amp to handle certain speaker loads. Jim T. wrote in one of his posts, perhaps on another thread, about a friend with a new HDTV who thought it looked great but yet didn't even had the proper connections to his DVD player. In a similiar vein, I'm always reading about how MP3s sound just as good as CDs, yes, when played back on your average computer audio system or through a pair of iPod earbuds, hardly worth listening to. Hook up the Krell and Sony amps to those stereos and they are going to sound the same but have them drive a pair of high end speakers and the results will be quite different, as long as you don't use MP3's.

What we are dealing with is a very difficult subject to fully and completely understand and I don't believe that anyone here claims to be an expert. However, we also take a small degree of pride in the audio playback systems we have managed to cobble together for ourselves in spite of all the in-fighting and snake oil and such. There's still penty of work to be done and I'm along for the ride.

misterc
misterc's picture
Offline
Last seen: 3 months 2 days ago
Joined: Sep 1 2005 - 11:45pm
Re: John Atkinson on subjective testing

Hi Dennis,

I'm going to use part of the "scientific method" to say something that's totally outrageous and, admittedly, baiting.

My observation: I have not seen anywhere on this forum, anyone's description of how to set up a DBT for audio components that will yield valid and meaningful results.

My hypothesis: No proponent of DBT for audio components on this forum knows what he/she is talking about when referring to DBT.

I am "testing" this hypothesis in my previous post.

Thanks.

tbng
tbng's picture
Offline
Last seen: Never ago
Joined: Jan 13 2006 - 10:00am
Re: John Atkinson on subjective testing

Go to an ABX Comparator website. It explains it all.

misterc
misterc's picture
Offline
Last seen: 3 months 2 days ago
Joined: Sep 1 2005 - 11:45pm
Re: John Atkinson on subjective testing

I see on this PC ABX website http://www.pcabx.com/#ten_req on the first page that this test is not a "properly designed" listening test. On further reading, I see that I'm supposed to test myself to hear (or not) the difference between a Bryston amp and one from Parasound by playing a music sample through my computer through a Kenwood or Pioneer receiver driving Polk or PSB speakers or a good set of headphones....Okay.....

My observation still stands.

Buddha
Buddha's picture
Offline
Last seen: 2 years 10 months ago
Joined: Sep 8 2005 - 10:24am
Re: John Atkinson on subjective testing

Dennis, that website describes a religion, it is dialog proof, circular.

It's the Leviticus of hi end audio.

Think some things over. That site is classic "my way or the highway" proselytizing. Seriously, you're being closed minded about this.

misterc
misterc's picture
Offline
Last seen: 3 months 2 days ago
Joined: Sep 1 2005 - 11:45pm
Re: John Atkinson on subjective testing

Hi Buddha,

I just don't see good scientific methodology here.

Thanks.

misterc
misterc's picture
Offline
Last seen: 3 months 2 days ago
Joined: Sep 1 2005 - 11:45pm
Re: John Atkinson on subjective testing

Again, on this website http://www.provide.net/~djcarlst/abx.htm , I see no mention of how to set up a "proper" double blind listening test for audio components. In the "data" provided on this site, I see that there's only 1 to 10 "listeners" being tested. In most endeavours of scientific research, such a small sample size does not provide convincing evidence of any kind.

WonkoTheSane
WonkoTheSane's picture
Offline
Last seen: Never ago
Joined: Jan 15 2006 - 4:06pm
Re: John Atkinson on subjective testing

I suppose, Mr. Brandt, that if you would like to tell me how to spend my money, you had better provide it for me.

  • X
    Enter your Stereophile.com username.
    Enter the password that accompanies your username.
    Loading