The 2011 Richard C. Heyser Memorial Lecture: "Where Did the Negative Frequencies Go?" Case Study 2: Loudspeakers

Case Study 2: Loudspeakers
During a visit to Canada's National Research Council many years ago, I noticed, stuck to the wall of the prototype IEC listening room, a page of results from one of Floyd Toole's seminal papers on the blind testing of loudspeakers. The scoring system was the one that Floyd developed and that I subsequently used for blind tests at Stereophile: "0" represents the worst sound that could possibly exist, "10" the perfection of live sound. On this scale, a telephone, for example, rates a "2." The speakers in Floyd's test pretty much covered the range of possible performance, yet their normalized scoring spread, from the worst to the best, was just 1.9 points. Other than some pathological designs, the audiophile speakers I test for Stereophile probably cover a range of 0.5 point on Floyd's scale; but, as I pointed out in my 1997 AES convention paper, our reviewers' ratings generally follow Floyd's findings: that people tended to prefer speakers with flat response and controlled dispersion.

But not always.

There are vocal advocates of high-sensitivity horn speakers. There are equally vocal advocates of low-sensitivity, large-panel speakers. The horn advocates, among whom was Stereophile's founder, the late J. Gordon Holt, enthuse about these speakers' "jump factor." I used to ask Gordon what he meant by that: was it dynamic range? No, it wasn't. Was it low distortion? No. His reply was always that there was something about the presentation of these speakers that more closely resembled what he experienced from live classical music.

What about panel speakers? Being dipoles, their interaction with the room is very different, so maybe that's why their advocates love them, not even considering the fact that, with a large panel, the listener is almost never far enough away from the speaker to be in the farfield, the result being more bass—which everyone likes, at least in two-channel audio. Of course, there are no cabinet resonances because there is no cabinet, and perhaps people's preference is due just to the fact that he very large radiating area inherently ensures more linear behavior when moving the same mass of air as a much smaller cone driver. But panel speakers' impulse responses look terrible. Well, I say "impulse response"; what I actually publish and use for my reviews is not the response of a loudspeaker to an impulse, but the impulse response calculated by cross-correlating an MLS signal or a chirp signal, based on the loudspeaker being a perfectly linear system. (John Vanderkooy has investigated the shortcomings of this assumption.)

Perhaps, then, the problem with measuring panel speakers is that the calculated impulse response does not accurately reflect the behavior of a large panel in the same manner as it does a conventional speaker's. At the 1989 AES Convention in New York, mathematician Manfred Schroeder discussed Chaotic systems in audio and mentioned, in passing, that a Chaotic system tended to produce subharmonics. And indeed, when I have measured panel speakers I have found that they produce subharmonics. You feed the panel speaker a 1kHz tone and in its output you find a 500Hz tone as well as the usual distortion harmonics at 2, 3, 4kHz, etc.

So in a large panel speaker, while the average position of the diaphragm follows the driving signal, the motions of the individual elements of that diaphragm are Chaotic. The speaker sounds much better than its measured behavior suggests, as all that randomness is integrated by human hearing's latency.


Live vs recorded; Steinway D vs Vivid GiyaG1 speakers

Or do people like the sound of panel speakers because the intensity of the sound they produce—the sound power per unit area of the radiating surface—is closer to what people hear live? This question was triggered by a live vs recorded demo in which I took part in 2009, in Maryland. Two pianists performed for an hour on a well-prepared Steinway D grand piano, which I recorded at 24/96. Then, after a break, the audience heard the recital again, this time played back through a system. (This meant miking the piano far more closely than I would do for a conventional recording, to avoid a double dose of the room acoustic when the recording was played back.)

The system was full-range, with low coloration and sufficient dynamic-range capability that I was able to match sound-pressure levels. But even though the SPL and the sound of the piano matched closely, listeners felt that the reproduced sound just didn't have the "bigness" of the real thing. I wasn't sure what they meant by "bigness." Yes, the piano's dispersion would excite the room very differently than would a pair of speakers, but perhaps these listeners were suggesting that they were aware that the intensity of the sound was not correct, even when the SPL and response were correct.

So when people say they like panel speakers that measure poorly, perhaps they're responding to those speakers' more accurate intensity, at least when it comes to large sources.

Food for thought?

COMMENTS
JohnnyR's picture

......blind testing mentioned by a person that doesn't believe in doing them. Aren't they "too difficult" according to yourself? Wait......nevermind just don't answer that because I know it will end up with you Mr Atkinson making some EXCUSE as to why you can't do blind tests for the magazine when it comes to reviewing products. ZzzzzzzzzzZzzzzz.

John Atkinson's picture

JohnnyR wrote:
Sure is a lot of blind testing mentioned by a person that doesn't believe in doing them.

A man sees what he wants to see and disregards the rest, eh? In the 15,000 words of this preprint, there are just a few paragraphs devoted to a discussion of blind testing.

JohnnyR wrote:
Aren't they "too difficult" according to yourself?

As I say, I took part in my first blind test 35 years ago and since then have been involved in well over a 100 such tests, as listener, proctor, or organizer. My opinion on their efficacy and how difficult it is to get valid results and not false negatives - ie reporting that no difference could be heard when a small but real audible difference exists - was formed as the result of that experience.

John Atkinson

Editor, Stereophile

GeorgeHolland's picture

In other words JohnnyR, Mr Atkinson DID find excuses not to use blind testing but found them worthwhile enough to mention them in his Heyser speech funnily enough.

JohnnyR's picture

A. Join a band as a second rate bass player

B. Fail to get a record contract

C.Pretend he's an electrical engineer

D.Get an offer to ruin.....ermm I meant run Stereophile

E. Proceed to ruin........ermmmmm excuse me......run Stereophile like Hitler would.

F. Spend his time talking online instead of actually doing any real work.

G. Make excuss about his busy schedule and why he can't do DBTs

H Profit!!!

Regadude's picture

Johnny makes plywood boxes in his mother's basement, and then claims he is a speaker expert.

Trolls audio websites. 

 

That is all.

ChrisS's picture

JRusskie seems to have become a failed audio (and DBT) expert in one easy step...

 

A. Failed...

dalethorn's picture

Hitler? I don't know whether to laugh or cry.

Ariel Bitran's picture

also, brings both of these to mind:

http://en.wikipedia.org/wiki/Reductio_ad_Hitlerum

http://en.wikipedia.org/wiki/Godwin's_law

ChrisS's picture

There it is, again...

ChrisS's picture

I think a synapse just exploded...

dalethorn's picture

I asked a question of Harman Testing Lab in the Computer Audiophile forum, and post #11 is the answer, which I thought was perfect. i.e., let the testee control the switch, so whatever adjustment time they need they can accomodate.

http://www.computeraudiophile.com/f12-headphones-and-speakers/behind-harman-testing-lab-14112/

Other observation: The comments in this Stereophile article about the noise floating free of the recording due apparently to having a higher playback bandwith - that made my day.

On R.C. Heyser: His description of the 1971 Sylmar quake, hearing a low-frequency "tone" prior to the shaking, got him started on comparing earthquakes and woofers and low-frequency playback in rooms, and he got beat up on a lot for those writings. But having been involved with both, those observations of his were like gold in the bank. Saved me lots of money.

Regadude's picture

Aren't you the dude who "tested" the Shure SRH 940 headphones and claimed they were almost as good as the 800HDs from Sennheiser?

I own the SRH940s. I was somewhat willing to believe you, but when I saw that your source was an Ipod, I just flicked your review in the trash.

You chose a 200$ source to test 325$ and 1500$ headphones?  And you believe your results mean something?

 

no

dalethorn's picture

My review of the 940, agreed upon by more than 95 percent of respondents, was vetted with specific music tracks listed in the review and subsequent comments, using ipods alone, ipods with analog amps, and desktop DACs and amps. Some of the naysayers are either impatient or just grumpy and disagreeable, which the comments in this Stereophile article perfectly illustrate.

Regadude's picture

How nice of you to call people who question your methodology grumpy, disagreable, etc. I question your methodology because it makes no sense to test 325$ headphones with an ipod. It makes even less sense to test 1500$ headphones with an ipod...

I actually listened to the Shure SRH940 and HD800 today, using my ipod Nano. You are correct, they both sound almost the same. They both sound like shit!

I know from experience that these headphones really shine with a good source, and especially, proper amplification. As good as the Shure is (I own them), it is no match for the HD800 when both are hooked up to decent equipment.

Your test is like testing the acceleration of a Ferrari and a Prius in your 12' driveway. You won't get much better results from the Ferrari in such a limited distance.

As for the 95% who agree with you, what gear did they use (ipod)? Were those 95% SRH940 owners and fanboys? 

dalethorn's picture

Saying "they both sound like shit" when one of those is a revered $1500 headphone of great pedigree and popularity among actual audiophiles is telling of your judgement. I would suggest taking a deep breath, pick a headphone and some music tracks you like, then tune out the matrix and just enjoy your music. You'll be less grumpy that way.

Regadude's picture

Don't try and hide behind my words! They both sound like shit through an ipod! What a weasel you are. I did not say the 800HD sounded like shit... BUT IT SOUNDED LIKE SHIT THROUGH AN IPOD. You rigged your test, Dale. 

Again, you use insults...

 

dalethorn's picture

A long time ago one person on a big audio site challenged people like you to be more specific instead of throwing words like s**t around. There were just two persons who offered test tracks with which to compare those headphones, but everyone else (like you) just threw out expletives, and a couple members even made serious threats. Now I have over 30 years experience comparing headphones like the Stax and Sennheiser series, and quite a bit of time invested with the Audio Engineering Society as well as reading Stereophile magazine, so even if nobody in the world were to agree with my conclusions, I'm certainly qualified to test and evaluate these devices. I'd suggest you refrain from purely ranting with words like s**t and offer specific testable facts, or just go away.

This article may help you:

http://dalethorn.com/Headphone_Ipod_Versus_Amp.txt

dalethorn's picture

BTW, and speaking of Prius and Ferrari, you may be one of the better off who don't mind paying $1500 for a headphone and then shelling out $2000 for a headphone amp. But let's pretend for a moment that you have to pinch pennies to get those luxurious goods, so when you finally have them you really appreciate what you have. In this latter case, were you to stumble across an ipod touch or iphone of recent manufacture, with the new Earpods ($29 purchased variety only) being driven by the Dirac DSP player, and had an opportunity to listen at length, it might make you ill to realize what you got for your $3500.

Regadude's picture

My headphone amp is a whopping 400$ (CI Audio VHP 2).

I do pinch pennies. The VHP 2 and the Shure 940 are mine.The Sennsheiser 800HD I borrowed from a friend.

Stop trying to cloud the issue with non relavant issues.

Regadude's picture

How about we let Stereophile test both headphones! First through an ipod, then with decent equipment!

Both headphones sound similar through an ipod.

Plug them into a receiver, they sound less like shit.

Plug them into a decent headphone amp, and they both sound MUCH BETTER. BUT the Sennsheiser pulls away and wipes the floor with the Shure.

Buy yourself a CI Audio headphone amp (the one I use), or any other decent headphone amp Dale... Before buying a 2500$ DAC to install on an ipod.

DUH...........................

GeorgeHolland's picture

The continuing saga of ChrisS being a troll and all around noncontributing poster. Classy act there ChrisS.

Mostly what I got from Mr Atkinson's article was the amount of photos of himself rather than Mr Heyser. I suppose a memorial lecture should be all about the presenter and how he sees fit to turn it into reasons why Stereophile does what it does regarding testing and the lack there of.

"Yes, what you think you are hearing might by dismissed as being imagination, but as the ghost of Professor Dumbledore says in Harry Potter and the Deathly Hallows, "Of course it's all happening in your head, Harry Potter, but why on earth should that mean that it is not real?"

Oh I see now. Stereophile's philosophy is based upon a ficticious Harry Potter character who spouts that what we hear is real.I'll be sure to let those people being treated for such voices in their heads that all is well and medication is not necessary.

"Footnote 2: For a long time, I've felt that the difference between an "objectivist" and a "subjectivist" is that the latter has had, at one time in his or her life, a mentor who could show them what to listen for. Raymond was just one of the many from whom I learned what to listen for."

Well if only that was true. Your readers tend to all be subjectivsts who aren't trained what to listen to but blindly ( I made a pun there since blind testing is not wanted at Stereophile) believe whatever they think they hear as the truth when in fact measuremnts will show that either they heard nothing related to the measurements or perhaps it's Dumbledore once more in their heads, make believe or wishful hearing.

All in all I found your presentation 90% about yourself and Stereophile and reasons why you defend subjective listening and gloss over things like cables and doing serous testing and throwing in various quotes and ambiguous stories barely related to the subect you were supposed to be talking about, unless that subject was muddling the understanding of Stereophile and telling everyone your own personal history instead of Mr Heyser's.

"Third, one well-known skeptic sitting in the audience tonight criticized my abstract a few weeks back on the grounds that I am just offering "hypotheses about stuff that might be just to stir the pot, while offering no real explanations."

Class act there Mr Atkinson, speak out and criticise someone you don't like or agree with in the audience who then has no opportunity to speak up and address the audience themselves. Very childish and immature.I already see how you do the same on here. "My way or the highway"

JohnnyR is once more sadly correct, Blind testing is not and wil not be used by Stereophile. Just because you Mr Atkinson, know how to push a few buttons to measure speakers, does not make you even close to the same league as Mr Heyser was. He furthered the cause of objectivity and made Audo magazine the best audio magizine there ever was. The Audio Critic is a very close second. Mr Aczel used to be one of the worst subjectivists around untill he admitted to being such and saw the light and spoke out against foolish subjective reviews. Testing will always be the most important part of audio magazines. Simply letting reviewers spout off about "blacker backgrounds" "lifted veils" or other such nonsense belittles both the reader and the so called reviewer not to mention the product under "test". Truth in testing is what Mr Heyser strove for not some silly review about rainbow foil, magic bowls or pebbles.

John Atkinson's picture

George Holland wrote:
Mostly what I got from Mr Atkinson's article was the amount of photos of himself rather than Mr Heyser. I suppose a memorial lecture should be all about the presenter and how he sees fit to turn it into reasons why Stereophile does what it does regarding testing and the lack there of. I found your presentation 90% about yourself and Stereophile...throwing in various quotes and ambiguous stories barely related to the subect you were supposed to be talking about, unless that subject was muddling the understanding of Stereophile and telling everyone your own personal history instead of Mr Heyser's.

You have misunderstood the nature of the lecture, GeorgeHolland. The Richard C. Heyser Memorial Lectures are not intended to be _about_ Dick Heyser but instead are offered to honor his memory. The invitation for me to be the October 2011 lecturer clearly stated that I was free to choose any subject I felt appropriate; my triple career as a musician/audio engineer/audio editor and the conclusions I have formed as a result of my 4 decades' experience in those careers were felt by the AES Technical Council members to be an eminently suitable subject. 

George Holland wrote:
Class act there Mr Atkinson, speak out and criticise someone you don't like or agree with in the audience who then has no opportunity to speak up and address the audience themselves. Very childish and immature.

The skeptic in question had emailed me before the lecture to let me know he would be in the audience and would take an active part in the anticipated Q&A session. I included this mention in the preprint to give him the necessary opening. As it turned out, he didn't attend, but I saw no reason to delete the point he made.

George Holland wrote:
Just because you Mr Atkinson, know how to push a few buttons to measure speakers, does not make you even close to the same league as Mr Heyser was.

I agree. As you can tell from my discussions of Dick Heyser's writing and thoughts in the preprint, I have an enormous amount of respect for what he achieved. I have never claimed to be his equal, nor would I. But again I must emphasize that the AES Heyser lectures are not intended to be _about_ Dick Heyser. That is your misunderstanding, I am afraid.

John Atkinson

Editor, Stereophile

Rolli666's picture

I am not sure as to why a person would even want to respond to these posts. Some problems really are better ignored as to invoke the spirit of go away.  I for one appreciated this lecture for what it is thank you.

John Atkinson's picture

Rolli666 wrote:
I for one appreciated this lecture for what it is thank you.

Appreciate your comment. It is rare that a magazine writer has the opportunity to gather all his thoughts in one place and I owe a debt of thanks to the Audio Engineering Society for inviting me to do so.

John Atkinson

Editor, Stereophile

 

ChrisS's picture

Whoa! There goes another synapse! Now they're both brain dead... I duelled extensively with JRusskie re: blind testing on the Forums and I daresay, he shot himself many times with his own arguments.

Saga? Can't wait for the movie version!

Sorry, Ariel... I'll try not to let it happen again.

Dunadan's picture

...which is a valuable insight, I think, from this article that Mr. Atkinson has generously preprinted. His stories--which, though I am not a longtime reader, I presume also reflect the history of Stereophile--mirror the tensions found in the philosophy of science between "reality" and "perception", and the question of whether our observations are truly disinterested or if our minds shape how and what we see (or hear). And, really, what exactly does a blind test resolve? That the reviewer likes one piece of equipment over another, presumably without factoring in aesthetics or cost? But "liking" one thing better than another is still a subjective judgment, and who's to say YOU would share that judgment? Never mind the problem of translating sounds heard by another into words on the magazine's page and then back again into the sounds heard by the reader when he or she auditions the gear. Or the issue of whether sound quality can be adequately described by a series of measurements, any more than colour can be described strictly in terms of wavelength, or consciousness in terms of vibrating atoms in your brain. That's not to say these things aren't significant, but to say that they add up to the sum of our human experience is absurd.

Thanks for a reflective and enjoyable article (at least in my mind).

GeorgeHolland's picture

The purpose of blind testing is soley to see if you can hear a difference between two units. After all the subjective rants of "oh this lifted several veils of crud from the liquid sound" or "I heard a definite improvement" I find it sad that Sterophile keeps saying that DBT or even SBT are not a valid way to test those claims. Either you can hear a difference or you can't yet those that claim they do shy away from proving that fact with a blind test. Afaid they might be shown to be wrong is my opinion, so Stereophile allows them to go merrily along in their own self delusions while spending way too much money on sham products.

ChrisS's picture

In a properly set up DBT, let's say for two amplifiers, what exactly will the listener be listening for?

 

(And what if the lamb heard a difference, but not the shepherd boy?)

GeorgeHolland's picture

Are you seriously so dense that you can't figure it out for yourself? Perhaps that's why you depend upon Stereophile to tell you what to buy?

http://home.provide.net/~djcarlst/abx.htm

http://www.matrixhifi.com/contenedor_ppec_eng.htm

Read and learn.

ChrisS's picture

So Georgie,

If the lamb can hear that Amp A can make the test system sing in a range of 10hz-60khz, but the shepherd boy can't hear in that range so can't tell the difference between Amp A and Amp B, which amp should the shepherd boy buy?

GeorgeHolland's picture

No you may address me as George or Mr Holland. Georie is an attempt at making fun but then again that's about all you know to do anyway.

Please ask a relevant question or just shut it.

The lamb would first have to prove they can hear a difference with a dbt

The shepard can buy whatever he wants.

The big bad wolf tried to convince both that the expensive boutique amp was the one to buy

They told him to shove it and bought a less expensive but well built amp and they all lived happily ever after except for Mr Big Bad who soon folded his shop due to no sales.

ChrisS's picture

So Georgie,

Let's make this question relevant....

Let's say we're testing two amplifiers with two listeners. The first listener is an 18 year old young lady who is trained in classical piano at a Grade 10 music conservatory level. She can hear that Amp A has an excellent range 16hz-40khz through the test system, but even though Amp B doesn't have the same range, she likes the "sound" of it better.  The second listener is the shepherd boy who's grown up now. He's 54 years old, likes big band jazz, but his hearing has been damaged by working with heavy machinery without hearing protection. He can't hear a difference between the two amps.

 

Which amplifier should the ex-shepherd buy?

GeorgeHolland's picture

The 18 year old can hear to 40KHz?  Were her parents bats?

"Like" doesn't have anything to do with blind testing. You don't pick which one you "like" you see if you can tell WHICH amp is playing. You don't even know how a dbt test is run I can see already.

 Who cares which amp they buy?  Maybe the people selling them do but that is completly irrevelant to dbt. You have no clue as to what blind testing is all about.

ChrisS's picture

Georgie,

Are you a real person or a computer generated figment from JRusskie's russian clone of an old IBM PC? Do you know anyone with normal hearing? Do you know how real people shop or do you isolate yourself in the sanctuary of your closed mind and order everything on-line after reading extensive reviews in Consumer Reports?

ChrisS's picture

You call those proper Double Blind studies?

GeorgeHolland's picture

Oh so you are an expert on double or single blind studies. The ABX system is a proven dbt way to do things. Just because the results have you so upset, you claim the people doing the testing are doing it wrong? Laughable. Tell me some more jokes.

ChrisS's picture

...Swinging down the street so fancy-free..."

 

In fact, Georgie, my major(s) for my undergraduate degree were in Developmental Psychology (including Perception) and Statistics (including Research Methodology). So yes, the set up and methodology shown in those links are crap and the results are laughable...

ChrisS's picture

Get thee to a local college and enroll in a first year research methodology course. Have fun learning!

GeorgeHolland's picture

Either address me as George or STFU you stupid little boy. I think you majored in being a twit and smart ass. Who can take anything you say as serious? Grow the fuck up already. You act out like a lil boy with the IQ of a rock.

Go tell the people who make the ABX test system what you just said and see how they laugh you out of the room. You bring nothing to this discussion other than what you don't agree with , with zero facts to back up your claims. Come on show us all how the ABX test methods aren't any good or why the blind testing done in the othe link was faulty. Better yet tell Harman Kardon that their blind testing techniques are faulty and not worth doing.

ChrisS's picture

Georgie,

You must be running out of neurons if you don't trust your own eyes and ears. Yes, the facts are out there.

JohnnyR's picture

.is what SBT and DBT are all about. Using your eyes to test audio products? Well yes I can SEE that YOU would have to look so you would know which one is "better".

ChrisS's picture

Whose ears? Why?

John Atkinson's picture

GeorgeHolland wrote:
I find it sad that Stereophile keeps saying that DBT or even SBT are not a valid way to test those claims.

Please do not put words in my mouth. That is not what I have said. What I _have_ written is that to design a blind test that limits the variables to just that which you are interrsted in and that produces valid results when there is a small but real audible difference is complicated and time-consuming. The literature is full of poorly designed and performed blind tests that have been proclaimed by audio skeptics as "proving" that there are no audible differences. Such people demonstrate both their ignorance of the Scientfiic Method and their unquesitoning faith in "Scientism."

John Atkinson

Editor, Stereophile

GeorgeHolland's picture

JohnnyR was right more EXCUSES.

It's pretty simple Mr Atkinson but then having the will or gumption to put a dbt into the line of testing is the first thing you have to have,

You obviously don't have that or just don't care so it's a moot point anyways.

JohnnyR's picture

.nor ever will when money is involved.

dalethorn's picture

I would hate to see magazines and websites like Stereophile become intimidated by naysayers who demand "proof" of everything they say, in advance or after the fact. There's a lot of that in mainstream media, due no doubt to controversial topics and false information being fed to reporters. But come on, people - this isn't a mainstream news outfit reporting life or death stories. We have here an incredibly rich article full of facts that can be researched and questioned with references that are well established over time. Instead we have people questioning the author's motives or his pursuit of truth? I think people who are looking for "The truth" should be looking in a religious forum, not a hi-fi forum. There's very little you can "prove" on these topics - the value here is the very informed opinion that costs you nothing.

Ariel Bitran's picture

nice comment Dale. as an attendee of this lecture, i can tell you it was surely enlightening.

GeorgeHolland's picture

I think you have it all backwards there friend.....

"I think people who are looking for "The truth" should be looking in a religious forum, not a hi-fi forum. There's very little you can "prove" on these topics - the value here is the very informed opinion that costs you nothing."

Religion is based upon belief and subjectivists cling to their belief, they don't go looking for proof or the scientific method. No you can't prove very much when the reviewers use subjective say so instead of actually measuring the units. Opinons are a dime a dozen or even less than that and worth next to nothing. Just look at all the opinions here.

dalethorn's picture

No, George, you don't get it. You're still stuck in religion, looking for proof of something. Here you get 'information' only, and if you want 'proof' of something, you have to do the work in proving it to yourself. What *you* believe, outside of yourself, is purely opinion. Perhaps all these people looking for truth or proof are just lazy, and trying to intimidate others into doing the work for them. Like bullies.

JohnnyR's picture

......the facts there pal. Your "arguement" is FLAT.  The only "info" Stereophile shows us is what the WANT to show us. Cables, power cords, magic bowls are off the list of even testing them in anyway what-so-ever. The reason? Ohhhhh we really don't know how to test those duhhhhh. How lame an EXCUSE is that?  Stereophile is SUPPOSED to be a magazine for information NOT excuses.

"Perhaps all these people looking for truth or proof are just lazy, and trying to intimidate others into doing the work for them. Like bullies."

 BWAHAHAHAHAH!!! that has to be on my "Top 10 WTF Things of 2012"

 You DIDN'T just say that did you???? So let me get this straight, Stereophiles job as you see it is to just fling out "say so" and it's up to the readers to wade through the muck and mire of those reviews to try and grasp one little bit of truth? Yeah right, you must really love it then because they rarely show any truth at all.

Ariel Bitran's picture

why is it so hard to accept that double-blind listening tests are difficult to achieve as JA has explained in his lecture?

the fact that our existences are commandeered by individual perception based on thousands of variables makes it very easy for me to understand, just as how one person may enjoy spicy foods but not grapefruit or where some may hear too much bass and others not enough. so many VARIABLES!!! culture, upbringing, what sounds you are surrounded by, traffic signals, your genetic structure, your actual physical position when listening. perception is a learned skill that we do not choose to accept, it just happens and it is different for every single person.

i think THESE are the sort of differences between individuals that make DBT difficult: everyone hears differently. there is no absolute sound.

the best example of how an ear and sonic preference can change is in the study of language and sounds. the chinese language has a completely different set of sounds to that of the english language, thus their speaking intonation, laughter, and music reflect their cultural and sonic inclinations. eastern and western and andean and greek and celtic and ... and ... all use completely different scales based on their preferences of sound learned over time through language and their environments.

Thus, i often wonder do hi-fi listeners across the globe prefer different sounding systems based on their installed sonic memory? or is there a constant in terms of preference across the globe? probably not. or even more interestingly, can one find similarities in preferences in sound based on linguistic sounds of an individual region? are the frequencies accented in the german language more easily noticed by a german in his hi-fi? DBTs are a waste of time. instead of focusing why not, it is much more fun to focus on the why.

the heart of all of this lies within JA's question: where do the negative frequencies go? there are aspects to our perception of sound that simply cannot be measured because they are based on individual perception which is different for every single one of us.

GeorgeHolland's picture

If blind testing is so difficult then how did the people that I linked to manage to do so?  Harman Kardon does blind testing at the drop of a hat. Go ask them how they do it so easily. Mr Atkinson's refusal to do so is simply an excuse as to not have to bring up why cables, power cords and other snake oil is indeed snake oil. He can merrily go along his way as he has for years now ignoring such products and letting his reviewers say whatever BS they want about the sham products and not have to worry one bit. He just doesn't care is the bottom line.

ChrisS's picture

Georgie Porgie,

That you cite these links as authoritative sources indicates the level of your understanding of testing methodology.

JohnnyR's picture

......nothing to back up whatever it is you are trying to say but it is amusing.

ChrisS's picture

Research methodology courses are taught in colleges and universities all over the world, even Russia...

Let us know when you and Georgie take one.

JohnnyR's picture

Nothing to cite other than your own wandering silly posts? Thought so.

Regadude's picture

Well little Johnny, at least Chris is A REAL psychologist. He's not a, you know, a hobbyist like yourself...

JohnnyR's picture

........credentials from ChrisSy just say so. Oh lets see I think I'll be a nuclear scientist now just because I say I am on the forums. There now it's a done deal. Besides how he acts out is more like a 3 year old than an adult. Some professional he is and tell us all again just what your expertise is? Trolling perhaps?

Regadude's picture

...credentials from Johnny the hobbyist speaker designer. Let's see some pictures of your Johnny brand speakers! Post some pics, or provide a link to a site where we can see these speakers of yours.

I demand to see these speakers of yours! 

ChrisS's picture

If JRusskie has a misguided and limited understanding of DBT, and....

Georgie has a misguided and limited understanding of DBT, then....

Are JRusskie and Georgie one and the same person?

Has anyone seen them in the same room together? Hmmmm.

Please, one of you (I guess it doesn't matter which...) ask Harman Kardon how they do their DBT's and how they use the results.

Thank you.

JohnnyR's picture

I haven't seen one iota of explanation from yourself yet as to why both of us and Harman Kardon and the other links George posted to are wrong. Still waiting ChrisSy.

ChrisS's picture

Has HK hired you guys as DBT consultants?

 

Hey JRusskie,

Can you answer this one?

If A=B and C=B, then A=?

If you pass the test, then perhaps someone will hire you... But you and Georgie might have to fight over the job.

JohnnyR's picture

"i think THESE are the sort of differences between individuals that make DBT difficult: everyone hears differently. there is no absolute sound."

The sole purpose of DBT is to see if the person listening can distinguish between A and B. If they can't then for all practical purposes there is no difference in the sound from A and B. You and ChrisSy seem to think it's all about what the person "likes". It's a straight forward test method and "likes" has nothing to do with it.

Please explain to us all how Harman Kardon manages to use DBT all the time and do it well? I will be awaiting your reply Ariel.

ChrisS's picture

So every household has a Harman Kardon product? And you and Georgie have living rooms that look like anechoic chambers? No fireplaces, of course....

ChrisS's picture

Hearing a difference between a Harman Kardon product and another product in a anechoic chamber means what to you, Georgie and JRusskie?

Do you know that Ford makes the best trucks in the world?

ChrisS's picture

Are you sure I didn't say 'licks". You know maybe tasting an audio product will yield just as useful results in a DBT.

ChrisS's picture

JRusskie,

Now run out to your nearest Boris' Convenience store and get yourself a can each of Pepsi (do you even have Pepsi in the Former-USSR?) and Coca-Cola and set up your own Pepsi (or whatever passes for cola in Russia) Challenge.

Wiki has a nice explanation of how to do a DBT...

Once you've done your very own Peps(k)i Challenge, please send us your conclusion. We're curious...

The next step now is to get everyone in your subsidized housing project to participate in your Pepski Challenge.

Gather up that data, compare it your own conclusion and let us know how useful that information is.

I'm sure you'll enjoy the challenge of your doing your very own DBT's! (You won't even have to ask Harman Kardon to use their anechoic chamber!)

John Atkinson's picture

JohnnyR wrote:
The sole purpose of DBT is to see if the person listening can distinguish between A and B. If they can't then for all practical purposes there is no difference in the sound from A and B.

And that's the problem with these tests. If a formal blind test gives results that are indistinguishable from what would be given by chance, formal statistical analysis tells us that this result does _not_ "prove" there was no difference in the stimulus being tested, only that if there _was_ a difference, it was _not_ detectable under the conditions of the test. No more general conclusion can be drawn from the results. And as I have said, it is very difficult to arrange so that those conditions don't themselves become interfering variables. Even the fact that it is a test at all can be an interfering variable, as I explain in this lecture preprint.

JohnnyR wrote:
Please explain to us all how Harman Kardon manages to use DBT all the time and do it well?

I have visited Harman's facility in Northridge and their blind testing set-up is impressive. They have worked hard to eliminate interfering variables and their testing is time- and resource-consuming and painstaking. Even so, they have to make compromises. Blind testing of loudspeakers, for example, is almots always performed in mono. And despite the rigor of their testing, you still have anomalous results, like the Mark Levinson No.53 amplifier, which was designed with such testing but fared poorly in the Stereophile review.

While formal blind tests are prone to false negative results - not detecting a difference when one exists - sighted listening is prone to false positives, ie, it detects a difference when none exists or perhaps exaggerates the degree of difference. As neither methodology is perfect, we go with the one that is manageable with our resources. We therefore offer our opinions for readers to reject or accept in the context of their own experience and I believe Stereophile does  a better job of that than any other review magazine or webzine.

If you are uncomfortable with that policy, then you should not read the magazine. And if I remember correctly, JohnnyR, you admitted in earlier discussions on this sute that you neither subscribe to Stereophile, nor do you buy the magazine on the newsstand. So why should anyone pay attention to your opinions on how the magazine conducts itself?

John Atkinson

Editor, Stereophile

JohnnyR's picture

So you are saying that if in a DBT the listeners could NOT tell a difference between two amps using music of their own choice then that doesn't prove the amps sound alike?  Funny stuff there Atkinson. For one who thinks you should trust your ears to evaluate components you just bashed the ONE single TRUE way to test by USING YOUR OWN EARS in a DBT.

"And that's the problem with these tests. If a formal blind test gives results that are indistinguishable from what would be given by chance, formal statistical analysis tells us that this result does _not_ "prove" there was no difference in the stimulus being tested, only that if there _was_ a difference, it was _not_ detectable under the conditions of the test."

Your own words are saying that "if there was a difference it was not detectable under the conditions of the test"........oh you mean like letting the listener use the music of their own choice and switch back and forth endless times between two amps and then guess wrongly enough times so that they can't tell which one was which? LMAO if that's not proof that both amps sound alike then what sort of test WOULD prove that they do?  Come on Atkinson you just don't like DBTs because they would show up so many components that people think sound "oh so better than the rest"

Opinions from you and your reviewers are the Gospel now folks. No need to test anything really just trust good ol'JA and his flunkies. Yay.

"If you are uncomfortable with that policy, then you should not read the magazine. And if I remember correctly, JohnnyR, you admitted in earlier discussions on this sute that you neither subscribe to Stereophile, nor do you buy the magazine on the newsstand. So why should anyone pay attention to your opinions on how the magazine conducts itself?"

Oh just maybe because  a lot of people care for this little thing called the TRUTH? When magazines like your's take liberties with the truth by having shoddy reviews instead of in depth testing, then it's everyone's and anyone's responsibility to speak up when crappy falsehoods are published and the readers are supposed to take it all on faith. That's why. I for one do not take your opinions on anything audio related as worthwhile at all for the simple reasons that you show so much promise when you measure speakers but fail to even bother with the snakeoli products that you let slide under the radar yet let your reviewers give them glowing reviews sans any testing what so ever. Maybe that's the sighted listening bias you just spoke about yet you fail to even try with those type of products to get to the real TRUTH.

ChrisS's picture

JRusskie,

If you like Harman Kardon marketing, but you're not sure if Ford makes the best trucks in the world, then get yourself an F-150 and whatever truck you used to rumble across Afghanistan with, do your DBT (just like  the Pepski Challenge) and let us know what you come up with...

You are marketing TRUTH now? How pure is it?

I know some construction workers who might be interested...

ChrisS's picture

JRusskie, Just looking at your response to John's post and comparing word-for-word what John wrote and what you think he says, there's such a huge world of difference!! There's a war in your head!

[Flame deleted by John Atkinson].

GeorgeHolland's picture

"Even so, they have to make compromises. Blind testing of loudspeakers, for example, is almots always performed in mono."

Well Mr Atkinson the reasoning behind testing speakers in mono is to eliminate the dreaded comb filter affect that would otherwise show up if a stereo pair were auditioned and the listener moved their head even a couple of inces. I'm surprised you didn't mention that fact but then again you think DBTs are hard to do, so if you don't know how to do them then indeed they are hard to do. *Chuckle*  Any DBT done should be auditoned is such a manner. The rest of your "excuses" for not doing them is the same old same old from you, nothing surprising there.

ChrisS's picture

So Georgie Porgie,

Let's say JRusskie is DBT'ing a $1500 speaker and a $500 speaker and can't hear a difference, and you are DBT'ing a $4500 speaker and a $4000 speaker and you happen to have enough working neurons to hear a difference... Which set of speakers should the ex-shepherd construction worker buy?

John Atkinson's picture

GeorgeHolland wrote:
John Atkinson wrote:
Even so, they have to make compromises. Blind testing of loudspeakers, for example, is almost always performed in mono.

Well Mr Atkinson the reasoning behind testing speakers in mono is to eliminate the dreaded comb filter affect that would otherwise show up if a stereo pair were auditioned and the listener moved their head even a couple of [inches].

That is a consideration, of course, but in my opinion a minor one. As I had understood from Floyd Toole back in the day, the additional complexity required  of  Harman's physical speaker shuffling apparatus to do blind speaker testing in stereo was not justified by the results, ie, they felt that the stereo performance could be predicted from the mono results.

I don't agree with that, but more importantly, this illustrates the thesis offered in my lecture, that when you move the testing situation a step away from how the product is going to be used, you can't be sure that the assumptions you make haven't invalidated the test. As I write in the abstract to the lecture, "perhaps some of things we discard as audio engineers bear further examination when it comes to the perception of music."

BTW, I am still waiting for you to acknowledge that the criticism you made of my lecture, that it was not about Richard Heyser, was incorrect.

John Atkinson

Editor, Stereophile

GeorgeHolland's picture

More excuses?  I see what Johnny meant. You are never wrong. I am pretty sure Haram Kardon knows what they are doing. Please address any criticisms to them not me.

I am afraid that comb filtering IS a big deal. That would explain why cables "sound" different. It's not the cable but the listener changing where their head is between "testing"

You will be waiting a long time for any ackowledgement about your "lecture". Stop being the primadonna already.

Regadude's picture

Georgie wrote:

"Stop being the primadonna already."

Look in the mirror and repeat those words!!!! laugh

ChrisS's picture

So how does one differentiate speakers that sound differently, amplifiers that sound differently, pre-amps, turntables, tonearms, cartridges, DAC's, etc., if a turn of one's head makes that much difference?

Where's your reliability, Georgie? Doesn't science depend on reliability?

JohnnyR's picture

I can see what George is up against in here with Tweedle Dum and Tweedle Dee tag teaming and showing their ignorance.

http://www.ethanwiner.com/believe.html

Golly look what he meant.I think Ethan was banned from here ages ago for showing up Fearless Leader and his cronies and out right showing how REAL science works. BWAHAHAHAHAHAH loser boys.

Regadude's picture

http://www.audioholics.com/news/editorials/diy-loudspeakers

John Atkinson's picture

Quote:
http://www.audioholics.com/news/editorials/diy-loudspeakers

Just bookmarking the link for future reference.

John Atkinson

Editor, Stereophile

JohnnyR's picture

......when you start doing a single DBT or even a SBT then you can talk about the "truth". Have you EVER designed and built your own speakers? Nahhhhhhhh you are too lazy or too "busy". Still finding plenty of time though to post online all the time though strangely enough.cheekyTill then you aren't an engineer so take your own advice and don't comment on speaker design anymore.

If you are going to save the link then please also save this link where discussion about it unfolded.

http://forums.audioholics.com/forums/loudspeakers/83412-diy-loudspeakers-can-you-build-better-than-professional-designs.html

As you can see the original post was just one of many OPINIONS about the topic that is if you bother to read it at all. There are various OPINIONS about the topic and notice just how many of the so called "hobbyists" ended up being professional speaker builders. If you just pick and choose certain OPINIONS from the thread then you are guilty of leaving out facts.

For starters read the sixth post down by Jinjuku regarding Jeff's post that pretty much sums up where DIY has progressed.

ChrisS's picture

Not real science either...

GeorgeHolland's picture

Frick and Frack strike again. Regadude and ChrisS always come up with strawman replies and ignore the links posted."Not real science "? How pompus can you get? Mr Winer measured the effects of comb filtering, what did you measure ChrisS the length of your nose when you typed that reply? You dismiss anything people link to yet show us nothing in return. Regadude, posting opinions isn't real science just so you both understand. Now run along lil boys and study real hard, maybe in another 20 years you might be able to hold your own in a discussion.

ChrisS's picture

Has Winer's results been verified?

Did you know that Harman Kardon makes the best audio products in the world? And Ford makes the best trucks, right?

JohnnyR's picture

Or is that above your abilities like thinking?

Go ahead, put on a pink or white noise source and move your head about and tell me the sound doesn't change. You won't bother so forget it ChrisSy.laugh

ChrisS's picture

When moving a microphone while recording a person's voice, the sound changes. Did the voice change?

GeorgeHolland's picture

You never answer a question , you just put forth silly questions of your own. That's what people do when they don't know or are scared to try.

Moving a microphone while recording a person's voice? If that's how you do things then no wonder you don't know what Johnny was talking about. Yes the sound changes as recorded by the microphone so what?  Genius.angle

ChrisS's picture

I'll answer you this one... You and JRusskie always answer your own questions that you pose to everyone in these discussions. There's no need to provide any answer to you. As well, your attitudes and limited knowledge of the application of research methodology make civil and thoughtful discourse impossible.

So my questions to you and JRusskie are formed to reveal how each of you think whenever you provide a response.

You provide enough information for me to say that I find the "best" use of my time in these discussions is to make fun of you and JRusskie.

Ariel Bitran's picture

makes some sense of the whole darn thing.

Regadude's picture

The only duo that strikes here George, is you and Johnny. You both STRIKE OUT!

ChrisS's picture

Who's doing the listening in your tests, JRusskie? Do you know any 18 year old musicians? Oh, of course not...But you probably keep company with a bunch of construction guys (lucky you!) with damaged hearing. They should all find that there's no difference between any products.

ChrisS's picture

Once again, comrade JRusskie, you are on your Quixotic journey down that twisty, winding path for "truth"... Being an upstanding citizen of the former-USSR, you should know about "truth", right?

DBT is SCIENCE, and SCIENCE is ALWAYS RIGHT, especially during the heyday of the Union of Soviet Socialist Republics and you've been hanging on to this idea since the dissolution of the USSR. Hanging on to something you really want to be TRUE.

Comrade JRusskie, just because you want it, doesn't make it so.

Ariel Bitran's picture

just wanted to let you know i haven't forgotten about you.

i've been south of the equator spending time with my father and brother, but now that I'm back in the Stereophile office, i'll answer your question in full a little later.

peace out homeslice.

be nice.

Regadude's picture

Asking Johnny to be nice... Wow, you are very optimistic Ariel! 

GeorgeHolland's picture

Ariel, how about doing your job and deleting post and start placing a 30 day BAN on Regadude and ChrisS for spamming this whole article day after day with nothing less than pure drivel and taunts? I've seen kids frums better moderated than this fiasco but it's already turned into a child's forum the way they act. You would think they had some dignity but we haven't seen any yet.

Ariel Bitran's picture

I believe the internet is pretty much open range, so I'm happy to just watch folks speak their minds using whatever manner to express themselves.

one thing that bothers me is when a member is unneccessarily rude to someone who is being respectful or even innocently posting unknowing of the snark they may receive. in those instances, I say something so new users are not discouraged to continue posting.

On the other hand, if you're being provocative with your initial comment, I can only expect others to react to your provocation. 

******

The other policy I encourage and have stated many times is as follows 

"don't feed the trolls"

******

Finally, we do delete comments from ANY member we feel is totally off-topic and insulting to an individual member, as we want the conversation to continue and bring light to the subject  at hand rather than turn into a name-calling game. That being said, I'd like to delete Regadude's comment above since it just makes fun of Johnny. Also, I probably shouldn't have typed be nice. i dont know what i was thinking. I think i just wanted a rhyme in there, and also to include a general suggestion of positivity, but deleting RegaDude's comment would delete your reply, and this is a conversation I'd like to keep around.

Regadude's picture

What did I do? In response to your comment of "be nice" to Johnny, I merely joked that you were asking a lot. A lot, because Johnny is often not nice. He is that way, that is a fact. I have no responsibility for his behavior. I just pointed it out.

Don't shoot the messenger!

ChrisS's picture

What is reliability?

Think, Georgie, think.

Ariel Bitran's picture

the two links provided earlier giving examples of some DBTs.

I found the matrixhifi test to be ignorable: who is the sample? how did they select these people? how are they representative of a population of listeners as a whole? in order to gather significance from these these tests, the first and most important step is determining your sample, sample size, and how you select your sample. this just seems like a bunch of friends having fun. also, since there were multiple components being switched at the same time, system synergies could have been the cause of the weaker sounding more 'hi-fi' system. maybe those components weren't right for each other, but the cheaper system just sounded better. at least in ABX, they only changed one piece at a time

what i found interesting in the ABX test was the user's ability to control the change of system component themselves. this helps eliminate the idea that the listener might feel like they are getting 'duped' or constantly searching/guessing for the difference.

Also regarding the ABX method, the # of times a difference was heard was 33. the # of times no difference was heard: 29. Interestingly, cables were the least discernable. 

DBT is time consuming and for signnificant results you need a large sample size (to represent a large population of listeners). With a small sample size, as in both of these tests, you risk a greatly flawed hypothesis and will lack confidence in your results. 

I don't want to whip out my textbooks, b/c i have other stuff to do, but 17 listeners is not nearly large enough of a sample size to even represent a population of 70,000 Stereophile readers (for example). Then we run into an even bigger problem of "who" is selected-->ie what type of sample you are trying to represent.

I've heard repeatedly that H/K does have a successful DBT model. I'm sure it takes them years to perform each experiment, and it is wildly expensive and time consuming. You need a large sample size for any of this stuff to matter, not a few dudes in a basement.

GeorgeHolland's picture

More excuses from Stereophile?  Why am I not surprised. You scoff and blow off any attempts at doing a DBT by the people I linked to yet when it comes to "reviewing" cables and amps it's suddenly okay to accept a sighted biased review as being true?  Laughable sir but then I suppose you being an emploee have to toe the line. So be it.

I give up. Go on and trust your sighted "tests" while ignoring a DBT or even a SBT. I know your boss won't let you do any. He "knows" it all. *eyeroll*

Regadude's picture

Hey George! Ariel provided you with a more than satisfactory answer. So why are you still complaining? You never heard the expression "agree to disagree"?

Ariel is right; sample size is crucial. Anyone who has a basic knowledge of statistics understands this. 

JohnnyR's picture

George was talking about simply doing a DBT or SBT among friends like he linked to. Let's say you and 5 pals think the "Humungo" amp  blows away every other amp you have listened to. So you set up a simple DBT or even a SBT to see if you can RELIABLY pick which amp is which WITHOUT sighted bias. In other words using your EARS???? I have read on the forums here how much people rely upon "What I heard" yet don't trust a simple test to prove that they can.

So lets say the "Humungo" amp is just simply "liquid, lifts 400 veils and makes the back ground blacker", what ever silly terms you wish to use, If so then you should be able to pick it out 100% of the time doing a DBT. If not and lets say you only choose right 50% of the time then that's proof you were only guessing and couldn't tell which was which. So much for saying sample size is critical. If no one can pick the "Humungo" amp from the other one then where are your rationalisations for saying it's better?

I can tell though that none of you have even bothered to try a SBT or DBT so it's all a waste of time speaking to the peanut gallery. I love how much money you both have WASTED on pricey products that only look good and sound the same. Funny as hell, go on spending YOUR money I love ityes

ChrisS's picture

JRusskie just beat himself with his own thinking...

Regadude's picture

 

Johnny wote:

"I can tell though that none of you have even bothered to try a SBT or DBT so it's all a waste of time speaking to the peanut gallery. I love how much money you both have WASTED on pricey products that only look good and sound the same. Funny as hell, go on spending YOUR money I love ityes"

How do you know what I've bought with my money? I will go on spending my money (but not on your plywood speakers you make in your basement), that's what it's made for!

 

JohnnyR's picture

.a fool and his money are soon parted.

Regadude's picture

Well I am not that much of a fool with my money. I have not yet bought any of your JohnnyR brand speakers! 

ChrisS's picture

"...came out to play,

Georgie Porgie ran away!"

Delusions of absolute truths cloaked in "science" aren't much to hide behind, eh Georgie...

I hear our Man of La Mancha, JRusskie, is looking for his Sancho Panza to accompany him on his journey in search for TRUTH.

Happy trails!

ChrisS's picture

Georgie, Look at all those scientists backing you up!

If you take a college level research methodology course, Georgie, you may not appear so laughable. At this point, there appears to be very little knowledge in what you say about reviewing audio products.

GeorgeHolland's picture

Here is how you review audio products........ looks in Stereophile and buys whatever the product of the month is. Spends too much money but doesn't care. Goes online and talks like a 5 year old and spews insults and THINKS he's smart. Case closed.

Regadude's picture

George, name ONE SINGLE product that I have bought. Just one. You and Johnny have this little fantasy in which you think you know everything and everyone. 

How about you list your gear? If you are so good and knowledgeable about buying audio products, do share with us the products that are George Holland worthy. If you actually have any audio gear...

GeorgeHolland's picture

ZzzzzzzzzZZzzzzzzzzzzzzz sorry but conversing with the likes of you is boring and makes my brain hurt considering the amount of BS you spew,

ChrisS's picture

Take one course of college-level research methodology and call us in the morning...

rl1856's picture

Do you like what you are hearing ?  If no, move on until you do.  If yes, then shut up and relax.  This is a hobby focused on the enjoyment of the creative output of artists.  It is not about how many proverbial angels can dance on the head of a pin.

Go listen to MUSIC !

hnipen's picture

Thanks John for a very interesting and exciting presentation, lots of interesting information here and I'm surprised, to say the least, from the lack of positive feedback.

There are many who are skeptical to some of the ways of doing measurements in Stereophile and in some ways I'm one of them too, especially the way speakers are measured so close, large array speakers like the bigger Dunlavy's and some others will not sum up very nicely in this way. We do, however, not live in a perfect world and Stereophile cannot afford an anechoic chamber, so this is probably the best they can do.

I wish John would share more of this kind of information as he has gathered lots of knowledge during a long interesting career at Stereophile and other places.

Go on John :-)

Merry Christmas

Cheers harald

absolutepitch's picture

John, thanks for getting this lecture pre-print available for us to read. I have been looking forward to this.

I agree that there is a lot fo information combined into one lecture that anyone would need a lot of time to learn and understand the details. Pardon me for paraphrasing some of your words below.

Regarding the null result of DBTs, your description of the interpretation is in agreement with what I remember from statistics classes. I might add that a statistician would include a probability value or confidence band with the interpretation (something to the effect that 'the null hypothesis of no-difference-detctable is accepted with high probability'), and equally for the case when a difference is detected with high probability. I personally think DBTs should be done for product reviews, but agree that valid DBT's are difficult and time consuming (expensive) to do correctly, as Dr. Toole has shown in his writings.

The example of the 'backwards' impulse being not agreeable to listeners is something I have noticed in reference to digital recording. It's a wave form that does not occur naturally in music production, so reproducing it should sound 'bothersome'.

I also agree with the previous post, that more articles like this would be welcomed, to further highlight how complicated this field really is.

bernardperu's picture

I have read your essay with great pleasure (all of it!) and I think it is a great example of the Liberal Arts and Science coming together. In the end, it feels like a piece of applied music philosophy, which I find fascinating. It also seems to be free of busines-oriented interests, as your opinion on cables clearly suggests. It is awesome and very unsual to meet an accomplished person who gives priority to his passions and principles over financial interests (as also expressed on your 2012 writing on the CES and Las Vegas). 

I consider myself to be an audiophile that turns off the lights and tries to connect his emotions with the music with a very relaxed mind (this seems to be a category in itself, as the un-relaxed passive listeners who cannot focus on the music on a mid to long term basis tend to be very opinionated). Having said this, I recently purchased a pair of Class D mono amps that can clearly connect me to the music (Hephaestus brand). I have not ever listened to amps which are over 15k. Within similar prices, class D seems to be the better choice (but how relative this can be, Jon!)

I will continue to follow your writings with deep admiration and I thank you for making a difference on my musical experience (which is passed on to my girlfriend and my child). 

 

Bernard

GeorgeHolland's picture

Clueless you are if you think Mr Atkinson is something special angle

ChrisS's picture

Yoda you are?

Andreasmaaan's picture

It's a pity that some proponents of DBT as the only valid methodology have used the comments thread here to launch personal attacks against JA. Personally, I found the lecture fascinating and thought-provoking, and I thought that the nuggets of personal history provided a powerful context for the thoughtful opinions expressed. For better or worse, Stereophile doesn't restrict itself to DBTs as their only reviewing tool, but JA does measure every piece of gear his reviewers review - a practice which ensures that the opinions of the reviewers are grounded in objective data, or otherwise as the case may be. I'm not sure why this approach, coupled with a reliance on an income stream from advertising, seems to place JA in line for so much personal vitriol. If similar attacks were levelled at me in my professional life, I'd be mortified and enraged.

To cut what is risking becoming a lengthy expression of indignation short: thank-you JA for a wise and thought-provoking read.

JohnnyR's picture

WRONG!

"For better or worse, Stereophile doesn't restrict itself to DBTs as their only reviewing tool, but JA does measure every piece of gear his reviewers review - a practice which ensures that the opinions of the reviewers are grounded in objective data"

Do you even READ the reviews? Seriously dude. Stereophile DOESN'T do DBTs at ALL! Atkinson has said repeatedly that they are "too difficult to do". Show me ONE single DBT he has done in a review please. Plus he does NOT "measure every piece of gear his reviewers review". Cables, power cords, record demagnitizers, just to name a few. You obviously have been reading another publication NOT Stereophile.

bernardperu's picture

JhonnyR

 

If Stereophile has the principle of not doing DBT and for you doing so is a matter of principle why oh earth would you even care to read Stereophile? Your statement above suggests you have very extensive knowledge of the content of Stereophile's articles. So you keep reading stereophile's reviews knowing they are irremediably flawed?? How perverse is this?

 

The real question lies on your double blind tested hatred. 

 

Whatever your answer is I will have to assume that your hatred is a broken record and it will just go on.

GeorgeHolland's picture

He probably reads them as I do for the laughs.smiley

JohnnyR's picture

......on your error filled post above so why should I comment on your assumptions? If you wish to worship Fearless Leader then by all means bow down and do so. I myself prefer to ask questions and be skeptical about his "methods" and the reason "why" he doesn't test ALL items under review.

John Atkinson's picture

Andreasmaaan wrote:
It's a pity that some proponents of DBT as the only valid methodology have used the comments thread here to launch personal attacks against JA.

Indeed. I am not sure why I have become a lightning rod for these audio skeptics. Perhaps it is because it is a matter of religious belief on their part - having had extensive experience of blind testing, I rejected it and thus became a heretic in their eyes. Or perhaps it is just our courtesy in allowing them the space on this website to express themselves.

Andreasmaaan wrote:
JA does measure every piece of gear his reviewers review - a practice which ensures that the opinions of the reviewers are grounded in objective data, or otherwise as the case may be.

As my reviewers do not see the measurements until after they have written the review, I take my hat off to them.

Andreasmaaan wrote:
To cut what is risking becoming a lengthy expression of indignation short: thank-you JA for a wise and thought-provoking read.

You're welcome and my thanks to everyone else who appreciated the preprint. It was an honor to have been invited by the Audio Engineering Society to give this lecture - it's not often that you get the opportunity to look back over a 4-decade career!

A Happy New Year to everyone who surfs this site.

John Atkinson

Editor, Stereophile

JohnnyR's picture

You ignore the errors and just plow along blissfully gobbling up the unwarranted praise.

"JA does measure every piece of gear his reviewers review"

As I pointed out above, along with other errors on the part of Andreasmaaan,you do NOT test every piece of gear your reviewers review, but then you pretended that you do in your response. Only certain items are tested by you. Please get your own facts straight if that's even possible for you to do so.

Regadude's picture

Johnny, I wish you a great new year 2013, in which you sell 10 times more of your home made speakers than in 2012. 

Since you sold zero of your speakers in 2012, you need to sell 10 X 0=0 speakers in 2013 to meet this objective. Maybe you can hire George Holland as your director of marketing and sales... 

Get cracking, you have only got 364 days left to sell your quota...

JohnnyR's picture

[flames deleted by John Atkinson]

Unlike yourself Raghead, I will be DESIGNING and BUILDING my own speakers and learning. You do know that word "learning"? Hmmmm maybe you stopped after age 10. You on the other hand will be on here *yawning* and acting the fool and spending your money on Stereophile reviewed crapola.yes

Regadude's picture

So according to you Johnny, everything Stereophile reviews is crapola?! There is not one good piece of equipment that has been reviewed by Stereophile? Hundreds of reviews, not one good product?

But you, in your basement, build quality speakers... 

I have challenged you before (as well as George) to name your gear. What kind of gear merits the Johnny seal of approval? 

As always, you always ignore my challenge (George chickened out to). It seems you can only complain, but cannot even provide a list of your gear. 

I guess ever seeing pictures of your "famous" speakers is also out of the question. 

All talk and no action. 

Regadude's picture

So Johnny, where are your speakers pics and specs?

In your imagination.... [flame deleted by JA]

GeorgeHolland's picture

Johnny never offered to show you anything other than his backside which I think is approriate considering the childish actions of yourself.What does it matter what our audio components are? Anything we mention would be ridiculed by yourself no doubt. Sorry but I'm not playing your silly game. Good God grow up and act like a man for once will you? angry

John Atkinson's picture

JohnnyR wrote:
As I pointed out above . . . you do NOT test every piece of gear your reviewers review, but then you pretended that you do in your response. Only certain items are tested by you. Please get your own facts straight if that's even possible for you to do so.

Out of morbid curiosity, I looked at the past 15 issues of Stereophile - we published 98 full reviews and follow-up reviews. Of those 98 reviews, 83 were accompanied with a full set of measurements, or 84.5%. So even if not every review includes measurements, the vast majority do, which supports Andreasmaaan's point.

I have explained to you before, in other comment threads, why we do not publish measurements of analog playback components, which is due to lack of resources. I have also explained to you why we do not publish measurements of tweaks, which is that it it is difficult to determine exactly what to measure. I don't see any reason to reopen those discussions, but that you continue to be fixated on these two issues and continue to raise them is sad.

And, as I have warned you, JohnnyR, I am continuing to delete comments from you, and others, that are nothing more than flames aimed at other readers.

John Atkinson

Editor, Stereophile

JohnnyR's picture

"Reviews" means just that, any item reviewed by your staff. If you think "tweaks" don't qualify then they should be labled as "Opinions" not a review.angle All of those full reviews you speak of were what speakers, amps, preamps, cd transposrts and the like?

Yeah I know why you don't test tweaks, that would involve you using those horrible SBT or DBT. That's a really nice system you have set up. "We don't believe in DBTs or do them or can't afford to do them"......"We don't know what to test on the tweaks"..........add the two up and you get "We don't have to show that tweaks actually do anything other than what our "reviewer" subjectivly alludes to in their sighted biased listening using their golden ears"......EXCUSES Mr Atkinson plain and simple. no

Once more I ask you to read and comment about this link:

http://forums.audioholics.com/forums/loudspeakers/83412-diy-loudspeakers...

You seem to think DIY can't do anything right in your previous post in answer to Raghead. Insulting intelligent people that actually build better speakers than the status quo can produce is beyond being pompus in your case.THAT is what's SAD.

Don't you have some real work to be doing instead of being on the forums? I thought that was Ariel's job. Let him earn his pay.

ChrisS's picture

JRusskie,

Look up "fixation" and "perseveration" in your dictionary. "Absolute truths" regarding audio products seems to exist only in your basement and in your head.

John Atkinson's picture

JohnnyR wrote:
John Atkinson wrote:
I looked at the past 15 issues of Stereophile - we published 98 full reviews and follow-up reviews. Of those 98 reviews, 83 were accompanied with a full set of measurements, or 84.5%. So even if not every review includes measurements, the vast majority do, which supports Andreasmaaan's point.

"Reviews" means just that, any item reviewed by your staff. If you think "tweaks" don't qualify then they should be labled as "Opinions" not a review.angle

I am not sure what you mean. You were objecting to Andreasmaan's statement that Stereophile accompanies its reviews with measurements as not being true. I offered the analysis above to show that he was correct.

JohnnyR wrote:
Once more I ask you to read and comment about this link:

http://forums.audioholics.com/forums/loudspeakers/83412-diy-loudspeakers...

You need to consult with your fellow traveler GeorgeHolland, who criticized me about linking to other sites :-)

JohnnyR wrote:
You seem to think DIY can't do anything right in your previous post in answer to [Regadude.] Insulting intelligent people that actually build better speakers than the status quo can produce is beyond being pompus in your case.THAT is what's SAD.

With respect, you are arguing with the voices in your head here. I haven't written anything about DIY speaker designers not being able to do "anything right." What I have said is that a DIY designer, like yourself, doesn't subject his loudpeakers to the scrutiny of a disinterested marketplace, something professional designers do as a matter of course. So the question whether or not DIY designs are superior to commercial designs isn't put under independent scrutiny. It remains unsupported conjecture, as far as I am concerned.

John Atkinson

Editor, Stereophile

GeorgeHolland's picture

Mr Atkinson, you could  compare DIY against manufactured speakers but of course you won't. Might upset those that can't make good speakers and I'm not talking about the DIY cheeky

Johnny plainly stated that "tweaks" are reviewed in your magazine and online yet they aren't tested, hence you don't test every component under review. You have a reading comprehension problem? How many components reviewed by Jason Serinus have been tested? uh huh, right.

So now you wuss out about commenting on links? That proves you had nothing to say in the first place about it so you shouldn't have opened your mouth.Cop out time for Mr Atkinson.

The only unsupported conjecture on here are your abilities to print honest reviews. Your ability to avoid testing  alot of audio components is well known by now. Very impressive spinning you do on here.wink

John Atkinson's picture

GeorgeHolland wrote:
Mr Atkinson, you could  compare DIY against manufactured speakers but of course you won't. Might upset those that can't make good speakers and I'm not talking about the DIY cheeky

Back in the early 1990s, Stereophile's Corey Greenberg was one of the judges for a DIY loudpeaker competition in San Francisco. There were 2 winning designs and as part of the prize, those 2 speakers were subject to a full set of measurements in Stereophile, published in the March 1992 issue. The speakers' measured performance was good but not great and certainly didn't embarass professional designers.

GeorgeHolland wrote:
Johnny plainly stated that "tweaks" are reviewed in your magazine and online yet they aren't tested, hence you don't test every component under review. You have a reading comprehension problem?

I don't believe so. JohnnyR was wrong, in that we very rarely publish full reviews of tweak products. Of the 98 reviews published in the magazine January 2012 through March 2013, there was just one review of a "tweak," Robert Deutsch on the HiFi Tuning Supreme Fuses in May 2012. ( I don't count the AudioQuest power strip reviewed by Kal Rubinson's review in December 2012 as Kal didn't make any comment on its sound quality, only on its utllity.)

GeorgeHolland wrote:
How many components reviewed by Jason Serinus have been tested? uh huh, right.

The correct answer is none. But that doesn't support JohnnyR's case either, as Jason's reviews are published on the website Secrets of Home Theater and High Fidelity, which has no connection with Stereophile. I hardly believe I am obliged to support reviews of products in competing publications with measurements in Stereophile.

John Atkinson

Editor, Stereophile

JohnnyR's picture

"Back in the early 1990s, Stereophile's Corey Greenberg was one of the judges for a DIY loudpeaker competition in San Francisco. There were 2 winning designs and as part of the prize, those 2 speakers were subject to a full set of measurements in Stereophile, published in the March 1992 issue. The speakers' measured performance was good but not great and certainly didn't embarass professional designers."

Golly "only" 20 years ago and as we all know, nothing new has happened since then to make speaker design any better (Insert BIG sarcasm face here) cheeky Talk about a lame example Atkinson.

Yeah yeah, "full reviews" whatever, Plenty of "little revews" though all the freakin time and all they are , are OPINIONS but you know what? You let your reviewers get away with murdering the intergrity of the audio world with their stupid banter about  "blacker backgrounds,lifted veils and HUGE improvements in the sound" without them having to justify their claims, Pretty slick there pal.

Oh and Jason Serinus "reviews" as posted about the recent RMAF isn't connected to Stereophile? Give us all a break, Your EXCUSES are even lamer this time than usual.

Yep typical Fearless Leader goobley gook. Did you major in Universisty in the double speak of Orwell's 1984? If not then you are gulity of plagiarizing the concept.

John Atkinson's picture

JohnnyR wrote:
Golly "only" 20 years ago and as we all know, nothing new has happened since then to make speaker design any better...

Again you are arguing with the voices in your head, JohnnyR. Of course a lot has happened in the past 20 years, especially the advent of low-cost measuring equipment. But you're missing my point, which is, as I wrote earlier today, that a DIY designer doesn't subject his loudpeakers to the scrutiny of a disinterested marketplace, something professional designers do as a matter of course. You have repeatedy claimed that your speakers are as good as if not better than commercial designs. However, unlike engineers like Kevin Voecks, Paul Barton, Richard Vandersteen, Jeff Joseph, etc, your ability to pay your mortgage and feed your family doesn't depend on your skill as a speaker designer. Their's does, and it makes a difference.

Look, if you seriously believe your speaker designs are fully competitve with commercial designs, email me your measurements. I'll get back to you with my interpretation of what they mean, assuming they are as comprehensive as what I publish with every Stereophile speaker review. Put up or shut up.

JohnnyR wrote:
Yeah yeah, "full reviews" whatever, Plenty of "little revews" though all the freakin time and all they are , are OPINIONS but you know what? You let your reviewers get away with murdering the intergrity of the audio world with their stupid banter about  "blacker backgrounds,lifted veils and HUGE improvements in the sound" without them having to justify their claim...

You're not making sense. "Andreasmaaan" was discussing reviews in the magazine being accompanied by measurements. I was discussing reviews in the magazine being accompanied by measurements. You seem to be discussing something that exists only in your mind. And unlike Professor Dumbledore's quote, that doesn't mean it's real :-)

JohnnyR wrote:
Oh and Jason Serinus "reviews" as posted about the recent RMAF isn't connected to Stereophile?

That was a show report. No-one other than yourself equates a magazine's coverage of a show with its formal equipment reviews. Are you seriously suggesting that magazines shouldn't publish show reports without including a full set of measurement data for each room they report on? Really? There isn't an audio magazine or website that does that.

John Atkinson

Editor, Stereophile

JohnnyR's picture

Go have another pink drink Fearless Leader.

Me let YOU interpet my frequency response? Don't make me LAUGH. You can't even interpept Wilson't ragged ass frequency response laugh

[usual flames deleted by John Atkinson]

John Atkinson's picture

JohnnyR wrote:
John Atkinson wrote:

Look, if you seriously believe your speaker designs are fully competitve
with commercial designs, email me your measurements. I'll get back to
you with my interpretation of what they mean, assuming they are as
comprehensive as what I publish with every Stereophile speaker review.

Me let YOU interpet my frequency response? Don't make me LAUGH.

You make my point for me, JohnnyR. It was a serious offer on my part. Professional loudspeaker designers send out their products for review and for the public scrutiny of possible customers. Amateur speaker designers, such as yourself, are often afraid to let anyone else judge their work.

John Atkinson

Editor, Stereophilee

Regadude's picture

I have been challenging Johnny tp provide pictures and or the specs of his speakers. He has never met my challenge.

Now the editor of Stereophile offers to look over his data. Johnny again makes up excuses to avoid providing any information on his heavenly speakers.

Big talker...

bernardperu's picture

Dear Jon,

What should have been an enlightening discussion of your great lecture has been contaminated by ill-conceived comments. Some individuals have posted comments that have created an environment of nastiness. 

I would seriously urge you to start deleting ill-conceived comments from now on, whether they use flamable language or not. Some people are just turned on by the flames, but others aren't. And those others would much rather focus on the joy of engaging in the music and exchanges related to it.

You do not run a state-owned site but a private one that is likely to become a lot less lucrative (in all senses) if such nasty comments are to become visible.

I am not even pointing out names and we all know who I am talking about.

I am just giving you feedback that comes from an avid reader of Stereophile online. 

Finally, the correction of my use of English was mentioned by one of these individuals. Please allow me to say that English is my second language and it is oxidized. Sorry about that. 

Thanks!

 

hollowman's picture

Thx for posting this lecture, JA!!
I also saw the Wilkinson/HTG YouTube video/interview in which you summed up some highlights of the AES lecture.
A question about the non-oversampling vs. oversampling debate ... you (JA) noted that people (subjective listeners) prefer the NOS DAC.
Was there a formal test (or survey) conducted that you may be referring to?

As far as NOS vs OS ... this has been an especially active (and debated) topic in the DIY community.
Older DACs (multibit R2R, not Sigma-Delta or one-bit MASH/bitstream) are commonly used in various DIY projects/experiments. And, all else held equal (e.g., all one does is shunt the OS chip in an older CD player), many audio hobbyists do, indeed, prefer NOS -- but many also do not (the ones who prefer OS note that once clean, well-regulated power is provided to the digital-filter IC, you get much better sound).
My own "breadboard" experiments convinces me OS is better, tho' the NOS sound has some (few!) nice qualities not present in OS. This is almost like the age-old tube vs. SS debate.
I am aware of the very $$ Zanden DAC, and that's NOS, as are the modern AudioNote CD player and the HiFiMan 602. But almost all other commercial DAC (or digital sections) equipment since the late 1980s uses digital filtering (oversampling). And, I believe, OS has won out the mass-market and high-end/audiophile market for sound, important reasons (e.g., production costs vs. fidelity; or the all-important implementation issue: it's not just the DAC chip, but, also, decoder, pwr supply, output stage, etc.)
It's difficult to isolate the benefits/drawbacks of digital-filtering -- other than the shunting/bypassing method I described above -- due to various manuf/model designs using wide array of topologies, parts, etc. In other words ... Zanden (NOS) vs. Theta or Wadia (OS).

One final note ...
The 1st- and 2nd-generation CDPs were criticized in the audiophile community for having sub-std. sound. TTBOMK, these early players were all NOS.
Then came DF chips like Philips SAA7220 (famous for the faceplate moniker: "Fourfold Oversampling Digital Filter"), and commercial (read: not modded or souped-up Philips units from Meridian/Cambridge/etc.), started to gain acceptance by serious audiophiles/reviewers.
Examples of this included various Philips/Magnavox, Yamaha and (of course) Sony models sold in department stores/mail-order-outlets at hugely-discounted prices.

So what happened to improve the sound? Was it mostly/partly OS? And/or was it better DACs? And/or better (analog) output stages? And/or better overall topologies, tighter tolerances, better PCB layout, more careful PS design?

Some of the zero-oversampling fanboyism is strange (other than the one-more-time-parasite-for-the-bored-audiophile pathology). In DIY threads and modern eBay Chinese kits (or even complete D/A NOS processors, mostly from China) new-old-stock DACs (like TDA1541 or TDA1545/1387) are used with just a simple output section and a $2 USB decoder (or S/PDIF receiver) to get data from some source.

Example:
The image below is a top-down view of a complete D/A processor based on a TDA1387 (Philips multibit R2R DAC IC from mid 1990s) -- it's a non-(zero-)oversampling config. (but uses 8 TDA1387s in parallel).
It's from China and sells for less than $125 (eBay, Taobao, etc.). I have no idea how it sounds.
The main point, however, is that: this is a fairly new offering in the marketplace ... but it's all OLD-SCHOOL ... simple topology/layout, R2R DAC, etc ... many engineers were doing this type of stuff in the mid-1980s, and henceforth.
So why is old new again?! Does NOS, indeed, sound better because most audiophiles (esp. young people, but also older geeks/reviewers who sold/got rid of their "vintage" gear) have heard nothing but oversampling or Delta-Sigma digital?
These probing queries go right to the heart of the AES lecture. Indeed, the Wilkinson video interview on Home Theater Geeks had this title: Episode 84: What Is Reality?
What is Reality?
http://twit.tv/show/home-theater-geeks/84

hollowman's picture

Another aspect of recorded/playback audio (and psychoacoustiics and related biology) is how human perception deals with:

-- NATURAL acoustic events, such as wind, leaves, birds, crickets, human voice
(IOW: random, short-duration, relatively QUIET events -- the type/duration of acoustics humans dealt with for most of the species' evolutionary history)

...vs. more "recent" acoustic exposures , such as ...

-- MUSIC/MEDIA(movies, tv/GAMING/URBAN ENVIRONMENTS
(these are much more continuous/linear, longer-duration, and louder events than what was statistically-signif. for most of hominid's evol. history).
With music, especially, there is so much more complexity (including the four std. music-theory qualities: harmony, tonality/timbre, rhythm, melody). Add words (song, opera, lyrics) .... and you have to, then, also engage the brain's symbolic/language interpreters ...hence, brain/mind acoustic organs are starting to get a real workout.
But let's not stop there: a good (= engaging) film with a complex orchestral score in the soundtrack makes things even more complicated -- i.e. the added visual-system integration! And the smell/flavor of popcorn and the sweetness/wetness/caffeine of that cola ...

I think the brain/mind can "learn" to cope with all this newer multi-tasking sensory environment. It (brain/mind) does seem to really like sensory "overload." Or movies and audio systems wouldn't be so popular.

To get to the point: Nature is a tough task-mistress. If you want to know her secrets, you've gotta have the passion for hard work (= scientific/empirical data colelction and careful analysis) AND passion for philosophical engagement. JA has done much of this in the 2011 AES lecture. Good work!

BTW: One (of the many, many) reasons classic objectivists' tests (like ABX) are flawed is that they (mostly) concentrate on short/excerpted chunks of music tracks.

Charleski's picture

You were setting up gear in the vestibule, and thus presumably moving around the area a bit. Is it not possible (indeed, highly probable) that in the short delay while they ‘demagnetized’ the LP you moved between a resonant null and resonant mode? Setting up the bass in small spaces can be challenging because moving one or two steps is often enough to produce a marked change in the resonances. We all know the vital importance of siting speakers correctly.

So yes, you heard a difference in bass. But no, this was probably the result of well-known acoustic factors and nothing to do with some kooky tweak.

X