Why John Atkinson Believes Measurements Matter

Back in 2013, I took the train to Stamford to give a presentation to the Connecticut Audio Society to help celebrate their 30th anniversary. On March 6 I returned to the CAS, but this time via Zoom. I talked about a subject close to my heart: measurements and their connections with accuracy and/or musical enjoyment. The video is now posted to the CAS YouTube channel (linked below)—it runs for 2+ hours but I think Stereophile readers will find what I had to say stimulating, perhaps even sometimes controversial.

My presentation takes up the first 21 minutes and is followed by a Q&A with the CAS members. (Great questions, guys!) At 1:18:00 I give a tour of my listening room, where two of my cats decide to make a cameo appearance.

My thanks to Samir Shah of the CAS for the invitation and to Chris Hart of CAS for moderating the event and preparing the video. Enjoy!

Many of the things I talk about in this video are covered in greater depth in articles that are posted to this website:

On why Stereophile started incorporating measurements in its reviews: Must We Test? Yes We Must (page 3) and God Lives in the Details.

On the one loudspeaker measurement that no-one does but might be significant when differentiating reproduced sound from live sound: The Recording Angel. Also mentioned in my 2011 Richard Heyser memorial lecture to the Audio Engineering Society in Case Study 2 Loudspeakers.

On blind vs sighted listening: Subjective Fact or Objective Fantasy.

On reviewers and reviewing: Measuring Sound Quality and Who Watches the Watchers?.

On my joining Hi-Fi News in 1976, my move to Stereophile in 1986, and my handing over the editorial reins to Jim Austin in 2019: From London to Santa Fe and The Next Generation.

On my careers in scientific research, teaching, and as a musician, before I joined Hi-Fi News & Record Review: Where Did the Negative Frequencies Go?.

On the subject of the ever-increasing prices of high-end audio products, which I mention in the video: Conspicuous Consumption, The Price Event Horizon, and The Upward Price Spiral.

And on J. Gordon Holt founding Stereophile on the concept that products should be judged by listening to them: 40 Years of Stereophile.

COMMENTS
mememe2's picture

Listening to this only reinforces the idea that we are never too old to learn something new by listening and thinking about something from new and different perspectives.

jimtavegia's picture

Good to be a fly on the wall. I was interested in John's comment about the LS50 not providing a realistic grand piano sound, when it sounds very good on vocalists. Would a 10" or 12" sub solve this problem?

I also like the idea of having filter choices in DAC and let the customer decide which they like of the music of the moment.

tiagoramossdg's picture

I too would like to know the answer to that one.

rschryer's picture

...for doing this video. Great stuff.

And as an added bonus, Kal Rubinson has a cameo.

georgehifi's picture

I've posted links to this vid on audio forums where shiller's are promoting bizarre snake oil voodoo products like these little black 5mm self adhesive dots you stick on the end of a fuse to make it sound better (no you don't get the fuse just the black dot) https://ibb.co/LpDpf3f with absolutely no tech info or measurements whatsoever from shiller or manufacturer, this sort of stuff needs to be completely eradicated from audio.

Cheers George

Anton's picture

It seems, the lower the bar for a product to enter the hobby, the higher the degree of potential BS.

It takes skill and knowledge to design and build and amp or speaker. A fuse button or small spec of something to stick on an actual component or in the room, not so much.

That is not meant as a comment on the likely awesome and multi-veil lifting order of magnitude improving little button you mentioned, just a general observation.

But, speaking of measuring things...;-D

thatguy's picture

it was just in 2019 when Stereophile finally dropped "P.W.B. Electronics Cream Electret and Rainbow Electret Foil" from their recommended components list. It was so hard to take any recommendations seriously with things like that on the list.

Jim Austin's picture

That was my first Recommended Components issue. I recall discussing it with Art Dudley and he agreed that it was time for it to go.

Jim Austin, Editor
Stereophile

thatguy's picture

That is good to know. I get it that it was likely a bit of tongue in cheek fun at the time, but it did make the rest of the lists harder to take seriously.

ChrisS's picture

I remember getting a free sample of the PWB Electronics Cream Electret and Rainbow Electret Foil from May and Peter Belt after reading Art's article (Listening #112) about them and their products.

I didn't hear a difference when I tried the cream and foil on my Linn system.

End of story.

Or is it?

drblank's picture

are only as good as the measurement itself and the equipment used to take the measurement, and then understanding what measurements will give the user an idea of what to listen for.

I think the aspect about audio world is that the room, placement within the room of equipment (speakers), and the listening position are the biggest single factor in how good your stereo sounds. But I see very little attention to that.

I have seen far too many photos of audio rooms where either there is no acoustic treatment, not enough acoustic treatment, the right type of acoustic treatment, speaker placement, etc. etc. where everything is not optimized for the best sound to the listener. Yet, reviewers tend to make a big deal over one piece of gear over another, yet the differences are sub-1dB differences, yet paying attention to the room, acoustic treatment, placement, etc. can attribute to MANY dB differences, sometimes in the double digit (for low frequencies) differences and improvements.

It just amazes me how much attention there is to gear vs the room and treatment. I guess passive room treatment isn't sexy as a nice shiny amplifier, DAC, pair of speakers, etc…

Here's a link to a room that someone built based on the design work of an acoustic treatment company's products/designs. It's probably one of the better examples of a top notch 2 channel listening room for a high end system.

I think any SERIOUS audiophile should take the time to check out these high end rooms to really understand what attention to room acoustics and treatment can do.

https://www.youtube.com/watch?v=eMLA5h0nh8s&t=2s

hollowman's picture

JA never produces boring content. This 2hr video flies!

mememe2's picture

Concerning the "LS50 not providing a realistic grand piano sound, when it sounds very good on vocalists. Would a 10" or 12" sub solve this problem?" Have not seen any reviews yet, but KEF's new KC62 has the potential to solve that problem.

tonykaz's picture

Mr. Atkinson has been High-End Audio's Highest Authority ( in print ), as far as I discern . I recognised Mr.JA when he was HFN&RR and I was importing the Magazine for one of my ( Grey Market ) Subscription Businesses. I also sold HFN&RR over the Counter at my Esoteric Audio Salon in Farmington Hills, Mi.

Since 1983ish, MR.JA has been my "Audiophile of the Year", there wasn't anyone else rising to that level ( until Tyll arrived about 10 years ago ).

My Reviewer of the Year could now be Mr. HR, Mr. Micallef or Steve G the Audiophiliac. Previously, Mr. JA was my top reviewer but these three gifted folks are outstanding at reaching thru their keyboards to touch the readership subscribers.. Phew! ( or YouTube Camera )

I'm not preoccupied with technical specs ( I'm still an Engineer and Machine Repairman ). I'm a low powered Amp/high efficiency loudspeaker/headphone audiophile that realises Mr.JA's & Tyll's instrumentation are critically needed to make intelligent sense of documented performance . Working without instrumentation is working blind.

Stereophile without Mr.JA's measurements will probably be the end of my Subscription, the brilliant reviews are about 50% of the reason to subscribe. Future Record Reviews are now 'my' work from listening thru streaming. My commitment to 33.3 & Koetsu rebuilding & exporting is a tired exercise that is diminishing with every OBIT page of every Newspaper. New kids can't afford Vinyl at today's prices.

Mr. JA is ( and has been ) the virtuoso voice of High-end audio, it's Man-of-Integrety. I am pleased that he didn't retire to join Tyll in living out on BLM land in a self sufficient Camper/Motorhome.

Tony in Venice

hollowman's picture

JA's reply about why AC conditioners improve sound when they are difficult to metric: "I don't know".
Yet most young "objectivists" -- freshly minted with university EE textbooks and (now) Wikipedia -- are truly ignorant (cognitive dissonant??) that most of current HUMAN science (ca. early 2021) is hardly probative of Nature.
In fact, at least 99.99999% of total physical/scientific underpinnings are UNKNOWN.
Even very basic stuff ...

https://www.theguardian.com/science/2013/sep/01/20-big-questions-in-science

For example: The 7 Biggest Unanswered Questions in Physics

What is matter made of? ...
Why is gravity so weird? ...
Why does time seem to flow only in one direction? ...
Where did all the antimatter go? ...
What happens in the gray zone between solid and liquid? ...
Can we find a unified theory of physics? ...
How did life evolve from nonliving matter?

Problems with Science itself:

Academia has a huge money problem.
Too many studies are poorly designed.
Replicating results is crucial — and rare.
Peer review is broken.
Too much science is locked behind paywalls.
Science is poorly communicated.
===========
So given these massive gaps in human knowledge, ca. 2021, why extend providential importance to a few limited-dimension metrics. Like the 2-D plots -- FR, jitter, time-domain, etc -- JA referred to?

JHL's picture

A really pertinent comment - scientism is indeed a belief system. The Science, as we presume it it, barely exists yet many of its fans cite what they *think* is settled when in fact hardly anything is. What is gravity? What powers the universe? What is matter? And why are the "laws" of material behavior built in the utterly ephemeral - how could every subatomic particle - if that's what they are - do what it must do simply because, faithfully and forever?

Nobody knows any of this but somehow we can divine sound not by hearing a thing, but by projecting an abstract, or even an outright belief, upon it.

John Atkinson's picture
hollowman wrote:
What is matter made of? ...
Why is gravity so weird? ...
Why does time seem to flow only in one direction? ...
Where did all the antimatter go? ...
What happens in the gray zone between solid and liquid? ...
Can we find a unified theory of physics? ...
How did life evolve from nonliving matter?

Andrew Thomas examines some of these questions in his inexpensive series of books Hidden in Plain Sight.

John Atkinson
Technical Editor, Stereophile

Jim Austin's picture

Science taken as a whole is the most successful knowledge-gathering effort in human history. It is easy to mystify it, because it is often so damn weird and intrinsically mysterious, but it's really very simple: It's a set of basic, logical steps for acquiring reliable, repeatable knowledge.

Which doesn't mean that the problems you (hollowman) raise don't exist. They do. But it's important to put things in context.

If you're an audio component or loudspeaker designer, it's important to be sure about what you think. You want your design decisions to be based on concrete, reliable, transferrable facts--for my truth and your truth to be the same. (Otherwise it's less likely that I'll respond to your designs the sam way you do--a poor foundation for a hi-fi business.)

And yet, much "research" in our industry is done in listening rooms by people who aren't interested in proving things. They hear what they hear, often quite clearly and unambiguously, and they act on it, trying new things, altering the design to get the sound they want. They're confident in what they hear and could probably pass any statistically valid test--but such tests are time-consuming and offer little if you're already sure and not interested in convincing others. It is possible to be sure--and correct--without producing rigorous, statistically significant data. You just can't prove it. Which in some circumstances doesn't matter.

And if you're an audiophile, the degree of certainty you require is your own business: Some want more, others less. There's little harm in a completely subjective approach, if that's what provides satisfaction. We each get to decide on our own criteria.

Which is to say that one need not be dismissive of science to make space for in audio for methods that are not strictly scientific. (And I'm not saying either of you intended to be dismissive of science. I can't tell.) Plenty of room already exists.

Science is wonderful and amazing and powerful, and I'd advise any designer to apply it rigorously to their work. But when it comes to judging the communication of human emotion and insight with music, science is not the ultimate arbiter--or even an arbiter at all. Science, in fact, aims to eliminate the human and personal from knowledge--to make knowledge transportable. And music communication is a very human, very personal thing.

Science is incomparably successful, but it doesn't apply to everything.

Since this is a thread about measurements, it's a fine place to reassure people that as long as I'm Editor, measurements will continue in Stereophile.

Jim Austin, Editor
Stereophile

hollowman's picture

I refer everyone to JA's 2005 comment (begins about 09:31 on MP3 file).
https://www.stereophile.com/news/050905debate/index.html
...something to the gist of ... when you're young, you know everything ... and when you're old-aged you realize how little you know. . Perhaps what Socrates meant near the end of his life with the famous ...
https://en.wikipedia.org/wiki/I_know_that_I_know_nothing

I think all this relates to the relatively recent scientific field of evolutionary psychology ... specifically, when you're young and your brain/mind system convinces you "you're right" ... and this makes sense in the ancient (non-technical) environment. Alas, you're only awake for so many hours/day ... and SELF-CONVINCING (maybe, self BRAINWASHING) decisions HAVE to be made by Nature.
HOWEVER ... with so many variables (domains or dimensions) of the modern environment, the ancient human brain can't possible TRACK or hold them all intact (recall how much super computer power it takes to do MUCH simpler things, like weather, stellar evolution, black holes, protein folding, etc.) ... so the human mind/brain comes to tracking only a few and makes a sunk-cost-fallacy (SELF DELUSIONAL) decision. (like ABX test or the fanboy ad nauseam comments on sites like hydrogenaudio, ASR or SBAF).
Even if you're a younger person (20s-30s) reading this ... and doubting what I'm saying .. try to think back to your CHILDHOOD and recall how many "CHILDISH" or "TEENAGEY" things you USED to believe ... and that are no longer valid.
As noted in the JA Zoom video, the relative absence of younger people at Hi-Fi shows (and their general absence in high end audio) may be this ancient, Natural programming at work.

pbarach's picture

(quote from Theodor Gottlieb, aka Brother Theodore)

Glotz's picture

NT

jimcomas's picture

Tottaly with you tonykaz and thank you. No measurements means no validity to me.Noone can recommend a component, just by saying: "believe me I know".

ChrisS's picture

..." Listen to this! "

Anton's picture

I think there are still some way cool things to measure that haven't been done yet...

Going back to a review of the Esoteric speakers several years back: a brand new pair of speakers shows up. I'd love to see the speaker's measurements before and after 'break in.' There is a world of coolness when you think about measuring break in.

Same goes for a brand new amp, cartridge, etc.

2) Cartridge measurement seems a bridge too far, but would be very interesting to see more of.

3) Measurements of 'listening seat' performance, primarily for speakers, to show what a difference we can make with sound with potentially minimal movement of gear. (We already get reports of in-room frequency response, so it seems possible to compare speaker positions in those small increments that seem to be so important.

4) Speaker/amp interaction would also be intriguing. Given how important that first watt is, imaging a graph of in room response for speakers or amp at the one watt level. It may be the most important watt, as they say....but could it also be the easiest watt?

5) Imagine amplifier, speaker, digital thing, preamp, etc....measured via being fed by different power conditioners and compare.

6) Perhaps contextualizing cable/power cord/wire performance, both alone, and when hooked up to a system.

None of that is meant as a complaint, merely musing about how close we are to being able to measure some cool stuff!

hollowman's picture

Yes, the number of TRACKED and MEASURED parameters (domains, specs) has been increasing and expanding since day one. For example, in DIGITAL, linearity (late 1980s) and jitter (early 1990s). And recently ... pre- and post-ring echoes. Etc. etc.
I superficiality looked at the early white papers and journal papers (1970s and 80s) from Philips and Sony. And really found no mention of these metrics. (Tho' they were probably known to aerospace, military and computer-industry scientists from the very beginning of digital science.)
YET ... despite those metrics being neglected (perhaps even unknown to the original inventing scientists), the early CD players (esp 3rd gen onward) are still find-sounding units ... even DESPITE their "poor" measured performance, chipsets like the Philips TDA154x, etc. still are favored by certain audiophiles.

dc_bruce's picture

Thanks to Mr. Atkinson and all who were responsible for this. FWIW, I, too, run a pair of LS 50s (not the "metas, yet) and I run them I believe successfully with a first generation REL subwoofer, the Q400E. I trade a bit ultimate low-end extension for limiting the room resonance "bump" to about 4 db. It's been many decades since I owned a pair of Magnepans (which I loved, but which have very demanding placement requirements); and I don't think the LS50 +subwoofer combination equals them in reproducing an acoustic piano. There's something about the way the Magnepans "load the room" that no pair of direct radiator speakers I've had has equaled.
Regarding horns, I've never heard a pair of Klipsch horn speakers that sounded to me any better than awful. However, at the home of a well-known designer of tube electronics I did hear a pair of Avantgarde Duos that really impressed me from the midrange up. The bass, supplied by conventional woofers with separate amplifiers, was a mess; but I attributed that to my host's not having taken the trouble to do all of the adjustments necessary to "dial them in."

hollowman's picture

Amir Majidimehr of AudioScienceReview.com (ASR) was mentioned a few times in the CAS/JA Zoom call.
Back in Jan, Majidimehr also spent about 2hrs with HionHiFi ... he has "slightly" different take on Measurements ...
https://www.youtube.com/watch?v=2WfWHC05lbg

Anton's picture

I still value the listening experience, but I am pleased to see his energy for measurements.

Some of my local audio kooks and I are embarking on our own little "first watt" experiment thanks to his review of a 70 dollar Amazon amplifier, the AIYIMA A07 TPA3255.

We also grabbed some SMSL and Nikou amplifiers (the Nikou was 23 bucks.)

We have a hefty array of very efficient (Klipsch, Lowther, Voxativ) speakers we will be playing with, and some different power supplies, and will see how easy this 'first watt' is to produce and listen to.

Amir had done a set of measurements on the Aiyima amp, which was the initial spark for this project.

Thank you for posting about his site, interesting stuff. (I hope he moves more listening impressions into his site and looks at correlation between what he measures and what he hears.

hollowman's picture

In the video, Majidimehr made some questionable (highly opinionated) comments/remarks. But he also made some very accurate ones. Perhaps the most important to STEREOPHILE is audiosciencereview.com SURPASSING stereophile.com in traffic metrics/ranking (i.e., Alexa web metrics).(NOTE: You can add Alexa to your browser via a free plugin (extension).) Also, Google often places ASR on or near the top of its SERP (often even ABOVE a product's own manuf. page or the Amazon page).
And Amir is correct ... that very quickly, ASR's Alexa has surpassed SBAF, hydrogenaudio, Audiokarma, etc.
Yes, ASR being ads-free helps. I'm sure Majidimehr's employment at Sony and Microsoft helped him accumulate quite a sizable nest egg ... so he can afford to buy AP555 and Kippel .... and go ads-free on his own "science"-biased site.
All this kind of reminds me of the cultism that occurs when a person that speaks well, and clearly, gains enormous power. Take Elon Musk ... he has been myth-busted many, many times for bogus or recycled claims (SpaceX, Solar City, "founder" of PayPal) ... but his celeb PERSONALITY and (most important) the brain-washed, wanna-believe fan-base, have made him the richest person in the US.

Charles E Flynn's picture

Stereophile, or audiosciencereview.com? I have not read enough of the latter to have a possibly-useful opinion.

JHL's picture

The latter has little connection to sound and less to human performance. It's lure is that it, like its patrons, apparently believes that a handful of base metrics are Special Insight and therefore a whole science (or close enough). It's a circular justification*.

S'Phile, on the other hand, arguably has the highest pertinence - as defined by the human element first, the technical element second, and the two in usable, even proper context to one another.

*which veers toward de-humanizing music itself, remarkably, at least for some.

DH's picture

"Science, in fact, aims to eliminate the human and personal from knowledge--to make knowledge transportable. And music communication is a very human, very personal thing."

Nonsense.
Hi-fi is a scientific/engineering pursuit regarding electronic music reproduction.
How we respond when we listen is something else.

Scientific enquiry aims to deal with only what is repeatable. Not what is imaginary. It has nothing to do with eliminating the human and personal from knowledge.

Science in audio is there to prevent the personal/subjective from being viewed as some sort of transportable truth. Any individual is free to enjoy audio how he/she pleases and under whatever personal subjective/emotional status he/she has. That personal reaction has zero validity for any other individual.

Your POV is a fake philosophic/"sciency" position designed to deny science and promote the purely individually subjective POV as having some sort of transportable truth which can be more broadly applied.

Even if some product has no scientific validity in it's claims and even if it can be measured and shown to be untrue.

Your POV is just an excuse for fake products that don't do what they claim.

The goal shouldn't be to promote products with distortion etc. (even if euphonic to some listeners), but to promote products with outstanding performance that also sound good to most listeners.

If an individual prefers listening to a tweak product which can't be shown to have any positive effect when measured (or, even to reduce performance); or to an amp with lots of distortion that's heard by some as "warmth", that's fine. We all listen with expectation bias and personal taste. But science is there to prevent us from promoting/reviewing such items as "hi-fi", "natural sounding" "lifelike", or "more real". Somehow superior.

If people want to listen to euphonic distortion, fine. But they should know that's what they are hearing. If they did, maybe they'd find a product that also sounded good to them and didn't add distortion.
I'd prefer something that measures well and sounds good. That's how I know my reproduction is an accurate reflection of what the artist recorded.

You and your magazine should promote that, instead of (sometimes making excuses for poorly performing or snakeoil products that add inaccuracy to playback by claiming that they are somehow outside science and convey emotion better than other products.

X