You are here

Log in or register to post comments
DavidEdwinAston
DavidEdwinAston's picture
Offline
Last seen: 1 week 4 days ago
Joined: Dec 1 2020 - 8:44am
Blind Testing

If I have missed the lengthy posts on this subject elsewhere on these forums, forgive me, or move this post to the correct forum.
I accept, a difficult and extremely time consuming process. The same music, the same volume balance, and time. It seems reasonable that a reviewer might think, "By golly, they do sound impressive, they are my pick!" And then over a period of days realise the initial qualities are simply tiresome over time!

Still, possible, I would have thought. I will get to the point. If some unstated manufacturers $1,000,000 Wham Bam Thankyou Mam speakers were tested blindly against, oh let's say Wharfedale Elysian 4's. Would the former speakers be every reviewers pick?

I will assume that every reviewer for this online publication would quickly identify the more expensive speaker without resort to measurements.
I suppose it must be obvious to anyone reading this that I could consider being able to afford the Elysian!
(I should perhaps say that when my thirty year old Spendor s100's do fail, they should in a blink, be replaced with the current equivalent, which are a couple of thousand dearer than the Wharfedale).

geoffkait
geoffkait's picture
Offline
Last seen: 1 day 32 min ago
Joined: Apr 29 2008 - 5:10am
One wonders why people often

One wonders why people often predict the outcomes of blind tests, as if they were a foregone conclusion. “A blind test would certainly prove that such and such is true.” Seems to me the proper way to obtain results for any test, including a blind test, would be to actually perform the test. Then let the chips fall where they may.

geoffkait
geoffkait's picture
Offline
Last seen: 1 day 32 min ago
Joined: Apr 29 2008 - 5:10am
More thoughts on blind

More thoughts on blind testing - results of a single blind test should be suspect if they are negative. The reason for this is that there are many things that can go wrong when testing is performed. On the outer hand positive results have more credibility since positive results overcame all potential pitfalls in testing. What are some pitfalls that can occur during testing? (1) Operator doesn’t follow instructions. (2) There are errors in the system. (3) There are external factors, e.g., weather, that prevent the system from being able to distinguish between test articles. (4) The system used for testing is not sufficiently revealing to distinguish between the test articles. (5) The operator’s hearing ability is not sufficiently good to distinguish between the test articles. (6) There is a problem with one or both of the test articles, e.g., something broke during shipping or miswired.

Repeatability and transferability. Credibility of results can be increased by performing multiple tests. Tests should be repeated on the same system with same or different operators as well as on different systems with different operators. The more data points that can be determined the greater the likelihood that correct conclusions will be drawn.

DavidEdwinAston
DavidEdwinAston's picture
Offline
Last seen: 1 week 4 days ago
Joined: Dec 1 2020 - 8:44am
You are right

Geoff. A blind test shoul be exactly that.
I would feel sympathy with the manufacturer of the million dollar speaker, if in fact the cheaper speaker was preferred.
Your second post highlights the large amount of time that would need to be taken in order to be fair to each product.
As an alternative, you could play a short section of music to a large number of "enthusiasts". They listen twice, the second speaker volume balanced with the first. Then ask the question,"Which did you prefer?" Not particularly scientific, but some validity, surely?

geoffkait
geoffkait's picture
Offline
Last seen: 1 day 32 min ago
Joined: Apr 29 2008 - 5:10am
I’ve actually heard million

I’ve actually heard million dollar systems that sounded horrible. For example, when the speakers and electronics aren’t broken in. Follow?

geoffkait
geoffkait's picture
Offline
Last seen: 1 day 32 min ago
Joined: Apr 29 2008 - 5:10am
I heard it through thr

I heard it through thr grapevince,

1. Why are audiophiles so afraid of blind tests?
2. If you would use blind tests you’d find out that there aren’t any differences between power cords, or cables, or fuses, or direction of wire.
3. I conducted a controlled blind test with a friend of mine and found out that there is no difference between a regular fuse and a fancy audiophile fuse.
4. Blind tests are part of the scientific method so negative results obviously means the item under test is a hoax.
5. If reviewers all used blind tests audiophiles would be more inclined to trust their reviews.

JackWlwi
JackWlwi's picture
Offline
Last seen: 5 months 1 week ago
Joined: Jun 12 2019 - 8:33pm
Nothing fixed except the excuses

Excuses .... always excuses.

Blind test means just that .. you don't know which is which.

There is literally not any case where a blind test is not superior to a sited test. Every single excuse give my the illustrious Mr. Kaitt applies to a sited test.

.. and who sets up a $1Mil system for listening with no break in? ... no one.

There is never a reason not to use blind testing.

geoffkait
geoffkait's picture
Offline
Last seen: 1 day 32 min ago
Joined: Apr 29 2008 - 5:10am
When results of a blind test

When results of a blind test are negative it doesn’t mean anything since there are many reasons why audio tests can be negative. Tis much better assume you’re doing something wrong than assume the device under test doesn’t work as advertised or that no difference between two devices exist. Audiophiles tend to fall under Murphy’s Law, anything that can go wrong will go wrong. The best laid plans of mice and audiophiles oft go awry.

Significance can be attached to test results only after a great many tests have been performed - tests by different individuals on the same system and by individuals on different systems. I.e., tests should be repeatable and transferrable. Otherwise they have no meaning. To insist that negative results of a blind test prove anything is sheer folly.

Geoff Kait
Machina Dynamica
Not Too Chicken to Change

JDFlood
JDFlood's picture
Offline
Last seen: 4 months 1 week ago
Joined: Jan 5 2014 - 9:27am
Listening for differences in

Listening for differences in sound quality of music is a skill. There are more obvious characteristics like tonal balance, imaging, bass... but there are many less obvious characteristics. Appreciating and articulating differences require extensive vocabulary and experience isolating and attributing sound changes to the language require lots of experience. Music is not a sine wave comparison of one tone... it requires the subconscious to appreciate the gestalt and then break down the results to attribute to specific differences. This takes time. Music’s constant changing nature and simultaneous appearance and disappearance of sounds makes it difficult. If blind testing was done with one day listening to one source then on day another for a week. Then experienced experience high end audio people would probably have little problem articulation differences. But A/B flipping in a few minutes is very problematic with complex multi variable sounds of music.

Kal Rubinson
Kal Rubinson's picture
Offline
Last seen: 3 hours 35 min ago
Joined: Sep 1 2005 - 9:34am
JDFlood wrote:
JDFlood wrote:

But A/B flipping in a few minutes is very problematic with complex multi variable sounds of music.

But it can help to make the subtle more obvious.

JDFlood
JDFlood's picture
Offline
Last seen: 4 months 1 week ago
Joined: Jan 5 2014 - 9:27am
Kal, To me A/Bing in real

Kal, To me A/Bing in real time can make the obvious more obvious... like a real difference on tonal balance or slam for instance. But for most other less obvious stuff,,, rhythm and pace, for instance.. for me it takes listening to the music and not the system and working backwards to figure out the character that is different.

Kal Rubinson
Kal Rubinson's picture
Offline
Last seen: 3 hours 35 min ago
Joined: Sep 1 2005 - 9:34am
Quote:
Quote:

But for most other less obvious stuff,,, rhythm and pace, for instance.. for me it takes listening to the music and not the system and working backwards to figure out the character that is different.

For me, rhythm and pace are characteristics of the music since no competent playback system can alter the speed of the music. To use them as criteria is to rely on subjective and subconscious inferences. So, yes, they are less obvious.

Old Audiophile
Old Audiophile's picture
Offline
Last seen: 1 week 5 hours ago
Joined: Jun 15 2017 - 7:34pm
Interesting conversation!

Interesting topic! I see value in Blind Testing and/or A/B shoot-outs. One of the things I very much enjoyed, back in the day, was frequenting a very large shop & warehouse operation called Tech Hi-Fi in Cambridge, MA. In those days, they could arrange for very quick A/B shoot-outs between various components. The large inventory shops like this had in those days helped. After proper set-up, a salesperson would hand you a button you would use to switch back & forth between two components literally at the flick of that button. A certain amount of equalization was often needed for most components to allow for things like speaker sensitivity, amplification power, etc. In retrospect, I suppose the gadgets, switching circuitry or electronics used to accomplish this probably introduced a variable but I found this helpful, nonetheless. It helped me whittle down the field for a couple pairs of speakers and a receiver I purchased there in the 70s. It was also a lot of fun listening to other customers with deeper pockets A/B'ing the higher-end stuff or sales people doing this just for fun sometimes. There were other shops I frequented in those days with similar quick A/B facility but I haven't seen anything like this in decades. Nowadays, inventory in most shops is much, much less and one has to rely upon how fleet of foot sales people are switching out components of interest.

geoffkait
geoffkait's picture
Offline
Last seen: 1 day 32 min ago
Joined: Apr 29 2008 - 5:10am
Switching out components and

Switching out components and cables and fuses is actually one of my pet peeves and I’ll tell you why. It’s because every time you break the sentivie electrical-mechanical interface on connectors it takes time for that interface to be restored. To heal as it were. Maybe a day, maybe longer. I also eschew trying to AB new cables or components as they take quite a long time to reach their max sound quality. What’s worse is in the beginning of break-in they can sound pretty atrocious. Same with fuses, capacitors, solder connections, etc. obviously these issues apply to controlled blind tests, too.

Which is part of the list of reasons why I say negative results of a controlled blind test is meaningless as too many things can and do go wrong. Think of testing like a murder trial - circumstantial evidence must accumulate quite a bit before the prosecutor will pursue the case. The detectives might be 100% sure the guy is guilty but it must be proven beyond a reasonable doubt in court. Results of audio tests must be repeatable and transferrable.

Geoff Kait
Machina Dynamica
Not too chicken to change

Catch22
Catch22's picture
Offline
Last seen: 1 week 5 days ago
Joined: Nov 21 2010 - 1:58pm
I've never been able to

I've never been able to envision a situation where a blind test of audio equipment would be preferable to a long-term listening experience. That's not to say I'm not an advocate for the scientific method, I just don't see blind testing audio components as having much usefulness. Having said that, they could be hugely entertaining for reasons other than being useful.

Kal Rubinson
Kal Rubinson's picture
Offline
Last seen: 3 hours 35 min ago
Joined: Sep 1 2005 - 9:34am
Catch22 wrote:
Catch22 wrote:

I've never been able to envision a situation where a blind test of audio equipment would be preferable to a long-term listening experience.

I can envision(!) that but have you ever been part of one?

Quote:

That's not to say I'm not an advocate for the scientific method, I just don't see blind testing audio components as having much usefulness.

I will offer you two.
1. It is the only meaningful way to assess the (potentially) audible differences in a population. This is useful to a designer, manufacturer and/or vendor to estimate success or acceptance since his or any one's anecdotal reaction is not likely to be universal or, even, typical.
2. It is highly enlightening to challenge one's own extended but ad hoc
evaluation to the scrutiny of eliminating visual and anticipation bias.
It may or may not align with your long-term sighted evaluations but it seems to me perverse not to want to know if it does.

Old Audiophile
Old Audiophile's picture
Offline
Last seen: 1 week 5 hours ago
Joined: Jun 15 2017 - 7:34pm
Interesting Points

Interesting and quite valid points, for sure. I agree that nothing replaces the empirical & experiential ability of living with a component or system under one's own roof for a good length of time. After all, this is how professional reviewers do their thing. However, for most mere mortals in the market for a major component, I still believe there is considerable value in A/B'ing components, when possible, and/or critical listening auditions in shops before purchase. I know some folks avoid this, preferring to purchase components, have them shipped to their homes and tolerate the hassle of shipping equipment back when it doesn't measure up to expectations and that's fine, as far as I'm concerned. More power to them! Chacun a son gout, so to speak! Regardless, I much prefer a bit of due diligence of reading & research, followed by scheduled critical listening auditions to narrow the competition before a purchase. More often than not, when comparing components of similar quality, price point and class, the sonic differences are relatively subtle. One has to muster all the audiophile skills garnered over the years, of course, to extrapolate what that or those components might sound like under your own roof. Fortunately, the fail-safe return policy of reputable shops & dealers offers all the insurance I need.

geoffkait
geoffkait's picture
Offline
Last seen: 1 day 32 min ago
Joined: Apr 29 2008 - 5:10am
An observation: audiophiles

An observation: audiophiles seem to require or at least prefer some kind of testing for components,mcablrs, and all manner of tweaks. But when it comes to video it appears no such requirements or preferences for testing exist. We tend to walk into the store and select the TV with the best picture in our price range. Why can’t audio be so easy? Please, no angry emails if there’s anyone out there reading this who uses controlled blind tests to pick his TV.

Kal Rubinson
Kal Rubinson's picture
Offline
Last seen: 3 hours 35 min ago
Joined: Sep 1 2005 - 9:34am
Old Audiophile wrote:
Old Audiophile wrote:

Interesting and quite valid points, for sure. I agree that nothing replaces the empirical & experiential ability of living with a component or system under one's own roof for a good length of time. After all, this is how professional reviewers do their thing.

That's the bottom line.

Quote:

However, for most mere mortals in the market for a major component, I still believe there is considerable value in A/B'ing components, when possible, and/or critical listening auditions in shops before purchase..................................................................Fortunately, the fail-safe return policy of reputable shops & dealers offers all the insurance I need.

Again, no argument but none of that relates to blind tests which is what I was talking about.

Of course, blind tests are not generally available for the average audiophile or reviewer but I urge anyone seriously interested in audio to take advantage of participating in such a test as part of becoming aware of one's own biases and reflecting on how one makes assessments.

Kal Rubinson
Kal Rubinson's picture
Offline
Last seen: 3 hours 35 min ago
Joined: Sep 1 2005 - 9:34am
geoffkait wrote:
geoffkait wrote:

An observation: audiophiles seem to require or at least prefer some kind of testing for components,mcablrs, and all manner of tweaks. But when it comes to video it appears no such requirements or preferences for testing exist. We tend to walk into the store and select the TV with the best picture in our price range. Why can’t audio be so easy?

Easy. I really don't care as much about video.

Quote:

Please, no angry emails if there’s anyone out there reading this who uses controlled blind tests to pick his TV.

I actually participated in a few of these and learned what I stated above.

Old Audiophile
Old Audiophile's picture
Offline
Last seen: 1 week 5 hours ago
Joined: Jun 15 2017 - 7:34pm
Understood, Kal!

Yes, Kal. I quite understand what you meant/mean by "blind testing". Thanks for elucidating this for those whom I may have lead astray by my ramblings. In fact, I believe it was one of your articles describing the blind testing procedures employed by Revel that I very much enjoyed. I would love the opportunity to participate in something like that, just for fun someday. The closest I've ever come to anything like that was the experiences I described at Tech HiFi in Cambridge and one other time at a store called Tweeter, in Holyoke, MA. On a couple occasions, rather than handing me the switching button or device to evaluate two components, I asked the salesperson to do the switching such that I never knew which component was playing at any given time. That was pretty cool! On the first occasion (1972), as a financially strapped student who definitely should not have been spending money on audio toys, I believe this very well may have quashed a bias of mine. I was listening to a Sansui 2000X and its Marantz competitor. The Marantz, as best I can recall, was rated at somewhere around 40 or 42 watts power and the Sansui at 38 or 39. I listened intently for quite some time before finally choosing the Sansui. Both, couldn't possibly have been more similar in performance with all types of music but I found the Sansui just half a hair more to my liking. Had I ultimately chosen the Marantz I would have been just as pleased. If I had known which was playing at any given time, I may very well have chosen the Marantz because of my bias for extra power, in general. I did the same thing choosing speakers on a couple occasions, listening back & forth between EPI, Advent, Bose, Studiocraft, Ohm and probably others I can't recall. Lotsa fun! Happy Memorial Day everyone!

Kal Rubinson
Kal Rubinson's picture
Offline
Last seen: 3 hours 35 min ago
Joined: Sep 1 2005 - 9:34am
Right. Such experiences

Right. Such experiences endow us with useful circumspection.

geoffkait
geoffkait's picture
Offline
Last seen: 1 day 32 min ago
Joined: Apr 29 2008 - 5:10am
geoffkait wrote:

geoffkait wrote:
An observation: audiophiles seem to require or at least prefer some kind of testing for components,mcablrs, and all manner of tweaks. But when it comes to video it appears no such requirements or preferences for testing exist. We tend to walk into the store and select the TV with the best picture in our price range. Why can’t audio be so easy?

Then Kal wrote,
“Easy. I really don't care as much about video.”

I do care as much about video so you can take it from me it’s easy to spot the best picture, or improvement to the existing picture, no controlled blind test necessary. :-)

Does it mean humans rely more on vision than sound? In the evolutionary sense. Maybe.

Geoff Kait
Machina Dynamica
Not too Chicken to Change

Kal Rubinson
Kal Rubinson's picture
Offline
Last seen: 3 hours 35 min ago
Joined: Sep 1 2005 - 9:34am
geoffkait wrote:
geoffkait wrote:

I do care as much about video so you can take it from me it’s easy to spot the best picture, or improvement to the existing picture, no controlled blind test necessary. :-)

But do you find that choice easier than with audio? If so, might it be because you can see two or more video screens at the same time while that is not possible with audio?

Quote:

Does it mean humans rely more on vision than sound?

Depends on the task and context.

geoffkait
geoffkait's picture
Offline
Last seen: 1 day 32 min ago
Joined: Apr 29 2008 - 5:10am
Yes, I find judging the

Yes, I find judging the picture quality easier than audio. Well, sometimes, not always. In fact I often use video picture quality to assess the degree of success or failure of certain tweaks that affect both audio and video. It’s usually very easy to see the contrast improve or the color intensity/vividness, color true-ness/faithfulness and of course blackness of background. Maybe 3 dimensionality and resolution. There’s a whole language. Ha ha

Examples of tweaks that affect both audio and video,

1. CD and Blu-ray tray treatment
2. CD and Bluray disc edge treatment
3. Clever Little Clock
4. All the PWB stuff
5. Dampers
6. Crystals
7. Contact enhancers
8. Power cords
9. Directionality of wire (e.g., HDMI cables)
10. Teleportation Tweak
11. Vibration isolation
12. Schumann frequency generator
13. Intelligent Chip

Old Audiophile
Old Audiophile's picture
Offline
Last seen: 1 week 5 hours ago
Joined: Jun 15 2017 - 7:34pm
With Kal on Audio!

Given the ultimatum to live without music\audio or without TV\video, there is no question I would bid a fond farewell to TV\video. Thankfully, I haven't moved from one residence to another in decades. However, when I was younger and had to do that sort of thing with some degree of regularity, the stereo or sound system was always... and I mean always... last moved and first set up in the new digs. Music makes this laborious process so much less objectionable!

I am by no means a video aficionado but I do know that most stores adjust video settings to compensate for display and/or fluorescent lighting in order to appeal to customers. Prior to buying a TV, I always prepare similarly but, admittedly, much less stringently, prior to the purchase of an audio component (i.e. reading & research; actual viewing in stores; discussion with friends; etc.). A videophile friend whom I rely on for TV advice, prepares just as stringently for video purchases as he does for audio purchases. As such, I assume audiophiles and videophiles are probably very similar with respect to how they go about pursuing their passions.

As for the very general question of whether humans rely more on vision than sound in the evolutionary sense, I think the answer would be: It depends. From an evolutionary sense, like most animals when the chips are down, humans absolutely rely upon all of their senses to stay one step ahead of predators. Talk with a combat veteran about this, if he or she is inclined to. Now would be a good time to do that sort of thing. Happy Memorial Day!

geoffkait
geoffkait's picture
Offline
Last seen: 1 day 32 min ago
Joined: Apr 29 2008 - 5:10am
In the evolutionary sense

In the evolutionary sense the sense of hearing only gets you so far. In order for a cave man or any man to make a rational decision as to whether it’s best to flee or stay and fight, or do nothing, it’s definitely preferable to see O O what is coming for you. It could be a Saber Tooth Tiger or it could be a Sloth. It might be a buddy, in the woods during hunting season many hunters are killed for just that reason, the shooter heard something move out there but couldn’t see what he was shooting at. It’s not rocket science, folks.

Kal Rubinson
Kal Rubinson's picture
Offline
Last seen: 3 hours 35 min ago
Joined: Sep 1 2005 - 9:34am
geoffkait wrote:
geoffkait wrote:

In order for a cave man or any man to make a rational decision as to whether it’s best to flee or stay and fight, or do nothing, it’s definitely preferable to see O O what is coming for you.

But he can see only what's coming for him from the front and is not hidden in the shrubs. His hearing can tell him it's there and, if necessary, to turn around.

Quote:

It might be a buddy, in the woods during hunting season many hunters are killed for just that reason, the shooter heard something move out there but couldn’t see what he was shooting at. It’s not rocket science, folks.

No, it's stupidity, something that natural selection has still not eliminated. After all, it's his buddy who is dead.

geoffkait
geoffkait's picture
Offline
Last seen: 1 day 32 min ago
Joined: Apr 29 2008 - 5:10am
Ever watch a meerkat? He

Ever watch a meerkat? He watches 360 degrees, that’s what early man did too, get the high ground and scan 360 degrees. That’s how he survived. Vision has priority.

Kal Rubinson
Kal Rubinson's picture
Offline
Last seen: 3 hours 35 min ago
Joined: Sep 1 2005 - 9:34am
Sure.

Sure.

geoffkait
geoffkait's picture
Offline
Last seen: 1 day 32 min ago
Joined: Apr 29 2008 - 5:10am
That’s how the military

That’s how the military has always worked. Take the high ground! That way you can see what’s coming and where to direct gunfire. The Ant Hill, Iwo Jima, Hamburger Hill, Pork Chop Hill, etc. For hearing we have radar. The sense of hearing has its place. Right behind the sense of vision. :-) The sense of taste is probably no. 3. Mmmm, hamburger, pork chops!

Early man was stalked by many animals, e.g., bears, saber tooth tigers, that viewed man as prey. Have you ever watched a lion or tiger stalk its prey? These prehistoric animals were adept at silent stalking, they had to eat too, so listening for them wouldn’t be nearly as effective as seeing them, you know, in terms of survival. Plus, by the time early man heard them it would have been too late. Fortunately, prehistoric animals that preyed on prehistoric man could not make themselves invisible.

  • X