To the Simple, Everything Appears Simple
I have no argument with that statement. But then, Madden went on to say, "Have someone hook up this gizmo on YOUR system, and then have you listen to it with the same song 10 times with and without it connected randomly, and if you get the 'better sound with it' right 9 times out of 10 then I would be convinced that it makes a difference to the sound."
Sounds like a simple test, but designing a blind test that can be used to confirm or deny that a real but small audible difference exists is far from simple. In the formal statistical analysis of the test results, you can't prove a negative; you can conclude only that, under the circumstances of the test, no difference could be detected. By contrast, a statistically significant positive identification can be regarded as universal proof that a difference is detectable. But that analysis depends on the test examining just one variablethe difference being examinedand, as I have repeatedly discussed in this magazine, the blind-testing methodology itself can be an interfering variable in the test. The fact that the listener is in a different state of mind in a blind test than he or she would be when listening to music becomes a factor.
Rigorous blind testing, if it is to produce valid results, thus becomes a lengthy and time-consuming affair using listeners who are experienced and comfortable with the test procedure. Otherwise, the results of the test become randomized, hence meaningless.
In the words of famed mastering engineer Bob Katz: "There is no such thing as a 'casual' blind test. Blind tests are a serious business. Experimenters need training how to perform blind tests well. Blind tests can fail (produce statistically invalid results) if the experimenter neglected one critical detail. Weeks of intensive study are required to learn how to perform blind tests. Then weeks of preparation to create the test. Then weeks of testing to follow."
Some probably think it paradoxical for the editor of a magazine based primarily on the concept of judging audio components by listening to them under sighted conditions to be commenting on blind-testing methodology. However, since the very first blind listening test I took part in, in 1977, organized by the late James Moir for Hi-Fi News magazine, I have been involved in well over 100 such tests, as listener, proctor, or organizer. My opinion on their efficacy and how difficult it is to get valid results and not false negativesie, reporting that no difference could be heard when a small but real audible difference existshas been formed as the result of that experience.
There is, in fact, a formal discipline devoted to the design of blind tests, based on recommendations formulated by the International Telecommunications Union in its document ITU-R BS1116-3 (footnote 1). Katz was summarizing the ITU guidelines and their consequences; the context for his comments was a workshop at the 139th Audio Engineering Society Convention (footnote 2), held last October in New York, on the audibility of possible improvements in sound quality made by recording and playing back audio with bit depths greater than the CD's 16 and sample rates higher than the CD's 44.1kHz.
This is a contentious subject. On the Stereophile website forum last summer, reader David Harper wrote, "Humans do not hear any difference between 16-bit/44.1kHz and any higher bit/sampling rate. This is established fact."
Harper was referring to a 2007 paper by E. Brad Meyer and David R. Moran that "proved" that there was no sonic advantage to high-resolution audio formats (footnote 3). Their conclusion ran counter to the experience of many recording engineers, academics, and audiophiles, but other than doubts over their methodology and the fact that their source material was of unknown provenance, Meyer and Moran's paper seemed to be the final formal word on the matter.
Until now. The AES workshop in which Bob Katz was taking part also featured presentations by legendary recording engineer George Massenburg (now a Professor at McGill University, in Montreal) and binaural recording specialist Bob Schulein. But it was the first presentationby Joshua Reiss, of Queen Mary University, in London, and a member of the AES Board of Governorsthat caught my attention.
Some 80 papers have now been published on high-resolution audio, about half of which included blind tests. The results of those tests, however, have been mixed, which would seem to confirm Meyer and Moran's findings. However, around 20 of the published tests included sufficient experimental detail and data to allow Dr. Reiss to perform a meta-analysisliterally, an analysis of the analyses (footnote 4). Reiss showed that, although the individual tests had mixed results, the overall result was that trained listeners could distinguish between hi-rez recordings and their CD equivalents under blind conditions, and to a high degree of statistical significance.
Footnote 1: See "Methods for the Subjective Assessment of Small Impairments in Audio Systems."
Footnote 2: See "Perceptual Evaluation of High Resolution Audio."
Footnote 3: Meyer, E. Brad, and Moran, David R. "Audibility of a CD-standard A/DA/A Loop Inserted into High-Resolution Audio Playback," JAES, September 2007, AES E-library 55 (9) 775779.
Footnote 4: Dr. Reiss reported that it didn't prove possible to include Meyer and Moran's data in the meta-analysis, because there were statistical inconsistencies. In particular, the number of listeners who scored positively was just 3 out of 55: far fewer than would be expected if the experimental data were truly random.