Horses for Courses
That I used the word evaluate and avoided measure sums up the controversy: Are we better served by measuring audio performance by objective physical standards, or by judging audio performance by subjective personal reactions?
At first blush, the objectivists would seem to have the better argument. The proposition that a component that "measures better" therefore "is better" seems hard to argue with.
But what I would like to explore are two follow-up questions that don't seem to get asked much: "How much 'better,' really?" and "Does that really make a difference?"
That's where the old maxim horses for courses comes in. This 19th-century expression of English horse-racing is most often taken to mean that you can't judge a horse in a vacuum. You judge a horse by its suitability for the track on which it will run. One horse will do better on a dry track; another will do better on a wet track.
In wider modern usage, horses for courses means that something that works for one person or situation might not be as suitable for another person or situation. However, I'd like to use horses for courses in its most straightforward sense: that you should judge a racehorse by its performance on a racetrack and not by any other criterion, such as how handsome the horse looks, how elegant its gait is, or how sweet or nasty its disposition.
Making the transition from horses to audio equipment, horses for courses implies that you should evaluate audio equipment by listening to music on it—seemingly a win for the subjectivists. However, that transition is fraught with logical pitfalls.
The difference between a horse's running a race and an audio system's playing The Birth of the Cool is that most races are clearly won by one horse or another. The winner is the horse that gets across the finish line first. However, deciding whether The Birth of the Cool sounds "better" on Loudspeaker A or Loudspeaker B is not as easy.
The decision matrix for judging a horse race has only one dimension, and that dimension is objective: which horse crossed the finish line first? The decision matrix for deciding which loudspeaker is "better," however, is multidimensional. Even confining the discussion to things that can be measured, there is frequency-response linearity, frequency extension, dispersion, dynamic range and linearity, and so on. Opening up the field to include criteria such as "musicality" and "pace" only makes a complicated situation more so.
For lack of a better way to do it, we measure the objective aspects of audio performance separately. This reductionistic analysis of performance has the virtue of objectivity—the measurements are what they are. But the reductionistic method, while objective, lacks a holistic dimension—how does the entire loudspeaker perform?
A unique, 40-page group test published some years ago in my favorite car magazine, Classic & Sports Car, employed a holistic but objective comparative measure of sports-car performance. The results were surprising, and may give us some guidance toward grounds for a cease-fire in the Objectivist vs Subjectivist Audio Wars.
The usual way to test sports-car performance is to separately time (or otherwise measure) acceleration, top speed, braking, g-forces on a skid pad, and sometimes a slalom run. These separate performance tests often show differences that appear to be significant.
If someone were to tell you that a 1967 Jaguar E-Type with four-wheel power disc brakes and wide tires comes to a dead stop from 60mph sooner than does a 1929 Bugatti Type 35B with cable-actuated manual drum brakes and skinny tires, you'd hardly be surprised. That the Jaguar has a higher top speed than the Bugatti will similarly fail to shock. But, in holistic terms, how much of a difference do such one-dimensional data really make?
Classic & Sports Car found that the answer is: not as much as one might think. For its test, C&S lined up $5 million worth of classic cars—10 important examples, from a 1904 Mercedes 28/32 Grand Prix to a 1997 Lotus Elise. Reasoning that the purpose of a sports car is to get around a road course as quickly as possible, C&S timed the cars on flying laps of a half-mile course consisting of a front straight, a back straight with a chicane, and asymmetrical curves connecting them. The flying lap, where the car starts at top speed rather than a standing start—and the short length of the course—meant that newer cars' acceleration from a standstill and top-speed advantages would not be decisive. All 10 cars were driven by John Surtees, still the only world champion on both two and four wheels.
The surprising result: Taking flying laps on the kind of road course that amateurs might run time trials on, the 1929 Bugatti's lap time was within fractions of a second of those set by a 1956 Mercedes 300SL Gullwing, a 1961 Ferrari 250SWB, a 1967 Jaguar E-Type, and a 1971 Mini Cooper S. To take one example, the Jaguar's 30.135 seconds was 0.314s faster than the Bugatti's 30.449s. Thirty-eight years of progress in engines, brakes, suspension, steering, and tires, and the newer car is only about a third of a second faster around the track. Horses for courses.
Not until Porsche's 911 Carrera 2.7RS of 1973, universally regarded as a true modern supercar, did a car set a lap time that was decisively better than the Bugatti's best time—27.919s vs 30.449s, or a difference of 2.53s. (The course record of 26.84s was set in a Pilbeam.)
What do Classic & Sports Car's results tell us? Perhaps that measuring one dimension of audio performance at a time may mislead us into thinking that the measured differences will make for more significant differences while listening to music than they actually do.
In the same way as the 1929 Bugatti held its own very nicely against decades of more modern competition, darTZeel's NHB-108 amplifier, which has "old-fashioned" distortion measurements, and Quad's ESL-57 and ESP's Concert Grand loudspeakers, which have "old-fashioned" frequency-response measurements (deliberately, in the case of the ESP), just might get around the road course of The Birth of the Cool in such a way as to confound objectivists and delight subjectivists. Horses for courses.
Classic & Sports Car's "holistic but objective" road test was a brilliant inspiration. Unfortunately, try as I might, I can't come up with an analogous holistic but objective audio test.
I guess we'll just have to listen—with open ears, minds, and hearts.