As We See It

Michael Fremer's review of the AudioPhysic Caldera III loudspeaker in this issue (p.81) reminded me of a subject I have written about many times in the past: what happens when a manufacturer submits a faulty sample for review. I formalized Stereophile's policy on this matter in late 1988, following both an unfortunate series of reviews in which the samples either arrived broken or broke during the auditioning, and my learning about how much went on behind closed doors at other audio magazines, where reviewers and editors too often appeared to collude with manufacturers.1 I wrote back then that:

"Once a writer has embarked upon a review for Stereophile, nothing can stop that review from eventually appearing in print....If a product sounds bad but appears to be working properly, then we will proceed with the review on the grounds that it is typical of production. If it eventually turns out that a loudspeaker's crossover was wired incorrectly, or that the wrong transistors were installed, or that the hum was due to a production fault, we will happily review the corrected version, but [the magazine's readers] will always learn about the original problem....[T]hey need to know that a manufacturer can't always supply a working sample, even when the user will be a reviewer.

"If a product does turn out to be faulty, or just doesn't work out of the box, we ask for a second sample....Similarly, when a manufacturer asks if he can send an updated version, we comply on the grounds that we need to be able to describe the most recent sample for the review to be relevant. However, the writers are instructed to include in their reviews all their experience with all the samples they've received, not just the most recent or best-functioning....

"After a writer has submitted his review copy to the magazine, we send a preprint of the edited report to the manufacturer in time for them to send us a 'Manufacturers' Comment.' That is often when the cow cakes hit the fan."

I returned to this subject in February 1989, emphasizing that everything that happens during the course of a Stereophile review is "on the record":

"If it is possible for a Stereophile reviewer to receive a faulty or unrepresentative sample of a component, with all that would be at stake were the product to receive a negative review, then it is probably more likely for one of the magazine's readers to do so. The magazine's primary responsibility is to its readers. Ergo, quality-control problems must be reported in the review, and to make exceptions for some companies would be both inconsistent and unfair.

"Against this, it could be argued that a reviewer is more likely to receive a defective sample than a reader, he or she often experiencing a sample from the first production run. After all, doesn't every magazine want the very first review sample of any product? In addition, the dealer is there to act as a buffer between a manufacturer's lack of QC and the consumer: if a company sends out products to its dealers that turn out to be defective, it gets them straight back. [However,] a magazine reviewer's listening room is a singularly inappropriate Beta-test site. The reviewer represents the interests of his or her readers, not those of the manufacturers. Not to inform the readers of bugs and failures, perhaps substituting a private word with the appropriate manufacturer, borders on behavior nearer to that of a consultant than a reviewer.

"It is also relevant to a review's findings whether or not the manufacturer can make their products to a consistent standard. It was the late Spencer Hughes of Spendor who succinctly defined the skills required of a manufacturer: first, to be able to design a worthy product; second, to be able to make it consistently and reliably. Each is as important as the other, and both are relevant to the consumer, not just the first."

Nothing that has happened in the 17 years since I wrote those words has persuaded me that this policy is misguided. The important point is that all manufacturers are accorded the same treatment. However, I admit that it is a hard-nosed policy when we are confronted with a new product from a new company that may have teething problems, which is why I also instituted our "Five Dealer Rule" back in the 1980s (most recently described in January 2003). I want the products we review to be real. Giving manufacturers a small commercial hurdle to overcome by acquiring five dealers goes a long way toward ensuring that this will be the case. Again, the first person other than its designer to audition a high-end product should not be a magazine reviewer.

I repeat: Merely sounding disappointing is not in itself evidence that a review sample is broken or defective. However, it does beg the question: How do we decide that a product is faulty? A component's producing no sound at all is really the only incontrovertible evidence. All other judgments are subjective. In the case of loudspeakers (or monoblock amplifiers), measuring or listening to the second sample in the pair will, of course, provide an immediate check on the first.

This is what happened with the Caldera III. When Audio Physic's Dieter Kratochwil received the preprint of the original review, he immediately suspected that the sample I had measured was broken, and sent me measurements of a Caldera III he had selected at random from production. This sample measured very differently in the treble from the sample (serial number 002A) on which I had performed a full set of measurements, so I agreed to measure the other of the original pair (s.n. 002B). As you can see from the revised review, 002B measured identically to 002A, meaning that if there was a fault, it was a systemic one. I therefore asked for Michael to be sent a second pair of Caldera IIIs (s.n. 057A/B), on which he reports favorably in the review. I would rather have used the time and space we spent on the second pair of Calderas on a review of another product, but...

Mr. Kratochwil was concerned about the fact that we had not informed him that s.n. 002A/B did not sound as good as we should have expected before the review was written. But as I have written above, it is my strict policy not to provide feedback to the manufacturer or distributor before they see the preprint. Stereophile publishes reviews, not collaborations. I am not saying that to publish reviews like that of the Caldera III is any more satisfying to me than it is to Audio Physic, only that it is necessary.

---

Footnote 1: For example, Larry Klein explained at a 1990 AES Conference that this sort of collaboration was Stereo Review's policy when he was that magazine's Technical Editor back in the early 1980s.