Ground Rules For Reviews

As Stereophile's Equipment Reports Editor, I get a lot of calls from readers asking how we choose the gear we review, and from manufacturers asking how to get their products reviewed. So I told JA to take the month off from writing this column so that I could talk about Stereophile's Equipment Reports section.

Stereophile publishes about 130 equipment reports annually but even a magazine as comprehensive as Stereophile can't review all the new audio products that are introduced each year. We look for significant additions to the hi-fi canon: products that offer a lot of performance for the money, or that advance the state of the art, or that have excited audiophiles, or that would excite audiophiles if only they knew about them. We must also include products that perform disappointingly.

Sometimes, we cannot cover a particular piece: some manufacturers refuse to submit their components for consideration, as with Wadia until recently—see RH's Wadia review in this issue. Others are willing to do so only if we grant them concessions we are unwilling to make—such as allowing them to select the reviewer themselves. Even though these products may be worthy of coverage, we must pass.

The biggest hurdle for many companies is getting to the point at which they qualify for review. We have few rules, but they must be met. We didn't formulate them capriciously: These rules are designed to eliminate the not-ready-for-prime-time players, thus ensuring that our readers can experience the products we write about and will be treated professionally by the firms that make those products.

Above all, a company must have a distribution network of no fewer than five US dealers. That's not many, you'd think, but it's surprisingly hard to convince five retailers to take on a new company's products. Most already have more lines than they can represent thoroughly—anything good enough to overcome that is likely to be pretty darned impressive.

"Do you have any idea," asked an aspiring manufacturer when told about the Five-Dealer Rule, "how hard it is to get five dealers without a review?" Actually, we do. That's the point.

We have reviewed merchandise that's only available via mail order, but we still try to ascertain that the firms are really in business: They must offer a satisfaction/money-back guarantee; they must advertise nationally it doesn't have to be in Stereophile); and we try to listen for a "buzz"—if audiophiles are interested and talking about this company/product, we should report on it. Then, case by case, we decide; in and of itself, selling direct does not ensure exemption from the Five-Dealer Rule.

We also look for a "mature" product. An amplifier or speaker that has gone through multiple generations in its first year of life is a product that hasn't been carefully thought out. We don't feel comfortable reviewing those items because they're probably not ready for scrutiny—and there are a lot of worthy products out there that are.

We had someone show up unannounced on our doorstep recently; he'd driven 2000 miles without an appointment just to bring us some components he'd made and was selling direct to consumers. They looked well-made and offered some interesting features, but they didn't qualify for consideration. "What if these were the best in the world?" he asked. "Wouldn't you consider it your duty to inform your readers?" Actually, our duty to our readers is paramount to every other consideration; that's why we couldn't review his products, even though they looked interesting. They had no track record—were they finished products or just prototypes? Besides, our readers need to be able to hear the products we write about and decide for themselves. That's another important reason for the Five-Dealer Rule: it guarantees that the stuff is out there.

Doesn't this favor larger, established firms over younger, struggling ones? In some ways it does—but then, consumers have a better idea of what to expect when dealing with established companies. Products they bring to market are generally mature and fully realized. These firms have fair and consistently honored warranty programs, not to mention widespread dealer networks. Such considerations bode well for potential customers. Our rules attempt to ensure that less well known, younger audio companies whose products we review are ready to offer similar services.

Sometimes we get fooled. A speaker manufacturer, unable to meet the Five-Dealer Rule, managed to convince me that he sold direct via nationally advertised mail-order. I had my doubts, but he seemed sincere, and I was too eager to please. My review, as it turned out, was highly critical of the speakers—which just weren't ready to be reviewed (or marketed, as dealers would have surely told him). He wasn't happy with the review, and I didn't enjoy writing it—nor am I wild about having used up precious space on his hobby. I could have used those pages to review something that you, the reader, needed to know about—something made by a firm that's really in business.

On another occasion, a manufacturer lied to us about meeting the Five-Dealer Rule. We published a positive review, which resulted in a slew of orders. However, consumers complained about not receiving merchandise in a timely manner, so we checked up. It turned out there was no dealer network, no factory, and no business—just a mail-drop and an answering service. We felt used—hell, we were used—just to get to you. That'll never happen again. The guy that perpetrated that hoax called me the other day to tell me about his new product. You'll never read about it in Stereophile.

Ultimately, we aren't in business to serve as the marketing arm of any audio company—whether new and deserving or old and established. We're here to report to you—honestly and directly. When it comes to that commitment, we don't deal, and we don't compromise.—Wes Phillips