Conflicting Data?

Last week, posted an article concerning Pollara, Inc.'s Canadian Recording Industry Association–commissioned 144-page report on the downloading habits of Canadian music consumers. We reported that the University of Ottawa's Dr. Michael Geist interpreted the Pollara data differently than the polling group did, in particular noting his conclusion that people who had downloaded music had legally purchased more music than their counterparts who had never done so.

On Wednesday, March 22, John Atkinson received—and forwarded to me—an email from Duncan McKie, president of Pollara, Inc., referring us to his "Research Analysis: Geist Claims Radio Research Contradicts CRIA Positions, Pollara Response," an 11-page document which alternates Geist's claims with Pollara's rebuttals. Pollara's take? Geist was "impertinent and presumptuous."

Having linked to both the CRIA survey and Geist's dissection of its findings, I felt it only fair to offer our readers access to the rebuttal as well. Read all of it and judge for yourself—but I had problems swallowing some of it, beginning with a niggling footnote on page two: "Our observation is that there is probably a good deal of 'overestimation' in these numbers by informants—however, we choose to use them 'as reported' despite what seems to be the overclaiming given the known statistics on music file commerce."

What makes this such a telling comment is that it is appended to Pollara's objection that Geist makes a misleading distinction between downloading activity and the music people had on their hard drives. The problem is that the question didn't ask about downloading, it asked people how they acquired the music on their computers. The footnote seems to editorialize that the stats are, in fact, cooked by the respondents, which seems a strange defense for a polling company to offer under any circumstances.

It also seemed to me that McKie's restatement of the polling questions in Pollara's rebuttal recasts several of these questions, which causes me to wonder just how rigorous the company's question-generating methodology was to begin with. It's an old saw in the legal profession that, in deposing a witness, one should never ask a question one doesn't know the answer to—that's just another way of saying "choose your questions to generate the answers you're looking for."

The concluding paragraph of McKie's "Research Analysis" states that the music industry's decline is solidly tied to the rise of downloading: "It is now considered almost common knowledge that this relationship exists, except among diehards and others whose contrary views imply a desire to protect what is increasingly considered an illegal and damaging practice." This is interesting, since I had assumed that the objective of the survey (at least as expressed on its page labeled "Objective") was "to provide objective, empirical research" rather than to parrot what is "almost considered common knowledge." Perhaps I am a "diehard," but I don't think Pollara's CRIA survey or its rebuttal to Geist have done that. Pollara, McKie, and the CRIA may well be correct in their analysis of the situation; all I'm saying is that, in my opinion, they still need to prove it.