SDMI Conducts Phase II Watermark Tests in Nashville
the little man who wasn't there.
I saw him there again today;
oh, how I wish he'd go away.
This childhood ditty about a malevolent, ephemeral figure pretty well sums up the audiophile position on watermarking: the mere thought of it is disturbing. Of the respondents to last week's Stereophile "Vote!" question, 90% say they hate the concept or plan to boycott watermarked music. That's quite a bit of animosity toward something that, at this point, is only experimental.
What's really disturbing is running up against your own inadequacies in trying to hear something when you don't know what you're listening for. That's exactly the situation that a dozen or so music- and audio-industry volunteers found ourselves in October 12 and 13, at Denny Purcell's Georgetown Masters Studios in Nashville, site of the SDMI Phase II watermark tests. Conducted by SDMI consultant Laurence Shear, the tests were intended to determine the audibility of the three remaining contenders for an acceptable watermarking technology, winnowed down from an original 16. Four different technologies had been scheduled for testing at this session, but SDMI had eliminated one of them the morning of October 12 due to its ease of detectability, Shear told us.
The setup was basically the same as it had been the previous summer, when we were here for the first round of SDMI testing. In the interim, Purcell had upgraded the monitors in his main mastering room from the Nova Applause loudspeakers to the massive Nova Evolution IIs. Driven by Pass Labs X-1000 amplifiers via biwired Cardas Neutral Reference cables, the big Novas sounded very well-balanced and very revealing—exactly what a mastering engineer needs. (Nova Audio, incidentally, is developing a self-powered loudspeaker for both home and professional use, to debut at the upcoming CES.) The signal source was a Sonic Solutions USP 24/96 workstation running Sonic Version 5.4 (beta) software, outputting a pair of time-aligned and level-matched 24-bit/96kHz AES datastreams, one clean and one watermarked. A handheld ABX switch box routed either datastream to a Z-Systems 16.16 Digital Detangler, whose single AES output fed a dCS 954 D/A converter. The analog output fed a Pass Labs Aleph P preamp; interconnects were Nordost Quatro Fil. Acoustically, Purcell's main room is very well-isolated from outside noises, and carefully treated inside for minimum reverberation. In short, both system and room would pass muster with the most fanatical audio perfectionist.
Unlike the Phase I testing, in which we were offered only one selection of music, here we had a choice of eight: an assortment of jazz, classical, and rock. As did several others, I chose a slow-tempo Diana Krall piece, thinking that the textures and dynamic contrasts of the more upbeat tracks would make spotting the watermarks more difficult.
On the first pass, designated "Technology 1," I reacted by gut instinct rather than attempting to analyze what I was hearing and scored 6 out of 8 correct. On the second pass, I couldn't resist trying to make conscious comparisons between "A," the clean copy, and "B," the watermarked version, and scored only 4 out of 8 correct—the same as random chance. On the third pass, I tried comparing only "B" to "X," thinking that might simplify the process, but still got only four correct.
Kevin Lee and Billy Curtis of Nova Audio scored similarly; consultant Chris English scored slightly better, but still well below the "95% confidence level" that SDMI is seeking. Engineer Chuck Ainley scored 8 of 8 correct on the first, 3 of 8 on the second, and 1 of 8 on the third; his first and third scores are the kinds of results that will encourage SDMI to pass or fail a watermarking technique, because such numbers approach the target confidence level. After each pass, Laurence Shear asked for and noted our subjective impressions.
Like his boss, Sony VP of engineering Malcolm Davidson, Shear is the embodiment of patience and diplomacy. He had just come from two days of testing in New York, and spent approximately 20 hours conducting tests in Nashville—all of it listening to the same few pieces of music. Although his test setup allowed him to play the watermarks without the masking effect of the music, we weren't allowed to hear those. (I theorized later that inverting the polarity of the clean copy and mixing it equally with the watermarked version would subtract the music and reveal the watermark.) Shear did show me the watermark that had been eliminated that morning. "Once you hear it, you'll always hear it," he said, echoing cautionary remarks that have been made by Malcolm Omar Hawksford, Tony Faulkner, and others. This particular watermark was an intermittent, low-level warble tone buried in the lower-midrange/upper-bass region. Shear was right. The tone wasn't apparent on first listening, but after I knew what to listen for, it was easy to spot—hence its elimination from the competition.
Detecting the other "Technologies" was obviously more difficult. While Ainley was in the studio, the rest of us sat around Purcell's waiting room and discussed what we had just done. Unlike the London tests, in which some people had complained about the noise level, Purcell's room was dead quiet. Still, several of us noted that we had been distracted by intermittent switching transients caused by the ABX box. I thought I had heard a pattern of "clicks" buried in the glottal noises in Krall's vocals, but I probably hadn't. Billy Curtis said that he, too, had heard some clicks, but had been unable to identify a pattern. Chris English thought he had heard an irregular pattern of sounds that followed the dynamics of the music. Given that he scored better than the rest of us, his impression may be closer to the facts—but none of us knew anything for certain. Later, we found out that the order of the tests had been randomized, which obviated any conclusions that we might have drawn in post-test discussions.
I have no doubt that many audiophiles are absolutely confident that they could have detected these watermarks the first time every time, to which I can only reply: Go ahead and try. I have no doubt that they would score the way most of us have in both phases of testing: 50/50, the same as random chance. Any skill—such as evaluating the performance of an audio system—is built up over time and with repetition. When you don't know what you're listening for, you might as well be guessing.
In fact, that's what most of us did. The mind plays plenty of tricks under stress, which in itself probably serves to obscure rather than reveal the pattern you hope to identify. SDMI officials have acknowledged that there will always be a tiny minority of people who can spot the watermarks with a high degree of accuracy, and that a technology that can fool all the people all the time may never be developed.
When I emerged from the test chamber, I went downstairs to Purcell's front office and checked my e-mail. There were several messages—from Jon Iverson, John Atkinson, Jonathan Scull, and Tony Faulkner—all alerting me to a story that had popped up that day on Salon.com, the San Francisco–based online magazine. The piece, by Janelle Brown, alludes to the success of hackers in breaking all of the SDMI watermarking technologies under test that day in Nashville. Brown's report relies heavily on unidentified sources and third-hand information, and has not been verified—or even picked up—by other publications.
As of today (Sunday, October 15), SDMI has been eerily quiet about the whole affair. Laurence Shear did tell us in Nashville that Malcolm Davidson couldn't be present due to "SDMI meetings in Los Angeles." Whether Janelle Brown's report is accurate or not isn't clear at the moment. Neither do we know if all the testing that has taken place over the past two years will ultimately take watermarking to SDMI's stated goal: copyright-secure but acoustically transparent high-resolution recordings. The visible drama—not to mention the backstage intrigue—is far from over.