Hard News by Russell Brown

This week's Media7 was a discussion about public service broadcasting and its absence. But it was also a story about how a bogus statistic could be created and then used repeatedly by those whose arguments it suited, to the extent that everyone thought it was the truth.

I noted in an earlier post this week that a weekly cumulative audience figure of 207,000 for TVNZ 7 had been cited twice in New Zealand Herald editorials pressing the case for TVNZ 7's defunding and demise. The first, headed Poor TVNZ 7 ratings justify pulling plug,  in April last year, said this:

The first of these is the $15 million cost of running the channel each year. The second is the mere 207,000 viewers a week that it attracts.  This really is television for minority interests taken to extreme. By way of contrast, TVNZ's One News attracts 600,000 viewers each and every night. 

The comparison between a weekly channel cume and the nightly ratings for what is often TVNZ's highest-rating programme was, to anyone familiar with such figures, a nonsense. But it didn't stop there.

The cited 207,000 weekly audience was a mystery. TVNZ does not report such a measure. Nielsen does not compile such a measure. AC Nielsen New Zealand's associate director Caroline Atford, told our researcher Sam Mulgrew that the figure was "rubbish".

TVNZ programmer Juliet Jensen had attempted to tell the Herald as much in a polite letter that noted that TVNZ had no idea where the incorrect figure had come from and offered the correct TVNZ 7 monthly cumes as recorded by Nielsen (1.47 million for December). The letter was not published and she never received a response.

It wasn't hard to determine that both the 207,000 figure and the inappropriate comparison with One News had been uttered by former Broadcasting minister Jonathan Coleman, in an interview for a Herald news story, to justify Cabinet's decision not to renew TVNZ 7's funding. But that didn't explain where the figure came from in the first place.

It occured to me, looking at the monthly cumes, that such a figure might have been conjured if someone had tried to calculate a weekly cume -- inevitably, a smaller number -- by dividing a monthly cume by four. This would of course be completely invalid. A monthly cume is the tally of all unique individuals who watched the channel in a month. The only way a weekly cume of 207,000 would be a quarter of a monthly cume is if 207,000 different people watched in each single week -- and no other week -- of the monthly period.

I asked Sam to check the month before Coleman's announcement for evidence of such a bamboozle. He did, and then looked back a bit further in tables Nielsen had provided to us. And he found it.

TVNZ 7's cume for the four weeks beginning on the 23rd of January 2011 (Nielsen always reports four-weekly "monthly" periods for the sake of consistency), the channel's quietest month of the year, was 827,800.

You're probably way ahead of me here, but: if you divide that number by four, you get -- more or less --207,000. Except, again, that’s not how it works. You don't get a weekly cume by dividing a monthly cume by four.

I discussed the bogus figure on the show with Throng's Rachel Cunliffe, a former lecturer in statistics at the University of Auckland. In theory, a the weekly cume broken out of a monthly cume of 827,800 could be anything between zero and 827,800, but, as Rachel said, it would be far more likely to be nearer the higher number, given that TV viewing is largely a weekly activity.

She agreed that a weekly cume of 600,000 -- the very figure used in a derisive comparison by the minister and the newspaper's editorial writer -- would be a reasonable estimate in the circumstances.

So: an audience figure that has come to be accepted as valid even by supporters of TVNZ 7's retention was probably incorrect by a factor of about 200%. (Update: let me rephrase that. It is more precise to say the figure was probably incorrect by a factor of about three, or that the actual audience was about 300% of the figure used by the minister and the newspaper.) And even that was the lowest survey month of the year.

The first blame for this idiot math must lie with the minister and his office. I am not paying my taxes for this kind of ineptitude.

But the Herald repeatedly used this figure from the minister without attempting to check it -- and did not seek to verify it even after being told it was bogus. It's hard to not feel that the ongoing feud between TVNZ and the Herald's owner APN (which extends to the absence of TVNZ 7 listings from the Herald and The Listener) was a factor here. The public is poorly served by this sort of  private jousting in matters of the public interest.

We got an email from Herald editor Shayne Currie just before the recording of the show, too late to be usefully included. But I'm happy to quote it in full here.:

The first point to note here is that Shayne Currie still hasn't acknowledged the original error. Instead, he has referred to a new and completely different figure (one which happens to be close to the original bogus number), for a daily audience. The second point is that the daily figure is probably correct. The third is that this daily figure is greater than the weekly figure Currie's newspaper has been quoting.

Sam called Caroline Atford the day after the recording. She did not confirm the figure that Currie quoted, but did not deny it. She did express (to put it mildly) great surprise that the Herald had such a figure, saying "[Currie] does not have access to that figure and he certainly shouldn't be telling you what that number is."

I disagree. The use of sources is a part of journalism. If you get accurate and relevant information from a source, you use it. (I have a good idea of who the source was, but it would be imprudent to say.) But Nielsen has a front door. You can knock on it any day, just like we did, and Nielsen will provide you with information and guidance. The Herald never did that, even when told its number was wrong.

I also agree with Shayne Currie that "the TVNZ7 audience to NZH audience comparison was never the point". I contributed to that impression in the earlier post, but only to demonstrate that the Herald was on perilous ground mocking TVNZ 7's audience, given that it was within the range of the paper's own readership. I didn't have access to Herald's monthly figures because they're not published but I'm more than happy to have them aired here.

So it's not a pissing contest with the Herald. But as a reader of the Herald, I'm interested in the Herald getting its act together on facts. I rely on it.

Two or three years ago I participated in a seminar on the judiciary and the media, speaking on behalf of the blogsophere. At one point, I addressed the point of what the blogosphere thought of the mainstream media. "To be honest," I said, "bloggers think you guys can't count." I looked up and saw Tim Murphy scowling at me from the audience. Well, I'm sorry, but it was true and it's still true.

At the time of the seminar, our own Keith Ng had published a blazing post about the Herald's staggeringly wrong use of statistics on school violence. In recent weeks the authors of Auckland University's excellent StatsChat blog have engaged in head>desk interaction over the Herald's use of statistics in this post on a story about marijuana potency, this one on a "well beyond dodgy" story based on a reader poll on attitudes towards smoking in public places, and one on the could-hardly-be-more wrong-if-it-tried infidelity story.

It's hardly just the Herald, of course. I'm sometimes amazed by the dismissive attitude of editors towards what must surely, in the age of data journalism, be a core competency.

But to the Herald's editors I would say this: you were given a bogus statistic by a government minister looking to frame a story in a particular way. Instead of scrutinising it, you parroted it repeatedly via your editorial voice. That's not exactly speaking truth to power.

On the evidence of this week, I'd have to say there's a place for public service broadcasting yet.

 

You can watch this week's episode of Media7 here.