The trouble with numbers

Counting on confusion

Why no one knows how manypeople are visiting their websites

What if everyone is wrong?

The mystery emerges

• Boston.com, BostonHerald.com complain of lowballing

The mystery emerges

• Boston.com, BostonHerald.com complain of lowballing

• Internal counts are three times higherthan Nielsen ratings

The mystery emerges

• Boston.com, BostonHerald.com complain of lowballing

• Internal counts are three times higherthan Nielsen ratings

• Whose count is accurate? Agencies’ or content-providers’?

Panel-based metrics

• Nielsen and comScore sample what sites audience is visiting

Panel-based metrics

• Nielsen and comScore sample what sites audience is visiting

• Afternoon traffic is undercounted, local sites are a challenge

Panel-based metrics

• Nielsen and comScore sample what sites audience is visiting

• Afternoon traffic is undercounted, local sites are a challenge

• Nielsen once gave washingtonpost.com 10m and comScore 17m

Server-side metrics

• An actual count of incoming traffic.What could go wrong?

Server-side metrics

• An actual count of incoming traffic.What could go wrong?

• Visitors are counted by device. Less accurateif they clear cookies.

Server-side metrics

• An actual count of incoming traffic.What could go wrong?

• Visitors are counted by device. Less accurateif they clear cookies.

• No way of separating out robotic hitsfrom search engines

What is a unique visitor?

• Most agree it is someone who visited once during time span

What is a unique visitor?

• Most agree it is someone who visited once during time span

• Unique visitors per month is most common Internet traffic measure

What is a unique visitor?

• Most agree it is someone who visited once during time span

• Unique visitors per month is most common Internet traffic measure

• Other measures: return visits, number of page views, time spent

Dearth of free data

• Compete.com is best known, samples a panel of 2 million people

Dearth of free data

• Compete.com is best known, samples a panel of 2 million people

• Quantcast gets good reviews from some, but code must be installed

Dearth of free data

• Compete.com is best known, samples a panel of 2 million people

• Quantcast gets good reviews from some, but code must be installed

• Alexa provides a lotof data, but results seem very odd

Was newspaper data better?

• No one knows how many people actually open up the paper

Was newspaper data better?

• No one knows how many people actually open up the paper

• Multipliers of “readers” versus “subscribers”is highly suspect

Was newspaper data better?

• No one knows how many people actually open up the paper

• Multipliers of “readers” versus “subscribers”is highly suspect

• Chicanery to make numbers look higheris not unusual

Credits

• Much of this presentation is based on “Confusion Online: Faulty Metrics and the Future of Digital Journalism,” by Lucas Graves, John Kelly and Marissa Cluck, Tow Center for Digital Journalism, Columbia University Graduate School of Journalism, September 2010.