Monday, August 25, 2008

Why audience measurement is so difficult

There are numerous resources for online audience measurement available today. One of the earliest forms was the page counter which appeared soon after the now retired scrolling text on web pages. 

Today some of the online analytics include Google AnalyticsQuantcast, NielsencomScore, Hitwise, and others. Though there seems to be quite a bit of controversy and debate over who's numbers are right, who's are wrong and why. A recent example has been the coverage over comScore and Google's Ad Planner and Trends for Web sites. After the announcement of Google's Ad Planner, comScore's stock price took a real hit. Headlines like Google Comscore Killer? didn't help the situation and helped stir a lot of reaction across the web. Once things started to settle down new headlines of a different tone "Google Ad Planner No comScore-Killer After All" started to appear.

A primary reason why audience measurement is so difficult, I believe is due to a lack of understanding of some of the fundamentals of audience measurement. This includes principals around sampling, panel recruitment and measurement. I am a data-junkie and data is only good as long as it is within context and when there is a thorough understanding of it's limitations, methodology and assumptions.

I think Fred Wilson summarizes things well in his post Checking Out Google Trends for Web sites

. . .  Everyone who provides third party measurement starts out with a data set that is skewed in some way. The trick is to understand how your data set is skewed and apply statistics to take that bias out. Firms like ComScore, Compete, Hitwise, and NetRatings who sell their data have invested heavily for many years in reconciling their data to server logs and internal analytics. And that makes their data better.
Instead of everyone wondering who's data is better, it would be nice to see an unbiased review of the different data sources summarizing the benefits, disadvantages, limitations, methodology and use cases.  

No comments: