Friday, July 18, 2008

Email panel response rates


The topic of response rates and online email panels can provide some interesting discussion amongst research professionals. Response rates in general are important for all research and is sometimes referred to as a measure of panel quality or a measure of risk on data quality. Outside of some statistical reasons, there is growing discussion on how to calculate and report response rates - or if you should at all.

Calculating response rates for an online email panel can be rather challenging given some of the questions around methodology, panel recruitment, email delivery, email bounce, shared accounts and so on. 

Online sample is a hot commodity these days, the industry is strong despite some real challenges. Though while I say it is 'hot', it is not yet a 'commodity' as methodology is still charging non-commodity prices. Enough on that for now so back to response rates.

Whether a researcher expecting a online sample provider to report response rates, or a sample provider reporting response rates, both need to realize some of the short-comings of such a metric. As an example, what does it really mean if someone says their panel can get a 70% + response rate? On the other hand, should anyone expect this from an online email panel.

Here are a few scenarios where a 70%+ response rate on an email panel may be possible:
  1. You, as a panel manager, do everything right (or at least most things). Recruitment is from credible sources, respondents are not 'touched' too many times that they get fatigued, respondents aren't ignored so much that they forget about you. Surveys are fun, engaging and relevant. Incentives are reasonable but not unrealistic. Your respondent, has developed some level of loyalty to your brand.
  2. Non-responsive panel members are purged on a regular basis. This leaves you with very responsive members, thus a higher response rate.
  3. Email members who have responded in the last few days.
  4. Primarily email 'new' members. New members are mostly likely to participate when they are fresh.
  5. A panel consisting primarily of professional respondents who take everything.
  6. Simply lie about the response rate.
Most cases are a mix of the above and hopefully #6 doesn't ever happen. In many cases email panel managers simply don't report response rates since low numbers don't look attractive and many attribute it to a reflection on panel quality.

So the next time you ask for a response rate, or report one, consider carefully what your metric really reflects. Is a 70% response rate really much better than a 4% response rate?

And for the record, the list above is not exhaustive, just a few extremes. Please add yours to the list.

No comments: