Monday, May 26, 2008

The problem with surveys

Surveys are broken. It's a bold statement but I'm not sure if it's far off from being true. Surveys are broken in part due to a lack of innovation in the industry. Market research and the survey related industries suffer from a lack of technical innovation and are generally engineering poor.

I would stand to argue that surveys have changed little over the past 50 years. Sure technology has made it somewhat easier and less expensive but let's take a look at a few changes in the survey as we know it.

Taking a look at some traditional survey techniques, we can start with the mall survey. The premise is that if you intercept people where they conduct a primary activity (shopping) that you'd get a valid sampling population. The basic survey mechanism involved an individual holding a clipboard to ask the unsuspecting shopper as they pass by if they would like to participate in a survey. If they said yes, the interviewer would ask a few basic screening question and if successful, conduct the interview based on a paper survey with question and logic notes. This method is slow, error prone, time consuming and expensive, though still a valid method.

Telephone polling changed the industry since the adoption of the telephone was near 100% in some areas. Using random digit dialing (RDD) a truer random sampling of individuals could be done and all you needed was a room full of phones and interviewers. Today this sector of the industry is facing many challenges due to do-not-call lists and the rise of cell phone use.  Still the survey hasn't changed much. In this case the survey was still mostly the same in either paper format or CATI.

The CATI (Computer Aided Telephone Interview) system added great efficiency to the telephone interviewer. Essentially a survey would be programmed and the interviewer would read off of a computer screen. This system supported basic survey logic (conditional skips) and computerized data collection.

Welcome the Internet and the CATI or paper survey can now be formatted in HTML and presented to an individual online with checkboxes, radio buttons, text boxes - all with logic, nice formatting and drastic cost savings.

Up until this point, the major breakthrough has been in the efficiency of deploying surveys. Online technologies drove the cost of implementing a survey to near commodity levels so to reach 500 or 1,000 individuals become very cost effective.

Though my real disappointment has been that the basic survey has not changed. The paper survey used by the mall or phone interviewer really hasn't changed much when it went live on the Internet.  The effect has largely been a direct translation of the paper survey to the web browser screen.

Surveys or market research in general has a fear of change. Any change in implementation creates fear and concern in validity of data. This fear has stifled innovation in methodologies and implementation. Why not challenge the boring typical linear survey? Sure there are those who are creating surveys with silly flash to flip pages or select items off of a shelf.

In my opinion the value or advantage of doing research online is being drastically under utilized. Take for example email panels. Instead of being innovative, everyone is still trying to build these massive 1,000,000+ email panels. I won't go on my feeling about panels, that's for another time.

Though there is some potential of advancement. Behavioral research is getting a fair bit of attention, and a few companies like Brain Juicer, UThink, comScore  Peanut Labs and HotSpex are getting some recognition for doing innovative things.

Online research is a sector that has been growing incredibly fast and is quite healthy. Though, due to this lack of innovation there are many concerns in it's direction. Some major firms are even starting to pull away from using online research due to the rise of professional survey takers, poor methodologies, weak panels and questionable sampling techniques.

Nearly 10 years ago, we were all trying to justify online research to clients. Today just about everyone is online because it's the place to be. The next generation of research is not far around the corner. It really only takes a little innovation to lead to big changes.


Emiel van Wegen said...

Thanks Paul for an eye-opening post on today's market research industry.

While you mention some niche market players who indeed are known for their innovative approach to surveys, these companies too base their product offering on panel research -- be it in a more innovative way than the big-MR firms.

The real challange for the market research industry in the next 12 years will be the impact of the shift away from measuring "claimed" behaviour (what did you do, when and how much) toward measuring actual behaviour. All new applications and services (think RFID, web 2.0, GPRS, bluetooth) are based on creating databases. These databases generate data for free. So the "only question left to ask" will be why.

Surinder said...

So what's the answer?

Nicolas said...

@surinder Paul started the answer in his post. We need to move away from linear, one-way surveys. Forget about surveys having a start and an end point, a screener and an incidence rate. We need to think in relationships with the customer base and go back and forth with your market. Listen to what they have to say. If you have this relationship with your customers or markets, then you don't need to screen thousands of people, you don't need to fight fraud, you don't need to collect the same data again and again, because you have them already.

Why force respondents to stay focused on your (boring, linear) questionnaire for more then a minute, when today the whole world is getting the idea that in order to keep somebody's attention you need to re-capture his attention every few seconds. (I should say the whole world, but not researchers)

I heard an interesting discussion a few days ago, that all TV and radio talk shows have moved from having one moderator (in the past) to at least two moderator (today), and that the number of guests have increased tremendously.

Why ? Because they figured out how to keep someone listening, and engaged: Every 20 sec another person needs to speak ! Otherwise people zap away.

If you compare today's successful TV shows and compare them to past shows, you'll see that instead of having just one crime (or love affair) as in the past today's shows have multiplied the crimes all happening in parallel. How boring does "Derrick" seam today compared to "24".

Market Researchers inhouse and agencies alike should wake up and really understand that the world has changed ... "Derrick" was tremendously successful in the last century, but in today's world it's outdated.

Paul Neto said...

Nicolas. Thanks for your comment. I think you are right on. There is a lot of competition for a person's attention and market research needs to become more innovative on how to deal with the different sources of data now available.

There are a few major obstacles that must be overcome.
1. We need researchers that are willing to explore these new sources and methods
2. There needs to be some technological innovation within the research firms (and many need to consider looking outside)
3. The need for a key client such as a P&G to embrace and support these new approaches.

These three things together will bring some new opportunities and methods into the headlines.

I'm not saying that there isn't some place for surveys as we know them. I'm just saying that there IS space for something different.

Murtaza said...

Hi Paul:

Thanks for including us in your list of innovators. As you mention in your post, consumers are shifting their communication habits away from phone and email. With more consumers spending more time on social networks, it’s crucial that market researchers follow this shift to meet them where they hang out. By leveraging these pre-existing communities, we feel the market research industry can get a better view of their target demographic. Also, because the surveys are completely relevant to the users’ online community, they become “fun activities”, and are encouraged to participate in the surveys, yielding a higher-quality sample from an engaged panel.

Do check out our blog at


thesis writing said...

this kind of blog always useful for blog readers, it helps people during research. your post is one of the same for blog readers.

read more said...

The information and the detail were just perfect. I think that your perspective is deep, its just well thought out and really fantastic to see someone who knows how to put these thoughts down so well. Great job on this.

The Barry Law Group said...

We certainly adore exactly how it’s simple upon my personal eye and also the truth is well crafted. I’m asking yourself generate an income may be informed each time a brand new publish may be created. I’ve bought for your feed that should have the desired effect! Possess a good day time!

home page said...

The information and the detail were just perfect. I think that your perspective is deep, its just well thought out and really fantastic to see someone who knows how to put these thoughts down so well. Great job on this.