Wednesday, June 18, 2008

Taking multiple surveys - What's the answer?

A recent post on Research Live reports how Greenfield Online is addressing the issue of panel quality with the development of a DRD system (Duplicate Respondent Detection). It is defined as being designed to stop panelists taking surveys more than once and that it will be applied to ALL projects from now on.

The part that I don't fully understand is the methodology which is described as:
"by assigning a unique ID which remains on their PC, and can detect multiple accounts"
To me this sounds like a cookie. If it is in fact a cookie, first, easy on the technology bragging, and secondly, I doubt this is very effective as I suspect the heavy survey takers are the heavy ad clickers and the ones that clear their cookies multiple times per day. If it's not a cookie, it would be interesting to get a bit more information on exactly what is being installed or stored on the users PC.

The article further quotes:
the new measures will help change the atmosphere in the online research market, which has for years been marred by unanswered questions over transparency and quality.
My 2 cents on this matter is that the answer lies in this very statement. For many years panels have been run providing little transparency on methodology and quality.  It is easy to find and join a panel that will send me 2 - 4 surveys per day. This immediately raises questions of quality. While all sample sources have their deficiencies and challenges, these can only be addressed and accounted for if the providers abide by this call to transparency.

The core problem which is being ignored, is that these mega panels are littered with professional respondents. It will be interesting to see whether attempts to remove these will merely result in mega-panels turning into micro panels of non-responsive members.

Many still consider RDD (random digit dialing) as the true form of survey recruitment despite it's own issues. Though this methodology has not translated well to the web - not yet at least.

Pete Comley is quoted summing it up nicely:

 "Pete Comley – of the UK’s Virtual Surveys – who recently compared reliance on panels to reliance on dwindling fossil fuel reserves, welcomed the quality initiatives but said industry-wide standards, rather than separate initiatives, are needed."


. . . and I'm hoping someone can provide further insight into these issues . . . ?

10 comments:

PeteC said...

Digital Fingerprints is not based on a cookie, as far I can see. Take a look at the white papers on PeanutLabs and MarketTools websites. I think they are a major step forward to preventing people defrauding us potentially. Big ifs though:

1. It really does not need to be an Industry/worldwide initiative and not one by these two companies - I hear on the grapevine it is now three companies - not two now - making it worse.
2. If we remove the professional respondents, we remove the 1/4% of people who do 30% of all our interviews. That is frightening and will beckon the day of 'peak panel' even quicker. Moreover whoever said that all thsoe who like filling out our surveys are bad.

Pete Comley
Virtual Surveys

Paul Neto said...

Hi Pete and thanks for your comment.

The digital fingerprint methodology implemented by Peanut Labs in their Optimus technology is quite interesting. Though it is still unclear whether the DRD system is along the same lines.

It definitely is an exciting time for the industry. There are some interesting innovations and new players in the game to help shake things up and help address some of the issues.

btw - the papers on the PeanutLabs site are excellent. For anyone's interest, here is a link:

http://www.peanutlabs.com/peanutlabs/files/documents/Optimus_Eliminating_Suspect_Respondents.pdf

http://www.peanutlabs.com/peanutlabs/files/documents/Optimus_MediaScreen_Study.pdf

PeteC said...

The MarketTools does use Digital finger printing based on your PC details (and not cookies). See: http://www.markettools.com/resources/files/WP_TrueSample.pdf

Pete

Olivier said...

Paul,

It's not a cookie and it is not stored on the computer. None of the companies having developed or licensed a similar technology want to give details, because they want to protect their innovation and business model as long as possible... I myself have been talking about de-duplication for some time now, without being able to share how we do it exactly at OTX, in order to stay ahead of competition in that field.

Generally speaking, the longer a technology like this stays in stealth mode, the longer it stays efficient because once it becomes mainstream, someone finds a way to abuse the system. Security is a series of hurdles and needs to evolve with hackers' capabilities.

I cannot speak for any of the digital fingerprinting technologies mentioned here, but basically digital fingerprinting is only new for market researchers. It has been used by the online banking industry for authentication purposes for years, and is based on device-specific information sniffed by the browser. Note that Peanut Labs has developped their technology internally, when MarketTools has licensed 41st Parameter's PC Print, and Greenfield has licensed RelevantView's RelevantID.

Olivier

Paul Neto said...

Thanks Olivier,

The digital fingerprint technology sure is an exciting move forward. It is also interesting that there are a few different companies implementing their own algorithms. I've even heard of using biometrics i.e. keystroke timing/patterns to identify a user.

I guess part of my beef was was the article and how it portrayed it as "a unique ID that remains on their computer".

And I totally agree, there needs to be some element of stealth and proprietary technology to ward off hackers, though I often wonder how tech-savvy are these people that want to break into everyone's surveys?

Thanks again!

Olivier said...

Gotcha. Yes, you are totally right, a digital fingerprint cannot stay on the computer... My point about hacking is that, if the technology becomes mainstream and also adopted by ad networks for other purposes than research, you can be sure that sooner or later there will be a lot of noise about privacy and some sort of "fingerprint blockers" will pop up all over the place - And then we will have to come up with something new... Hence the need to stay in stealth mode as long as possible.

David said...

>>It's not a cookie and it is not stored on the computer.

Hi Oliver, technically that isn’t correct.
There are other ways to store data beyond cookies these days.

Many of these companies are using a combination of watermarking and fingerprinting technologies in combination with each other.

Some, including RelevantID appear to be using a bit of survey session behavioral tracking to improve their quality measures (how long people spend per page, whether they straight line, etc)….. instead of relying on self-reported data from each research company.

A few groups, such as your company and mine already have been doing much of this for many many years.

OpenID potentially could offer another possible method via a third party database comparison, though it isn’t quite clear yet what the final offering will be.

Krgrds,
David

Dan Coates said...

Staying a step ahead of those attempting to game the system requires significant resources.

We've just announced a partnership with RelevantView to offer the RelevantID technology to users of the Globalpark platform.

The rationale behind our 'partnering' our way into this functionality strikes at the heart of this discussion thread ... engaging in a game of cat and mouse is so technically demanding and time-consuming that we consciously decided to partner with RelevantView (who also powers Greenfields DRD platform).

We looked at others and felt that the RelevantView was greater than or equal to alternatives, but didn't have the inherent conflict of interest that ensue when a panel company tries to sell its technology to other panel companies.

Paul Neto said...

Great discussion and comments. Thanks all.

Dan - as you stated, staying ahead of the game requires significant resources.

While great progress is being made, I still am left wondering whether the source of the problem is being addressed. I suspect that a lot of these 'professional' survey takers exist largely due to email panels. Inherently there is something wrong with how email panels have evolved and this itself needs to be challenged.

The answer is still not clearly apparent though many, including ourselves, are trying our best to address it.

Email panels + incentives + an expectation for high response rates leads to an environment where these multiple survey takers flourish.

If we challenge incentives and response rates, are we left with anything much better or any step further.

The internet has made it relatively easy and inexpensive to reach 1,000's in a very short amount of time. This was never the case with mall or phone surveys. Research has been relatively spoiled over the past decade and we are just starting to pay for it.

Or is it the case that the problem really isn't all that bad?

Anonymous said...

>>Email panels + incentives + an expectation for high response rates leads to an environment where these multiple survey takers flourish.


I don't think so.

Personally, I believe the problem is much larger for affiliate based networks such as Greenfield's router/river approach.

Each referring site receives a commission every time a completed survey takes place. A respondent can take up to N number of projects one after another.

The motivation of the affiliate is to *push* as many potential respondents to their front door every day.
They are leaving it up to the sample companies to decide if the person is legit.

The motivation of the respondents is to take as many projects as possible. In some cases, the quota need is spelled out.. "We're looking for people ages XX through YY for a chance to win $10".


Bottom line, the problem is worse than e-mail because it is easier to game it from a respondent and affiliate standpoint.