Monday, December 10, 2012

Where Market Research and Big Data Intersect

Market research, or at least survey based market research, is often referred to as Little data. Big data is the hot topic these days and is slowly infiltrating all industries. The discussion today is around the meeting of Little data and Big data and how the two complement each other. The definition of big data varies greatly depending who you ask.

Big data as I describe it, is a data source that meets some of these criteria:
  1. The quantity of data is great. Managing, accessing, and processing is a challenge
  2. The format is often unstructured, or at least seems to be unstructured
  3. Data source is very complex
  4. The user just isn't very sure what to make of it 
Little data, survey data, or market research in general is most efficient at adding a human interpretation to data. Some call this the 'why' or the 'insights'. You may wonder where do these two data concepts intersect.

A powerful component of Big data is machine learning. This field of expertise is essential in helping blend the two concepts of Big and Little data. For any of those interested in this field, I strongly recommend spending a few evenings or weeks listening to Yaser S. Abu-Mostafa's lectures on ML available through iTunesU.

As explained by Yaser, machine learning is a candidate as a technique for any problem where:
  1. Some pattern exists, or we believe one to exist
  2. We cannot easily describe this pattern mathematically
  3. Data exists
Some of the challenges in MR are related to the changing data sources. Whether it be text analytics, social media, web metrics, or other, the vast amount of data being made available is overwhelming. Excel, Word and general databases are no longer sufficient to dealing with the data we must manage.

Machine learning is a technique that I believe MR firms will need to adopt in-house, or through strategic partners to be able to deal with the data challenges they face today. This would surely help assist reducing data to the most essential metrics that market research can apply it's magic to.

My prediction is that machine learning will play an important role to define the intersection between little data and big data.

No comments: