Big data as I describe it, is a data source that meets some of these criteria:
- The quantity of data is great. Managing, accessing, and processing is a challenge
- The format is often unstructured, or at least seems to be unstructured
- Data source is very complex
- The user just isn't very sure what to make of it
A powerful component of Big data is machine learning. This field of expertise is essential in helping blend the two concepts of Big and Little data. For any of those interested in this field, I strongly recommend spending a few evenings or weeks listening to Yaser S. Abu-Mostafa's lectures on ML available through iTunesU.
As explained by Yaser, machine learning is a candidate as a technique for any problem where:
- Some pattern exists, or we believe one to exist
- We cannot easily describe this pattern mathematically
- Data exists
Machine learning is a technique that I believe MR firms will need to adopt in-house, or through strategic partners to be able to deal with the data challenges they face today. This would surely help assist reducing data to the most essential metrics that market research can apply it's magic to.
My prediction is that machine learning will play an important role to define the intersection between little data and big data.