Turning Fresh Air into Diamonds - rethinking the priorities of consumer insight & market research users

In most areas of economic life, value is seen as a function of scarcity.

This basic tenet explains why, for the most part, unicorns are considered pretty special and why goldfish get flushed down toilets at the end of their natural lives. Things which are abundant are usually deemed worthless even though some, such as fresh air, may have considerable utility.

Historically, the market research and ‘insight’ industry has followed the principles of scarcity economics. 50, 40, even 20 years ago, when it was still difficult - and costly - for businesses to gather the views of their customers or gauge consumer opinion, this made sense.

Now, thanks to the digital revolution and explosion of social media in the last 10 years, it makes less sense.

We are now in a post-scarcity world, where the costs of accessing consumer opinion or insight into their behaviour have been drastically reduced. Even if we disregard all of the behavioural data that organisations collect as a matter of course from their customers, the swathes of attitudinal data that consumers freely - and willingly - share through consumer complaints, social media, and online sentiment represent a fantastic opportunity for consumer insight and market research professionals.

Not only is this data cheaper to collect, it is often better data too, compared to traditional research-provoked data. Because it is naturally occurring, it is more relevant to and reflective of how people live their lives and is more likely to open up new and unexpected issues not anticipated by interview questions or group discussion guides.

However, this post-scarcity landscape also represents a challenge to the market research industry.

The market research industry model has developed almost exclusively around the increasing rationalization of data collection and research management. The quick turnaround of groups, surveys and projects demanded by clients encourages the belief that data is king; commissioning a project on Friday and expecting to learn something meaningful by Monday is symptomatic of an industry that treats analysis and interpretation as expensive and unquantifiable irritants - despite its protestations to the contrary.

As data becomes abundant, the challenge for market research suppliers AND users is how to reorient their priorities, resources and, well, pretty much everything to reflect this new reality. One option is to concentrate on further rationalization: automated data collection, automated analysis, and automated reporting. Forget analysis, save money on data collection. Which is what the largest research companies have done, and will continue to do.

The alternative, for both buyers and suppliers, is to develop the organizational flexibility that allows you to build relationships with specialist suppliers (linguists, semioticians, ethnographers, domain experts, cultural analysts, growth hackers) who have the skills to turn abundant naturally occurring data into compelling nuggets of insight. A larger proportion of research budgets is then spent on analysing data, rather than collecting it.

In this way, we more accurately reflect the reality that the priority of research users lies not in the collection of data, but in the production of insight.