What the History of Oil can Teach us about the Future of Data

If data is the new oil, then what can we learn about data from oil? Oil has been taken from the ground for thousands of years.  I’m going to focus on the modern history of oil starting in the nineteenth century to power the industrial world as we know it.  To help here, oil’s modern history can be thought of in three phases:

Early Industrialization 

Through the 19th century, steam engines and whale oil and paraffin (an oil derivative) were already in wide use.  New ways to refine oil resulted in a better way support to these activities.  Prospectors then actively sought big reserves along with innovative ways to extract and refine the oil.  As big reserves were found, the cycle accelerated. 

We are in this stage with data (well beyond the beginning, but still there). The pre-refining stage, when oil was raw and dirty and the price not commoditized although its abundance known.  Companies have clear use cases that they know can be supported by data.  However, the challenges are those of an emerging market: 

• Finding clean data to support the highest-value use cases in a stable, consistent format is hard work (and at times not yet possible).
• Companies refine data themselves for their custom use cases.  They use data to try and gain competitive advantage, which they can do because data is not yet a commodity. 
• Where data ‘refiners’ exist in the marketplace, they are generally use-case specific.
• There is an ecosystem of tools that support the refining process that are used by both operating companies and data suppliers. 

In most (but not all) environments data at scale isn’t yet commoditized as in input.

Stable and Steady Growth

Think of this as mid-20th century oil.  Supply is plentiful and stable. Legacy use cases (transportation, power, etc.) are using oil in quantity: Big Oil.  New use cases emerge that hadn’t been imagined before plentiful oil was discovered (Plastics!).  We are seeing some of this now in data, and for some use-cases we’ve comfortably transitioned into this phase.  Digital personalization is like early plastics: the market knows it is possible, is doing it well but it is still evolving fast. 

Supply Saturation and Shock Cycles

Yes, data can reach this phase.  

Saturation:  Analytic models become ‘tired’ and ‘oudated.’  There will be situations like these where seemingly plentiful data chases few worthwhile use cases.  Data science in these scenarios will be pushing at boundaries of low-margin profit.

Shock:  Data isn’t like a billboard that can be viewed by one or a thousand people with no degradation.  As use cases for the same dataset expand, technical solutions are employed to make that expansion possible.  Code needs to be written to avoid database table locks, data gets replicated to multiple locations, bandwidth and compression are needed.  Single physical instances of data will have use cases compete for its use, and derivative instances will serve lower priority use cases.  The debate over net-neutrality is an early foreshadowing of the governance issues this stage will demand.

Conclusion 

Oil has a fourth phase since it is a naturally occurring scarce resource, call it ‘depletion’, during which it will need to be replaced by alternatives. This isn’t going to happen with data, but when the Saturation/Shock cycle is reached entrepreneurs will need to look beyond data as a driver of profitability.  Maybe the best already do.