From Big Data to Small & Wide Data: what you need to know

A shift away from Big Data to Small and Wide Data is unlocking new opportunities for innovation and data-driven decision-making.

With the emergence of AI, data fabric and composable analytics solutions, organisations are increasingly able to examine a combination of small and large - and structured and unstructured - data. Combined with the correct data strategy, these data sources can help organisations uncover useful insights in small and even micro data tables.

To illustrate, while traditional data sources may provide a column for the colour of an item, an AI-friendly (wide) data source could have multiple columns - or features - that ask "Is it red? Is it yellow? Is it blue?" Each of the additional columns demand special consideration from the database engine to unlock the true value of wide data sources.

Organisations are likely to continue leveraging their access to big, small and wide data sources as a key competitive capability. In fact, Gartner analysts predict that by 2025, 70% of enterprises will move from big data to small and wide data (or data that comes from a variety of sources), thus enabling more context for analytics and intelligent decision-making.

But what's the difference between these data types, and why should organisations care?

Big vs Small vs Wide Data

Big Data has been prevalent since computer software and hardware gained the ability to process huge data sets. At first used by scientists and researchers to conduct meaningful statistical analyses, by the mid-2000s Big Data had become a must-have for every large enterprise. The Economist even famously called data 'the new oil', and companies duly set about generating massive data lakes and mining them for value.

Big Data is great for big-picture analyses and gaining a better view into broader trends. In short, it's a good tool for understanding whether you are looking at a man or a horse.

Small and Wide Data is better at focusing on specific bits of information to gain distinct insights. To use the man or horse example, Small and Wide Data is more about understanding what type of horse you're looking at, what colour the man's eyes are, and why both are in the picture in the first place.

Wide Data specifically ties together disparate data from a wide variety of data sources to understand aspects such as behaviour. For example, wide data analyses can help retailers understand how likely a shopper is to purchase a specific item based on the items in their basket.

Small Data takes on a more individual element. Small Data analyses focus on collecting and understanding smaller data sets sourced from a single organisation. It's basically the opposite of big data, and requires a separate data strategy as small data is not readily gained from big data sets.

 Towards small, wide data use cases

The sheer cost of big data initiatives make a compelling case for greater adoption of wide and small data capabilities. In addition to enormous complexity introduced by vast pools of big data and difficulties with extracting value from big data sets, any strategy depending on big data also requires often scarce and expensive skills. This level of investment is unfortunately beyond the means of most organisations.

In contrast, the more company-specific insights gained from small data sets can more easily be leveraged to improve decision-making, while insights from wide data should be integrated into organisational decision-making to improve the quality of outcomes from such decisions.

Companies exploring the potential of AI to augment decision-making capabilities can find enormous benefit from wide data. The smaller data sets are easier to manage than big data lakes, and thus more likely to remain up-to-date and of immediate relevance. Applying algorithms to augment decision-making using wide data typically produces more accurate and timely insights than relying on large-scale analyses using big data.

To return to the man and horse example, wide data can help companies not only better understand different attributes - hair colour, eye colour, age, etc. - of the man as well as key characteristics (breed, colour, age, size) of the horse, but also draw from other data sources to understand the lineage of the horse, the man's family connections and hobbies, and whether the man rode the horse to where they now are.

These types of hyper-contextual data insights can bring greater clarity to organisational processes and help companies better understand their customers, employees and operating environment.

As the volume of data grows to unmanageable levels, companies will increasingly have to turn to small and wide data to support the business.

Companies wishing to achieve true data-driven decision-making capabilities should start exploring the potential of small and wide data in specific use cases and embark on a process of discovery to unlock the power of data to improve business outcomes.