Where Big Data Projects Fail

Over the past 6 months I have seen the number of big data projects go up significantly and most of the companies I work with are planning to increase their Big Data activities even further over the next 12 months. Many of these initiatives come with high expectations but big data projects are far from fool-proof. In fact, I predict that half of all big data projects will fail to deliver against their expectations.

Failure can happen for many reasons, however there are a few glaring dangers that will cause any big data project to crash and burn. Based on my experience working with companies and organizations of all shapes and sizes, I know these errors are all too frequent. One thing they have in common is they are all caused by a lack of adequate planning.

So, in no particular order, here are some of the most common causes of failure in business big data projects that I’ve come across.

Not starting with clear business objectives

It’s easy to get caught up in hype – and Big Data has certainly been hyped. When so many people (including me) are shouting about how earth-shatteringly important it is, and anyone not on board is likely to sink, it isn’t surprising that a lot of people start with the “how” without first considering the “why”.

What people who fall into that trap often failed to appreciate is that analytics in business in about problem solving – and first you need to know what problem you are trying to solve.

I worked with an airline which had thrown itself into a range of Big Data projects with great enthusiasm – cataloguing and collecting information on everything from meal preferences to the impact delays would have on drinks orders. Another client – a retailer – had 258 separate data projects on the go when they called me in. Some were interesting – such as by mining all of their stock and purchase data they had found that a particular bottle of wine sold exceptionally well on a Tuesday, and even more so if it was raining. But so what?  The issue is that shelf-space to pre-assigned and can’t be increased for this brand for just this one day. The only option is to ensure the allocated shelf-space is regularly restocked on Tuesdays. In isolation that insight isn’t going to provide them with huge growth or positive change.

Sometimes you will get lucky and hit on an interesting insight taking this approach, but it’s highly inefficient. In fact it’s a bit like sitting an exam and not bothering to read the question, simply writing out everything you know on the subject and hoping it will include the information the examiner is looking for.

Not making a good business case

A lot of people go into Big Data with a “me too!” attitude. The barriers to entry are constantly dropping, which is a great thing – open source software is becoming more accessible all the time, and there are an ever-growing number of “software-as-a-service” companies which often hugely cut down the need for infrastructure investment. On top of that, people like me are always saying that if you aren’t in, you’re going to get left behind!

Well, that’s all true – but you still need to establish why there is a particular need for your business to allocate time and resources to it. However much data you collect (but particularly if it’s big) you will need to make sure it is kept clean and secure and there will be ongoing costs associated with this that might not be anticipated. In short, you need to know why your business needs to use Big Data before you start doing it. If you don’t – wait until you do.

Management Failure 

I admit, this is something of a catch-all and can affect any kind of business initiative. But in a time and resource-intensive Big Data initiative (skilled data scientists generally expect to be paid at least $100,000 a year) management failure can have disastrous consequences.

Sometimes it’s because those holding the purse strings haven’t taken into account some long-term or ongoing cost associated with the project, or sometimes the senior project managers just can’t talk productively to the data scientist workers in the lab. Sometimes it will be because senior managers don’t trust the algorithms – many got where they are today on gut instinct – and they aren’t going to start letting a computer tell them what to do now.

Poor communication

To say the world of Big Data is made up of egghead scientists and corporate suits with dollar signs in their eyes is to use stereotypes which are offensive to both groups – but it works to illustrate the problem here. Big Data in business is about the interface between the analytical, experimental science that goes on in data labs, and the profit and target chasing sales force and boardroom. They are not always natural bedfellows and things can get lost in translation – with sometimes tragic consequences.

For me, the Space Shuttle Challenger disaster serves as an example – it may have been well before the term Big Data was coined but the analysts at NASA were dealing with very large amounts of information for the time – monitoring sensors equipped throughout the shuttle. Their reports to the higher-ups at mission control went into a great amount of detail and included information that would have shown the significant risks of the shuttle breaking up – if the mission controllers had been able to spot it among the superfluous data. If they had been presented with a document titled “The shuttle is likely to crash because …” things would have ended far more happily.

I feel that lessons have not yet been learnt. Those with responsibility for reporting need to think “who is this data for, and how can I package it to make sure the message gets through?” Analysts with one of my clients – a healthcare company – recently created a report for senior management which was 217 pages long. By replacing much of the text with infographics, we cut it down to 15 pages which still contained all the essential information but presented it far more clearly.

Not having the right skills for the job

Or just as fatally, not having the right skills at the right time. As I’ve explained in the examples here, companies are often fond of starting up data projects “left, right and centre” without thinking enough about how this might impact resources in the future. And skilled data science staff are certainly a very valuable resource. Having started a project with no clearly defined aim, in my experience businesses will often come unstuck when they do come across a valid opportunity for meaningful analysis and find their skilled staff otherwise engaged.

Data science staff are expensive and in extremely limited supply. Companies can benefit here from some out-of-the-box thinking – for example one of my banking client told me that while they have a lot of business analysts, they aren’t trained in Big Data and aren’t really data scientists. As there is a huge crossover in skills between the disciplines, I suggested that offering their existing staff specific Big Data training would almost certainly be cheaper than hiring in a whole new team of specialists. We identified the key skills gaps and developed a customised course to move people from business analyst to big data scientists. We also complemented the course with the many online resources where anyone can learn the necessary fundamentals for free. In addition to the training, the bank looked to universities and colleges which often offer the service of students or academics to provide analytical support to businesses. Today, the bank also sponsors a number of PhD students that are using their business’s own data for their study.

Mismanagement can come from many angles and the UK National Health Service’s fatally botched National Programme for IT is a prime example. The plan to bring all patient medical records into a central database was described as the “biggest IT failure ever seen” and was scrapped after more than £10 billion ($14.9 billion) had been spent.

But what's your view of on why big data projects fail? Please share your thoughts in the comments below.

Thank you for reading my post. This post first appeared in my column for Forbes. Here is a link to the original post: Where Big Data Projects Fail and here are some other recent Forbes articles I have written:

• Meet 7 Brand New Big Data Billionaires
• How Big Data Is Changing Healthcare
• 7 Attitudes That Kill Big Data Projects
• Big Data: Saving 13,000 Lives A Year By Predicting Earthquakes?
• Big Data: The Winning Formula In Sports

Here at LinkedIn I regularly write about management, technology and the mega-trend that is Big Data. If you would like to read my regular posts then please click 'Follow' and feel free to also connect via Twitter, Facebook and The Advanced Performance Institute.

And here are some related LinkedIn posts:

• Big Data Explained in Less Than 2 Minutes - To Absolutely Anyone
• Big Data: 25 Eye-Opening Facts Everyone Should Know
• Big Data: The Predictions For 2015
• Big Data: The Key Vocabulary Everyone Should Understand
• Big Data: The 4 Layers Everyone Must Know
• 10 Awesome Ways Big Data Is Used Today To Change Our World

About : Bernard Marr is a globally recognized expert in big data, analytics and enterprise performance. He helps companies improve decision-making and performance using data. His new book is Big Data: Using Smart Big Data, Analytics and Metrics To Make Better Decisions and Improve Performance'.

Photo: Lester Balajadia / Shutterstock.com