CIO, Do I Need to Implement BigData?

INTRODUCTION

As digitization has become an integral part of everyday life, data collection has resulted in the accumulation of huge amounts of data that can be used in various beneficial application domains. Effective analysis and utilization of big data is a key factor for success in many business and service domains. This paper discusses and compares different definitions of the big data and explores the opportunities, challenges and benefits of incorporating big data applications for your business. In addition it attempts to identify the requirements that support the implementation of big data applications . The review reveals that several opportunities are available for utilizing big data ; however, there are still many issues and challenges to be addressed to achieve better utilization of this technology.

Data is being generated from multiple sources resulting in the formation of what is currently known as big data. Data sources are around us everywhere, smart phones, computers, environmental sensors, cameras, GPS (Geographical Positioning Systems), and even people. Various applications like social media sites, digital pictures and videos, commercial transactions, advertising applications, games and many more helped accelerate data generation in the past few years.

In this Article I will summaries and shorten the concepts and i will focus in the practical side of big data uses, i will not go deeply in the theoretical and scientific part of big data.

 Characteristics and features of big data

1. Volume: refers to the size of data that has been created from all the sources.
2. Velocity: refers to the speed at which data is generated, stored, analyzed and processed. An emphasis is being put recently on supporting real-time big data analysis.
3. Variety: refers to the different types of data being generated. It is common now that most data is unstructured and cannot be easily categorized or tabulated.
4. Variability: refers to how the structure and meaning of data constantly changes especially when dealing with data generated from natural language analysis for example.
5. Value: refers to the possible advantage big data can offer a business based on good big data collection, management and analysis.

Benefits and opportunities

1. Efficient resource utilization: With many resources becoming either scarce or very expensive, it is important to integrate solutions to have better and more controlled utilization of these resources. Starting with technological systems such as Enterprise resource planning (ERP) and Geographic Information System (GIS) will be useful. With monitoring systems at work, it will be easier to spot waste points and better distribute resources while controlling costs, and reducing energy and natural resources consumption.
2. Higher levels of transparency and openness: The need for better management and control of the different applications, will drive the interoperability and openness to higher levels. Data and resource sharing will be the norm. In addition, this will increase information transparency for everyone involved. This will encourage collaboration and communication between entities and creating more services and applications. One example is the US government that collected and released a wide range of data, publications, and content in the name of transparency and openness. These offered the citizens and the government entities the chance to exchange and use the data effectively.
3. Product Development: Companies like Netflix and Procter & Gamble use big data to anticipate customer demand. They build predictive models for new products and services by classifying key attributes of past and current products or services and modeling the relationship between those attributes and the commercial success of the offerings. In addition, P&G uses data and analytics from focus groups, social media, test markets, and early store rollouts to plan, produce, and launch new products.
4. Predictive Maintenance: Factors that can predict mechanical failures may be deeply buried in structured data, such as the year, make, and model of equipment, as well as in unstructured data that covers millions of log entries, sensor data, error messages, and engine temperature. By analyzing these indications of potential issues before the problems happen, organizations can deploy maintenance more cost effectively and maximize parts and equipment uptime.
5. Customer Experience: The race for customers is on. A clearer view of customer experience is more possible now than ever before. Big data enables you to gather data from social media, web visits, call logs, and other sources to improve the interaction experience and maximize the value delivered. Start delivering personalized offers, reduce customer churn, and handle issues proactively.
6. Fraud and Compliance: When it comes to security, it’s not just a few rogue hackers—you’re up against entire expert teams. Security landscapes and compliance requirements are constantly evolving. Big data helps you identify patterns in data that indicate fraud and aggregate large volumes of information to make regulatory reporting much faster.
7. Machine Learning: Machine learning is a hot topic right now. And data—specifically big data—is one of the reasons why. We are now able to teach machines instead of program them. The availability of big data to train machine learning models makes that possible.
8. Operational Efficiency: Operational efficiency may not always make the news, but it’s an area in which big data is having the most impact. With big data, you can analyze and assess production, customer feedback and returns, and other factors to reduce outages and anticipate future demands. Big data can also be used to improve decision-making in line with current market demand.
9. Drive Innovation: Big data can help you innovate by studying interdependencies among humans, institutions, entities, and process and then determining new ways to use those insights. Use data insights to improve decisions about financial and planning considerations. Examine trends and what customers want to deliver new products and services. Implement dynamic pricing. There are endless possibilities.

How to Start ?

Big data gives you new insights that open up new opportunities and business models. Getting started involves three key actions:

1. Integrate

Big data brings together data from many disparate sources and applications. Traditional data integration mechanisms, such as ETL (extract, transform, and load) generally aren’t up to the task. It requires new strategies and technologies to analyze big data sets at terabyte, or even petabyte, scale.

During integration, you need to bring in the data, process it, and make sure it’s formatted and available in a form that your business analysts can get started with.

2. Manage

Big data requires storage. Your storage solution can be in the cloud, on premises, or both. You can store your data in any form you want and bring your desired processing requirements and necessary process engines to those data sets on an on-demand basis. Many people choose their storage solution according to where their data is currently residing. The cloud is gradually gaining popularity because it supports your current compute requirements and enables you to spin up resources as needed.

3. Analyze

Your investment in big data pays off when you analyze and act on your data. Get new clarity with a visual analysis of your varied data sets. Explore the data further to make new discoveries. Share your findings with others. Build data models with machine learning and artificial intelligence. Put your data to work.

Guidelines for building a successful big data implementation

Align Big Data with Specific Business Goals

More extensive data sets enable you to make new discoveries. To that end, it is important to base new investments in skills, organization, or infrastructure with a strong business-driven context to guarantee ongoing project investments and funding. To determine if you are on the right track, ask how big data supports and enables your top business and IT priorities. Examples include understanding how to filter web logs to understand e-commerce behavior, deriving sentiment from social media and customer support interactions, and understanding statistical correlation methods and their relevance for customer, product, manufacturing, and engineering data.

Ease Skills Shortage with Standards and Governance

One of the biggest obstacles to benefiting from your investment in big data is a skills shortage. You can mitigate this risk by ensuring that big data technologies, considerations, and decisions are added to your IT governance program. Standardizing your approach will allow you to manage costs and leverage resources. Organizations implementing big data solutions and strategies should assess their skill requirements early and often and should proactively identify any potential skill gaps. These can be addressed by training/cross-training existing resources, hiring new resources, and leveraging consulting firms.

Optimize Knowledge Transfer with a Center of Excellence

Use a center of excellence approach to share knowledge, control oversight, and manage project communications. Whether big data is a new or expanding investment, the soft and hard costs can be shared across the enterprise. Leveraging this approach can help increase big data capabilities and overall information architecture maturity in a more structured and systematic way.

Top Payoff Is Aligning Unstructured with Structured Data

It is certainly valuable to analyze big data on its own. But you can bring even greater business insights by connecting and integrating low density big data with the structured data you are already using today.

Whether you are capturing customer, product, equipment, or environmental big data, the goal is to add more relevant data points to your core master and analytical summaries, leading to better conclusions. For example, there is a difference in distinguishing all customer sentiment from that of only your best customers. Which is why many see big data as an integral extension of their existing business intelligence capabilities, data warehousing platform, and information architecture.

Keep in mind that the big data analytical processes and models can be both human- and machine-based. Big data analytical capabilities include statistics, spatial analysis, semantics, interactive discovery, and visualization. Using analytical models, you can correlate different types and sources of data to make associations and meaningful discoveries.

Plan Your Discovery Lab for Performance

Discovering meaning in your data is not always straightforward. Sometimes we don’t even know what we’re looking for. That’s expected. Management and IT needs to support this “lack of direction” or “lack of clear requirement.”

At the same time, it’s important for analysts and data scientists to work closely with the business to understand key business knowledge gaps and requirements. To accommodate the interactive exploration of data and the experimentation of statistical algorithms, you need high-performance work areas. Be sure that sandbox environments have the support they need—and are properly governed.

Align with the Cloud Operating Model

Big data processes and users require access to a broad array of resources for both iterative experimentation and running production jobs. A big data solution includes all data realms including transactions, master data, reference data, and summarized data. Analytical sandboxes should be created on demand. Resource management is critical to ensure control of the entire data flow including pre- and post-processing, integration, in-database summarization, and analytical modeling. A well-planned private and public cloud provisioning and security strategy plays an integral role in supporting these changing requirements.

Top Big Data Companies and solutions

• Apache Hadoop
• Prolifics
• Cloudera - Hortonworks
• Clairvoyant
• ScienceSoft
• Xplenty
• IBM
• HP Enterprise
• Teradata
• Oracle
• SAP
• EMC
• Amazon
• Microsoft
• Google
• VMware
• Splunk
• Alteryx
• Cogito

How Much it will cost me?

To answer this question, you need start with the basics and your strategy of how you are planning to implement? most of big data tools are open source and free of use, but this is not applicable for all cases. of-course you need to start with the ROI exercise and consider the following in terms of budgeting:

• Software licenses requirements and cost
• Consulting services requirements and cost
• Infrastructure requirements (cloud/on-promise) and cost

Thanks And Regards

Radwan Ayoub