Big Data for Global Development

The U.N. High-Level Panel report on the post-2015 agenda has a section entitled “Wanted: A data revolution” that has caught the attention of development practitioners.

This “data revolution” requires greater collaboration between the public and private sector to share information and improve evidence-based policy-making and planning. The HLP emphasizes that this cannot be achieved through business as usual. New partners, new approaches and even new sources of data are needed:

“Better data and statistics will help governments track progress and make sure their decisions are evidence-based; they can also strengthen accountability. This is not just about governments. International agencies, CSOs and the private sector should be involved. A true data revolution would draw on existing and new sources of data to fully integrate statistics into decision making, promote open access to, and use of, data and ensure increased support for statistical systems.”

The age of ‘Big data’

This call to action is music to the ears of those who have been at the forefront of a transformative trend: ”Big Data for Development.”

Innovations in technology and greater affordability of digital devices worldwide have ushered in an age of “Big Data,” an umbrella term for the explosion in the quantity and diversity of high-frequency digital data, and the innovations which allow for its storage and analysis. Digital data is being produced, in real time, at an unprecedented rate, even across the developing world, just as we all go about our daily lives. In 2000, only a quarter of the world’s stored information was digital. The rest was preserved on paper and other analog media like film. Today, 98 percent of all stored data is in digital form.

The digital divide is also closing faster than anticipated. There are over 6 billion active mobile phones in use worldwide, and 5 billion of those are in developing countries. Services such as the wildly popular m-Pesa allows people to make financial transactions over their mobiles, to the point that by 2011, m-Pesa’s local transaction volume in Kenya alone exceeded transactions made through Western Union globally. And it may be surprising to know that Indonesians in Jakarta tweet more than residents of any other city in the world. All this global usage of mobile phones, social media and internet searches leaves a digital trail of ”data exhaust” which, when anonymized, aggregated and analyzed, can reveal significant new insights and trends about human behavior.

This opens up a world of new opportunities for development: from early warning, to real-time trend analysis, to measurement, monitoring and evaluation.

Global Pulse: the UN’s Big Data innovation lab

It’s well-known that private sector companies already analyze so-called “Big Data” to understand their customers in real time, and improve their services and products. The United Nations must do the same for its constituents: People around the world who are losing jobs, getting sick, or can’t afford food or housing.

The Global Pulse initiative was launched by U.N. secretary-general Ban Ki-moon in late 2009 and serves as an innovation lab that is developing the capacity to responsibly use new sources of data and real-time analytics.

Global Pulse’s vision involves raising awareness of the opportunities presented by Big Data, building bridges between data science expertise, data providers and development sector practitioners, and helping to catalyze adoption of the use of new digital data sources and analytics into institutional practice. Through collaborative R&D projects with a wide range of partners, we test the usefulness of new sources of data like social media, or mobile phone data to solve a specific challenge, or serve as a proxy indicator for development progress.

What does this mean for the development community?

So what does Big Data mean for the NGO staffer in Pretoria or health worker in Guyana? We don’t know yet. There is no playbook, no ready to roll-out best practices or guidelines. The only way to test the utility of new data sources for development is through learning from other sectors, building communities of practice and a common understanding of the concepts, and sharing what we learn.

That is why the backbone of Global Pulse’s strategy is our network of “pulse labs.” Our labs in New York, Jakarta and soon in Kampala provide a space where partners from government, private sector and across the U.N. system can brainstorm practical challenges, design exploratory research projects, prototype applications and share findings.

For example, analysis of social media has been used by our researchers to offer insights about leading and lagging indicators of important socio-economic events like unemployment. We’re working with USAID to conduct a feasibility study about whether new digital data sources can help identify barriers to financial inclusion in Kenya. We’re building on a project initiated by UNICEF which looked at huge numbers of blog posts, online comments and tweets to gain insights into the attitude of parents towards immunization of their children against diseases like diphtheria, tetanus and measles. Attitudinal insights can be used to improve communication with parents to allay their fears about inoculating their children.

And in Indonesia, we used social media analytics to understand the public perception about the impacts of food and fuel price changes. We saw that when social media trends are combined with official statistics, there is an opportunity to get a real-time approximation of official indicators that are already used for social protection planning.

A practical guide to Big Data for Development

As a first step in capacity-building, we’ve just published a simple guide that offers an overview of Big Data in the development context. ”Big data for development: A primer” introduces basic terms, summarizes case studies, lists the key challenges, and the opportunities.

It is important to acknowledge some of the challenges that lie ahead — including privacy issues, analytical challenges and the digital divide — in order to find ways to address them. In a world where data is ever more important in making decisions, human intuition, common sense and understanding are crucial for deep analysis that can be used for public good. Ultimately, Big Data is more than a buzzword. It offers a new tool in our development toolkit, and must be approached with a nuanced appreciation of its power, and also of its limitations.

There’s a long way to go, and a lot to learn, but we hope our new overview will help as a guide to get started.