What do you do with missing data in public policy analysis?

1 Why does missing data matter?

Missing data can have serious consequences for public policy analysis, depending on the amount, pattern, and mechanism of the missingness. Missing data can introduce bias, reduce statistical power, increase uncertainty, and limit the scope of the analysis. For example, if a survey of public opinion on a policy issue has a low response rate, it may not reflect the true views of the population. If a longitudinal study of the impact of a policy intervention has a high dropout rate, it may not capture the long-term effects of the policy. If a dataset of public service delivery has missing values due to systematic errors, it may not represent the actual performance of the service.

2 How to identify missing data?

The first step in dealing with missing data is to identify and document them. This involves checking the source and quality of the data, looking for patterns and anomalies, and reporting the extent and distribution of the missingness. There are different ways to visualize and summarize missing data, such as frequency tables, histograms, scatter plots, heat maps, and matrix plots. These tools can help you to see how much data is missing, where it is missing, and whether it is related to other variables or time.

3 How to classify missing data?

The second step in dealing with missing data is to classify them according to their mechanism or reason. This can help you to choose an appropriate method to handle them. There are three main categories of missing data mechanisms: missing completely at random (MCAR), missing at random (MAR), and missing not at random (MNAR). MCAR means that the missingness is unrelated to any variable in the dataset, and therefore does not introduce bias. MAR means that the missingness is related to some observed variables in the dataset, but not to the missing values themselves, and therefore can be accounted for by adjusting for those variables. MNAR means that the missingness is related to the missing values themselves, and therefore cannot be ignored or adjusted for without introducing bias.

4 How to handle missing data?

The third step in dealing with missing data is to choose a method to handle them. There are three main types of methods: deletion, imputation, and modeling. Deletion means that you remove the cases or variables that have missing data from the analysis. This is simple and straightforward, but it can reduce the sample size, lose information, and introduce bias. Imputation means that you replace the missing values with plausible estimates based on the available data. This can preserve the sample size, retain information, and reduce bias, but it can also introduce error, uncertainty, and complexity. Modeling means that you use statistical techniques that account for the missing data mechanism and incorporate the uncertainty of the missing values into the analysis. This can provide valid and reliable results, but it can also require strong assumptions, sophisticated methods, and specialized software.

5 How to choose a method?

The final step in dealing with missing data is to choose a method that suits your data, research question, and analytical goal. There is no one-size-fits-all solution, and different methods may have different advantages and disadvantages. Some of the factors that you should consider when choosing a method are: the type and mechanism of the missing data, the amount and pattern of the missingness, the distribution and relationship of the variables, the type and level of the analysis, and the availability and suitability of the software. You should also compare and evaluate the results of different methods, and report the methods and results transparently and accurately.

6 How to learn more?

Missing data is a complex and nuanced topic that requires careful attention and critical thinking. There are many resources and references that can help you to learn more about missing data, such as books, articles, courses, and webinars. You can also consult experts and peers who have experience and expertise in dealing with missing data, and seek feedback and advice on your analysis. Missing data is not a problem to be avoided, but an opportunity to be explored and understood.

7 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?