What are the most common pitfalls when imputing missing data?

Pitfall 1: It's often neglected to investigate why data is missing. Understanding if the gaps are random or patterned with other data is crucial. This insight is key, as missing values, once analyzed, can significantly enhance a model's predictive power. Pitfall 2: In time series forecasting, addressing missing values should follow the decomposition of the series into trends and seasonality. Impute the deseasonalized data first; then reintroduce the seasonal component. Imputing on the series' individual elements is typically not recommended.

One thing i find helpful is to first assess the significance of missing data. If you have 100 rows and 50 data missing, spend time to investigate the flow on how these missing data came into place. If you have million rows and 50 missing data just delete them to avoid putting more assumptions and bias into the model.

One of the most overlooked principle in statistics is Occam's razor, find a solution with the least number of assumptions. Data imputation involves adding extra assumption on your data. It gives you the convenience of having more data, but with distorted reality. Missing data is part of reality and any analyst should think on how to deal with data as it is before applying further assumptions. The other problem is how much can I impute? In my humble opinion, 0%. However, if anyone wants to go with an imputation mechanism, then you need to be extremely careful about your assumptions on the missingness and the statistical properties of the method you are applying. Sensitivity analyses are must in such cases.

I always tell this to my students and fellow junior colleagues : Missing data is a feature. It may be missing for a reason. First you need to verify the root cause of missing data points. Second, before even considering imputation, you need to ask whether being missing is a meaningful feature itself. Especially, if the missing values are dominant. To do this, I always consider the N/A itself as a category and perform analysis, sometimes using ML models. For continous features, this may not be possible. However, you can always discretize them and still treat missing as a category.

Missing data may have meaning in and of itself. While the first instinct of a data scientist might be to impute missing values, it may be worth creating a binary variable called `<feature>_missing` in addition to imputation. Better yet, it has been shown that using missing values without imputation on supported gradient boosting packages such as xgboost has better out-of-sample performance vs. imputation.

Using a single imputation method can drastically change the results, so be careful with the method(s) you use. Here are some more methods: 1. Linear Interpolation: For time-series or ordered data, you can estimate missing values by interpolating between neighboring data points. Multiple Imputation: 2. Generate multiple datasets with different imputed values, which can account for uncertainty in the imputation process. Analyze each dataset separately and then combine the results. 3. Domain-Specific Imputation: Use domain knowledge to impute missing values. 4. Manual Imputation: Manually impute missing values if you have a small dataset or if the missing values are crucial and can't be reliably estimated by other methods.

Bias: Imputation methods can introduce bias if not chosen carefully, leading to inaccurate results. Overconfidence: Imputing data can give a false sense of completeness, making researchers overlook the true extent of uncertainty. Loss of Information: Imputed values may not reflect the true nature of the missing data, resulting in a loss of valuable information. Inappropriate Imputation Methods: Selecting the wrong imputation technique for the data type can lead to inaccurate results. Propagation of Errors: Errors in imputed values can propagate through subsequent analyses, compounding the problem. Assumption Violations: Imputation methods often assume data is missing at random, which may not be true in practice.

Single Imputation method can be misleading too.Let’s say we are talking about mean imputation, mean is volatile to the outliers so if we are imputing the missing data that might be not be accurate or close to actual missing data and can fall into outliers too. The best way of imputation or dealing with missing data can be if the data has less missing values and if it is not going to impact your analysis then simply drop such records bit if it is significant then try to sit with domain experts with your team and understand their thoughts why is it missing and what’s its impact would really help you in choosing imputation methods from wide variety of imputation techniques, just like single model cannot be used for all type of problem.

You need to be careful of the side of the data gap. I have tested many different interpolation methods, and machine learning seems to be the best, especially for bigger data gaps. I mean, it gives you an opportunity to investigate the performance of the model. LSTM is one good example.

Imputation is a double edged sword. Like many concepts and methods in statistical inference there is no ready to use cookbook for it. But based on my experience, the rules-of-thumbs and default methods like mean imputing etc do not well apply to real-world use cases. For each context, you would need to come up with your own strategy.

Handling missing data is a critical aspect of data analysis. Common pitfalls include ignoring missing data, using inappropriate imputation methods, overimputing, losing valuable information during imputation, and making incorrect assumptions about the nature of missing data. It's crucial to transparently report how missing data is handled, avoid data leakage, and be cautious when using predicted values for imputation. Sensitivity analysis should be conducted to assess the impact of imputation choices. Avoid imputing missing values with outliers, and always check the distribution of imputed values. By addressing these pitfalls, analysts can ensure the accuracy and reliability of their data analysis.

Most overlooked aspect in imputation is generally the assumption that a missing feature for a data point cannot contain information to increase model accuracy. All idea of imputation whether it is filling with medians or something else is actually to leverage other non-missing features of that instances in modeling. Missing continous variables with 0s may provide an idea to the ML model on the representation of missing values, however it may disrupt the functional approximation or the feature can be naturally 0 as well.

I’d say often people disregard class imbalances when imputing the data. For example, if you have 99% of data from one class and 1% from another it is very dangerous to impute data as the likelihood that it will impute the data from the dominant class. Before imputing you might manually look for more data from the second class and only then impute data.

Filling in the missing parts of a story. Sometimes we assume things to complete the tale, like guessing the hero's actions. But what if our guesses about the hero aren't quite right? Similarly, when we fill in missing data, we assume things about the numbers we don't have. It's like saying, "I think this number should be around 10." But what if it's not close to 10? Checking assumptions is like double-checking our story guesses with clues. We use special tests and graphs to see if our assumptions match the real facts in the data. It's like saying, "Let's make sure our story about the hero matches the evidence in the book!" This way, we get a more accurate and believable tale (or data).

It is crucial to ponder - "How is Imputing without considering the impact on downstream analyses"? The imputed data will be used in downstream analyses, such as regression analysis, machine learning, and hypothesis testing. One must consider how the imputation method will impact the results of these downstream analyses. For example, some imputation methods can introduce bias into the results of downstream analyses. "Imputing without considering the computational cost is also worth giving a thought." Some imputation methods can be computationally expensive, especially for large datasets. It is important to consider the computational cost of the imputation method before using it.

It depends on the sanple size of the data set. It can go both ways 1. Generates strong outcome based on the similarity of mean statistical analysis performed. 2. It can also skew the data if imputed valuses are outliers.

Before working with actual data, it is highly recommended to test the method on a complete sample of the data by introducing randomly generated missing values (NAs). This allows for a comparison between the imputed values and the original values. By doing so, we gain insights into the nature of missing values and can select an imputation method that closely resembles the randomly created NAs

Ignoring the Impact of Imputation on Analysis: Imputation can introduce variability and affect the results of subsequent analyses. It's essential to account for the uncertainty introduced by imputation by using appropriate methods such as multiple imputation or sensitivity analyses

Best way is to design everything as a composable, modular pipeline. Even certain components of scikit-learn or your very own procedures do not conform to scikit-learn pipeline, you can always override scikit-learn classes and properly modify them to work as an end to end pipeline. This way you can inject imputations in pipeline and optimize the choices and parameters in an end to end cross validation.

underestimating or overestimating some values of a given treatment, especially if the number of observations per treatment is small.

In my experience, I like to work as if I would not be able to remember any longer what I did. (I learned that perspective from an older colleague.) Therefore, document precisely your assumptions and how you have tested them, and save your raw data, so that others can reevaluate your work anytime.

It may be a good idea to run statistical hypothesis testing to assess values of missing and non missing values, and present the results in a structured, illustrative graphs.

Fixing missing data is not easy. It involves making choices that can affect your results. So, it's important to share how you did it, like the percentage of missing data, how you filled it in, and how you decided what was the best way. This helps others trust your results and try the same methods.

Not Documenting the Imputation Process: Proper documentation of the imputation process is crucial for transparency and reproducibility. It allows others to understand and validate the imputation methods used and the impact on subsequent analyses.

An additional pitfall to consider is imputing data before splitting it into (train/dev/test) sets, this is a problem whether we are using statistics based or model based imputation. The problem is that imputing training data by using information about the dev/test data leads to data leakage: the training data knows something about the test data that it is not supposed to know, which makes the test results represent inaccurate estimation of the model performance, leading to failure in production.

One of the most common pitfall is not performing the EDA and root cause analysis to understand why the data is missing. Data could be missing for various reasons 1. Missing data from survey results: this is the most common type of missing data for survey’s where people decline to answer certain questions. 2. Calculation problem: sometimes computing some numerical features may lead into missing values like division by zero. 3. Non random missing data: it could also happen that the missing data inherently shows presence of certain category. In that case we could just replace them with “NA”. To conclude, its very important to understand why the data is missing to be able to handle the imputation correctly.

The common pitfalls of imputing missing data in brief pointers: 1. Ignoring missing data. 2. Complete case analysis (deleting rows). 3. Using default values. 4. Over-imputation (guessing too much). 5. Not exploring missing data patterns. 6. Not assessing imputation impact. 7. Not documenting the process.

Always consider using missing features as a covariate itself by converting context in to a full categorical one, if you think the ratio of missing cases are considerable and use case necessitates. Then, after building the statistical model you can interpret the influence and further take action accordingly.

For data imputation there could be a lot of cases that you could make mistakes. The most important thing before data imputations is to understand the data and see if it’s necessary to do data imputation or not. There’s always this idea that you better fill the null values with something but sometimes the null values could give you lot information about the data that you have, so before doing such cleanings try to understand the nature of the dataset.