How can you use robust methods to identify outliers and noise in data?

In my experience as a data scientist and educator, I've observed that many budding analysts assume that data follows ideal conditions. However, real-world data often deviates from these ideals. While traditional statistics provide us with invaluable tools, relying solely on them can be misleading in the face of outliers or noise. Robust methods are our safeguards against such imperfections. For instance, when training machine learning models, outliers can lead to overfitting, where a model might perform exceptionally on training data but poorly on unseen data. Thus, understanding and employing robust methods becomes essential not only for statistical accuracy but also for building reliable AI and ML models.

It really depends on your research question. Based on your question, you may find an appropriate approach/method to identify outliers. What I usually do, as a data scientist in Medical field, is using visualization (e.g., box-plot or histogram) to see how data are distributed in different variables. Based on what I see on the plots, I try to pinpoint the outliers, say for example, using MAD to flag the people who are a certain number of MADs away from the median.

The easiest and effective ways can be from following 1. +- 1sd or 1.5 sd (standard deviation) if data is following normal distribution 2. Find IQR (inter quartile range) data outside Q1-1.5×IQR & Q3+1.5×IQR gives good result 3. Plot frequency distribution of data and play with probabilities.

Robust methods in robust statistics literature are the methods which yields the estimates that are less sensitive to outliers i.e., the bad data in the training dataset. Depending on their location, there are three types of outliers in the data: vertical outliers, good leverage points, and bad leverage points. The good leverage points do not harm the regression fit. But it is necessary to employ a mechanism that downweights the bad leverage and vertical outliers data points in linear and non-linear regression problems.

Thomas Dietz, Ph.D. It is important to understand why an outlier is an outlier. Sometimes it is just an error. Sometimes it is the result of an underlying mechanism that is different for one or a few observations than for the rest of the data. And remember that many methods of scanning data, such as looking a univariate distributions, may not detect outliers in multidimensional space. See: Dietz, T., et al. (1987). "Estimation with Cross-National Data: Robust and Nonparametric Methods." American sociological review 52: 380-390. Dietz, T., et al. (1991). "On the Utility of Robust and Resampling Procedures." Rural Sociology 56: 461-474.

In my experience without understanding data type we should not think about robust methods of identifying noise and underlying. Noise of image and tabular data is different. Outlier is always with respect to samples that is available for analysis. For example one event is occurring every 6 years and we data of last 7 years then we can conclude that event is outlier but it is not. Also methods vary from domain, scenario and data type

We offer a course in Robust Methods which you can find in our Eigenvector Recorded Courses section of our website, and of course we support robust methods for PCA and PLS regression in our PLS_Toolbox and Solo software. They are powered by the LIBRA toolbox from Mia Hubert and co-workers.

Robust methods for identifying outliers and noise in data are essential for maintaining the quality of datasets, especially in fields such as data science and machine learning. Isolation Forest is an unsupervised learning algorithm that's particularly efficient for outlier detection. It works by randomly selecting a feature and then randomly selecting a split value between the maximum and minimum values of that feature. The idea is that outliers are fewer in number and will, therefore, be isolated with fewer steps compared to regular observations.

True outliers can be removed from the data or corrected/imputed/set to missing if the data is in error. For instance, if you are taking the average of 12 weights, the largest is 100 lbs and the next largest is 15 lbs, then 100 lbs would be an outlier. A data point being beyond a certain percentile or Z score does not make it an outlier. These values are still part of your distribution. If your data is skewed you could transform it, such as through binning (categorization) or a log transform. In a skewed distribution, the mean and median will be different. You should choose the statistic that best represents your purpose. The mean is the expected value while the median evens out the probability of being above or below.

Robust methods typically do not identify outliers rather their outputs are not affected very much by outliers. A simple example is the median as a measure of central tendency rather than the mean. For data points (1,2,3) the median and mean are both 2 whereas for (1,2,96) the mean is 33 but the median is still 2. So changing the 3 to 96 didn't change the median at all but had a big effect on the mean. It is important to remember that sometimes "outliers" are real and important. If you are analyzing sales data for candy, the week before Halloween probably looks like an outlier but you will want to keep that data in your analysis as candy sellers reliably do make more money then.

In my experience, dealing with outliers and noises, first step is to evaluate the impact of these on the outcomes and define if it is material. If not, one can simply accept it and apply on-going monitoring. If yes, apply the treatments - different methods can be used. It is practical to have some options to compare based on benefit, cost, time before making final decisions. Finally, it is important to have all data available to re-evaluate and assess these options after a period of time.

As I said above, data cleaning and choosing an analytic technique appropriate to your data will solve most issues surrounding so called outliers. True outliers are best handled through data cleaning. Skewed distributions are best handled through transforming variable values.

First, I like to refer the initially identified outlier as a suspect outlier. These should be investigated. Great example: we are looking at monthly gift card sales to ensure we have sufficient supply. We may encounter suspect outliers in June and December. Simple investigation will confirm that graduation and Christmas contribute to the outliers. Next step from this: rather than looking at annual gift card sales, would consider monthly. To evaluate the impact of suspect outliers, run the analysis with the outlier and without the suspect outlier. What impact did this have upon the objective for your analysis?

Unfortunately, there is neither free lunch nor a silver bullet when it comes to identify and process outliers. The choice for methodology depends particularly on the context and the nature of the data at hand. As said, the onthological ambuiguity starts with the definition of what an outlier is, which also depends on the context and the interpretation. After a conclusion is reached for the definition of an anomaly or noise, the processing method is also subjective to nature of the data. However, there exists several commonly used techniques and procedures that each seasoned data scientist is acquinted with. These include proper transformation and normalization, segmentation, etc.

The dealing in robust methods can be done by minimizing a different function of residuals than the squared residuals which is commonly used in weighted least squares estimation. These different functions proposed in the literature that offer robustness are called M-estimators. This is an open research topic.

There's a hard truth that everyone must eventually come to terms with: there is no one-size-fits-all solution. For tabular data it's vital to discern between two primary characteristics: scale and covariance. It's necessary to verify statistical assumptions, normality, and to apply relevant scaling, like converting to a logarithmic scale. While tree-based algorithms like isolation forests are popular, they might not efficiently capture covariance due to their inherent structures which cannot perform extrapolation of multivariate relationships, prompting a careful evaluation of their suitability. Robust methods, like RPCA, that strive to reconstruct latent linear structures could excel with complex numerical datasets if linearity is viable.

It depends on the data type, dim, and the distribution itself. Every day now we have new papers talking about certain distributions with robust estimation methods.

The choice of the best robust method depends on your data's characteristics and the specific goals of your analysis. It often involves a combination of data exploration, research, experimentation, and validation to make an informed decision.

Any programming language is useful to implement the algorithm needed. R is the most powerful tool in this topic since most statisticians include packages to solve this problem according to the model and distribution.

Using python packages as an example, you can use seaborn boxplot, histplot, violinplot, or kdeplot to help identify the shape of the distribution. You can readily use .min .max or quantiles on a column of a pandas dataframe. You can then use if then logic to remove or alter values in a dataframe. One approach is to bin/categorize the data by quantile such that extreme values have the same category. You might also set extreme values to missing or impute them based on your knowledge of the data, a statistical model, or the mean/median/mode of the field.

To implement robust methods with software tools, select a programming language and environment like Python or R. Load your data and choose a specific robust method, such as robust regression or outlier detection. Import relevant libraries or packages, write code for the chosen method, and consider data pre-processing if needed. Evaluate and validate the method, creating visualizations and iteratively fine-tuning it. Document your code and report findings, ensuring clear documentation and communication. Consider specialized libraries, scalability, version control (e.g., Git), and thorough testing. Collaborate effectively and share results to foster collaboration and ensure the robustness and accuracy of your data analysis.

The amount of work you put into this process depends on the amount of time you have and how important the task is. Trimming datasets based on quantile is a lazy solution, but especially in the case of exploratory analysis, it might be an acceptable one. Binning or transforming data is often an acceptable approach when in depth data cleaning and analysis are not practical. Finally, don't be afraid to leave data as it is. If a data point is accurate, it is a legitimate part of your data and conclusions especially when that data point is or likely will be repeated.

The context of the problem at hand is an important factor in deciding what to do with outliers. Healthcare datasets often contain extreme values-especially when considering cost variables. Identifying outliers is a critical step, but simply removing them can be problematic for real-world outcomes evaluations. Setting a threshold or outlier definition is necessary, but these data represent the health care costs or utilization of actual people. An analysis that excludes them may make the math work better, but those excluded high-cost events still happened and need to be accounted for. Using transformations and reclassifying variables can help limit data loss, but in real-world evaluations consider reporting results with and without outliers.

Robust methods in diverse fields ensure reliable results in challenging data or outliers. Identifying outliers and noise involves techniques like robust statistics, visualizations, and specialized algorithms, helping to spot irregular data points. Implementing robust methods with software tools requires choosing a suitable programming language, importing libraries, writing code, and evaluating the method's performance. Documentation and reporting are vital for clear communication. Specialized libraries and testing improve accuracy. Collaboration and sharing facilitate teamwork. Employing these robust methods and techniques ensures more accurate, stable, and reliable data analysis and modeling, especially when data quality is a concern.

To identify outliers and noise in data, use robust methods like the IQR or MAD to detect values significantly different from the median. These methods are less sensitive to extreme values, making them practical for real-world data analysis.