What are the most common feature selection methods in Python?

Feature selection is the process of choosing the most relevant and informative variables from a large set of data for a specific machine learning task. It can help improve the performance, interpretability, and generalization of your models, as well as reduce the computational cost and complexity. In this article, you will learn about some of the most common feature selection methods in Python and how to apply them using popular libraries such as scikit-learn, pandas, and numpy.

1 Filter methods

Filter methods are based on the statistical properties of the features, such as correlation, variance, or mutual information. These methods are easy and fast to implement, but may not capture interactions between features and the target variable, nor be optimal for a specific model. In Python, there are several examples of filter methods which can be used. For instance, you can use df.corr() to calculate the correlation matrix of a pandas dataframe and select the features with high correlation with the target variable or low correlation with each other. Additionally, VarianceThreshold() from scikit-learn can be used to remove features with low variance. Lastly, mutual_info_classif() or mutual_info_regression() from scikit-learn can be used to estimate the mutual information between each feature and the target variable.

Add your perspective

Gaurav Gokhale

Data Scientist
Report contribution
In the majority of the scenarios you will see that correlation combined with the contextual analysis will enable you to make informed decisions on the feature selection. You can also leverage the technique of elimination by looking at the low variance features as against the target attribute.

Like
Aditya Vardhan

Helping client to take decision based on Data. Experience in different components of BFSI.
Report contribution
Most of the cases combination of correlation and mutual information will be good enough to select features. Combined with trend.

Like

2 Wrapper methods

Wrapper methods are a type of feature selection technique that selects a subset of features based on the performance of a machine learning algorithm. These methods capture interactions between features and the target variable and can be tailored to a specific model. However, they are computationally expensive and can lead to overfitting. Python offers several wrapper methods, such as RFE() and RFECV() from scikit-learn to perform recursive feature elimination, SelectFromModel() from scikit-learn to select features based on coefficients or feature importances, and SequentialFeatureSelector() from mlxtend to perform forward or backward feature selection. All of these methods start with all features and eliminate the least important ones until reaching a desired number of features or the best cross-validation score.

Add your perspective

Ravi singh

Head of AI Foundations @ Schibsted | Driving Data-Driven Business Growth | Ex HBOMax
Report contribution
Wrapper methods aren't just about selecting features for ML models; they're a deep dive into understanding feature interplay. Sure, they're compute-heavy, but they uniquely tailor features to your specific model, boosting its smarts. Python's scikit-learn offers cool tools like RFE() for recursive feature elimination, and mlxtend's SequentialFeatureSelector() for smart, step-wise selection. Pro tip: mix these with filter methods to balance efficiency and accuracy. They're great for complex datasets, just keep an eye on overfitting. Remember, in feature selection, it's not just what you pick, it's how and why!

Like

3 Embedded methods

Embedded methods are regularization techniques integrated into some machine learning algorithms, such as lasso, ridge, or elastic net for linear models, or random forest or gradient boosting for tree-based models. These methods can shrink or penalize the coefficients of less relevant features, balancing the trade-off between bias and variance while being faster and more robust than wrapper methods. In Python, embedded methods such as Lasso() or LassoCV() from scikit-learn can fit a linear model with L1 regularization, driving some coefficients to zero and eliminating the corresponding features. Meanwhile, Ridge() or RidgeCV() from scikit-learn can fit a linear model with L2 regularization, reducing the magnitude of some coefficients and making them less influential. Additionally, ElasticNet() or ElasticNetCV() from scikit-learn can fit a linear model with a combination of L1 and L2 regularization to balance sparsity and stability of feature selection. Furthermore, RandomForestClassifier() or RandomForestRegressor() , as well as GradientBoostingClassifier() or GradientBoostingRegressor() , from scikit-learn can provide feature importances based on average decrease in impurity or mean squared error across the trees.

Add your perspective

4 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?

Add your perspective

What are the most common feature selection methods in Python?

1

2

3

4

1 Filter methods

2 Wrapper methods

3 Embedded methods

4 Here’s what else to consider

Data Science

Rate this article

Thanks for your feedback

More articles on Data Science

More relevant reading

What are the most common feature selection methods in Python?

1

2

3

4

1 Filter methods

2 Wrapper methods

3 Embedded methods

4 Here’s what else to consider

Data Science

Rate this article

Thanks for your feedback

Explore Other Skills