How do you compare and contrast different models using validation metrics?

When you build statistical models, you want to make sure they perform well on new data, not just on the data you used to train them. That's why you need validation metrics, which are numerical measures of how well your models fit the data and generalize to unseen situations. In this article, you'll learn how to compare different models using validation metrics, and how to choose the best one for your purpose.

1 Splitting the data

The first step in model validation is to split your data into two or more subsets: a training set, a validation set, and optionally, a test set. The training set is used to fit the models, the validation set is used to compare and tune the models, and the test set is used to evaluate the final model on a completely independent data. You can split the data randomly, or use techniques like cross-validation or bootstrap to create multiple splits and reduce variability.

Add your perspective

Michael W.

AI | LinkedIn Top Voice - AI & Deep Learning | Speaker | Board Director | Data Scientist | Investment Strategy | IEEE Reviewer | Medium Top Writer
Report contribution
This is not always the case, Some models especially within the finance domain and scientific research are explanatory models rather than predictive models. In these cases you don’t need to apply test train validations splits

Like
Sk Raju

Data Scientist - Aditya Birla Capital | IIT Kanpur
Report contribution
In addition to that, when partitioning data for model validation, it's paramount to consider several factors to ensure a representative and unbiased distribution between training and validation sets. Addressing the potential challenge of imbalanced target variables, it is advisable to implement stratified sampling. In the context of time series analysis, a chronological split is recommended to mitigate the risk of future data leakage into the training set. This strategic approach aims to enhance the generalizability of the model, promoting its effectiveness in real-world scenarios.

Like

2 Choosing the metrics

The next step is to choose the validation metrics that are relevant for your problem and your models. There are many types of validation metrics, such as accuracy, precision, recall, F1-score, R-squared, mean squared error, log-likelihood, AIC, BIC, and so on. The choice of metrics depends on the type of data, the type of models, and the objectives of the analysis. For example, if you are doing classification, you might use accuracy or F1-score, while if you are doing regression, you might use R-squared or mean squared error.

Add your perspective

Sk Raju

Data Scientist - Aditya Birla Capital | IIT Kanpur
Report contribution
The selected metrics should yield stable and consistent results, ensuring that minor fluctuations in the data or model do not result in substantial changes to the evaluation metrics.

Like
İlgüsu Öksüz

PhD Student | Researcher | Developmental Psychologist
Report contribution
As a latent profile/class modeler, we use numerous metrics to identify the most accurate and optimal profile number for the given data. The latent model analysis aims to identify subgroups among a heterogeneous population. Consequently, based on these metrics, the researcher evaluates and compares the prior model by increasing the number of profiles. Later, several model-fit indices are used to acquire the best model fit. For LCA and LPA, Log-likelihood (LL), Akaike information criteria (AIC), Bayesian information criteria (BIC), Lo-Mendell-Rubin adjusted likelihood ratio test (LMRT), and Entropy was the goodness-of-fit indices considered during the best model fit estimation.

Like
Michael W.

AI | LinkedIn Top Voice - AI & Deep Learning | Speaker | Board Director | Data Scientist | Investment Strategy | IEEE Reviewer | Medium Top Writer
Report contribution
It’s also depends on the attributes of the data, for example accuracy shouldn’t be for classifying data that displays class imbalance

Like

3 Comparing the models

Once you have chosen the validation metrics, you can compare the models by applying them to the validation set and calculating the values of the metrics for each model. You can then rank the models according to the metrics, or use graphical tools like ROC curves, precision-recall curves, or residual plots to visualize the trade-offs between the models. If you are using accuracy as a metric, you can compare the models by looking at the percentage of correct predictions they make on the validation set.

Add your perspective

İlgüsu Öksüz

PhD Student | Researcher | Developmental Psychologist
Report contribution
When it comes to comparing the models based on the previously mentioned model fit indices, the most important metric is entropy. entropy measures the uncertainty of aggregated classification, determining the accurate classification of individuals into their profiles. Having said that, values closer to 1 imply a sharper distinction between profiles (Ahlborg et al., 2019). Additionally, if the p-value is more remarkable than .05, the lower profile model fits better. Therefore, for the most contributional results, one might need a p<0.001.

Like

4 Contrasting the models

Comparing the models by validation metrics is not enough to choose the best one. You also need to contrast the models by considering factors such as complexity, interpretability, robustness, scalability, and domain knowledge. For example, if you have two models with similar validation metrics, but one is simpler and easier to explain than the other, you might prefer the simpler one. Or, if you have a model that performs well on the validation set, but is sensitive to outliers or assumptions, you might want to test it on the test set or use other methods to check its validity.

Add your perspective

Michael W.

AI | LinkedIn Top Voice - AI & Deep Learning | Speaker | Board Director | Data Scientist | Investment Strategy | IEEE Reviewer | Medium Top Writer
Report contribution
Other factors to consider include the size of the training data set, some models require large data sets to train properly whilst other May perform better on smaller data sets. Also if you know before hand that the relationship between the input and outputs are non linear this should also be considered in modle selection

Like

5 Selecting the best model

The final step is to select the best model based on the validation metrics and the other factors. There is no universal rule for selecting the best model, as it depends on your specific situation and goals. However, some general principles are to avoid overfitting, which means that the model fits the training data too well but does not generalize well to new data, and to avoid underfitting, which means that the model does not fit the data well enough and misses important patterns. You can also use techniques like regularization, feature selection, or ensemble methods to improve your models and reduce overfitting or underfitting.

Add your perspective

Michael W.

AI | LinkedIn Top Voice - AI & Deep Learning | Speaker | Board Director | Data Scientist | Investment Strategy | IEEE Reviewer | Medium Top Writer
Report contribution
It’s incorrect to say avoid over/underfitting as they are a function of the process of model building rather than a proactive choice in model design. It’s more accurate to say reduce and mitigate under/overfitting as mentioned in the last sentence

Like
İlgüsu Öksüz

PhD Student | Researcher | Developmental Psychologist
Report contribution
For selecting the best model, the model-fit indices or metrics matter, and the justification behind the best model also matters. For instance, in LPA analysis, if you yielded the best model fit as a 5-profile model with one profile having only 0.4% of the population, then it might be reasonable to consider a 4-profile model if the model-fit indices' scores are closer. So, it is not only related to numbers but also to your hypothesis and previous literature findings within the field.

Like

6 Applying the best model

Once you have selected the best model, you can apply it to the test set, if you have one, to get an unbiased estimate of its performance on new data. You can also use the best model to make predictions, draw conclusions, or provide recommendations based on your data. However, you should always remember that validation metrics are not perfect, and that your model might still have limitations or uncertainties. Therefore, you should always monitor your model, update it with new data, and validate it again as needed.

Add your perspective

Michael W.

AI | LinkedIn Top Voice - AI & Deep Learning | Speaker | Board Director | Data Scientist | Investment Strategy | IEEE Reviewer | Medium Top Writer
Report contribution
In addition, monitoring should include monitoring generalisation error (prior to validating/training again) as well as data drift

Like

7 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?

Add your perspective

How do you compare and contrast different models using validation metrics?

1

2

3

4

5

6

7

1 Splitting the data

2 Choosing the metrics

3 Comparing the models

4 Contrasting the models

5 Selecting the best model

6 Applying the best model

7 Here’s what else to consider

Statistical Modeling

Rate this article

Thanks for your feedback

More articles on Statistical Modeling

More relevant reading