How do you present uncertain Machine Learning predictions?

Machine learning models often produce predictions that are not 100% certain, but rather have some degree of confidence or probability attached to them. This uncertainty can be due to various factors, such as data quality, model complexity, or inherent randomness. How do you communicate this uncertainty to your stakeholders, clients, or users in a clear and meaningful way? In this article, we will explore some best practices and tips for presenting uncertain machine learning predictions.

1 Why uncertainty matters

Uncertainty is a crucial element of machine learning, since it indicates the restrictions and presumptions of the models and the data. Ignoring or concealing uncertainty can cause overconfidence, misinterpretation, or poor decision making. On the contrary, recognizing and displaying uncertainty can aid in validating and improving your models by uncovering sources of error and bias. It can also provide context and detail to your predictions by displaying the range of potential outcomes and the confidence intervals. Acknowledging uncertainty can engage and inform your audience by clarifying the reasoning and limitations of your models and the data. Moreover, it can manage expectations and risks by pointing out the scenarios and trade-offs related to using your predictions.

Add your perspective

Helen Wall

LinkedIn [in]structor | Data Science Consulting
Report contribution
I think it's important to build uncertainty into the model presentations, even if it's to a high-level audience. We can just adjust how we communicate what this uncertainly means and why it's important based on the audience we're presenting it to. Including uncertainty to the context of our work is a lot like guardrail along the sides of roads. We shouldn't be driving on the shoulder of the road, but if we find ourselves out there, we still have room to maneuver back to where we should be.

Like

2 How to measure uncertainty

In order to present uncertainty, you must first measure it. There are various ways to quantify uncertainty in machine learning, depending on the type and purpose of your model. Probabilistic models, like logistic regression, Bayesian networks, or neural networks with softmax activation, explicitly output probabilities or distributions for each prediction. These probabilities or distributions can then be used to calculate metrics such as entropy, variance, or confidence intervals. Non-probabilistic models, such as linear regression, support vector machines, or decision trees, do not output probabilities or distributions but rather point estimates. Bootstrapping, cross-validation, and ensemble methods can be used to estimate the uncertainty around these point estimates. Model-agnostic methods are independent of the specific model or algorithm and instead rely on the data and prediction task. Sensitivity analysis, permutation tests, and adversarial examples can be used to measure how robust your predictions are to changes in the input data or output labels.

Add your perspective

3 How to visualize uncertainty

Once you have measured uncertainty, you need to visualize it in a way that is easy to understand and interpret. There are various visualizations that can be used to convey uncertainty, depending on the nature and dimensionality of your data and predictions. For example, error bars or bands are simple yet effective ways to show the uncertainty around point estimates like means, medians, or regression lines. Such visualizations can be used for one-dimensional or two-dimensional data such as time series, scatter plots, or bar charts. Additionally, box plots or violin plots are useful for showing the distribution and variability of your predictions, particularly for categorical or ordinal data. Contour plots or surface plots are also powerful for showing the uncertainty over a continuous space like a map, a grid, or a function. They can be used for two-dimensional or three-dimensional data such as geographic data, spatial data, or function approximation.

Add your perspective

Helen Wall

LinkedIn [in]structor | Data Science Consulting
Report contribution
Time series forecast models are a great example of how to effectively visualize uncertainty. There's a line that extends past the historical data to represent the most likely outcome for each future time period. Around this line though, we see a shaded band representing the confidence interval for each future prediction. As we go farther into the future, this shaded area around the line expands because we're much less certain about the outcome a year from now than we are about the outcome next month.

Like

4 How to communicate uncertainty

Visualizing uncertainty is not enough, you also need to communicate it in a clear and effective way. To do this, you must consider the level of detail and granularity for your audience and purpose, the language and terminology used, and guidance for making informed decisions. You should avoid overwhelming or confusing your audience with too much or too little information. If needed, provide the option to drill down and explore the data in more depth. Use plain and consistent language that explains what your predictions mean, how confident you are, and what the sources and implications of uncertainty are. Additionally, offer clear and actionable insights, suggestions, or alternatives that help your audience to make rational decisions based on your predictions and uncertainty.

Add your perspective

5 How to evaluate uncertainty

In order to evaluate the quality and impact of your uncertainty presentation and communication, you can ask your audience for their opinions or preferences through feedback and surveys. You can also test their understanding, interpretation, or decision-making with A/B testing, randomized trials, or observational studies. Additionally, you can measure your audience's performance, behavior, or outcomes with objective or subjective metrics such as accuracy, error, confidence, trust, or satisfaction.

Add your perspective

6 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?

Add your perspective

How do you present uncertain Machine Learning predictions?

1

2

3

4

5

6

1 Why uncertainty matters

2 How to measure uncertainty

3 How to visualize uncertainty

4 How to communicate uncertainty

5 How to evaluate uncertainty

6 Here’s what else to consider

Machine Learning

Rate this article

Thanks for your feedback

More articles on Machine Learning

More relevant reading

How do you present uncertain Machine Learning predictions?

1

2

3

4

5

6

1 Why uncertainty matters

2 How to measure uncertainty

3 How to visualize uncertainty

4 How to communicate uncertainty

5 How to evaluate uncertainty

6 Here’s what else to consider

Machine Learning

Rate this article

Thanks for your feedback

Explore Other Skills