What is uncertainty?

Uncertainty in the in context of HTA is, “a lack of confidence in the conclusions of an HTA owing to limitations in the available evidence.” 

As covered in previous explainers, HTA is a process that supports evidence-based medicine by using clinical and health economic evidence to inform policy decision-making for health technologies. No one piece of evidence can provide all the answers for this process, with evidence coming from a range of different sources. The collation of evidence is limited by the availability of data and the practicalities or feasibility of generating evidence. It is these limitations that drive this “lack of confidence”. This is important as the level of certainty will drive decisions relating to whether a health technology should be used at all and if so, in which population, for how long and at what cost.

Sources of uncertainty

A review of HTA guidance will highlight that uncertainty comes from different sources - clinical evidence, economic evaluation, and overall budget impact/affordability. Looking at each of these areas individually, we see a range of factors driving uncertainty.

Clinical Evidence: If one considers a set of registrational trials for a new health technology, these are normally tightly controlled clinical trials, in specific patient populations and will be powered to show an effect on predefined outcome measures over a period of treatment and follow-up. The objective of these studies will be to provide evidence of the balance of benefits and risks in using the new health technology in the study population, primarily to support regulatory approval. These trials are also likely to be used to support HTA.  The nature of these trials may not provide evidence reflecting how the health technology will be used in clinical practice for a given jurisdiction or over extended periods. For example, the clinical characteristics of the study populations may differ compared with a general clinical population, which could impact the outcomes observed. The trial duration may not provide sufficient follow-up for final outcomes to be evaluated and treatment duration within the trial may also vary compared with expected clinical practice. Whilst providing the required evidence for registrational approval, these have the potential to leave gaps in evidence and understanding of the impact of how the health technology would be used in everyday practice, driving uncertainty in the clinical evidence.

Economic Evaluation: In model-based analyses, there are 4 types of uncertainty observed, variability, heterogeneity, structural and parametric uncertainty. Variability refers to stochastic uncertainty, that is different outcomes observed in the same patients due to random chance. Uncertainty via heterogeneity is the difference between patients which can be explained e.g. age, baseline performance status or disease severity. Structural uncertainty is driven by the choices made when deciding the structure of the model – for example the inclusion of different health states or the way the intervention or disease pathways are modelled. Structural uncertainty can include the choice of comparators, inclusion of events, different methods of estimation and clinical uncertainty. Finally, parametric uncertainty is driven by the parameter estimates used within the model and the associated uncertainty inherent within the data.

Budget impact or affordability: Uncertainty related to budget impact or affordability can be a function of the overall certainty related to cost-effectiveness (if relevant) and/or the overall size of the eligible population for the health technology. Uncertainty here can drive very different budget and affordability implications.

Managing uncertainty

Recognising the potential impact of uncertainty in HTA allows decision-makers to account for these in their policy decisions. Managing this uncertainty within a decision problem can be done in different ways by identifying and characterising the uncertainty within an evaluation and generating additional evidence to address gaps in the evidence base. 

Clinical evidence: a review of an HTA will highlight areas of alignment and difference within the clinical evidence base compared with the decision problem, focusing on areas of the PICO and how generalisable the data is to the jurisdiction. For example, are the trial subjects representative of the population targeted within the decision problem? Other areas could include trial duration, follow-up, dosing/time on treatment or selected outcomes. Depending on the differences, additional evidence generation may provide an opportunity to address the identified evidence gaps and provide more certainty in the data. 

Economic evaluation: uncertainty within economic evaluation can be characterised via different approaches, depending on the type of uncertainty. 

Variability can be characterised using the standard deviation observed within the data generated and via methods such as a Monte Carlo microsimulation. In Monte Carlo microsimulation the parameter value can be compared to a mean value derived from the microsimulation. This is obtained by using a large number of microsimulation iterations to identify the central tendency of a population, where random chance dictates how that population will differ. 

Heterogeneity can be characterised via subgroup analysis, with different population characteristics driving parameter estimations and different outcomes. Examples of this could be patients with differing tumour characteristics.

Structural uncertainty can be characterised via the use of multivariate sensitivity analysis/scenario analysis to understand the outcomes using different combinations of parameter estimates. Another method is model averaging, where a weighted average outcome is generated based on the outcomes of multiple different models. This method is designed to account for the inherent complexity in designing a single representative model and that multiple models could be credibly used to address a decision problem.

Parametric uncertainty has been characterised using univariate and multivariate sensitivity analysis, the standard method however is probabilistic sensitivity analysis. This allows one to understand the uncertainty of parameter values across multiple parameters at the same time, alongside probability distributions. Uncertainty is then characterised using Monte Carlo simulation to randomly select parameter values from each of the resulting parametric distributions.

Budget impact/affordability: whilst the characterisation of uncertainty from clinical evidence and economic evaluation will help manage uncertainty relating to the overall budget, it is important for decision-makers to have confidence in estimates used for the cost of care and population size during the decision period. Like economic evaluation, structural and parametric uncertainty can exist in budget impact analysis; relevant data however is often limited and therefore sensitivity analysis is limited in how it can meaningfully quantity any associated uncertainty. Scenario analysis can help characterise the uncertainty, alongside the generation of relevant evidence to address any data gaps.

Accounting for uncertainty in decisions

Decision-makers can mitigate identified uncertainty using different levers, depending on the source and impact of the uncertainty. These levers can include price adjustment, limiting the eligible population, treatment duration; requiring additional evidence generation and in the case of managing budgets, applying caps to limit exposure. 

In summary, uncertainty is, “a lack of confidence in the conclusions of an HTA”, is driven by the inherent limitations of evidence generation and is important as it directly impacts decisions relating to whether a health technology should be used and if so, in which population, for how long and at what cost. Uncertainty can be found in clinical evidence, economic evaluation and analyses relating to budget impact/affordability. It is possible to characterise uncertainty across these different pieces of evidence and take steps to manage or mitigate any risk associated with the uncertainty. These steps include price adjustments, access limitations, additional evidence generation and applying caps to manage budget exposure.

1. HTA Glossary.net (2024) Definition of uncertainty. Accessed 21st July 2024
2. Health technology appraisals from ICER and NICE guidance (sources available on request)
3. Gray et al. (2011) Applied methods of cost-effectiveness analysis in health care. Oxford University Press
4. Sullivan, S et al. (2014) Budget Impact Analysis – Principles of Good Practice. Value in Health 17 (2014) 5-14