NICE real-world evidence framework

Eligibility criteria

For most studies, the eligibility criteria should mimic a hypothetical pragmatic trial by reflecting the clinical pathways (including diagnostic tests) and patients seen in routine care in the NHS. For external control studies, the focus should be on matching the eligibility criteria from the interventional study rather than the broader target population. As in a trial, eligibility criteria should be based on variables recorded before treatment assignment.

If heterogeneity is anticipated in the intervention effects, subgroup analysis can be done. The subgroups should be defined upfront when planning the study.

Treatment strategies

Treatment strategies include the intervention of interest and any comparators. Comparators could be different levels of an exposure (for example, different doses of a medicine), a different intervention, or the absence of intervention. In observational data it is very difficult to emulate a placebo-controlled trial because of higher risk of selection bias and intractable confounding.

Comparators that are for the same (or similar) treatment indication (that is, active comparators) are preferred to comparison with those not receiving an intervention. Active comparators reduce the risk of confounding by indication by ensuring greater similarity of patients having different interventions. If routine follow-up procedures are similar across interventions this also reduces the risk of detection bias. The active comparator should ideally reflect established practice in the NHS.

For studies of interventions, new (or incident) user designs are generally preferred to studies of prevalent users (those who have already been using the intervention for some time) because of the lower risk of selection bias and better emulation of trial designs. Prevalent users have, by definition, remained on-treatment and survived for some period of follow up. When making use of already collected data, new users are typically defined using an initial period in which the individual was not observed to use the intervention of interest (known as the 'washout' period in pharmacoepidemiology). A further advantage of new-user designs is the ability to estimate time-varying hazards from treatment initiation. The inclusion of prevalent users may be needed if the effects of interventions are cumulative, there are too few incident users in the data, or follow up is limited (Vandenbroucke and Pearce 2015, Suissa et al. 2016).

Data on comparators would ideally come from the same period as the intervention as well as from the same healthcare system and settings. This is to minimise any differences between treatment groups resulting from differences in care access, pathways (including diagnostic tests) or time-based trends in outcomes.

Assignment procedure

In randomised controlled trials, individuals (or groups) are randomly assigned to interventions. If possible, providers, patients and analysts are blinded to this assignment. Neither random assignment nor blinding are possible in observational studies. With sufficient information on confounders, random assignment can, however, be approximated through various analytical approaches (see the section on analysis).

In some applications, individuals will meet eligibility criteria at multiple time points. For example, they may start treatment more than once after a sufficient period without exposure (or 'washout' period). There are several approaches to deal with this including using only the first eligible time point, a random eligible time or all eligible time points (Hernán and Robins 2016).

Follow-up period

The start and end of follow up must be defined. The start of follow up should ideally begin at the same time at which all eligibility criteria are met and the intervention is assigned (or just after). If a substantial latency period is expected between treatment initiation and outcomes, it may be necessary to define an induction period before which outcomes are not counted. This can reduce the risk of reverse causation, in which the outcome influences the exposure.

The follow-up period should be long enough to capture the outcomes of interest but should not exceed the period beyond which outcomes could be reasonably impacted by the intervention (known as the exposure-effect window). Censoring events should be clearly defined and will depend on the causal effect of interest.

Outcomes

Primary and secondary outcomes should be defined and can include both patient and health system outcomes (such as resource use or costs). Patient outcomes should reflect how a patient feels, functions, or how long a patient lives. This includes quality of life and other patient-reported outcome measures. Objective clinical outcomes (such as survival) are typically subject to a lower risk of bias than subjective outcomes if outcome detection or reporting could be influenced by known treatment history.

For a surrogate outcome there should be good evidence that changes in the surrogate outcome are causally associated with changes in the final patient outcomes of interest (Ciani et al. 2017).

While outcome ascertainment is not blinded in observational data, analysts can be blinded to outcomes before finalising the analysis plan (see the section on analysis).

Causal effect of interest

Researchers should describe the causal effect of interest. Trials are usually designed to estimate 1 of 2 causal effects: the effect of assignment to an intervention (intention-to-treat) or the effect of adhering to treatment protocols (per-protocol). It is not usually possible to estimate the effect of treatment assignment using observational data because this is not typically recorded. However, it can be proxied using treatment initiation (the as-started effect). The equivalent of the per-protocol effect is sometimes called the on-treatment effect.

The as-started effect is usually of primary interest to NICE. However, if treatment discontinuation (or switching) is substantial or is not expected to reflect routine practice or outcomes in the NHS, it is important to present results from the on-treatment analysis. On-treatment analyses may also be most appropriate for the analysis of safety and adverse events. The on-treatment effect can also be extended to cover dynamic treatment strategies such as treatment sequences or other complex interventions which are of interest to NICE.

Analysis plan

The analysis plan should describe how the causal effect of interest is to be estimated, taking into account intercurrent events. Intercurrent events are events occurring after treatment initiation (such as treatment switching or non-adherence) that affect the interpretation of the outcome of interest. This is supported by the estimand framework (for further information, see ICH E9 [R1] addendum on estimands and sensitivity analysis in clinical trials).

The relevance of intercurrent events will depend on the causal effect of interest. In an as-started analysis, treatment discontinuation, switching or augmentation can usually be ignored. However, if these changes are substantial there is a risk of increasing exposure misclassification over time. In most cases this would bias estimates of effect towards the null.

In an on-treatment analysis or when modelling dynamic treatment strategies, the follow up is often censored once the patient stops adhering to the treatment plan plus some biologically informed effect window. For medicines (and some devices) continued exposure is proxied by dates of prescriptions and expected period of use (for example, derived from number of days' supply), with some grace period between observations permitted. Particular attention needs to be given to the possibility of informative censoring, which causes bias if censoring depends on outcomes and differs across interventions, and time-varying confounding.

Further content on statistical analysis including addressing confounding, informative censoring, missing data and measurement error is presented in the analysis section.

Panel 1 shows examples of using the target trial approach:

Identification and selection of confounders

Potential confounders should be identified before analysis, based on a transparent, systematic and reproducible process. Key sources of evidence are published literature and expert opinion. Consideration should be given to the presence of time-varying confounders. These affect the outcome and future levels of the exposure and can be affected by previous levels of the exposure. They are especially relevant when modelling time-varying interventions or dynamic treatment strategies or addressing informative censoring.

Developers should outline their assumptions about the causal relationships between interventions, covariates and outcomes of interest. Ideally, this would be done using causal diagrams known as directed acyclic graphs (Shrier and Platt 2008).

Inappropriate adjustment for covariates should be avoided. This may result from controlling for variables on the causal pathway between exposure and outcomes (overadjustment), colliders or instruments. Confounders that may change value over time should be recorded before the index date, except when using statistical methods that appropriately address time-varying confounding.

The selection of covariates may use advanced computational approaches such as machine learning to identify a sufficient set of covariates, for example, when the number of potential covariates is very large (Ali et al. 2019, Tazare et al. 2022). The use of these methods should be clearly justified and their consistency with causal assumptions examined. Choosing covariates based on statistical significance should be avoided.

Selecting methods for addressing confounding

Adjusted comparisons based on clear causal assumptions are preferred to naive (or unadjusted) comparisons. Statistical approaches should be used to address confounding and approximate randomisation (see the section on assignment procedure).

Various approaches can be used to adjust for observed confounders including stratification, matching, multivariable regression and propensity score methods, or combinations of these. These methods assume no unmeasured confounding. Simple adjustment methods, such as stratification, restriction and exact matching, may be appropriate for research questions in which confounding is well understood and there are only a small number of confounders that are well recorded.

If there are many potential confounders, more complex methods such as multivariable regression and propensity score (or disease risk score) methods are preferred. Propensity scores give the probability of receiving an intervention based on observed covariates. Several methods use propensity scores including matching, stratification, weighting and regression (or combinations of these). General discussions of the strengths and weaknesses of these different approaches can be found in Ali et al. 2019. The choice of method should be justified and should be aligned with the causal effect of interest.

There is mixed evidence on the relative performance of regression and propensity score methods for addressing confounding bias (Stürmer et al. 2006). However, using propensity score methods may have advantages in terms of the transparency of study conduct:

Regression and propensity score methods may also exclude some participants to enhance the similarity of people across intervention arms or levels. When using such methods, trade-offs between internal validity, power and generalisability should be considered. For studies of comparative effects, internal validity should generally be prioritised.

Time-varying confounders should typically not be adjusted for using the above methods. It may be acceptable for on-treatment analyses if confounders that vary over time are not affected by previous levels of the intervention but this is uncommon. G-methods including marginal structural models with weighting are preferred (Pazzagli et al. 2017, Mansournia et al. 2017). Adjustment for time-varying confounders requires high-quality data over the whole follow-up period.

Various sensitivity and bias analyses can be used to adjust for bias because of residual confounding or to explore its likely impact (see the section on assessing robustness of studies). This may be informed by external data on confounder-outcome relationships or data from a data-rich subsample of the analytical database, if available (Ali et al. 2019). Negative controls (that is, outcomes that are not expected to be related to the intervention) may also be useful (Lipsitch et al. 2010).

If there are multiple potential sources of suitable real-world data to provide external control to trial data, developers should consider whether to estimate effects separately for each data source or to increase power by pooling data sources. Data sources should only be pooled when there is limited heterogeneity between sources in terms of coverage and data quality. Individual estimates of effects for each data source should always be provided.

External controls can also be used to supplement internal (or concurrent) controls in randomised controlled trials. There are several methods available to combine internal and external controls, which place different weight on the external data (NICE Decision Support Unit report on sources and synthesis of evidence).

Instrument-based methods (or quasi-experimental designs) can be used to address unobserved confounding (Matthay et al. 2019). Further technical guidance on methods for addressing baseline confounding due to observed and unobserved characteristics using individual patient-level data is given in NICE's Decision Support Unit technical support document 17.

Informative censoring

Censoring occurs in longitudinal studies if follow up ends before the outcome is fully observed. It can happen because the data collection period ends (administrative censoring), loss to follow up, occurrence of events such as treatment switching, non-adherence, or death depending on the analysis. It may be induced by analytical strategies such as cloning to avoid time-related biases in studies without active comparators (Hernán and Robins 2016).

Censoring can create bias if it is informative (that is, it is related to the outcomes and treatment assignment). For example, in on-treatment analyses, if people on an experimental drug were less likely to adhere to the treatment protocol because of a perceived lack of benefit this could lead to informative censoring. When modelling effects on-treatment or dynamic treatment strategies, censoring because of treatment switching is likely to be informative. Methods to address informative censoring are similar to those for time-varying confounding such as marginal structural models with weighting or other G-methods (Pazzagli et al. 2017). Methods for dealing with missing data may also be used (see the section on missing data).

Missing data

The impact of missing data depends on the amount of missing data, the variables that have missing data, and the missing data mechanism. Developers should compare patterns of missingness across exposure groups and over time, if relevant, considering causes of missingness and whether these are related to outcomes of interest. Missing data on outcomes may arise for a number of reasons including non-response to questionnaires or censoring.

If the amount of missing data is low and likely to be missing completely at random, complete records analysis will be sufficient. Advanced methods for handling missing data include imputation, inverse probability weighting and maximum likelihood estimation. Most of these methods assume the missing data mechanism can be adequately modelled using available data (that is, missing at random). If this is not the case, sensitivity or bias analysis may be preferred (see the section on assessing robustness). A framework for handling missing data is provided in Carpenter and Smuk 2021.

Measurement error and misclassification

Measurement error describes the extent to which measurements of study variables deviate from the truth. For categorical variables, this is known as misclassification. The impact of measurement error depends on the size and direction of the error, the variables measured with error, and whether error varies across intervention groups. Measurement error can induce bias or reduce the precision of estimates.

Random measurement error in exposures tends to (but does not always) bias estimates of treatment effects towards the null (van Smeden et al. 2020). Random measurement error in continuous outcomes reduces the precision of estimates but provides unbiased estimates of comparative effects. For risk ratios and rate ratios, non-differential misclassification of a categorical outcome provides unbiased estimates of comparative effects when specificity is 100%, even if sensitivity is low. So, it is often recommended to define outcome variables to achieve high specificity.

Differential measurement error in exposures, covariates or outcomes generally produces biased estimates of comparative effects but the direction of bias can be hard to predict. If data is available on the likely structure and magnitude of measurement error (for example, through an internal or external validation study), this information can be incorporated into analyses using calibration or other advanced methods (van Smeden et al. 2020).

Assessing external validity

This section focuses on methods to assess and address external validity bias resulting from differences in patient characteristics (for example, age, disease risk scores) between the analytical sample and the target population. Importantly, differences in patient characteristics may not be the only, or most important, sources of external validity bias. Developers should also consider differences in: setting (for example, hospital type and access to care), treatment (for example, dosage or mode of delivery, timing, comparator therapies, concomitant and subsequent treatments) and outcomes (for example, follow-up, measurements, or timing of measurements). Identifying a suitable data source, using a target trial approach and using internally valid analysis methods remain the primary approaches by which external validity can be achieved.

To assess external validity, an explicit definition of the target population is needed and suitable reference information. Information can be drawn from published literature, context-relevant guidelines, or bespoke analysis of data from the target population alongside information gathered during the data suitability assessment.

To assess differences between the analytical sample and target population for patient characteristics, several tests are available:

In studies of relative treatment effects, differences observed between the analytical sample and target population do not necessarily lead to concerns about external validity bias unless those differences are considered to be important treatment effect modifiers. This depends on the causal effect of interest, the extent of heterogeneity in the treatment effect, and whether this has been adequately modelled. Assumptions about the causal relationships between interventions, outcomes, and other covariates can be outlined to help identify potential treatment effect modifiers, for example, using directed acyclic graphs (Shrier and Platt 2008). Under certain conditions, treatment effect modification can also be investigated statistically (Degtiar and Rose 2022).

In studies of absolute treatment effects, assessment of external validity requires consideration of all differences that are prognostic of the outcome of interest, not only treatment effect modifiers.

Methods to minimise external validity bias

Methods to adjust for external validity bias are similar to those which adjust for confounding bias, including matching, weighting, and outcome regression methods. These approaches can also be combined for additional robustness.

Degtiar and Rose 2022 provides further guidance on these methods, including approaches for when only summary-level data is available for the target population. Adjustment approaches are unlikely to perform well when the target population is poorly represented in the analytical sample, that is, where there is insufficient overlap for important covariates, or across strata of these variables. Successful application of these methods also depends on good internal validity of analyses and consistency in measurements of outcomes, treatments, and covariates across settings.

Where the sample is drawn from an entirely different population to the target population, judgements of similarity will require stronger assumptions. Pre-specified, empirical assessments of 'transportability' for the decision context could provide supportive evidence (for example, see Ling et al. 2023). In all cases, sensitivity analyses are recommended to explore potential violation of study assumptions (for example, see Dahabreh et al. 2023, Nguyen et al. 2018).