NICE real-world evidence framework

Selection bias at study entry

Selection bias at study entry can arise for several reasons including selection of patients based on eligibility criteria related to the exposure and outcome, or from deviations between the date the patient meets eligibility criteria, the date treatment is assigned, and the start of follow up. Common types of time-related bias are prevalent-user bias, lead time bias, immortal time bias and depletion of susceptibles. Discuss the potential for selection bias at study entry and how this was addressed or investigated through study design, statistical analysis or sensitivity analysis.

Selection bias at study exit

A common cause of selection bias because of how individuals exit a study is informative censoring. This may be because of loss to follow up or the occurrence of censoring events. Discuss the possibility of informative censoring and how this was addressed in the analysis.

Addressing confounding

Describe the risk of confounding from unmeasured (or unknown) confounders, poorly measured confounders, or time-varying confounding. This should be informed by a systematic identification of potential confounders, clear causal assumptions including the possibility of time-varying confounding, and differences in baseline characteristics between comparison groups.
Show how you dealt with any identified risk of confounding through study design (such as selection of a suitable active comparator) and analysis (using an appropriate statistical model, accounting for time-varying confounding). If possible, provide empirical data on the balance of baseline characteristics after adjustment.

If concerns remain about residual confounding, show its impact on results has been assessed using sensitivity or bias analysis.

Confirm that no covariates were inappropriately adjusted to induce bias. For example, show that no covariates on the causal pathway between interventions and outcomes were adjusted for (overadjustment). This may result from the use of covariates measured after the index date. Avoid adjustment for colliders or instruments. This can be informed by causal diagrams.

Detection bias

Describe the potential for detection bias resulting from differences in healthcare practices across comparison groups (for example, because of differential frequency or intensity of follow up, or different tests) or length of follow up.

Describe how these have been dealt with through study design (for example, use of comparator with similar follow up) or analysis (for example, adjustment for healthcare use before index date).

Measurement error and misclassification

Describe the potential for bias from measurement error or misclassification (this should be informed by assessment of data suitability). Consider which variables are inaccurate, whether this is random or systematic, and how it differs across comparison groups.

Show you addressed risks of bias through statistical analysis (for example, by incorporating external data or calibration) or assessed its impact on results using sensitivity or bias analysis.

Missing data

Describe the potential for bias from missing data (this should be informed by assessment of data suitability). Consider which variables have missing data, whether this is random or systematic, and how it differs across comparison groups.

Show how you have addressed risks of bias using statistical methods (such as multiple imputation) and demonstrating their validity. If missingness may not be explainable by observed variables or has unknown mechanisms, sensitivity or bias analysis can be used to explore the impact of different missing 'not at random' assumptions.

Reverse causation

Describe the risk of reverse causation between the intervention and the outcome arising from causal relationships between variables, time lag between recording of data on interventions and outcomes, or care pathways.
Describe how risk of reverse causation was addressed through study design (for example, induction periods or longitudinal follow up), analysis (for example, instrumental variables), or assessed through sensitivity analysis.

Selection bias at study entry

Previous observational studies of the effects of the timing of dialysis initiation are at high risk of lead time and immortal time bias resulting from non-alignment of the time at which eligibility criteria were met, treatment assignment, and start of follow up. The study emulated a target trial informed by the IDEAL trial. To avoid issues with misspecification of time zero, the study used the cloning, censoring, and weighting method. Patients are cloned and assigned to each treatment according to eGFR (one of 15 treatment strategies in the base case) and are censored once they deviated from a given treatment strategy. The approach was validated by replicating results from the IDEAL trial over the range of eGFR values seen in the trial.

Selection bias due to the choice of population was not an issue in this population-based study.

Selection bias at study exit

Selection bias can be induced by the censoring when patients stop adhering to the 'treatment strategy' if this is related to patient characteristics. Inverse probability of censoring weights were estimated using baseline and time-varying confounders to address censoring-induced selection bias.

Loss to follow up is very low.

Addressing confounding

The outcome model adjusted for baseline measurements including demographics, laboratory measurements, prior treatment and hospitalisations. Time-varying confounders were adjusted for in censoring weights including current and previous measurements of eGFR.

Data was not available on other potentially important confounders including muscle mass stores, uraemic symptoms, volume status, quality of life, or physical activity, and data was only available for subset of the cohort on urine albumin-creatinine ratio and plasma potassium. To assess the possibility of residual confounding, the study did the following sensitivity analyses:

Detection bias

Outcomes included 5‑year all-cause mortality and major adverse cardiovascular events (composite of cardiovascular death, non-fatal myocardial infarction, or non-fatal stroke). These are likely to be accurately observed regardless of small differences in level of surveillance, for example, resulting from earlier dialysis treatment.

Measurement error and misclassification

Timeliness and accuracy of variables extracted from the Swedish Renal Registry have previously been demonstrated. In particular, cardiovascular comorbidities have a very high positive predictive value, generally between 85% to 95%.

eGFR was calculated with the Chronic Kidney Disease Epidemiology equation from routine plasma creatinine measurements. This has been shown to be accurate to within 30% of measured glomerular filtration rate 85% of the time.

Missing data

Data on initiation of dialysis and key outcomes are thought to be complete. Data on mandatory items such as eGFR is also very high.

For non-mandatory data items in the registry, missingness was greater. For example, body mass index was missing in 26% of patients, urinary albumin to creatinine ratio in 44%, and potassium in 29%. This was assumed to be missing completely at random and determined by the preferences of the attending physician. Sensitivity analysis in the subset of people with data available showed had no impact on results.

Reverse causation

Reverse causation is not expected to be a problem in this analysis.

Selection bias at study entry

The study compared people enrolled in phase 2 trials assigned alectinib against patients from routine care in the US initiating ceritinib. Several steps were taken to minimise the risk of selection bias:

Matching inclusion criteria in the real-world data to the population included in the trial

Excluding additional patients from the trial with prior lines of therapy not observed in the real-world data

Using real-world data over a similar time period to that covered in the trial

Using a new-user, active comparator design

To help demonstrate the validity of the approach, the comparison was repeated using only real-world data and similar results were found.

Selection bias at study exit

This was an as-started analysis with limited loss to follow up. Censoring is not thought to be informative.

Addressing confounding

Key prognostic variables were prospectively identified by a systematic review.

Key known confounders were captured in the data albeit with limitations. See below for information on addressing missing data and misclassification of key confounders.

Observed confounders measured at or before baseline were used to estimate propensity scores. Estimation used the inverse probability of treatment weights method. There was no evidence of large differences in covariate patterns between treatment groups after adjustment (standardised mean difference was less than 0.1 for all variables).

In sensitivity analysis, adjustment for additional variables did not change results.

Quantitative bias analysis was used to assess how strong a confounding effect an unknown confounder would need to have to eliminate the estimated treatment effect. The estimated e-value was 2.4 which would require a level of confounder-mortality and confounder-treatment association substantially higher than seen for any measured confounders.

Detection bias

The outcome of mortality was not thought to be subject to detection bias.

Measurement error and misclassification

Data on mortality is sufficiently well captured in the real-world data with sensitivity of 91% and specificity of 96%.

There were concerns that central nervous system metastases were misclassified (underreported) in the real-world data due to limited surveillance. A sensitivity analysis found that the prevalence in the real-world data would have to be 40% larger to eliminate the estimated treatment effect.

Missing data

Missing data on baseline performance status (European Cooperative Oncology Group [ECOG] score) was high in the real-world data (32%) and this is a key prognostic variable. The main analysis assumed data was missing completely at random in a complete case analysis.

Because this assumption was expected to be invalid, sensitivity analysis was performed using multiple imputation assuming data was missing at random. Results were consistent with the complete case analysis.

Quantitative bias analysis was performed to address remaining concerns about missing not at random data, when ECOG scores are worse than expected by the imputation model. Using threshold analysis the study conclusions remained similar under any reasonable assumptions about the ECOG scores in those with missing values.

Reverse causation

Reverse causation is not expected to be a problem in this analysis.