Corporate document

Overview

Overview

Key messages

  • Real-world data can improve our understanding of health and social care delivery, patient health and experiences, and the effects of interventions on patient and system outcomes in routine settings.

  • As described in the NICE strategy 2021 to 2026 we want to use real-world data to resolve gaps in knowledge and drive forward access to innovations for patients.

  • We developed the real-world evidence framework to help deliver on this ambition. It does this by:

    • identifying when real-world data can be used to reduce uncertainties and improve guidance

    • clearly describing best practices for planning, conducting and reporting real-world evidence studies to improve the quality and transparency of evidence.

  • The framework aims to improve the quality of real-world evidence informing our guidance. It does not set minimum acceptable standards for the quality of evidence. Users should refer to relevant NICE manuals for further information on how recommendations are made (see the section on NICE guidance).

  • The framework is mainly targeted at those developing evidence to inform NICE guidance. It is also relevant to patients, those collecting data, and reviewers of evidence.

  • Table 1 summarises key considerations for conducting real-world evidence studies. The following core principles should be followed to generate high-quality and trusted real-world evidence:

    • ensure data is of good provenance, relevant and of sufficient quality to answer the research question

    • generate evidence in a transparent way and with integrity from study planning through to study conduct and reporting

    • use analytical methods that minimise the risk of bias and characterise uncertainty.

  • The framework provides in-depth guidance and tools to support the implementation of these core principles across different uses of real-world evidence. It is structured as follows:

  • The framework is a living framework that will be updated periodically to reflect user feedback, learnings from implementation including exemplar case studies, developments in real-world evidence methodology, and to extend its scope to include additional guidance on priority topics.

  • We encourage companies planning to use real-world data in their submissions to engage early with NICE's Advice service on how to make best use of real-world data as part of their evidence-generation plans.

Table 1

Summary of key considerations in planning, conducting and reporting real-world evidence studies
Stage of evidence generation Key considerations

Planning

  • Clearly define the research question including, as relevant, definitions of population eligibility criteria, interventions, outcomes and the target quantity of estimation

  • Plan the study in advance and make protocols (including a data analysis plan) publicly available

  • Choose data that is of good provenance and of sufficient quality and relevance to address the research question (see the section on assessing data suitability)

  • When planning primary data collection, consider how to implement this collection in a patient-centred manner while minimising the burden on patients and healthcare professionals

  • Use data in accordance with local law, data governance processes, codes of practice and the requirements of the data controller

Conduct

  • Use a study design and statistical methods appropriate to the research question, considering the key risks of bias

  • Use sensitivity and/or bias analysis to assess the robustness of studies to key risks of bias and uncertain data curation or analytical decisions

  • Create and implement quality assurance standards and protocols to ensure the integrity and quality of the study

Reporting

  • Report study design and analytical methods in sufficient detail to enable independent researchers to fully understand what was done and why, critically appraise the study and reproduce it

  • Reporting should also cover:

    • provenance, quality, and relevance of the data (see the section on assessing data suitability)

    • data curation

    • patient attrition from initial data to the final analyses

    • characteristics of patients (including missing data) and details of follow up overall and across key population groups

    • results for all planned and conducted analyses (clearly indicating any analyses that were not pre-planned)

    • assessment of risk of bias and generalisability to the target population in the NHS

    • limitations of the study and interpretation of what the results mean

Real-world data and its role in NICE guidance

  • Real-world data refers to data relating to patient health or experience or care delivery collected outside of highly controlled clinical trials. It can come from many different sources including patient health records, administrative records, patient registries, surveys, observational cohort studies and digital health technologies.

  • Real-world data is already widely used to inform NICE guidance to, for example:

    • characterise health conditions, interventions, care pathways and patient outcomes and experiences

    • design, populate and validate economic models (including estimates of resource use, quality of life, event rates, prevalence, incidence and long-term outcomes)

    • develop or validate digital health technologies (for example, digital technologies may use a clinical algorithm developed using real-world data)

    • identify, characterise and address health inequalities

    • understand the safety of medical technologies including medicines, devices and interventional procedures

    • assess the impact of interventions (including tests) on service delivery and decisions about care

    • assess the applicability of clinical trials to patients in the NHS.

  • Real-world data that represents the population of interest is NICE's preferred source of evidence for most of these applications. Such data is regularly used for these purposes in NICE guidance, but its use could be more commonplace, especially of routinely collected data.

  • Randomised controlled trials are the preferred source of evidence on the effects of interventions. Randomisation ensures that any differences in baseline characteristics between groups are because of chance. Blinding (if applied) prevents knowledge of treatment allocation from influencing behaviours. However, randomised trials are sometimes unavailable or are not directly relevant to decisions about patient care in the NHS.

  • Randomised trials may not be available for several reasons, including:

    • randomisation is considered unethical or unfeasible (for instance, for some rare or severe diseases with unmet need)

    • technical challenges make randomisation impractical, which is most common for medical devices and interventional procedures

    • funding is not available for a trial (for example, when the intervention is already used in routine practice).

  • Even if randomised evidence is available, it may not be sufficient for decision making in the NHS for several reasons including:

    • the comparator does not reflect standard of care in the NHS

    • relevant population groups are excluded

    • there are major differences in patient behaviours, care pathways or settings that differ from implementation in routine practice

    • follow up is limited

    • unvalidated surrogate outcomes are used

    • learning effects are present

    • trials were of poor quality.

  • Non-randomised studies are already widely used to estimate the effects of medical devices and procedures and public health interventions, for which trials are less common. They are becoming more widely used in initial assessments of medicines, as more are granted regulatory approval based on uncontrolled single-arm trials. Finally, the increased focus on the lifecycle evaluation of technologies and lived experiences of patients relies on non-randomised studies after initial approvals. The most common non-randomised studies using real-world data to assess comparative effects are observational cohort studies and single-arm trials with real-world external control.

  • Real-world data could be used more routinely to fill evidence gaps and speed up patient access. For this promise to be realised, real-world evidence studies must be performed transparently and with integrity, use fit-for-purpose data, and address the key risks of bias.

  • We are communicating our view on best practices for the conduct of real-world evidence studies to ensure they are generated transparently and are of good quality. This is essential to improving trust in real-world evidence studies and their use in decision making.