Tools and resources

Evidence standards framework for digital health technologies: user guide

How the framework was developed

How the framework was developed

NICE developed the first version of the ESF in 2018 and has published details separately on how the evidence standards were developed (Unsworth et al. 2021). Based on stakeholder comments and feedback received in early 2019, we made changes to the framework. These changes were mainly additional clarifications and further explanation of the evidence standards.

In June 2021, NICE started a wider update of the ESF, which included:

  • alignment with regulatory frameworks in development

  • addressing stakeholder feedback obtained via a survey in October 2019

  • inclusion of digital health technologies (DHTs) that use adaptive algorithms (that is, algorithms that continually change).

Experts in artificial intelligence (AI) in healthcare at Birmingham University, Imperial College London and the Turing Institute did research and advised NICE around the challenges of DHTs with adaptive algorithms and how these can be addressed by appropriate evidence standards in the framework. Between October 2021 and March 2022, the following stakeholders provided comments and feedback:

  • industry representatives, developers and companies

  • healthcare evaluators

  • regulators

  • academic experts in health technology assessment, DHTs and AI

  • clinical experts with special interests in DHTs and AI

  • national and international health technology assessment organisations

  • organisations responsible for promoting innovations such as Academic Health Science Networks (AHSNs), Digital Accelerators and the NHS AI Lab.

The work carried out by the academic consortium is described in the accompanying report by the academic consortium led by Imperial college and in the report by Unsworth et al. (2022).

During this update of the ESF, the NICE team also worked closely with other national bodies interested in the evaluation of DHTs such as Medicines and Healthcare products Regulatory Agency (MHRA), Health Technology Wales and the Scottish Health Technology Group. There was also significant interest from overseas including health technology assessment (HTA) organisations who are also exploring how these types of technologies should be evaluated. During the revision of the ESF we had discussions with groups working in France, Australia, Italy and Latin America.

We updated the classification system based on stakeholder feedback and to align with medical device regulatory frameworks in development. For tier C, we aligned the classification groups to the software as a medical device (SaMD) classification framework proposed by the International Medical Device Regulators Forum (IMDRF), of which the MHRA is a member. For tier B, we simplified the classification groups so that most tier B technologies can be covered by 1 of the 3 groups. There are no changes to tier A.

We updated the evidence standards based on stakeholder feedback, and research and input from our academic partners. These changes include:

  • updating and amending existing standards

  • expanding standards so they are also relevant to DHTs with adaptive algorithms

  • adding new standards such as standards for environmental sustainability and good data practice

  • restructuring the standards to reflect a DHT's life cycle.

The updated ESF covers 5 broad areas: design factors, describing value, demonstrating performance, delivering value and deployment considerations (see the section on deciding which evidence standards are relevant).

The digital healthcare environment is rapidly evolving and we aim to regularly review and update the ESF so that it reflects developments in areas such as regulation and standards for use of DHTs in the UK health and social care system.


This page was last updated: