As digital health technologies (DHTs) develop at an increasing pace, we've worked with stakeholders, system partners and thought leaders to develop standards that ensure new DHTs are clinically effective and offer value to the health and care system: the evidence standards framework (ESF).
What is the evidence standards framework?
The ESF is a set of evidence standards for a wide range of DHTs. Evaluators and decision makers in the health and care system can consistently use to help them identify DHTs that are likely to offer benefits to users and to the health and care system.
View the framework Read the user guide
In this short video, Mark Salmon, Programme Director at NICE, describes why NICE has created the Evidence Standard Framework for Digital Health Technologies, how it can benefit developers of digital technology and why developers should be interested in evidence generation.
Who is the ESF for?
The ESF is for people in the health and care system who are responsible for:
- identifying and evaluating new DHTs
- conducting formal product reviews
- authorising DHT product coverage, funding, or reimbursement
- conducting ongoing clinical and economic product evaluations in real-world settings.
We refer to these people as evaluators in the ESF.
The ESF is also for companies that develop or distribute DHT for use in the health and care system.
We refer to these as companies in the ESF.
What does the standard help me to do?
For companies that develop or distribute, and for evaluators and decision makers in the health and care system, the ESF:
- provides a classification system that helps to assign specific technologies to one of the tiers of standards within the framework
- provides a set of standards that can be consistently used by developers
- makes it easier to understand what good levels of evidence for digital health technologies look like.
For companies that develop or distribute DHTs, the ESF also:
- helps to understand how to demonstrate the effectiveness and value of a DHT when engaging with evaluators and decision makers in the health and care system.
For evaluators and decision makers in the health and care system the ESF also:
- helps to make more informed and consistent decisions when evaluating, commissioning or purchasing DHTs.
What the standard does not do
As a company, meeting the standard does not mean your DHT has been assessed or endorsed by NICE. The following webpages give advice on how NICE identifies and assesses medical technologies that include digital technologies:
- NHS Innovation Service, a new service used to connect technology innovators to several public sector organisations within the UK health care system.
- NICE selects technologies (including diagnostics) for the Medical Technologies Evaluation Programme (MTEP).
Meeting the standard also does not mean your DHT has been given regulatory approval as a device or as a service.
The ESF is aligned to regulatory processes wherever possible. You can also approach the following organisations for guidance:
- Medicines & Healthcare products Regulatory Agency (MHRA) are responsible for regulating medical devices. They will be able to offer advice on submissions to their organisation.
- Care Quality Commission (CQC) are responsible for regulating services. They will be able to offer guidance on registration with their organisation.
How can I use the framework?
- Read the full ESF document to view all the standards.
- Read the user guide to understand more about the ESF and how to use it.
- View example classifications (Excel) to see the ESF tier that may apply to a range of technologies.
- View case studies to see how the ESF might apply to some technologies.
- Use the budget impact tool (Excel) to estimate the costs and impacts to current practice that arise from using the DHT.
Who currently uses the framework?
The ESF is used locally and nationally. Several organisations have provided more information about how they use the framework.
Health Technology Wales (HTW) uses the ESF to assist with technology appraisal topic selection. It guides HTW’s view on whether a technology is mature enough to go forward for appraisal or not.
Decision making in this rapidly growing area is something everyone is grappling with. Up until now, we’ve had little support in this area. Health Technology Wales truly welcomes this update to the ESF from NICE.
Case study: Health Technology Wales
The Accelerated Access Collaborative within NHS England has incorporated the NICE evidence standards framework (ESF) for digital health technologies into their Artificial Intelligence (AI) in Health and Care Award.
Having an ESF that is regularly refreshed will help to ensure that the evidence we generate today will meet the system requirements of tomorrow.
We would like to hear from users who are using the ESF at digitalhealth@nice.org.uk.
How was this framework developed?
2019
We released the first version of the ESF in March 2019. It consisted of a set of standards that supported innovation and ensured an appropriate level of rigour and assurance for the health and care system.
2021
At the end of 2019, we surveyed people’s experiences of using the framework. Based on this, we published an updated framework in April 2021.
The feedback we received, and the changes made to the framework, are summarised in this feedback report (Word).
2022
We updated the framework to include artificial intelligence (AI) and data-driven technologies with adaptive algorithms. We aligned it with regulatory requirements and made it easier to use.
Following an open consultation, we published an updated framework in August 2022.
Read the consultation framework (Word) and our response (Word).
AI collaboration
The project has worked with an academic collaboration partner made up of academics from Imperial University, Birmingham University and the Turing Institute to identify standards for AI and data driven technology.
The Academic collaboration conducted a three-stage research programme:
- a scoping literature review
- a survey and interviews
- a Delphi Consensus Process.
Details of how this research programme was run and the findings from each stage can be found in this final report (Word).
The authors of this work are:
- Xiaoxuan Liu, Trystan MacDonald, Alastair K Denniston, Institute of Inflammation and Ageing, University of Birmingham, Birmingham, UK.
- Viknesh Sounderajah, Hutan Ashrafian, Ara Darzi, Institute of Global Health Innovation, Imperial College London, London, UK.
- Carolyn Ashurst, Adrian Weller, Chris Holmes, The Alan Turing Institute, London, UK.
This content is published with agreement from NHS AI lab.
Where can I find out more?
If you have any further questions, please contact us at digitalhealth@nice.org.uk.