Evidence standards framework for digital health technologies

What is the difference between the ESF and medical device regulation by the Medicines and Healthcare products Regulatory Agency?

Medical devices and in vitro diagnostics regulation, which is under Medicines and Healthcare products Regulatory Agency's (MHRAs) remit in the UK, is a legally mandated requirement for any digital health technology (DHT) within remit for regulation. The aim of regulation is to ensure that only safe and effective medical devices can be placed on the market. MHRA has provided guidance on medical device stand-alone software including apps to help companies to understand whether their DHT is within remit for regulation as a medical device or in vitro diagnostics.

The ESF is a set of non-mandated standards designed by NICE. The ESF describes the types and levels of evidence that different types of DHT should be able to demonstrate in order to be commissioned in the UK health and care system. The ESF standards include good design practices, describing and evidencing a value proposition, demonstrating clinical effectiveness and some issues relating to effective deployment of DHTs.

The ESF can be used to evaluate any DHT that is commissioned in the UK health and care system, regardless of whether it is regulated as a medical device or in vitro diagnostics, or not.

My DHT is a medical device and is UK Conformity Assessed marked, do I have to duplicate evidence presented to the UK-approved bodies in the ESF?

Within the ESF, there are some topics, such as user acceptability and demonstrating effectiveness, that might also be within the scope of regulations by MHRA for UK Conformity Assessed (UKCA) marking. It should not be assumed that these standards are met based on UKCA marking alone, and relevant details should be provided for the evaluator to understand the available evidence.

What is the difference between the ESF and the Digital Technology Assessment Criteria?

Both the Digital Technology Assessment Criteria (DTAC) and the NICE ESF help assess DHTs that enter the health and care system in the UK. They also help companies to understand what is needed when entering the health and care system. The DTAC covers clinical safety, data protection, technical security, interoperability and usability and accessibility standards. The ESF covers standards on design factors, describing value, demonstrating performance, delivering value and deployment considerations, so that companies can demonstrate and evaluators can evaluate the effectiveness and value for DHTs. Most standards covered in the ESF are not within the scope of DTAC. There might be topic overlap with a few standards such as ESF standard 1 and D1 in the DTAC around user acceptability.

My DHT passed the DTAC, do I have to duplicate evidence presented during the DTAC assessment in the ESF?

The DTAC covers clinical safety, data protection, technical security, interoperability and usability and accessibility standards. The ESF covers standards on design factors, describing value, demonstrating performance, delivering value and deployment considerations, so that companies can demonstrate and evaluators can evaluate the effectiveness and value of DHTs. Most standards covered in the ESF are not within the scope of DTAC. There might be topic overlap with a few standards such as ESF standard 1 and DTAC D1 around user acceptability. In such cases, the same evidence could be presented and the company might refer to the DTAC assessment.

What is the difference between NICE guidance and the ESF?

The ESF is a non-mandated tool that is designed to be used by evaluators in the health and care system to help them to decide whether to commission a DHT within their organisation. NICE guidance is produced by NICE teams, according to published methods. NICE guidance involves a more in-depth evaluation of the clinical and economic evidence for a DHT and results in a national-level recommendation on whether the technology should be used.

NICE's Centre for Health Technology Evaluation (CHTE) produces guidance across a range of health technologies, including DHTs. The health technology evaluation methods and processes used are described in NICE health technology evaluations: the manual. This manual describes the evidence considered in the evaluations to demonstrate the effectiveness of the technology and its economic value. Only DHTs that are UKCA or CE marked (under the transition agreement) are considered for NICE guidance development. The identification and selection for technologies including DHTs for guidance development are described in NICE health technology evaluation topic selection: the manual.

The ESF has been designed to apply to a broader range of DHTs, including tier A and tier B DHTs that are unlikely to be regulated as medical devices. The intended users of the ESF are commissioners and evaluators in the health and care system making purchasing decisions, and companies developing DHTs for use in the health and care system. The ESF is a tool to guide local evaluations, not a NICE evaluation process, and NICE does not use the ESF to produce NICE guidance. The ESF is a framework that has been produced to promote more consistency in the evaluation of DHTs across the NHS.

There is a common approach between the methods used for NICE guidance development and the standards described in the ESF. Both use an evidence‑based approach to ensure clinical effectiveness and value for money of the technologies used in the health and care system. NICE guidance identifies technologies for national adoption and so requires a detailed systematic review of the evidence, and its recommendations are based on independent committee decisions. The ESF is designed for local or regional evaluations of any DHT and so involves a simpler evaluation process and usually requires lower levels of evidence, for example, system service DHTs do not need clinical studies.

What is the difference between NICE medtech innovation briefings and the ESF?

NICE also produces medtech innovation briefings (MIBs). These summarise available information on a technology including the description of the technology, its use, a review of the relevant published evidence and the likely costs of the DHT. A MIB is NICE advice about a medical device or diagnostic and does not include recommendations about the use of the technology.

The ESF has been designed to apply to a broader range of DHTs, including tier A and tier B DHTs that are unlikely to be regulated as medical devices. The intended users of the ESF are commissioners and evaluators in the health and care system making purchasing decisions, and companies developing DHTs for use in the health and care system. The ESF is a tool to guide local evaluations, not a NICE evaluation process, and NICE does not use the ESF to produce MIBs. The ESF is a framework that has been produced to promote more consistency in the evaluation of DHTs across the NHS.

If my DHT passes the ESF, does this mean it is approved by NICE?

Meeting the ESF standards does not mean that a DHT is approved by NICE, and does not mean that the DHT has NICE guidance. The ESF is intended to be a tool to help inform local evaluations of a DHT.

Does meeting the ESF standards equate to an NHS Approval for use?

No, meeting the ESF is not an approval for use within the NHS. For the innovator it means you have an appropriate level of evidence for the type of technology, and so that's a good start. However, the quality of the evidence will need to be assessed by local evaluators (clinicians and commissioners in the NHS) and decisions made locally about approving your technology for use. If an innovator is successful in these steps then it would be legitimate to say: Our technology has been assessed by [Evaluator] against the NICE ESF and has been approved for use in [Region/Locality] for the [intended purpose].

My DHT has a positive ORCHA review, what does this mean to evaluators?

ORCHA (Organisation for the Review of Health and Care Apps) adapted an earlier version of the ESF in its evaluation process. The updated ESF has a different remit to an ORCHA evaluation. Although evaluators in the health and care system might be guided by an ORCHA evaluation outcome, they might ask for additional evidence as outlined in the ESF.

Does meeting the ESF standards link to a reimbursement process?

The ESF does not link to any local or national reimbursement schemes. There is currently no centralised reimbursement for DHTs in the UK.

How can I access help to develop evidence to support the claimed benefits of my DHT?

Here are some organisations that can support companies to develop their evidence‑generation plans:

Funding opportunities exist from organisations including:

What is a budget impact analysis? Which costs should be included?

The NICE ESF requires companies to provide a budget impact analysis (BIA) to inform the economic assessment of a DHT. The aim of a BIA is to give an estimate of the impact of the DHT on the decision-maker's budgets, usually over the next 5 years.

The key elements of a BIA include:

For each cost or benefit item included in the BIA, the number of resources required or saved, and their unit costs should be reported. The totals for all cost items and monetary benefits should be provided. For these totals, a breakdown should be given, to show whether these costs and benefits relate to cash costs or savings (such as money spent on consumables) or capacity costs or savings (such as reducing hospital admissions) together with the incremental cost or saving.

Key points to note include:

Where can I find information to support the development of a BIA?

NICE provides a budget impact template alongside the ESF. This provides an initial guide for the development of a BIA. Because each DHT is unique, the information contained in the BIA and how the BIA is populated will depend on the DHT. The budget impact template includes a worksheet that provides links to useful cost and data sources. The template also links to examples of BIAs for technologies that have received positive NICE guidance. Other useful resources to learn more about BIA include

Where can I find information to support the development of a cost–utility analysis?

Section 4.2.14 of NICE health technology evaluations: the manual describes the cost–utility approach in more detail. The principles are relevant to pharmacological technologies and DHTs.

Where can I find information to support the development of a cost–consequence analysis?

Section 7 of the Developing NICE guidelines: the manual describes the cost–consequence approach in more detail. The UK Health Security Agency published some guidance on how to use a cost–consequence analysis to evaluate a DHT.

Do DHTs that are already commissioned in the health and care system need to be evaluated using the ESF now?

The ESF has been designed to help inform a commissioning decision. DHTs that have already been commissioned could be evaluated using the ESF at their next recommissioning decision point.

Is the perspective in the ESF broader than a health technology assessment remit?

The revised ESF is based on standards presented in 5 groups: design factors, describing value, demonstrating performance, delivering value and deployment considerations.

The 4 describing value standards include information about the intended purpose of the technology, the target population and their current pathway of care, the proposed pathway with the DHT and the expected benefits associated with its use. This information is typically used to define the decision problem of a health technology assessment (HTA) process.

The HTA process usually focuses on evaluating the clinical effectiveness and value for money of a technology. These aspects are covered in the demonstrating performance and delivering value parts of the ESF.

The design factors standards include domains which may be assessed as part of the regulatory process but not all DHTs in the ESF will be medical devices and need regulation. Some of these standards, such as credibility with UK professionals and considerations of health and care inequalities, are important components of HTA.

The deployment considerations are a new area for the ESF, but these are essential for commissioners of data-driven DHTs. Feedback during the development of the standards highlighted the importance of understanding that many DHTs need significant recalibration when being implemented at a new site. For example, a DHT that is analysing imaging input data may produce slightly different results from different imaging hardware (even different models of the same machine). To help with this, standard 19 requests information about the requirements for implementing and embedding the new data-driven DHT. Similarly, we know that human factors play an important role in the successful implementation of these technologies. The effectiveness of data-driven DHTs depends on the effectiveness of the human team, not just on the AI algorithms. Staff need to understand and trust the new systems, and central to that will be communications and meeting training needs. This is covered in standard 20.

How should digital platforms be evaluated using the ESF?

A digital platform is defined as a digital tool, such as a website, that allows users access to several separate digital solutions (programmes or modules). This could be a single website that allows access to separate DHTs, such as different treatment modules or programmes for different mental health conditions.

The ESF is intended to be used to evaluate each digital solution or module within a platform independently. This is because each of these modules may be used by different patients or end users, and the health outcomes that can be used to measure their effectiveness are different. This means that it is more appropriate to evaluate these separately.

Will the ESF be updated over time?

We have designed the ESF to reflect current best practice in the evaluation of DHTs, including AI technologies. We expect that best practice will change further over the coming years and we will continue to review and update the ESF as needed. We anticipate that the first review of the ESF will be in 2023, after MHRA publishes updated regulations for medical devices.