Tools and resources

Evidence standards framework for digital health technologies: user guide

How to use the framework

How to use the framework

Deciding whether the technology is within the scope of the framework

We developed the ESF for a broad spectrum of digital health technologies (DHTs) that are of interest to the health and social care system.

If you can answer all the following 4 questions with 'Yes', then it is likely that the technology is covered by the ESF:

  1. Is the technology an app, software or online platform that is intended to benefit people's health or care or the wider health and social care system?

  2. Does the technology have a medical, health or wellness, or system efficiency purpose?

  3. Does the technology offer value to the health and social care system?

  4. Is the technology available and likely to be commissioned in the UK health and social care system?

If you answer 'Yes' to any of the following questions, then it is unlikely that the technology is covered by the ESF:

  1. Is the technology designed for training health or care professionals or facilitating data collection in research studies?

  2. Is the technology and software integral to or embedded in a medical device? Embedded software is likely to be regulated as software in a medical device (SiMD) rather than software as a medical device (SaMD).

  3. Is the technology a surgical robot?

Deciding what category the technology falls in

To decide what category the DHT falls in, we have used a tiered approach (consisting of 3 tiers) based on the potential risk to the person whose health or wellbeing is affected by the DHT, and to the system:

  • Tier A: the technology is intended to release costs or staff time or to improve efficiency.

  • Tier B: the technology is intended to help patients and citizens to manage their own health and wellness.

  • Tier C:

    • the technology is intended to treat or diagnose a specific condition or guide treatment, diagnosis and care choices

    • the technology is intended to have direct health outcomes

    • the technology is a medical device or an in vitro diagnostic or a screening tool.

Tier A has the lowest associated risk and tier C has the highest associated risk. Tier A includes only 1 category, whereas tier B includes 3 categories and tier C includes 4 categories.

The categories are defined by the intended purpose of the DHT. Tables 1 to 3 describe examples of technologies in the different tiers. Further examples can be found in the ESF classification examples spreadsheet.

To identify the category for a technology, consider the tier and category descriptions in the following 3 tables indicating the intended purpose of technologies within that group. The examples shown and those included in the ESF classification examples spreadsheet are designed to help users understand how the classification works in practice. These examples have been classified based on their publicly available descriptions.

Table 1 Examples of tier A technologies

Category within tier

Example technologies

System service

There are no direct patient, health or care outcomes from this technology.

  • An app intended for documenting patient encounters via a mobile or desktop.

  • An app intended to work as a wireless microphone, optimised for use with specific software.
Table 2 Examples of tier B technologies

Category within tier

Example technologies

Communicating about health and care

Communicating with health professionals or others, to help service users to manage their health and care.

  • A messaging service intended for within-team and practitioner-to-patient messaging.

Health and care diaries

Health and care diaries to help service users to manage their own health and wellness.

  • An app intended for tracking vital health and lifestyle metrics using automatically captured data, as well as data manually entered by the user

  • An app intended for patients to track treatment and record fatigue, appetite and pain in order to help spot trends and improve care during breast cancer treatment

Promoting good health

Population-level information to help people and service users to maintain healthy lifestyles and manage conditions.

  • An app intended to support mental health by providing courses on mindfulness and meditation, and sleep-specific solutions such as relaxing sleep music and sleepcasts.

  • An app intended to provide information and self-care advice to help manage the symptoms of lymphoedema.
Table 3 Examples of tier C technologies

Category within tier

Example technologies

Inform clinical management

Digital health technologies (DHTs) that record and calculate data and transmit the data to a professional, carer or third-party organisation, to inform clinical management decisions in the future. Information provided by the DHT will not trigger an immediate or near-term action.

  • An app intended for heart health monitoring that reports the results of heart rates measured by a smartphone's camera.

  • An app intended for diabetes management including tracking and diary keeping options.

Drive clinical management

Information provided by the DHT will be used to aid in treatment, aid in diagnoses, to triage or identify early signs of a disease or condition, or to guide next diagnostics or next treatment interventions.

  • Software intended to analyse data and estimate fractional flow reserve from coronary CT angiography.

  • A portable electrocardiogram recorder for detecting atrial fibrillation, compatible with an app intended to record and share the results with a healthcare professional.

Diagnose a condition

Information provided by the DHT will be used to take an immediate or near-term action to diagnose, screen or detect a disease or condition.

  • An artificial intelligence (AI) algorithm intended to fast-track the diagnosis of suspected lung cancer and reduce the radiology department's workload

  • An implantable cardiac monitor that can monitor heart rhythms for several years; algorithms on the device intended to detect potential atrial fibrillation and send the data for clinician review

Treat a condition

Information provided by the DHT will be used to take an immediate or near-term action to treat, prevent or mitigate by means of providing therapy to a human body.

  • An app intended to facilitate self-management for chronic obstructive pulmonary disease, which includes pulmonary rehabilitation.

  • A web-based programme intended for self-help sleep improvement based on cognitive behavioural therapy for insomnia.

Some DHTs might have intended purposes that fall into more than 1 tier, in which case, the highest tier should be used to define its tier. For example, a DHT might provide both a health diary and treatment for a condition. Of these 2 intended purposes, providing treatment is the higher risk. So, this technology would be classed as a tier C technology.

Some DHTs have various levels of intended purpose within the same tier depending on the level of service that is commissioned. For these technologies, the highest category that is being considered for commissioning should be used to define its classification. This is particularly relevant for tier C technologies for which evidence requirements differ between the 4 categories. For example, a DHT that enables data sharing for a specific condition with a healthcare professional for later review and that also provides treatment could be classed as tier C: inform clinical management or treat a condition, depending on what level of service is being commissioned.

Deciding which evidence standards are relevant

The ESF includes 21 standards that are arranged across 5 areas of a DHT's life cycle: design factors, describing value, demonstrating performance, delivering value and deployment considerations (for more detail, see the section on how the framework was developed).

Figure 1 The 5 groups of evidence standards relating to different aspects of the product life cycle

Most standards (17 out of 21) are tier agnostic, meaning that they are relevant to technologies in tiers A, B and C.

  • Design factors: The 9 standards cover general aspects of good design principles and identify key aspects of the design process that impact the DHT's value to the health and care system. Most of these areas are also in the remit of published technical standards, as referred to in the ESF standard 1, and might be within the remit of regulation of medical devices or in vitro diagnostics. The extent to which the 9 areas are covered in the different applications might vary. Of the 9 standards:

    • Standards 1 to 6 apply to all tiers.

    • Standards 7 to 9 apply to tiers B and C (these standards are less relevant to technologies in tier A). Standard 7 addresses items such as reliable health information, standard 8 addresses credibility with UK professionals, and standard 9 addresses safeguarding assurance.

  • Describing value: The 4 standards describe the system service, wellness, care or health problem to be addressed by the DHT. They provide key information about the target population and positioning of the technology in the health and care system. They also identify, value and qualitatively compare the costs and outcomes of the DHT to inform the value proposition of the DHT. For DHTs that are early in their life cycle, we acknowledge that numerical estimates for costs and outcomes or benefits might not be available, and so a qualitative description might be sufficient. The 4 standards provide important background information that clearly specifies the problem the DHTs are addressing and helps to inform the assessment of the evidence described in the demonstrating performance and delivering value standards. Companies might work with evaluators to refine the value proposition of the DHT. The 4 standards apply to all tiers.

  • Demonstrating performance: The 3 standards describe the evidence to establish the level of the DHT's performance. Proven clinical effectiveness is important for considering the potential patient and health benefits of DHTs that treat and diagnose or guide clinical care choices. Standard 14 is specific to tier C technologies to demonstrate clinical effectiveness. Likewise, proven real-world performance is important for considering the potential use of the DHTs in clinical practice. Monitoring performance ensures that the DHT continues to deliver its anticipated value after deployment and implementation. This is particularly important for data-driven technologies that may update frequently. Standards 15 and 16 are relevant to all technologies. Monitoring after deployment is likely to also be in the remit of the Medicines and Healthcare products Regulatory Agency (MHRA) regulations. The type of evidence presented to the different authorities might vary depending on the remit and process of the specific evaluation.

  • Delivering value: The 2 standards help to understand the affordability and value for money of the DHT. They measure and quantitively compare the costs and outcomes identified in the describe value standards (standards 10 to 13). All DHTs should present a budget impact analysis (BIA) so the evaluator can understand the potential impact of the DHT on their budget when adopting the DHT. A BIA only considers costs and benefits that are monetised; these costs and benefits can relate to cash (for example, consumables) or capacity (for example, hospital admissions). For DHTs with higher financial risk (that is, when the costs of commissioning, purchasing or implementing the DHT are deemed to be substantial within the context of relevant budget), a cost–utility analysis or cost–consequences analysis might be needed to understand the value for money of the DHT.

  • Deployment considerations: We acknowledge that deployment of DHTs might be more complex than deployment of pharmaceuticals and we identified 3 key deployment considerations. These considerations are important for successful deployment and implementation of a DHT but do not directly contribute to the value of a technology. The 3 standards apply across all tiers.

ESF standards that are especially relevant for data-driven DHTs

The ESF is also relevant for data-driven DHTs that have fixed or adaptive machine learning algorithms. These technologies might have increased risks not seen for other technologies; therefore, we developed some standards with this in mind. These standards are not exclusively for data-driven DHTs but include aspects that are more relevant to these technologies. These standards are:

  • Design factors, standards 4, 5 and 6:

    • Consider health and care inequalities and bias mitigation.

    • Embed good data practices in the design of the DHT.

    • Define the level of professional oversight.

  • Demonstrating performance, standards 15 and 16:

    • Show real-world evidence that the claimed benefits can be realised in practice.

    • The company and evaluator should agree a plan for measuring changes in the DHT's performance over time.

  • Deployment consideration, standards 20 and 21:

    • Ensure transparency about requirements for deployment.

    • Describe strategies for communication, consent and training processes to allow the DHT to be understood by end users.

ESF standards for early deployment DHTs

We acknowledge that some promising DHTs may not have all the evidence to support their value-for-money claim and that such evidence might best be generated in a real-world setting through an evidence-generation programme. The company and evaluator, in this case the potential early adopter, should assess the feasibility of the evidence generation within the timelines of the evidence-generation programme. The evidence-generation plan should be designed so that the generated evidence should support full deployment of the DHT following the evidence-generation programme. This means that DHTs entering the programme should have a certain maturity so that the evidence-generation plan can address gaps in the evidence and increase the likelihood of a successful completion of the evidence-generation programme. We have developed an early deployment subset of 16 standards which might help identify DHTs that can be considered for such programmes. Evidence‑generation programmes might be initiated by different authorities and therefore might differ in their requirements and length of programme. The early deployment subset contains:

  • Design factors: The 9 standards identify key aspects of the design process that impact the DHT's value to the health and care system, including ensuring the technology has the appropriate technical standards for safety and reliability. Standards 7 to 9 do not apply to tier A DHTs.

  • Describing value: The 4 standards apply across all tiers and present the value proposition of the technology.

  • Demonstrating performance: 1 standard that shows the company and the evaluator should reach an agreement for ongoing data collection.

  • Deployment considerations: 2 standards that describe the deployment considerations and the communication strategies that are in place for service users and health and care professionals.

Figure 2 The subset of standards relevant to digital health technologies (DHTs) that can be considered in an evidence-generation programme during early deployment

Example use cases

Evaluators who want to purchase or commission DHTs

Evaluators who are assessing DHTs to inform a purchasing or commissioning decision can use the NICE ESF as a standardised way of assessing the potential value of a DHT. This could provide a way for evaluations to be done consistently between DHTs and provide a structured format for information relevant to the evaluation.

The evaluator would work with the company to identify the intended purpose of the DHT and its appropriate evidence tier. The evaluator can then ask the company to provide any evidence needed to meet the relevant evidence standards.

For the standards involving ongoing evidence-generation plans, the company and evaluator would work together to agree what performance and usage data would be collected, and how and when this would be reported to the evaluator.

Evaluators in the health and care system who are appraising the value of a DHT

Evaluators may use the ESF to assess the suitability of a DHT for inclusion in innovation programmes, or to compare DHTs within the same clinical area.

The evaluator would identify the intended purpose of the DHT and its appropriate evidence tier, and would assess the breadth and relevance of the available evidence.

Companies who want to sell a DHT to the health or care system

Companies who have a DHT that they believe is ready for adoption in the health and care system at a local, regional or national level can use the ESF to show that they meet the evidence standards relevant to their DHT.

The company would work with the evaluator to identify the intended purpose of the DHT and its appropriate evidence tier. The company would provide any evidence needed to meet the relevant evidence standards. The NICE ESF describes the evidence that is likely to be needed to present to an evaluator for assessment.

Companies who want to generate evidence for their DHT within a health or care setting

Companies who have developed a DHT that is a prototype at planned operational level and is ready for demonstration in the health and care system, can use the ESF to understand the level of evidence they need to produce to aid commissioning at a later stage.

The company would work with the evaluator to identify the intended purpose of the DHT and its appropriate evidence tier. The company and evaluator would work together to agree what performance and usage data would be collected, and how and when this would be reported to the evaluator.

Companies who are at early stages of product development

Companies who are at the early stages of product development can use the NICE ESF as guidance to understand the level of evidence needed to be considered for an evidence-generation plan.

Examples of ESF use cases can be found here.


This page was last updated: