Corporate document

How to meet the standards

How to meet the standards

Design factors

Standard 1: the digital health technology (DHT) should comply with relevant safety and quality standards

Applies to DHTs in tiers A, B and C.

Information that can be used to meet standard 1

Companies should demonstrate that all safety and quality standards relevant to their DHT have been met.

Examples of standards that may apply to different DHTs include:

Standard 2: incorporate intended user group acceptability in the design of the DHT

Applies to DHTs in tiers A, B and C.

Information that can be used to meet standard 2

Describe how representatives from intended user groups were involved in the design, development or testing of the DHT. Depending on who is intended to operate the DHT, the intended users may include patient groups and service users, or health and care professionals. Describe how user acceptability was appraised and provide any available data to show user acceptability with the DHT.

Section D1 of the NHS DTAC also refers to usability and accessibility. DHTs that have evidence of meeting ISO's standard IEC 62366-1, application of usability engineering to medical devices, may be considered to have already demonstrated compliance with this standard.

Standard 3: consider environmental sustainability

Applies to DHTs in tiers A, B and C.

Information that can be used to meet standard 3

The NHS has set ambitions to have a net zero carbon footprint by 2040. Environmental sustainability should be factored into all stages of the life cycle of the DHT starting at the design of the DHT. The company should provide a narrative description of any expected environmental sustainability benefits and negative impacts from using the DHT. This should focus on impacts on greenhouse gas emissions, in line with the NHS carbon footprint and carbon footprint plus.

Standard 4: consider health and care inequalities and bias mitigation

Applies to DHTs in tiers A, B and C.

Information that can be used to meet standard 4

Health inequalities considerations should be factored into the design of the DHT. Describe how this has been approached and how this has been included in the design of the DHT. Describe any specific positive impacts and any efforts to reduce negative impacts on health inequalities.

If the DHT has a claim of addressing a health or care inequality, it should also show evidence that the DHT contributes to:

  • challenging health inequalities in the UK health and social care system, or improving access to care among hard-to-reach populations

  • promoting equality, eliminating unlawful discrimination and fostering good relations between people with protected characteristics (as described in the Equalities Act 2010) and others.

For early deployment (ED) DHTs being used in evidence-generation programmes, plans for collecting evidence to support the health inequalities claims should be provided.

NHS Digital's guide on digital inclusion for health and social care provides information for companies and providers to understand digital inclusion and steps that can be taken to evaluate and support digital inclusion. The Open Data Institute's Data Ethics Canvas is a tool that can be used to identify and manage ethical issues in projects that use data. This includes exploring the impacts of the project on different people and identifying sources of bias.

For data-driven DHTs (including those with artificial intelligence), the company should describe any actions taken in the design of the DHT to mitigate against algorithmic bias that could lead to unequal impacts between different groups of service users or people.

Standard 5: embed good data practices in the design of the DHT

Applies to DHTs in tiers A, B and C.

Information that can be used to meet standard 5

Good data practices are essential to creating high-quality data-driven DHTs. Any datasets used to train, validate or develop the DHT should be of a high quality. One indicator of quality is that the following information can be provided by the company:

  • which datasets (title, source, version) were used for training and validating the DHT

  • the size of the training and validation datasets

  • how the data was labelled and 'ground-truth' established

  • why these datasets were collected, and by what means (manual input, through monitors or other devices)

  • diversity (demographics, age, clinically relevant subgroups) in these datasets used and how this reflects the intended target population for the DHT

  • any synthetic training or validation data should be highlighted; synthetic data should be supported by real data on how any decision thresholds have been set and how these align to current care.

For DHTs that incorporate machine learning, companies should follow the MHRA guiding principles on good machine learning practice for medical device development.

Standard 6: define the level of professional oversight

Applies to DHTs in tiers A, B and C.

Information that can be used to meet standard 6

During the design of the DHT, the company should define the anticipated level of professional oversight needed when the DHT is used in practice. This must be clearly described. The level of professional oversight should be acceptable to relevant health and care professionals. Professional oversight may include (among others):

  • expert review of each decision or output on a case-by-case basis

  • periodic overarching review of the trends in the decision outputs of the DHT, to ensure that the decisions are aligned to, or calibrated against, best practice

  • monitoring for occasions where the DHT's decision output has been overridden by professionals.

The level of professional oversight should be proportionate to the level of risk associated with failure of the DHT to perform as expected. Higher levels of professional oversight may be needed when the consequences of the DHT's failure are serious or critical.

Standard 7: show processes for creating reliable health information

Applies to DHTs in tiers B and C.

Information that can be used to meet standard 7

Health information is text, video or other educational material for people, patients or healthcare professionals, to help them to better understand their health and care. This could include information about conditions, tests or treatments. The developer should be able to show that processes are in place to maintain any health information provided by the DHT, which are:

  • valid (aligned to best available sources, such as NICE guidance, relevant professional organisations or recognised UK patient organisations, and appropriate for the target population)

  • accurate

  • reviewed and updated by relevant experts (such as health and care professionals in the relevant field) at defined intervals, such as every year

  • sufficiently comprehensive.

Standard 8: show that the DHT is credible with UK professionals

Applies to DHTs in tiers B and C.

Information that can be used to meet standard 8

Show that relevant health or care professional(s) working in the UK health and social care system have either been involved in designing, developing or testing the DHT, or given their support to the UK deployment of the DHT.

Be able to show that the DHT is viewed as useful and relevant by professional experts or expert groups in the relevant field. This could include providing evidence to support key factors such as the choice of behaviour change techniques used in the DHT.

Standard 9: provide safeguarding assurances for DHTs where users are considered to be in vulnerable groups, or where peer-to-peer interaction is enabled

Only applies to DHTs in tiers B and C, if the DHT is intended to be used by people in vulnerable groups, such as children or at-risk adults, or where peer-to-peer interaction is enabled through the DHT. NHS England defines an at-risk adult as an adult 'who may be in need of community care services by reason of mental or other disability, age or illness; and who is or may be unable to take care of him or herself, or unable to protect him or herself against significant harm or exploitation'.

Information that can be used to meet standard 9

Show that appropriate safeguarding measures are in place around peer support and other communication functions enabled through the DHT:

  • Describe who has access to the platform and their roles within the platform.

  • Describe why these people or groups are suitable and qualified to have access.

  • Describe any measures in place to ensure safety for vulnerable users and in peer‑to‑peer communication, for example, through user agreements or moderation.

Describing value

Standard 10: describe the intended purpose and target population

Applies to DHTs in tiers A, B and C.

Describe the target population and intended purpose for the DHT. Include any inclusion and exclusion criteria that apply.

Describe the expected uptake profile of the DHT.

Information that can be used to meet standard 10

The target population for tier C DHTs is likely to be defined by a particular health condition and position in the care pathway. For regulated medical devices, the intended population for the DHT will be defined in the information associated with the device, and the target population can match this. For tier B DHTs, the target population is the group of people who are intended to benefit from using the DHT. For tier A DHTs, the target population is the group that benefits from using the DHT; which could be service users, clinical staff or administrative staff. Any important subgroups should also be identified.

The size of the target population should be calculated using appropriate and current national or local sources (for example, accurate epidemiological data of prevalence and incidence of the relevant health problem), or expert estimates if this is not available. Note that NICE's resource impact assessment manual describes an approach to calculating population size.

The expected uptake profile describes the proportion of people within the target population who are expected to use the DHT, and their usage rates. This may be impacted by digital literacy within the intended user population, availability of necessary connectivity, and access to necessary hardware or devices.

Demonstrate that the expected uptake profile is:

  • calculated using uptake rates from pilot data or other usage data from the company

  • validated as an accurate representation of what is expected (including any variations by subgroup and over time) by showing agreement and support from relevant professionals in the UK health and social care system

  • mindful of subgroups with different expected uptake rates and how these may change over time.

Standard 11: describe the current pathway or system process

Applies to DHTs in tiers A, B and C.

Information that can be used to meet standard 11

Use national clinical guidelines, national guidance or academic literature and consultation with healthcare professionals and service users to map out the existing care pathway(s) or system processes for the intended purpose and target population (as described in standard 10).

Use a comprehensive, detailed and stepwise approach (for example, using a flow chart).

The representation of current care or system processes should be comprehensive and relevant to the intended purpose and the target population, and should be checked and validated by relevant professionals in the UK health and social care system.

If there is no existing care pathway or system process, the impact of adopting the technology should be clearly specified using an approach that can be used as a basis for a potential economic evaluation.

If there is more than 1 existing care pathway or system process, describe each of them.

Standard 12: describe the proposed pathway or system process using the DHT

Applies to DHTs in tiers A, B and C.

Information that can be used to meet standard 12

Provide details of how the proposed care pathway or system process using the DHT will be different to the current pathway or system process described for standard 11, including:

  • whether using the DHT would replace an existing technology or step in current care, would complement current care or whether it would be in addition to current care

  • any changes that would need to be made to infrastructure, service provision and workforce, compared with current care or process

  • whether the proposed pathway crosses between existing system care boundaries, such as between primary and secondary care

  • changes needed to implement, operate and maintain the proposed pathway or process using the DHT

  • need for training and education for health and care professionals or end users, in order to effectively implement and use the DHT

  • any influential contextual issues that may act as barriers for enablers to implementation.

Standard 13: describe the expected health, cost and resource impacts compared with current care or system processes

Applies to DHTs in tiers A, B and C. The information used to meet this standard may be used to inform the budget impact analysis in standard 17, and health economic evaluation in standard 18.

Information that can be used to meet standard 13

To assess impact of the DHT, we need to understand and compare the health and system benefits from the current pathway or process and the proposed pathway or process using the DHT:

  • Describe the health benefits and other outcomes (such as system efficiency, care outcomes, or structural and procedural effects) associated with current practice. If possible, quantify the uncertainty associated with these figures (for example, with confidence intervals or probability distribution).

  • Describe the anticipated health benefits and other outcomes (such as system efficiency, care outcomes, or structural and procedural effects) associated with using the DHT. If possible, quantify the uncertainty associated with these figures (for example, with confidence intervals or probability distribution).

Structural and procedural effects could include access to care, health literacy, adherence to care plans, or coordination of care.

Also, it is important to understand whether there are any expected additional costs or cost savings as well as resource impact from the DHT compared with current practice:

  • Describe the costs and resource use associated with current practice. If possible, quantify the uncertainty associated with these figures (for example, with confidence intervals or probability distribution).

  • Describe the expected costs and resource use associated with using the DHT. If possible, quantify the uncertainty associated with these figures (for example, with confidence intervals or probability distribution).

The sources for this information should:

  • be from the most robust evidence available, for example from clinical studies on the DHT and on current care options (if available), real-world evidence, observational studies or from expert opinion

  • be the same as referred to in the performance and effectiveness standards (if these apply).

Evidence synthesis can be used if there are several studies, in which case a sensitivity analysis should be shown.

Estimates should have minimal bias and all uncertainties should be accurately characterised.

Demonstrating performance

Standard 14: provide evidence of the DHT's effectiveness to support its claimed benefits

Applies to DHTs in tier C.

Information that can be used to meet standard 14

For tier C DHTs, evidence for effectiveness must be shown.

The evidence should show that using the DHT impacts on clinical management of the relevant condition, in a setting relevant to the UK health and social care system or that it provides reliable test results that can be used to impact clinical management. Outcomes relevant to the intended purpose (value proposition) and claimed benefits of the DHT should be captured.

Choice of study design should be guided by the intended purpose (value proposition) and claimed benefits of the DHT, and comparative studies are generally more informative than non-comparative studies. Some general guidance on assessing the quality of evidence includes:

  • The results of studies done in a setting that is similar to the UK health and care system (such as where the care pathway is similar, patients have similar care options, or similar kinds of staff are involved in care) are more easily generalisable to the UK system than those of studies done in settings that are very different to the UK system.

  • Prospective studies are often considered to be more valuable than retrospective studies because they can be designed to capture the most relevant outcomes, and have lower risk of bias in terms of who is included in the studies.

  • Studies that are published in peer-reviewed journals have usually had some independent assessment of their quality before publication.

  • The studies used to support claimed benefits should be done on the DHT in question. If supporting evidence from similar DHTs is presented, then a clear rationale should be provided explaining how that evidence is generalisable.

  • There are different ways to appraise the quality of research studies. Appendix H of NICE's developing NICE guidelines: the manual provides a comprehensive list of checklists that can be used to assess risk of bias or quality of different study types.

  • NICE has published a framework on best practice in developing real-world evidence, which describes how real-world evidence should be assessed.

Qualitative studies can be useful for showing patient and healthcare professional views and experience of using the DHT.

If the DHT is intended to be used for situations or conditions in which accurate, timely diagnosis or treatment action is vital to avoid death, long-term disability or other serious deterioration of health, the increased risk associated with the situation or condition is likely to mean that less uncertainty in the performance of the DHT is acceptable.

Additional evidence could include any well-designed studies or data in addition to those mentioned below for the 4 different functional groups in tier C. This could be real-world data, prospective or retrospective studies that can serve to reduce uncertainty about the performance of the DHT.

Effectiveness of DHTs that inform clinical management

Evidence to support the claimed benefits of the DHT can include real-world evaluations of its clinical utility, or test accuracy studies for diagnostics.

Effectiveness of DHTs that drive clinical management

The evidence should include 1 or more high-quality studies to support the claimed benefits of the DHT, done in a setting relevant to the UK health and social care system, and show improvements in relevant outcomes, such as:

  • clinically relevant outcomes

  • patient-relevant outcomes

  • test accuracy

  • time to diagnosis.

This could include any of the following study designs:

  • test accuracy studies, using an appropriate reference standard

  • a concordance study (to show agreement with currently used tests)

  • interventional studies

  • prospective observational studies (including real-world evidence).

For DHTs that drive a diagnosis, test accuracy alone does not demonstrate clinical utility and may need to be linked to existing studies (including studies on other technologies) reporting the downstream clinical consequences of the diagnosis or test outcome.

Effectiveness of DHTs that treat a specific condition

One or more high-quality interventional studies (experimental or quasi-experimental design) to support the claimed benefits of the DHT, done in a setting relevant to the UK health and social care system and showing improvements in relevant outcomes, such as:

  • clinically relevant outcomes

  • patient-relevant outcomes.

The choice of study design should be appropriate for the intended purpose of the DHT. Randomised controlled trials would be preferrable where this study design is appropriate. High quality, comparative real-world study designs may also be acceptable.

The comparator should be a care option that reflects the current NHS care pathway, such as a commonly used active intervention.

User acceptability and engagement measures may also be useful.

Performance of DHTs that diagnose a specific condition

One or more high-quality studies done using the DHT, to support the claimed benefits of the test. This may include test accuracy studies, using an appropriate reference standard, or a concordance study to show agreement with current practice. Relevant outcomes may include:

  • test accuracy

  • time to diagnosis (if this is a claimed benefit of the test).

Test accuracy alone does not demonstrate clinical utility and may need to be linked to existing studies (including studies on other technologies) reporting the downstream clinical consequences of the diagnosis or test outcome.

Standard 15: show real-world evidence that the claimed benefits can be realised in practice

Applies to DHTs in tiers A, B and C.

Information that can be used to meet standard 15

Evidence to show that the DHT has been successfully piloted in the UK health and social care system, showing that it is relevant to:

  • current service provision in the UK (for tier A DHTs) or

  • current best practice in the UK (for tier B and tier C DHTs).

This may include a statement from pilot site(s) to confirm that during testing, the DHT:

  • was acceptable to users

  • performed its intended purpose to the expected level

  • successfully integrated into current service provision or current best practice

  • caused no unintended negative impacts on service users or services

  • showed improvements in outcomes (costs saved, efficiencies achieved, health and care improvements)

  • was used in line with expectation (who, how, for how long).

For tier C DHTs intended to be used for critical conditions or situations, this real‑world evidence can be used to reduce uncertainties in any interventional studies that have been done to meet standard 14.

For DHTs that are expected to have high costs or system impact (such as requiring significant service redesign), then higher levels of real-world evidence may help to reduce uncertainty. This could include larger-scale studies or longer-term outcomes.

For DHTs whose performance may be affected by local deployment factors (such as DHTs using artificial intelligence), this may include deploying the DHT to run offline or evaluating it 'in silent mode'.

Silent mode evaluations allow the DHT's performance on local data inputs to be observed (but not used in care decisions), before the DHT is integrated into clinical or care pathways. This can show whether the DHT's performance reaches the expected levels using input data generated in the local environment.

Standard 16: the company and evaluator should agree a plan for measuring usage and changes in the DHT's performance over time

Applies to DHTs in tiers A, B and C.

Information that can be used to meet standard 16

The company and evaluator should agree a plan for ongoing data collection to report:

  • ongoing usage of the DHT in the target population in line with the expected usage profile (to be agreed at commissioning)

  • anonymised, aggregate data to show service user outcomes or any other outcomes collected by the DHT (where appropriate).

In addition, for DHTs whose performance is expected to change over time (such as DHTs that use artificial intelligence or machine-learning algorithms, or DHTs that are expected to be updated in subsequent versions), the company and evaluator should agree on post-deployment reporting of changes in performance. This may include:

  • future plans for updating the DHT, including how regularly the algorithms are expected to retrain, re-version or change functionality

  • the sources of retraining data, and how the quality of this data will be assessed

  • processes in place for measuring performance over time, to detect any impacts of planned changes or environmental factors that may impact performance

  • processes in place to detect decreasing performance in certain groups of people over time

  • whether there is an independent overview process for reviewing changes in performance

  • an agreement on how and when changes in performance should be reported and to whom (evaluators, patients, carers, health and care professionals).

If the intended purpose of the DHT changes, or if additional functions are added that change the intended purpose and ESF classification of the DHT, then a new evaluation should be done. For DHTs that are medical devices, changes to the intended purpose would require regulatory re-evaluation and may result in reclassification as a medical device, and additional evidence to support the change.

Delivering value

Standard 17: provide a budget impact analysis

Applies to DHTs in tiers A, B and C.

Information that can be used to meet standard 17

Provide a budget impact analysis relevant to the setting the DHT is used in. This can be done using information about the value proposition given in response to standards 10 to 13, and the outcomes from studies shown in standard 14, or the real-world evidence in standard 15.

For tier B and C DHTs, the budget impact analysis should include:

  • size of target population and uptake estimates

  • all direct costs associated with the technology and implementing the technology, including cost of the technology (purchasing, updating, maintenance), costs of staffing and training, costs of supportive IT infrastructure needed to implement the technology

  • all direct costs associated with the comparator

  • relevant indirect costs associated with the technology and the comparator, reference test or current practice.

For tier A DHTs, a simpler analysis may be more appropriate. This would include a comparison of direct and indirect costs and resource impacts between the DHT and current practice.

Estimates of resource use should include:

  • length of hospital or care home stay

  • number of hospitalisations

  • outpatient or primary care consultations

  • changes in infrastructure, use and maintenance.

Show that the costs used are relevant to the UK health and care system and they should relate to NHS and personal social services resources. Suitable sources include:

Show that the estimates for resource use are based on clinical practice, which can be based on data from:

  • a clinical study

  • real-world data including from pilot studies

  • information obtained from relevant clinical or social care professionals

  • other appropriate sources.

State the source of the data for the cost and resource estimates. State whether the estimates are recognised as accurate and comprehensive by a relevant health and social care professional. Include any expected variations for different groups of service users.

Explore the uncertainty of the estimate obtained from the budget impact analysis by varying the assumptions used (for example, using best- and worst-case values for target population size, resource use or outcomes from the performance studies in standard 13), to investigate how these variations impact the analysis.

Standard 18: for DHTs with higher financial risk, provide a cost-effectiveness analysis

Applies to DHTs in tiers A, B and C.

Information that can be used to meet standard 18

We define a DHT with higher financial risk as: where the costs of commissioning, purchasing or implementing the DHT are deemed to be substantial within the context of the relevant budget and system priorities. This will vary between different commissioning organisations, and contributing factors may include:

  • coverage of commissioning of the technology, for example for large-scale or national adoption of the DHT

  • the extent of changes needed within an organisation to use the DHT; this could include changes to IT systems, staffing or care pathways

  • the extent of implementation costs needed to use the DHT.

When needed, a cost-effectiveness analysis in the form of cost–utility or cost‑consequences analysis should be done to inform the budget impact analysis in standard 17. A cost-effectiveness analysis can be done if a DHT:

  • provides similar or greater benefits at higher cost

  • provides marginally lower benefits for significantly lower costs.

Cost–utility analysis (described in section 4.2.14 of the NICE health technology evaluations manual) uses utility as a common outcome. It considers people's quality of life and the length of life they will gain as a result of an intervention or a programme. An appropriate standard measure should be used for utility data (such as EQ-5D).

If a cost–utility analysis is not possible (for example, when outcomes cannot be expressed using a utility measure such as the QALY), a cost–consequences analysis may be considered (see developing NICE guidelines: the manual). A cost‑consequences analysis can consider all the relevant health and non-health effects of a DHT across different settings and reports them without aggregation. The UK Health Security agency has published advice on using cost–consequences analysis for DHTs. NICE uses cost-comparison analysis to develop guidance on DHTs that are likely to be cost saving. This is a similar approach to cost‑consequences analyses.

The NICE health technology evaluations: the manual gives further guidance on how different health economic analyses can be done.

For all analyses, explore the uncertainty of the obtained estimate by using sensitivity and scenario analyses.

Deployment considerations

Standard 19: ensure transparency about requirements for deployment

Applies to DHTs in tiers A, B and C.

Information that can be used to meet standard 19

The company should provide clear descriptions of the data used in deployment. This should include:

  • a full description of the input data for the DHT that includes a data dictionary

  • quantifying the level of tolerance that the DHT has for incomplete data (such as inputs that are missing or of insufficient quality), and how outlier data is handled

  • a data flow map for deployment of the DHT to allow efficient deployment

  • data requirements for the DHT, such as specific data formats, data standardisation requirements (such as DICOM), completeness or quality

  • the minimum infrastructure requirements for deploying the DHT.

Standard 21: ensure appropriate scalability

Applies to DHTs in tiers A, B and C.

Information that can be used to meet standard 21

The company should ensure that load testing has been done, to show that the DHT can perform to the scale needed (for example, having servers that can scale to manage the expected number of service users).

Describe the process for load testing and how this relates to the expected uptake for the DHT.