Developing NICE guidelines: the manual

Interventions funded by the NHS and PSS with health outcomes

For decision problems that evaluate an intervention solely commissioned by the NHS and do not have a clear focus on non-health outcomes, the reference case for 'interventions funded by the NHS and PSS with health outcomes' should be chosen.

All relevant NHS and PSS costs that change because of an intervention should be taken into account. Important non-NHS and PSS costs should also be identified and considered for inclusion in sensitivity analysis, or to aid decision making. These may include costs to other central government departments and local government. Service recommendations are likely to have additional costs, which include implementation costs not usually included in the analysis and costs to other government budgets, such as social care. Implementation costs should be included in a sensitivity analysis, where relevant, while costs to other government budgets can be presented in a separate analysis to the base case.

Biosimilar medicines are considered to differ from the original product in price only (in line with the NICE technology appraisal position statement on the use of biosimilars). It may be necessary to consider modelling scenarios for different levels of uptake for biosimilars, and whether there is any relevant Medicines and Healthcare products Regulatory Agency (MHRA) or government guidance on switching to biosimilars.

Productivity costs and costs borne by people using services and carers that are not reimbursed by the NHS or PSS should usually be excluded from any analyses. That is, a societal perspective will not normally be used.

More details on methods of economic evaluation for interventions with health outcomes in NHS and PSS settings can be found in NICE health technology evaluations: the manual (2022).This includes a reference case, which specifies the methods we consider to be the most appropriate for analysis when developing technology appraisal guidance. The reference case is consistent with the NHS objective of maximising health gain from limited resources.

Interventions funded by the public sector with health and non-health outcomes

For decision problems where the interventions evaluated are commissioned in full or in part by non‑NHS public sector and other bodies, the reference case for 'interventions funded by the public sector with health and non-health outcomes' should be chosen. For the base-case analysis, a cost-utility analysis should be done using a cost per QALY where possible.

This reference case may be most appropriate for public health interventions paid for by an arm of government, and would consider all the costs of implementing the intervention, and changes to downstream costs. In some cases, the downstream costs are negative, and refer to cost savings. For example, an intervention such as increasing physical activity, whose effects may include preventing type 2 diabetes, may be paid for by local government, but may result in cost savings to the NHS in the form of fewer or delayed cases of diabetes. A public sector cost perspective would aggregate all these costs and cost savings. A narrower local government cost perspective would consider only the cost of implementation, whereas an NHS cost perspective would consider only the cost savings. When examining interventions that are not paid for by an arm of government (such as workplace interventions), the perspective on costs should be discussed and agreed with staff who have responsibility for quality assurance.

Productivity costs should usually be excluded from both the reference-case and non-reference-case analyses; exceptions (for example, when evaluating interventions in the workplace) can only be made with the agreement of staff with responsibility for quality assurance.

For public health interventions, all direct health effects for people using services or, when relevant, other people such as family members or unpaid carers will be included. Non-health effects may also be included. When needed, the perspective will be widened to include sectors that do not bear the cost of an intervention, but receive some kind of benefit from it.

Interventions with a social care focus

For decision problems where the interventions evaluated have a clear focus on social care outcomes, the reference case on 'interventions with a social care focus' should be chosen. For the base-case analysis, a cost-utility analysis should be done using a cost per QALY approach where possible.

Public sector funding of social care for individual service users is subject to eligibility criteria based on a needs assessment and a financial assessment (means test). Therefore, users of social care may have to fund, or partly fund, their own care. A public sector perspective on costs should still be adopted, but should consider different scenarios of funding.

A public sector perspective is likely to be a local authority perspective for many social care interventions, but downstream costs that affect other public sector bodies may be considered where relevant, especially if they are a direct consequence of the primary aim of the intervention.

When individuals may pay a contribution towards their social care, 2 further perspectives may also be relevant:

The value of unpaid care may also be included in sensitivity analysis, or to aid decision making. The value of unpaid care should be set at the market value of paid care. Productivity costs should usually be excluded from both the reference-case and non-reference-case analyses; exceptions can only be made with the agreement of staff with responsibility for quality assurance.

For social care interventions, the usual perspective on outcomes will be all effects on people for whom services are delivered including, when relevant, family members or unpaid carers. When needed, the perspective may be widened to include sectors that do not bear the cost of an intervention, but receive some kind of benefit from it.

Other perspectives

Other perspectives (for example, employers) may also be used to capture significant costs and effects that are relevant to the interventions. Agree use of other perspectives with staff who have responsibility for quality assurance before use.

Searching for economic evidence

Advice on the approach to searching for economic evidence is detailed in the chapter on identifying the evidence: literature searching and evidence submission.

Selecting relevant economic evaluations

The process for sifting and selecting economic evaluations for assessment is essentially the same as for effectiveness studies (see the section on identifying and selecting relevant evidence in the chapter on reviewing evidence). It should be targeted to identify the papers that are most relevant to current UK practice and therefore likely to inform the committee's decision making.

Inclusion criteria for sifting and selecting papers for each review should specify populations and interventions relevant to the review question. They should also specify:

The review should also usually focus on economic evaluations that compare both the costs and consequences of the alternative interventions under consideration. Cost-utility, cost-benefit, cost-effectiveness, cost-minimisation or cost-consequences analyses (see box 7.1 in the section on economic analyses) can be considered depending on what the committee deems to be the most relevant perspective and likely outcomes for the question. Non-comparative costing studies, 'burden of disease' studies and 'cost of illness' studies should usually be excluded. However, non-comparative costing studies (such as econometric, efficiency, simulation, micro-costing and resource use, and time-series) may be included for some service delivery questions, or flagged if they might be useful for economic modelling. Sometimes, the published economic evidence is extremely sparse. In such cases, the inclusion criteria for studies may be broadened. The development team will decide to do this in consultation with staff who have responsibility for quality assurance and, when appropriate, with the committee or its chair.

Assessing the quality of economic evaluations

All economic evaluations relevant to the guideline should be appraised using the methodology checklists (see the appendix on appraisal checklists, evidence tables, GRADE and economic profiles). These should be used to appraise published economic evaluations, as well as unpublished papers, such as studies submitted by registered stakeholders and academic papers that are not yet published. The same criteria should be applied to any new economic evaluations done for the guideline (see the section on approaches to original economic evaluation).

Consideration and inclusion of economic evaluations will depend on the applicability of evidence to our decision-making context (usually the reference case), the amount of higher-quality evidence and the degree of certainty about the cost effectiveness of an intervention (when all the evidence is considered as a whole). Lower-quality studies are more likely to be excluded when cost effectiveness (or lack of it) can be reliably established without them. The reasons for such exclusions should be explained in the guideline's evidence review documents.

Sometimes reported sensitivity analyses show whether the results of an evaluation or study are robust despite methodological limitations. If there is no sensitivity analysis, judgement is needed to assess whether a limitation would be likely to change the results and conclusions. If necessary, a checklist, such as the health technology assessment checklist for decision-analytic models (Philips et al. 2004), may also be used to give a more detailed assessment of the methodological quality of economic evaluations and modelling studies. Judgements made, and reasons for these judgements, should be recorded in the guideline's evidence review documents.

Summarising and presenting results for economic evaluations

Cost-effectiveness or net benefit estimates from published or unpublished studies, or from original economic evaluations done for the guideline, should be presented in the guideline's evidence review documents, for example, using an 'economic evidence profile' (see the appendix on appraisal checklists, evidence tables, GRADE and economic profiles), as well as a statement on the cost effectiveness with respect to our decision threshold. This should include relevant economic information (applicability, limitations, costs, effects, cost effectiveness or net benefit estimates, as appropriate). Costs do not need to be adjusted to present value, but costs from other countries should be converted to pound sterling using an exchange rate from an appropriate and current source (such as HM Revenue and Customs or Organisation for Economic Co-operation and Development).

It should be explicitly stated if economic information is not available or if it is not thought to be relevant to the review question.

Separate health economic evidence statements are no longer required.

Economic plans

The full economic plan initially identifies key areas of the scope as priorities for further economic analysis and outlines proposed methods for addressing review questions about cost effectiveness. The full economic plan may be modified during development of the guideline; for example, as evidence is reviewed, it may become apparent that further economic evaluation is not needed or may not be possible for some areas that were initially prioritised. A version of the economic plan setting out the questions prioritised for further economic analysis, the population, the interventions and the type of economic analysis is published on the NICE guideline's webpage at least 6 weeks before the guideline goes out for consultation (see the chapter on developing review questions and planning the evidence review). The reasons for the final choice of priorities for economic analysis should be explained in the guideline's evidence review documents.

Discussing the economic plan with the committee early in guideline development is essential to ensure that:

The number and complexity of new analyses depends on the priority areas and the information needed for decision making by the committee. Selection of questions for further economic analysis should be based on systematic consideration of the potential value of economic analysis across all key issues.

Economic analysis is potentially useful for any question in which an intervention, service or programme is compared with another. It may also be appropriate in comparing different combinations or sequences of interventions, as well as individual components of the service or intervention. However, the broad scope of some guidelines means that it may not be practical to conduct original economic analysis for every component.

The decision about whether to carry out an economic analysis therefore depends on:

Economic modelling may not be warranted if:

General principles

Regardless of the methodological approach taken, the general principles described in this section should be observed. Any variation from these principles should be described and justified in the guideline's modelling report or evidence review documents. The decision problem should be clearly stated. This should include a definition and justification of the interventions or programmes being assessed and the relevant groups using services (including carers).

Developing conceptual models linked to topic areas or review questions may help the health economist to decide what key information is needed for developing effectiveness and cost-effectiveness analyses (see the chapter on the scope for details). Models developed for public health and service delivery topics are likely to relate to several review questions, so most recommendations will be underpinned by some form of modelled analysis.

The choice of model structure is a key aspect of the design-oriented conceptual model. Brennan's taxonomy of model structures (Brennan et al. 2006) should be considered for guidance on which types of models may be appropriate to the decision problem.

Even if a fully modelled analysis is not possible, there may be value in the process of model development, because this will help to structure committee discussions. For example, a model might be able to demonstrate how a change in service will affect demand for a downstream service or intervention.

For service delivery questions, the key challenge is linking changes in service to a health benefit. This obviously poses a challenge when doing health economic analyses, but it will also be difficult finding high-quality evidence of effectiveness. Modelling using scenario analysis is usually needed to generate the health effects used within the health economic analyses. Because of the considerable resource and health impact of any recommendations on service delivery, its cost effectiveness must be considered, either analytically or qualitatively (see the appendix on service delivery – developing review questions, evidence reviews and synthesis).

Economic analysis should include comparison of all relevant alternatives for specified groups of people affected by the intervention or using services. Any differences between the review questions and the economic analysis should be clearly acknowledged, justified, approved by the committee and explained in the guideline's modelling report or evidence review documents. The interventions or services included in the analysis should be described in enough detail to allow stakeholders to understand exactly what is being assessed. This is particularly important when calculating the cost effectiveness of services.

An economic analysis should be underpinned by the best-quality evidence. The evidence for treatment effectiveness should be based on and be consistent with that identified for the relevant effectiveness review question. However, sometimes models may differ from the guideline review, such as when additional outcomes or additional timepoints might be extracted from the effectiveness evidence, that were not prioritised in the effectiveness review.

Where there is not enough evidence in the effectiveness review, usually a research recommendation will be made and modelling will not go ahead. Occasionally, a model will be developed based upon indirect evidence, bespoke analysis of real-world data or expert opinion. This should be discussed with staff responsible for quality assurance and clearly explained and justified in the guideline.

The structure of any economic model should be discussed and agreed with the committee early in guideline development. The reasons for the structure of the model should be clear. Potential alternatives should be identified and considered for use in sensitivity analysis. If existing economic models are being used, or are informing a new analysis, particularly economic models used previously in the guideline's development, the way these models are adapted or used should be clear.

Clinical endpoints that reflect how a patient feels, functions, or how long a patient lives are considered more informative than surrogate outcomes. When using 'final' clinical endpoints in a model is not possible and data on other outcomes are used to infer the effect on mortality and health-related quality of life, there should be evidence supporting the outcome relationship (see NICE health technology evaluations: the manual [2022] for the 3 levels of evidence for surrogate relationships that can be considered in decision making). The uncertainty associated with the relationship between the surrogate endpoints and the final outcomes should be quantified and captured in the model's probabilistic analysis.

For evaluations of diagnostic testing, there may be some direct benefits from the knowledge gained and some direct harm from the testing, but most of the outcomes come downstream because of treatment or preventive measures being started, modified or stopped. Diagnostic tests can sometimes be evaluated using clinical trials, but this is unusual. If direct data on the impact of a diagnostic test strategy on final outcomes is not available, it will be necessary to combine evidence from different sources. A linked-evidence modelling approach should be used that captures the following components:

Clinical trial populations often differ from the population of interest and so might not capture the absolute effects of an intervention. Quantifying the baseline risk of health outcomes and how the condition would naturally progress with the comparator or comparators can be a useful step when estimating absolute health outcomes in the economic analysis. This can be informed by observational studies. Relative treatment effects seen in randomised trials may then be applied to data on the baseline risk of health outcomes for the populations or subgroups of interest. The methods used to identify and critically evaluate sources of data for these estimates should be reported.

When outcomes in an economic evaluation are known to be related, a joint synthesis of structurally related outcomes is recommended whenever possible, to account for correlation and to increase precision. Examples of these relations include:

Studies using survival outcomes, or time-to-event outcomes, often measure the relative effects of treatments using hazard ratios (HRs), which may either be constant over time (proportional hazards) or change over time. When incorporating such outcomes into a model, the proportional hazards assumption should always be assessed (see NICE's Decision Support Unit technical support document 14).

Sometimes there will be a lack of robust quantitative evidence for a key model parameter. In such situations, informal or formal expert opinion might sometimes be used to identify a plausible distribution of values.

For service delivery questions, any analysis will need to consider resource constraints. These might be monetary, but might also be resources such as staff, beds, equipment and so on. However, affordability should not be the sole consideration for service recommendations; the impact of any proposed changes on quality of care needs to be considered.

Before presenting final results to a committee for decision making, all economic evaluations should undergo rigorous quality assessment and validation to assess inputs, identify logical, mathematical and computational errors, and review the plausibility of outputs. The HM Treasury's review of quality assurance of government models (2013) provides guidance on developing the environment and processes required to promote effective quality assurance. This process should be recorded within the guideline's modelling report or evidence review documents.

Quality assurance of an economic evaluation may take various forms at different stages in development, as detailed in the HM Treasury Aqua Book (2015). It can range from basic steps that should always occur, such as disciplined version control, extensive developer testing of their own model, and independent testing by a colleague with the necessary technical knowledge, to external testing by an independent third party and independent analytical audit of all data and methods used. For developer health economists testing their own evaluation, or those of others ('model busting'), useful and practical validation methods include:

Results should be reported of any analyses done to demonstrate external validity. However, relevant data should not be omitted just to enable external validation (for example, not including trials so that they can be used for subsequent validation).

Conventions on reporting economic evaluations should be followed (see Husereau et al. 2022) to ensure that reporting of methods and results is transparent. For time horizons that extend beyond 10 years, it may be useful to report discounted costs and effects for the short (1 to 3 years) and medium (5 to 10 years) term. The following results should be presented where available and relevant:

When comparing multiple mutually exclusive options, an incremental approach should be adopted. This should be done by:

Comparisons with a common baseline intervention should not be used for decision making (although should be included in the incremental analysis if it reflects a relevant option).

Any comparison of interventions in an economic model that are not based on head-to-head trial comparisons should be carefully evaluated for the between-study heterogeneity, and potential for modifiers of treatment effect should be explored. Limitations should be noted and clearly discussed in the guideline's modelling report or evidence review documents. Ideally, when options have not been directly compared, and, more generally, when there are more than 2 options, a network meta-analysis should be considered as the best option for evidence synthesis to inform the model (see the chapter on reviewing evidence).

Economic models developed for the guideline are available to registered stakeholders during consultation on the guideline. These models should be fully executable and clearly presented.

Different approaches to economic analysis

There are different approaches to economic analysis (see box 7.1 in the section on economic analyses for examples). If economic analysis is needed, the most appropriate approach should be considered early during the development of a guideline, and reflect the content of the guideline scope.

There is often a trade-off between the range of new analyses that can be done and the complexity of each piece of analysis. Simple methods may be used if these can provide the committee with enough information on which to base a decision. For example, if an intervention is associated with better effectiveness and fewer adverse effects than its comparator, then an estimate of cost may be all that is needed. Or a simple decision tree may provide a sufficiently reliable estimate of cost effectiveness. In other situations, a more complex approach, such as Markov modelling or discrete event simulation, may be warranted.

The type of economic analysis that should be considered is informed by the economic perspective specified in the scope of the guideline, and the extent to which the effects resulting from the intervention extend beyond health.

Measuring and valuing effects for health interventions

The measurement of changes in health-related quality of life should be reported directly from people using services (or their carers). The value placed on health-related quality of life of people using services (or their carers) should be based on a valuation of public preferences obtained from a representative sample of the UK population, using a choice-based valuation method such as the time trade‑off or standard gamble. We prefer the QALY as the measure of health effects, and the EQ‑5D as the instrument to measure health-related quality of life in adults.

For some economic analyses, a flexible approach may be needed, reflecting the nature of effects delivered by different interventions or programmes. If health effects are relevant, the EQ‑5D‑based QALY should be used. When EQ‑5D data are not available from the relevant clinical studies included in the clinical evidence review, EQ‑5D data can be sourced from the literature. The methods used for identifying the data should be systematic and transparent. The justification for choosing a particular data set should be clearly explained. When more than 1 plausible set of EQ‑5D data is available, sensitivity analyses should be carried out to show the impact of the alternative utility values.

When EQ‑5D data are not available, published mapped EQ‑5D data should be used, or they may be estimated by mapping other health-related quality-of-life measures or health-related effects observed in the relevant studies to the EQ‑5D if data are available. The mapping function chosen should be based on data sets containing both health-related quality-of-life measures. The statistical properties of the mapping function should be fully described, its choice justified, and it should be adequately demonstrated how well the function fits the data. Sensitivity analyses exploring variation in the use of the mapping algorithms on the outputs should be presented.

In some circumstances, EQ‑5D data may not be the most appropriate or may not be available. Qualitative empirical evidence on the lack of content validity for the EQ‑5D should be provided, demonstrating that key dimensions of health are missing. This should be supported by evidence that shows that EQ‑5D performs poorly on tests of construct validity and responsiveness in a particular patient group. This evidence should be derived from a synthesis of peer-reviewed literature. In these circumstances, alternative health-related quality of life measures may be used and must be accompanied by a carefully detailed account of the methods used to generate the data, their validity, and how these methods affect the utility values.

When necessary, consideration should be given to alternative standardised and validated preference-based measures of health-related quality of life that have been designed specifically for use in children. The standard version of the EQ‑5D has not been designed for use in children. We do not recommend specific measures of health-related quality of life in children and young people. A generic measure that has shown good psychometric performance in the relevant age ranges should be used. Technical support document 8 by NICE's Decision Support Unit (Brazier et al. 2011) summarises the psychometric performance of several preference-based measures.

As outlined in NICE health technology evaluations: the manual (2022) and the accompanying NICE position statement on use of the EQ-5D-5-level (5L) valuation set for England (updated October 2019), we do not currently recommend using the EQ‑5D 5L valuation set. Guideline development teams should use the 3-level (3L) valuation set for reference-case analyses, when available.

The QALY remains the most suitable measure for assessing the impact of services, because it can incorporate effects from extension to life and experience of care. It can also include the trade‑offs of benefits and adverse events. However, if linking effects to a QALY gain is not possible, links to a clinically relevant or a related outcome should be considered. Outcomes should be optimised for the lowest resource use. The link (either direct or indirect) of any surrogate outcome, such as a process outcome (for example, bed days), to a clinical outcome needs to be justified. However, when QALYs are not used, issues such as trade‑offs between different beneficial and harmful effects need to be considered.

Measuring and valuing effects for non-health interventions

For some decision problems (such as for interventions with a social care focus), the intended outcomes of interventions are broader than improvements in health status. Here broader, preference-weighted measures of outcomes, based on specific instruments, may be more appropriate. For example, social care quality-of-life measures are being developed and we will consider using 'social care QALYs' if validated, such as the ASCOT (Adult Social Care Outcome Toolkit) set of instruments used by the Department of Health and Social Care in the Adult Social Care Outcomes Framework indicator on social care-related quality of life.

Similarly, depending on the topic, and on the intended effects of the interventions and programmes, the economic analysis may also consider effects in terms of capability and wellbeing. For capability effects, we may consider use of the ICECAP‑O (Investigating Choice Experiments for the Preferences of Older People CAPability measure for Older people) or ICECAP‑A (Investigating Choice Experiments for the Preferences of Older People CAPability measure for Adults) instruments when developing methodology in the future. If an intervention is associated with both health- and non-health-related effects, it may be helpful to present these elements separately.

Economic analysis for interventions funded by the NHS and PSS with health outcomes

Economic analyses done for decisions about interventions with health outcomes funded by the NHS and PSS should usually follow the reference case in table 7.1 in the section on the reference case. Advice on how to follow approaches described in NICE health technology evaluations: the manual (2022) is provided in the NICE Decision Support Unit's technical support documents. Departures from the reference case may sometimes be appropriate; for example, when there are not enough data to estimate QALYs gained. Any such departures must be agreed with members of staff with responsibility for quality assurance and highlighted in the guideline's modelling report or evidence review documents with reasons given.

Economic analysis for interventions funded by the public sector with health and non-health outcomes

The usual perspective for the economic analysis of public health interventions is that of the public sector. This may be simplified to a local government perspective if few costs and effects apply to other government agencies.

As public health and wellbeing programmes are funded by the public sector, in particular by local authorities, we have broadened our approach for the appraisal of interventions in these areas. Local government is responsible not only for the health of individuals and communities, but also for their overall welfare. The tools used for economic evaluation must reflect a wider remit than health and allow greater local variation in the outcomes captured. The nature of the evidence and that of the outcomes being measured may place more emphasis on cost-consequences analysis and cost–benefit analysis for interventions in these areas.

Whenever there are multiple outcomes, a cost-consequences analysis is usually needed, and the committee weighs up the changes to the various outcomes against the changes in costs in an open and transparent manner. However, for the base-case analysis, a cost-utility analysis should be undertaken using a cost per QALY approach where possible.

A wider perspective may be used, and will usually be carried out using cost–benefit analysis. When a wider perspective is used, it must be agreed with staff who have responsibility for quality assurance and highlighted in the guideline's modelling report or evidence review documents with reasons given.

Economic analysis for interventions with a social care focus

For social care interventions, the perspective on outcomes should be all effects on people for whom services are delivered (people using services or carers). Effects on people using services and carers (whether expressed in terms of health effects, social care quality of life, capability or wellbeing) are the intended outcomes of social care interventions and programmes. Although holistic effects on people using services, their families and carers may represent the ideal perspective on outcomes, a pragmatic and flexible approach is needed to address different perspectives, recognising that improved outcomes for people using services and carers may not always coincide.

Whenever there are multiple outcomes, a cost-consequences analysis is usually needed, and the committee weighs up the changes to the various outcomes against the changes in costs in an open and transparent manner. However, for the base-case analysis, a cost-utility analysis should be done using a cost per QALY approach where possible.

Any economic model should take account of the proportion of care that is publicly funded or self-funded. Scenario analysis may also be useful to take account of any known differences between local authorities in terms of how they apply eligibility criteria. Scenario analysis should also be considered if the cost of social care varies depending on whether it is paid for by local authorities or by individual service users; the value of unpaid care should also be taken into account where appropriate. A range of valuation methods exists to cost this type of care, so methods chosen should be clearly described and sensitivity analyses using other methods should be considered.

It is expected that the analytical difficulties involved in creating clear, transparent decision rules around the costs that should be considered, and for which interventions and outcomes, will be particularly problematic for social care. These should be discussed with the committee before any economic analysis is undertaken and an approach agreed.

Identification and selection of model inputs

An economic analysis uses decision-analytic techniques with outcome, cost and utility data from the best available published sources.

The reference case across all perspectives (see table 7.1 in the section on the reference case) states that evidence on effects should be obtained from a systematic review with a preference on randomised controlled trials. Some inputs, such as costs, may have standard sources that are appropriate, such as national list prices or a national audit, but for others appropriate data will need to be sourced.

Additional searches may be needed; for example, if searches for evidence on effects do not provide the information needed for economic modelling. Additional information may be needed on:

Although it is desirable to conduct systematic literature reviews for all such inputs, this is time-consuming and other pragmatic options for identifying inputs may be used. Informal searches should aim to satisfy the principle of 'saturation' (that is, to 'identify the breadth of information needs relevant to a model and sufficient information such that further efforts to identify more information would add nothing to the analysis' [Kaltenthaler et al. 2011]). Studies identified in the review of evidence on effects should be scrutinised for other relevant data, and attention should be paid to the sources of parameters in analyses included in the systematic review of published economic evaluations. Alternatives could include asking committee members and other experts for suitable evidence or eliciting their opinions, this could include for example, using formal consensus methods such as the Delphi method or the nominal-group technique. If a systematic review is not possible, transparent processes for identifying model inputs (such as effectiveness, adverse events, diagnostic accuracy, prognosis, quality of life, resource use and costs) should be reported; the internal quality and external validity of each potential data source should be assessed, and their selection justified. If more than 1 suitable source of evidence is found, consideration should be given to synthesis or exploration of alternative values in sensitivity analyses.

For some questions, there may be good reason to believe that relevant and useful information exists outside of literature databases or validated national data sources. Examples include ongoing research, a relatively new intervention and studies that have been published only as abstracts. Typically, the method for requesting information from stakeholders is through a call for evidence (see the appendix on call for evidence and expert witnesses).

For some guidelines, econometric studies provide a supplementary source of evidence and data for bespoke economic models. For these studies, the database 'Econlit' should be searched as a minimum.

Real-world data

'Real-word data' is defined in the NICE real-world evidence framework (2022) as 'data relating to patient health or experience, or care delivery collected outside the context of a highly controlled clinical trial'. Data from electronic health records, registries, audits and other sources of real-world data may be used to better define and inform parameter estimates for economic models. For some parameters, real-world data are considered the most appropriate evidence source (for example, jurisdiction-specific administrative databases can be used to estimate control or untreated event probabilities, resource use counts and unit costs), or such data may be more up to date and closer to reality (for example, in relationship to real-world costs and populations), so they should be the main source of such data in economic models. Real-world evidence should not be considered a substitute for published evidence from randomised controlled trials when assessing differences in outcomes between interventions. However, real-world evidence may be used where evidence from trials is absent or insufficient, or to answer different, but related, research questions alongside good published randomised controlled trial evidence (for example, effectiveness versus efficacy, effects in subgroups of interest, UK-specific head-to-head data and to derive final from surrogate trial outcomes).

To obtain real-world data, it may be necessary to negotiate access with the organisations and individuals that hold the data, or to ask them to provide a summary for inclusion in the guidance if published reports are insufficient. Any processes used for accessing data will need to be reported in the economic plan and in the guideline. Given the difficulties that organisations may have in extracting audit data, such requests should be focused and targeted: for example, identifying a specific audit and requesting results from the previous 3 years.

Additional guidance on the use of non-randomised studies and real-world data is provided by the NICE real-world evidence framework (2022), which:

Further guidance on searching and selecting evidence for key model inputs is also provided by Kaltenthaler et al. (2011) and Paisley (2016).

Cost data

Some information on unit costs may be found in the Personal Social Services - Research Unit (PSSRU) report on unit costs of health and social care or NHS England's National Cost Collection. Information on resource impact costings can be found in NICE's process guide on resource impact assessment. The NHS Supply Chain catalogue provides costs of clinical consumables, capital medical equipment and non-medical products. Some information about public services may be better obtained from national statistics or databases, rather than from published studies. Philips et al. (2004) provides a useful guide to searching for data for use in economic models.

In cases where current costs are not available, costs from previous years should be adjusted to present value using inflation indices appropriate to the cost perspective, such as the hospital and community health services index and the PSS pay and prices index, available from the PSSRU report on unit costs of health and social care (Jones 2021), or the Office for National Statistics (ONS) consumer price index.

Wherever possible, costs relevant to the healthcare system in England should be used. However, in cases where only costs from other countries are available these should be converted to Pounds Sterling using an exchange rate from an appropriate and current source (such as HM Revenue and Customs or Organisation for Economic Co-operation and Development).

Usually the public list prices for technologies (for example, medicines or medical devices) should be used in the reference-case analysis. However, as outlined in NICE health technology evaluations: the manual (2022), reference-case analyses should be based on prices that reflect as closely as possible the prices that are paid in the NHS for all technology evaluations. When there are nationally available price reductions (for example, for medicines procured for use in secondary care through contracts negotiated by the NHS Commercial Medicines Unit), the reduced price should be used in the reference-case analysis to best reflect the price relevant to the NHS. The Commercial Medicines Unit publishes information on the prices paid for some generic medicines by NHS trusts through its Electronic Market Information Tool (eMIT), focusing on medicines in the 'National Generics Programme Framework' for England. Analyses based on price reductions for the NHS will be considered only when the reduced prices are transparent and can be consistently available across the NHS, and when the period for which the specified price is available is guaranteed. When a reduced price is available through a patient access scheme that has been agreed with the Department of Health and Social Care, the analyses should include the costs associated with the scheme. If the price is not listed on eMIT, then the current price listed on the British National Formulary (BNF) should be used. For medicines that are predominantly dispensed in the community, prices should be based on the Drug Tariff. In the absence of a published list price and a price agreed by a national institution (as may be the case for some devices), an alternative price may be considered, provided that it is nationally and publicly available. If no other information is available on costs, local costs obtained from the committee may be used.

Quality of life data

Preference-based quality-of-life data are often needed for economic models. Many of the search filters available are highly sensitive and so, although they identify relevant studies, they also detect a large amount of irrelevant data. An initial broad literature search for quality of life data may be a good option, but the amount of information identified may be unmanageable (depending on the key issue being addressed). It may be more appropriate and manageable to incorporate a quality of life search filter when performing additional searches for key issues of high economic priority. When searching bibliographic databases for health-state utility values, specific techniques outlined in Ara (2017), Golder et al. (2005) and Papaioannou et al. (2010) may be useful, and specific search filters have been developed that may increase sensitivity (Arber et al. 2017). The provision of quality of life data should be guided by the health economist at an early stage during guideline development so that the information specialist can adopt an appropriate strategy. Resources for identifying useful utility data for economic modelling are the dedicated registries of health-state utility values such as ScHARRHUD and Tufts CEA Registry and the NICE Decision Support Unit's technical support documents.

Exploring uncertainty

The committee should discuss any potential bias, uncertainties and limitations of economic models. Sensitivity analysis should be used to explore the impact that potential sources of bias and uncertainty could have on model results.

Deterministic sensitivity analysis should be used to explore key assumptions and input parameters used in the modelling, as well as to test any bias resulting from the data sources selected for key model inputs. This should test whether and how the model results change under alternative, plausible scenarios. Threshold analysis should also be considered, particularly for highly uncertain parameters, to explore the impact of the parameter on the incremental cost-effectiveness ratio (ICER) or net-health benefit.

'Tornado' histograms may be a useful way to present deterministic results. Deterministic threshold analysis might inform decision making when there are influential but highly uncertain parameters. However, if the model is non-linear, deterministic analysis will be less appropriate for decision making.

Probabilistic sensitivity analysis should be used to account for uncertainty arising from imprecision in model inputs. The use of probabilistic sensitivity analysis will often be specified in the health economic plan. Any uncertainty associated with all inputs can be simultaneously reflected in the results, so the preferred cost-effectiveness estimates should be those derived from probabilistic analyses when possible. In non-linear decision models where outputs are a result of a multiplicative function (for example, in Markov models), probabilistic methods also provide the best estimates of mean costs and outcomes. The choice of distributions used should be justified. When doing a probabilistic analysis, use enough model simulations to minimise the effect of Monte Carlo error. Reviewing the variance around probabilistic model outputs (net health benefits or ICERs) as the number of simulations increases can provide a way of assessing whether the model has been run enough times or more runs are needed. Presentation of the results of probabilistic sensitivity analysis could include scatter plots or confidence ellipses, with an option for including cost-effectiveness acceptability curves and frontiers.

When probabilistic methods are unsuitable, or not possible, the impact of parameter uncertainty should be thoroughly explored using deterministic sensitivity analysis, being mindful of correlated parameters. The decision not to use probabilistic methods should be justified in the guideline's modelling report or evidence review documents.

Consideration can be given to including structural assumptions and the inclusion or exclusion of data sources in probabilistic sensitivity analysis. In this case, the method used to select the distribution should be outlined in the guideline (Jackson et al. 2011).

Discounting

Cost-effectiveness results should reflect the present value of the stream of costs and benefits accruing over the time horizon of the analysis. For the reference case, the same annual discount rate should be used for both costs and benefits. We consider that it is usually appropriate to discount costs and health effects at the same annual rate of 3.5%.

Sensitivity analyses using 1.5% as an alternative rate for both costs and health effects may be presented alongside the reference‑case analysis, particularly for public health guidance. When treatment restores people who would otherwise die or have a very severely impaired life to full or near full health, and when this is sustained over a very long period (normally at least 30 years), cost-effectiveness analyses are very sensitive to the discount rate used. In this circumstance, analyses that use a non-reference-case discount rate for costs and outcomes may be considered. A discount rate of 1.5% for costs and benefits may be considered by the committee if it is highly likely that, based on the evidence presented, long-term health benefits are likely to be achieved. However, the committee will need to be satisfied that the recommendation does not commit the funder to significant irrecoverable costs.

Subgroup analysis

The relevance of subgroup analysis to decision making should be discussed with the committee. When appropriate, economic analyses should estimate the cost effectiveness of an intervention in each subgroup.

Local considerations

For service delivery questions, cost-effectiveness analyses may need to account for local factors, such as the expected number of procedures and the availability of staff and equipment at different times of the day, week and year. Service delivery models may need to incorporate the fact that each local provider may be starting from a different baseline of identified factors (for example, the number of consultants available at weekends). It is therefore important that these factors are identified and considered by the committee. Where possible, results obtained from the analysis should include both the national average and identified local scenarios to ensure that service delivery recommendations are robust to local variation.

Service failures

Service designs under consideration might result in occasional service failure – that is, where the service does not operate as planned. For example, a service for treating myocardial infarction may have fewer places where people can be treated at weekends compared with weekdays because of reduced staffing. Therefore, more people will need to travel further away by ambulance and the journey time will also be longer. Given the limited number of ambulances, a small proportion may be delayed, resulting in consequences in terms of costs and QALYs. Such possible service failures should be taken into account in effectiveness and economic modelling. This effectively means that analyses should incorporate the 'side effects' of service designs.

Service demand

Introducing a new service or increasing capacity will often result in an increase in demand. This could mean that a service does not achieve the predicted effectiveness because there is more demand than was planned for. This should be addressed either in the analysis or in considerations.