Methods for the development of NICE public health guidance (third edition)

6.2.1 Critical appraisal of economic studies

The search for economic studies should have identified papers that are most relevant to current public sector practice in the area under consideration and hence, likely to inform the PHACs' decision-making. It should have focused on 'full' economic evaluations that compare both the costs and the health consequences of the alternative options under consideration.

The process for sifting and selecting economic studies for critical appraisal is essentially the same as for effectiveness studies (see chapters 4 and 5). The Centre for Public Health Excellence (CPHE) project team and the review team should discuss and agree the inclusion criteria. These should include the populations and interventions that relate to the review question and should also specify:

6.2.2 Assessing economic evaluation studies

Estimates of resource use obtained from public health studies should be reviewed using the studies identified from the process described in section 6.2.1. Reservations about the applicability of these estimates to routine public sector practice should be noted in the evidence tables. The PHACs should take these reservations into account.

The criteria for appraising costs, cost-effectiveness ratios and net benefits are different because these estimates are usually obtained using some form of modelling. Modelling can include:

The review team should appraise all the economic evaluation studies (both published and unpublished) used to inform public health recommendations, using the economic evaluations checklist (see appendix I). Unpublished economic evaluations may include studies submitted by stakeholders or pre-publication academic papers (see sections 4.4.4, 4.4.5 and 5.3.2.3). The same criteria should be applied to any additional economic evaluations conducted for the guidance.

Given the small number of economic evaluations usually obtained through the literature searches, title and abstract screening, full-paper screening and quality appraisal should normally be undertaken by 2 reviewers. Any disagreement between the 2 reviewers, at any stage, should be resolved by discussion or recourse to a third reviewer.

The checklist includes a section on the methodological quality of the study (that is, the extent to which it fulfils its stated objectives), a section on its applicability to the review question and a section on the NICE health economics methodology used for decision-making. It can be used to judge overall methodological quality as follows:

The robustness of the study results (that is, its methodological limitations) may sometimes be apparent from reported sensitivity analyses. If not, judgement will be needed to assess whether a limitation is likely to change the results and conclusions.

The checklist can also be used to judge the overall applicability of the study in the context of the guidance:

If necessary, the health technology assessment modelling checklist for decision-analytic models (Philips et al. 2004) may also be used to assess the methodological quality of modelling studies in more detail.

When assessing the evidence of effectiveness and cost effectiveness, the PHACs should consider the implications of any unexplained differences between model results.

The review team should record its judgements based on the economic evaluation checklist (and health technology assessment modelling checklist, if appropriate) in an appendix to the guidance. The checklist 'comments' column should be used to record the reasons for these judgements and, where necessary, additional details about the studies.

Where good evidence is scarce, the criteria for inclusion should be discussed (or revisited) with the CPHE project team.

6.2.3 Summarising and presenting cost-effectiveness results

The review team should present cost, cost effectiveness or net-benefit estimates from published or unpublished studies, or from economic analyses conducted for the guidance, in an economic evidence table. (See appendix K for the table template.) Where economic information is not available (or it is not thought to be relevant to the question) this should be explicitly stated. An overall assessment of any study limitations and its applicability should also be included. Issues that the PHACs should consider when assessing the economic evidence should be noted in the 'Notes' column.

The economic evaluation results should include a 'best-available' estimate or range for the incremental cost, the incremental effect and, where relevant, the incremental cost-effectiveness ratio or net-benefit estimate. A summary of the degree of uncertainty in the estimates should also be presented, reflecting the results of deterministic or probabilistic sensitivity analyses, or stochastic analyses of trial data, as appropriate.

The review team should include a short economic evidence statement at the end of the economic evidence synthesis. This should summarise key features of the evidence on cost effectiveness in a similar format to that used to summarise the effectiveness evidence (see section 5.5).

6.3.1 Type of economic analysis

Because local government will (after March 2013) be largely responsible for implementing public health programmes, NICE has broadened its approach to the appraisal of public health interventions. Local government is responsible not only for the health of individuals and communities, but also for their overall welfare. It is also less centralised than the NHS. The tools of economic evaluation must reflect both of these aspects (wider remit than health and greater local element). The main change to the approach to economic evaluation is to place more emphasis on cost–consequences analysis (CCA) and cost–benefit analysis (CBA) than has been the case in previous methods manuals.

Cost-effectiveness analysis and cost–utility analysis (CUA) will still be required routinely, for several reasons:

6.3.2 Perspective

The standard perspective for the economic analysis of public health interventions is typically that of the public sector. This allows the costs and benefits of more than 1 central government department or body, and local government, to be taken into account.

In addition to the public sector perspective, the perspective of the department that administers the interventions should be used. For example, if local government is involved almost exclusively, or is the body administering the intervention, then a local government perspective should be used as well as the public sector perspective. If the NHS is involved almost exclusively, or is the body administering the intervention, then an NHS perspective should be used as well as the public sector perspective.

When appropriate, and in agreement with the CPHE project team, results may also be presented from other perspectives. For example, an employer's perspective would be taken to demonstrate the business case for a public health intervention on, say, workplace smoking. The transfer of many public health functions to local government means that a local government perspective is likely to be the most common for public health programmes.

Whenever CBA (with WTP estimated by aggregating individually elicited WTP) has been used, the appropriate perspective is a societal one.

A modified CBA (with WTP being estimated as a government-sector WTP out of its budget) would use a public sector perspective. However, this is only likely if the government sector involved is the health sector, because that is the only sector with a reasonably well-defined government-sector WTP in the form of the threshold incremental cost-effectiveness ratio (ICER) of £20,000 to £30,000 per QALY gained.

This set of perspectives differs from that used for NICE technology appraisals and clinical guidelines: these only consider the NHS and personal social services (PSS). The broadening of the perspective set for public health guidance reflects the wider remit of public health and the changes in the way that it will be delivered after March 2013.

Table 6.1 Summary of the public health reference case^[8]

6.3.3 More than one department (or local government) is involved

No standard method has been devised of apportioning costs and who should bear them when more than 1 government department (or local government) is involved in the delivery of an intervention. The problem is greatest when 1 government department (or local government) reaps the benefits of an intervention but another department is required to implement it: for example, a school being asked to find time in its timetable to deliver a health intervention. No method is suggested in this guide for dealing with such problems.

6.3.4 Discounting

There is currently some debate about the most appropriate rate of discounting future costs and benefits.

Since public health interventions usually act over the long term, it has been decided to use a discount rate of 1.5% for all costs and benefits, so as to be in line with the option used in the Guide to the methods of technology appraisal for interventions which have effects lasting for many years. In practical terms, for many interventions in public health where the intervention occurs over a relatively short period earlier in people's lives but the benefits accrue mostly at the end of their lives, the use of this set of discount rates will give approximately the same result as a 3.5% discount rate on costs and a 1.5% rate on benefits used by the Department of Health's (DH), in line with their interpretation of the Treasury Green Book.

In sensitivity analysis, an annual discount rate of 3.5% on both costs and benefits will also be routinely used in sensitivity analysis, to allow comparisons with previous public health and Centre for Health Technology Evaluation guidance and with current Centre for Health Technology Evaluation and Centre for Clinical Practice guidance where this is based on 3.5% for costs and benefits.

6.3.5 Other aspects of modelling

The general approach to modelling to be taken must be agreed with the CPHE project team prior to implementation. Where available, consideration should be given to linking the model structure to the logic model used to develop the research questions. The logic model illustrates the causal pathways between human behaviour, the social, environmental and biological determinants of health and potential interventions and outcomes (see chapter 2 and appendix A for details).

Sensitivity analysis should be used to explore the impact that potential sources of bias and uncertainty could have on model results.

Deterministic sensitivity analysis should be used to explore key structural assumptions: testing whether and how the model results change under alternative, plausible scenarios. It should also be used to test any bias resulting from the data sources selected for key model parameters.

Probabilistic sensitivity analysis should be used to explore the uncertainty arising from imprecision in model parameters. Any uncertainty associated with all parameters can be simultaneously reflected in the results. In non-linear decision models, probabilistic methods also provide the best estimates of mean costs and outcomes. However, models incorporating probabilistic methods are more time-consuming to construct and may not always be a priority for health economists working on public health guidance. In such cases, deterministic sensitivity analysis should be used, and the decision not to use probabilistic methods should be justified in the guidance.

The Guide to the methods of technology appraisal includes other useful advice for health economists developing economic models for use in public health guidance.

6.3.6 General principles

Regardless of the modelling approach adopted, economic analysis should be based on the following principles:

6.3.7 Identifying and selecting model inputs

The review team and the CPHE project team should discuss whether or not any additional searches are needed. Additional searches may be necessary if the effectiveness searches do not provide the information needed for economic modelling. For example, information on:

It is not necessary to conduct formal, systematic literature searches for all the types of information required for economic modelling (although effectiveness data used in the modelling should be taken from the effectiveness reviews). For example, it is often better to get information on unit costs from: national list prices, the Personal Social Services Research Unit (PSSRU) Unit costs of health and social care report or the DH tariff. Information on costing can also be found in the NICE document Assessing cost impact: Methods guide (2011)) and through discussion with the costing analyst in the NICE implementation team. In addition, it might be better to obtain some information about epidemiology or public services from national statistics or databases, rather than from literature studies.

Ideally, systematic literature reviews should be conducted for other model inputs, for instance to model the relative risk of coronary heart disease in relation to physical activity levels. However, this is time-consuming and the health economist should look at other options first. Possibilities include: searching the public health effectiveness evidence that was used to structure the key questions (and perhaps other relevant questions) in the scope; or liaising with the review team, the PHACs and other experts.

If an additional literature search is necessary, the review team should discuss and agree this with the information scientist and the CPHE project team. If longer-term follow-up data are required, a literature search to identify cohort studies may be appropriate. The report by Philips et al. (2004) is a useful guide to data searching methods for economic models.

6.3.8 Subgroup analysis

Where appropriate, economic model(s) should be run to estimate the cost effectiveness of an intervention in different population subgroups. The subgroups of relevance will be agreed with CPHE project team.

6.4.1 Using cost–utility analysis (CUA)

If there is strong evidence that an intervention dominates the alternatives (that is, it is both more effective and less costly), it should normally be recommended. However, if 1 intervention is more effective but also more costly than another, then the ICER should be considered. The cost per QALY gained should be calculated as the difference in mean cost divided by the difference in mean QALYs for 1 intervention compared with the other.

If 1 intervention appears to be more effective than another, the PHACs will have to decide whether it represents reasonable 'value for money' as indicated by the relevant ICER. In doing so, the committee should also refer to the principles outlined in NICE's report Social value judgements: principles for the development of NICE guidance, second edition (2008). This states:

However, such thresholds do not exist outside the health sector, making it difficult to judge whether the benefits accruing to the non-health sectors are cost effective. The PHACs should take into account the factors it considers most appropriate. These could include non-health-related outcomes that are valued by the rest of the public sector (like reduced levels of public disorder resulting from reduced alcohol consumption).

Decisions about whether to recommend interventions should not be based on cost effectiveness alone. The advisory committees responsible for developing the guidance should also take into account other factors, such as the need to prevent discrimination and to promote equity. They will consider trade-offs between efficient and equitable allocations of resources. These factors should be explained in the considerations section of the guidance (see section 7.6). If a structured public health question is not considered for further economic analysis, the PHACs should still consider the likely cost effectiveness of the associated recommendations. If available, this assessment may be based on published estimates of cost effectiveness or, if not, a qualitative judgement based on prior knowledge and undertaken in an open and transparent process.

6.4.2 Using cost–benefit analysis (CBA)

Where the WTP is aggregated individual WTP, the NICE committee should be aware that this is likely to exceed public-sector WTP, sometimes by quite a margin. If a conversion factor between these 2 forms of WTP has been established in any particular circumstance, the committee should take this into account. In the absence of a conversion factor, the committee should consider the likelihood of a discrepancy between the 2 measures in making recommendations that rely on a CBA.

The committee should also attempt to determine whether any adjustment should be made to convert 'ability-to-pay' estimates into those that prioritise on the basis of need and the ability of an intervention to meet that need.

After accounting for the above factors, committees should not recommend interventions whose net present value (NPV) is estimated to be negative unless they believe that non-economic objectives override the economic loss involved. Given a choice of interventions with positive NPVs, committees should prefer the intervention that maximises the NPV, unless non-economic objectives override the economic loss incurred by choosing an intervention that does not maximise NPV.

6.4.3 Using cost–consequences analysis (CCA)

NICE committees should ensure that, where possible, the different sets of consequences do not double count costs or benefits. The way that the sets of consequences have been implicitly weighted should be recorded as openly, transparently and as accurately as possible. CCA then requires the decision-maker to decide which interventions represent the best value, preferably using systematic and transparent process. Various tools are available to support this part of the process.