NICE process and methods

6 The appraisal of the evidence and structured decision-making

6 The appraisal of the evidence and structured decision-making

6.1 Introduction

6.1.1 The purpose of this section is to explain how the Appraisal Committee appraises the evidence and makes the judgements that lead to its final conclusions.

6.1.2 The Appraisal Committee is an independent advisory body. Members include people who work in the NHS, relevant academic disciplines, pharmaceutical and medical devices industries and lay members. The Appraisal Committee makes recommendations to the Institute regarding the clinical and cost effectiveness of treatments for use within the NHS. It is also the role of the Appraisal Committee not to recommend treatments if the benefits to patients are unproven, or if the treatments are not cost effective. The Institute is responsible for the dissemination of the final guidance to the NHS.

6.1.3 When formulating its recommendations to the Institute, the Appraisal Committee has discretion to consider those factors it believes are most appropriate to each appraisal. In doing so, the Appraisal Committee has regard to the provisions and regulations of the Health and Social Care Act 2012 relating to NICE, and NICE's legal obligations on equality and human rights. The Act expects NICE, in undertaking its general duties, to have regard to:

  • The broad balance between the benefits and costs of providing health services or social care in England.

  • The degree of need of people in England for health services or social care.

  • The desirability of promoting innovation in providing health services or social care in England.

6.1.4 In reaching its decision, the Appraisal Committee bases its recommendations on the evidence presented, including statements from consultees and commentators and the views expressed by clinical specialists, commissioning experts and patient experts at the Committee meeting. Formulating the 'Considerations' section of the guidance represents an important component of the Appraisal Committee's work. This section identifies the key evidence taken into account by the Appraisal Committee and its views on this evidence. It highlights any areas of contention and uncertainty that have arisen during the Appraisal Committee's discussions of the evidence and presents a general description of the Committee's views on the written and oral inputs that have informed their decision.

6.1.5 Usually, the Appraisal Committee's provisional recommendations are released in an appraisal consultation document for widespread consultation with consultees, commentators and the public. In reviewing responses to consultation, the Committee is principally interested in comments on its preliminary recommendations within the context of the evidence base reviewed at its first meeting and its consideration of that evidence. The comments received on the key issues identified at the first meeting are carefully reviewed.

6.1.6 The Appraisal Committee considers the impact of the consultation comments on:

  • the preliminary recommendations on the use of the technology

  • the other sections of the appraisal consultation document

  • recommendations for further research

  • issues for implementation, including:

    • resource availability to support implementation (for example, workforce planning and training, and new clinics)

    • the extent of any changes in current clinical practice

    • the implementation criteria agreed between the Institute and the Department of Health

  • the timing of the appraisal review, because of the timing and potential impact of research in progress (for example, new RCTs).

6.1.7 The Appraisal Committee considers the comments and, if appropriate amends its recommendations, exercising judgement on the nature and importance of the comments from consultation. The content of the 'Considerations' section is modified to clarify the key evidence considered by the Appraisal Committee, its views on this evidence and any areas of contention that have arisen during the appraisal. This section also highlights, in general terms, the written and oral inputs that the Appraisal Committee has used to inform its judgement.

6.1.8 The Appraisal Committee takes into account advice from the Institute on the appropriate approach to making scientific and social value judgements. Advice on social value judgements is informed by the work of the Citizens Council, NICE's advisory bodies, and NICE's Board, as well as legislation on human rights, discrimination and equality as reflected in NICE's equality scheme. Principles that describe the social value judgements that should, generally, be considered by the Appraisal Committee have been provided in the Institute's document, Social value judgements: principles for the development of NICE guidance..

6.1.9 The credibility of the guidance produced by the Institute is dependent on the transparency of the Appraisal Committee's decision-making process. It is crucial that the Appraisal Committee's decisions are explained clearly with reference to all the available evidence, and that the contributions of clinical specialists, commissioning experts, patient experts and the views of people who responded to consultation during the appraisal are considered. The reasoning for the Committee's decision will be explained, with reference to the factors that have been taken into account, in the 'Considerations' section of the guidance.

6.1.10 The language and style used in the documents produced by the Committee are governed by the following principles:

  • The need for clarity in explaining how the Appraisal Committee has come to its conclusions. Of particular importance is the 'Considerations' section of the guidance document, which summarises the key issues that have been debated and the rationale for the conclusions drawn. It also includes a table that documents how the Appraisal Committee has taken account of each of the main components of the decision.

  • The understanding that the text of the documents does not need to reiterate all the factual information that can be found in the information published alongside the guidance. This requires careful judgement so that enough information and justification is given in the appraisal consultation document or final appraisal determination to enable the reader to understand what evidence the Appraisal Committee considered and, if appropriate, who provided that evidence.

6.1.11 The Appraisal Committee is not empowered to alter the Direction from the Secretary of State for Health on the implementation of the Institute's guidance regarding the mandatory requirement placed upon health commissioners to make funds available for implementation of the Institute's appraisal guidance within 3 months of publication. However, the Appraisal Committee may consider circumstances in which this implementation period should be varied and advise the Institute accordingly. When appropriate, the Committee's consideration is limited to those circumstances in which it is apparent that either the technology cannot be acquired and/or the NHS will not be in a position to use it within the 3-month period.

6.1.12 The Appraisal Committee does not normally make recommendations regarding the use of a drug outside the terms of its marketing authorisation, as published in the manufacturer's summary of product characteristics. For technologies that are not subject to licensing procedures (for example, medical devices), evidence of acceptable quality of manufacturing processes, such as the CE mark, will be required and the technology will be evaluated in the context of the instructions for use. Exceptionally, the Department of Health may direct the Appraisal Committee to make recommendations about a technology outside of the terms of its marketing authorisation or instructions for use.

6.1.13 Evidence relating to use of the technology under appraisal outside the terms of its marketing authorisation may be considered during the assessment phase of the appraisal and may inform the Appraisal Committee's deliberations regarding the licensed use of the drug.

6.1.14 The Committee is not able to make recommendations on the pricing of technologies to the NHS but can consider a patient access scheme subject to the arrangements detailed in the technology appraisal process guide(s).

6.2 Appraisal of the evidence

Structured decision-making: comparators

6.2.1 The Committee has to make judgements on the appropriateness and relevance of comparator technologies because this is crucial to the consideration of the clinical and cost-effectiveness evidence.

6.2.2 When selecting the most appropriate comparator(s), the Committee will consider:

  • established NHS practice in England

  • the natural history of the condition without suitable treatment

  • existing NICE guidance

  • cost effectiveness

  • the licensing status of the comparator.

6.2.3 The Committee will normally be guided by established practice in the NHS when identifying the appropriate comparator(s). When the assessment suggests that an established practice may not be considered a good use of NHS resources relative to another available treatment, the Committee will decide whether to include it as an appropriate comparator in the appraisal, after reviewing an incremental cost–utility analysis. The Committee's overall decision on whether it is a valid comparator will be guided by whether it is recommended in other extant NICE guidance, and/or whether its use is so embedded in clinical practice that its use will continue unless and until it is replaced by a new technology. The Committee will also take into account the uncertainty associated with the estimates of clinical and cost effectiveness, and whether the new technology under appraisal could provide a cost-saving alternative.

6.2.4 The Appraisal Committee can consider as comparators technologies that do not have a marketing authorisation (or CE mark for medical devices) for the indication defined in the scope when they are considered to be part of established clinical practice for the indication in the NHS. Long-standing treatments often lack a sponsor to support the licensing process. Specifically when considering an 'unlicensed' medicine, the Appraisal Committee will have due regard for the extent and quality of evidence, particularly for safety and efficacy, for the unlicensed use.

Structured decision-making: clinical effectiveness and health-related factors

6.2.5 The Appraisal Committee has the discretion to take account of the full range of clinical studies that have been carried out and is not expected to restrict itself to considering only certain categories of evidence. This requires the Appraisal Committee to consider all of the evidence presented to it, including RCTs, observational studies and any qualitative evidence related to the experiences of patients, carers and clinical specialists who have used the technology being appraised or are familiar with the relevant condition. In evaluating the evidence base, the Appraisal Committee will exercise its judgement when deciding whether particular forms of evidence are fit for purpose in answering specific questions.

6.2.6 The importance given to these various kinds of evidence depends on both the overall balance and quality of the evidence from different sources, and the suitability of a particular type of evidence to address issues under consideration. In general, greater importance is given to evidence derived from high-quality studies with methodology designed to minimise bias.

6.2.7 The Appraisal Committee's judgements on clinical effectiveness take account of the following factors:

  • The nature and quality of the evidence derived from:

    • the analysis of the independent academic groups

    • the written submissions of the consultees

    • the views expressed by the clinical specialists, particularly their experience of the technology in clinical practice

    • the views of the patient experts and carers on the experiences of patients who have used the technology.

  • Uncertainty generated by the evidence and differences between the evidence submitted for licensing and that relating to effectiveness in clinical practice.

  • The possible differential benefits or adverse outcomes in different groups of patients.

  • The impact of benefits and adverse outcomes associated with the technology as seen from the patient's perspective.

  • The position of the technology in the overall pathway of care and the alternative treatments that are established in clinical practice.

6.2.8 The extent to which the above factors are taken into account in making judgements about the evidence of clinical effectiveness is a matter for the Committee's discretion.

6.2.9 In the reference case, the Committee will regard all QALYs as being of equal weight. However, when considering the overall health benefits, the Appraisal Committee can accept analysis that explores a QALY weighting that is different from that of the reference case when a technology appraisal concerns a 'life-extending treatment at the end of life', or in other circumstances when instructed by the NICE board.

6.2.10 In the case of a 'life-extending treatment at the end of life', the Appraisal Committee will satisfy itself that all of the following criteria have been met:

  • the treatment is indicated for patients with a short life expectancy, normally less than 24 months and

  • there is sufficient evidence to indicate that the treatment has the prospect of offering an extension to life, normally of a mean value of at least an additional 3 months, compared with current NHS treatment.

    In addition, the Appraisal Committees will need to be satisfied that:

  • the estimates of the extension to life are sufficiently robust and can be shown or reasonably inferred from either progression-free survival or overall survival (taking account of trials in which crossover has occurred and been accounted for in the effectiveness review) and

  • the assumptions used in the reference case economic modelling are plausible, objective and robust.

6.2.11 When the conditions described in section 6.2.10 are met, the Appraisal Committee will consider:

  • the impact of giving greater weight to QALYs achieved in the later stages of terminal diseases, using the assumption that the extended survival period is experienced at the full quality of life anticipated for a healthy individual of the same age and

  • the magnitude of the additional weight that would need to be assigned to the QALY benefits in this patient group for the cost effectiveness of the technology to fall within the normal range of maximum acceptable ICERs, with a maximum weight of 1.7.

6.2.12 Treatments recommended following the application of the 'end-of-life' criteria listed in section 6.2.10 will not necessarily be regarded or accepted as standard comparators for future appraisals of new treatments introduced for the same condition. Second and subsequent extensions to the marketing authorisations for the same product will be considered on their individual merits.

Amendments have been made to sections 6.2.10–12 to support the joint NHS England and NICE proposals for the management of the Cancer Drugs Fund from April 2016.

Structured decision-making: cost effectiveness

6.2.13 The Institute is asked to take account of the overall resources available to the NHS when determining cost effectiveness. Therefore, decisions on the cost effectiveness of a new technology must include judgements on the implications for healthcare programmes for other patient groups that may be displaced by the adoption of the new technology.

6.2.14 The potential budget impact of the adoption of a new technology does not determine the Appraisal Committee's decision. The Committee does take account of how its advice may enable the more efficient use of available healthcare resources. In general, the Committee will want to be increasingly certain of the cost effectiveness of a technology as the impact of the adoption of the technology on NHS resources increases. Therefore, the Committee may require more robust evidence on the effectiveness and cost effectiveness of technologies that are expected to have a large impact on NHS resources.

6.2.15 The Appraisal Committee takes account of how the incremental cost effectiveness of the technology being appraised relates to other interventions or technologies currently or potentially applied in the NHS. In addition, as far as possible, the Committee will want to ensure that their judgements regarding the cost-effective use of NHS resources are consistently applied between appraisals.

6.2.16 When the evidence on key parameters used to estimate cost effectiveness (for example, clinical effectiveness and effect on health-related quality of life) has serious limitations and/or when a variety of assumptions have been necessary in the cost-effectiveness modelling, the additional uncertainty this generates is a key factor in underpinning the judgements of the Committee. The Appraisal Committee is likely to consider more favourably technologies for which evidence on cost effectiveness is underpinned by the best-quality clinical data than those for which supporting evidence is dependent to a large extent on theoretical modelling alone. However, the Committee is aware that the evidence base will necessarily be weaker for some technologies, such as technologies used to treat patients with very rare diseases.

6.2.17 The Committee's judgements on cost effectiveness are influenced by the following factors:

  • The strength of the supporting clinical-effectiveness evidence.

  • The robustness and appropriateness of the structure of the economic models. In particular, the Committee considers carefully whether the model reflects the decision problem at hand and the uncertainties around the assumptions on which the model structure is based.

  • The plausibility of the inputs into, and the assumptions made, in the economic models.

  • The Committee's preferred modelling approach, taking into account all of the economic evidence submitted.

  • The range and plausibility of the ICERs generated by the models reviewed.

  • The likelihood of decision error and its consequences.

6.2.18 The Appraisal Committee will consider carefully which individuals benefit most from the technology and whether there are subgroups of individuals for whom the effectiveness evidence suggests differential cost effectiveness. The Appraisal Committee may recommend the use of an intervention for subgroups of the population only when there is clear evidence that the characteristics defining the subgroup influence the effectiveness and/or cost effectiveness of the intervention.

6.2.19 In cases when treatment restores people who would otherwise die or have a very severely impaired life to full or near full health, and when this is sustained over a very long period (normally at least 30 years), cost-effectiveness analyses are very sensitive to the discount rate used. In this circumstance, analyses that use a non-reference-case discount rate for costs and outcomes may be considered. A discount rate of 1.5% for costs and benefits may be considered by the Appraisal Committee if it is highly likely that, on the basis of the evidence presented, the long-term health benefits are likely to be achieved. Further, the Appraisal Committee will need to be satisfied that the introduction of the technology does not commit the NHS to significant irrecoverable costs.

Structured decision-making: non-health factors

6.2.20 In general the Committee uses the most plausible ICER as the primary consideration when making judgements about the acceptability of technologies as a cost-effective use of NHS resources. However, its overall conclusions are also affected by the following additional considerations:

  • Whether or how its judgements have a bearing on broader social considerations to the extent that these are covered by NICE's principles on social value judgements.

  • Whether a substantial proportion of the costs (savings) or benefits are incurred outside of the NHS and personal and social services, or are associated with significant benefits other than health, only when requested specifically by the Department of Health as part of the remit.

6.2.21 The concept that underlies the Committee decision-making is that of the opportunity cost of programmes that could be displaced by the introduction of new technologies. This way, NICE seeks to maximise the health benefit gained from a fixed NHS budget. This principle is correct if the sole purpose of the health service is to improve health. While this may be the primary purpose of the NHS, it is acknowledged that care delivered by the NHS could have other benefits that are considered socially valuable but are not directly related to health and are not easily captured in a cost per QALY analysis. Techniques exist to consider the trade-off between health benefits and non-health benefits quantitatively. These techniques require that all relevant criteria are identified in advance, quantified and then weighted to reflect aspects of social value in a way that can be regarded as legitimate by all stakeholders. At present the introduction of such techniques into the Committee's decision-making is considered unsuitable. Therefore the Committee will take non-health objectives of the NHS into account by considering the extent to which society may be prepared to forego health gain in order to achieve other benefits that are not health related.

6.3 Decision-making

6.3.1 The Appraisal Committee does not use a precise maximum acceptable ICER above which a technology would automatically be defined as not cost effective or below which it would. Given the fixed budget of the NHS, the appropriate maximum acceptable ICER to be considered is that of the opportunity cost of programmes displaced by new, more costly technologies. NICE does not have complete information about the costs and QALYs from all competing healthcare programmes in order to define a precise maximum acceptable ICER. However, NICE considers that it is most appropriate to use a range as described in sections 6.3.2 to 6.3.5. Furthermore, consideration of the cost effectiveness of a technology is a necessary, but is not the sole, basis for decision-making. Consequently, the Institute considers technologies in relation to this range of maximum acceptable ICERs, such that the influence of other factors upon the decision to recommend a technology is greater when the ICER is closer to the top of the range.

6.3.2 Below a most plausible ICER of £20,000 per QALY gained, the decision to recommend the use of a technology is normally based on the cost-effectiveness estimate and the acceptability of a technology as an effective use of NHS resources. When the estimated ICERs presented are less than £20,000 per QALY gained and the Committee judges that particular interventions should not be provided by the NHS, the recommendations will make specific reference to the Committee's view on the plausibility of the inputs to the economic modelling and/or the certainty around the estimated ICER. This might be affected, for example, by sensitivity analysis or limitations to the generalisability of findings regarding effectiveness.

6.3.3 Above a most plausible ICER of £20,000 per QALY gained, judgements about the acceptability of the technology as an effective use of NHS resources will specifically take account of the following factors:

  • The degree of certainty around the ICER. In particular, the Committee will be more cautious about recommending a technology when they are less certain about the ICERs presented.

  • Whether there are strong reasons to indicate that the assessment of the change in health-related quality of life has been inadequately captured, and may therefore misrepresent the health utility gained.

  • The innovative nature of the technology, specifically if the innovation adds demonstrable and distinctive benefits of a substantial nature which may not have been adequately captured in the reference case QALY measure.

  • The technology meets the criteria for special consideration as a 'life-extending treatment at the end of life' (see section 6.2.10)

  • Aspects that relate to non-health objectives of the NHS (see sections 6.2.20 and 6.2.21).

6.3.4 As the ICER of an intervention increases in the range of £20,000 to £30,000 per QALY gained, the Committee's judgement about the acceptability of the technology as an effective use of NHS resources will make explicit reference to the relevant factors listed in section 6.3.3.

6.3.5 Above a most plausible ICER of £30,000 per QALY gained, the Committee will need to identify an increasingly stronger case for supporting the technology as an effective use of NHS resources, with regard to the factors listed in section 6.3.3.

6.4 Research recommendations

6.4.1 When the evidence of clinical effectiveness or impact of a technology on other health outcomes is either absent, weak or uncertain, the Appraisal Committee may recommend that the technology is used only in the context of research or while the technology is recommended as an option, research is also conducted. Before issuing such recommendations the Committee will consider the following factors:

  • the need for and potential value to the NHS of additional evidence that can inform the development of NICE guidance and clinical practice on the use of the technology

  • the uncertainty in the analysis and what could be gained by reconsidering the decision in the light of research findings

  • whether the research is feasible in circumstances when the Appraisal Committee recommends the intervention for NHS use outside the context of research

  • irrecoverable costs incurred from introducing the technology

  • the likely net benefits for all NHS patients of use only in a research setting during the time that the recommended research is being conducted.

    In considering these factors the Committee will balance the potential net benefits to current NHS patients of a recommendation not restricted to research with the potential net benefits to both current and future NHS patients of being able to produce guidance and base clinical practice on a more secure evidence base.

6.4.2 Recommendations on the use of technologies only in the context of research will not include consideration of which organisation (public or private) will fund the research. The Appraisal Committee will consider:

  • the likelihood that the research needed will be commissioned and successfully report

  • the time it is likely to take for research findings to be available to inform subsequent NICE guidance and clinical practice

  • other factors which may impact on the value of evidence generation, such as other research that is underway or likely to be commissioned and completed.

    In considering these factors the Committee may seek advice from research commissioners, the wider research and clinical communities and consultees.

6.4.3 When the Committee recommends use of a technology and that research is conducted, it considers the factors in sections 6.4.1 and 6.4.2. The Committee will need to be satisfied that the additional research is feasible in the circumstances in which the intervention has been recommended.

6.4.4 In all cases, when technologies are being recommended only in the context of research, the Committee will explore whether overall, the potential value to the NHS of the recommended research is likely to represent good value in the context of limited research resources.