Guide to the methods of technology appraisal 2013

5.1 Framework for estimating clinical and cost effectiveness

Directions on particular aspects of NICE health technology assessment and economic evaluation are presented below. The position statement of the Institute is set out (in bold), followed by explanation and justification.

5.2 Synthesis of evidence on health effects

5.2.1 The objective of the analysis of clinical effectiveness is an unbiased estimate of the mean clinical effectiveness of the technologies being compared. The analysis of clinical effectiveness must be based on data from all relevant studies of the best available quality and should consider the range of typical patients, normal clinical circumstances, clinically relevant outcomes, comparison with relevant comparators, and measures of both relative and absolute effectiveness with appropriate measures of uncertainty. The Institute has a preference for RCTs directly comparing the intervention with 1 or more relevant comparators and these should be presented in the reference-case analysis if available.

5.3 Measuring and valuing health effects

5.3.1 For the cost-effectiveness analyses health effects should be expressed in QALYs. For the reference case, the measurement of changes in health-related quality of life should be reported directly from patients and the utility of these changes should be based on public preferences using a choice-based method. The EQ-5D is the preferred measure of health-related quality of life in adults.

5.3.2 A QALY combines both quality of life and life expectancy into a single index. In calculating QALYs, each of the health states experienced within the time horizon of the model is given a utility reflecting the health-related quality of life associated with that health state. The duration of time spent in each health state is multiplied by the utility. Deriving the utility for a particular health state usually comprises 2 elements: measuring health-related quality of life in people who are in the relevant health state and valuing it according to preferences for that health state relative to other states (usually perfect health and death).

5.3.3 Health-related quality of life, or changes in health-related quality of life, should be measured directly by patients. When it is not possible to obtain measurements of health-related quality of life directly from patients, data should be obtained from the person who acts as their carer in preference to healthcare professionals.

5.3.4 The valuation of health-related quality of life measured in patients (or by their carers) should be based on a valuation of public preferences from a representative sample of the UK population using a choice-based method. This valuation leads to the calculation of utility values.

5.3.5 Different methods used to measure health-related quality of life produce different utility values; therefore, results from different methods or instruments cannot always be compared. Given the need for consistency across appraisals, one measurement method, the EQ-5D, is preferred for the measurement of health-related quality of life in adults.

5.3.6 The EQ-5D is a standardised and validated generic instrument that is widely used and has been validated in many patient populations. The EQ-5D comprises 5 dimensions of health: mobility, ability to self-care, ability to undertake usual activities, pain and discomfort, and anxiety and depression. For each of these dimensions it has 3 levels of severity (no problems, some problems, severe problems). The system has been designed so that people can describe their own health-related quality of life using a standardised descriptive system. A set of preference values elicited from a large UK population study using a choice-based method of valuation (the time trade-off method) is available for the EQ-5D health state descriptions. This set of values should be applied to measurements of health-related quality of life to generate health-related utility values.

5.3.7 In some circumstances adjustments to utility values, for example for age or comorbidities, may be needed.

5.3.8 If not available in the relevant clinical trials, EQ-5D data can be sourced from the literature. When obtained from the literature, the methods of identification of the data should be systematic and transparent. The justification for choosing a particular data set should be clearly explained. When more than 1 plausible set of EQ-5D data is available, sensitivity analyses should be carried out to show the impact of the alternative utility values.

5.3.9 When EQ-5D data are not available, these data can be estimated by mapping other health-related quality of life measures or health-related benefits observed in the relevant clinical trial(s) to EQ-5D. The mapping function chosen should be based on data sets containing both health-related quality of life measures and its statistical properties should be fully described, its choice justified, and it should be adequately demonstrated how well the function fits the data. Sensitivity analyses to explore variation in the use of the mapping algorithms on the outputs should be presented.

5.3.10 In some circumstances the EQ-5D may not be the most appropriate. To make a case that the EQ-5D is inappropriate, qualitative empirical evidence on the lack of content validity for the EQ-5D should be provided, demonstrating that key dimensions of health are missing. This should be supported by evidence that shows that EQ-5D performs poorly on tests of construct validity and responsiveness in a particular patient population. This evidence should be derived from a synthesis of peer-reviewed literature. In these circumstances alternative health-related quality of life measures may be used and must be accompanied by a carefully detailed account of the methods used to generate the data, their validity, and how these methods affect the utility values.

5.3.11 When necessary, consideration should be given to alternative standardised and validated preference-based measures of health-related quality of life that have been designed specifically for use in children. The standard version of the EQ-5D has not been designed for use in children. An alternative version for children aged 7–12 years is available, but a validated UK valuation set is not yet available.

5.3.12 A new version of the EQ-5D, the EQ-5D-5L, has been developed in which there are 5 levels of severity (no problem, slight problems, moderate problems, severe problems and unable to or extreme problems) for each of the 5 dimensions of health (see section 5.3.6). The EQ-5D-5L may be used for reference-case analyses. The descriptive system for the EQ-5D-5L has been validated, but no valuation set to derive utilities currently exists. Until an acceptable valuation set for the EQ-5D-5L is available, the validated mapping function to derive utility values for the EQ-5D-5L from the existing EQ-5D (-3L) may be used (available from www.euroqol.org).

In August 2017, NICE issued a position statement on the use of the EQ‑5D‑5L valuation set. Companies and academic groups should refer to this statement.

5.4 Equity considerations in cost-effectiveness analysis

5.4.1 In the reference case, an additional QALY should receive the same weight regardless of any other characteristics of the people receiving the health benefit.

5.4.2 The estimation of QALYs, as defined in the reference case, implies a particular position regarding the comparison of health gained between individuals. Therefore, in the reference case, an additional QALY is of equal value regardless of other characteristics of the individuals, such as their socio-demographic characteristics, their age, or their level of health. The Committee has discretion to consider a different equity position, and may do so in certain circumstances and when instructed by the NICE Board (see section 6).

5.5 Evidence on resource use and costs

5.6 Discounting

5.6.1 Cost-effectiveness results should reflect the present value of the stream of costs and benefits accruing over the time horizon of the analysis. For the reference case, the same annual discount rate should be used for both costs and benefits (currently 3.5%).

5.6.2 The specific discount rate varies across jurisdictions and over time. The Institute considers that it is usually appropriate to discount costs and health effects at the same annual rate of 3.5%, based on the recommendations of the UK Treasury for the discounting of costs.

5.6.3 Sensitivity analyses using rates of 1.5% for both costs and health effects may be presented alongside the reference-case analysis (see section 6.2.19).

5.7 Modelling methods

5.7.1 Full documentation and justification of structural assumptions and data inputs should be provided. When there are alternative plausible assumptions and inputs, sensitivity analyses of their effects on model outputs should be undertaken.

5.7.2 Modelling provides an important framework for synthesising available evidence and generating estimates of clinical and cost effectiveness in a format relevant to the Appraisal Committee's decision-making process. Models are required for most technology appraisals. Situations when modelling is likely to be required include those when:

5.7.3 Providing an all-embracing definition of what constitutes a high-quality model is not possible. In general, estimates of treatment effect should be based on the results of the systematic review, structural assumptions should be fully justified and data inputs should be clearly documented and justified in the context of a valid review of the alternatives.

5.7.4 The methods of quality assurance used in the development of the model should be detailed and the methods and results of model validation should be provided. In addition, the results from the analysis should be presented in a disaggregated format and should include a tabular presentation of information on estimates of life-years gained, mortality rates (at separate time points if appropriate) and the frequency of selected clinical events predicted by the model.

5.7.5 Clinical end points that reflect how a patient feels, functions, or how long a patient survives are regarded as more informative than surrogate end points (such as laboratory tests and imaging findings). When the use of 'final' clinical end points is not possible and 'surrogate' data on other outcomes are used to infer the effect of treatment on mortality and health-related quality of life, evidence in support of the surrogate-to-final end point outcome relationship must be provided together with an explanation of how the relationship is quantified for use in modelling. The usefulness of the surrogate end point for estimating QALYs will be greatest when there is strong evidence that it predicts health-related quality of life and/or survival. In all cases, the uncertainty associated with the relationship between the end point and health-related quality of life or survival should be explored and quantified.

5.7.6 Clinical trial data generated to estimate treatment effects may not sufficiently quantify the risk of some health outcomes or events for the population of interest or may not provide estimates over a sufficient duration for the economic analysis. The methods used to identify and critically appraise sources of data for economic models should be stated and the choice of particular data sets should be justified with reference to their suitability to the population of interest in the appraisal.

5.7.7 Modelling is usually required to extrapolate costs and health benefits over an extended time horizon. Assumptions used to extrapolate the impact of treatment over the relevant time horizon should have both external and internal validity and be reported transparently. The external validity of the extrapolation should be assessed by considering both clinical and biological plausibility of the inferred outcome as well as its coherence with external data sources such as historical cohort data sets or other relevant clinical trials. Internal validity should be explored and when statistical measures are used to assess the internal validity of alternative models of extrapolation based on their relative fit to the observed trial data, the limitations of these statistical measures should be documented. Alternative scenarios should also be routinely considered to compare the implications of different methods for extrapolation of the results. For example, for duration of treatment effects, scenarios might include when the treatment benefit in the extrapolated phase is: (i) nil; (ii) the same as during the treatment phase and continues at the same level; or (iii) diminishes in the long term.

5.7.8 In RCTs, participants randomised to the control group are sometimes allowed to switch treatment group and receive the active intervention. In these circumstances, when intention-to-treat analysis is considered inappropriate, statistical methods that adjust for treatment switching can also be presented. Simple adjustment methods such as censoring or excluding data from patients who crossover should be avoided because they are very susceptible to selection bias. The relative merits and limitations of the methods chosen to explore the impact of switching treatments should be explored and justified with respect to the method chosen and in relation to the specific characteristics of the data set in question. These characteristics include the mechanism of crossover used in the trial, the availability of data on baseline and time-dependent characteristics, and expectations around the treatment effect if the patients had remained on the treatment to which they were allocated.

5.8 Exploring uncertainty

5.8.1 It is important for the model to quantify the decision uncertainty associated with a technology (that is, the probability that a different decision would be reached if the true cost effectiveness of each technology could be ascertained before making the decision).

5.8.2 Models are subject to uncertainty around the structural assumptions used in the analysis. Examples of structural uncertainty may include how different states of health are categorised and how different pathways of care are represented. These structural assumptions should be clearly documented and the evidence and rationale to support them provided. The impact of structural uncertainty on estimates of cost effectiveness should be explored by separate analyses of a representative range of plausible scenarios.

5.8.3 Examples of when this type of scenario analysis should be conducted are:

5.8.4 Uncertainty about the appropriateness of the methods used in the reference case can also be dealt with using sensitivity analysis, but these analyses must be presented separately.

5.8.5 A second type of uncertainty arises from the choice of data sources to provide values for the key parameters, such as different costs and utilities, estimates of relative effectiveness and their duration. The implications of different estimates of key parameters must be reflected in sensitivity analyses (for example, through the inclusion of alternative data sets). Inputs must be fully justified and uncertainty explored by sensitivity analysis using alternative input values.

5.8.6 The choice of data sources to include in an analysis may not be clear-cut. In such cases, the analysis should be re-run, using the alternative data source or excluding the study about which there is doubt, and the results reported separately. Examples of when this type of sensitivity analysis should be conducted are:

5.8.7 A third source of uncertainty arises from parameter precision, once the most appropriate sources of information have been identified (that is, the uncertainty around the mean health and cost inputs in the model). Distributions should be assigned to characterise the uncertainty associated with the (precision of) mean parameter values. Probabilistic sensitivity analysis is preferred. This enables the uncertainty associated with parameters to be simultaneously reflected in the results of the model. In non-linear decision models, probabilistic methods provide the best estimates of mean costs and outcomes. The mean value, distribution around the mean, and the source and rationale for the supporting evidence should be clearly described for each parameter included in the model. The distributions chosen for probabilistic sensitivity analysis should not be arbitrarily chosen, but chosen to represent the available evidence on the parameter of interest, and their use should be justified. Formal elicitation methods are available if there is a lack of data to inform the mean value and associated distribution of a parameter. If there are alternative plausible distributions that could be used to represent uncertainty in parameter values, this should be explored by separate probabilistic analyses of these scenarios.

5.8.8 Evidence about the extent of correlation between individual parameters should be carefully considered and reflected in the probabilistic analysis. Assumptions made about the correlations should be clearly presented.

5.8.9 The computational methods used to implement an appropriate model structure may occasionally present challenges in conducting probabilistic sensitivity analysis. The use of model structures that limit the feasibility of probabilistic sensitivity analysis should be clearly specified and justified. Models should always be fit for purpose, and should enable a thorough consideration of the decision uncertainty associated with the model structure and input parameters. The choice of a 'preferred' model structure or programming platform should not result in the failure to adequately characterise uncertainty.

5.8.10 Appropriate ways of presenting uncertainty in cost-effectiveness data parameter uncertainty include confidence ellipses and scatter plots on the cost-effectiveness plane (when the comparison is restricted to 2 alternatives) and cost-effectiveness acceptability curves. The presentation of cost-effectiveness acceptability curves should include a representation and explanation of the cost-effectiveness acceptability frontier. Uncertainty should also be presented in tabular form. In addition to details of the expected mean results (costs, outcomes and ICERs), the probability that the treatment is cost effective at maximum acceptable ICERs of £20,000–£30,000 per QALY gained and the error probability (that the treatment is not cost effective) should also be presented, particularly when there are more than 2 alternatives.

5.8.11 The use of univariate and best- or worst-case sensitivity analysis is an important way of identifying parameters that may have a substantial impact on the cost-effectiveness results and of explaining the key drivers of the model. However, such analyses become increasingly unhelpful in representing the combined effects of multiple sources of uncertainty as the number of parameters increase. The use of probabilistic sensitivity analysis can allow a more comprehensive characterisation of the parameter uncertainty associated with all input parameters.

5.9 Companion diagnostics

5.9.1 The use of a technology may be conditional on the presence or absence of a particular biomarker (for example a gene or a protein). If a diagnostic test to establish the presence or absence of this biomarker is carried out solely to support the treatment decision for the specific technology, the associated costs of the diagnostic test should be incorporated into the assessments of clinical and cost effectiveness. A sensitivity analysis should be provided without the cost of the diagnostic test. When appropriate, the diagnostic accuracy of the test for the particular biomarker of treatment efficacy should be examined and, when appropriate, incorporated in the economic evaluation.

5.9.2 The appraisal will take account of any requirements of the marketing authorisation, including tests to be completed and the definition of a positive test. In clinical practice in the NHS, it may be possible that an alternative diagnostic test procedure to that used in the clinical trials of the technology is used. When appropriate, the possibility that using an alternative test (which may differ in diagnostic accuracy from that used in the clinical trials) may affect selection of the patient population for treatment and the cost effectiveness of the treatment will be highlighted in the appraisal guidance.

5.9.3 It is expected that assessments of multiple companion diagnostic test options will generally be undertaken in the NICE diagnostics assessment programme. For further information see the NICE diagnostics assessment programme manual.

5.10 Analysis of data for patient subgroups

5.10.1 For many technologies, the capacity to benefit from treatment will differ for patients with differing characteristics. This should be explored as part of the reference-case analysis by providing estimates of clinical and cost effectiveness separately for each relevant subgroup of patients. The characteristics of patients in the subgroup should be clearly defined and should preferably be identified on the basis of an expectation of differential clinical or cost effectiveness because of known, biologically plausible mechanisms, social characteristics or other clearly justified factors. When possible, potentially relevant subgroups will be identified at the scoping stage with consideration being given to the rationale for expecting a subgroup effect. However, this does not preclude the identification of subgroups later in the process; in particular, during the deliberations of the Appraisal Committee.

5.10.2 Given the Institute's focus on maximising health gain from limited resources, it is important to consider how clinical and cost effectiveness may differ because of differing characteristics of patient populations. Typically, the capacity to benefit from treatment will differ between patients, and this may also impact on the subsequent cost of care. There should be a clear justification and, if appropriate, biological plausibility for the definition of the patient subgroup and the expectation of a differential effect. Post hoc data 'dredging' in search of subgroup effects is to be avoided and will be viewed sceptically.

5.10.3 The estimate of the overall net treatment effect of an intervention is determined by the baseline risk of a particular condition or event and/or the relative effects of the technology compared with the relevant comparator treatment. The overall net treatment effect may also be determined by other features of the people comprising the population of interest. It is therefore likely that relevant subgroups may be identified in terms of differences in 1 or more contributors to absolute treatment effects.

5.10.4 For subgroups based on differences in baseline risk of specific health outcomes, systematic identification of data to quantify this is required. It is important that the methods for identifying appropriate baseline data for the purpose of subgroup analysis are provided in sufficient detail to enable replication and critical appraisal.

5.10.5 Care should be taken to specify how subgroup analyses are undertaken, including the choice of scale on which any effect modification is defined. The statistical precision of all subgroup estimates should be reflected in the analysis of parameter uncertainty. The characteristics of the patients associated with the subgroups presented should be clearly specified to allow the Appraisal Committee to judge the appropriateness of the analysis with regard to the decision problem.

5.10.6 The standard subgroup analyses performed in RCTs or systematic reviews seek to determine whether there are differences in relative treatment effects between subgroups (through the analysis of interactions between the effectiveness of the technology and patient characteristics). The possibility of differences emerging by chance, particularly when multiple subgroups are reported, is high and should be taken into account. Pre-specification of a particular subgroup in the study or review protocol, with a clear rationale for anticipating a difference in efficacy and a prediction of the direction of the effect, will increase the credibility of a subgroup analysis.

5.10.7 In considering subgroup analyses, the Appraisal Committee will take specific note of the biological or clinical plausibility of a subgroup effect in addition to the strength of the evidence in favour of such an effect (for example, if it has a clear, pre-specified rationale and is consistent across studies). The evidence supporting biological or clinical plausibility for a subgroup effect should be fully documented, including details of statistical analysis.

5.10.8 Individual patient data are preferred, if available, for the estimation of subgroup-specific parameters. However, as for all evidence, the appropriateness of such data will always be assessed by considering factors such as the quality of the analysis, how representative the available evidence is to clinical practice and how relevant it is to the decision problem.

5.10.9 Consideration of subgroups based on differential cost may be appropriate in some circumstances; for example, if the cost of managing a particular complication of treatment is known to be different in a specific subgroup.

5.10.10 When considering subgroups, the Appraisal Committee pays particular attention to its legal obligations on equality and human rights.

5.10.11 Types of subgroups that are not considered relevant are those based solely on the following factors:

5.10.12 Analysis of 'treatment continuation rules', whereby cost effectiveness is maximised based on continuing treatment only in those who achieve a specified 'response' within a given time, should not be analysed as a separate subgroup. Rather, the strategy involving the 'continuation rule' should be analysed as a separate scenario, by considering it as an additional treatment strategy alongside the base-case interventions and comparators. This enables the costs and health consequences of factors such as any additional monitoring associated with the continuation rule to be incorporated into the economic analysis. Additional considerations for continuation rules include:

5.11 Presentation of data and results

5.12 Impact on the NHS