The social care guidance manual

1.1 Is the study population appropriate for the review question?

The study population should be defined as precisely as possible and should be in line with that specified in the guidance scope and any related review protocols.

This includes consideration of appropriate subgroups that require special attention. For many interventions, the capacity to benefit will differ for participants with differing characteristics. This should be explored separately for each relevant subgroup as part of the base-case analysis by the provision of estimates of effectiveness and cost effectiveness. The characteristics of participants or communities in each subgroup should be clearly defined and, ideally, should be identified on the basis of an a priori expectation of differential effectiveness or cost effectiveness as a result of biologically, sociologically or economically plausible known mechanisms, social characteristics or other clearly justified factors.

Answer 'yes' if the study population is fully in line with that in the review question(s) and if the study differentiates appropriately between important subgroups. Answer 'partly' if the study population is similar to that in the review question(s) but: (i) it differs in some important respects; or (ii) the study fails to differentiate between important subgroups. Answer 'no' if the study population is substantively different from that in the review question(s).

1.2 Are the interventions and services appropriate for the review question?

All relevant alternatives should be included, as specified in the guidance scope and any related review protocols. These should include routine and best practice in UK social care, existing NICE guidance and other feasible options.

Answer 'yes' if the analysis includes all options considered relevant for the review question, even if it also includes other options that are not relevant. Answer 'partly' if the analysis omits 1 or more relevant options but still contains comparisons likely to be useful for the guidance. Answer 'no' if the analysis does not contain any relevant comparisons.

1.3 Is the social care system in which the study was conducted sufficiently similar to the current UK social care context?

This relates to the overall structure of the social care system within which the interventions were delivered. For example, an intervention might be delivered on a residential basis in one country whereas in the UK it would be provided in the community. This might significantly influence the use of resources and costs, thus limiting the applicability of the results to a UK setting. In addition, old UK studies may be severely limited in terms of their relevance to current practice.

Answer 'yes' if the study was conducted within the UK and is sufficiently recent to reflect current practice. For non-UK or older UK studies, answer 'partly' if differences in the setting are unlikely to substantively change the cost-effectiveness estimates. Answer 'no' if the setting is so different that the results are unlikely to be applicable in the current social care context.

1.4 Are the perspectives clearly stated, and what are they?

The decision-making perspective of an economic evaluation determines the range of costs that should be included in the analysis. For social care guidance, one perspective that will usually be used is that of the public sector organisations (such as local authorities) delivering the interventions. Sometimes costs will be borne and benefits will accrue outside the public sector. When they are borne or accrue predominantly by other public sectors agencies, it will also be appropriate to use a public sector perspective. For social care topics the importance of the value of unpaid care in contributing to outcomes may be an important element of the cost perspective. In topics where interventions have a material effect on employment, the perspective may also need to reflect that. Where cost effectiveness using a narrower perspective is clearly established, however, the requirement to embrace a wider perspective is much reduced. Answer 'yes' if the study clearly and correctly states the perspective used, and whether that perspective is appropriate. Answer 'partly' if the perspective stated is not the perspective used. Answer 'no' if the study does not state the perspective or that the perspective is not appropriate.

1.5 Are all direct effects on individuals included, and are all other effects included where they are material?

For a personal social services (PSS) (and where appropriate, an NHS perspective), outcomes should include all direct effects, whether for individuals directly affected or, when relevant, other people (often other family members or carers). This is consistent with an objective of maximising benefits from available public sector resources. Any significant characteristics of social care provision that have a value to people that is independent of any direct effect on outcomes should be noted. These characteristics include the convenience with which social care is provided and the level of information available for service users and carers.

Where the perspective is wider (a public sector perspective), outcomes may, where relevant, include not just the individuals directly targeted but also their families, friends and the community in general.

If a wider public sector perspective is used, answer 'yes' if the measure of outcome excludes effects that are not directly related to the social care intervention (or if such effects can be excluded from the results). Answer 'partly' if the analysis from a public sector perspective includes some non-social care effects but these are small and unlikely to change the cost-effectiveness results. Answer 'no' if the analysis incorrectly includes or excludes significant non-social care effects that are likely to change the cost-effectiveness results for a particular perspective.

If a societal perspective is used, answer 'yes' if the measure of outcome includes non-social care effects. Answer 'partly' if the analysis includes some non-social care effects but these are small and unlikely to change the cost-effectiveness results. Answer 'no' if the analysis incorrectly includes or excludes significant non-social care effects that are likely to change the cost-effectiveness results for a particular perspective.

1.6 Are all future costs and outcomes discounted appropriately?

The need to discount to a present value is widely accepted in economic evaluation, although the specific rate is variable across jurisdictions and over time. NICE considers that it is usually appropriate to discount costs and health effects at the same rate. The annual rate of 3.5%, based on the recommendations of the UK Treasury for the discounting of costs, should be applied to both costs and effects. Sensitivity analyses using rates of 1.5% for both costs and effects may be presented alongside the reference-case analysis.

Answer 'yes' if both costs and effects are discounted at 3.5% per year (or at another rate considered appropriate). Answer 'partly' if costs and health effects are discounted at a rate similar to the rate considered appropriate (for example, costs and effects are both discounted at 3% per year where the appropriate rate is 3.5%). Answer 'no' if costs and/or health effects are not discounted, or if they are discounted at a rate (or rates) different from the rate considered appropriate (for example, 5% for both costs and effects, or 6% for costs and 1.5% for effects where the appropriate rate is 3.5%). Note in the comments column what discount rates have been used. If all costs and health effects accrue within a short time (roughly a year), answer 'NA'.

1.7 How is the value of effects expressed?

The QALY is a measure of a person's length of life weighted by a valuation of their health-related quality of life (HRQoL) over that period. For social care, the QALY may not be the most appropriate measure of effects; other measures based on social care-related quality of life or capability may be used.

Answer 'yes' if the effectiveness of the intervention is measured using QALYs or an appropriate social care-related equivalent; answer 'no' if not. Use the comments column to describe the measure of effects used. There may be circumstances when such measures cannot be obtained or where the underlying assumptions are considered inappropriate. In such situations answer 'no', but consider retaining the study for appraisal. Similarly, answer 'no' but retain the study for appraisal if it does not include appropriate measures of effects but is still thought to be useful for GDG decision-making: for example, if the evidence indicates that an intervention might be dominant, and estimates of the relative costs of the interventions from a cost-minimisation study are likely to be useful. When economic evaluations not using appropriate measures of effects are retained for full critical appraisal, use the comments column to note why.

1.8 Are costs and outcomes from other sectors (including the value of unpaid care, where relevant) fully and appropriately measured and valued?

Studies in social care often include costs accruing to other sectors of the economy or benefits gained by these sectors. Not all of these benefits can be translated into measures of effects (for example, the ability to return to work earlier). Answer 'yes' if all the costs and all the benefits have been included, if they are appropriately measured and if they are appropriately valued. Answer 'partly' if omissions are not material and answer 'no' if some major cost or benefit is omitted, is improperly measured or improperly valued. Use the comments column to describe costs and outcomes relating to other sectors or unpaid care.

1.9 Overall judgement

Classify the applicability of the economic evaluation to the social care guidance, the current social care situation and the context for NICE guidance as 1 of the following:

2.1 Does the model structure adequately reflect the nature of the topic under evaluation?

This relates to the choice of model and its structural elements (including cycle length in discrete time models, if appropriate). Model type and its structural aspects should be consistent with a coherent theory of the social care needs under evaluation. The selection of care pathways, whether individual states or branches in a decision tree, should be based on the underlying biological, sociological or economic processes of the topic under study and the potential impact (benefits and adverse consequences) of the intervention(s) of interest.

Answer 'yes' if the model design and assumptions appropriately reflect the health condition and intervention(s) of interest. Answer 'partly' if there are aspects of the model design or assumptions that do not fully reflect the health condition or intervention(s) but these are unlikely to change the cost-effectiveness results. Answer 'no' if the model omits some important aspect of the health condition or intervention(s) and this is likely to change the cost-effectiveness results. Answer 'NA' for economic evaluations based on data from a study which do not extrapolate intervention outcomes or costs beyond the study context or follow-up period.

2.2 Is the time horizon sufficiently long to reflect all important differences in costs and outcomes?

The time horizon is the period of analysis of the study: the length of follow-up for participants in a trial-based evaluation, or the period of time over which the costs and outcomes for a cohort are tracked in a modelling study. This time horizon should always be the same for costs and outcomes, and should be long enough to include all relevant costs and outcomes relating to the intervention. A time horizon shorter than lifetime could be justified if there is no differential mortality effect between options, and the differences in costs, social care-related quality of life or other relevant outcomes relate to a relatively short period.

Answer 'yes' if the time horizon is sufficient to include all relevant costs and outcomes. Answer 'partly' if the time horizon may omit some relevant costs and outcomes but these are unlikely to change the cost-effectiveness results. Answer 'no' if the time horizon omits important costs and outcomes and this is likely to change the cost-effectiveness results.

2.3 Are all important and relevant outcomes included?

All relevant outcomes should include direct social care or other effects relating to harms from the intervention as well as any potential benefits.

Answer 'yes' if the analysis includes all relevant and important harms and benefits. Answer 'partly' if the analysis omits some harms or benefits but these would be unlikely to change the cost-effectiveness results. Answer 'no' if the analysis omits important harms and/or benefits that would be likely to change the cost-effectiveness results.

2.4 Are the estimates of baseline outcomes from the best available source?

The sources and methods for eliciting baseline probabilities should be described clearly. These data can be based on 'natural history' (outcomes in the absence of intervention), sourced from cohort studies. Baseline probabilities may also be derived from the control arms of experimental studies. Sometimes it may be necessary to rely on expert opinion for particular parameters.

Answer 'yes' if the estimates of baseline health outcomes reflect the best available evidence as identified from a recent well-conducted systematic review of the literature. Answer 'partly' if the estimates are not derived from a systematic review but are likely to reflect outcomes for the relevant group of people in England (for example, if they are derived from a large UK-relevant cohort study). Answer 'no' if the estimates are unlikely to reflect outcomes for the relevant group of people in England.

2.5 Are the estimates of relative intervention effects from the best available source?

Evidence on outcomes should be obtained from a systematic review, defined as the systematic location, inclusion, appraisal and synthesis of evidence to obtain a reliable and valid overview of the data relating to a clearly formulated question.

Synthesis of outcome data through meta-analysis is appropriate provided that there are sufficient relevant and valid data obtained using comparable measures of outcome.

Head-to-head randomised controlled trials (RCTs) provide the most valid evidence of the effects of interventions. However, such evidence may not always be available. Therefore, data from non-randomised studies may be required to supplement RCT data. Any potential bias arising from the design of the studies used in the assessment should be explored and documented.

When assessing multiple interventions that have not been compared within a single RCT, data from a series of pairwise head-to-head RCTs should be presented. Consideration should also be given to presenting a combined analysis using a mixed treatment comparison framework if it is considered to add information that is not available from the head-to-head comparison.

The principles of good practice for standard meta-analyses should also be followed in mixed and indirect treatment comparisons.

The methods and assumptions that are used to extrapolate short-term results to final outcomes should be clearly presented.

Answer 'yes' if the estimates of the effect of intervention appropriately reflect all relevant studies of the best available quality, as identified through a recent well-conducted systematic review of the literature. Answer 'partly' if the estimates of the effect of intervention are not derived from a systematic review but are similar in magnitude to the best available estimates (for example, if the economic evaluation is based on a single large study with effects similar to pooled estimates from all relevant studies). Answer 'no' if the estimates of the effect of intervention are likely to differ substantively from the best available estimates.

2.6 Are all important and relevant costs included?

Costs related to the topic of interest and incurred in additional years of life gained as a result of intervention should be included in the base-case analysis. Costs that are considered to be unrelated to the topic or intervention of interest should be excluded. If introduction of the intervention requires additional infrastructure to be put in place, consideration should be given to including such costs in the analysis.

Answer 'yes' if all important and relevant resource use and costs are included given the perspective and the research question in the economic study under consideration. Answer 'partly' if some relevant resource items are omitted but these are unlikely to affect the cost-effectiveness results. Answer 'no' if important resource items are omitted and these are likely to affect the cost-effectiveness results.

2.7 Are the estimates of resource use from the best available source?

It is important to quantify the effect of the interventions on resource use in terms of physical units (for example, days in care or contacts with social care practitioners) and valuing those effects in monetary terms using appropriate prices and unit costs. Evidence on resource use should be identified systematically. When expert opinion is used as a source of information, any formal methods used to elicit these data should be clearly reported.

Answer 'yes' if the estimates of resource use appropriately reflect all relevant evidence sources of the best available quality, as identified through a recent well-conducted systematic review of the literature. Answer 'partly' if the estimates of resource use are not derived from a systematic review but are similar in magnitude to the best available estimates. Answer 'no' if the estimates of resource use are likely to differ substantively from the best available estimates.

2.8 Are the unit costs of resources from the best available source?

Resources should be valued using the prices relevant to the agencies that deliver the interventions. A first point of reference in identifying costs and prices should be any current official listing published by relevant government departments.

When the acquisition price paid for a resource differs from the public list price, the public list price should be used in the base-case analysis. Sensitivity analysis should assess the implications of variations from this price. When cost data are taken from the literature, the methods used to identify the sources should be defined. When several alternative sources are available, a justification for the costs chosen should be provided and discrepancies between the sources explained. When appropriate, sensitivity analysis should have been undertaken to assess the implications for results of using alternative data sources.

Answer 'yes' if resources are valued using up-to-date prices relevant to the appropriate sectors. Answer 'partly' if the valuations of some resource items differ from current relevant unit costs but this is unlikely to change the cost-effectiveness results. Answer 'no' if the valuations of some resource items differ substantively from current relevant unit costs and this is likely to change the cost-effectiveness results.

2.9 Is an appropriate incremental analysis presented or can it be calculated from the data?

An appropriate incremental analysis is one that compares the expected costs and outcomes of one intervention with the expected costs and outcomes of the next-best non-dominated alternative.

Standard decision rules should be followed when combining costs and effects, and should reflect any situation where there is dominance or extended dominance. When there is a trade-off between costs and effects, the results should be presented as an incremental cost-effectiveness ratio (ICER): the ratio of the difference in mean costs to the difference in mean outcomes of a technology compared with the next best alternative. Where benefits are expressed as QALYs, in addition to ICERs, expected net monetary or health benefits can be presented using values placed on a QALY gained of £20,000 and £30,000. However, it may not be possible to place such values on other measures of benefits that are used in social care economic evaluation.

For cost-consequences analyses, appropriate incremental analysis can only be done by selecting one of the consequences as the primary measure of effectiveness, providing the consequences are independent of one another.

Answer 'yes' if appropriate incremental results are presented, or if data are presented that allow the reader to calculate the incremental results. Answer 'no' if: (i) simple ratios of costs to effects are presented for each alternative compared with a standard intervention; or (ii) if options subject to simple or extended dominance are not excluded from the incremental analyses.

2.10 Are all important parameters whose values are uncertain subjected to appropriate sensitivity analysis?

There are a number of potential selection biases and uncertainties in any evaluation (trial- or model-based) and these should be identified and quantified where possible. There are 3 types of bias or uncertainty to consider:

Answer 'yes' if an extensive sensitivity analysis was undertaken that explored all key uncertainties in the economic evaluation. Answer 'partly' if the sensitivity analysis failed to explore some important uncertainties in the economic evaluation. Answer 'no' if the sensitivity analysis was very limited and omitted consideration of a number of important uncertainties, or if the range of values or distributions around parameters considered in the sensitivity analysis were not reported.

2.11 Is there any potential conflict of interest?

The British Medical Journal (BMJ) defines competing interests for its authors as follows: "A competing interest exists when professional judgment concerning a primary interest (such as patients' welfare or the validity of research) may be influenced by a secondary interest (such as financial gain or personal rivalry). It may arise for the authors of a BMJ article when they have a financial interest that may influence, probably without their knowing, their interpretation of their results or those of others."

Whenever a potential financial conflict of interest is possible, this should be declared.

Answer 'yes' if the authors declare that they have no financial conflicts of interest. Answer 'no' if clear financial conflicts of interest are declared or apparent (for example, from the stated affiliation of the authors). Answer 'unclear' if the article does not indicate whether or not there are financial conflicts of interest.

2.12 Overall assessment

The overall methodological study quality of the economic evaluation should be classified as 1 of the following: