The social care guidance manual

6.1.1 Research studies

Before acquiring papers for assessment, the information specialist or systematic reviewer should sift the evidence identified in the search and discard irrelevant material. First, the titles of the retrieved citations should be scanned and those that fall outside the topic of the guidance should be excluded. A quick check of the abstracts of the remaining papers should identify those that are clearly not relevant to the review questions and can be excluded.

Next, the remaining abstracts should be scrutinised against the inclusion and exclusion criteria agreed by the Guidance Development Group (GDG). Abstracts that do not meet the inclusion criteria should be excluded. Any doubts about inclusion should be resolved by discussion with the GDG before the results of the study are considered. Once the sifting is complete, full versions of the selected studies can be acquired for assessment. Studies that fail to meet the inclusion criteria once the full version has been checked should be excluded; those that meet the criteria can be assessed. Because there is always a potential for error and bias in selecting the evidence, double sifting (that is, sifting by 2 people) of a random selection of abstracts should be performed periodically (Edwards et al. 2002).

6.1.2 Conference abstracts

Conference abstracts can be a good source of information in systematic reviews. For example, conference abstracts can be important in identifying published trials that may be missed and ongoing trials that are due to be published, or in estimating the amount of not-fully-published evidence (and so guiding calls for evidence and judgements about publication bias). However, the following should be considered when deciding whether to include conference abstracts as a source of evidence:

Therefore:

6.1.3 Economic evaluations

The process for sifting and selecting economic evaluations for assessment is essentially the same as for social care studies. Consultation between the information specialist, the economist and the systematic reviewer is essential when deciding the inclusion criteria; these decisions should be discussed and agreed with the GDG. The review should be targeted to identify the papers that are most relevant to current practice and to GDG decision-making. The review should also usually focus on 'full' economic evaluations that compare both the costs and consequences of the alternative interventions and any services under consideration.

Inclusion criteria for filtering and selection of papers for review by the economist should specify relevant populations and interventions for the review question. They should also specify the following:

6.2.1 Introduction

This section applies to the assessment of both qualitative and quantitative evidence.

The review team should assess the quality of evidence selected for inclusion in the review using the appropriate quality appraisal checklist (see section 6.2.2). This is a key stage in the guidance development process because the quality rating of studies will be reflected in the evidence statements (see section 6.4). These, in turn, are taken into account in the recommendations (along with other factors and considerations, see section 9.2).

Some of the more commonly used study types and their abbreviations are:

The quality of individual studies should be assessed using an appropriate quality appraisal checklist. This is to make a judgement about both the quality of execution of the study and its fitness-for-purpose in terms of answering the review question(s). Factors that influence judgements on the 'trustworthiness' of the study, such as its relevance to the review questions and how 'convincing' the results are, should be clearly described in the review.

Some studies, particularly those using mixed methods, may report quantitative, qualitative and economic outcomes. In such cases, each aspect of the study should be separately assessed using the appropriate checklist.

Similarly, a study may assess the effectiveness of an intervention using different outcome measures, some of which are more reliable than others (for example, self-reported levels of school attendance compared with a formal measure of attendance from the school). In such cases, the study might be rated differently for each outcome, depending on the reliability of the measures used. For further information on how to integrate evidence from qualitative and quantitative studies, see Dixon-Woods et al. (2004).

6.2.2 Quality assessment

Quality assessment is a critical stage of the evidence review process.

6.3.1 Data extraction and evidence tables

The evidence tables can also be used as data extraction templates for included studies.

Evidence tables can help determine whether it is possible to calculate a summary estimate of effect, if applicable (see Conducting and presenting a meta-analysis in section 6.3.4).

6.3.2 Narrative summaries of quantitative or qualitative studies

The narrative summary provides an opportunity to place a study and its findings in context. It should highlight key factors influencing the results observed, and give an interpretation of the results and more on the detail presented in the evidence tables (see section 6.3.1).

The narrative summary should conclude with a short discussion, followed by 1 or more evidence statements. These should reflect the key findings, the quantity, quality and consistency of the evidence, and its applicability to the review question (including its applicability to the target population).

Narrative summaries of all studies and interventions should be incorporated in the main findings of the evidence review. They should be organised by review question and could be divided into smaller subcategories, such as outcome measure, setting or subpopulation. The summary should be brief and, where possible, use tables or other methods to summarise and present key elements or features of the evidence.

6.3.3 Summary tables

If appropriate, short summary tables can be included with the main findings (usually preceding an evidence statement) or in the appendices. For example, these might:

6.3.4 Other presentations of quantitative data

There are a range of ways to summarise and illustrate the strength and direction of quantitative evidence about the effectiveness of an intervention. Some of the most commonly used methods are described below, although this is not an exhaustive list.

6.3.5 Other presentations of qualitative data based on analytic and structured techniques

The nature of qualitative evidence is such that it is unhelpful to set a prescriptive method for its synthesis and description. Qualitative evidence occurs in many forms and formats.

In some cases, the evidence may be synthesised and then summarised. In other cases, a narrative description may be adequate. The approach used depends on the volume and consistency of the evidence. If the qualitative literature is extensive, then a synthetic approach is preferable. If the evidence is more disparate and sparse, a descriptive approach may be more appropriate.

6.4.1 Introduction

This section applies to both qualitative and quantitative reviews. As described in section 6.3.2, each evidence review should include a narrative summary and should conclude with a short discussion and 1 or more supporting evidence statements.

The evidence statements should reflect the strength (quality, quantity and consistency) of the evidence and make a statement about its applicability. They may also highlight a lack of evidence. They should provide an aggregated summary of the evidence (from 1 or more studies) in relation to a key question or issue. In the case of intervention studies, they should also reflect what is plausible, given the evidence available about what has worked in similar circumstances.

Evidence statements are structured and written to help the GDG formulate and prioritise recommendations. They help it decide:

Evidence statements that support the recommendations should be included in the final guidance document.

6.4.2 Structure and content of evidence statements

One or more evidence statements are prepared for each review question or its subsidiary questions. (Subsidiary questions may cover a type of intervention, specific population groups, a setting or an outcome.)

Once all the data have been collected, consideration should be given on how to group the evidence. For example, it could be grouped according to the similarity of the populations, interventions and outcomes covered in the studies.

However, the decision will be highly context-specific and will depend on the amount, breadth and depth of evidence. A separate evidence statement for each study should be avoided. Evidence statements based on so many studies that the statement becomes too generic and therefore meaningless are also to be avoided.

Short evidence statements should be presented, by outcome where possible, summarising the key features of the evidence on effectiveness (including harms as appropriate) and cost effectiveness.

The evidence statements should include the number of studies and participants, the quality of the evidence and the direction of estimate of the effect. An evidence statement may be needed even if no evidence is identified for an important outcome. Evidence statements may also note the presence of relevant ongoing research.

6.5.1 NICE guidance

Recommendations taken from published NICE guidance should be quoted verbatim. Published NICE guidance should be fully referenced and the evidence underpinning the recommendations left unchanged, provided it is not out of date.

6.5.2 Other published guidance

Relevant published guidance from other organisations may be identified in the search for evidence. If these are not from NICE accredited sources, they should be assessed for quality using the AGREE II (appraisal of guidelines research and evaluation II) instrument (Brouwers et al. 2010). The aim is to ensure they have sufficient documentation to be considered.

There is no cut-off point for accepting or rejecting a piece of guidance, and each GDG will need to set its own parameters. These should be documented in the methods section of the guidance, along with a summary of the assessment. The results should be presented as an appendix to the guidance.

Reviews of evidence from other guidance that cover questions formulated by the GDG may be considered as evidence if:

The GDG should create its own evidence summaries or statements to include in the standard template. Evidence tables from other guidance should be referenced with a direct link to the source website or a full reference of the published document. The GDG should formulate its own recommendations, taking into consideration the whole body of evidence.

Recommendations from other guidance should not be quoted verbatim, except for recommendations from Department of Health or Department for Education social care policy or legislation.

6.6.1 Are the evidence-review criteria inclusive?

All relevant inequalities data should be included in the reviews. At the data extraction stage, reviewers are prompted to refer to the PROGRESS-Plus criteria (age, sex, sexual orientation, disability, family origin, religion, place of residence, occupation, education, socioeconomic position and social capital; Oliver et al. 2008). Review inclusion and exclusion criteria should also take the relevant groups into account.

6.6.2 Have relevant data been appropriately extracted and presented in the evidence statements?

Equalities evidence should be considered during the drafting of reviews. It should be included in the data extraction process and should appear in the summary evidence statements.

6.6.3 What is the state of the evidence base?

This question aims to identify whether there are any gaps in the evidence in relation to inequalities. It also aims to identify whether the evidence has uncovered gaps in the scope of the guidance in relation to inequalities.