The guidelines manual

The review addresses an appropriate and clearly focused question that is relevant to the guideline review question

If the question addressed by the systematic review is not clearly stated, it will be difficult to determine whether the review is adequate to answer the guideline review question. If the question is not clear, the systematic review is unlikely to be a good one because it difficult to be systematic in addressing an unclear question. The review report should give a clear description of the population considered, the interventions, exposures or tests evaluated, comparators, and outcomes evaluated. Inclusion and exclusion criteria should be clearly described. Outcomes considered should be clearly described within the methodology, including a precise definition and acceptable methods of measuring. The appropriateness of the question addressed in the systematic review for answering the guideline review question can be determined by comparing these components. If the review does not consider all of the outcomes that are judged to be important to your guideline review question, you may still be able to use the outcome data but may need to review the individual studies to obtain other outcome data.

The review collects the type of studies you consider relevant to the guideline review question

You should be clear about the characteristics of studies that you consider will adequately address your guideline review question. These may relate to minimum design or quality characteristics (for example, randomised trials only). Systematic reviews should report the types of studies they sought, including any inclusion/exclusion criteria used. You can use this information to quickly assess the review's suitability for your purpose.

The literature search is sufficiently rigorous to identify all the relevant studies

Systematic and rigorous searches can help to minimise publication biases and identify as many relevant data as possible. Exact search terms depend on the review question, but there are core databases that should have been searched for every question. As a minimum, a well-conducted review should look at Embase and MEDLINE. For questions about interventions in particular, the Cochrane Database of Systematic Reviews (CDSR), the Database of Abstracts of Reviews of Effects (DARE)and the Cochrane Central Register of Controlled Trials (CENTRAL) should also be searched. The dates on which the searches were carried out should be given in the review. Good-quality reviews will also attempt to identify relevant studies by handsearching of key journals and examining reference lists of retrieved studies for further references.

If the methods used to locate studies are not clearly reported, it will be difficult to determine whether the review is likely to have missed important relevant studies. Ideally, the search strategy used should be reported in sufficient detail that the process could be replicated.

Any restrictions applied to the search (such as language or year of publication) should also be reported. You should consider how these might have influenced the findings of the review.

Advice from the information specialist (and/or other members of the Guideline Development Group) working on the guideline may be useful to determine whether any important search terms have been omitted.

If the search described in the review is judged to be inadequate to identify all relevant studies, it may be possible to expand the search by including additional databases or extra search terms within the search strategy, or by updating the search to identify more recently published studies. Any additional studies identified by this expanded search should be appraised for quality using the appropriate NICE checklist (see appendices C–I). They should appear individually in separate rows in an evidence table.

Study quality is assessed and reported

The inclusion of poor-quality studies within a review can result in biased estimates of effect. A well-conducted systematic review should have used clear criteria to assess whether individual studies had been appropriately designed and conducted, before deciding whether to include or exclude them. These criteria should be clearly described and should be reported for each study included. The quality appraisal checklists in appendices C–I, as appropriate for the type of question and study design, can be used as a guide to the types of quality criteria that should be considered.

If there is no indication of such a quality assessment, the review is unlikely to be reliable enough to be used in formulating guideline recommendations. It may be necessary to obtain and quality appraise the individual studies as part of your review.

An adequate description of the methodology used is included, and the methods used are appropriate to the question

In common with primary research, the approach used to analyse the data should be described and justified where appropriate. This may include the choice of statistical test used to analyse the outcome data, meta-analytical techniques and approaches to dealing with heterogeneity, including the specification of any subgroup analyses and sensitivity analyses.