Methods for the development of NICE public health guidance (third edition)

1 Does the review address an appropriate and clearly-focused question that is relevant to the guidance topic's research question/s?

If the research question/s addressed by the review is not clearly stated it will be difficult to determine whether the review is adequate to answer the question addressed by the guidance topic. If the question is not clear, it is unlikely to be a good review as it is difficult to be systematic in addressing an unclear question. The review should give a clear description of the population considered; the interventions included; comparators; and outcomes evaluated. Inclusion and exclusion criteria should be clearly described. Outcomes considered should be clearly described within the methodology, including a precise definition and how validated. The appropriateness of the question addressed in the review for answering the research question/s considered within the guidance can be determined by comparing these components. If the review does not consider all of the outcomes that are judged to be important to the evidence review, the outcome data presented may be able to be used, although the individual studies may also need to be reviewed in order to obtain other outcome data.

2 Does the review include the types of study/ies relevant to the key research question/s?

You should be clear about the characteristics of studies that you consider will adequately address your guidance question/s. This may relate to minimum design or quality characteristics, for example, randomised trials only. Reviews should report the types of studies they sought, including any inclusion/exclusion criteria used, and you can use this to quickly assess the review's suitability for your purposes.

3 Is the literature search sufficiently rigorous to identify all the relevant studies?

Systematic and rigorous searches to identify as much relevant data from as many sources as possible can help to minimise publication biases. Exact search terms depend on the question, but there may be core databases that should be searched for every question, these databases are likely to be different for each guidance topic. Hand-searching of key journals may suggest a good quality review, and good quality reviews should examine reference lists of retrieved studies for further references. If the methods used to locate studies are not clearly reported, it will be difficult to determine whether the review is likely to have missed important, relevant studies. Ideally, the search strategy used should be reported in sufficient detail so that the process could be replicated.

Any restrictions applied to the search (for example, language, or year of publication) should also be reported. You should consider how these might influence the findings of the review.

Advice from an information scientist working on the guidance may be useful to decide whether any important search terms have been omitted.

If the search is judged to be inadequate to identify all relevant studies, it may be possible to expand the search by including additional databases or extra search terms within the search strategy, or by updating the search to identify more recently published studies. Any additional studies identified by this expanded search should then follow the assessment process for primary studies.

4 Is the study quality appropriately assessed and reported?

The inclusion of low quality studies within a review can result in biased estimates of effect. A well-conducted systematic review should have used appropriate and clear criteria to assess whether individual studies had been appropriately designed and conducted before deciding whether to include or exclude them. These criteria should be clearly described and should be reported for each included study. The other quality appraisal checklists in the appendices can be used as a guide to the types of quality criteria that should be considered.

If there is no indication of such an assessment, the review is unlikely to be reliable enough to be used in formulating recommendations. In such circumstances the relevant primary studies from the review should be obtained and quality appraised as part of the evidence review.

5 Is an adequate description of the analytical methodology used included, and are the methods used appropriate to the question?

In common with primary research, the approach used to analyse the data should be described and justified where appropriate. This may include the choice of statistic used to analyse the outcome data, meta-analytic techniques, and approaches to dealing with heterogeneity including the specification of any subgroup analyses and sensitivity analyses.

Year of publication

Typically CPHE advocate including primary studies from a specified date onwards. However, a review published since that date may include older primary studies. A decision will need to be made as to how to process such reviews. An example of an approach that has been used is to accept the review if >80% of the primary studies covered by the review meet the specified topic inclusion criteria. An alternative approach could be to specify a different year of publication for review-level material compared to year of publication for primary studies.

Inclusion/exclusion criteria

Reviews may well cover criteria that accord both with the guidance topic/scope inclusion criteria and its exclusion criteria. For example, the guidance topic referral and scope are explicit that the inclusion criteria are males and females under 16 years of age and will only focus on non-treatment-related outcomes. However, a review paper's criteria may cover adolescents aged up to 25 years and include treatment as a component of the interventions delivered. In such situations a decision will need to be made as to how to process such review papers. Is the review included but it is clearly documented that it also includes studies which cover 1 or more the topics/scope's exclusion criteria and therefore any findings associated with this review need to be treated with caution? Or is the review automatically excluded if the data relevant to the guidance topic is not sufficiently disaggregated? Or is the review only used as a source to identify additional primary studies which meet the specified inclusion criteria?

Double counting of studies between reviews and primary studies

If review-level material and primary studies are quality assessed and data extracted it is likely that a number of the primary studies included in the evidence review will also be reported in the review-level material. In such circumstances it is important that a clear description of the number of overlapping studies and an outline of any limitations associated with the overlap should be provided (for example, the potential for over-emphasising study findings through double counting, or the possibility that different conclusions or interpretations of the primary study's findings can be made).

Double counting of studies across review papers

Where review-level material has been quality assessed and data extracted there may be a significant overlap between the primary studies included across different review-level material. In such circumstances it is important that a clear description of the number of overlapping studies and an outline of any limitations associated with the overlap should be provided (for example, where different reviews which include in the main the same primary studies, reaching different conclusions about the evidence).

Whatever approaches are undertaken to address the above, these need to be clearly described within the methods section of the evidence review and any limitations associated with the selected approach clearly documented.