6.6 Presenting evidence for reviews other than reviews of primary studies

The principles described above remain relevant when reporting evidence not based on systematic reviews of primary studies done by NICE. A description of some of these alternative approaches and when they may be appropriate is given in the chapter on developing review questions and planning the evidence review. However, additional factors need to be considered in many of these situations and are described in this section. When reviews have used either multiple options described in this section or an option combined with a systematic review of primary studies, the different approaches should be reported separately according to the appropriate reporting approach outlined in this chapter. A description of how these sources of evidence were either combined or interpreted together by the committee should also be given.

Reporting reviews based on a published systematic review or qualitative evidence synthesis

In some cases, evidence reviews may be based on previously published systematic reviews or qualitative evidence syntheses done outside of NICE, rather than an original review. In such cases, where that review is publicly available, presentation of review content in NICE evidence reviews should be limited to those sections where additional material or analysis has been undertaken. If a published and free to access review has been used with no adaptation, it should be cited in the relevant sections and appendices of the NICE evidence review and a hyperlink to the original review provided, with no reproduction of the review content. If the review used is not free to access, then the relevant content should be summarised within the guideline.

Examples of additions that may be made to published reviews include adding new data to an out-of-date review, including additional outcomes or subgroups, re-analysing data using different statistical strategies, re-evaluating GRADE quality assessments, and combining separate reviews in a network meta-analysis. If a review has been updated by NICE to include additional material or analysis, a link should be provided to the relevant original review with a full citation in line with the NICE style guide on referencing and citations. Only the relevant updated sections should be written up in the NICE evidence review.

An evidence summary should still be provided in the evidence review, which makes clear which parts of the cited reviews were used as evidence within the guideline, and summarises any changes or additional analyses undertaken, if relevant. When considering the confidence we have in the findings of a published review, both the quality of the overall review (as assessed using the checklists recommended in the appendix on appraisal checklists, evidence tables, GRADE and economic profiles), and the quality of the studies within that review should be taken into account.

Reporting reviews based on a published individual participant data meta-analysis

Evidence reviews based on a published individual patient data (IPD) meta-analysis should follow the same principles as reviews based on other published systematic reviews. Reviewers can make use of the PRISMA-IPD checklist to assess the reporting standards of published IPD analyses, and Wang 2021 includes a checklist that can be used for quality assessment of IPD meta-analyses.

In most cases it is not possible to update an IPD meta-analysis within a guideline, and therefore an approach must be decided if there are additional relevant studies not included within the analysis (for example, additional studies published after the searches in the published review). A number of possible approaches can be followed:

Only include the IPD meta-analysis in the review, and exclude any additional studies.
Include the IPD meta-analysis review, and additionally report aggregated results for the studies not included in the IPD analysis.
Include the IPD meta-analysis review, and additionally report aggregated results for all studies within the review, both those included within the IPD meta-analysis and those not included.

The approach taken should be described and justified within the review. It should take into account the number and proportion of studies not included in the IPD meta-analysis, whether those studies are systematically different to the studies included, and whether the studies not included would be likely to lead to different overall conclusions.

Reporting reviews based on multiple published systematic reviews or qualitative evidence syntheses

Sometimes an evidence review may report the results of multiple systematic reviews, either as a result of a review of reviews being done, or because multiple relevant reviews are otherwise identified. Each review should be reported following the advice in the section on reporting reviews based on a published systematic review or qualitative evidence synthesis.

Additionally, the evidence review should report on any overlaps between the included reviews (for example, where multiple included reviews cover the same intervention or include some of the same studies), or any important differences between the methodologies of the included reviews. How these overlaps or differences were dealt with when assessing evidence and making recommendations should be reported.

Reporting reviews based on formal consensus methods

When formal consensus methods, such as Delphi panels or nominal group technique, are used as a way of generating or interpreting evidence, at minimum the following information should be reported in the evidence review:

How the participants involved in the formal consensus exercise were selected.
How the initial evidence or statements presented as part of the formal consensus exercise were derived.
The methodology used for the formal consensus exercises, including any thresholds used for retaining or discarding statements.
The results of each round or iteration of the formal consensus exercise.
How the results of the formal consensus exercise were then used to inform the recommendations made.

Reporting reviews based on previously published guidance

If systematic reviews or qualitative evidence syntheses done as part of a published non-NICE guideline are used as evidence within a NICE guideline, those reviews should be assessed following the advice in the section above on reporting reviews based on a published systematic review or qualitative evidence synthesis. No assessment of other aspects of the guideline is needed, because only the evidence from the reviews is being used, not any other part of the non-NICE guideline.

If parts of the non-NICE guideline other than evidence reviews are used (for example, if the recommendations made are themselves used as evidence, not just the underlying reviews) then the guideline should be assessed for quality using the AGREE II instrument. There is no cut-off point for accepting or rejecting a guideline, and each committee needs to set its own parameters. These should be documented in the methods of the guideline, and the full results of the assessment included in the evidence review. In addition to the assessment of the quality of the guideline, the following should also be included in the review at minimum:

A summary of the content from the non-NICE guideline used to inform the NICE guideline (for example, the recommendations considered).
A description of the justifications presented in the non-NICE guideline (for example, why those recommendations were made).
A description of how the NICE committee interpreted that content, including any concerns about quality and applicability, and how it informed their own discussions and recommendations.
A clear link between which parts of the non-NICE guideline informed the final recommendations in the NICE guideline.

Reporting reviews or using recommendations from previously published NICE guidelines

If systematic reviews or qualitative evidence syntheses done as part of published NICE guidelines are considered relevant and appropriate, they can be used as evidence within a different NICE guideline. These reviews can be included as part of the evidence when:

the review question in the guideline in development is sufficiently similar to the question addressed in the published guideline
the evidence base is unlikely to have changed significantly since the publication of the related published NICE evidence review.

When evidence reviews from another guideline are used to develop new recommendations, the decision should be made clear in the methods section of the guideline in development, and the committee's independent interpretation and discussion of the evidence should be documented in the discussion section. The evidence reviews from the published guideline (including review protocol, search strategy, evidence tables and full evidence profiles [if available]) should be included in the guideline in development. They then become part of the whole evidence base for the new guideline and are updated as needed in future updates of the guideline.

If parts of a published NICE guideline (or multiple guidelines) other than evidence reviews are used (for example, if recommendations made are themselves used as evidence, not just the underlying reviews) and new recommendations are formulated, the committee's discussion and decision should be documented clearly in the review. This should include areas of agreement and difference with the committee for the published guideline (for example, in terms of key considerations – balance of benefits and harms or costs, and interpretation of the evidence).

The following should be included in the review at minimum:

A summary of the content from the published NICE guideline used to inform the guideline in development (for example, the recommendations considered).
A description of the justifications presented in the published NICE guideline (for example, why those recommendations were made).
A description of how the committee interpreted that content, including any concerns about applicability, and how it informed their own discussions and recommendations, including how the recommendations from the published NICE guideline were extrapolated to the guideline in development. It is not routinely necessary to do an assessment of the published NICE guideline using the AGREE II instrument. However, in certain circumstances such an assessment may be useful (for example, if it is an older NICE guideline that used different methods to those currently in use), and if an assessment is undertaken the results should be reported in the review.
A clear link between which parts of the published NICE guideline informed the final recommendations in the guideline in development and why new recommendations were warranted (including why the original recommendations could not be adopted without change).

Reporting reviews using primary analysis of real-world data

Reviewers should follow the advice in the NICE real-world evidence framework when reporting primary analyses of real-world data undertaken by NICE. At minimum, the level of detail provided should match that which would be provided in a published research article. It should also be enough to enable an independent researcher with access to the data to reproduce the study, interpret the results, and to fully understand the strengths and limitations of the study.

More information on what is required and links to relevant reporting tools are provided in the NICE real-world evidence framework.

Reporting reviews using calls for evidence or expert witnesses

If evidence for a review has been obtained using either a call for evidence or an expert witness, follow the reporting advice in the appendix on calls for evidence and expert witnesses.

Reporting reviews using additional consultation or commissioned primary research

If evidence for a review has been obtained using either additional consultation or commissioned primary research, follow the reporting advice in the appendix on additional consultation and commissioned primary research.

6.7 References and further reading

AGREE Collaboration (2003) Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project. Quality and Safety in Health Care 12: 18–23

Booth A, Lewin S, Glenton C. et al. (2018) Applying GRADE-CERQual to qualitative evidence synthesis findings–paper 7: understanding the potential impacts of dissemination bias. Implementation Sci 13:12

Brouwers M, Kho M, Browman G et al. for the AGREE Next Steps Consortium (2010) AGREE II: advancing guideline development, reporting and evaluation in healthcare. Canadian Medical Association Journal 182: E839–42

Caldwell D, Ades A, Dias S et al. (2016) A threshold analysis assessed the credibility of conclusions from network meta-analysis. Journal of Clinical Epidemiology 80: 68–76

Caldwell D, Welton N (2016) Approaches for synthesising complex mental health interventions in meta-analysis Evidence-Based Mental Health 19:16

Collins G, Reistma J, Altman D et al. (2015) Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): The TRIPOD Statement. Annals of Internal Medicine 162: 55–63

Colvin C, Garside R, Wainwright M et al. (2018) Applying GRADE-CERQual to qualitative evidence synthesis findings-paper 4: how to assess coherence. Implementation Sci 13:13

Glenton C, Carlsen B, Lewin S et al. (2018) Applying GRADE-CERQual to qualitative evidence synthesis findings—paper 4: how to assess coherence. Implementation Sci 13:14

Gough D, Oliver S, Thomas J, editors (2012) An introduction to systematic reviews. London: Sage

GRADE working group (2004) Grading quality of evidence and strength of recommendations. British Medical Journal 328: 1490–4

The GRADE series in the Journal of Clinical Epidemiology

Guyatt G, Oxman A, Schünemann H et al. (2011) GRADE guidelines: a new series of articles in the Journal of Clinical Epidemiology. Journal of Clinical Epidemiology 64: 380–2

Higgins J, Thomas J, Chandler J et al., editors (2021) Cochrane handbook for systematic reviews of interventions, version 6.2

Higgins J, López-López J, Becker B, et al (2019) Synthesising quantitative evidence in systematic reviews of complex health interventions BMJ Global Health 2019;4:e000858

Johnsen J, Biegel D, Shafran R (2000) Concept mapping in mental health: uses and adaptations. Evaluation and Programme Planning 23: 67–75

Lewin S, Bohren M, Rashidian A et al. (2018) Applying GRADE-CERQual to qualitative evidence synthesis findings—paper 2: how to make an overall CERQual assessment of confidence and create a Summary of Qualitative Findings table. Implementation Sci 13:10

Lewin S, Booth A, Glenton C et al. (2018) Applying GRADE-CERQual to qualitative evidence synthesis findings: introduction to the series. Implementation Sci 13:2

Lizarondo L, Stern C, Carrier J, et al. Chapter 8: Mixed methods systematic reviews. In: Aromataris E, Munn Z (Editors). JBI Manual for Evidence Synthesis. JBI, 2020. Available from https://synthesismanual.jbi.global. https://doi.org/10.46658/JBIMES-20-09

Moons K, Altman D, Reistma J et al. (2015) Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): explanation and elaboration. Annals of Internal Medicine 126: W1–W73

Munthe-Kaas H, Bohren M, Glenton C et al. (2018) Applying GRADE-CERQual to qualitative evidence synthesis findings—paper 3: how to assess methodological limitations. Implementation Sci 13:9

NICE Decision Support Unit (2020) Sources and Synthesis of Evidence: Update to Evidence Synthesis Methods CHTE-2020_final_20April2020_final.pdf (sheffield.ac.uk) [online; accessed 31 March 2022]

NICE Decision Support Unit Evidence synthesis TSD series [online; accessed 31 August 2018]

Noyes J, Booth A, Lewin S et al. (2018) Applying GRADE-CERQual to qualitative evidence synthesis findings–paper 6: how to assess relevance of the data (nih.gov). Implementation Sci 13:4

Phillippo D, Dias S, Ades A et al. (2017) Sensitivity of treatment recommendations to bias in network meta-analysis. Journal of the Royal Statistical Society; Series A

Phillippo D, Dias S, Welton N et al. (2019) Threshold Analysis as an Alternative to GRADE for Assessing Confidence in Guideline Recommendations Based on Network Meta-analyses. Annals of Internal Medicine 170(8): 538-46

Puhan M, Schünemann H, Murad M et al. (2014) A GRADE working group approach for rating the quality of treatment effect estimates from network meta-analysis. British Medical Journal 349: g5630

Salanti G, Del Giovane C, Chaimani A et al. (2014) Evaluating the quality of evidence from a network meta-analysis. PloS one. 9(7): e99682

Thomas J, O'Mara-Eves A, Brunton G. (2014) Using qualitative comparative analysis (QCA) in systematic reviews of complex interventions: a worked example. Systematic Reviews 3:67

Thomas J, Petticrew M, Noyes J, et al. Chapter 17: Intervention complexity. In: Higgins JPT, Thomas J, Chandler J et al (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.3 (updated February 2022). Cochrane, 2022 Chapter 17: Intervention complexity | Cochrane Training [online; accessed 31 March 2022]

Viswanathan M, McPheeters M, Murad M et al. (2017) AHRQ series on complex intervention systematic reviews—paper 4: selecting analytic approaches - Journal of Clinical Epidemiology. J Clin Epidemiol 90:28

Wang H, Chen Y, Lin T, et al. (2021) The methodological quality of individual participant data meta-analysis on intervention effects: systematic review BMJ 373:n736

Welton N, Caldwell D, Adamopoulos E, et al. (2009) Mixed Treatment Comparison Meta-Analysis of Complex Interventions: Psychological Interventions in Coronary Heart Disease. Am J Epidemiol 169:1158

Whiting P, Rutjes A, Westwood M et al. and the QUADAS‑2 group (2011) QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Annals of Internal Medicine 155:529

How are you taking part in this consultation?

6 Reviewing evidence