NICE process and methods
6 Developing the evidence summary
6 Developing the evidence summary
6.1 Equality and diversity
Evidence summaries are developed in line with NICE's equality scheme and declaration of interests policy.
6.2 Process overview and timelines
The key steps in developing evidence summaries are summarised below. The process reflects their status as advice products and when the advice is needed.
Timelines are agreed for each evidence summary during scoping. It usually takes 12 weeks to produce a standard evidence summary. A longer timeline may be agreed if a large volume of evidence is expected from the literature search.
NICE will consider developing a rapid evidence summary within a shorter time, if needed. Some steps in the process may be omitted or shortened and this will be explained in the published evidence review (see section 6.6). The rapid evidence summary will be concise; as a minimum, resource impact will not be assessed.
The key steps in developing an evidence summary are:
-
develop and agree final scope and population, intervention, comparator and outcomes (PICO) with clinical advisers and external topic experts as needed and/or the organisation commissioning the topic
-
send search request to information services
-
contact the company, when applicable, with data request
-
search for evidence
-
sift and select the evidence
-
appraise and summarise the evidence
-
produce initial draft documents
-
do internal technical check of revised draft documents
-
send revised draft documents to company (if applicable), external topic experts, targeted stakeholders, other relevant NICE teams (including resource impact assessment if applicable) and Medicines and Healthcare products Regulatory Agency (MHRA; if applicable) for review
-
review comments received and produce revised draft documents for sign off (including advisory statement, if applicable)
-
content sign off by associate director, clinical adviser or programme director
-
review comments received and produce revised draft documents for NICE's guidance executive or publications executive
-
if applicable, company invited to check for any factual errors and informed of publication date
-
submit final draft documents for guidance executive or publications executive sign off
-
sign off by guidance executive or publications executive
-
review comments received and produce final documents
-
submit to the organisation commissioning the topic, if applicable
-
publication on the NICE website.
6.3 Scoping
A scope is developed by either NICE or the organisation commissioning the topic, depending on how the topic has been referred to NICE. If it is developed by NICE, clinical advisers and external topic experts advise on the content as needed. The scope outlines the key review questions, PICO, and inclusion and exclusion criteria for the review. The final scope is signed off by NICE or the organisation commissioning the topic.
During scoping the following are confirmed:
-
key contacts at the company (if applicable; see section 6.4)
-
key contacts at the MHRA (if applicable)
-
external topic experts
-
stakeholder organisations
-
literature search terms (see section 6.5).
Options within the process, such as whether an advisory statement and resource impact tools will be produced, are also agreed during scoping.
6.4 Information from companies
When appropriate NICE asks companies to provide information, which may include:
-
key published clinical trials for the indication being reviewed
-
ongoing or recently completed studies that have not yet published in full
-
regulatory status, including whether or not the company (or another company) expects to hold a UK marketing authorisation for the medicine for this indication within the next 2 years
-
likely licensing and marketing timeline
-
the usual dose, or best estimate from the available data
-
the presentation of the medicine, including form, strength and pack size
-
incidence and prevalence of the indication, alternative treatments and estimated usage and cost.
6.5 Literature search
6.5.1 Searching for evidence
NICE's information services do a literature search according to the agreed scope and PICO. The aim is to find the best available evidence on the effectiveness, safety and resource impact of the medicine. In exceptional circumstances, the literature search may include preprints from medRxiv and bioRxiv, for example during a public health emergency.
The search strategy and quality assurance of the search process is included as an appendix in the evidence review.
6.5.2 Selecting the evidence
Evidence identified from the literature search is reviewed to find relevant primary research that addresses the use of the medicine within the defined indication and population under review. If robust systematic reviews of randomised controlled trials (RCTs) or RCTs are available, they form the basis of the review. However, the best available evidence may include evidence other than RCTs, such as observational studies.
First sift
The first sift reviews the title and abstract of the study against the scope and PICO and removes evidence of low relevance. This may include non-English language studies, or conference abstracts or studies that have not been published in full (because these cannot be critically appraised). Note that preprints may be considered for inclusion in exceptional circumstances.
Second sift
The second sift of full papers further excludes articles that do not meet the criteria in the scope.
When all relevant studies have been identified, the best available evidence is selected for inclusion in the evidence review. Usually no more than 3 studies are prioritised for inclusion, using these principles:
-
systematic reviews of RCTs are prioritised first, followed by single RCTs
-
if 1 or more systematic reviews or RCTs are included, lower-quality studies (for example cohort or case-control studies, or case series) would only be included if they provide additional data on outcomes not available from the higher-quality studies
-
if further prioritisation is needed, other factors would be considered such as:
-
size of study (number of study participants)
-
date of publication
-
how well the data are reported
-
whether an active comparator was used, and whether this reflects usual UK practice
-
whether the population in the study reflects the typical UK population for which this medicine is likely to be used.
-
If no relevant evidence is identified, the development team will consider if broadening the search to include a wider population may provide useful information for decision making.
A summary of included studies and those studies excluded at second sift (with reasons for non-inclusion) are included as appendices in the evidence review.
Relevant regulatory information such as a European public assessment report (EPAR) or national public assessment report (if this has been published) are also reported to supplement the included studies, if needed.
6.5.3 Appraising the prioritised evidence
The development team appraises the included studies to assess risk of bias or quality of studies using a NICE quality appraisal checklist suitable for the type of evidence being reviewed. This quality assessment is included in an appendix in the evidence review.
6.6 Writing the evidence review
The development team drafts the evidence review document using a standard NICE template. A modified template may be used depending on the organisation commissioning the topic, for example to support development of a commissioning policy.
The evidence review will usually include the following sections (some may be omitted or shortened if a rapid evidence summary is developed):
Background
A short summary of the clinical problem, current practice and any existing related NICE guidance.
Product overview
A brief discussion of the medicine(s), including:
-
mode of action
-
regulatory status
-
dosing information
-
antimicrobial resistance (if the evidence summary is for an antimicrobial, relevant information, usually from the English surveillance programme for antimicrobial utilisation and resistance report or summary of product characteristics [SPC], is included).
Effectiveness
A short narrative summary of the clinical effectiveness evidence, usually presented by outcome or by clinical question. Meta-analyses will not be carried out.
Safety
A short narrative summary of the evidence on safety outcomes. Background safety information from other sources may also be included, such as:
-
the SPC or EPAR for precautions, warnings and undesirable effects
-
published advice from medicines regulators.
Person-centred factors
A summary of person-centred factors that may be important for decision making, such as medicines adherence issues, based on a review of the evidence.
Limitations of the evidence
A critical review of the strengths and weaknesses of the relevant evidence.
Resource implications
Cost-effectiveness reviews are not carried out. Basic cost information is obtained from:
If it is agreed during scoping that resource impact tools will be developed, NICE produces these following the processes to estimate the resource impact of NICE guidance.
Development of the evidence review
A link to this process guide including any deviations from the usual process, for example if some sections have been omitted or shortened because of a rapid development timeline. Details of the external topic experts and their declarations of interest are also included.
6.7 Writing the evidence summary
The development team drafts the shorter evidence summary document, based on the content of the more detailed evidence review. This summary includes the following sections, although an advisory statement is included only if a commissioning policy is not being developed, and this has been agreed during scoping:
Advisory statement
A short evidence‑based statement about the medicine's likely place in therapy (which does not include a cost-effectiveness review), developed with input from external topic experts. It is advisory only: it is not a recommendation or formal NICE guidance.
Likely place in therapy
The development team summarises the medicine's likely place in therapy by bringing together all the key factors for decision making, in the context of the wider evidence base for managing the condition, particularly if NICE guidance is available.
The development team also takes into account the responses to questions (based on user feedback) put to the external topic experts after they review the draft evidence summary documents (including any supporting resources):
-
Which patients would be considered for this medicine?
-
Is there a particular care setting that would be best for this medicine (for example secondary care)?
-
Would the practical use of the medicine in a real-world setting differ from trial populations and settings? If so, how?
-
Would there need to be a significant change to the care pathway to support adoption of this medicine? If so, give details.
-
Are there any specific barriers or levers to the adoption of this medicine? (Other than general issues such as existing national guidance, views of opinion leaders and professional societies, and the evidence on the medicine.)
-
Are there any specific implications for people using this medicine, or their carers, compared with current practice? (For example, how convenient is it to take? Are there increased monitoring visits?)
-
Are there any specific implications around commissioning and procurement?
-
Would use of the medicine affect finance or budgets and staff capacity or resources?
-
Where do you see the place in therapy for this medicine?
The development team may finalise the likely place in therapy at a meeting with the external topic experts.
If a commissioning policy is being developed, the likely place in therapy section will not be completed. There will be a hyperlink to the policy from the NICE website.
Factors for decision making
A summary of the most important factors to be considered to help inform decision making are taken from the evidence review. This includes key information on effectiveness and safety evidence, limitations of the evidence, person-centred factors and resource impact. If the medicine is an antimicrobial, this will also include considerations about good antimicrobial stewardship.
6.8 Targeted review
The development team sends the draft evidence summary documents to:
-
targeted stakeholder organisations
-
identified external topic experts
-
the organisation commissioning the topic, if applicable
-
the company, if applicable
-
the MHRA, if applicable.
Any comments received are considered when revising the drafts. Actions are also recorded. Feedback to commentators is available on request to NICE.
6.9 Quality assurance
The evidence summary is quality assured by NICE staff who are not part of the development team. This involves a detailed check of all content, to ensure all sections contain statements and conclusions that are fair and balanced. The evidence summary must accurately reflect the evidence reviewed and be substantiated by an explicit and appropriate source of evidence.
The evidence summary is signed off by the programme director, clinical adviser or associate director.
If applicable, the company is given the opportunity to review the near-final draft to check for any factual errors (1 working day). Any necessary corrections are made by the development team.
NICE's guidance executive or publications executive reviews the evidence summary and, if appropriate, approves it for publication, ensuring that the process has been followed in its development. If applicable, the company is informed of the scheduled publication date, and may request an embargoed copy to be sent to them 24 hours before publication.
6.10 Publication
The final evidence summary documents (both the evidence review and the shorter evidence summary) are published on the NICE website. For published evidence summaries, see the NICE website. An alert to the published evidence summary is also circulated through the medicines and prescribing alert service.