Developing NICE guidelines: the manual

Scoping searches

Scoping searches are top-level searches to support scope development. The purpose of the searches is to investigate the current evidence around the topic, and to identify any areas where an evidence review may be beneficial and any research gaps. The results of the searches are used to draft the scope of the upcoming guideline or update and to inform the discussions at scoping workshops (if held). Scoping searches do not aim to be exhaustive.

In some cases, scoping searches are not required when it is more efficient to use the surveillance review (see the chapter on the scope).

The sources searched at scoping stage will vary according to the topic, type of review questions the guideline or update will seek to address, and type of evidence sought. Each scoping search is tailored using combinations of the following types of information:

All scoping searches are fully documented and if new issues are identified at a scoping workshop, the search is updated. A range of possible sources considered for scoping searches is provided in the appendix on suggested sources for scoping.

Health inequalities searches

The purpose of these searches is to identify evidence to help inform the scope, health inequalities briefing, or the equality and health inequalities assessment (EHIA). They help identify key issues relevant to health inequalities on the topic, for example covering protected characteristics, groups experiencing or at risk of inequalities, or wider determinants of health.

The searches involve finding key data sources, such as routinely available national databases, audits or published reports by charities, non-governmental bodies, or government organisations.

Surveillance searches

Surveillance determines whether published recommendations remain current. The searches are tailored to the evidence required. This may include searches for new or updated policies, legislation, guidance from other organisations, or ongoing studies in the area covered by the evidence review.

If required, published evidence is identified by searching a range of bibliographic databases relevant to the topic. Surveillance searches generally use the same core set of databases used during the development of the original evidence review. A list of sources is given in the appendix on sources for evidence reviews.

The search approach and sources will vary between topics and may include:

Searches usually focus on randomised controlled trials and systematic reviews, although other study types will be considered where appropriate, for example for diagnostic questions.

The search period starts at either the end of the search for the last update of a guideline evidence review, or at the last search date for any previous surveillance check. Where appropriate, living evidence surveillance could be set up to continuously monitor the publication of new evidence over a period of time until impact reaches the threshold for actions. For more information on NICE guideline recommendation surveillance, see the chapter on ensuring that guideline recommendations are current and accurate and appendix on surveillance - interim principles for monitoring approaches of guideline recommendations.

Search protocols

Search protocols form part of the wider guideline review protocol (see the appendix on the review protocol template). They pre‑define how the evidence is identified and provide a basis for developing the search strategies.

Once the final scope is agreed, the information specialist develops the search protocols and agrees them with the development team before the evidence search begins.

A search protocol includes the following elements:

Sources

Searches are done on a mix of bibliographic databases, websites and other sources, depending on the subject of the review question and the type of evidence sought.

For most searches there are key sources that are prioritised, and other potentially relevant sources that can be considered. It is important to ensure adequate coverage of the relevant literature and to search a range of sources. However, there are practical limits to the number of sources that can be searched in the standard time available for an evidence review.

The selection of sources varies according to the requirements of the review question.

Clinical intervention sources

For reviews of the effectiveness of clinical interventions the following sources are prioritised for searching:

Clinical safety sources

In addition to the sources searched for clinical interventions, the following should be prioritised for clinical safety review questions:

Antimicrobial resistance sources

For reviews of antimicrobial resistance, the following sources should be prioritised:

Cost-effectiveness sources

For reviews of cost effectiveness, economic databases are used in combination with general bibliographic databases, such as MEDLINE and Embase (see appendix G on sources for economic reviews).

Economic evaluations of social care interventions may be published in journals that are not identified through standard searches. Targeted searches based on references of key articles and contacting authors can be considered to identify relevant papers.

Topic-specific sources

Some topics we cover may require the use of topic-specific sources. Examples include:

Searching for model inputs

Evidence searches may be needed to inform design-oriented conceptual models. Examples include precise searches to find representative NHS costs for an intervention or finding out the proportion of people offered an intervention who take up the offer.

Some model inputs, such as costs, use national sources such as national list prices or national audit data. In some cases, it may be more appropriate to identify costs from the academic literature. Further advice on methods to identify model inputs are also informed by Paisley (2016) and Kaltenhaler et al. (2011). See also the chapter on incorporating economic evaluation.

Real-world data

Information specialists can identify sources of real-world data (such as electronic health records, registries, and audits) for data analysts to explore further. The Health Data Research Innovation Gateway can be used to identify datasets. The NICE real-world evidence framework (2022) has additional guidance on searching for and selecting real-world data sources.

Grey literature

For some review questions, for example, where significant evidence is likely to be published in non-journal sources and there is a paucity of evidence in published journal sources, it may be appropriate to search for grey literature. Useful sources of grey literature include:

Committee members may also be able to suggest additional appropriate sources for grey literature.

A list containing potential relevant sources is provided in the appendix on sources for evidence reviews.

Developing search strategies

The approach to devising and structuring search strategies is informed by the review protocol. The PICO (population, intervention, comparator and outcome) or SPICE (setting, perspective, intervention, comparison, evaluation) frameworks may be used to structure a search strategy for intervention review questions. For other types of review questions, alternative frameworks may be more suitable.

It is sometimes more efficient to conduct a single search for multiple review questions, rather than conducting a separate search for each question.

Some topics may not easily lend themselves to PICO- or SPICE-type frameworks. In these cases, it may be better to combine multiple, shorter searches rather than attempting to capture the entire topic using a single search. This is often referred to as multi-stranded searching.

In some instances, for example where the terminology around a topic is diffuse or ill defined, it may be difficult to specify the most appropriate search terms in advance. In these cases, an iterative approach to searching can be used.

In an iterative approach, searching is done in several stages, with each search considering the evidence that has already been retrieved (for example, see Booth et al. 2020). Searching in stages allows the reviewers to review the most relevant, high-quality information first and then make decisions for identifying additional evidence if needed.

Decisions to use iterative approaches are agreed by the development team and staff with responsibility for quality assurance because it can affect timelines.

Updating previous work

Where high-quality review-level evidence is available on a topic, the review team may choose to update or expand this previous work rather than duplicating the existing findings. In these cases, the original review searches are re-run and expanded to account for any differences in scope and inclusion criteria between the original review and the update.

Cost-effectiveness searches

There are several methods that can be used to identify economic evaluations:

The rationale for the selected approach is recorded in the search protocol.

Where searches are needed to populate an economic model, these are usually done separately.

Identifying search terms

Search terms usually consist of a combination of subject headings and free‑text terms from the titles and abstracts of relevant references.

When identifying subject headings, variations in thesaurus and indexing terms for each database should be considered, for example MeSH (Medical Subject Headings) in MEDLINE and Emtree in Embase. Not all databases have indexing terms and some contain records that have not yet been indexed.

Free‑text terms may include synonyms, acronyms and abbreviations, spelling variants, old and new terminology, brand and generic medicine names, and lay and medical terminology.

For updates, previous search terms, including those from surveillance searches, are reviewed and used to inform new search terms. New or changed terms are identified, as well as any changes to indexing terms. This also applies when an existing review, for example a Cochrane review, is being updated to answer a review question.

Key studies can be a useful source of search terms, as can reports, guidelines, topic-specific websites, committee members and topic experts.

Some websites and databases have limited search functionality. It may be necessary to use fewer search terms or do multiple searches of the same resource with different search term combinations.

It may be helpful to use frequency analysis or text mining to develop the search-term strategy. Tools such as PubReMiner and Medline Ranker can help, either by highlighting search terms that might not otherwise be apparent, or by flagging terms of high value when exhaustive synonym searching is unfeasible or inadvisable.

Search limits

The application of limits to search strategies will reflect the eligibility criteria in the review protocol. Typically, English language limits, date limits, and the exclusion of conference abstracts and animal studies are usually done as a matter of routine.

Search filters

A search filter is a string of search terms with known (validated) performance. When a particular study design is required for a review question, relevant search filters are usually applied to literature search strategies.

Other search filters relating to age, setting, geography, and health inequalities are also applied as relevant. The most comprehensive list of available search filters is the search filter resource of the InterTASC Information Specialists' SubGroup. This resource also includes critical appraisal tools, which are used for filter selection.

Economics-related filters

A variety of search filters of relevance to cost effectiveness are available. These include filters for economic evaluations, quality of life data, and cost-utilities data. It may be necessary to use more than 1 filter to identify relevant data. In addition, it may be appropriate to add geographic search filters, such as those for the UK or Organisation for Economic Co-operation and Development (OECD) countries, to retrieve economic studies relevant to the UK or OECD (Ayiku et al. 2017, 2019, 2021).

Use of machine learning-based classifiers

Machine learning-based classification software has been developed for some study types (for example the Cochrane RCT classifier, Thomas et al. 2020). These classifiers apply a probability weighting to each bibliographical reference within a set of search results. The weighting relates to the reference's likelihood to be a particular study type, based on a model created from analysis of known, relevant papers. The weightings can then be used to either order references for screening or be used with a fixed cut-off value to divide a list of references into those more likely to be included, and those that can be excluded without manual screening.

We support the use of machine classifiers if their performance characteristics are known, and if they improve efficiency in the search and screening process. However, caution is needed when using classifiers, because they may not be as effective if used on data that is different to the type of data for which they were originally developed. For example, the Cochrane RCT classifier is reported to have over 99% recall for health studies but showed "unacceptably low" recall for educational research (Stansfield et al. 2022).

Priority screening, a type of machine classifier that orders references for manual sifting based on previous sifting decisions, is considered in the chapter on reviewing evidence.

Additional search techniques

Additional search techniques are used alongside database searching when it is known, or reasonably likely, that relevant evidence is not indexed in bibliographic databases, or when it will be difficult to retrieve relevant evidence from databases in a way that adequately balances recall and precision. Additional search techniques include forward and backward citation searching, journal hand-searches and contacting experts and stakeholders.

Existing reviews may provide an additional source of primary studies, with reference lists being used as an indirect method of identifying primary research.

Various tools, including Citationchaser and Web of Science, are available to speed up the process of citation searching. These may not be as comprehensive as manual reference list checking (due to limitations of the underlying data sources), but the trade-off in terms of speed is generally acceptable.

All search techniques should follow the same principles of transparency, rigour and reproducibility as other search methods.

If possible, additional search techniques should be considered at the outset and documented in the search protocol. They should also be documented in the supporting appendices for the final evidence review.