5 Identifying the evidence: literature searching and evidence submission

5.1 Introduction

The systematic identification of evidence is an essential step in clinical guideline development. Systematic literature searches undertaken to identify evidence of clinical and cost effectiveness should be thorough, transparent and reproducible. These searches will also minimise 'dissemination biases' (Song et al. 2000), such as publication bias and database bias, that may affect the results of reviews.

This chapter is aimed primarily at information specialists in the National Collaborating Centres (NCCs) and in NICE^[7]. It provides advice on the sources to search and on how to develop strategies for systematic literature searches to identify clinical and economic evidence. It also provides advice on other areas of information management that form an important part of the clinical guideline development process. These include using reference management software, acquiring the full text of articles and documenting the search process. Calls for submissions of evidence from stakeholders and undertaking baseline assessments of service activity (for service guidance) are also covered. The scoping search undertaken when drafting the scope of a clinical guideline is described in section 2.3.3.

5.2 Searching for clinical evidence

5.2.1 Databases and other sources to search

The databases and other sources that should be searched to identify evidence of clinical effectiveness depend on the review question.

5.2.1.1 Core and subject-specific databases

The core databases listed in table 5.1 should be searched for every review question. Additional subject-specific databases and other resources may also need to be searched, depending on the subject area of the review question and the type of evidence sought. Links are provided in table 5.1 for sources that are freely available.

Table 5.1 Databases that should be searched

Question type

Databases

Review questions about interventions, diagnosis, prognosis^a, patient experience and service delivery

Core databases:

MEDLINE/MEDLINE In-Process

Embase

Cochrane Database of Systematic Reviews – CDSR (Cochrane Reviews)^b

Database of Abstracts of Reviews of Effects – DARE (Other Reviews)^c

Cochrane Central Register of Controlled Trials – CENTRAL (Clinical Trials)^b

Health Technology Assessment (HTA) database (Technology Assessments)^c

Subject-specific databases (this list is not exhaustive):

AMED (Allied and Complementary Medicine Database)

The Campbell Collaboration Library of Systematic Reviews

CINAHL (Cumulative Index to Nursing and Allied Health Literature)

ERIC (Education Resources Information Center)

PEDro (Physiotherapy Evidence Database)

PsycINFO^d

^aCDSR and DARE do not need to be searched for questions about prognosis.

^bAccessible via the Cochrane Library. Database names in parentheses are those used in the Cochrane Library. CENTRAL only needs to be searched when evidence from controlled trials is sought.

^cAccessible as part of the Cochrane Library and via the Centre for Reviews and Dissemination (CRD). The CRD website hosts the most up-to-date version of the databases. Database names in parentheses are those used in the Cochrane Library.

^d PsycINFO is searched as an additional core database by the NCC for Mental Health.

An awareness of the strengths and weaknesses of each database is important when undertaking a systematic literature search. The different databases index different journals, use different subject headings, cover different time periods and provide different amounts of bibliographic information. For example, Embase is considered to be stronger than MEDLINE in its coverage of the pharmacology, toxicology, drug research and psychiatric literature, but contains only selected coverage of the dental and nursing literature. On the other hand, MEDLINE contains a much better developed collection of scope notes for its subject heading (MeSH) terms, which can assist development of the search strategy. There will be overlap in the records retrieved from the different databases for a particular review question; the extent of this overlap for MEDLINE and Embase is reported as being between 10 and 87% depending on the topic (Lefebvre et al. 2008). Therefore cross-database searching, although time-consuming, is necessary in order to comprehensively identify evidence for clinical guideline development.

5.2.1.2 Other sources of information

The sources listed in table 5.2 – which include databases and websites – can provide useful information about ongoing research, patient experience, clinical audits and statistics to help guide Guideline Development Group (GDG) decision-making. This list is not intended to be exhaustive; the 'Searching for studies' chapter in the 'Cochrane handbook' offers a good overview and further examples of sources to search (Lefebvre et al. 2011).

Table 5.2 Other sources of information

Source	Website
International Standard Randomised Controlled Trial Number Register	www.controlled-trials.com/isrctn
International Clinical Trials Registry Platform (WHO)	http://apps.who.int/trialsearch/
IFPMA Clinical Trials Portal	http://clinicaltrials.ifpma.org
ClinicalTrials.gov (US National Institutes of Health service)	http://clinicaltrials.gov
UK Clinical Research Network (UKCRN) Study Portfolio database	http://public.ukcrn.org.uk/search
Web of Knowledge	www.isiwebofknowledge.com/
Conference Papers Index	www.csa.com/factsheets/cpi-set-c.php
The King's Fund	www.kingsfund.org.uk
DH-Data	http://ds.datastarweb.com/ds/products/datastar/sheets/dhss.htm
Hospital Episode Statistics	www.hesonline.nhs.uk
Patient Episode Database for Wales	www.wales.nhs.uk/sitesplus/922/page/50308
National or regional registers, for example cancer registers	Search by type of cancer to locate appropriate register
National or regional audits	Search by topic or geographical area for appropriate audit data.
Information about patient experiences	www.healthtalkonline.org www.youthhealthtalk.org
Surveys of patients' experiences	Search for relevant patient organisation websites; condition-specific or topic-specific as appropriate.

NCCs are not expected to routinely search other sources of information, and there is no requirement to hand search journals for studies.

5.2.2 How to search for clinical evidence

Many of the principles listed in this section are also relevant to searching for economic evidence (see section 5.3).

5.2.2.1 Devising an overall search strategy

Review questions can be broken down into different parts, which can then be used to devise a search strategy. For example, using the PICO (population, intervention, comparator and outcome) framework (see box 4.1), a search strategy can be constructed for terms relating to the population; this can be combined with terms relating to the interventions and comparators (if there are any) to be evaluated. It is important to remember that not all components of a review question will always be mentioned in the abstracts or subject headings of database records – in particular, outcomes are often not mentioned. Therefore it may not be advisable to include these components when developing a strategy. For guidelines that are being updated, previous strategies can be used to inform search strategy design.

5.2.2.2 Identifying search terms

Search strategies should usually consist of a combination of subject headings and 'free-text' terms from the titles and abstracts of relevant studies (see also section 5.2.2.3). Subject headings are used to identify the main theme of an article; however, not all conditions or diseases will have a subject heading, so it is important to use free-text terms too. When identifying subject headings it is important to include variations in thesaurus and indexing terms for each database; for example, MeSH in MEDLINE and the Cochrane Library, and Emtree in Embase. Free-text terms may include synonyms, acronyms, abbreviations, differences in terminology across national boundaries, different spellings, old and new terminology, brand and generic drug names, and lay and medical terminology. Misspellings or 'typos' may also affect a search, particularly with records in the process of being indexed, for which there may be only a title and no abstract or subject headings.

5.2.2.3 Sensitivity and precision

The key attributes of a search strategy are:

Sensitivity: the number of relevant records retrieved by a search strategy as a proportion of the total number of relevant records (normally represented by a gold standard) (Jenkins 2004).
Precision: the number of relevant records retrieved by a search strategy as a proportion of the total number of records retrieved (Jenkins 2004).

Both of these will be influenced by the time period covered and by the search terms used. Although it is important that searches for systematic reviews attempt to identify all the relevant literature, there needs to be a trade-off between conducting an exhaustive search that will need additional resources versus undertaking a more modest search that may miss some studies. Identifying key studies for a review question can assist in checking search sensitivity; such studies can also act as a guide to search terms.

5.2.2.4 Grouping review questions

It is useful to identify review questions that overlap and so can be grouped together for searching purposes. For example, questions about the most effective treatments for a condition may involve comparing several interventions. This may make it possible to carry out one search that covers all the interventions. Questions that have the population and intervention in common but a different comparator can be grouped together by identifying and combining search terms for the population and intervention only.

5.2.2.5 Limiting searches

Using certain parameters to limit searches can improve precision without unduly affecting sensitivity.

Date parameters. These depend on the clinical guideline topic and on when the majority of the research was published. The date range for the search should be agreed by the GDG, in consultation with experts in the area. If relevant good-quality published systematic reviews exist (see chapter 6), additional searching may be limited to updating the reviews, covering the time period since the searches for the published reviews were conducted. However, existing reviews may not address all of the relevant outcomes, in which case new searches may be needed. Consider contacting authors of published reviews for updates, particularly for reviews found in the Cochrane Database of Systematic Reviews.
Animal studies can be excluded from the search results in some databases. In Ovid, for MEDLINE the search strategy is:

1 Final search set

2 Exp Animals/ not Humans/

3 1 not 2.

If a decision has been taken to limit a review to studies reported in English, the appropriate database limit function can be used to improve precision.
Depending on the review question, it may be appropriate to limit searches to particular study designs. The best way to do this is to use an appropriate search filter rather than limiting searches by the publication type field (see sections 5.2.2.6 and 5.2.2.7).
Sometimes it may be appropriate to limit searches by age. This can be useful to identify citations relating to children, but is often not necessary for those relating to adults. A search filter is listed on the InterTASC website (see section 5.2.2.7).
Limiting searches by sex is not recommended.

5.2.2.6 Searching step-by-step by study design

For review questions on the effectiveness of interventions, it may be more efficient to search for systematic reviews, followed by randomised controlled trials (RCTs), followed by cohort or case–control studies. This will prevent unnecessary searching and review work. An absence of good-quality RCTs covering all the key outcomes may mean expanding the search to retrieve observational studies. The use of relevant search filters (see section 5.2.2.7) can help to identify study types and thus assist in this method of searching.

5.2.2.7 Search filters

Search filters can be used to make searching more efficient and effective by saving time and bringing consistency and focus to the searching process. Search filters may be developed using a range of research-based and non-research-based methods. The most reliable filters are likely to be those that describe explicit methods, including how the search terms were identified and combined, and how the performance of search strategies was tested using collections of relevant records (ideally different from the records used to identify or extract the search terms) (Jenkins 2004). Research-based filters for finding RCTs and other study designs include the Cochrane Highly Sensitive Search Strategies for identifying RCTs in MEDLINE (Lefebvre et al. 2011) and filters developed by the McMaster University Hedges team for MEDLINE and Embase. The most comprehensive listing of available search filters can be found on the NICE InterTASC Information Specialists' Sub-Group (ISSG) website, which lists filters by study design, database and interface.

When choosing a search filter, it is important to consider the age of the filter (to take account of changes such as indexing or interface changes), and whether it maximises sensitivity or precision. The most useful search filters for clinical guideline work are likely to be those for identifying specific study designs such as RCTs or economic evaluations.

5.3 Searching for economic evidence

The approach to searching for economic evidence should be systematic, but targeted to identify studies that are most relevant to current NHS practice and hence likely to inform GDG decision-making.

Two types of search might be required for economic evidence:

First, a systematic search for economic evaluations relevant to the guideline and applicable to current NHS practice should be performed. This should cover all review questions with potential cost or resource implications and should not be limited to the modelling priorities identified in the economic plan. This search should be conducted by the information specialist, in consultation with the health economist.
Additional searches may be necessary to identify other information required for economic modelling. This may include information about prognosis, adverse effects, quality of life, resource use or costs that is not always available from the clinical searches conducted for the guideline. The requirement for additional searches should be discussed by the information specialist and the health economist. (See section 7.2.3 for more details about identifying model inputs, including searching for quality-of-life data.)

Much of the advice provided in section 5.2.2 about how to search for clinical evidence is relevant to systematic searches for economic evaluations.

5.3.1 Initial search to identify economic evaluations

The majority of the search for economic evaluations should be completed near the beginning of the guideline development process as an initial broad search. The first step is a search of a key health economics database using the patient population terms, as for the initial clinical background search. Other core databases should then be searched for the patient population terms with the addition of a published economics search filter.

A suggested strategy for searching for economic evaluations in the initial broad search is:

NHS EED (NHS Economic Evaluation Database)^[8], and HEED (Health Economic Evaluations Database) if subscribed to – all years
HTA database – all years.

This initial broad search should be extended to identify recent papers that have not yet been referenced in the economics databases, by searching MEDLINE (including MEDLINE In-Process) and Embase with a published economics search filter (see section 5.2.2.7), covering the most recent complete year.

Search filters to identify economic evaluations can maximise precision (for example, the economics search filters developed and validated as having high precision by the McMaster Hedges team) or sensitivity (for example, the CRD search filter developed to identify economic evaluations for NHS EED). Information specialists should use their judgement about whether maximising precision or sensitivity is more appropriate when selecting search filters to identify economic evidence (see sections 5.2.2.3 and 5.2.2.7).

Other subject-specific databases may be searched at this stage, at the discretion of the information specialist.

5.3.2 Further searches to identify economic evaluations

Further searches for economic evaluations may be needed for some review questions. The purpose of these searches is to try to ensure that all relevant economic evaluations are identified; some may not be retrieved by the initial search because of the inclusion criteria of the economics databases (for example, economic evaluations indexed in Embase have been sought for inclusion in NHS EED only since 2002). The need for additional searches and the criteria (such as date parameters) for any additional searches should be established by the health economist in consultation with the information specialist. As a minimum, MEDLINE and Embase should be searched; additional databases should be searched as appropriate. It may also be worthwhile to use a highly sensitive economics search filter (for example, the CRD filter – see sections 5.2.2.7 and 5.3.1). The searches may be executed when required or alongside the clinical searches, depending on the preference of the health economist in consultation with the information specialist.

5.4 Quality assurance of search strategies

Efforts should be made to check the quality and accuracy of search strategies during the development of the clinical guideline. Although it will not usually be possible to check all strategies for every search, the following approaches can be used to ensure that the key studies are retrieved.

Ask GDG members to identify key clinical studies or economic evaluations that are already published, in order to gather useful search terms.
Check search strategies used in existing published systematic reviews.
Run searches with and without certain search terms and assess the differences between the results obtained.
Check the bibliographies of included studies to ensure that all relevant papers have been retrieved by the search strategy used.
If relevant papers have not been retrieved by the search strategy, investigate and amend the strategy if appropriate.

5.5 Reference management software

Electronic records of the references retrieved by searches should be stored using reference management software such as EndNote, Reference Manager or ProCite. Records can be exported from bibliographic databases such as MEDLINE and imported automatically into the software using import filters. Details of references can also be added manually.

In addition to storing records of references, consideration should be given to using reference management software for the following:

Coding the references with additional information, such as the source of the reference, the review question it was identified to answer, the study design and selection decisions. Coding should be determined and agreed by the NCC technical team before working with a reference management database to ensure consistency of use.
Providing links to the full text of articles, where possible.
Logging the ordering and/or receipt of articles.
Keeping track of the printed copies of papers.
Linking to word processing packages using output styles to facilitate the automatic generation of in-text citations and reference lists for the full version of the guideline.

5.6 Acquiring the full text of references

The full text of references can be obtained from several sources:

Free online journal articles: many journals provide free access to some or all of their content. Several apply this to all material more than 1 or 2 years old; others provide access to particular types of articles only (for example, the British Medical Journal provides free access to all research articles). Individual articles can be purchased from the websites of most journals that do not allow free access, but this can be expensive.
Some websites provide links to medical journal web pages with freely available articles. Two that are useful are Free Medical Journals and Genamics JournalSeek.
NHS Evidence and its Welsh equivalent, NHS Wales e-Library for health, provide free access to some journals for all NHS staff and staff in organisations such as NICE and the NCCs that work exclusively for the NHS. An Athens log-in is needed to access journal content provided by NHS Evidence, which can be obtained by applying to the Guidance Information Services team at NICE.
Free online reports: many institutions make their reports and guidelines freely available online, so it is worth checking the relevant websites.
Libraries: many libraries that stock a wide range of journals, books and reports will have an inter-library loan or document delivery service. All will supply articles within copyright law and some will loan documents. There is usually a charge for this service, and for loans the cost of postage is usually extra. Some libraries provide articles at a reduced cost if an annual subscription is taken out. Three major libraries offering this level of service are the British Library, the British Medical Association (BMA) Library and the Royal Society of Medicine Library. A British Library account also allows users to pay for articles from other libraries that accept payment in this way. Some of the NCCs are based in, or associated with, a medical institution that has its own library.

5.7 Documenting the search strategy

An audit trail should be kept of the searches for both clinical and economic evidence that are conducted during the clinical guideline development process, so that the process for identifying the evidence is transparent and reproducible.

5.7.1 Internal documentation

The following information should be recorded for each search conducted during the clinical guideline development process:

Details of the question for which the search was conducted.
The names of the databases and database host systems used.
The database coverage dates; for example, Ovid MEDLINE 1950 to February week 3 2012.
The date on which the search was conducted.
The search strategy (this should be stored in an easily accessible form such as Microsoft Word or ASCII plain text).
Any limits applied to the search or to study designs searched for.
The number of records retrieved from each database.
A text file and/or database (from EndNote, Reference Manager or ProCite) of results.

Enough detail should be provided to allow searches to be repeated when the guideline requires updating (see section 14.1).

5.7.2 Full guideline

A description of the searching process should be included in the methods section of the full version of the clinical guideline (see section 10.1.1). This should include:

details of the scoping search (see section 2.3.3)
details of the development of the search strategies
dates on which the searches were carried out, including any re-run searches (see section 5.9)
any limits placed on the type of evidence searched for and details of methodological search filters, if used
names of the databases and database host systems and any other sources searched
date or language limits applied to searches.

Search strategies will be published on the NICE website 5–7 weeks before consultation on the draft guideline starts, and will also be available to stakeholders during consultation. They should also be published at the same time as the final full guideline. It may be helpful to publish the search strategies for each literature search for all databases.

5.8 Timing of searches

Searches should be prioritised according to the clinical and economic evidence required for each GDG meeting. Additional searching time may be needed for guideline topics that involve a lot of pharmacological areas, for which there are likely to be large numbers of published papers. This should be taken into consideration early in the process and should be accounted for in the planning. Specific searches will need to be carried out for each of the review questions and the economic evidence that will be discussed at the planned GDG meetings.

5.9 Re-running searches

5.9.1 Clinical evidence

The searches undertaken to identify clinical evidence for each review question need to be re-run to identify any further evidence that has been published since the search was run initially. The final re-run of searches should be done 6–8 weeks before submission of the draft guideline to NICE. This can be done either by using database and website automatic alerting systems on each search or by executing re-runs of searches at one or two time points before the consultation.

Search strategies should be checked when re-running the search to ensure that all subject headings are still mapping to the appropriate heading, as these can change, and also to see if there are any new terms or headings that could be used (for example, MeSH headings are evaluated and can change annually). An awareness of how and when databases are indexed and updated should guide the re-run, because there may be times when indexing stops temporarily or when repetition of articles is more common. This can affect the value of re-running the search. It is worth noting that records identified by re-runs may not necessarily be 'new'. They may have been identified in the initial search in a different database that has a shorter indexing time lag, or they may have been identified in the same database but now have a revised entry date as a result of a revision of the indexing.

5.9.2 Economic evidence

As for clinical searches, economic evaluation literature searches should be re-run 6–8 weeks before submission of the draft guideline to NICE. The re-runs can be executed either question by question (that is, for the questions for which additional searches for economic evaluations were conducted) or, as a minimum, on the initial broad search only (see section 5.3.2). This will largely be determined by the requirements of the health economist. Re-runs of selective searches for model inputs may be repeated after guideline consultation, but only at the request of the health economist, who is able to determine whether there is time to incorporate any new information in a revised model (see also section 7.2.3). It is not usually necessary for the health economist to re-run other searches (for example, quality of life), but they should discuss this with the information specialist.

5.10 Calls for evidence from stakeholders

For some questions, the GDG and NCC staff may have good reason to believe that information exists that has not been found using standard searches. Examples include ongoing research in a field, if a technology is relatively new, studies that have been published only as abstracts (see section 6.1.2), data about the off-label use of drugs, data on adverse effects, economic models, and studies of the experiences of patients, carers or healthcare or other professionals.

In these situations, the NCC may call for evidence. This call goes to all registered stakeholders. It should specify the question being addressed and details of the type of evidence being sought, for example in terms of PICO framework and study design for questions of effectiveness. A call for evidence may be made at any point during development of a clinical guideline, and stakeholders should usually be given 4 weeks to respond. The NCC may choose not to issue any calls for evidence for a guideline.

If the NCC and GDG think it is likely that the regulatory authorities hold relevant data about a drug that has not been submitted in response to a call for evidence, they may approach the appropriate regulatory authority to release those data. This should be done through the Centre for Clinical Practice at NICE.

5.10.1 Confidential information

In addition to published studies, stakeholders may submit relevant unpublished data or studies in response to a call for evidence. When the NCC sends out a call for evidence, it should ask stakeholders that respond to complete a checklist that lists and identifies the location of all confidential information contained in their submission. This checklist is available from the NICE webboard for NCCs. The NCCs should keep the checklists for their records to ensure that the draft and final versions of the full guideline do not contain confidential information.

Box 5.1 summarises what may and may not be considered confidential by NICE.

Box 5.1 Information on what may and may not be considered confidential

Data that may be included as confidential include those that may influence share price values ('commercial in confidence') or are intellectual property ('academic in confidence'; that is, awaiting publication).

Confidential information should be kept to an absolute minimum; for example, just the relevant part of a sentence, a particular result from a table or a section of code.

NICE will not allow a whole study to be designated confidential. As a minimum, a structured abstract of the study or economic model will have to be made available for public disclosure during consultation on the guideline.

Results derived from calculations using confidential data will not be considered confidential unless releasing those results would enable back-calculation to the original confidential data.

In addition to completing the checklist, stakeholders should indicate the part of their submission that contains the confidential information, for example by using a highlighter pen on a hard copy, or the highlighter function in an electronic version. These markings should then be maintained on those sections so that the GDG knows which parts are confidential. When the draft and final versions of the full guideline are prepared for publication, the NCC should ensure that these sections are replaced by a note stating that confidential information has been removed, so that readers know exactly where confidential data have been used.

Following the principles in box 5.1, the amount of confidential information should be kept to a minimum. As a minimum, a summary should be publicly available by the time of the consultation on the guideline. NICE needs to be able to justify the recommendations in clinical guidelines on the basis of the evidence considered by the GDG. NICE and the NCC will therefore work with the data owners to agree a balance between confidentiality and transparency^[9].

5.10.2 Information not eligible for submission

Stakeholders are asked not to submit the types of evidence listed in box 5.2, as these will not be considered.

Box 5.2 Stakeholder material not eligible for consideration by the GDG

Studies with weak designs if better designed studies are available

Promotional literature

Papers, commentaries and editorials that interpret the results of a published paper

Representations and experiences of individuals (unless assessed as part of a well-designed study or survey)

5.10.3 Documenting evidence from stakeholder submissions

Information received from stakeholders in response to a call for evidence should be entered into a reference management database (as described in section 5.5), and the details cross-checked against evidence identified through database searching. It should be assessed in the same way as published studies identified through the searches (see section 6.2.1).

5.11 Additional requirements for service guidance

In addition to evidence identified through routine literature searches, the GDG requires information describing the current configuration of clinical services, the level of activity and any significant regional variations. This will help the GDG to:

identify the gaps between current clinical practice, service provision and patient experience and what the GDG concludes should be in place
shape the guidance and formulate recommendations that are likely to have the greatest impact on the service as well as on clinical outcomes.

A detailed baseline assessment of service activity is needed, and should be conducted before the GDG starts work. This should be available for consideration early in the guideline development process, and ideally early enough to inform the scope. The following data sources might be used in providing an overall picture of service configuration and activity:

hospital episode statistics (HES)
patient episode data Wales (PEDW)
national or regional registers (for example, cancer registers)
national or regional clinical audits
surveys of patients' experiences
morbidity statistics from general practice.

5.12 Further reading

Centre for Reviews and Dissemination (2009) Systematic reviews: CRD's guidance for undertaking reviews in health care. Centre for Reviews and Dissemination, University of York

Jenkins M (2004) Evaluation of methodological search filters – a review. Health Information and Libraries Journal 21: 148–63

Lefebvre C, Eisinga A, McDonald S et al. (2008) Enhancing access to reports of randomized trials published world-wide – the contribution of EMBASE records to the Cochrane Central Register of Controlled Trials (CENTRAL) in The Cochrane Library. Emerging Themes in Epidemiology 5: 13

Lefebvre C, Manheimer E, Glanville J (2011) Searching for studies. In: Higgins JPT, Green S, editors. Cochrane handbook for systematic reviews of interventions, version 5.1.0 (updated March 2011). The Cochrane Collaboration.

Song F, Eastwood AJ, Gilbody S et al. (2000) Publication and related biases. Health Technology Assessment 4: 1–115

The guidelines manual

NICE process and methods