NICE process and methods

4 Identifying the evidence

4 Identifying the evidence

4.1 Introduction

This chapter describes how evidence is identified for reviews of public health information. It outlines the stages of developing a robust, transparent search protocol and planning a search strategy. It also identifies a variety of sources to search and additional methods of finding relevant information. Whether evidence is identified by an internal search team from NICE or by an external search team, the same methods and processes still apply. In the text below, 'search team' applies to any search team identifying the evidence for the review.

The Centre for Public Health Excellence (CPHE) encourages the use of search methods that balance precision and sensitivity. The aim is to identify the best available evidence to address a particular public health question, without producing an unmanageable volume of results. This involves a forensic search that includes:

  • creating precise search questions and identifying the study types needed to answer those questions

  • matching key databases to the questions being asked (and not necessarily trawling all available databases just because they exist)

  • adopting a pragmatic and flexible approach that allows a continual review of how best to find evidence

  • having an understanding of the existing evidence base.

For the technicalities involved in developing a search strategy read 'Unit six: finding the evidence in systematic reviews of health promotion and public health interventions' (available from the Cochrane Public Health Group website). For technical advice on developing a search strategy for systematic reviews, read section 6.4 of the Cochrane handbook for systematic reviews of interventions (Lefebvre et al. 2011).

4.1.1 Issues to consider when searching for public health literature

Public health encompasses a wide range of disciplines including health, economics, psychology and the social sciences. Literature searches for public health evidence are not always straightforward:

  • Searches are often long and complex and can present a technical challenge because of the databases being searched.

  • Search strategies have their limitations. Public health information resources do not use a standard indexing vocabulary or thesaurus and the thesauruses used by clinical databases only cover a limited number of public health concepts. The use of natural language varies, and studies, outcomes, measures and populations are not described in a consistent way.

  • The broad multidisciplinary nature of public health means that searches are carried out across a wide range of databases – currently, there are no dedicated national databases that bring this information together.

  • There is a lack of particular types of evidence, such as controlled trials. This limits the methodological coverage of systematic reviews if the review process follows the imperatives of evidence-based clinical medicine.

The CPHE project team will support innovative and flexible approaches to searching, as it is not possible to know in advance where the best available evidence or interventions of interest are likely to be located. The aim is to retrieve a relevant and manageable set of results. The logic model developed when the initial briefing paper was drawn up (or when the scope was developed) should help inform the approach taken.

All search processes should be transparent, clearly documented and reproducible. They should also be agreed with the CPHE project team.

The search process itself should be as comprehensive as possible, bearing in mind time and resource limitations. The type of search strategy used will depend on the logic model and the scope questions (see chapter 2). It will also depend on the type of review required: effectiveness, cost effectiveness, correlates and qualitative reviews all require different types of evidence.

4.1.2 Procedure

The search stage consists of 3 phases:

  • constructing the search protocol

  • developing the search strategy

  • gathering the evidence, conducting searches and documenting the process.

About 1 week should be allocated for each phase. The following sections describe the stages and processes for each phase. The principles set out below will be the same, whether the protocol identifies the need for an exhaustive or a more flexible search.

4.2 Phase 1: constructing the search protocol

4.2.1 Introduction

The search team should develop a search protocol from the review protocol (see section 3.4.1) and this should be agreed with the CPHE project team and the review team. The search protocol sets out how evidence will be identified and provides a basis for the search team to develop a detailed search strategy. Development of the search protocol is likely to be an iterative process involving the CPHE project team, the search team and the review team. The search protocol is normally added as an appendix to each review protocol (see chapter 3).

The search protocol should clearly state the:

  • search approach taken (for example, systematic/exhaustive, emergent)

  • rationale for the chosen approach

  • search question(s) and key concepts

  • electronic sources that will be searched (core, additional and economic databases plus any websites)

  • plans for any additional searches (for example, citation or hand-searching) (see section 4.4.2)

  • main study types that will be identified (for example, primary, review-level)

  • inclusion and exclusion criteria

  • restrictions (if any) on searches.

4.2.2 Electronic resources

Identifying public health evidence involves searching a wide range of electronic resources, in addition to the traditional scientific and clinical databases. The list of information sources should be individually tailored for each review to ensure they are relevant to the guidance topic. This list should be agreed between the search team, the review team and the CPHE project team.

Searches should include a mix of: core databases, subject-specific databases and other resources, depending on the subject of the research question and the level of evidence sought.

4.2.2.1 Databases

The databases searched must be relevant to the public health topic in terms of their coverage and content. Where there are a large number of possibilities, it would be expedient to prioritise those most likely to produce relevant evidence. (For example, MEDLINE is unlikely to be a useful source of information for a review of social and emotional wellbeing in primary education.) Databases cover different time periods, index different journals, use different subject headings and provide different amounts of bibliographic information. Consequently, each has its strengths and weaknesses. A list of core and topic-specific sources can be found in appendix B.

4.2.2.2 Websites

Websites can be a useful source of grey literature for public health reviews, particularly as a search of traditional, peer-reviewed literature may not produce much information. A list of core websites for consideration can be found in appendix B. These can be supplemented by information from topic-specific websites. Careful selection of websites is required to ensure that the type of evidence available is likely to be relevant: finding relevant data is more important than doing an exhaustive search.

4.2.3 Restrictions

NICE public health reviews are usually restricted to English language studies.

The start date for searches is determined by the nature of the evidence base and the time available to process data (the rationale needs to be documented). The NICE process allows stakeholders to identify whether any restrictions would exclude important data.

Study-type limits or filters should be used with caution, due to the broad nature of public health evidence and the fact that the:

  • majority of sociological and social science databases do not provide adequate indexing by study design

  • quality of indexing for – and the vocabulary used in – study methodologies and designs varies extensively and, in some instances, is poor.

4.3 Phase 2: developing the search strategy

During this phase, the search team 'translates' the concepts from the search protocol, including all the synonyms that will be used (thesaurus terms and free-text/keywords) into a search strategy. Development of the search strategy is an iterative process between the search team, the review team and the CPHE project team. The latter will agree its final form. Support and advice is available from the appointed NICE information services team lead.

4.3.1 Health inequalities and searches

It is essential that the list of synonyms used is exhaustive and that they describe a homogenous group or setting. They should not describe characteristics of subgroups, as this increases the risk of omitting a group (being inequitable or discriminatory). This detail should only be included by default. For example, if the population group is 'older people' a search for 'older people' should pick up subpopulations such as 'disabled older people' or 'black and minority ethnic older people'. Similarly, if the setting is 'communities and religious places', the search terms should cover relevant faith settings (such as 'church', 'temple' and 'mosque') not characteristics of the settings, such as whether the churches are gothic or neo-classical.

4.3.2 Sensitivity versus precision

The strategy needs to balance sensitivity (ability to identify relevant information) and precision (ability to exclude irrelevant documents). However, the need for an exhaustive search (involving additional resources) also needs to be balanced against a more modest search that may miss some studies. The balance will depend on the nature of the review questions and the available evidence.

4.4 Phase 3: gathering the evidence, conducting searches and documenting the process

During this phase, the search team translates the search strategy (as necessary) for use with various databases. The results should be downloaded into Reference Manager or EndNote (or other reference management software). Items that cannot be downloaded into bibliographic software can be recorded in a Word document or spread sheet.

4.4.1 Economic searches

The search for economic evidence should identify papers that are most relevant to public sector practice and hence likely to inform the Public Health Advisory Committee (PHAC) decision-making. A systematic search for economic evaluations should be undertaken in NHS Economic Evaluation Database (EED) and EconLit (as appropriate). Other databases and websites can be searched using appropriate economic filters, where these are available. Economic evidence can also be identified when sifting effectiveness or qualitative search results.

It is advisable to simplify the agreed search strategy for the economic searches because a complex search may exclude relevant studies. For example, instead of searching for population group and setting and intervention and the problem, it might be more reliable to just search for the public health problem. If this produces too many results, then additional concepts can be added.

4.4.2 Additional searches

If the main searches have not retrieved enough relevant material and the search needs to be widened, the review team may carry out additional types of searches. These could include: 'snowballing' to find citations, a search of the grey literature, journal hand-searches or making contact with experts and stakeholders.

4.4.2.1 Citations using 'snowballing'

A search can be usefully extended by looking for articles that cite other, more specific articles containing additional relevant references. However, it depends on whether the database software can perform this search; even if it is possible, such a search will only retrieve cited articles from journals indexed in the same database.

4.4.2.2 Grey literature

A search of the 'grey literature' can help identify material that will not be picked up by mainstream sources (such as the MEDLINE database).

This type of search can be difficult and time consuming: both a database and an Internet search may be necessary. It is essential to be clear about the type of material needed. In particular, it is useful to distinguish between data that might supplement the effectiveness literature (for example, ongoing evaluative research) and information that could aid implementation.

4.4.2.3 Hand-searching

Hand-searching involves a manual search through the contents tables of selected journal titles for relevant articles. There is no requirement to do this and it can be time consuming. However, it is worth doing if the reviewers are aware of any relevant journal titles that are not included in the bibliographic databases being searched. Hand-searching can also be worthwhile if the database searches have failed to retrieve much relevant evidence (though it should be limited to a few relevant, specialised journals). Bibliographic details of any studies identified should be added manually to the database of references that have been downloaded.

Hand-searching is a useful way of identifying experts who may be able to identify other relevant research (see section 4.4.2.4 below).

4.4.2.4 Contacting experts

Ongoing research may be needed to inform the PHAC of important studies likely to be published or completed during the development – or soon after publication – of the public health guidance. Some types of research, notably intervention trials, are often documented in databases of ongoing research. However, these are not always up-to-date and it is advisable to ask experts in the area.

Experts can be identified and contacted via research networks, relevant journal abstracts or via relevant reference lists. The PHAC members may also be able to help; try to give them advance notice, taking into account the time involved and their availability.

Any additional evidence received should be entered into the bibliographic database. The number of articles identified by this means must be specified in the methods section of the review.

4.4.2.5 Using review-level material to identify primary studies

Review-level material (for example, systematic reviews, literature reviews and meta-analyses) may provide an additional source of primary studies. Relevant reviews can be identified using an appropriate checklist. The reference lists in the reviews can be used to identify potentially relevant primary studies.

The Centre for Reviews and Dissemination (CRD), Cochrane and Campbell databases are useful sources of robust, quality reviews.

4.4.3 Quality assuring the search process

The search process must be quality assured to maintain a high, consistent standard for identifying the evidence. The search protocols and MEDLINE search strategies of external search teams are quality assured by NICE information services. The same quality assurance mechanism applies to internal search teams.

4.4.4 Documenting the search process

The search process needs to be transparent and replicable. For these reasons, as well as to aid quality assurance, it is important to document it. The search team should save the following files once the searches are complete:

  • Word document containing the search strategies for each resource searched. Each strategy should include audit information, as shown in appendix C. The document should clearly state the title of the review and the review type (that is, effectiveness/qualitative/economic).

  • Final de-duplicated Reference Manager database.

  • Word document of other results (for those records that cannot be downloaded into Reference Manager, for example, website results).

4.4.5 Submission of evidence from stakeholders

For some review questions, members of the PHAC and the CPHE project team may have good reason to believe that information exists, even though it has not been found using the standard searches. For example, they may be aware of:

  • ongoing research (if an intervention is relatively new)

  • studies that have only been published as abstracts

  • data on adverse effects relevant to the interventions being studied

  • economic models

  • studies of public or professional views and experiences.

In these situations, the CPHE project team may ask stakeholders for evidence (these calls for evidence may be made at any point during development of the guidance). The team will specify the question being addressed, along with details of the type of evidence being sought. For example, for questions of effectiveness, details of the intervention, participants, comparisons, outcome and study design may be needed.

Stakeholders are usually given 4 weeks to respond.

4.4.6 Confidential information

In addition to published studies, stakeholders may submit unpublished data or studies in response to a call for evidence. They will be asked to complete a checklist that identifies the location of any confidential information contained in their submission. Box 4.1 summarises what may be considered confidential by NICE.

Box 4.1 Information on what may be considered confidential

Information that may influence share price values ('commercial in confidence') or is deemed intellectual property (that is, awaiting publication) is deemed confidential.

The relevant part of a sentence, a particular result from a table or a section of code is deemed confidential (that is, information deemed confidential should be kept to an absolute minimum).

NICE will not agree to a whole study being designated confidential. At a minimum, a structured abstract of the study or economic model will have to be made available for public disclosure during consultation on the guidance.

Results derived from calculations using confidential data are not considered confidential unless releasing those results would enable back-calculation to the original data.

Stakeholders should also indicate on the submission itself which part contains the confidential information. For example, they could use a highlighter pen on a paper copy or the highlighter function in an electronic version. These markings should be maintained throughout the development process. When the draft and final versions of the guidance are prepared for publication, the CPHE project team should ensure that these sections are replaced by a note stating that confidential information has been removed, so that the public is aware of exactly where confidential data have been used. These criteria also apply to 'academic in confidence' data.

Following the principles outlined in box 4.1, the amount of confidential information should be kept to an absolute minimum, and, as a minimum, a summary of this information should be publicly available by the time of consultation on the guidance. NICE needs to be able to justify the recommendations made in its public health guidance on the basis of the evidence considered by the PHAC. To this end, NICE will work with the data owners to find an agreed balance between confidentiality and transparency[3].

4.4.7 Documenting evidence from stakeholders

The CPHE project team should enter information received from stakeholders into a table or bibliographic database (see section 4.4.4). Details should be cross-checked against evidence identified through the database searches. It should be assessed in the same way as evidence identified from published studies (see chapter 5).

4.5 Equality and diversity

All searches should be inclusive, capturing evidence related to all groups identified in the Equality Act (2010) (or to groups that are particularly disadvantaged with respect to the topic under consideration). Search strategies should only be narrowed to specific groups if these have been specified during the topic or scoping development phases.

4.6 References

Lefebvre C, Manheimer E, Glanville J (2011) Searching for studies. In: Higgins JPT, Green S, editors. Cochrane handbook for systematic reviews of interventions, version 5.1.0 (updated March 2011). The Cochrane Collaboration.


[3] For example, see the Agreement between the Association of the British Pharmaceutical Industry (ABPI) and the National Institute for Health and Clinical Excellence (NICE) on guidelines for the release of company data into the public domain during a health technology appraisal.