NICE process and methods
5 Identifying the evidence: literature searching and evidence submission
5 Identifying the evidence: literature searching and evidence submission
5.1 Introduction
The systematic identification of evidence is an essential step in developing social care guidance. Systematic literature searches should be thorough, transparent and reproducible. Searches should also minimise 'dissemination biases' (Song et al. 2000), such as publication bias and database bias, that may affect the results of reviews.
This section is aimed primarily at information specialists in the NICE Collaborating Centre for Social Care (NCCSC). It provides advice on the sources to search and on how to develop strategies for systematic literature searches to identify social care and economic evidence. It also provides advice on other areas of information management that form an important part of the social care guidance development process.
Calls for submissions of evidence from stakeholders and undertaking baseline assessments of service activity (for service guidance) are also covered.
The scoping search undertaken when drafting the scope of social care guidance is described in section 2.3.2.
5.2 Searching for evidence
NICE encourages the use of search methods that balance precision and sensitivity. The aim is to identify the best available evidence to address a particular question, without producing an unmanageable volume of results.
NICE supports innovative and flexible approaches to searching, because it is often not possible to know in advance where the best available evidence is likely to be located. The use of iterative searching (sometimes referred to as emergent searching) in which the evidence base is not pre-defined is welcomed, as is the use of grey literature sources, such as charity and government department websites.
The search for evidence involves:
-
creating precise search questions and identifying the study types needed to answer those questions
-
using an appropriate search approach – iterative or systematic
-
matching key sources to the questions being asked (and not necessarily trawling all available sources just because they exist)
-
adopting a pragmatic and flexible approach that allows a continual review of how best to find evidence and where
-
having an understanding of the existing evidence base.
Identifying evidence for social care guidance involves searching a wide range of electronic resources. The list of information sources should be individually tailored for each review to ensure they are relevant to the guidance topic.
Searches should include a mix of core databases, subject-specific databases and other resources, depending on the subject of the research question and the level of evidence sought.
Database search strategies should be developed using an industry standard database (for example, Social Care Online) and translated into other sources. Database searches should be supplemented by alternative search approaches as appropriate to the topic, for example, hand searching, pearl growing and citation searching.
For innovative and alternative search methods (for example, iterative searching), a rationale for the approach and for subsequent search iterations should be included with any search strategies.
5.2.1 Databases and other sources to search
The databases and other sources that should be searched to identify evidence depend on the review question.
Core and subject-specific databases
The core databases listed in box 8 should be searched for most review questions, although this will be dependent on the specific question. Additional subject-specific databases and other resources may also need to be searched, depending on the subject area of the review question and the type of evidence sought.
Box 8 Sources for the review question searches (listed in alphabetical order)
Searches may also include a newspaper database (for example, Proquest Newspaper Library) depending on the topic. For information about service user experience (including children and young people):
Websites of relevant organisations that may report research on service users' views or experiences (NICE's Public Involvement Programme can advise further). |
|
a Accessible via the Cochrane Library. Database name in parentheses is that used in the Cochrane Library. b Accessible as part of the Cochrane Library and via the Centre for Reviews and Dissemination (CRD). The CRD website hosts the most up-to-date versions of the databases. Database names in parentheses are used in the Cochrane Library. |
An awareness of the strengths and weaknesses of each database is important when undertaking a systematic literature search. The different databases index different journals, use different subject headings, cover different time periods and provide different amounts of bibliographic information. There will be overlap in the records retrieved from the different databases for a particular review question. Therefore cross-database searching, although time-consuming, is necessary to comprehensively identify evidence for social care guidance development.
Other sources of information
The sources listed in table 3 – which include databases and websites – can provide useful information about ongoing research, service user experience, practice audits and statistics to help guide Guidance Development Group (GDG) decision-making.
Table 3 Other sources of information
|
Source |
Website |
|
Care Quality Commission |
|
|
Conference Papers Index |
|
|
Economic and Social Data Service |
|
|
Health and Social Care Information Centre |
|
|
Hospital Episode Statistics |
|
|
Information about experiences |
|
|
International Standard Randomised Controlled Trial Number Register |
|
|
National or regional audits |
Search by topic or geographical area for appropriate audit data |
|
National or regional registers |
Search to locate appropriate register |
|
Office for National Statistics |
|
|
Personal Social Services Research Unit |
|
|
Poverty site |
|
|
Surveys of user experiences |
Search for relevant service user organisation websites; condition, service-specific or topic-specific as appropriate |
|
The King's Fund |
|
|
UK National Statistics Publication Hub |
|
|
Web of Knowledge |
5.2.2 How to search for social care evidence
Many of the principles listed in this section are also relevant to searching for economic evidence (see section 5.3).
Devising an overall search strategy
Review questions can be broken down into different parts, which can then be used to devise a search strategy. For example, using a structured approach such as the PICO (population, intervention, comparator and outcome) or the SPICE (setting, perspective, intervention or phenomenon of interest, comparison, evaluation) framework, a search strategy can be constructed for terms relating to the population; this can be combined with terms relating to the interventions and comparators (if there are any) to be evaluated.
It is important to remember that not all components of a review question will always be mentioned in the abstracts or subject headings of database records – in particular, outcomes are often not mentioned. Therefore, it may not be advisable to include these components when developing a strategy. For guidance that is being updated, previous strategies can be used in search strategy design.
Additional searches
Because of the range of evidence needed for social care guidance, different approaches to searching are needed (such as snowball citation searching or hand searching). As with database searching, these should be considered at the outset and follow the same principles of transparency and documentation.
5.3 Searching for economic evidence
The approach to searching for economic evidence should be systematic, but targeted to identify studies that are most relevant to current social care practice in the UK and hence likely to be relevant for GDG decision-making.
Two types of search might be needed for economic evidence:
-
A systematic search for economic evaluations relevant to the guidance and applicable to current social care practice in the UK should be performed. This should cover all review questions with potential cost or resource implications and should not be limited to the modelling priorities identified in the economic plan. This search should be conducted by the information specialist in consultation with the economist (see section 5.3).
-
Additional searches may be necessary to identify other information needed for economic modelling. This may include information about prognosis, adverse effects, quality of life, resource use or costs, which is not always available from the searches conducted for the guidance. The information specialist and the economist should discuss the need for additional searches. See section 7.2.11 for more details about identifying model inputs, including searching for quality-of-life data.
Much of the advice in section 5.2.2 about how to search for social care evidence is also relevant to systematic searches for economic evaluations.
5.3.1 Initial search to identify economic evaluations
Most of the search for economic evaluations should be completed near the beginning of the guidance development process as an initial broad search. The first step is a search of a key economics database using the service user population terms, as for the initial topic background search. Other core databases should then be searched for the service user population terms with the addition of a published economics search filter.
A suggested strategy for searching for economic evaluations in the initial broad search is:
-
NHS EED (NHS Economic Evaluation Database)[2], and HEED (Health Economic Evaluations Database) if subscribed to – all years.
-
Econpapers.
-
CEA Registry.
-
HTA database – all years.
This initial broad search should be extended to identify recent papers that have not yet been referenced in the economics databases, by searching key subject-specific databases covering the most recent complete year.
Other subject-specific databases may be searched at this stage, at the discretion of the information specialist.
5.3.2 Further searches to identify economic evaluations
Further searches for economic evaluations may be needed for some review questions. The purpose of these searches is to try to ensure that all relevant economic evaluations are identified; some may not be retrieved by the initial search because of the inclusion criteria of the economics databases.
The need for additional searches and the criteria (such as date parameters) for them should be established by the economist in consultation with the information specialist. It may also be worthwhile to use a highly sensitive economics search filter.
The searches may be executed when needed or alongside other new searches, depending on the preference of the economist in consultation with the information specialist.
5.4 Publishing search strategies
Search strategies are published on the NICE website 5–7 weeks before consultation on the draft guidance starts, and are available to stakeholders during consultation. They should also be published at the same time as the final guidance.
5.5 Re-running searches
Searches undertaken to identify evidence for each review question may be re-run to identify any further evidence that has been published since the search was run initially. If this is done, it will be 6–8 weeks before the draft guidance is submitted to NICE.
5.6 Calls for evidence from stakeholders
For some questions, there may be good reason to believe that information exists that has not been found using standard searches. Examples include ongoing research in a field, if a service or intervention is relatively new, studies that have been published only as abstracts (see section 6.1.2), data on adverse effects, economic models and studies of the experiences of service users, carers or social care practitioners or other professionals.
In these situations, a call for evidence may be made to all registered stakeholders. This call should specify the question being addressed and details of the type of evidence being sought, for example the structured framework being used and study design for questions of effectiveness. A call for evidence may be made at any point during guidance development and stakeholders should usually be given 4 weeks to respond.
If it is likely that the regulatory authorities hold relevant data that have not been submitted in response to a call for evidence, the appropriate regulatory authority may be approached to release those data.
5.6.1 Confidential information
In addition to published studies, stakeholders may submit relevant unpublished data or studies in response to a call for evidence.
Box 9 summarises what may and may not be considered confidential by NICE.
Box 9 Information on what may and may not be considered confidential
|
Data that may be included as confidential include those that may influence share price values (commercial in confidence) or are intellectual property (academic in confidence; that is, awaiting publication). Confidential information should be kept to an absolute minimum; for example, only the relevant part of a sentence, a particular result from a table or a section of code. NICE does not allow a whole study to be designated confidential. As a minimum, a structured abstract of the study or economic model must be made available for public disclosure during consultation on the guidance. Results derived from calculations using confidential data are not considered confidential unless releasing those results would enable back-calculation to the original confidential data. |
Stakeholders should complete a checklist that lists and identifies the location of all confidential information contained in their submission. Stakeholders should also mark the part of their submission that contains the confidential information; for example, by using a highlighter pen on a hard copy, or the highlighter function in an electronic version. These markings should be maintained on those sections so that the GDG knows which parts are confidential.
Following the principles in box 9, the amount of confidential information should be kept to a minimum. As a minimum, a summary should be publicly available by the time of the consultation on the guidance. NICE needs to be able to justify the recommendations in social care guidance on the basis of the evidence considered by the GDG. NICE and the NCCSC will therefore work with the data owners to agree a balance between confidentiality and transparency.
5.6.2 Information not eligible for submission
Stakeholders are asked not to submit the types of evidence listed in box 10, because these will not be considered.
Box 10 Stakeholder material not eligible for consideration by the GDG
|
5.6.3 Contacting experts
Ongoing research may be needed to tell the GDG of important studies likely to be published or completed during the development – or soon after publication – of the social care guidance. Some types of research, notably intervention trials, are often documented in databases of ongoing research. However, these are not always up-to-date and it is advisable to ask experts in the area.
Experts can be identified and contacted via research networks, relevant journal abstracts via relevant reference lists, or GDG members.
5.7 Additional information for service guidance
In addition to evidence identified by routine literature searches, when developing service guidance (see section 1.4.1) the GDG needs information describing the current configuration of social care services, the level of activity and any significant regional variations. This helps the GDG to:
-
identify the gaps between current social care practice, service provision and service user and carer experience, and what the GDG concludes should be in place
-
shape the guidance and formulate recommendations that are likely to have the greatest effect on the service as well as on outcomes.
A detailed baseline assessment of service activity is needed, and should be conducted before the GDG starts work. This should be available for consideration early in the guidance development process, and ideally early enough to be taken into account in the scope. The following data sources might be used in providing an overall picture of service configuration and activity:
-
national or regional registers
-
National Adult Social Care Intelligence Service reports, including social care activity, expenditure, workforce, user experience and joint strategic needs assessments
-
Office of National Statistic reports
-
local authority datasets
-
national or regional practice audits
-
surveys of service users' or carers' experiences.
Such information is also useful to the NICE costing and commissioning lead when developing the cost impact report and can be used in the needs assessment process undertaken by the NCCSC adoption support lead, as part of the adoption support development work.
Where a topic does include a substantive service guidance component, approaches as described in the NICE Interim methods guide for developing service guidance may be used. Such methods will be agreed with NICE from the outset and clearly documented in the final guidance.
5.8 Equality and diversity
All searches should be inclusive, capturing evidence related to all groups identified in the Equality Act (or to groups that are particularly disadvantaged with respect to the topic under consideration). Search strategies should be narrowed to specific groups only if these have been specified during the topic or scoping development phases.
5.9 References and further reading
Jenkins M (2004) Evaluation of methodological search filters – a review. Health Information and Libraries Journal 21: 148–63
Lefebvre C, Manheimer E, Glanville J (2011) Searching for studies. In: Higgins JPT, Green S, editors. Cochrane handbook for systematic reviews of interventions, version 5.1.0 (updated March 2011). The Cochrane Collaboration.
Song F, Eastwood AJ, Gilbody S et al. (2000) Publication and related biases. Health Technology Assessment 4: 1–115
[2] Accessible as part of the Cochrane Library and via the Centre for Reviews and Dissemination (CRD). The CRD website hosts the most up-to-date version of NHS EED.