Interim methods guide for developing service guidance 2014

7.1 Searching for evidence

Evidence in this context refers not only to studies that directly inform the review question, but also to primary data that might inform parameters identified in the design-oriented conceptual model. A number of evidence and data sources might have been identified during scoping, but additional evidence and data should still be systematically searched for, including grey literature as needed.

When a systematic review of published literature is assessed as being appropriate, the methods outlined in chapter 5 of 'The guidelines manual' should be used to search for and identify evidence and data.

If undertaking a systematic search, the search methods should balance precision and sensitivity. The aim is to identify the best available evidence without producing an unmanageable volume of results. A search protocol should be developed prior to undertaking the search. The protocol will indicate the sources to be searched and the rationale for searching, plans to use any supplementary search techniques and any limits to be applied to the search.

It is important to ensure adequate coverage of the relevant literature and to search a range of sources, to minimise bias. However, there should be a clear rationale for the inclusion of additional sources, with only those likely to yield results being prioritised. Appendix 1 can provide a useful starting point for identifying potential sources.

Consideration may be given to identifying evidence about the efficiency and productivity of healthcare delivery. These include studies that examine the numbers of patients treated or identified for a given constraint (such as financial).

These additional requirements are likely to result in significantly more potential studies or data sources being identified than for clinical guidelines. Therefore consideration should be given to methods of identifying and including relevant evidence and data, such as iterative searching techniques.

As noted above, supplementary search techniques, such as citation searching on known relevant studies, could be considered to make searching more efficient. The types of studies that will be needed to address the identified areas will be varied, and a search of the grey literature is likely to be needed. Some of the evidence may be in the form of modelling studies that may not be published within health‑related journals, but rather in those related to operation research, statistical and mathematical methods. Because these are less likely to be indexed in healthcare databases such as Medline, consideration should be given to sources likely to retrieve this type of evidence. Science Citation Index is an example of a useful source.

Some of the evidence may be in the form of modelling studies that may not be published within health‑related journals, but rather in those related to operation research, statistical and mathematical methods, which would not be indexed within Medline or Embase, but would be found in databases such as the Science Citation Index.

Searching for relevant observational data can be very time consuming, so the design‑oriented conceptual model should be used to evaluate the usefulness of doing this. For example, registry data can be a potential source of estimates of treatment effects. But if the Committee cannot estimate the extent or direction of any inherent biases, and has no way of placing limits on these estimates, the value of these studies is greatly diminished and the usefulness of searching for this evidence is low.

Before requesting evidence or data from stakeholders, there needs to be careful consideration of how such evidence or data will inform either the working of the design‑oriented conceptual model or how the model should be structured. For example, financial reports from health organisations may be used to quantify the financial trade-offs, but consideration needs to be given to whether single reports can provide sufficient detail.

7.2 Calls for evidence or data from stakeholders

For some questions, there may be good reason to believe that relevant and useful information exists outside of literature databases or validated national data sources. Examples include ongoing research in a field, if a service is relatively new, and studies that have been published only as abstracts (see section 6.1 of 'The guidelines manual').

Typically, the method for requesting information from stakeholders is through a 'call for evidence'. This is to allow all registered stakeholders to have an equal opportunity to provide relevant information they may have access to. It is anticipated that developing service guidance will rely significantly on information from NHS, patient and other relevant clinical organisations, which is why a list should be compiled during scoping of stakeholders that might be able to provide such information. The type of evidence that might be requested includes:

7.3 Registries and audits

Data from registries and audits may be used to inform both estimates of effectiveness and any modelling. To obtain such data, it may be necessary to negotiate access with the organisations and individuals that hold the data, or to ask them to provide a summary for inclusion in the guidance. Any processes used for accessing data will need to be reported in the protocol and in the guidance. Given the difficulties that organisations may have in extracting audit data, such requests should be focused and targeted: for example, identifying a specific audit and requesting results from the previous 3 years.

7.4 Economic studies

Searching for existing economic evidence relating to services may differ from economic searches for questions about interventions, diagnosis or prognosis. When undertaking a systematic search for economic evidence, the principles and guidance outlined in section 5 of 'The guidelines manual' should be followed. Examining the economic evaluations may also differ for service guidance. Existing economic evaluations are likely to focus on local or regional populations rather than national averages. Studies may not measure commonly used health outcomes such as mortality or measures to calculate the quality‑adjusted life-year (QALY). Evidence is likely to include measures of resource use, and process measures such as length of lists, number of falls or throughput of patients. Searching for, selecting and reviewing such information will differ for service guidance.

The following inclusion criteria should be taken into account when deciding whether to include economic studies identified in the search; final decisions will depend on the service being assessed: