NICE process and methods

4 Developing review questions and planning the systematic review

4 Developing review questions and planning the systematic review

At the start of guidance development, the key issues listed in the scope need to be translated into review questions. In some instances, this may be done as part of the scoping process (see section 2). The review questions must be clear, focused and closely define the boundaries of the topic. They are important both as the starting point for the systematic literature review and as a guide for the development of recommendations by the Guidance Development Group (GDG). The development of the review questions should be completed soon after the GDG is convened.

This section describes, in principle, how review questions are developed, formulated and agreed. It describes the different types of review question that may be used, and provides examples. It also provides information on how to plan the systematic review.

4.1 Number of review questions

The number of review questions for each social care guidance topic depends on the topic and the breadth of the scope (see section 2). However, the number of review questions must be manageable for the GDG and the NICE Collaborating Centre for Social Care (NCCSC) within the agreed timescale. As a guide, 10–15 is a reasonable number of review questions for standard social care guidance.

This is based on the estimate that, on average, it is feasible to present a maximum of 2 systematic reviews at a GDG meeting. However, review questions vary considerably in terms of the number of studies included and the complexity of the question and analyses. For example, a review question might involve a complex comparison of several service models involving many individual studies. At the other extreme, a question might address the effects of a single, simple intervention and have few relevant studies.

4.2 Developing review questions from the scope

Review questions should address all areas covered in the scope, and should not introduce new aspects not specified in the scope. However, they contain more detail than the scope, and should be seen as building on the key issues in the scope.

Review questions are refined and agreed by all GDG members through discussions at GDG meetings. The different perspectives among GDG members help to ensure that the right review questions are identified, thus enabling the literature search to be planned efficiently. On occasion, the questions may need refining once the evidence has been searched.

Review questions are then used to develop protocols that detail how questions will be addressed.

4.2.1 Economic aspects

This section relates to the specification of questions for reviewing the effectiveness evidence.

Evidence about economic aspects of the key issues should also be sought from published economic evaluations and by conducting new modelling where appropriate. Methods for identifying and reviewing the economic literature are discussed in sections 5 and 6; economics modelling is discussed in section 7.

When developing review questions, it is important to consider what information is needed for any planned economic modelling. This might include, for example, information about quality of life, rates of adverse effects or use of social care services.

4.3 Formulating and structuring review questions

A good review question is clear and focused. It should relate to a specific service user problem or concern, because this helps to identify the relevant evidence.

Service user experience should be considered when developing all structured review questions. Review questions that focus on a specific element of service user (and, where appropriate, carer) experience may also merit consideration in their own right.

4.3.1 Types of evidence

Social care guidance recommendations are based on research and other types of evidence about what works generally, why it works and what might work (and how) in specific circumstances. Recommendations may also need to be based on evidence of service user and carer experience of different types of intervention – and other issues related to context, ethics and theory (Tannahill 2008).

Recommendations will often therefore be based on evidence from multiple sources.

4.3.2 Types of review

Several high-quality reviews of the best available evidence are used to develop every piece of NICE social care guidance. These reviews explicitly address questions based on the scope. Rather than relying on the standard hierarchy of evidence, with randomised controlled trials (RCTs) at the top, a wide range of study designs and methodologies should be used to answer these questions (Petticrew and Roberts 2002). (See section 4.3.1.)

4.3.3 Types of review questions

The review questions are based on the key issues in the scope and the views of social care practitioners, decision makers and other stakeholders. However, the scope may include several other questions and potential considerations that reflect the nature of the specific issue being tackled and its context.

In addition to questions of effectiveness and cost effectiveness, there are often questions about the acceptability and accessibility of interventions, and service user or practitioner experiences.

The nature and type of questions determines the number and type of reviews and the type of evidence that is most suitable (for example, intervention studies and qualitative data).

Whatever method is used, the process for developing questions is the same.

Review questions should be clear and focused. The exact structure of each question depends on what is being asked, but it is likely to cover one of the following:

  • extent and nature of the social care issue

  • interventions that work in ideal circumstances and might work in specific circumstances or settings (the extent to which something works, how and why)

  • a relevant programme theory or theory of change

  • views and experiences of the target population (people who may be affected by the recommendation), including how acceptable and accessible they find the intervention

  • practitioners' views, experiences and working methods (including any barriers to and factors supporting adoption of the intervention)

  • cost effectiveness

  • potential for an intervention to do harm.

At least 1 effectiveness review is developed for every piece of social care guidance. The decision on whether or not to use additional types of review depends on the topic area and the type, depth and breadth of relevant evidence available. Sometimes, a review may draw on a combination of different sources of evidence or types of data (for example, combining mapping information and qualitative data).

4.4 Planning the review

For each review, a review protocol that outlines the background, the objectives and the planned methods should be prepared.

4.4.1 Structure of the review protocol

The review protocol should include the components outlined in table 2.

Table 2 Components of the review protocol

Component

Description

Review question

Review questions as agreed by the GDG.

Objectives

Short description; for example 'To estimate the effectiveness and cost effectiveness of…' or 'To describe the views of…'.

Criteria for considering studies for the review

Detailed components of the review questions, for example the PICO (population, intervention, comparator and outcome) or SPICE (setting, perspective, intervention or phenomenon of interest, comparison, evaluation) framework or similar.

Including the study designs selected.

How the information will be searched

Methods of searching (such as databases, hand searching, citation searches).

Sources to be searched and any limits applied to the search strategies; for example, publication date, study design, language.

The review strategy

Methods to be used to review the evidence, outlining exceptions and subgroups.

Whether meta-analysis will be used and, if so, how it will be conducted.

The review protocol is an important opportunity to look at issues relating to equalities that were identified in the scope, and to plan how these should be addressed. For example, if it is anticipated that the effects of an intervention might vary with service user age the review protocol should outline the plan for addressing this in the review strategy.

4.4.2 Process for developing the review protocol

All review protocols should be included in the draft of the guidance that is prepared for consultation. Any changes made to a protocol in the course of the work should be described. Review protocols are also published on the NICE website 5–7 weeks before consultation on the guidance starts.

4.5 Colloquial evidence

Most types of review focus on gathering and assessing research evidence. However, 'colloquial evidence' – about values, practice, judgement, operational considerations and interests – is also central to developing social care guidance. It can take the following forms.

4.5.1 Expert testimony

An expert witness may be invited to give expert testimony when:

  • reviews have uncovered significant gaps in the evidence

  • available evidence conflicts significantly

  • the GDG wishes to seek the views and experiences of specific groups of practitioners or service users and carers.

Expert testimony can be used to provide a range of information about social care approaches and aspects of service delivery, including:

  • context – for example, the policy or commissioning context

  • effectiveness – for example, preliminary results from ongoing interventions or services

  • service design and delivery – for example, detailed information on how a particular service is implemented with different groups of people

  • experience – for example, views and experiences of groups of service users, carers or practitioners.

Experts may be identified via stakeholders, via GDG members, or in the course of carrying out the reviews (for example, key authors or researchers). The Public Involvement Programme will help to identify service user experts. Before the GDG meeting, the expert witness will be asked to prepare an expert testimony summary, including references to any relevant published work. This is treated as evidence and subject to consultation, along with any reviews.

Expert testimony takes the form of a short, focused presentation to the GDG, followed by discussion.

4.6 Equality and diversity

Specific issues in relation to groups identified in the Equality Act (or groups who are particularly disadvantaged in the topic under consideration) should be addressed. These issues should be identified during the topic selection and scope development. They should also be considered when developing the review questions.

4.7 References and further reading

Lomas J, Culyer T, McCutcheon C et al. (2005) Conceptualizing and combining evidence for health system guidance: final report. Ottawa: Canadian Health Services Research Foundation

Petticrew M, Roberts H (2002) Evidence, hierarchies, and typologies: horses for courses. Journal of Epidemiology and Community Health 57: 527–9

Tannahill A (2008) Beyond evidence – to ethics: a decision making framework for health promotion, public health and health improvement. Health Promotion International 23: 380–90