Developing NICE guidelines: the manual

Conceptual frameworks and logic models

When review questions are being developed, it can sometimes be useful to use a conceptual framework or a logic model (or reuse ones that were developed for the scope). These display pathways through which actions and interventions are expected to lead to differences in outcomes. They can be used to potentially identify which interventions are most likely to be effective when targeted at particular places in a pathway. Conceptual frameworks and logic models can be produced by the development team or taken from previously published literature in the topic area.

Conceptual frameworks or logic models are helpful in many topics and can be particularly useful when:

An understanding of the expected mechanisms by which outcomes occur can help guide which interventions are most worthwhile to study in particular settings or questions.

Conceptual frameworks and logic models are used to aid the committee in developing review questions and protocols that are most likely to produce useful results. However, they are not intended to constrain the recommendations the committee may make in the guideline (for example, the committee is not limited to making recommendations that fit within the causal pathways in the framework or model). Examples of conceptual frameworks and logic models are given in figure 1 and figure 2.

Review questions about the effectiveness of an intervention

A helpful structured approach for developing questions about interventions is the PICO (population, intervention, comparator and outcome) framework (see box 4.1). The setting for the question should also be specified if relevant.

For each review question, factors that may affect the outcomes and effectiveness of an intervention, including any wider social factors that may affect health and any health inequalities, should be considered (see the section on considering health inequalities when preparing review protocols). Outcomes (on both benefits and harms) and other factors that are important should be specified in the review protocol. In general, a range of 5 to 9 outcomes should be defined. Guidance on prioritising outcomes is provided by the GRADE working group.

When designing review questions and protocols, it is important to consider possible intercurrent events (events that occur after starting treatment and either preclude the observation of the variable, or affect its interpretation, such as death, non-adherence to treatment or stopping treatment) and how these will be dealt with in any analysis in the guideline, if different identified studies analyse the data in different ways. Clinical trials are increasingly following the Estimand framework (European Medicines Agency), which attempts to increase the clarity of the precise treatment effect that an individual study is estimating.

Examples of review questions on the effectiveness of interventions are shown in box 4.2.

Review questions about pharmacological interventions will usually only include medicines with a UK marketing authorisation, based on regulatory assessment of safety and efficacy. Use of a licensed medicine outside the terms of its marketing authorisation (off-label use) may be considered in some circumstances; for example, if this use is common practice in the UK, if there is good evidence for this use, or if there is no other medicine licensed for the indication. Off-label use is particularly common in pregnant women and in children and young people because these groups have often been excluded from clinical trials during medicine development.

Medicines with no UK marketing authorisation for any indication (unlicensed medicines) will not usually be considered in a guideline because there is no UK assessment of safety and efficacy to support their use. Unlicensed medicines may be included in exceptional circumstances, for example in complex conditions when there are no other treatment options. This should be agreed with the medicines adviser responsible for quality assurance (see also the section on recommendations on medicines, including off-label use of licensed medicines).

The Medicines and Healthcare products Regulatory Agency (MHRA) is responsible for ensuring that medicines meet the required standards of safety, that is, expected benefits outweigh risks of harmful effects. Therefore, we take account of national medicines safety advice and do not assess the safety of licensed medicines unless prior agreement has been reached with the MHRA (these discussions can be initiated via the medicines adviser responsible for quality assurance). In particular, we rarely consider review questions that just cover the clinical risk-benefit balance of a drug versus placebo or no treatment for its licensed indication. National medicines safety advice does not require formal quality assessment but should be reported in the evidence review document and used to inform committee discussions. See the chapter on interpreting the evidence and writing the guideline for more information.

When the effectiveness of a medicine is being considered, it is expected that the evidence review will include medicines safety outcomes, such as adverse events. These outcomes are also likely to be included as part of assessing the overall impact on quality of life in any cost-effectiveness analysis, or as part of supporting shared decision making.

Sources of national medicines safety advice include:

If there is a need for additional information on drug safety for a particular topic (for example, pregnancy or breastfeeding) advice on how to obtain this should be sought from the medicines adviser responsible for quality assurance.

When including an off-label use of a licensed medicine, it is usually possible to extrapolate national medicines safety advice, for example, if the population is similar and the recommended dosage is likely to be within the licensed dose range. The approach should be discussed with the committee as early as possible and agreed with the team responsible for quality assurance.

In some circumstances, primary evidence on medicines safety may be needed, for example, for unlicensed medicines, or off-label use of licensed medicines when it is not expected that extrapolating safety information from the licensed uses is appropriate. This should be identified as early as possible, stated in the review protocol, and agreed with the team responsible for quality assurance. It is important to note that different evidence types will often be needed for assessing safety, because both rare serious events and long-term events are important, and are often not captured in research studies.

Review questions about complex interventions

Skivington et al. 2021b states: 'An intervention might be considered complex because of properties of the intervention itself, such as the number of components involved; the range of behaviours targeted; expertise and skills required by those delivering and receiving the intervention; the number of groups, settings, or levels targeted; or the permitted level of flexibility of the intervention or its components'.

Review questions about the effectiveness of complex interventions may need additional analyses to understand the complexity of the interventions, or additional types of evidence to answer different aspects of the question such as qualitative research or real-world evidence. For example, additional evidence might address the views of people using the service or intervention, or barriers to use, as reported by practitioners or providers, or the impact on health inequalities. In this case, 2 related review questions (quantitative and qualitative) may be used to address the issues. These different forms of data can be considered separately, or together (integrated) if needed to understand the underlying problem. A review of effectiveness may also include evidence of the intervention's mechanism of action, that is, evidence of how the intervention works. All these examples of questions about the effectiveness of interventions may be addressed by 2 related review questions or by a mixed methods review.

When formulating questions to assess complex interventions, routinely consider whether to factor in the sociocultural acceptability and accessibility of an intervention, as well as contextual factors that impact on intervention feasibility. Qualitative evidence synthesis is one method of exploring these factors (Booth et al. 2019). An extended question framework (PerSPEcTiF) is proposed to recognise these wider issues, while also being particularly suited to qualitative evidence synthesis and complex intervention reviews (see table 21.5a in chapter 21 of the Cochrane Handbook for Systematic Reviews of Interventions). The PerSPEcTiF model of question formulation includes the following elements: Per = Perspective, S = Setting, P = Phenomenon of interest or problem, E = Environment, (c) = Comparison (optional), Ti = Time or timing, F = Finding.

Examples of review questions that include analysis targeting complex interventions are shown in box 4.3.

Box 4.3 Examples of review questions that include analysis targeting complex interventions

Review questions about the accuracy of diagnostic tests

Review questions about diagnosis are concerned with the performance of a diagnostic test or test strategy to identify the presence of a current condition in people. They begin at the point in the diagnostic process when a professional has already made an initial diagnosis, or diagnoses, based on their clinical judgement. Diagnostic tests to confirm or rule out the initial diagnosis can include history-taking, symptoms, signs, identification tools, laboratory or pathological examination, and imaging tests.

Broadly, there are 2 types of review questions about diagnostic tests:

Questions looking at the accuracy of multivariable diagnostic prediction models are covered in the section on review questions about predicting an individual prognosis or identifying an individual diagnosis.

In studies of the accuracy of a diagnostic test, the results of the test under study (the index test) are compared with those of the best available test (the reference standard). It is important to be clear when deciding on the question what the exact proposed use of the test is (for example, as an identification tool, an initial 'triage' test or after other tests).

The PICTO (population, index test, comparator, target condition and outcome) framework can be useful when formulating review questions about diagnostic test accuracy (see box 4.4). However other frameworks (such as PPIRT; population, prior tests, index test, reference standard, target condition) can be used if helpful.

A review question about diagnostic test accuracy is usually best answered by a cross-sectional study in which both the index test and the reference standard are performed on the same sample of people. Cohort and case–control studies are also used to assess the accuracy of diagnostic tests, but these types of study design are more prone to bias (and often result in inflated estimates of diagnostic test accuracy). Further advice on the types of study to include in evidence reviews of diagnostic test accuracy can be found in the Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy.

Examples of review questions on diagnostic test accuracy are shown in box 4.5.

Although assessing test accuracy is important for establishing the usefulness of a diagnostic test, the value of a test usually lies in how useful it is in guiding treatment decisions or the provision of services, or supporting shared decision making, and ultimately in improving outcomes. Review questions aimed at establishing the value of a diagnostic test in practice can be structured in a similar way to questions about interventions. The best study design is a test-and-treat RCT. These compare outcomes for people who have a new diagnostic test (in combination with a management strategy) with outcomes of people who have the usual diagnostic test and management strategy. These types of study are not very common so evidence about diagnostic test accuracy is usually also needed.

Review questions about prognosis

Information about prognosis can be used within guidelines to:

Review questions about prognosis address the likelihood of an outcome for an individual person from a population at risk for that outcome, based on the presence of a proposed prognostic factor or factors (see box 4.6). A helpful structured approach for developing questions about prognosis is the PICOTS (population, index prognostic factor, comparator prognostic factors, outcome, timing, setting) framework.

Review questions about prognosis may be closely related to questions about aetiology (cause of a disease or need) if the outcome is viewed as the development of the disease or need based on a number of risk factors.

Questions looking at the accuracy of multivariable prognostic prediction models are covered in the section on review questions about predicting an individual prognosis or identifying an individual diagnosis.

A review question about prognosis is best answered using a prospective cohort study with multivariable analysis. Case-control studies and cross-sectional studies are not usually suitable for answering questions about prognosis because they do not estimate baseline risk; they give only an estimate of the likelihood of the outcome for people with and without the prognostic factor. When developing a review question on prognosis, it is also important to consider possible confounding factors and whether some prognostic factors are modifiable and others non-modifiable.

Review questions about predicting an individual prognosis or identifying an individual diagnosis

Statistical analyses can be used to develop prediction models for a specific diagnosis or prognosis. These models are usually developed using multivariable modelling methods. Multivariable prediction models are developed to help healthcare professionals estimate the probability or risk that a specific disease or condition is present (diagnostic prediction models) or that a specific event will occur in the future (prognostic prediction models). They are usually developed using a multivariable model – a mathematical equation that relates multiple predictors for a particular person to the probability of or risk for the presence (diagnosis) or future occurrence (prognosis) of a particular outcome. Other names for a prediction model include risk prediction model, predictive model, prognostic (or prediction) index or rule, and risk score.

Diagnostic prediction models can be used to inform who should be referred for further testing, whether treatment should be started directly, or to reassure patients that a serious cause for their symptoms is unlikely. Prognostic prediction models can be used for planning lifestyle or treatment decisions based on the risk for developing a particular outcome or state of health in a given period.

Prediction model studies can be broadly categorised into those that develop models, those that validate models (with or without updating the model) and those that do both. Studies that report model development aim to derive a prediction model by selecting the relevant predictors and combining them statistically into a multivariable model. Logistic and Cox regression are most frequently used for short-term (for example, disease absent versus present, 30‑day mortality) outcomes and long-term (for example, 10‑year risk) categorical outcomes, respectively. Studies may also focus on quantifying how much value a specific predictor (for example, a new predictor) adds to the model. Outcomes that are important should be agreed by the committee and specified in the review protocol.

Quantifying the predictive ability of a model using the same data from which the model was developed (often referred to as apparent performance) tends to overestimate performance. Studies reporting the development of new prediction models should always include some form of validation to quantify any optimism in the predictive performance (for example, calibration and discrimination). There are 2 types of validation: internal validation and external validation. Internal validation uses only the original study sample with methods such as bootstrapping. External validation evaluates the performance of the model with data not used for model development. The data may be collected by the same investigators or other independent investigators, typically using the same predictor and outcome definitions and measurements, but using a different sample (for example, from a later time period). If validation indicates poor performance, the model can be updated or adjusted on the basis of the validation data set. For more information on validating prediction models, see Steyerberg et al. 2001, 2003, 2009; Moons et al. 2012; Altman et al. 2009; and Justice et al. 1999.

For more information on developing review questions about prediction models, see the TRIPOD statement in the Annals of Internal Medicine and the TRIPOD statement: explanation and elaboration in the Annals of Internal Medicine.

Although assessing predictive accuracy is important for establishing the usefulness of a prediction model, the value of a prediction model lies in how useful it is in guiding treatment or management decisions, or the provision of services, and ultimately in improving outcomes. Review questions aimed at establishing the value of a prediction model in practice, for example, to compare outcomes of people who were identified from a prediction model (in combination with a management strategy) with outcomes of people who were identified opportunistically (in combination with a management strategy) can be structured in the same way as questions about interventions.

Qualitative review questions

In some circumstances, specific questions should be formulated about the views and experience of people using services, family members and carers, and the public. The views and experiences of those commissioning and providing services may also be relevant. Qualitative questions do not have to be linked to an effectiveness question; they can stand alone in a single evidence review. Qualitative questions can cover a range of dimensions, including:

As for other types of review question, questions that are broad and lack focus (for example, 'What is the experience of living with condition X?') should not be asked. The question should be more focused by including the phenomenon of interest such as acceptability, accessibility, preferences, information and support needs, feasibility or implementation.

The PICO (Population, phenomena of Interest, Context) framework and the SPIDER framework in McMaster University's National Collaborating Centre for Methods and Tools registry are examples of frameworks that can be used to structure qualitative evidence synthesis (QES) review questions. Examples of QES review questions are shown in box 4.8.

Other evidence used for views and experiences

While qualitative studies often answer a review question about the views or experiences of people using or providing services, family members or carers or the public, other options include quantitative patient preference studies. Information on views and experiences is also becoming increasingly available as part of some intervention studies, for example, collecting qualitative data from trial participants. When qualitative and quantitative data are generated from the same trial in this way, they are referred to as 'trial siblings'. For more information see chapter 21 of the Cochrane Handbook for Systematic Reviews of Interventions.

When there is a lack of evidence on issues important to people affected by the guideline (including families and carers, where appropriate), the development team should consider seeking additional information outside of that from formal literature searching. This could be through a call for evidence (see the section on calls for evidence from stakeholders in the chapter on identifying the evidence: literature searching and evidence submission). It could also be through additional consultation, commissioned primary research, or by approaching experts who may have access to additional data sources (such as surveys of people's views and experiences) or who may be able draw on their experience of working in the field. For more information, see:

Mixed methods approaches

For further information on mixed methods reviews, see chapter 8 of the JBI Manual for Evidence Synthesis.

Lizarondo et al. 2020 states: 'The core intention is to combine quantitative and qualitative data (from primary studies) or integrate transformed quantitative evidence and qualitative evidence to create a breadth and depth of understanding that can confirm or dispute evidence and ultimately answer the review question/s posed'.

'Dependent on the nature of the review question mixed methods systematic reviews may allow for:

A mixed method approach is only needed when a question cannot be answered by separate quantitative or qualitative evidence synthesis and when multiple perspectives are needed to understand the underlying problem. For example, if the method of intervention delivery affects whether people engage with it and that has an impact on its effectiveness. Another example is when integrating the 2 types of evidence provides more evidence or explanation than the separate quantitative and qualitative evidence reviews. Reasons to use a mixed methods review may include:

Using a mixed methods approach requires integration of evidence. That is:

When developing a mixed methods review question, it is helpful to consider a number of things to help guide the method to take, such as sequence of synthesis, approach to integration and the nature of the question (for more information, see sections 8.2 and 8.3 in chapter 8 of the JBI Manual for Evidence Synthesis).

Examples of mixed methods review questions are shown in box 4.9.

Review questions about service delivery

Guidelines sometimes cover areas of service delivery. These might include the relative effectiveness of different models of service delivery, how delivery of services could improve, how delivery of services impact on health inequalities, or what the core components of services are and how different components could be re‑configured.

A review question about the effectiveness of service delivery models is usually best answered by a pragmatic RCT, if it is feasible to do one. However, a wide variety of methodological approaches and study designs can be used, including non-randomised studies that report observational data (including routine healthcare and audit data), experimental and qualitative evidence. Other types of questions on service delivery are also likely to be answered using evidence from study types other than RCTs. For example, to determine whether an intervention will work for a particular subgroup or setting that does not have specific evidence from an RCT, we might want to know how the intervention works, which will require evidence of the relevant underlying mechanisms.

Depending on the type of review questions, the PICO framework may be appropriate but other frameworks can be used.

When a topic includes review questions on service delivery, approaches described in the chapter on incorporating economic evaluation and the appendix on service delivery – developing review questions, evidence reviews and synthesis may be used. Such methods should be agreed with staff with responsibility for quality assurance and should be clearly documented in the guideline.

Review questions about descriptive epidemiology

Some epidemiological reviews describe the problem under investigation and can be used to inform other review questions. For example, an epidemiological review of incidence or prevalence of a condition would provide baseline data for further evidence synthesis, an epidemiological review of accidents would provide information on the most common accidents, as well as morbidity and mortality statistics, and data on inequalities in the impact of accidents. These review questions may also be necessary to provide input data for economic modelling.

The structure of the question and the type of evidence will depend on the aim of the review.

Other epidemiological reviews describe relationships between epidemiological factors and outcomes – a review on associations. If an epidemiological review has been carried out, information will have been gathered from observational studies on the nature of the problem. However, further analysis of this information – in the form of a review on associations – may be needed to establish the epidemiological factors associated with any positive or negative behaviours or outcomes.

Review questions that consider cost effectiveness

In most NICE guidelines, it is expected that considerations around cost effectiveness will be included for all review questions. Therefore, it is not necessary to explicitly mention cost effectiveness in the review question itself. If some review questions for a guideline will not consider cost effectiveness, it should be specified which questions will and will not be including these considerations. For more information on review questions that consider cost effectiveness, see the chapter on incorporating economic evaluation.

Structure of the review protocol

The protocol should describe any differences from the methods described in this manual (see the chapters on identifying the evidence: literature searching and evidence submission, reviewing evidence, and incorporating economic evaluation), rather than duplicating the manual.

Templates for the 4 common review types (intervention, diagnostic accuracy, qualitative and prognosis) are available in the appendix on the review protocol templates. These templates can be amended to match the specifics of an individual review question. For reviews not covered by these templates, a similar level of detail should be provided in the review protocol.

When a guideline is updating an evidence review from a published NICE guideline, the protocol from the published guideline, if available, should be used as a starting point for developing the new review protocol. It should be updated based on any changes since the original protocol was developed (such as new interventions and comparators, and extensions of the population). The level of changes needed is likely to depend on the length of time since the original protocol was developed. No more changes than necessary should be made, as the closer the protocol remains to the original, the easier it will be to reuse data extracted in that original review.

Process for developing the review protocol

The review protocol should be drafted by the development team, with input from the guideline committee, and then reviewed and approved by staff with responsibility for quality assurance. This should take place after agreeing the review question and before starting the evidence review.

Although an original systematic review of primary studies is the most common method to answer a review question, there are several other possible alternatives that should be considered, such as:

More than one of these methods may need to be used for some review questions, if different parts of the question need different approaches. It will often not be possible to describe evidence reviews using any of these methods in the format of a standard review protocol. When this is not possible a narrative description of the review plan can be produced instead. This should clearly describe the planned approach to the review, and the reason why this approach was preferred over doing an original review of published primary data.

When considering which of these potential approaches to use, it is important to consider the trade-off between the optimal evidence to address a question, and the additional time and resources needed to gather that evidence. In particular, it should be considered whether any additional work is likely to lead to different recommendations being made.

All review protocols should consider registering on the PROSPERO database before the data extraction commences, if possible and appropriate. The review questions are published on the NICE website at least 6 weeks before consultation on the draft recommendations. Any changes made to a protocol during guideline development should be agreed with staff who have responsibility for quality assurance and the version on the website updated. Any deviations from the signed-off and published protocol should be clearly stated and justified in the evidence review document. If protocols are published anywhere else (for example, on the PROSPERO database) development teams should ensure the versions of the protocol are consistent.

Considering health inequalities when preparing review protocols

When developing review protocols it is important to identify any health inequalities that may be relevant to the review question. This should involve identifying any issues from the equality and health inequalities assessment that are particularly relevant to the particular question, as well as documenting any new issues identified by the committee.

The committee will need to consider these issues and any gaps in the evidence when interpreting the result of the review. However, in some circumstances it may be advisable to make modifications to the review protocol to ensure health inequalities are appropriately addressed, for example by:

Other evidence

Depending on the nature of the guideline topic and the review question, other sources of relevant evidence such as reports, audits or service evaluations from the published or grey literature may be included. Often these will not need identifying from a systematic literature search (for example, if there is a national organisation responsible for producing reports on a particular subject). This should be agreed with staff who have responsibility for quality assurance and documented in the review protocol. When it is necessary to assess the quality, reliability and applicability of this evidence, it should be assessed according to standard processes (see the appendix on appraisal checklists, evidence tables, GRADE and economic profiles).

See also the chapter on linking to other guidance (which also covers using evidence from non-NICE guidance).