Methods for the development of NICE public health guidance (third edition)

1.1 Is the source population or source area well described?

This question seeks to determine the study's source population or area (that is, to whom or what the study aims to represent). The source population is usually best identified by referring to the study's original research question.

It is important to consider those population demographic characteristics such as age, sex, sexual orientation, disability, ethnicity, religion, place of residence, occupation, education, socioeconomic position and social capital^[14] that can help to assess the impact of interventions on health inequalities and may help guide recommendations for specific population subgroups.

1.2 Is the eligible population or area representative of the source population or area?

To determine if the eligible population or area (for example, smokers responding to a media advertisement, areas of high density housing in a particular catchment area) are representative of the source population (for example, smokers or areas of high density housing), consider the means by which the eligible population was defined or identified and the implicit or explicit inclusion and exclusion criteria used. Were important groups likely to have been missed or under-represented? For example, were recruitment strategies geared toward more affluent or motivated groups? (For example, recruitment from more affluent areas or local fitness centres.) Were significant numbers of potentially eligible participants likely to have been inadvertently excluded? (For example, through referral to practitioners not involved in the research study.)

1.3 Do the selected participants or areas represent the eligible population or area?

Consider whether the method of selection of participants or areas from the eligible population or area was well described (for example, consecutive cases or random sampling). Were any significant sources of biases likely to have been introduced? Consider what proportion of selected individuals or clusters agreed to participate. Was there a bias toward more healthier or motivated individuals or wealthier areas?

Also consider whether the inclusion and exclusion criteria were well described and whether they were appropriate given the study objectives and the source population. Strict eligibility criteria can limit the external validity of intervention studies if the selected participants are not representative of the eligible population. This has been well-documented for RCTs where recruited participants have been found to differ from those who are eligible but not recruited, in terms of age, sex, race, severity of disease, educational status, social class and place of residence (Rothwell 2005).

Finally, consider whether sufficient detail of the demographic (for example, age, education, socioeconomic status, employment) or personal health-related (for example, smoking, physical activity levels) characteristics of the selected participants were presented. Are selected participants representative of the eligible population?

2.1 Allocation to intervention or comparison. How was confounding minimised?

Consider the method by which individuals were allocated to either intervention or control conditions. Random allocation of individuals (as in RCTs) to receive 1 or other of the interventions under investigation, is considered the most reliable means of minimising the risk of selection bias and confounding.

If an appropriate method of randomisation has been used, each participant should have an equal chance of ending up in each of the intervention groups. Examples of random allocation sequences include random numbers generated by computer, tables of random numbers and drawing of lots or envelopes. However, if the description of randomisation is poor, or the process used is not truly random (for example, if the allocation sequence is predictable, such as date of birth or alternating between 1 group and another) or can otherwise be seen as flawed, this component should be given a lower quality rating.

2.2 Were the interventions (and comparisons) well-described and appropriate?

2.3 Was the allocation concealed?

If investigators are aware of the allocation group for the next individual to be enrolled in the study, there is potential for people to be enrolled in an order that results in imbalances in important characteristics. For example, a practitioner might feel that people with mild rather than severe mental health problems would be more likely to do better on a new, behavioural intervention and be tempted to only enrol such individuals when they know they will be allocated to that group. This would result in the intervention group being, on average, less severe at baseline than control group. Concealment of treatment group may not always be feasible but concealment of allocation up until the point of enrolment in the study should always be possible.

Information should be presented in the paper that provides some assurance that allocations were not known until at least the point of allocation. Centralised allocation, computerised allocation systems and the use of coded identical containers would all be regarded as adequate methods of concealment. Sealed envelopes can be considered as adequate concealment if the envelopes are serially numbered, sealed and opaque, and allocation is performed by a third party. Poor methods of allocation concealment include alternation, or the use of case record numbers, date of birth or day of the week.

If the method of allocation concealment used is regarded as poor, or relatively easy to subvert, the study should be given a lower quality rating. If a study does not report any concealment approach, this should be scored as 'not reported'.

2.4 Were participants and investigators blind to exposure and comparison?

Blinding refers to the process of withholding information about treatment allocation or exposure status from those involved in the study who could potentially be influenced by this information. This can include participants, investigators, those administering care and those involved in data collection and analysis.

Unblinded individuals can bias the results of studies, either intentionally or unintentionally, through the use of other effective co-interventions, decisions about withdrawal, differential reporting of symptoms, or influencing concordance with treatment.

The terms 'single blind', 'double blind' and even 'triple blind' are sometimes used in studies. Unfortunately, they are not always used consistently. Commonly, when a study is described as 'single blind', only the participants are blind to their group allocation. When both participants and investigators are blind to group allocation the study is often described as 'double blind'. It is preferable to record exactly who was blinded, if reported, to avoid misunderstanding.

It is important to note that blinding of participants and researchers is not always possible, and it is important to think about the likely size and direction of bias caused by failure to blind in making an assessment of this component.

2.5 Is the exposure to the intervention and comparison adequate?

2.6 Is contamination acceptably low?

2.7 Were other interventions similar in both groups?

This question seeks to establish if there were any important differences between the intervention groups aside from the intervention received. If some patients received additional intervention (known as 'co-intervention'), this additional intervention is a potential confounding factor in the presence of which can make it difficult to attribute any observed effect to the intervention rather than to the other factors.

2.8 Were there other confounding factors?

2.9 Were all participants accounted for at study conclusion?

Section 2 also aims to assess the likelihood of attrition bias being introduced into a study.

Attrition bias occurs when there are systematic differences between the comparison groups with respect to participants lost, or differences between participants lost to the study and those who remain. Attrition can occur at any point after participants have been allocated to their intervention groups. As such, it includes participants who are excluded post-allocation (and may indicate a violation of eligibility criteria), those who fail to complete the intervention and those who fail to complete outcome measurement (regardless of whether or not the intervention was completed).

It is a concern if the number of participants who were lost to follow-up (that is, dropped out) is high – typically >20%, although it is not unreasonable to expect a higher drop-out rate in studies conducted over a longer period of time.

Consideration should also be given to the reasons why participants dropped out. Participants who dropped out of a study may differ in some significant way from those who remained in the study. Drop-out rates and reasons for dropping out should be similar across all treatment groups. In good quality studies, the proportion of participants lost after allocation is reported and the possibility of attrition bias considered in the analysis.

2.10 Did the setting reflect usual UK practice?

2.11 Did the intervention or control comparison reflect usual UK practice?

3.1 Were the outcome measures reliable?

This question seeks to determine how reliable (that is, how consistently the method measures a particular outcome) and valid (that is, the method measures what it claims to measure) the outcome measures were. For example, a study assessing effectiveness of a smoking cessation intervention may report on a number of outcomes using a number of different tools, including self-reported smoking rates (a subjective outcome measure that is often unreliable) and biochemically validated smoking rates (an objective outcome measure that is likely to be more reliable).

If the outcome measures were subjective, it is also important to consider if the participant or researcher was blinded to the intervention or exposure (see question 2.4) as blinding may rescue the reliability of some subjective outcome measures.

3.2 Were the outcome measurements complete?

3.3 Were all important outcomes assessed?

3.4 Were outcomes relevant?

3.5 Were there similar follow-up times in exposure and comparison groups?

It is possible to overcome differences in the length of follow-up between groups in the analyses, for example, by adjusting the denominator to take the time into account (by using person-years).

3.6 Was follow-up time meaningful?

The duration of post-intervention follow-up of participants should be of an adequate length to identify the outcome of interest.

4.1 Were the exposure and comparison groups similar at baseline? If not, were these adjusted?

Studies may report the distributions or important differences in potential confounding factors between intervention groups. However, formal tests comparing the groups are problematic – failure to detect a difference does not mean a difference does not exist, and multiple comparisons of factors may falsely detect some differences that are not real.

It is important to assess whether all likely confounders have been considered. Confounding factors may differ by outcome, so potential confounding factors for all of the outcomes that are of interest will need to be considered.

4.2 Intention to treat analysis?

4.3 Was the study sufficiently powered to detect an intervention effect (if one exists)?

For cluster RCTs in particular, it is important to consider whether the cluster design has been appropriately taken into account in calculating required sample size for adequate power.

4.4 Were estimates of effect size given or calculable?

4.5 Were the analytical methods appropriate?

There are a large number of considerations in deciding whether analytical methods were appropriate. For example, it is important to review the appropriateness of any subgroup analyses (and whether pre-specified or exploratory) that are presented. Although subgroup analyses can often provide valuable information on which to base further research (that is, are often exploratory), it is important that findings of subgroup analyses are not over (or under) emphasised. Meaningful results from subgroup analyses are beset by the problems of multiplicity of testing (in which the risk of a false positive result increases with the number of tests performed) and low statistical power (that is, studies generally only enrol sufficient participants to ensure that testing the primary study hypothesis is adequately powered) (Assmann et al. 2000). In a good quality paper, subgroup analyses are restricted to pre-specified subgroups and are often confined to primary outcome measures. Data are analysed using formal statistical tests of interaction (that assess whether intervention effect differs between subgroups) rather than comparison of subgroup p values. A correction for multiple testing is performed where appropriate (for example, 'Bonferroni correction' where a stricter significance level is used to define statistical significance). The results are delineated carefully, and full details of how analyses were performed are provided (Assmann et al. 2000; Guillemin 2007).

The appropriateness of some analytical methods will also depend on the study design under investigation. For example, with cluster RCTs, because participants are randomised at the group level and are not independent 'units' (as is the case with RCTs based on individuals without clustering), and outcomes are often assessed at the individual level, statistical adjustments are necessary before pooled intervention and control group outcomes can be compared.

Likewise, it is also important to consider whether the degree of similarity or difference in clusters has been considered in analyses of cluster RCTs. Good quality cluster-RCTs will determine the intra-class correlation coefficient of their study (a statistical measure of the interdependence in each cluster that is calculated by taking the ratio of the variance between groups compared with variance in groups).

Studies may also report other forms of statistical analysis such as regression, time series, factor analysis and discriminant analysis, as well as epidemiological or economic modelling. Economic modelling is covered in more detail in chapter 6. The other topics are specialised and advice should be sought from the NICE project team before attempting to assess such studies.

5.1 Are the study results internally valid (that is, unbiased)?

5.2 Are the findings generalisable to the source population (that is, externally valid)?