NICE process and methods

Appendix D Glossary of study designs

Appendix D Glossary of study designs

Before-and-after (BA) studies

An approach where the dependent variables are measured before and after an intervention has been delivered. The intervention can either be delivered by the investigator or by others (observational before and after study). An approach that is often called a pre–post study. Study participants in pre- and post-intervention stages can either be the same (A) – as is often the case for simple 1-to-1 intervention studies – or different (B) – as is often the case for assessing large scale interventions.

Before-and-after (BA) studies

Quality appraisal of BA studies

BA studies are generally considered to have lower internal validity than study designs in which outcomes in the intervention (exposed) group are compared with outcomes in a concurrent (unexposed) control group. Important things to consider in appraising a BA study is whether or not there is any evidence for a prevailing 'temporal trend' (for example, a general reduction in population smoking rates) that may confound study findings and whether or not there is any indication of selection bias (for example, have the most or least motivated individuals been selected for participation?). A good quality BA study will demonstrate clear and consistent inclusion and exclusion criteria in subject selection. Likewise, causality between intervention and outcome will often be strengthened when observed changes are significant and occur soon after the intervention.

BA studies should not be confused with non-randomised controlled trials.

Case–control studies

A comparative observational study in which the investigator selects people who have an outcome of interest (for example, developed a disease) and others who have not (controls), and then collects data to determine previous exposure to possible causes. Case–control studies are often reserved for early hypothesis testing or for investigating the causes of rare outcomes.

Case–control studies

Quality appraisal of case–control studies

Case–control studies are generally considered to have lower internal validity than study designs in which outcomes in the intervention (exposed) group are prospectively compared with outcomes in a concurrent (unexposed) control group. Important things to consider in appraising a case–control study is whether or not there is any indication of potential confounding factors; whether there is any indication of selection bias (for example, have the same selection and exclusion criteria been applied equally to both cases and controls?), and whether or not there is likely to be significant recall bias (for example, whether the investigator is able to reliably determine previous exposure to factor of interest for both cases and controls).

Cluster randomised controlled trial

A trial where the unit of randomisation is a cluster of participants (for example, a school). See randomised controlled trial (RCT).

Quality appraisal of cluster randomised controlled trials

As for (individual) RCTs, important things to consider in appraising a cluster RCT is whether or not the method of randomisation is truly random (compared to pseudo-randomisation procedures); whether the allocation of clusters to either intervention or control could have been influenced by the person doing the allocation (allocation concealment); and whether the trial is externally valid – that is, the extent to which the findings of a study are applicable or generalisable beyond the confines of the study itself. It is also important to consider whether appropriate analyses were conducted (that is, for cluster design, analyses of sample size, power, and effect size should be performed on clusters – as the unit of randomisation – rather than individuals).

Cohort studies

An observational study in which a group or 'cohort' of people are observed over time in order to see who develops the outcome of interest. An approach that is often called a longitudinal study. Cohort studies differ from experimental studies such as randomised or non-randomised controlled trials because individuals effectively allocate themselves according to the extent of their exposure to the risk factor of interest. Prospective cohort studies involve following groups of people forward in time to assess who develops the outcome of interest, often by conducting a series of cross-sectional studies. Conversely, in retrospective cohort studies, both the exposure and outcomes of interest all take place in the past relative to the starting point of the study.

Cohort studies

Quality appraisal of cohort studies

Cohort studies are generally considered to be the most reliable observational study design and are particularly useful for examining the effects of harmful exposures. Although they do not allow for the random allocation of study participants to receive or not receive the exposure of interest, they do allow for random sampling from the source population which can minimise some sources of selection bias and confounding. Important things to consider in appraising cohort studies include whether or not there is any indication of selection bias or confounding; in the case of retrospective cohort studies, whether or not there is likely to be significant recall bias (that is, whether the investigator is able to reliably determine previous exposure to factor of interest for both cases and controls), and in the case of prospective cohort studies, whether or not there is likely to be significant withdrawal bias (that is, where a significant number of participants have been lost to follow-up).

Controlled before-and-after (CBA) study

See non-randomised controlled trial (NRCT).

Correlation study

A correlation study is an observational study in which the association (or correlation) between 2 or more variables is investigated. An approach that is often called an ecological or association study. Like cross-sectional studies, correlation studies are descriptive rather than analytical and cannot be used to estimate the relationship between cause and effect.

Correlation study

Quality appraisal of correlation studies

The most important things to consider in appraising the quality of correlation studies is whether or not there are any potential confounding factors and whether there is likely to be significant sources of measurement bias (that is, whether exposures and outcomes are assessed using reliable criteria).

Cross-sectional study

A cross-sectional study is an observational study in which the source population is examined to see what proportion has the outcome of interest, or has been exposed to a risk factor of interest, or both. Cross-sectional studies are generally used to determine the population prevalence of outcomes or exposures. An approach that is often called a survey or a prevalence study. Although cross-sectional studies can often provide useful estimates of disease burden for a particular population, they are less reliable for determining prevalence of very rare conditions or conditions with a short duration. As cross-sectional studies are descriptive rather than analytical, they cannot be used to estimate the relationship between cause and effect.

Cross-sectional study

Quality appraisal of cross-sectional studies

The most important things to consider in appraising the quality of cross-sectional studies is whether or not there are any potential confounding factors and whether or not there is likely to be significant sources of measurement bias (that is, whether exposures and outcomes are assessed using reliable criteria).

Ecological study

See correlation and cross-sectional studies.

Focus group study

A specific type of interview study (see interview study) where a group of people (usually 6–12) is interviewed by 1 or more facilitators or interviewers. The method explicitly includes and uses the group interaction to generate data. Data are managed in a similar way to other interview data, in that the discussions are usually recorded and transcribed and analysis is undertaken on the transcripts rather than the verbal data.

Interrupted time series

An approach in which multiple (more than 2) observations are made on the same individuals, or groups of individuals, over time.

Interrupted time series

Quality appraisal of interrupted time series

The most important things to consider in appraising the quality of interrupted time series are whether or not outcomes were assessed before and after an intervention was delivered; whether it is clear precisely when an intervention took place (that is, in order to compare outcomes before and after intervention); and like before-and-after studies, whether or not there is any evidence for a prevailing 'temporal trend' (for example, a general reduction in population smoking rates) that may confound study findings.

Interview study

A qualitative method of data collection where participant's views are elicited via verbal interviews. Interviews can be structured (that is, reliant on set questions), semi-structured (the interviewer has a general idea of topics and themes to cover and will steer the interview towards these participants) or unstructured (the format and direction of the interview is set by the participant). In addition, interviews can be 1-to-1, or with couples or small groups (see focus groups). They can be conducted in person, on the telephone, or more recently online. Interviews are usually recorded and transcribed and analysis is undertaken on the transcripts rather than the verbal data.

Longitudinal studies

See cohort studies.

Meta-analysis

Should not be confused with systematic review. Meta-analysis is a statistical technique that enables the findings from multiple primary studies (often identified during a systematic review) to be combined.

Non-participant observation

A qualitative methodology where the researcher observes participants as they engage with the phenomenon being researched.

Non-randomised controlled trial (NRCT)

These are trials where participants (or clusters) are allocated to receive either intervention or control (or comparison intervention) but the allocation is not randomised – an approach often called a controlled before-and-after (CBA) study.

Non-randomised controlled trial (NRCT)

Quality appraisal of NRCTs

NRCT's are generally considered to be less reliable than randomised controlled trials as non-randomisation of participants makes selection, participant and observer biases much more likely and confounding factors may exist. The most important thing to consider in appraising NRCT's is whether or not there are likely to be significant baseline differences between groups.

Participant observation

A qualitative methodology where the researcher joins in with the participants as they engage with the phenomenon being researched.

Pre–post study

See before-and-after (BA) studies.

Prevalence study

See cross-sectional study.

Prospective cohort study

See cohort studies.

Randomised controlled trial (RCT)

These are trials where participants (or clusters) are randomly allocated to receive either intervention or control. If well implemented, randomisation should ensure that intervention and control groups only differ in their exposure to treatment.

Randomised controlled trial (RCT)

Quality appraisal of RCTs

RCTs are generally considered to be the most rigorous experimental study design as the randomisation of participants helps to minimise confounding and other sources of bias. Important things to consider in appraising an RCT is whether or not the method of randomisation is truly random (compared to pseudo-randomisation procedures such as consecutive admissions to a clinic); whether the allocation of participants to either intervention or control could have been influenced by the person doing the allocation (allocation concealment); and whether the trial is externally valid – that is, the extent to which the findings of a study are applicable or generalisable beyond the confines of the study itself. It is not unusual for an RCT to have strong internal validity, but poor external validity for example, if stringent selection criteria for entry into the study mean that the study participants fail to reflect the characteristics of the source population.

Retrospective cohort study

See cohort studies.

Survey

Surveys can be used as part of a quantitative or a qualitative methodology. For quantitative surveys, see cross-sectional study. Qualitative surveys try to elicit qualitative data from surveys or questionnaires by providing open ended questions that try to encourage a lengthy response and then treat this data as qualitative data by coding and analysing it.

Systematic review

A systematic review can be defined as a summary of the literature that uses explicit and systematic methods to identify, appraise and summarise the literature according to predetermined criteria. If this description is not present, it is not possible to make a thorough evaluation of the quality of the review.