The social care guidance manual

1.1 The study addresses an appropriate and clearly focused question

Unless a clear and well-defined question is specified, it will be difficult to assess how well the study has met its objectives or how relevant it is to the question you are trying to answer.

1.2 The cases and controls are taken from comparable populations

Study participants may be selected from the target population (all individuals to which the results of the study could be applied), from the source population (a defined subset of the target population from which participants are selected) or from a pool of eligible people (a clearly defined and counted group selected from the source population). A study that does not include clear definitions of the source population may be rejected.

1.3 The same exclusion criteria are used for both cases and controls

All selection and exclusion criteria should be applied equally to cases and controls. Failure to do so may introduce a significant degree of bias into the results of the study.

1.4 What was the participation rate for each group (cases and controls)?

Differences between the eligible population and the study participants are important because they may influence the validity of the study. A participation rate can be calculated by dividing the number of study participants by the number of people who are eligible to participate. It is more useful if it is calculated separately for cases and controls. If the participation rate is low, or there is a large difference in rate between cases and controls, the study results may be invalid because of differences between participants and non-participants. In these circumstances the study should be downgraded, and rejected if the differences are very large.

1.5 Participants and non-participants are compared to establish their similarities or differences

Even if participation rates are comparable and acceptable, it is still possible that the participants selected to act as cases or controls may differ from other members of the source population in some significant way. A well-conducted case–control study will look at samples of those not participating among the source population to ensure that the participants are a truly representative sample.

1.6 Cases are clearly defined and differentiated from controls

The method of selection of cases is of critical importance to the validity of the study. Investigators have to be certain that cases are truly cases, but must balance this with the need to ensure that the cases admitted into the study are representative of the eligible population. The issues involved in case selection are complex, and should ideally be evaluated by someone with a good understanding of the design of case–control studies. If there is no information on how cases were selected it is probably safest to reject the study as a source of evidence.

1.7 It is clearly established that controls are not cases

Just as it is important to be sure that cases are true cases, it is important to be sure that controls do not have the outcome under investigation. Controls should be chosen so that information on exposure status can be obtained or assessed in a similar way to that used for the selection of cases. If the methods of control selection are not described, the study may be rejected. If different methods of selection are used for cases and controls, the study should be evaluated by someone with a good understanding of the design of case–control studies.

1.8 Measures were taken to prevent knowledge of primary exposure from influencing case ascertainment

If there is a possibility that case ascertainment was influenced by knowledge of exposure status, assessment of any association is likely to be biased. A well-conducted study should take this into account in the design of the study.

1.9 Exposure status is measured in a standard, valid and reliable way

The inclusion of evidence from other sources or previous studies that demonstrate the validity and reliability of the assessment methods, or the fact that the measurement method is a recognised procedure, should increase confidence in study quality.

1.10 The main potential confounders are identified and taken into account in the design and analysis

Confounding is the distortion of a link between exposure and outcome by another factor that is associated with both exposure and outcome. The possible presence of confounding factors is one of the principal reasons why observational studies are not more highly rated as a source of evidence. The report of the study should indicate which potential confounders have been considered, and how they have been assessed or accounted for in the analysis. Social care practitioner (or healthcare professional where appropriate) judgement should be used to consider whether all likely confounders have been taken into account. If the measures used to address the potential effects of confounders are considered inadequate, the study should be downgraded or rejected, depending on how serious the risk of confounding is considered to be. A study that does not address the possibility of confounding should be rejected.

1.11 Have confidence intervals been provided?

Confidence intervals are the preferred method for indicating the precision of statistical results, and can be used to differentiate between an inconclusive study and a study that shows no effect. Studies that report a single value with no assessment of precision should be treated with caution.

1.12 Overall assessment of internal validity

Rate the study for internal validity according to the list below:

++ All or most of the checklist criteria have been fulfilled, where they have not been fulfilled the conclusions are very unlikely to alter

+ Some of the checklist criteria have been fulfilled, where they have not been fulfilled, or not adequately described, the conclusions are unlikely to alter

− Few or no checklist criteria have been fulfilled and the conclusions are likely or very likely to alter