Guidance

Appendix B: Summary of the methods used to develop this guidance

Appendix B: Summary of the methods used to develop this guidance

Introduction

The reports of the reviews and economic appraisal include full details of the methods used to select the evidence (including search strategies), assess its quality and summarise it.

The minutes of the Programme Development Group (PDG) meetings provide further detail about the Group's interpretation of the evidence and development of the recommendations.

All supporting documents are listed in appendix E.

Key questions

The key questions were established as part of the scope. Initially they formed the starting point for the reviews of evidence and facilitated the development of recommendations by the PDG. The overarching question was: What are the most appropriate generic and specific interventions to support attitude and behaviour change at population and community levels? The subsidiary questions were:

1. What is the aim/objective of the intervention?

2. How does the content of the intervention influence effectiveness?

3. How does the way that the intervention is carried out influence effectiveness?

4. Does effectiveness depend on the job title/position of the deliverer (leader)? What are the significant features of an effective deliverer (leader)?

5. Does the site/setting of delivery of the intervention influence effectiveness?

6. Does the intensity (or length) of the intervention influence effectiveness/duration of effect?

7. Does the effectiveness of the intervention vary with different characteristics within the target population such as age, sex, class and ethnicity?

8. How much does the intervention cost (in terms of money, people and time)? What evidence is there on cost effectiveness?

9. Implementation: what are the barriers to implementing effective interventions?

These questions were refined further in relation to the topic of each review (see reviews for further details).

Reviewing the evidence of effectiveness

Six reviews of the evidence, 1 cost-effectiveness review and 1 economic modelling report were conducted. In addition, a number of important theoretical and methodological principles were taken into account.

The empirical evidence about behaviour change is very varied and methodologically diverse. Areas of focus can include 1 or more of the following:

  • the individual, including the psychological processes affecting individuals

  • social factors

  • large-scale policy and legislative arrangements

  • empirical investigations and observations

  • propositional and modelling approaches.

Identifying the evidence

It is not always appropriate – or even possible – to carry out controlled trials or gather experimental evidence for public health interventions, including those covering legislation or policy. The search process initially followed standard NICE processes. However, as relatively little evidence on behaviour change addresses effectiveness or cost effectiveness, the review of the literature was extended to cover theoretical, descriptive and empirical studies of a type not normally reviewed for NICE guidance.

The goal of the primary studies varied and included efficacy, effectiveness, the theoretical elegance of models, implementation and programme evaluation. Some studies included all or some of these elements. The economic modelling for this guidance reflected the state of the literature.

There are few evidenced-based reviews on the effect of behaviour change interventions on social and health inequalities. There is evidence that the uptake of interventions or response to health education messages differs by social circumstances, and this has historically, widened the health inequalities gap. Evidence about interventions intended to narrow the health inequalities gap had to be drawn from the outcomes and methods described in other sorts of literature.

Databases were searched to identify the evidence relevant for each review. Since very different types of evidence were being gathered for each review, no common core set of databases was searched.

Further details of the databases, search terms and strategies used are included in the individual review reports.

Summary of reviews

  • Review 1 included systematic reviews and meta-analyses which focused on public health, health promotion or primary care-led interventions which contained an educational or behavioural component.

  • Review 2 (part 1) included reviews of intervention studies that evaluated the effectiveness of road safety interventions. Part 2 included reviews of intervention studies that evaluated the effectiveness of 'pro-environmental behaviour'.

  • Review 3 (part 1) included reviews that provided an overview of conceptual, theoretical or research issues in relation to resilience, coping and salutogenesis. It also included reviews of interventions explicitly linked to 1 of these theories. Part 2 included reviews of empirical evidence on positive adaptation in conditions of socio-structural adversity.

  • Review 4 included reviews of 4 behaviour change models.

  • Review 5 included reviews of empirical data on the effectiveness of interventions designed to change knowledge, attitude, intention and behaviour with respect to smoking, physical activity and healthy eating. Specific attention was focused on whether or not effectiveness was influenced by the individual's position in the life course, the intervention's mode of delivery or the social and cultural context.

  • Review 6 included reports on the strategies used by marketeers to influence low-income consumers and any evidence of effectiveness.

Quality appraisal

Papers included in the reviews and additional empirical and theoretical data were assessed where appropriate for methodological rigour and quality using the NICE methodology checklist. This is set out in the NICE technical manual 'Methods for development of NICE public health guidance'. Each study or paper was described by study type and graded (++, +, -) to reflect the risk of potential bias arising from its design and execution.

Study type

  • Meta-analyses, systematic reviews of randomised controlled trials (RCTs) or RCTs (including cluster RCTs).

  • Systematic reviews of, or individual, non-randomised controlled trials, case-control studies, cohort studies, controlled before-and-after (CBA) studies, interrupted time series (ITS) studies, correlation studies.

  • Non-analytical studies (for example, case reports, case series).

  • Expert opinion, formal consensus, theoretical articles.

Study quality

++ All or most of the checklist criteria have been fulfilled. Where they have not been fulfilled the conclusions are thought very unlikely to alter.

+ Some of the checklist criteria have been fulfilled. Those criteria that have not been fulfilled or not adequately described are thought unlikely to alter the conclusions.

- Few or no checklist criteria have been fulfilled. The conclusions of the study are thought likely or very likely to alter.

The studies or papers were also assessed for their applicability to the UK where this was possible and the evidence statements were graded as follows:

A. Relevant – review makes direct reference to a UK population.

B. Probably relevant – review from outside UK but most likely equally applicable to UK settings.

C. Possibly relevant – review from outside UK and needs interpreting with caution for a UK setting.

D. Not relevant – review is from outside UK and is not relevant to a UK setting.

Summarising the evidence and making evidence statements

The review data were summarised in evidence tables (see full reviews). The findings from the reviews were synthesised and used as the basis for a number of evidence statements relating to each key question. The evidence statements reflect the strength (quantity, type and quality) of evidence and its applicability to the populations and settings in the scope.

Economic appraisal

The economic appraisal consisted of a review of economic evaluations and a model of cost effectiveness.

Review of economic evaluations

A systematic search of Medline, Embase, NHS EED, OHE HEED, NCCHTA, CEA Registry (Harvard University) was undertaken in June 2006, using a specified set of search terms, as well as inclusion and exclusion criteria. Following a review of 4,122 abstracts and 225 papers, 26 papers were retained for full review, using a standard set of piloted questions. The data extracted included: background, population characteristics, interventions and alternatives, main features and findings and 3 sets of quality review criteria.

Cost-effectiveness analysis

An economic model was constructed to incorporate data from the reviews of effectiveness and cost effectiveness. The results are reported in: 'The cost-effectiveness of population level interventions to lower cholesterol and prevent coronary heart disease: extrapolation and modelling results on promoting healthy eating habits from Norway to the UK'. See the economic model and results.

Fieldwork

Fieldwork was carried out to evaluate the relevance and usefulness of NICE guidance and the feasibility of implementation. It was conducted with policy makers, commissioners, service providers and practitioners whose work involves changing people's health behaviour. They included those working in local and national government, the NHS and in charitable organisations.

The fieldwork comprised:

  • Qualitative interviews carried out by Dr Foster Intelligence with 97 individuals, either in small groups or individually, across 30 sites. Participants included: representatives from the DH, other government departments and arm's length bodies; directors of public health in PCTs and strategic health authorities; public health advisers, health promotion staff and NHS practitioners (including GPs, practice nurses, community midwives, health visitors and health advisers); community-based school nurses; health trainers; and commissioners, service providers and practitioners working in local and national charities.

The fieldwork was conducted in London, Greater Manchester and the West Midlands to ensure there was ample geographical coverage. Grid analysis was used to determine common ground and differences of opinion.

The main issues arising from the fieldwork are set out in fieldwork findings in appendix C. See the full fieldwork report.

How the PDG formulated the recommendations

At its meetings held between July 2006 and February 2007, the PDG considered the evidence of effectiveness and cost effectiveness and theoretical and methodological evidence. Initially, discussions focused on the evidence outlined in the reviews (see appendix B). The PDG also considered evidence on cost effectiveness, evidence from fieldwork, additional review material and a range of theoretical and methodological approaches (see appendix C).

In addition, at its meeting in May 2007 it considered comments from stakeholders and the results from fieldwork to determine:

  • whether there was sufficient evidence (in terms of quantity, quality and applicability) to form a judgement

  • whether, on balance, the evidence demonstrates that the intervention is effective or ineffective, or whether it is equivocal

  • where there is an effect, the typical size of effect.

The PDG developed draft recommendations through informal consensus, based on the theoretical ideas that informed its view of behaviour, and the degree to which the available effectiveness evidence could support these ideas.

The draft guidance, including the recommendations, was released for consultation in April 2007. The guidance was signed off by the NICE Guidance Executive in September 2007.