Guidance

Appendix B: Summary of the methods used to develop this guidance

Appendix B: Summary of the methods used to develop this guidance

Introduction

The reports of the review and economic appraisal include full details of the methods used to select the evidence (including search strategies), assess its quality and summarise it.

The minutes of the PHIAC meetings provide further detail about the Committee's interpretation of the evidence and development of the recommendations.

All supporting documents are listed in appendix E.

Key questions

The key questions were established as part of the scope. They formed the starting point for the review of evidence and facilitated the development of recommendations by PHIAC. The overarching question was:

Which workplace-based policies and initiatives that aim to increase employees' physical activity levels are effective and cost effective, and what are the barriers to participation?

Subsidiary questions were:

1. What is the aim/objective of the intervention?

2. How does the way it is delivered influence effectiveness?

3. Does the degree to which employees are involved in the planning, implementation and review of interventions influence their effectiveness?

4. Does the duration, frequency or intensity of the intervention influence its impact?

5. Does the type of workplace influence effectiveness?

6. What are the most effective and appropriate interventions for different sectors of the workforce such as: men and women, younger and older workers, people from different socioeconomic backgrounds or minority ethnic groups and temporary or casual workers?

7. Does the intervention have an impact on health inequalities?

8. Does effectiveness vary according to the type of job people do?

9. What are the key components of the intervention that motivate individuals to become more physically active?

10. What are the barriers and facilitators to implementation – for both employers and employees?

11. Is there any evidence of cost effectiveness?

12. How can employers be encouraged to promote physical activity at work?

13. What are the resource needs of large, medium and small enterprises in promoting physical activity at work?

Reviewing the evidence of effectiveness

A review of effectiveness was conducted.

Identifying the evidence

The following databases were searched for relevant systematic reviews, experimental studies and qualitative studies (from 1996–2006):

  • ABI Inform

  • Applied Social Science Index and Abstracts (ASSIA)

  • CENTRAL

  • Cinahl

  • Cochrane Database of Systematic Reviews (CDSR)

  • Database of abstracts of reviews of effects (DARE)

  • Embase

  • PsycINFO

  • Social Policy and Practice

  • Social Science Citation Index

  • SportDiscus

  • Transport.

Searches of a range of websites were performed to identify any further projects. The electronic searches were supplemented by hand-checking of the references of all papers included at the 'screening full papers' stage.

Selection criteria

Studies were included in the effectiveness review if:

  • the intervention aimed to increase physical activity levels

  • the intervention was aimed at employed adults

  • the intervention was initiated or endorsed by the employer

  • physical activity was an outcome measure

  • there was evidence of a change in physical activity levels – or a change of behaviour relating to physical activity

  • the paper was published in English in 1996 or later

  • the review included papers published in 1996 or later.

Studies were excluded if:

  • the intervention was aimed at self-employed or unemployed adults

  • the intervention involved a modification to the built or natural environment

  • there was no report or measure of a change in physical activity

  • the document was a dissertation

  • the papers focused on costs or cost effectiveness only.

Quality appraisal

Included papers were assessed for methodological rigour and quality using the NICE methodology checklist, as set out in the NICE technical manual 'Methods for development of NICE public health guidance' (see appendix E). Each study was described by study type and graded (++, +, -) to reflect the risk of potential bias arising from its design and execution.

Study type

  • Meta-analyses, systematic reviews of randomised controlled trials (RCTs) or RCTs (including cluster RCTs).

  • Systematic reviews of, or individual, non-randomised controlled trials, case-control studies, cohort studies, controlled before-and-after (CBA) studies, interrupted time series (ITS) studies, correlation studies.

  • Non-analytical studies (for example, case reports, case series).

  • Expert opinion, formal consensus.

Study quality

++ All or most criteria have been fulfilled. Where they have not been fulfilled the conclusions are thought very unlikely to alter.

+ Some criteria fulfilled. Those criteria that have not been fulfilled or not adequately described are thought unlikely to alter the conclusions.

- Few or no criteria fulfilled. The conclusions of the study are thought likely or very likely to alter.

The interventions were also assessed for their applicability to the UK and the evidence statements were graded as follows:

A. Likely to be applicable across a broad range of settings and populations

B. Likely to be applicable across a broad range of settings and populations, assuming they are appropriately adapted

C. Applicable only to settings or populations included in the studies – broader applicability is uncertain

D. Applicable only to settings or populations included in the studies.

Summarising the evidence and making evidence statements

The review data was summarised in evidence tables (see full review).

The findings from the review were synthesised and used as the basis for a number of evidence statements relating to each key question. The evidence statements reflect the strength (quantity, type and quality) of evidence and its applicability to the populations and settings in the scope.

Additional evidence

PHIAC wanted to know whether evidence from the US was consistent with evidence found in the review from the UK, Europe, Australia, New Zealand and Canada. The US evidence was surveyed by the CPHE team and presented to PHIAC as an additional evidence paper on summary of the US evidence as it relates to the draft workplace physical activity recommendations.

Economic analysis

The economic appraisal consisted of a review of economic evaluations and a cost-effectiveness analysis.

Review of economic evaluations

In addition to scanning the effectiveness evidence the following databases were searched:

  • Econlit

  • Health Economics Evaluation Database (HEED)

  • NHS Economics Evaluation Database (NHS EED).

Literature published in English was considered from 1990 onwards. No geographical restrictions were placed on the search strategy.

A search of grey literature was also undertaken. The IDEAS economic database was searched specifically for cost-effectiveness information. The cost-effectiveness studies resulting from the grey literature search were highlighted by the effectiveness review team.

Studies that met the inclusion criteria were rated to determine the strength of the evidence using the Drummond checklist (Drummond MF, Jefferson TO [1996] Guidelines for authors and peer reviewers of economic submissions to the BMJ. The BMJ Economic Evaluation Working Party. British Medical Journal 313: 275–83).

Cost-effectiveness analysis

Economic models were constructed to incorporate data from the reviews of effectiveness and cost effectiveness. The results of the economic models are reported in: 'An economic analysis of workplace interventions that promote physical activity'.

Fieldwork

Fieldwork was carried out to evaluate the relevance and usefulness of NICE guidance for practitioners and the feasibility of implementation. It was conducted with practitioners, employers, employee representatives and others who are involved in promoting workplace physical activity. They included: those working in occupational health, health and safety and health promotion in the NHS and local authorities; those in the voluntary sector with a remit for improving physical activity; small, medium and large private businesses; organisations representing the interests of businesses; employee organisations (for example, trades unions).

The fieldwork comprised:

  • a series of focus group meetings held in London by Greenstreet Berman with representatives from the public, private and voluntary sectors

  • a series of telephone interviews carried out nationally by Greenstreet Berman with small, medium and large employers and other representatives from the public, private and voluntary sectors.

It was commissioned to ensure there was ample geographical coverage. The main issues arising from these two studies are set out in appendix C under fieldwork findings. The full fieldwork report is 'Fieldwork on the promotion of physical activity in the workplace'.

How PHIAC formulated the recommendations

At its meeting in October 2007 PHIAC considered the evidence of effectiveness and cost effectiveness to determine:

  • whether there was sufficient evidence (in terms of quantity, quality and applicability) to form a judgement

  • whether, on balance, the evidence demonstrates that the intervention is effective or ineffective, or whether it is equivocal

  • where there is an effect, the typical size of effect.

PHIAC developed draft recommendations through informal consensus, based on the following criteria.

  • Strength (quality and quantity) of evidence of effectiveness and its applicability to the populations/settings referred to in the scope.

  • Effect size and potential impact on population health and/or reducing inequalities in health.

  • Cost effectiveness (for the NHS and other public sector organisations).

  • Balance of risks and benefits.

  • Ease of implementation and the anticipated extent of change in practice that would be required.

Where possible, recommendations were linked to an evidence statements (see appendix C for details). Where a recommendation was inferred from the evidence, this was indicated by the reference 'IDE' (inference derived from the evidence).

The draft guidance, including the recommendations, was released for consultation in December 2007. At its meeting in February 2008, PHIAC considered comments from stakeholders and the results from fieldwork and amended the guidance. The guidance was signed off by the NICE Guidance Executive in April 2008.