Guidance

Appendix B: Summary of the methods used to develop this guidance

Appendix B: Summary of the methods used to develop this guidance

Introduction

The reviews and economic analysis include full details of the methods used to select the evidence (including search strategies), assess its quality and summarise it.

The minutes of the PHIAC meetings provide further detail about the Committee's interpretation of the evidence and development of the recommendations.

All supporting documents are listed in appendix E.

Key questions

The key questions were established as part of the scope. They formed the starting point for the reviews of evidence and were used by PHIAC to help develop the recommendations. The overarching questions were:

Question 1: What types of road design or modification to the road and street environment are effective and cost effective in reducing road injuries among children and young people aged under 15?

Question 2: What are the barriers and facilitators to implementing environmental modifications and road/street designs relating to the reduction of road injuries?

The subsidiary question was:

Question 3: What are the barriers and facilitators to implementing environmental modifications and designs relating to the reduction of vehicle speeds and road injuries?

These questions were made more specific for the reviews (see reviews for further details).

Reviewing the evidence

One review of effectiveness and cost effectiveness was conducted, and 1 review of barriers and facilitators.

Identifying the evidence

The following databases were searched for evaluations (prospective or retrospective) of relevant interventions that used comparative designs (randomised controlled trials [RCTs], non-randomised controlled trials, before-and-after studies, or natural experiments); full economic evaluations and high quality costing studies conducted in the UK or countries of a similar level of economic development, patterns of transport use and urban environment; primary qualitative research involving the analysis of written or spoken evidence regarding attitudes towards, or experiences of, the relevant interventions, qualitative surveys of attitudes towards, or experiences of the relevant interventions:

  • Applied Social Science Index and Abstracts (ASSIA)

  • Bibliomap

  • Centre for Review and Dissemination

  • Database of Abstracts of Reviews of Effects (DARE)

  • Database of Promoting Health Effectiveness Reviews (DoPHER)

  • EPPI CENTRE databases

  • ERIC

  • Health Management Information Consortium (HMIC)

  • MEDLINE

  • MEDLINE In Process

  • National Health Service Economic Evaluations Database (NHSEED)

  • NHS Economic Evaluation Database (Health Technology Assessment)

  • PsycINFO

  • SafetyLit

  • Social Science Citation Index

  • Transport Research Information Service (TRIS)

  • Trials Register of Promoting Health Interventions TRoPHI

A follow up targeted search was done in TRIS and MEDLINE of specific named programmes and additional traffic-calming methods determined from the results of the original database searches.

Quality appraisal

Included papers were assessed for methodological rigour and quality using the NICE methodology checklist, as set out in the NICE technical manual 'Methods for the development of NICE public health guidance' (see appendix E). Each study was graded (++, +, –) to reflect the risk of potential bias arising from its design and execution.

Study quality

++ All or most of the methodology checklist criteria have been fulfilled. Where they have not been fulfilled, the conclusions are thought very unlikely to alter.

+ Some of the methodology checklist criteria have been fulfilled. Those criteria that have not been fulfilled or not adequately described are thought unlikely to alter the conclusions.

– Few or no methodology checklist criteria have been fulfilled. The conclusions of the study are thought likely or very likely to alter.

Summarising the evidence and making evidence statements

The review data was summarised in evidence tables (see full reviews).

The findings from the reviews were synthesised and used as the basis for a number of evidence statements relating to each key question. The evidence statements were prepared by the external contractors/public health collaborating centres (see appendix A). The statements reflect their judgement of the strength (quantity, type and quality) of evidence and its applicability to the populations and settings in the scope.

Economic analysis

The economic analysis consisted of a review of economic evaluations (the cost effectiveness part of review 1) and a cost-effectiveness modelling (report 3).

Cost-effectiveness review (part of review 1)

A wide range of electronic databases was searched, including some that are specific to the areas of transport policy/research and safety policy/research. Papers or reports were sought that reported quantitative comparative evaluations of local or regional interventions to reduce injuries in children aged under 15 by road/street design or by modifying the road/street environment and highway design (for example, measures to reduce speed and 20 mph zones).

Studies were included if they were full economic evaluations of relevant types of intervention or scheme, and high quality costing studies conducted in the UK or countries of a similar level of economic development, patterns of transport use and urban environment.

Studies were excluded if they were cost-of-illness studies, or other studies that did not involve assessing the cost and related benefits/effectiveness of particular interventions (or class of intervention). Of 19 identified as potentially relevant 13 were included, all of which were cost–benefit analyses.

Cost-effectiveness modelling

A number of assumptions were made that could underestimate or overestimate the cost effectiveness of the interventions (see modelling report for further details).

Economic modelling was undertaken to explore the cost effectiveness of mixed priority route schemes, mandatory 20 mph zones and advisory 20 mph limits. The results are presented as net present values as well as costs per quality adjusted life-year. (Net present value or NPV determines the total monetary benefit of an intervention less its costs - compared with an alternative intervention – when discounted to its present value. A positive NPV occurs when the sum of the discounted benefits exceeds the sum of the discounted costs.) They can be found in the cost-effectiveness modelling report of road and street design-based interventions aimed at reducing unintentional injuries in children.

Fieldwork

Fieldwork was carried out to evaluate how relevant and useful NICE's recommendations are for practitioners and how feasible it would be to put them into practice. It was conducted with practitioners and commissioners who are involved in unintentional injury and transport services. They included those working in community and charitable organisations, local transport planning departments, directors of public health and NHS and government representatives.

The fieldwork comprised:

  • Six half-day workshops (2 each in Birmingham, London and Manchester) with practitioners including community and charitable organisations, local transport planning departments, directors of public health and NHS, government representatives.

  • Seven focus groups (in Hull, London, Manchester, Milton Keynes, Oxford, Portsmouth and Stockport) with transport planning departments in local authorities and within the private sector, including:

    • chartered civil engineers

    • members of the social inclusion team

    • road safety engineers

    • transport planners

    • road safety managers.

  • Telephone interviews with:

    • transport planners

    • a policy planner.

The studies were commissioned to ensure there was ample geographical coverage. The main issues arising from these studies are set out in fieldwork findings in appendix C. See the full report on prevention of unintentional road injury in under-15s: road design.

How PHIAC formulated the recommendations

At its meeting in July 2009 PHIAC considered the evidence to determine:

  • whether there was sufficient evidence (in terms of strength and applicability) to form a judgement

  • where relevant, whether (on balance) the evidence demonstrates that an intervention or programme is effective or ineffective or whether the evidence is inconclusive

  • where relevant, the typical size of effect (where there is one)

  • whether the evidence is applicable to the target groups and contexts covered by the guidance.

PHIAC developed draft recommendations through informal consensus, based on the following criteria:

  • Strength (type, quality, quantity and consistency) of the evidence.

  • The applicability of the evidence to the populations/settings referred to in the scope.

  • Effect size and potential impact on the target population's health.

  • Impact on inequalities in health between different groups of the population.

  • Equality and diversity legislation.

  • Ethical issues and social value judgements.

  • Cost effectiveness (for the NHS and other public sector organisations).

  • Balance of harms and benefits.

  • Ease of implementation and any anticipated changes in practice.

PHIAC noted that effectiveness can vary according to context. For instance, the effectiveness of interventions on mixed priority routes varied with the initial casualty rate.

Where possible, recommendations were linked to evidence statements (see appendix C for details). Where a recommendation was inferred from the evidence, this was indicated by the reference 'IDE' (inference derived from the evidence).

The draft guidance, including the recommendations, was released for consultation in November 2009. At its meeting in January 2010, PHIAC amended the guidance in light of comments from stakeholders, experts and the fieldwork. The guidance was signed off by the NICE Guidance Executive in March 2010.