Guidance

Appendix B: Summary of the methods used to develop this guidance

Appendix B: Summary of the methods used to develop this guidance

Introduction

The reviews, primary research, commissioned reports and economic modelling report include full details of the methods used to select the evidence (including search strategies), assess its quality and summarise it.

The minutes of the Programme Development Group (PDG) meetings provide further detail about the Group's interpretation of the evidence and development of the recommendations.

All supporting documents are listed in appendix E.

Key questions

The key questions were established as part of the scope. They formed the starting point for the reviews of evidence and were used by the PDG to help develop the recommendations. The overarching questions were:

1. Which approaches are effective and cost effective in preventing or reducing unintentional injuries among children and young people aged under 15?

2. Which approaches are effective and cost effective in preventing or reducing unintentional injuries among children and young people aged under 15 from disadvantaged families?

3. Which types of approach effectively (and cost effectively) support and help develop the skills of professionals and others involved in childhood injury prevention?

4. What type of monitoring systems are effective and cost effective in recording and detecting changes in the type, incidence and prevalence of unintentional injuries among children and young people aged under 15?

5. What are the barriers and facilitators to implementing initiatives to prevent unintentional injuries among children and young people aged under 15?

These questions were made more specific for each review (see reviews for further details).

Reviewing the evidence

Effectiveness reviews

Five reviews of effectiveness were conducted. One compared international practice (review 1), one covered quantitative correlates (review 2) and three were reviews of effectiveness (reviews 3–5).

Identifying the evidence

The following databases were searched for the effectiveness reviews (from 1990 to January 2009 [review 1], 1990 to February 2009 [review 2], 1990 to April 2009 [review 3], 1990 to June 2009 [review 4] and 1990 to July 2009 [review 5]):

  • Cochrane Database of Systematic Reviews

  • Database of Abstracts of Reviews of Effectiveness (DARE)

  • EPPI Centre databases (Bibliomap, DoPHER, TRoPHI)

  • Health Management Information Consortium (HMIC)

  • Kings Fund catalogue and Department of Health data

  • Health Technology Assessment (HTA)

  • MEDLINE

  • NHS Economic Evaluation Database (NHS EED)

  • SafetyLit

  • Social Science Citation Index

  • The Campbell Collaboration

In addition, the following databases were searched, as appropriate, for individual reviews (from 1990 to January 2009 [review 1], 1990 to February 2009 [review 2], 1990 to April 2009 [review 3], 1990 to June 2009 [review 4] and 1990 to July 2009 [review 5]):

  • Assia

  • Cinahl

  • Cochrane Injuries Group Register

  • EconLit

  • Embase

  • ISI Web of Science

  • International Transport Research Documentation (ITRD)1

  • PsycINFO

  • SPORTDiscus

  • Transport Research Information Service (via the TRIS)

  • Transport Research Laboratory

Website searches included:

For review 1, searches were primarily conducted by snowball sampling of key organisations and individual contacts, supplemented by Internet searches, including the web pages of international and national organisations. For reviews 2–5, electronic searches of relevant bibliographic databases and selected websites were supplemented by communication with experts and organisations involved in the relevant research or policy areas.

Further details of the databases, search terms and strategies are included in the review reports.

Selection criteria

Studies were included in reviews 1 and 2 if they were published between 1997 and 2009 in English. In addition:

  • Review 1 included studies which reported separately for children in at least two countries (or 'country-sized' regions).

  • Review 2 focused on observational research and intervention studies which quantified the association or relationship between unintentional injuries among children and two or more variables such as exposure to a particular environment or socioeconomic status.

Studies were included in reviews 3–5 if they:

  • were published between January 1990 and February 2009 in English

  • used comparative studies to compare groups of people, places or activities

Quality appraisal

For reviews 1 and 3 to 5, the included papers were assessed for methodological rigour and quality using the NICE methodology checklist, as set out in the NICE technical manual 'Methods for the development of NICE public health guidance' (see appendix E). Each study was graded (++, +, –) to reflect the risk of potential bias arising from its design and execution.

Study quality

++ All or most of the checklist criteria have been fulfilled, where they have not been fulfilled the conclusions are very unlikely to alter.

+ Some of the checklist criteria have been fulfilled. Those criteria that have not been fulfilled or not adequately described are unlikely to alter the conclusions.

– Few or no checklist criteria have been fulfilled. The conclusions of the study are likely or very likely to alter.

The main reasons for studies being assessed as (–) were:

  • lack of control or comparison group

  • lack of baseline equivalence/data

  • inadequately described interventions

  • inadequate analysis and reporting of data.

For reviews 2 to 5, the studies were also assessed for their applicability to the area under investigation and the evidence statements were graded as follows:

  • Directly applicable.

  • Partially applicable.

  • Not applicable.

Summarising the evidence and making evidence statements

The review data was summarised in evidence tables (see full reviews).

The findings from the reviews were synthesised and used as the basis for a number of evidence statements relating to each key question. The evidence statements were prepared by the public health collaborating centres (see appendix A). The statements reflect their judgement of the strength (quality, quantity and consistency) of evidence and its applicability to the populations and settings in the scope.

Cost effectiveness

There was a review of economic evaluations and an economic modelling exercise.

Review of economic evaluations

This sought to identify and review economic evaluations published since 1990 of relevant legislation, regulation or other strategic approaches of interest. The search was undertaken in two stages.

  • First the RefMan database was searched for 'hits' from the five reviews and two related pieces of NICE public health guidance (preventing unintentional injuries to children on the road and in the home).

  • Second, a new search was carried out in EconLit and NHSEED (NHS Economic Evaluation Database) using text words and thesaurus terms covering all types of injuries among children.

Economic modelling

An economic model was constructed to explore the cost-effectiveness of jurisdiction-wide strategic approaches to prevent unintentional injuries among children aged under 15 years.  The exploratory analyses were conducted from a UK public sector perspective.

Two different strategic policies were explored: to reduce unintentional injuries among children and adults on the road and at home. The former focused on legislation or regulations, supported by other activities, introducing mandatory 20mph zones in high casualty residential areas. The latter focused on legislation or regulations, supported by other activities, to promote installation of thermostatic mixer valves in family social housing where children are aged less than 5 years.

Due to a paucity of data, the model explored which factors might be important in determining cost effectiveness.

The results are reported in: Economic modelling of legislation/regulations and related national strategies to promote the wider use of: 20 mph zones in residential areas, and TMVs in social housing for families.

Fieldwork

Fieldwork was carried out to evaluate how relevant and useful NICE's recommendations would be for practitioners and how feasible it would be to put them into practice.

It was conducted with practitioners and commissioners who are involved in preventing unintentional injuries among children and young people. This included those working in primary care trusts (PCTs), local safeguarding children boards and accident prevention and road safety teams. It also included health visitors, nurses and policy leads, within the NHS, those working in the fire and police services, leisure and play services, and environmental health and housing.

The fieldwork comprised:

  • Seven discussion groups conducted in Lancashire, South East London and Sussex by Word of Mouth research consultancy.

  • Forty-nine face-to-face and telephone interviews conducted by Word of Mouth with staff from Lancashire, South East London and Sussex. In addition to the groups listed above, participants also included: an assistant school head, a cycle events organiser and a cycle retailer, further education college curriculum managers, healthy schools managers, paediatricians, staff from children's centres, a safer communities manager, a school governor, social workers and voluntary sector children's services managers.

The main issues arising are set out in appendix C under 'Fieldwork findings'. The full fieldwork report Strategies to prevent unintentional injury among under-15s is available online.

How the PDG formulated the recommendations

At its meetings between February 2009 and July 2010, the Programme Development Group (PDG) considered the evidence, expert testimony and cost effectiveness to determine:

  • whether there was sufficient evidence (in terms of strength and applicability) to form a judgement

  • where relevant, whether (on balance) the evidence demonstrates that the intervention or programme can be effective or is inconclusive

  • where relevant, the typical size of effect (where there is one)

  • whether the evidence is applicable to the target groups and context covered by the guidance.

The PDG developed draft recommendations through informal consensus, based on the following criteria:

  • Strength (type, quality, quantity and consistency) of the evidence.

  • The applicability of the evidence to the populations and settings referred to in the scope.

  • Effect size and potential impact on the target population's health.

  • Impact on inequalities in health between different groups of the population.

  • Equality and diversity legislation.

  • Ethical issues and social value judgements.

  • Cost effectiveness (for the NHS and other public sector organisations).

  • Balance of harms and benefits.

  • Ease of implementation and any anticipated changes in practice.

Where possible, recommendations were linked to an evidence statement(s) (see appendix C for details). Where a recommendation was inferred from the evidence, this was indicated by the reference 'IDE' (inference derived from the evidence).

The draft guidance, including the recommendations, was released for consultation in May 2010. At its meeting in July 2010, the PDG amended the guidance in light of comments from stakeholders and experts and the fieldwork. The guidance was signed off by the NICE Guidance Executive in October 2010.