Guidance

Appendix B: Summary of the methods used to develop this guidance

Appendix B: Summary of the methods used to develop this guidance

Introduction

The reviews and cost effectiveness modelling report include full details of the methods used to select the evidence (including search strategies), assess its quality and summarise it.

The minutes of the PHIAC meetings provide further detail about the Committee's interpretation of the evidence and development of the recommendations.

All supporting documents are listed in appendix E.

Key questions

The key questions were established as part of the scope. They formed the starting point for the reviews of evidence and were used by PHIAC to help develop the recommendations. The overarching questions were:

Question 1: Which interventions involving the supply and/or installation of home safety equipment are effective and cost effective in preventing unintentional injuries among children and young people aged under 15 in the home?

Question 2: Are home-risk assessments effective and cost effective in preventing unintentional injuries among children and young people aged under 15?

Question 3: What are the barriers to, and facilitators of, interventions involving the supply and/or installation of home safety equipment and/or home-risk assessments?

These questions were made more specific for the reviews (see reviews for further details).

Reviewing the evidence

Two evidence reviews were carried out: one on effectiveness and cost effectiveness and one on the barriers to, and facilitators of, the prevention of unintentional injury in children in the home.

Identifying the evidence

The following databases were searched from 1990 up to March 2009, using a single strategy to identify relevant primary and qualitative research (no study design filters were applied):

  • Applied Social Science Index and Abstracts (ASSIA)

  • Bibliomap

  • Centre for Review and Dissemination databases

  • CINAHL (Cumulative Index of Nursing and Allied Health Literature)

  • Cochrane Library database of systematic reviews

  • Database of Abstracts of Reviews of Effects (DARE)

  • Database of Promoting Health Effectiveness Reviews (DoPHER)

  • EconLit

  • Evidence for Policy and Practice Information and Co-ordinating (EPPI) Centre databases

  • Health Management Information Consortium (HMIC)

  • ISI Web of Knowledge Social Science Citation Index (SSCI)

  • Science Citation Index Expanded (SCI-EXPANDED)

  • MEDLINE

  • National Health Service Economic Evaluations Database (NHSEED)

  • NHS Economic Evaluation Database (HTA)

  • PsycINFO

  • SafetyLit

  • Trials Register of Promoting Health Interventions (TRoPHI)

A follow-up targeted search of named programmes was conducted in MEDLINE and using the search engine Google.

The following websites were also searched:

Further details of the databases, search terms and strategies are included in the reviews.

Selection criteria

Studies were included in the effectiveness and cost effectiveness review if they:

  • were published from 1990 to March 2009 in English

  • were conducted in member countries of the Organisation for Economic Cooperation and Development (OECD)

  • reported injury related outcomes (for example, a reduction in injuries from smoke inhalation, an increase in the number of smoke alarms installed and improved knowledge of how to prevent other injuries in the home).

Studies were excluded if they did not:

  • compare the injury-related outcome prior to or without the intervention report injury-related outcomes for children or young people aged under 15 (for examples, see above; however, studies that reported injury-related outcomes among, for example, those aged 5–18 years would be included if most of the data related to children aged 15 years or under.)

  • for the cost-effectiveness review only, assess the cost and related benefits or effectiveness of the intervention (or class of intervention).

Quality appraisal

Included papers were assessed for methodological rigour and quality using the relevant NICE methodology checklist, as set out in the NICE technical manual 'Methods for the development of NICE public health guidance' (see appendix E). Each study was graded (++, +, –) to reflect the risk of potential bias arising from its design and execution.

Study quality

++ All or most of the methodology checklist criteria have been fulfilled. Where they have not been fulfilled, the conclusions are thought very unlikely to alter.

+ Some of the methodology checklist criteria have been fulfilled. Those criteria that have not been fulfilled or not adequately described are thought unlikely to alter the conclusions.

– Few or no methodology checklist criteria have been fulfilled. The conclusions of the study are thought likely or very likely to alter.

Summarising the evidence and making evidence statements

The review data was summarised in evidence tables (see full reviews).

The findings from the reviews were synthesised and used as the basis for evidence statements relating to each key question. The evidence statements were prepared by the public health collaborating centre (see appendix A). The statements reflect their judgement of the strength (quantity, type and quality) of evidence and its applicability to the populations and settings in the scope.

Economic analysis

The economic analysis consisted of a review of economic evaluations (the cost effectiveness part of review 1) and a cost-effectiveness model (report 3).

Cost effectiveness review (part of review 1)

As indicated above, a single search strategy was used to identify relevant economic evaluations from a wide range of databases (listed earlier).

Cost-effectiveness modelling

Two economic models were constructed to incorporate data from the evidence reviews.

First, the intervention model was used to analyse the effectiveness of an intervention to increase the number of people using a particular safety feature (such as a smoke alarm or stair gate) in the home.

The second stage outcomes model used the levels of installed safety equipment in the population (derived from the first model) to predict the number of resulting injuries and fatalities over the lifetime of the population cohort. It involved a cost–utility analysis undertaken from the NHS and personal social services perspective.

A number of assumptions were made which could underestimate or overestimate the cost effectiveness of the interventions (see review modelling report for further details).

The results are reported in: Preventing unintentional injuries among under-15s in the home. Cost-effectiveness modelling of home-based interventions aimed at reducing unintentional injuries in children.

Fieldwork

Fieldwork was carried out to evaluate how relevant and useful NICE's recommendations are for practitioners and how feasible it would be to put them into practice.

It was conducted with practitioners and commissioners who are involved in preventing unintentional injuries among under-15s. They included: unintentional injury prevention specialists; practitioners working on local home-safety initiatives, including safety equipment distribution schemes; and practitioners with a broader remit for the welfare of children aged 0–15. The latter included: children's centre managers, health visitors, housing managers, public health practitioners, school nurses, social workers and others working in the NHS, local authorities, police and fire services, and voluntary sector organisations.

The fieldwork comprised nine focus groups carried out in different local authority areas and one in-depth interview. They were conducted by GHK (with Noble Denton) and involved a total of 65 participants.

The focus groups and in-depth interview were commissioned to ensure there was ample geographical coverage. The main issues arising are set out in appendix C fieldwork findings. See the full report, Preventing unintentional injuries in the home among under-15s: providing safety equipment and home-risk assessments: fieldwork report.

How PHIAC formulated the recommendations

At its meeting in September 2009 PHIAC considered the evidence of effectiveness and cost effectiveness to determine:

  • whether there was sufficient evidence (in terms of quantity, quality and applicability) to form a judgement

  • whether, on balance, the evidence demonstrates that the intervention is effective, ineffective or equivocal

  • where there is an effect, the typical size of effect.

PHIAC developed draft recommendations through informal consensus, based on the following criteria.

  • Strength (quality and quantity) of evidence of effectiveness and its applicability to the populations/settings referred to in the scope.

  • Effect size and potential impact on the target population's health.

  • Impact on inequalities in health between different groups of the population.

  • Cost effectiveness (for the NHS and other public sector organisations).

  • Balance of risks and benefits.

  • Ease of implementation and any anticipated changes in practice.

Where possible, recommendations were linked to evidence statements (see appendix C). Where a recommendation was inferred from the evidence, this was indicated by the reference 'IDE' (inference derived from the evidence).

The draft guidance, including the recommendations, was released for consultation in November 2009. At its meeting in January 2010, PHIAC amended the guidance in light of comments from stakeholders, experts and the fieldwork. The guidance was signed off by the NICE Guidance Executive in March 2010.