Appendix B Summary of the methods used to develop this guidance | Social and emotional wellbeing: early years | Guidance

Introduction

The reviews, primary research, commissioned reports and economic modelling report include full details of the methods used to select the evidence (including search strategies), assess its quality and summarise it.

The minutes of the Public Health Interventions Advisory Committee (PHIAC) meetings provide further detail about the Committee's interpretation of the evidence and development of the recommendations.

All supporting documents are listed in appendix E.

Key questions

The key questions were established as part of the scope. They formed the starting point for the reviews of evidence and were used by PHIAC to help develop the recommendations. The overarching questions were:

1. What are the most effective and cost-effective early education, childcare and home-based interventions for helping improve and maintain the cognitive, social and emotional wellbeing of vulnerable children and their families?

2. Which progressive early education, childcare and home-based interventions are effective and cost effective in terms of promoting the cognitive, social and emotional wellbeing of vulnerable children and their families at: 0–3 months, 3 months to 1 year, 1–2 years, and other early-life stages?

3. How can vulnerable children and families who might benefit from early education, childcare and home-based interventions be identified? What factors increase the risk of children experiencing cognitive, social and emotional difficulties? What is the absolute risk posed by these different factors – and in different combinations?

4. How can home-based interventions reduce a child's vulnerability and build resilience to help achieve positive outcomes? In particular, how can interventions help develop a strong and positive child–parent attachment?

5. How can early education and childcare interventions reduce vulnerability and build resilience to help achieve positive outcomes and generally prepare children for school?

6. Which characteristics of an intervention are critical to achieving positive outcomes for vulnerable children and families?

7. What lessons can be learnt from current UK-based programmes aimed at promoting the social and emotional wellbeing of children under 5?

These questions were made more specific for each review (see reviews for further details).

Reviewing the evidence

Effectiveness reviews

Two reviews of effectiveness were conducted. One looked at review-level evidence (review 1), the other focused on primary evaluation studies of UK programmes (review 2). The latter included related qualitative evidence on factors influencing uptake and implementation.

Identifying the evidence

A number of databases and websites were searched for review level and evaluation studies from January 2000. See each review for details of the databases searched.

Additional methods used to identify evidence were as follows:

reference list search of included papers (for reviews 1 and 2)
cited reference searches of included studies in the Web of Knowledge, Scopus and Google Scholar
additional searches in Medline and the Web of Knowledge for key UK programmes
consultation with an expert advisory group.

Selection criteria

Studies were included in the effectiveness reviews (reviews 1 and 2) if the:

populations included vulnerable children aged 0–5 and their families
interventions were 'progressive' and
were provided at home, within early education or childcare settings and
aimed to improve the social and emotional health and cognitive ability of vulnerable under-5s and their families.

Studies were excluded if they focused on:

tools and methods used to assess the risk and diagnose social and emotional problems or a mental health disorder
clinical or pharmacological treatments
support provided by specialist child mental health services.

See each review for details of the inclusion and exclusion criteria.

Other reviews

Review 3 focused on the risk factors associated with children experiencing social, emotional and cognitive difficulties.

Identifying the evidence

The Millennium Cohort database (maintained by the Centre for Longitudinal Studies) was searched for review 3. All records were hand-searched at the title/abstract level to identify relevant publications. See the review for details.

Selection criteria

Studies were included in review 3 if any aspect of a child's social and emotional wellbeing were reported (including behaviour, development and mental health).

Quality appraisal

Included papers were assessed for methodological rigour and quality using the NICE methodology checklist, as set out in the NICE technical manual 'Methods for the development of NICE public health guidance' (see appendix E). Each study was graded (++, +, –) to reflect the risk of potential bias arising from its design and execution.

Study quality

++ All or most of the checklist criteria have been fulfilled. Where they have not been fulfilled, the conclusions are very unlikely to alter.

+ Some of the checklist criteria have been fulfilled. Those criteria that have not been fulfilled or not adequately described are unlikely to alter the conclusions.

– Few or no checklist criteria have been fulfilled. The conclusions of the study are likely or very likely to alter.

Summarising the evidence and making evidence statements

The review data was summarised in evidence tables (see full reviews).

The findings from the reviews were synthesised and used as the basis for a number of evidence statements relating to each key question. The evidence statements were prepared by the external contractors (see appendix A). The statements reflect their judgement of the strength (quality, quantity and consistency) of evidence and its applicability to the populations and settings in the scope.

Commissioned reports

Three expert reports were commissioned.

Expert report 1 summarised the evidence from primary evaluation studies on progressive interventions to promote the social and emotional wellbeing of vulnerable children aged under 5 years. The evidence came from the UK, US, the Netherlands and elsewhere.
Expert report 2 looked at programmes to promote the social and emotional wellbeing of vulnerable children aged under 5 years. It included the results of applying the 'Evidence2Success' standards of evidence.
Expert report 3 looked at the costs and benefits of intervening early with vulnerable children and families to promote their social and emotional wellbeing.

Cost effectiveness

There was a review of economic evaluations and an economic modelling exercise.

Review of economic evaluations

A systematic search of key health and medical databases was undertaken for relevant economic evaluation studies. The inclusion and exclusion criteria were the same as for the systematic review of UK interventions (review 1). Included studies were then quality-assessed.

Economic modelling

The economic modelling comprised two parts: an econometric analysis and the development of an economic model.

An econometric analysis of longitudinal data was undertaken to:

understand the factors determining aspects of social, psychological and cognitive development in early childhood
establish a link between early childhood development and adult outcomes
predict the effects of childhood interventions on long-term outcomes.

An economic model was developed to determine the long-term outcomes of the intervention (home visiting, early education and childcare). It incorporated data from the reviews of effectiveness and the economic evaluation and outputs from the econometric analysis.

The results are reported in the economic modelling reports – see appendix E.

Fieldwork

Fieldwork was carried out to evaluate how relevant and useful NICE's recommendations are for practitioners and how feasible it would be to put them into practice. It was conducted with commissioners and practitioners who are involved in early years services in local authorities, the NHS and the community, voluntary and private sectors. Parents and carers of vulnerable children aged under 5 were also consulted.

The fieldwork comprised:

Sixteen discussion groups with commissioners and practitioners. Two were held in each of 8 local authority areas (Barking and Dagenham, Birmingham, Cambridgeshire, Luton, Northamptonshire, Reading, Sheffield and Tower Hamlets).
Eight discussion groups involving a total of 41 parents and carers. These were held in 8 local authority areas (Barking and Dagenham, Birmingham, Cambridgeshire, Luton, Northamptonshire, Reading, Sheffield and Tower Hamlets).

The main issues arising from the fieldwork are set out in fieldwork findings in appendix C. See also the full fieldwork reports 'The social and emotional wellbeing of vulnerable children (early years): views of professionals' and 'The social and emotional wellbeing of vulnerable children (early years): views of parents and carers'.

How PHIAC formulated the recommendations

At its meetings in January 2012, the Public Health Interventions Advisory Committee (PHIAC) considered the evidence, expert reports and cost effectiveness to determine:

whether there was sufficient evidence (in terms of strength and applicability) to form a judgement
where relevant, whether (on balance) the evidence demonstrates that the intervention or programme/activity can be effective or is inconclusive
where relevant, the typical size of effect (where there is one)
whether the evidence is applicable to the target groups and context covered by the guidance.

PHIAC developed draft recommendations through informal consensus, based on the following criteria:

Strength (type, quality, quantity and consistency) of the evidence.
The applicability of the evidence to the populations/settings referred to in the scope.
Effect size and potential impact on the target population's health.
Impact on inequalities in health between different groups of the population.
Equality and diversity legislation.
Ethical issues and social value judgements.
Cost effectiveness (for the NHS and other public sector organisations).
Balance of harms and benefits.
Ease of implementation and any anticipated changes in practice.

Where possible, recommendations were linked to evidence statements (see appendix C for details). Where a recommendation was inferred from the evidence, this was indicated by the reference 'IDE' (inference derived from the evidence).