NICE process and methods
3 Determining the evidence for review and consideration
3 Determining the evidence for review and consideration
3.1 Introduction
NICE public health guidance is informed by a variety of types of evidence (Lomas et al. 2005) (see section 1.5). This chapter describes some of the types of evidence reviews that the Centre for Public Health Excellence (CPHE) may commission or undertake to provide good quality, scientific evidence to answer different types of research questions. It also outlines how to develop specific research questions for these reviews. It concludes by describing how other types of evidence may be used to help develop the guidance.
3.2 Types of evidence
The public health advisory committees (PHACs) need both scientific and other types of evidence about what works generally, why it works, and what might work (and how) in specific circumstances. The committees need evidence from multiple sources, extracted for different purposes and through different methods. In making its recommendations, the committees need to consider all of this evidence within an ethical and theoretical framework (Tannahill 2008).
3.2.1 Scientific evidence
Scientific evidence is explicit (codified and propositional), systemic (uses transparent and explicit methods for codifying), and replicable (using the same methods with the same samples will lead to the same results). It can be context-free or context-sensitive (Lomas et al. 2005):
-
Context-free scientific evidence explains universal truths about what might generally work, in ideal circumstances. It is knowledge produced from a scientific framework that assumes that objects of the natural world are real, objective, measurable and exist independent of the observer and context (Woolgar 1988). It can be derived from systematic reviews or meta-analyses of quantitative research of intervention efficacy or effectiveness, primary empirical research or theoretical models or propositions.
-
Context-sensitive scientific evidence is concerned with what works and how, why or whether it might work in specific real-life circumstances. It includes information on attitudes, implementation, organisational capacity, forecasting, economics and ethics. It is mainly derived from social scientific and behavioural research methods including: quantitative and qualitative research studies, surveys, theories, cost-effectiveness analyses and mapping reviews. Sometimes, it is even derived from systematic reviews and the other research techniques described above. Context-sensitive evidence can be used to complement context-free evidence, providing the basis for more specific and practical recommendations (see figure 3.1). It may be used to:
-
supplement evidence on efficacy or effectiveness (for example, to look at differential effectiveness according to occupation, educational attainment and income)
-
construct logic models (see section 2.2.1) and causal pathways (for example, to explain what factors predict teenage parenthood)
-
provide information about the characteristics of the population (including its social and physical circumstances) and about the process of implementation
-
describe psychological processes and behaviour change.
-
Figure 3.1 Scientific evidence: context-sensitive complements context-free
|
(Source: Lomas et al. 2005 adapted from Davies 2005)
3.2.2 Colloquial evidence
'Colloquial evidence' can complement the scientific evidence or provide missing information on context (see figure 3.2). It may come from expert testimony (see section 3.5.1), from members of the advisory committees, or from stakeholder comments (see section 3.5.2). It includes evidence about values (including political judgement), practical considerations (resources, professional experience or expertise and habits or traditions) and the interests of specific groups (views of lobbyists and pressure groups).
Figure 3.2 Colloquial evidence complements scientific evidence
|
(Source: Lomas et al. 2005 adapted from Davies 2005)
3.3 Types of review and types of research question
NICE public health guidance is routinely informed by high-quality reviews of the scientific evidence. These reviews explicitly address research questions based on the key questions in the scope. Rather than relying on the standard hierarchy of evidence (with randomised controlled trials (RCTs) at the top), a wide range of study designs and methodologies should be used to answer these questions (Petticrew and Roberts 2002) (see section 1.5.1 for details). This is because:
-
Public health covers a broad range of (often multiple) interactions between the different effects of an intervention (even when the intervention is relatively simple). Public health methods have to reflect this complexity.
-
In the traditional evidence hierarchy, RCTs are usually accorded the highest status. However, it is difficult to run RCTs to test many public health issues since they tend to produce an over-inflated effect estimate (because they are conducted in ideal circumstances). They are also usually limited to answering questions on the efficacy – and sometimes the effectiveness – of interventions.
-
Table 3.1 shows the type of evidence that should be used to address different types of research question and the type of standard CPHE review it will involve. It builds on a general typology of public health evidence for social interventions with children (Petticrew and Roberts 2002, adapted from Muir Gray 1996).
3.3.1 Types of review research question
The key questions in the scope are based on the Department of Health's (DH) original referral, the topic area and the views of practitioners, decision makers and other stakeholders. All public health guidance aims to recommend the most effective ways to promote health, prevent disease and reduce health inequalities. However, the scope may include several other questions and potential considerations that reflect the nature of the specific issue being tackled and its context (see sections 2.4.1 and 2.4.2).
The CPHE project team and the review team should use the key questions and considerations from the final scope, along with its other parameters (populations, activities and outcomes) to develop more specific research questions for the reviews. In addition to questions of effectiveness and cost effectiveness, there will often be questions about the epidemiology of a problem or issue, the acceptability and accessibility of interventions, and the experiences of service users or practitioners.
The type of research questions will determine the number and type of reviews and the type of evidence they include (for example, intervention studies and qualitative data). Whatever method is used, the process for developing questions is the same.
The CPHE project team and the review team should draft clear, focused review questions. The exact structure of each question will depend on what is being asked, but it is likely to cover 1 of the following:
-
extent and nature of the public health problem
-
factors, causal mechanisms and the role of the various vectors (see appendix A)
-
interventions that work in ideal circumstances and might work in specific circumstances (the extent to which something works, how and why)
-
a relevant programme theory or theory of change
-
views and experiences of the target population (people who may be affected by the recommendation), including how acceptable and accessible they find the intervention
-
practitioners' views, experiences and working methods (including any barriers to and factors supporting implementation of the intervention)
-
cost effectiveness
-
potential for an intervention to do harm.
Table 3.1 Types of evidence and standard[1] CPHE reviews to address different research questions
|
The different types of review that may be carried out to answer these questions are set out below – along with how to develop appropriate research questions in each case.
Public health guidance will usually be informed by at least 1 effectiveness review. The decision on whether or not to use other, additional types of review will depend on the topic area and the type, depth and breadth of relevant evidence available. Sometimes, a review may draw on a combination of different sources of evidence or types of data (for example, combining mapping information and qualitative data). The nature of the area, the technical expertise and topic knowledge of the CPHE, and feedback from stakeholders at the scoping stage may all influence this decision.
The remainder of this chapter gives a general description of when different types of review may be considered and the process of developing appropriate research questions for them. It does not rule out the possibility of using 'hybrid' reviews (for example, combining mapping information and qualitative data), but simply illustrates a range of potential approaches.
3.3.2 CPHE effectiveness reviews
The CPHE project team and the review team should explicitly consider what type of study will provide the best evidence to answer a question on effectiveness. Traditional systematic reviews of effectiveness answer specific questions about the efficacy or effectiveness of a particular intervention. Usually, they draw on RCTs as this type of study is most likely to give an unbiased estimate of the effects. However, for many public health topics it can be difficult or unethical to assign populations to control and intervention groups (for example, for interventions aiming to change policy). In such cases, a non-randomised controlled trial (see appendix D) might be a more appropriate way of establishing cause and effect. The Medical Research Council (MRC) has produced guidance on evaluating complex interventions (Craig et al. 2008) and using natural experiments to evaluate population health interventions (Craig et al. 2011).
3.3.2.1 Review research questions
At least 1 review question, possibly more (depending on the populations and outcomes of interest) will be needed for each intervention considered. The CPHE project team must ensure these questions are clear and focused.
Because the advisory committees need to take into account factors that could affect the outcomes and effectiveness of an intervention, it may be helpful to list outcomes and other key criteria. Effectiveness reviews could also consider context and implementation issues, as far as the available evidence permits – although these may also be dealt with in a separate evidence review.
The population, intervention, comparison and outcome (PICO) framework is a helpful, structured approach to formatting intervention questions and should certainly be used as a starting point. Other approaches or questions may also be used, where helpful.
Box 3.1 PICO guide to review questions on intervention effectiveness
|
Population: Which populations are we interested in? How can they best be described? Do any subgroups need to be considered? Intervention: Which intervention, activity or approach should be used? Comparison: What are the main alternatives to the intervention being considered? Outcome: What outcomes should be considered? (Examples include increases in knowledge, skills or availability of services; changes in behaviour; mortality rates; morbidity rates; changes to quality of life; and adverse effects.) |
Once the review questions have been framed, the review team should identify key words as potential search terms.
Examples of research questions
-
What types of mass-media intervention help prevent children and young people from taking up smoking? Are the interventions delaying rather than preventing the onset of smoking?
-
Which of the harm-reduction services offered by needle and syringe programmes (including advice and information on safer injecting, onsite vaccination services, and testing for hepatitis B and C and HIV) are effective in reducing blood-borne viruses and other infections among people who inject drugs?
-
What types of intervention and programme are effective in increasing physical activity levels among children under 8 – particularly those who are not active enough to meet the national recommendations for their age – or help to improve their core physical skills?
-
Does brief advice from GPs increase adult patients' physical activity levels?
-
What are the most effective school-based interventions for changing young people's attitudes to alcohol use?
3.3.3 CPHE epidemiological reviews
CPHE epidemiological reviews describe the public health problem.
The topic referral and scope development stages may draw on observational studies to gauge the nature of the public health problem (see chapter 2). However most topics will benefit from further epidemiological reviews to determine the focus of the guidance. For example, such a review of accidents would provide information on the most common ones, as well as morbidity and mortality statistics. This would help focus the research questions. The resulting recommendations would be based on a logical appraisal of the problem and its causes and hypotheses about the kinds of interventions that could address them.
The CPHE project team will commission an epidemiological review if:
-
the exact nature of the public health problem is unclear
-
those who could benefit from the guidance cannot be clearly defined
-
the cause of the public health problem is unclear.
Examples of research questions
-
What are the patterns of physical activity among children from different populations and of different ages in England?
-
Which populations of children are least physically active and at which developmental stage are all children least physically active?
-
What effect does physical activity have on children's health and other outcomes in the short and long term?
3.3.4 CPHE correlates reviews
CPHE correlates reviews describe relationships between epidemiological factors and outcomes.
If an epidemiological review has been carried out, information will have been gathered from observational studies on the nature of the public health problem. However, further analysis of this information – in the form of a correlates review – may be needed to establish the epidemiological factors associated with any positive or negative behaviours or outcomes.
Examples of research questions
-
What factors are associated with children or adolescents' physical activity and how strong are those associations?
-
What are the barriers to and facilitators for participation in physical activity?
-
How do the barriers and facilitators differ for the least active subpopulations and age groups?
3.3.5 CPHE reviews of qualitative evidence
Where there is a mature body of relevant evidence, or where the evidence is important enough to merit a substantive piece of work, the CPHE project team may commission a review of qualitative research.
Qualitative studies may be the primary source of evidence to answer questions on:
-
client or practitioner experiences (including information on what works, for whom and under what circumstances)
-
patient and practitioner views
-
opportunities for and barriers to improvement (including issues of access or acceptability)
-
variations in delivery and implementation for different groups, populations or settings
-
barriers to and facilitators that aid implementation
-
social context and the social construction and representation of health and illness
-
background on context, from the point of view of an observer (and not necessarily that of a user or practitioner)
-
an explanation of the associations between interventions and outcomes.
Qualitative information can also be used to enhance logic models (see chapter 2).
Of course, some evidence from quantitative research may be able to answer these types of questions: this can be picked up by the effectiveness (or other) reviews. Alternatively, the review team may choose to combine quantitative evidence with qualitative evidence in a single review (such as a review of barriers and facilitators), when appropriate.
Examples of research questions
The CPHE project team, in conjunction with the review team, should fine-tune questions posed by the scope to produce research questions for each qualitative review. These questions should be relevant, specific (about factors such as the populations or settings involved), actionable and realistic in relation to the area covered by the review.
The PICO framework (see box 3.1) may help ensure the views and experiences of the target population are integrated into questions about intervention effectiveness. Specific aspects of these views and experiences, the social context or the social construction of an issue or area can be tackled in separate review questions. (However, if the review does not involve an intervention, the PICO framework may not be appropriate). Questions that are too broad and lack focus should be avoided (for example, 'What is the experience of looked after children?'). Examples of good questions include:
-
How do different groups of practitioners, clients or stakeholders perceive the issue (for example, broken down by profession, age, sex or ethnicity)?
-
What social and cultural beliefs, attitudes or practices might affect this issue?
-
How do different groups perceive the intervention or available options? What are their preferences?
-
What approaches are used in practice? How effective are they, according to different groups of practitioners, clients or stakeholders?
-
What is a desired, appropriate or acceptable outcome as far as the client groups are concerned? What outcomes are important to them? What do professional, client or stakeholder groups perceive to be the barriers to, and opportunities for, change in this area?
-
What status or importance do professional or public groups attach to NICE guidance?
-
What do the target populations think about current or proposed practice?
-
Why do people make the choices they do or behave in the way that they do?
-
How is a public health issue represented in the media and popular culture?
-
How does an issue such as access or acceptability affect the delivery and effectiveness of an approach?
Once the research questions have been agreed between the review team and the CPHE project team, the review team should develop and implement a search and appraisal process (see chapters 4 and 5). As with other types of review, the review team should agree a set of inclusion and exclusion criteria with the CPHE project team. This should be consistent with the scope and research questions.
Examples of research questions for qualitative evidence that supplements a CPHE effectiveness review
The process for developing research questions for an effectiveness review that will include qualitative evidence is similar to that described in section 3.3.2. A qualitative component should be included if there are issues that could be better assessed using qualitative evidence. For example, to assess why an intervention does or does not work.
Research questions should be developed by the CPHE project team and the review team, using the scope and the main review questions. Again, care should be taken to ensure the questions are relevant, specific, realistic and actionable. They might include:
-
How is the intervention perceived by different client or practitioner groups?
-
How acceptable is the intervention?
-
How accessible is the intervention or service to different client groups? What factors affect its accessibility?
-
Does the mode or structure of delivery (including characteristics of relevant health or other professionals, the setting and language) affect user perceptions?
3.3.6 CPHE cost-effectiveness reviews
Cost-effectiveness reviews aim to identify, appraise and present data from economic studies (see chapter 6). Cost-effectiveness data may be considered as part of each effectiveness review undertaken for a piece of guidance, or in just 1 or 2 key areas (depending on the topic and the available data). Collation of this data is vital for any subsequent modelling. It may be integrated in the effectiveness reviews or presented as a separate report.
3.3.7 CPHE mapping reports
Where evidence from published formal research is limited, a mapping report could provide a snapshot of current practice. It might address, for example, current professional practice or multidisciplinary theoretical accounts. It could draw on published or grey literature[2] – or both.
A mapping report can help with the development of other reviews, supplement the knowledge gained from other reviews, help inform recommendations and help guide the implementation process. It does not provide a quality assessment of the material. There are 2 types:
-
A map of the literature describes the type (research method, design or theoretical basis or approach) and quantity of published material. Selected databases of important literature are sampled and a typology developed. These reports are useful where there is a lot of material or when several keywords are used for indexing complex topics. They can help to identify or clarify areas where a review of effectiveness might be useful and to determine the scope and breadth of the guidance.
-
A map of current policy and practice is useful when the available evidence is largely non-UK based. It can help deal with applicability issues and identify how to overcome barriers to effective implementation. This type of mapping study can encompass published information such as practice surveys (based on context-sensitive scientific evidence). Alternatively, it can draw on practitioner views and reports (colloquial evidence) to highlight the conditions needed for interventions to work in practice (detail that is often absent from published material). Mapping policy and practice can help to ensure the context in which the recommendations are made is up to date. It is important because some of the most innovative action is never written up, due to a lack of expertise or resources.
3.3.8 CPHE reviews of reviews
Generally, review-level material (for example, systematic reviews, literature reviews and meta-analyses) will only be used as an additional source of potentially relevant primary studies. In such cases, the review team should screen reference lists of reviews to identify additional primary studies that might not have been captured in the electronic or other searches. Generally, review-level material will not be quality-assessed or data-extracted for inclusion in the evidence reviews because it:
-
rarely covers inclusion and exclusion criteria relevant to the guidance topic's referral and parameters (for example, comparable research questions, relevant outcomes, settings, population groups or time periods)
-
often aggregates different outcome or study types, some of which may be part of the topic's exclusion criteria and would need to be disaggregated
-
often covers both clinical and public health-related data that is either difficult or impossible to disaggregate
-
rarely provides enough data to develop recommendations: obtaining detail on the implementation of specific interventions is particularly problematic and, as a result, it is often necessary to refer to the primary studies.
Review-level material can be useful when developing the scope and when defining research questions, outcomes and outcome measures for the evidence reviews. The discussion section can help identify some of the limitations or difficulties associated with a topic area. It can also provide a critical account of the state of the evidence base for a specific topic (see chapter 2).
Review-level material should only be quality-assessed, data-extracted and integrated with primary studies in the evidence reviews in the following, exceptional circumstances:
-
If the parameters (for example, research question, inclusion and exclusion criteria) of a recent, good quality review (see screening checklist in appendix J) are sufficiently similar to the guidance topic's research parameters to be able to answer 1 or more specific research question. In such cases, a search should be undertaken for primary studies published after the search date covered by the identified review.
-
If a series of systematic reviews or meta-analyses for a specific well-defined topic have led to a definitive decision or consensus on a specific research question and investment in further research has stopped.
-
If a topic referral draws heavily on published theories (for example, on how to support attitude and behaviour change).
-
If the evidence base for the specific topic is so large that resource limitations make it impossible to cover all available primary studies.
The review team should discuss and agree with the CPHE project team when 'exceptional circumstances' dictate that review-level evidence should be included in a review. See appendix J for further details.
3.4 Planning reviews
The review team will prepare a protocol that outlines the background, objectives and methods for each review. It should be agreed with and signed off by the CPHE project team. The protocol provides an explicit public record of how the review is to be carried out, making it possible for it to be repeated by others at a later date. The protocol should also make it clear how inequality issues will be considered, where appropriate.
3.4.1 Structure of review protocol
The review protocol should describe any deviations from the methods described in this manual and should include the components outlined in table 3.2. It should not duplicate the methodology stated in this manual.
Table 3.2 Components of the review protocol
|
Component |
Description |
|
Guidance title |
Full and short title of guidance. |
|
Review team |
Provide names, affiliations and number of whole-time equivalent members of the review team. Where lead or key staff are working on other NICE reviews, please state. |
|
Summary of the scope |
|
|
Groups that will be covered |
Based on the scope, additional detail as required. |
|
Groups that will not be covered |
As above. |
|
Activities or interventions that will be covered |
As above. |
|
Activities or interventions that will not be covered |
As above. |
|
Other aspects of the scope |
Include other relevant information. Outline schematically, where relevant, how the scope relates to NICE's public health conceptual framework (see chapter 2 and appendix A). |
|
Review(s) |
|
|
Overview |
Provide a brief overview of the review(s) aims and objectives. If a logic model or conceptual framework is to be used, provide a brief outline including how it will inform the search process. |
|
Review questions |
Make these clear, focused and realistic, adopting the PICO approach (if relevant). Include any sub-questions. |
|
Review outcomes |
Specify all the outcomes of interest relevant to the review populations, subpopulations, settings etc. |
|
Methods |
Include details of:
Include in the search protocol:
If appropriate these details can be listed by review. Separate search protocol(s) to be developed as part of the first phase of the review should be appended to that review protocol. |
|
Economic analysis modelling (if relevant) |
Describe all components. |
|
Timetable |
Provide a table showing agreed dates for:
Where the focus of some reviews is still to be determined, insert the date(s) for agreeing the protocol of each one; once agreed, append to this document. |
|
Deliverables |
Describe what products will be delivered or presented to each meeting or deadline for reports. |
|
Additional information |
Any other relevant information. |
|
Date and version of protocol |
|
|
Appendix |
Include a 'logic model' to show how the review(s) helps to address the scope. |
3.4.2 Process for developing the review protocol
As a first step, the CPHE project team and the review team should develop detailed research questions for the review, using the key questions and considerations from the final scope, along with the other parameters of the guidance (populations, activities and outcomes) – see section 3.3.1.
The types of question will dictate which methodology will be used and the types of evidence that will be considered. All review protocols should be included as appendices in the review. Any changes made to a protocol in the course of guidance development should be described in the appendix.
3.5 Types of colloquial evidence
The different types of review outlined in this chapter mainly focus on gathering and assessing scientific evidence (mapping reports are an exception – see section 3.3.7). However, 'colloquial evidence' – about values, practice, political judgement, operational considerations and interests – is also key to developing public health guidance. It takes the following forms.
3.5.1 Expert testimony
An expert witness may be invited to give testimony when:
-
reviews have uncovered significant gaps in the evidence (or the CPHE project team is aware from the outset that the formal evidence is likely to be limited)
-
the available evidence conflicts significantly
-
the PHAC wishes to seek the views and experiences of specific groups of researchers, practitioners, clients or service users.
Expert testimony can be used to provide a range of information about public health interventions and programmes, including:
-
context – for example, the policy or commissioning context
-
effectiveness – for example, preliminary results from ongoing interventions or services
-
service design and delivery – for example, detailed information on how a particular service is implemented with different groups of people
-
experience – for example, views and experiences of groups of clients or practitioners.
'Experts' can be drawn from both professional and lay communities. They are selected and invited for their knowledge about or involvement in an intervention, programme, service or policy that is of particular interest to the committee in relation to the guidance. Figure 3.3 below sets out a simple protocol for expert testimony in public health guidance development.
Figure 3.3 Colloquial evidence complements scientific evidence
|
The CPHE team develop research questions from gaps in the evidence that have been identified by the PHAC as they develop the guidance. Experts are then identified to help fill those gaps.
Experts may be identified via stakeholders, via PHAC members, or in the course of carrying out the reviews (for example, key authors or researchers). The patient and public involvement programme (PPIP) will help to identify community experts. All expert witnesses are agreed with the Chair in advance of the relevant PHAC meeting. Once key experts have been identified, they are contacted by the CPHE team and invited to speak to the PHAC about a specific issue or question. The invitation is sent with an expert testimony summary template (see appendix N), which the topic expert completes.
Expert testimony takes the form of a short, focused presentation to the committee, followed by discussion. Once they have accepted the invitation to present, expert witnesses are asked to complete part B of the expert testimony summary form (see appendix N), which requires them to provide a summary of their testimony, including references to any relevant published work. This is treated as evidence and subject to consultation, along with any reviews.
3.5.2 Consultation with stakeholders
Stakeholder views are routinely sought during the development of public health guidance: at the scoping stage and during consultation on the draft guidance. See sections 4.3 and 4.5 of The NICE public health guidance development process (third edition, 2012).
The CPHE project team always considers and responds to comments made during these consultation exercises and the PHAC takes them into account when developing the final guidance. All stakeholder comments – and CPHE project team responses – are available on the NICE website.
3.5.3 Fieldwork
On occasion, the CPHE project team carries out fieldwork with key practitioner groups on the draft recommendations (see appendix M). If fieldwork has been conducted the findings are taken into account by the PHACs when developing the final guidance. The findings are also published as a report on the NICE website.
3.5.4 Committee discussion and expertise
Each PHAC includes professional and community members with both general and specialist expertise in public health. Each PHAC includes a Chair, and a core of members selected for their technical expertise (for example, a health economist) and a community member. In addition to the core membership, topic expert members join a PHAC for each new guidance topic. Topic expert membership of each PHAC includes a community member and professionals, practitioners and technical experts drawn from the NHS, education, social care, environmental health, local government or the voluntary sector. If any additional skills or expertise are required on the PHAC, individuals may be co-opted to 1 or more meetings.
For further details on the composition of PHACs and the recruitment process, see sections 5.2, 5.3 and 7.4 of The NICE public health guidance development process (third edition, 2012). 3.5.5 Mapping reports of local practice
Mapping reports can help to clarify local practice, research or theory. For more information, see section 3.3.7.
3.5.6 Primary research and commissioned reports
Sometimes when the guidance is aimed at a particular population group where there is little evidence, the CPHE may commission consultation with the populations intended to benefit from the recommendations, to gain their views. This may be needed:
-
where there are important gaps in the evidence on the views and experiences of the target population
-
where there are gaps in membership of the PHAC in terms of client/service user perspectives, for example, for guidance covering children and young people or people with learning disabilities.
3.6 Equality and diversity
Specific issues in relation to groups identified in the Equality Act (2010) (or groups who are particularly disadvantaged with respect to the topic under consideration) should be addressed. These issues should be identified during the topic selection and scope development phases. They should also be considered during development of the review questions.
3.7 References and further reading
Craig P, Dieppe P, McIntyre S et al. on behalf of the MRC (2008) Developing and evaluating complex interventions: new guidance. London: Medical Research Council
Craig P, Cooper C, Gunnell D et al. on behalf of the MRC (2011) Using natural experiments to evaluate population health interventions: guidance for producers and users of evidence. London: Medical Research Council
Harden A, Garcia J, Oliver S et al. (2004) Applying systematic review methods to studies of people's views: an example from public health research. Journal of Epidemiology and Community Health 58: 794–800
Kelly MP, Swann C, Morgan A et al. (2002) Methodological problems in constructing the evidence base in public health. London: Health Development Agency
Kelly MP, Moore TA (2012) The judgement process in evidence-based medicine and health technology assessment. Social Theory and Health 10:1-19. doi:10.1057/sth.2011.21
Lomas J, Culyer T, McCutcheon C et al. (2005) Conceptualizing and combining evidence for health system guidance: final report. Ottawa: Canadian Health Services Research Foundation
Muir Gray JM. (1996) Evidence based healthcare. London: Churchill Livingstone
Ogilvie D, Hamilton V, Egan M et al. (2005) Systematic reviews of health effects of social interventions: 1. Finding the evidence: how far should you go? Journal of Epidemiology and Community Health 59: 804–8
Ogilvie D, Egan M, Hamilton V et al. (2005) Systematic reviews of health effects of social interventions: 2. Best available evidence: how low should you go? Journal of Epidemiology and Community Health 59: 886–92
Petticrew M (2003) Why certain systematic reviews reach uncertain conclusions. British Medical Journal 326: 756–8
Petticrew M, Roberts H (2002) Evidence, hierarchies, and typologies: horses for courses. Journal of Epidemiology and Community Health 57: 527–9
Popay J, Rogers A, Williams G (1998) Rationale and standards for the systematic review of qualitative literature in health services research. Qualitative Health Research 8 (3): 341–51
Popay J, editor (2006) Moving beyond effectiveness in evidence synthesis: methodological issues in the synthesis of diverse sources of evidence. London: National Institute for Health and Clinical Excellence
Rychetnik L, Frommer M, Hawe P et al. (2002) Criteria for evaluating evidence on public health interventions. Journal of Epidemiology and Community Health 56: 119
Tannahill A (2008) Beyond evidence – to ethics: a decision making framework for health promotion, public health and health improvement. Health Promotion International 23 (4): 380–90
Victora C, Habicht J, Bryce J (2004) Evidence-based public health: moving beyond randomized trials. American Journal of Public Health 94 (3): 400–5
Woolgar S (1988) Science: the very idea. London: Routledge