Appendix B: Approaches to additional consultation and commissioned primary research
Closed for comments This consultation ended on at Request commenting lead permission
Appendix B: Approaches to additional consultation and commissioned primary research
This appendix outlines the processes and minimum reporting standards for additional consultation and commissioned primary research.
Additional consultation
An additional consultation to the routine stakeholder consultation only happens on an exceptional basis.
Additional consultation is a targeted engagement exercise to get feedback, independent from that provided by the committee. It may involve health and social care professionals, people using services and their family or carers, or both. It can be conducted either by NICE or an externally commissioned organisation.
Deciding if additional consultation is needed
Reasons for additional consultation will vary depending on the topic and may become apparent at different stages of guideline development. Reasons include:
concerns over draft recommendations, including the impact on health inequalities and equalities more broadly
draft recommendations based on committee consensus, with outstanding committee uncertainty
certain population groups not being represented on the guideline committee and routine consultation being unlikely to get a response from these population groups.
The aim of additional consultation may be to obtain feedback on:
Examples of using additional consultation
Examples of when additional consultation has been used include:
NICE's guideline on jaundice in babies under 28 days: Healthcare professionals working in neonatology and midwives were consulted on the consensus bilirubin thresholds for managing hyperbilirubinaemia in babies of 38 weeks or more gestational age. The additional consultation was conducted before routine stakeholder consultation. The aim was to get topic expert views on the consensus bilirubin thresholds before wider public consultation. For more information see appendix P of the addendum to the full guideline on the NICE website.
NICE's guideline on social, emotional and mental wellbeing in primary and secondary education: A series of online focus groups were run with children and young people aged 6 to 17 years from a range of diverse backgrounds across England. This aimed to gain insight into their perceptions of social, emotional, and mental wellbeing provision and processes in primary and secondary education. For more information see the focus group report on the NICE website.
The process
The process is:
decide whether additional consultation is needed
establish the aim and objectives
follow the recruitment process for consultees
establish the consultation method and process
agree a proposal and timeframe
report findings.
It is like the process for routine stakeholder consultation. Additional consultation usually happens before or alongside the routine consultation.
The developer should contact the quality assurance team for an initial discussion as soon as the need for additional consultation is identified. If the work is likely to involve people using services or their family or carers, the developer should discuss their plans with the public involvement lead. If the aim is to test the feasibility of implementing guideline recommendations, it may be useful to involve a member of the field team for advice and support.
Details of the additional consultation must be clearly stated in a proposal document and in the methods document. The proposal needs to be signed off by the quality assurance team.
The information received from the additional consultation and responses to consultees should be published alongside the final guideline. The committee discussion section in the relevant evidence review and the rationale section in the guideline should say how the information affected the recommendations.
The proposal
The proposal should cover:
the aim and objectives of the additional consultation
characteristics and number of additional consultees needed
recruitment methods for consultees, the approach to declaration of interests, and obtaining consent
ethical considerations, including whether formal ethical approval is needed
consultation methods
data analysis or synthesis methods
confidentiality (if additional consultation happens before the routine stakeholder consultation)
the timeframe for additional consultation
how feedback will be used to inform the recommendations
Commissioned primary research
Commissioned primary research usually uses focus groups, interviews or online surveys, and involves health and care professionals, people using services and their family or carers, or both. It may include quantitative analysis of real-world data sources such as audit datasets.
Deciding if commissioned primary research is needed
Reasons for commissioned primary research will vary depending on the topic and may become apparent at different stages of pre-consultation guideline development. Reasons include:
a lack of evidence to answer specific review questions
a call for evidence or expert witnesses is not appropriate or is not expected to produce the evidence needed (see appendix J on call for evidence and expert witnesses)
the committee's consensus is not sufficient for making recommendations
There may be a lack of evidence on:
a specific topic (or that which is applicable to the UK)
the views and experiences of people affected by the guideline
specific populations listed in the guideline scope (for example, children and young people under 18).
The aim could be:
to fill gaps in the evidence, which may include obtaining data (including real-world data) on:
specific interventions
specific populations (there is a lack of research on children and young people in general)
to obtain the views and experiences of people delivering or using services.
Examples of using commissioned primary research
Examples of how guidelines have used commissioned primary research:
NICE's guideline on looked-after children and young people: Primary focus group research was commissioned to explore the views and experiences of looked-after children for a set of review questions covering:
care and placement stability
relationships and contact
health and wellbeing
learning
moving back to birth families or special guardianship
preparing care leavers for independent living.
NICE's guideline on myalgic encephalomyelitis (or encephalopathy)/chronic fatigue (ME/CFS) syndrome: Primary focus group research was commissioned to explore the views and experiences of children and young people with ME/CFS for a set of review questions covering:
identification and assessment before diagnosis
diagnosis of ME/CFS
management of ME/CFS
monitoring and review
information, education, and support for people with suspected or diagnosed ME/CFS and their families and carers
NICE's guideline on BMI: preventing ill health and premature death in black, Asian and other minority ethnic groups: Analysis of a UK dataset (ADDITION-Leicester) was commissioned to supplement the published non-UK literature. The data was analysed to assess the following question: what are the cut-off points for BMI and waist circumference among adults from black, Asian and other minority ethnic groups living in the UK that are 'risk equivalent' to the current thresholds set for white European populations?
The process
The process is:
decide whether commissioned primary research is needed
establish the aim and objectives
follow the commissioning and recruitment processes
establish the research methods and process
consider whether ethical approval is needed
agree a proposal and timeframe (including key milestones that align with guideline development timeframes; see box 1 for an example of key milestones)
report findings).
The research should happen before routine stakeholder consultation and the committee should use the findings as part of the evidence base for developing recommendations.
The developer should contact the quality assurance team for initial discussion as soon as the need for commissioned primary research is identified. If the work is likely to involve people using services or their family or carers, the developer should discuss their plans with the public involvement lead.
Once the aim of the primary research is agreed, the developer should discuss the commissioning process with the quality assurance team.
Details of the commissioned primary research must be clearly stated in a proposal document as well as in the guideline methods document. The proposal should be discussed with the quality assurance team and approved by the centre director.
All outputs (including the final report) should be made available during routine stakeholder consultation. The committee discussion section in the relevant evidence review and the rationale section in the guideline should say how the information affected the recommendations.
Both the proposal and the final report of the additional consultation should be available as appendices on publication of the guideline.
The proposal
The proposal should cover:
the aim and objectives of the commissioned primary research
number and characteristics of participants to be recruited
considerations of consent, confidentiality and data protection
recruitment strategy
ethics considerations, including whether formal ethical approval is needed
sampling method
data analysis methods
feedback mechanism for participants
reporting standards
anticipated timeframe and costs
milestones and expected outputs, for example, the final report may summarise themes from participants' views to inform or fine-tune the final recommendations.
The report
The report of the primary research should follow the same structure as the proposal. It should include sections on the aim and objectives, recruiting participants, methods used, analysis of data and all the findings from the primary research.
The developer should conduct a formal critical appraisal of the final report, using appropriate checklists from the appendix on appraisal checklists, evidence tables, GRADE and economic profiles. For example, the CASP checklist will be suitable for commissioned primary research using qualitative methods such as focus groups or 1-to-1 interviews.
Commissioning an external contractor
Primary research is conducted by an external contractor.
The contractor should:
be an academic or research organisation, or an organisation that works with people affected by the guideline and has research expertise
be separate from the developer and the committee, unless there are exceptional circumstances (for example, when specific expertise or access to specialist networks can only be provided via a committee member)
have a good record of research in the proposed field (for example, qualitative or participatory research), and ideally experience in the topic area, as well as expertise in working with people affected by the guideline.
The developer may be asked to help the contractor, for example by generating a list of research participants.
If an external contractor is commissioned, the commissioning process should follow NICE's Standing Financial Instructions. This involves developing a project specification, issuing invitations to tender, and selecting a contractor based on clear and auditable criteria.
Recruiting participants
The recruitment strategy should take into account the aims and objectives of the commissioned primary research, the topic, the groups, the range of views needed, and other relevant issues.
When planning primary research with children and young people, school holidays and exam schedules should be considered.
Equality issues may require getting a representative spread of practitioners or people using services, but it may also mean focusing on seldom-heard groups or people with recent experience of working with them.
Different sampling methods may be used to recruit participants. Sampling should be guided by the topic and will depend on the:
health and care professional groups delivering the service
the make-up or case-mix of the population affected by the guideline (including factors such as age groups, geographical issues, and access), to ensure appropriate representation of all people affected
scope of the guideline
review questions
inclusion criteria for the research.
'Snowballing' (gathering participants via other participants or networks) and purposive or other non-random techniques may be used to ensure all relevant groups are represented.
Random sampling (randomly selecting participants from the relevant groups) or quota sampling (selecting a fixed number of participants, randomly or purposively from these groups) may be useful for large-scale surveys or if there are many potential participants but not enough of them in each relevant geographical area.
The proposal should explicitly state the groups of participants to be recruited, the recruitment strategy, including sampling method, the number of participants to be recruited, considerations of consent, confidentiality and data protection. The developer or external contractor should ensure the sampling frame and sample take account of equality issues.
Establishing methods and processes
The research method should take into account the topic, the groups involved and other issues.
The NICE real-world evidence framework provides advice on situations where primary quantitative analysis of real-world data may be appropriate and outlines best practices for identifying and assessing data sources and doing the analysis.
When involving people affected by the guideline, the methods and materials used should be tailored to the age, ability and culture of participants.
Commissioned primary research may use group-based methods, 1-to-1 or paired in-depth interviews or surveys. In some cases – for example, if a range of groups are involved – a combination of approaches may be used.
Group-based methods
Group-based methods include focus groups, participative workshops and 'virtual' (electronic) groups. These may be appropriate when:
potential participants have clear 'professional identities' and the 'field' is well established
it is possible to contact enough people in a geographical area to set up a focus group or workshop
the issues discussed are unlikely to be confidential or sensitive and anonymity will not be necessary.
The following may be considered:
More than 1 participative workshop or focus group or virtual meeting; these should take place in more than 1 geographical area and will normally be a half day but may take up to a day. If it is not feasible to organise this many workshops or groups, the decision on how many should be convened must be agreed with NICE.
If it suits the needs of the project, separate participative workshop or focus group or virtual meetings can be arranged for different practitioner or user groups. This will depend on the number of participants and should be agreed with NICE.
For some topic areas, academic researchers may be included in the commissioned primary research. In such cases, a separate meeting should be convened for them, using the same processes. This should be agreed with NICE.
Topic guides, prompts or supporting materials must be developed in collaboration with the guideline committee, and agreed by NICE.
1-to-1 or paired in-depth interviews
Interviews may be carried out face-to-face, by telephone or online. They may be appropriate when:
it is not possible to get groups together because the topic is a relatively new area, the number of possible participants is limited or there are geographical or time constraints
participants may have mobility difficulties or be unable to travel
the issues discussed are likely to be confidential or sensitive and anonymity may be needed
in-depth responses are needed.
Interviews may be structured or semi-structured, depending on the topic and the groups involved. Semi-structured interviews allow complex or difficult issues to be explored and so are likely to be more useful than structured interviews. All interviews should focus on areas in which views and experiences are needed, or on draft recommendations.
Individual or paired interviews are usually more expensive to set up than group work, and the need for in-depth or individual contact should be weighed against the available resources at the planning stage.
Surveys
Group-based methods and interviews are the best way to find out opinions. But they may not be suitable in all circumstances, for example, because of the sensitivity of the topic, confidentiality issues, or difficulties in recruiting participants. In such cases, surveys that use semi-structured and open-ended questions could be more appropriate. Surveys may be carried out by telephone, online, on paper or by using vote casting or polling.
Surveys gather opinions in a quick and less obtrusive manner than group-based approaches and interviews. The responses can also be quantified. But surveys do not allow the same depth of exploration and, generally, should only be used if other methods are unsuitable. Formal consensus methods such as Delphi survey and RAND appropriateness could be modified for the survey if appropriate.
Analysis of data
There are different ways of analysing data from commissioned primary research, depending on the methods used for data collection and whether it is qualitative or quantitative. Some descriptive summary statistics should be provided, for example, characteristics of participants and attendance or response rates.
Quantitative analysis should include a clear statistical data analysis plan, including a priori plans for sub-group analysis or co-variate adjustments.
Group-based methods and interviews are likely to generate qualitative data. Analysis may be performed using qualitative research software, or by hand, but the method should be fully reported in the proposal and the final report.
Qualitative data can be broken down into common and consistent themes for each of the questions asked, using, for example, a content analysis approach. Usually, 1 researcher should prepare an initial analysis, which should be verified by 'blind' coding and sorting of a sample of the transcript by a second researcher. For examples of this kind of analysis, see part 3 (chapters 7 to 13) of Silverman (2004) or Ritchie and Spencer (1993).
Once the analysis is complete, participants' quotes may be selected to illustrate each theme. These quotes should be coded to keep participants anonymous and to allow the quotes to be distinguished. When transcripts are processed, ensure confidentiality and data protection are fully considered. As with data from clinical trials, transcripts should be kept for at least 5 years (see the National Institute for Health and Care Research's clinical toolkit).
Surveys are likely to involve a mixture of quantitative and qualitative data. Quantitative data may be analysed and presented using summary statistics or may involve more advanced approaches such as regression analysis and multi-variable adjustments (which need agreeing a priori). When formal consensus methods such as Delphi survey and RAND appropriateness have been modified for the survey, specific analytical methods, for example, thresholds for agreement, should be stated in the proposal and the final report.
The developer or external contractor should ensure the methods for analysing the data and reporting standards are discussed and agreed with NICE.
Feedback mechanism
The developer or external contractor should agree with NICE a process for giving feedback to all participants. Providing feedback to participants should be specified in contracts. This should include an evaluation exercise, a follow-up session or sharing interim findings via email.
Agreeing the proposal
The proposal for the commissioned primary research should include information on the:
aim and objectives
recruiting participants
methods used
ethics considerations, including whether formal ethical approval is needed
timeframe of the primary research
data analysis
feedback mechanism.
The proposal and the final report of the commissioned primary research should be included as part of the guideline or guideline appendices.
The developer and the external contractor should agree with NICE the approaches and methods to use, including a summary of the issues to be covered. Similarly, the methodology and any questions or support materials used must be developed and agreed with NICE. For example, NICE should:
be briefed by the developer and external contractor in detail before work begins
agree final documents (including the proposal) and comment on draft recruitment letters
agree on topic areas and research questions
agree sampling frames and samples, and other supporting materials
agree approaches on how to get participants from key groups involved, including people who work with or are from seldom heard groups or those who share characteristics protected under equality legislation
have access to transcripts of all data
discuss and agree techniques for data analysis and themes for data presentation
comment on the commissioned primary research report before the final draft is submitted.
Obtaining ethical approval
Most primary research commissioned for NICE guidelines falls under the category of 'service evaluation' and so is outside the remit of NHS research ethics committees. However, the quality assurance team, the developer and the external contractor should consider ethical issues each time commissioned primary research is planned, to ensure appropriate expertise, and that policies and procedures for the safety and welfare of participants are in place. If there is any doubt, the developer or external contractor should consult the national Research Ethics Service. The developer or external contractor is responsible for seeking ethical approval, if it is needed.
For topics covering children and young people, NICE's patient and public involvement policy includes a set of principles for involving them and has an appendix about safeguarding. The national Research Ethics Service should also be consulted for topics covering children and young people and other vulnerable groups such as adults with learning disabilities or frail older people.
Milestones | Timeline/Date |
|---|---|
Start-up meeting with contractor Recruitment of research participants Data collection period Contractor submits draft report to developer, who shares with committee Developer and committee provide comments on draft report, including a formal quality assessment of report by developer. Contractor submits revised report to developer, who shares with committee Contractor submits slides for presentation to committee (if necessary), who shares with committee Presentation of report at committee meeting Contractor submits final report after committee meeting |
How are you taking part in this consultation?
You will not be able to change how you comment later.
You must be signed in to answer questions