Tools and resources

Developing NICE guidelines: the manual – appendices A to P

Appendix B: Approaches to additional consultation and commissioned primary research

Appendix B: Approaches to additional consultation and commissioned primary research

This appendix outlines the processes and minimum reporting standards for additional consultation and commissioned primary research.

Additional consultation

An additional consultation to the routine stakeholder consultation only happens on an exceptional basis.

Additional consultation is a targeted engagement exercise to get feedback, independent from that provided by the committee. It may involve health and social care professionals, people using services and their family or carers, or both. It can be conducted either by NICE or by an externally commissioned organisation.

Deciding if additional consultation is needed

Reasons for additional consultation will vary depending on the topic and may become apparent at different stages of guideline development. Reasons include:

  • concerns over draft recommendations, including the impact on health inequalities and equalities more broadly

  • draft recommendations based on committee consensus and where there is committee uncertainty

  • certain population groups not being represented on the guideline committee and routine consultation being unlikely to get a response from these population groups.

The aim of additional consultation may be to obtain feedback on:

  • the relevance and acceptability of the draft recommendations from people affected by the guideline

  • the feasibility of implementing the draft recommendations

  • whether the draft recommendations reflect current practice, or the way services are currently organised.

Examples of using additional consultation

Examples of when additional consultation has been used include:

The process

The process is:

  • decide whether additional consultation is needed

  • establish the aim and objectives

  • follow the recruitment process for consultees

  • establish the consultation method and process

  • agree a proposal and timeframe

  • report findings.

It is like the process for routine stakeholder consultation. Additional consultation usually happens before or alongside the routine consultation.

The developer should contact the quality assurance team for an initial discussion as soon as the need for additional consultation is identified. If the work is likely to involve people using services or their family or carers, the developer should discuss their plans with the public involvement lead. If the aim is to test the feasibility of implementing guideline recommendations, it may be useful to involve a member of the field team for advice and support.

Details of the additional consultation must be clearly stated in a proposal document and in the methods document. The proposal needs to be signed off by the quality assurance team.

The information received from the additional consultation and responses to consultees should be published alongside the final guideline. The committee discussion section in the relevant evidence review and the rationale section in the guideline should say how the information affected the recommendations.

The proposal

The proposal should cover:

  • the aim and objectives of the additional consultation

  • characteristics and number of additional consultees needed

  • recruitment methods for consultees, the approach to declaration of interests, and obtaining consent

  • ethical considerations, including whether formal ethical approval is needed

  • consultation methods

  • data analysis or synthesis methods

  • confidentiality (if additional consultation happens before the routine stakeholder consultation)

  • the timeframe for additional consultation

  • how feedback will be used to inform the recommendations

  • how to respond to those who provide feedback.

Commissioned primary research

Commissioned primary research usually uses focus groups, interviews or online surveys, and involves health and care professionals, people using services and their family or carers, or both. It may include quantitative analysis of real-world data sources such as audit datasets.

Deciding if commissioned primary research is needed

Reasons for commissioned primary research will vary depending on the topic and may become apparent at different stages of pre-consultation guideline development. Reasons include:

  • a lack of evidence to answer specific review questions

  • a call for evidence or expert witnesses is not appropriate or is not expected to produce the evidence needed (see the appendix on call for evidence and expert witnesses)

  • the committee's consensus is not sufficient for making recommendations

There may be a lack of evidence on:

  • a specific topic (or that which is applicable to the UK)

  • the views and experiences of people affected by the guideline

  • specific populations listed in the guideline scope (for example, children and young people under 18).

The aim could be:

  • to fill gaps in the evidence, which may include obtaining data (including real-world data) on:

    • specific interventions

    • specific populations (there is a lack of research on children and young people in general)

  • to obtain the views and experiences of people delivering or using services.

Examples of using commissioned primary research

Examples of how guidelines have used commissioned primary research:

  • NICE's guideline on looked-after children and young people: Primary focus group research was commissioned to explore the views and experiences of looked-after children for a set of review questions covering:

    • care and placement stability

    • relationships and contact

    • health and wellbeing

    • learning

    • moving back to birth families or special guardianship

    • preparing care leavers for independent living.

  • NICE's guideline on myalgic encephalomyelitis (or encephalopathy)/chronic fatigue syndrome (ME/CFS): Primary focus group research was commissioned to explore the views and experiences of children and young people with ME/CFS for a set of review questions covering:

    • identification and assessment before diagnosis

    • diagnosis of ME/CFS

    • management of ME/CFS

    • monitoring and review

    • information, education, and support for people with suspected or diagnosed ME/CFS and their families and carers.

The process

The process is:

  • decide whether commissioned primary research is needed

  • establish the aim and objectives

  • follow the commissioning and recruitment processes

  • establish the research methods and process

  • consider whether ethical approval is needed

  • agree a proposal and timeframe (including key milestones that align with guideline development timeframes; see box 1 for an example of key milestones)

  • report findings.

The research should happen before routine stakeholder consultation and the committee should use the findings as part of the evidence base for developing recommendations.

The developer should contact the quality assurance team for initial discussion as soon as the need for commissioned primary research is identified. If the work is likely to involve people using services or their family or carers, the developer should discuss their plans with the public involvement lead.

Once the aim of the primary research is agreed, the developer should discuss the commissioning process with the quality assurance team.

Details of the commissioned primary research must be clearly stated in a proposal document as well as in the guideline methods document. The proposal should be discussed with the quality assurance team and approved by the centre director.

All outputs (including the final report) should be made available during routine stakeholder consultation. The committee discussion section in the relevant evidence review and the rationale section in the guideline should say how the information affected the recommendations.

Both the proposal and the final report of the additional consultation should be available as appendices on publication of the guideline.

The proposal

The proposal should cover:

  • the aim and objectives of the commissioned primary research

  • number and characteristics of participants to be recruited

  • considerations of consent, confidentiality and data protection

  • recruitment strategy

  • ethics considerations, including whether formal ethical approval is needed

  • sampling method

  • data analysis methods

  • feedback mechanism for participants

  • reporting standards

  • anticipated timeframe and costs

  • milestones and expected outputs, for example, the final report may summarise themes from participants' views to inform or fine-tune the final recommendations.

The report

The report of the primary research should follow the same structure as the proposal. It should include sections on the aim and objectives, recruiting participants, methods used, analysis of data and all the findings from the primary research.

The developer should conduct a formal critical appraisal of the final report, using appropriate checklists from the appendix on appraisal checklists, evidence tables, GRADE and economic profiles. For example, the CASP checklist will be suitable for commissioned primary research using qualitative methods such as focus groups or 1-to-1 interviews.

Commissioning an external contractor

Primary research is conducted by an external contractor.

The contractor should:

  • be an academic or research organisation, or an organisation that works with people affected by the guideline and has research expertise

  • be separate from the developer and the committee, unless there are exceptional circumstances (for example, when specific expertise or access to specialist networks can only be provided via a committee member)

  • have a good record of research in the proposed field (for example, qualitative or participatory research), and ideally experience in the topic area, as well as expertise in working with people affected by the guideline.

The developer may be asked to help the contractor, for example by generating a list of research participants.

If an external contractor is commissioned, the commissioning process should follow NICE's Standing Financial Instructions. This involves developing a project specification, issuing invitations to tender, and selecting a contractor based on clear and auditable criteria.

Recruiting participants

The recruitment strategy should take into account the aims and objectives of the commissioned primary research, the topic, the groups, the range of views needed, and other relevant issues.

When planning primary research with children and young people, school holidays and exam schedules should be considered.

Equality issues may require getting a representative spread of practitioners or people using services, but may also mean focusing on seldom-heard groups or people with recent experience of working with them.

Different sampling methods may be used to recruit participants. Sampling should be guided by the topic and will depend on the:

  • health and care professional groups delivering the service

  • make-up or case-mix of the population affected by the guideline (including factors such as age groups, geographical issues, and access), to ensure appropriate representation of all people affected

  • scope of the guideline

  • review questions

  • inclusion criteria for the research.

'Snowballing' (gathering participants via other participants or networks) and purposive or other non-random techniques may be used to ensure all relevant groups are represented.

Random sampling (randomly selecting participants from the relevant groups) or quota sampling (selecting a fixed number of participants, randomly or purposively from these groups) may be useful for large-scale surveys or if there are many potential participants but not enough of them in each relevant geographical area.

The proposal should explicitly state the groups of participants to be recruited, the recruitment strategy, including sampling method, the number of participants to be recruited, considerations of consent, confidentiality and data protection. The developer or external contractor should ensure the sampling frame and sample take account of equality issues.

Establishing methods and processes

The research method should take into account the topic, the groups involved and other issues.

The NICE real-world evidence framework provides advice on situations where primary quantitative analysis of real-world data may be appropriate and outlines best practices for identifying and assessing data sources and doing the analysis.

When involving people affected by the guideline, the methods and materials used should be tailored to the age, ability and culture of participants.

Commissioned primary research may use group-based methods, 1-to-1 or paired in-depth interviews or surveys. In some cases – for example, if a range of groups are involved – a combination of approaches may be used.

Group-based methods

Group-based methods include focus groups, participative workshops and 'virtual' (electronic) groups. These may be appropriate when:

  • potential participants have clear 'professional identities' and the 'field' is well established

  • it is possible to contact enough people in a geographical area to set up a focus group or workshop

  • the issues discussed are unlikely to be confidential or sensitive and anonymity will not be necessary.

The following may be considered:

  • More than 1 participative workshop or focus group or virtual meeting; these should take place in more than 1 geographical area and will normally be a half day but may take up to a day. If it is not feasible to organise this many workshops or groups, the decision on how many should be convened must be agreed with NICE.

  • If it suits the needs of the project, separate participative workshop or focus group or virtual meetings can be arranged for different practitioner or user groups. This will depend on the number of participants and should be agreed with NICE.

  • For some topic areas, academic researchers may be included in the commissioned primary research. In such cases, a separate meeting should be convened for them, using the same processes. This should be agreed with NICE.

  • Topic guides, prompts or supporting materials must be developed in collaboration with the guideline committee, and agreed by NICE.

1-to-1 or paired in-depth interviews

Interviews may be carried out face-to-face, by telephone or online. They may be appropriate when:

  • it is not possible to get groups together because the topic is a relatively new area, the number of possible participants is limited or there are geographical or time constraints

  • participants may have mobility difficulties or be unable to travel

  • the issues discussed are likely to be confidential or sensitive and anonymity may be needed

  • in-depth responses are needed.

Interviews may be structured or semi-structured, depending on the topic and the groups involved. Semi-structured interviews allow complex or difficult issues to be explored and so are likely to be more useful than structured interviews. All interviews should focus on areas in which views and experiences are needed, or on draft recommendations.

Individual or paired interviews are usually more expensive to set up than group work, and the need for in-depth or individual contact should be weighed against the available resources at the planning stage.

Surveys

Group-based methods and interviews are the best way to find out opinions. But they may not be suitable in all circumstances, for example, because of the sensitivity of the topic, confidentiality issues, or difficulties in recruiting participants. In such cases, surveys that use semi-structured and open-ended questions could be more appropriate. Surveys may be carried out by telephone, online, on paper or by using vote casting or polling.

Surveys gather opinions in a quick and less obtrusive manner than group-based approaches and interviews. The responses can also be quantified. But surveys do not allow the same depth of exploration and, generally, should only be used if other methods are unsuitable. Formal consensus methods such as Delphi survey and RAND appropriateness could be modified for the survey if appropriate.

Analysis of data

There are different ways of analysing data from commissioned primary research, depending on the methods used for data collection and whether it is qualitative or quantitative. Some descriptive summary statistics should be provided, for example, characteristics of participants and attendance or response rates.

Quantitative analysis should include a clear statistical data analysis plan, including a priori plans for sub-group analysis or co-variate adjustments.

Group-based methods and interviews are likely to generate qualitative data. Analysis may be performed using qualitative research software, or by hand, but the method should be fully reported in the proposal and the final report.

Qualitative data can be broken down into common and consistent themes for each of the questions asked, using, for example, a content analysis approach. Usually, 1 researcher should prepare an initial analysis, which should be verified by 'blind' coding and sorting of a sample of the transcript by a second researcher. For examples of this kind of analysis, see part 3 (chapters 7 to 13) of Silverman (2004) or Ritchie and Spencer (1993).

Once the analysis is complete, participants' quotes may be selected to illustrate each theme. These quotes should be coded to keep participants anonymous and to allow the quotes to be distinguished. When transcripts are processed, ensure confidentiality and data protection are fully considered. As with data from clinical trials, transcripts should be kept for at least 5 years (see the National Institute for Health and Care Research's clinical toolkit).

Surveys are likely to involve a mixture of quantitative and qualitative data. Quantitative data may be analysed and presented using summary statistics or may involve more advanced approaches such as regression analysis and multi-variable adjustments (which need agreeing a priori). When formal consensus methods such as Delphi survey and RAND appropriateness have been modified for the survey, specific analytical methods, for example, thresholds for agreement, should be stated in the proposal and the final report.

The developer or external contractor should ensure the methods for analysing the data and reporting standards are discussed and agreed with NICE.

Feedback mechanism

The developer or external contractor should agree with NICE a process for giving feedback to all participants. Providing feedback to participants should be specified in contracts. This should include an evaluation exercise, a follow-up session or sharing interim findings via email.

Agreeing the proposal

The proposal for the commissioned primary research should include information on the:

  • aim and objectives

  • recruiting participants

  • methods used

  • ethics considerations, including whether formal ethical approval is needed

  • timeframe of the primary research

  • data analysis

  • feedback mechanism.

The proposal and the final report of the commissioned primary research should be included as part of the guideline or guideline appendices.

The developer and the external contractor should agree with NICE the approaches and methods to use, including a summary of the issues to be covered. Similarly, the methodology and any questions or support materials used must be developed and agreed with NICE. For example, NICE should:

  • be briefed by the developer and external contractor in detail before work begins

  • agree final documents (including the proposal) and comment on draft recruitment letters

  • agree on topic areas and research questions

  • agree sampling frames and samples, and other supporting materials

  • agree approaches on how to get participants from key groups involved, including people who work with or are from seldom heard groups or those who share characteristics protected under equality legislation

  • have access to transcripts of all data

  • discuss and agree techniques for data analysis and themes for data presentation

  • comment on the commissioned primary research report before the final draft is submitted.

Obtaining ethical approval

Most primary research commissioned for NICE guidelines falls under the category of 'service evaluation' and so is outside the remit of NHS research ethics committees. However, the quality assurance team, the developer and the external contractor should consider ethical issues each time commissioned primary research is planned, to ensure appropriate expertise, and that policies and procedures for the safety and welfare of participants are in place. If there is any doubt, the developer or external contractor should consult the national Research Ethics Service. The developer or external contractor is responsible for seeking ethical approval, if it is needed.

For topics covering children and young people, NICE's patient and public involvement policy includes a set of principles for involving them and has an appendix about safeguarding. The national Research Ethics Service should also be consulted for topics covering children and young people and other vulnerable groups such as adults with learning disabilities or frail older people.

Box 1: examples of key milestones
Milestones Timeline/Date

Start-up meeting with contractor

Recruitment of research participants

Data collection period

Contractor submits draft report to developer, who shares with committee

Developer and committee provide comments on draft report, including a formal quality assessment of report by developer.

Contractor submits revised report to developer, who shares with committee

Contractor submits slides for presentation to committee (if necessary), who shares with committee

Presentation of report at committee meeting

Contractor submits final report after committee meeting


This page was last updated: