NICE real-world evidence framework

Developing the research question and study protocol

The study should provide a rationale for why a qualitative approach is considered the best way to meet the research aims. Available question formulation frameworks may be useful to clearly specify key components of the research question, for example: population, phenomena of interest, context (PICo); sample, phenomena of interest, design, evaluation, research type (SPIDER); and setting, perspective, intervention, comparison, evaluation (SPICE). See Cooke et al. (2012) and Booth (2006) for more information.

Developers should clearly outline and justify their selected approach prior to undertaking data collection. As with other studies involving the collection of patient data, a predefined protocol and quality assurance processes are necessary. The study protocol should cover: the research question and aims; rationale for a qualitative approach; the theoretical framework underpinning the qualitative approach; study design; data collection methods; analytical approach; and ethical and regulatory compliance. While qualitative research protocols may iterate over the course of research, this should be subject to agreement from regulatory bodies and strict version control.

Study ethics and data protection

It is essential to demonstrate ethical conduct and appropriate governance arrangements. Data should be collected, stored, processed and deleted in accordance with current data protection laws, with safeguards and appropriate transparency information provided to participants. Approvals from the Health Research Authority (HRA) or local organisation review and agreement should be in place. Information on attained ethical approvals and patient consent should be provided (see also the framework section on data collection).

The arrangements for safe and ethical conduct of qualitative studies should be described, such as any adaptations for vulnerable populations, how effects of the study were handled during and after the study (when dealing with sensitive topics), and any safety monitoring. The process of informed consent and study information provided to the participants should be available on request.

Participant anonymity should be ensured when presenting data and findings. When using verbatim extracts from participants in the study to substantiate findings, these should not lead to the study participants being identifiable. Where qualitative data is collected through web surveys, mechanisms should be in place to ensure survey anonymity and protection from unauthorised access.

Sampling strategy

Purposive sampling is commonly used in qualitative research and involves identifying and selecting participants who can offer relevant, information-rich data (Palinkas et al. 2015). The type of participants selected is specific to the outlined research aim. A more homogenous sample may be required for studies exploring perspectives of patients with a common set of characteristics. Maximum variation sampling may be appropriate for studies aiming to capture a wide range of perspectives. In general, convenience sampling, which recruits participants based on their availability, is considered less rigorous than purposive sampling.

In some cases, the mode of data collection may systematically exclude potential participants, affecting study findings. For example, interviews conducted online, or the mode of survey administration (online, email, postal) may inadvertently cause exclusion due to variation in digital literacy or access to technology.

Qualitative methods typically emphasise sampling until a point of data saturation has been reached, that is, until the researchers are no longer able to identify additional insights from the data. Continued data collection and analysis until saturation may be challenging and decisions to stop data collection may be based on other factors, such as requirements to complete within a feasible timeframe, participant response rate and availability.

Reporting with sufficient detail on study sampling will help reviewers consider whether the included sample can sufficiently capture diverse views of interest and if the insights are likely to be transferable to broader populations of interest. This can be supported by clearly describing:

Data collection

Semi-structured interviews, focus groups, and open-ended questions in surveys have been commonly applied to collect qualitative data in evidence considered by NICE. The suitability of each method depends on the research aims. For example:

The study report should clearly describe the data collection method used and the setting. The selected method should be justified in relation to its ability to support the research aims.

For interviews and focus groups, rapport within the researcher–participant relationship is important for the natural flow of the conversation, as well as participant engagement and comfort in expressing their views openly, especially when covering sensitive topics. The mode of data collection should be specified, such as whether collected as audio or video recording or any other method and if repeat interviews were undertaken. Field notes may also be taken, which may describe nonverbal cues and other observations to aid interpretation of the data.

Written surveys are typically administered online, by e-mail or as a postal survey. Online surveys may allow for wider and greater recruitment. However, testing ease of completing the online format, preventing multiple entries, and the potential to miss out on valuable data due to digital exclusion should be considered in study planning. Several e-based platforms are available to conduct surveys and, where used, their compliance with information governance requirements should be ensured.

For all outlined methods of data collection, researchers should consider the content, tone, order and phrasing of questions asked. Reflexivity (that is, considering how a researcher's own values and preferences may impact the findings of the research) is important to maintain throughout data collection. Care should be taken that the questions are not leading and can support participants in openly expressing their views.

Involvement of relevant stakeholders is recommended, including members of the target population (for example, patients or clinicians) as well as subject matter experts in the design of questions included in the interview, focus group guide, or survey. Key considerations for developing questions include unambiguity, brevity, explanation of terminologies and sensitivity (especially when exploring difficult topics).

Prior to study initiation, piloting questions and approaches within a subset of the target sample can help ascertain the questions' validity in meeting research aims and anticipate possible miscommunication. Feedback from these can then be used to refine, reorder or clarify the questions. Piloting is especially important for written surveys as there is usually no further interaction with participants to allow for prompts and clarifications.

Materials relating to the data collection methods can support a full understanding of the approach used. For example, the interview questions, interview schedule (such as whether a single or longitudinal interviews, delivered before or after intervention) and interview guide, or the full survey. These may be provided as part of the study appendix or on request. Details of piloting work undertaken and the groups involved in the design of interview or survey materials should be reported in the study methods sections, including patient and public involvement.

Data analysis

The process of qualitative data analysis includes transcribing data, data coding and categorisation, identification of themes, and interpretation. Thematic analysis is commonly applied in qualitative evidence presented at NICE and is the focus of this section (Byrne 2022, provides a worked example).

Studies with a high volume of data may use transcribing software or involve multiple transcribers. Accuracy of the transcription in these studies should be reviewed by the researcher involved in data collection.

In thematic analysis, transcribed data is coded and categorised through repeated review. Use of multiple researchers or independent coders can enhance credibility in this process. Typically, in a thematic analysis the coding list is developed in an iterative manner and is adapted and refined with ongoing data collection and analysis. Where large amounts of text are generated in qualitative data, software packages can assist with this process and support transparent and systematic handling and organisation of the data (for example, searching and assigning codes).

Through scrutiny of the coded data, researchers can observe similarities and differences in the data and develop summaries based on a hierarchical coding structure. For some research questions, these initial findings may be sufficient to present as general descriptive analysis for initial insights or can be used to inform further data collection and analysis with the aim of constructing explanatory theoretical models. To demonstrate a thorough nature of enquiry, it is recommended to investigate 'negative' cases that challenge the views of the majority, or conflict with previous findings, as these may be insightful or may support further targeted data collection.

The interpretation of data is inevitably influenced by a researcher's understanding or perspectives, as compared to standardised numerical approaches. Therefore, understanding how the values and preferences of the researcher may impact the findings of the research or 'researcher reflexivity' is an important aspect of rigour in a qualitative study. This can be explored through a description and consideration of involved researchers' qualities (such as their role, characteristics, research training and experience, and position on the issue discussed) as well as other potential conflicts of interest (for example, funding source or researcher allegiance). Statements of researcher reflexivity are generally recommended as good practice to understand the position of researchers in relation to the research and participants.

Overall findings of the study and interpretation should be clearly described. Reported themes should be substantiated with the data collected, for example with verbatim extracts, text excerpts or field notes. A full description of the approach used to develop code lists, perform thematic analysis, and assess research reflexivity in the study report will enable informed study assessment.

Reporting and assessment of qualitative studies

Several checklists are available that aim to provide a guide to reporting qualitative studies in a structured and transparent way.

The COREQ reporting tool provides guidance for reporting primary qualitative studies collected through interviews and focus groups. Reporting standards for how qualitative data is handled, analysed and interpreted can be similarly applied for qualitative data that is collected as free text embedded within a quantitative survey.

Additional elements that are important and should be provided in reporting a qualitative study include:

Developing NICE guidelines: the manual recommends the Critical Appraisal Skills Programme (CASP) checklist for the assessment of previously conducted primary qualitative studies.