3 Approach to evidence generation
3.1 Evidence gaps and ongoing studies
No ongoing study has been identified that could address the evidence gaps for this technology.
3.2 Data sources
There are several data collections that have different strengths and weaknesses that could potentially support evidence generation. NICE's real-world evidence framework provides detailed guidance on assessing the suitability of a real-world data source to answer a specific research question.
Local or regional data collections that have historically supported evidence generation in this clinical area may be able to collect the specific outcome measures needed to support evidence generation. For example, the primary clinical trial that informed NICE's early value assessment guidance on virtual reality technologies for treating agoraphobia or agoraphobic avoidance was done in 9 NHS trusts and the underlying data infrastructure supporting that work could be repurposed.
The Mental Health Services Dataset (MHSDS) is a mandated national data collection that could collect the necessary data. But it may not routinely collect all the outcome measures that were identified in the early value assessment for this evidence generation plan. Also, data may not have been submitted for all people using mental health services and there are potential issues with data quality. NHS England has suggested that modification of the MHSDS could take up to 2 years, so it is unlikely that it could be modified in time to support data collection for this evidence generation plan.
The Clinical Record Interactive Search (CRIS) system is a subnational dataset based on information from the South London and Maudsley NHS Foundation Trust's clinical records. It may include data on clinical outcomes of people with psychotic disorders that could be used to increase and improve the data collected from the study proposed in this plan.
The quality and coverage of real-world data collections are of key importance when used in generating evidence. Active monitoring and follow up through a central coordinating point is an effective and viable approach of ensuring good-quality data with broad coverage.
3.3 Evidence collection plan
Real-world comparative study
The suggested approach to addressing the evidence gaps for the technology is a comparative interrupted time series (CITS) study using individual-level data collection. In practice, there is considerable variation across services in the delivery of care for severe agoraphobic avoidance in people with psychosis. Standard care may include cognitive behavioural therapy (CBT), antidepressants or monitoring from community mental health services. A CITS design will allow for evaluation of the technology's effects, controlling for variation across services while limiting the risk of time-varying cofounders. This study design also supports continuous enrolment of people in smaller populations. This study should include people who have severe agoraphobic avoidance with psychosis, and who would be expected to be offered virtual reality (VR) therapy in usual practice.
Data should be collected for everyone who has severe agoraphobic avoidance with psychosis, having care in community mental health services that are enrolled in the study. The data collection period should be long enough to ensure there is 6‑month follow-up data while people have standard care. People should be offered VR therapy through the technology after 6 months of standard care. Data should then be collected from people offered therapy through the technology for a further 6 months. The study should compare outcomes in the 6 months before offering VR therapy with the 6 months after offering VR therapy. In both observation periods, data should be collected at regular time intervals but must include all of the following:
-
the point of entry to the study (the beginning of 6 months of standard care)
-
6 months after standard care but before starting VR therapy
-
6 weeks after starting VR therapy (the point at which VR therapy would be expected to finish)
-
6 months from being offered treatment with the technology.
This study could be done within a single service or, ideally, replicated across multiple services to show how the technology can be implemented across a range of services, representative of the variety of care in the NHS. Outcomes can reflect other changes that occur over time in the population, unrelated to the interventions. So, additional robustness can be achieved by collecting data from services which have not implemented the technology. These services should be as similar as possible in clinical practice and patient demographics to the services in which the technology has been deployed. This could control for changes in outcomes over time that might have occurred anyway over the same time period.
High-quality data on patient characteristics is needed to assess who the technologies would be suitable for and who engages with the technologies. Important characteristics should be identified with input from clinical experts during the protocol development. Loss to follow up, and reasons for loss to follow up over the data collection period should be reported.
Data collection could be approached using multiple sources, for example:
-
engagement and drop-out information generated through the technology
-
some information collected through suitable real-world data sources
-
other information collected by clinicians and research staff working with the person, for example clinical measures and quality-of-life data.
Data collection should follow a predefined protocol and quality assurance processes should be put in place to ensure the integrity and consistency of the data collection. NICE's real-world evidence framework provides guidance on the planning, conduct, and reporting of real-world evidence studies of comparative treatment effects and outlines potential sources of bias. Guidance from the NICE real-world evidence framework also includes reporting standards for interrupted time series studies.
3.4 Data to be collected
The following information has been prioritised for collection:
-
At baseline, clinical diagnosis for agoraphobia in people with psychosis who are having difficulty leaving the house, as assessed in line with the World Health Organization's International Classification of Diseases (ICD)-10.
-
Information on standard care provided in the mental health services. Standard care may include CBT, antidepressants or monitoring from community mental health services. Details of processes and NHS staff should also be collected because they may vary between services.
-
In people having VR therapy with the technology, the number of extra sessions delivered beyond the standard 6‑week therapy.
-
The Oxford Agoraphobic Avoidance Scale (O‑AS) or the Agoraphobia Mobility Inventory (AMI) score at regular time intervals (see section 3.3).
-
Changes in agoraphobic symptoms, defined according to a standardised measure.
-
Recovery, defined according to a standardised measure of agoraphobic symptoms, with a predefined threshold, at regular time intervals (see section 3.3).
-
Relapse to severe agoraphobic symptoms, that is, the number of people with previous recovery who relapse to severe agoraphobic avoidance at a later time point.
-
EQ‑5D or Recovering Quality of Life questionnaire scores at regular time intervals.
-
Any adverse events through use of the technology or VR therapy.
-
Access and uptake: this should include the number and proportion of people who were able to access treatment for severe agoraphobic avoidance before and after deployment of the technology in the service.
-
Engagement and information on stopping VR therapy, including reasons for stopping. This should include the number of people starting VR therapy, the number of sessions completed, and whether it was stopped because of improvements in symptoms, adverse effects, or other reasons.
-
Information about individual characteristics at baseline, including sex, age, ethnicity, clinical diagnosis, medicines, comorbidities and past medical history including previous interaction with tiered mental health services, and other important characteristics identified with input from clinical experts.
-
Costs of VR therapy with the technology, including any costs associated with providing the VR headset, licence fees, maintenance and replacement of the hardware, and costs associated with the delivery of extra sessions when needed.
-
Other health service use costs including NHS staff grade and time needed to support the use of the technology, and cost and time associated with staff training.