3 Approach to evidence generation | Tools and resources | ProKnow cloud-based system for radiotherapy data storage, communication and management: early value assessment | Guidance

3.1 Evidence gaps and ongoing studies

Table 1 summarises the evidence gaps and ongoing studies that might address them. More information on the studies in this table can be found in table 9 in the external assessment group report. After publication of NICE's health technology assessment of ProKnow, no evidence directly related to the evidence gaps was identified (search completed 27 April 2023). No new ongoing studies were identified.

**Table 1 Evidence gaps and ongoing studies**
Evidence gap	Existing and real-world evidence	Ongoing studies
Impact on quality assurance for radiotherapy treatment planning	Limited evidence available	Ongoing study
Impact on staffing and treatment planning resources	Limited evidence available	Ongoing study
Changes in access to radiotherapy treatment	Limited evidence available	No ongoing studies
Impact on staff training	Limited evidence available	Ongoing study

In March 2022, NHS England (NHSE) commissioned an evaluation of ProKnow across 49 specialist cancer centres. Funding has been provided until March 2025 as part of the Radiotherapy Transformation Programme. This programme aims to improve the quality and reduce variability of radiotherapy service delivery across the NHS. Data collection within the NHSE evaluation is not mandatory, and uptake of ProKnow is optional and variable.

3.2 Data sources

There are several data sources, with different strengths and weaknesses, that could potentially support evidence generation. NICE's real-world evidence framework provides detailed guidance on assessing the suitability of a real-world data source to answer a specific research question.

Because data on patient radiotherapy treatment plans is uploaded to and analysed in ProKnow, this would be the main data source for evidence generation. In the NHSE evaluation, each NHS Trust with access to ProKnow acts as its own data controller. Treatment plan information can be uploaded to local workspaces from local treatment planning software. From there, it can also be uploaded to national workspaces in an anonymised format. National collection workspaces are organised into themes based on cancer type, population and treatment modality. Plans uploaded to national datasets are being used to develop national guidance and service improvement toolkits. They are also being used to develop sets of scorecards for treatment plans in areas with enough data, such as lung stereotactic ablative radiotherapy, which could be used for system quality improvements.

Linking ProKnow data to other routinely collected data, such as cancer registries and the National Radiotherapy Dataset, could provide additional information, such as clinical outcomes. But this would be challenging to achieve within the evidence generation period, and would provide limited information related to the evidence gaps.

The quality and coverage of real-world data sources are of key importance when they are used in evidence generation. Active monitoring and follow up through a central coordinating point is an effective and viable way of ensuring good-quality data with broad coverage. It could also be assisted through national linkage efforts. Currently, all specialist radiotherapy services in England have access to ProKnow as part of the NHSE evaluation. Central active monitoring of the national workspace is overseen by the ProKnow Clinical Leadership and Implementation Group.

3.3 Evidence collection plan

The proposed approach to addressing the evidence gaps for ProKnow is a combination of real-world data collection and cross-sectional survey-based data collection. Real-world data collection could be through a time-series study or an alternative quasi-experimental design over the course of the NHSE evaluation period.

Cross-sectional survey

Feedback from staff with ProKnow access could be collected periodically (for example, annually) by survey. This should cover user experience on key issues, including the impact of ProKnow on quality assurance for radiotherapy treatment planning and on staff time and resources. This study approach could also collect data on uptake of the contouring accuracy training module for the technology, providing evidence for the impact on staff training (see section 2.2).

Real-world time-series study

The impact of the technology on quality assurance for radiotherapy treatment planning, staffing and resources can be measured through a time-series analysis of data collected from each centre. This would allow periodic analyses of data collected as part of the NHSE evaluation. It would also allow the adoption of more formal evaluation methods such as before and after studies, interrupted time-series analysis and stepped wedge analyses. This could provide increasingly robust quantitative evaluations of the relative impacts of changes in ProKnow use or address specific questions as they arise during the NHSE evaluation.

Data should be analysed and compared across defined timepoints showing trends over time. This includes:

local and national workspace uptake
variability (positive deviance) and spread of treatment plans according to plan quality metrics (such as scorecards or benchmarking against national standards)
peer reviews
local and national workspace use.

This may highlight centres that are performing well, which may provide lessons to emulate, and centres that are performing less well, which may provide opportunity for improvement. This approach allows for monitoring of adherence to existing clinical guidance, and quantifies the variation in clinical practice over time.

Changes in access to specific treatments, such as stereotactic ablative radiotherapy, could also be considered within this study design.

Data analysis should ideally capture the use of ProKnow for prospective treatment planning and quality assurance.

Data collection should follow a predefined protocol. Also, quality assurance processes should be put in place to ensure the integrity and consistency of data collection. See NICE's real-world evidence framework, which provides guidance on the planning, conduct, and reporting of real-world evidence studies.

3.4 Data to be collected

The following information has been prioritised for collection within each of the recommended study designs:

Survey data, reported on an annual basis, including:
- number of audits done and planned
- usability and ease of retrieving and storing data
- use of ProKnow contouring accuracy training module
- perceived impact of ProKnow on:
  - treatment planning
  - access to peer review for treatment planning
  - quality assurance activities such as scorecard development or analysis, or completion of audits
  - number of treatment planning errors and their causes
  - any adverse events and their causes and
  - staff time and resource.
- open questions covering:
  - impacts of ProKnow on improving treatment planning and quality assurance activities, such as peer review and audit
  - negative impacts of ProKnow, for example, on staff time or resources
  - development of guidance through using ProKnow and case examples of the impact of the technology on practice
  - the impact of using ProKnow on day-to-day clinical practice, including any perceived benefits or challenges
time-series data on:
- essential outcomes, such as:
  - number of patient treatment plan uploads to local workspaces across all NHS centres per month
  - number of patient treatment plan uploads to local workspaces that are uploaded to national workspaces (by cancer type and treatment modality) across all NHS centres per month
  - any adverse events associated with ProKnow, for example, data transfer resulting in changing of labelling conventions or dosimetry data because of lack of interoperability with other treatment planning or software systems
- desirable outcomes, such as:
  - plan variance and spread (between acceptable and major or minor deviations) according to scorecard metrics, which includes dosimetry data
  - quarterly or annual data analysis of treatment plans uploaded for 1 to 3 core metrics per scorecard.

3.5 Evidence generation period

This should be 2 years to align with the commissioning position of the technology, and to allow for the evidence to be sufficiently mature to support future decision making.