Artificial intelligence (AI)-derived software to help clinical decision making in stroke

Concordance study

A concordance study is used to assess the agreement between 2 or more methods. The evidence gaps that this will address are:

Each patient case would include clinical data available at the time of scanning, and may include unenhanced CT, CT angiography (CTA) and CT perfusion (CTP) images, for review alongside each other, as appropriate according to standard care.

This study will assess the concordance between the treatment decision reached for each included case of suspected stroke by:

Data should be selected carefully to be representative of existing sites, considering responses to the user survey. This is to make sure acute and comprehensive stroke centres, relevant staff roles and experience levels, and specific AI software are represented.

Representative image sets would be provided by stroke centres, anonymised, and processed by e-Stroke, RapidAI, Viz, and no AI software. They would then be presented to, and assessed retrospectively by, recruited healthcare professionals with appropriate qualifications and experience, recommending no treatment, thrombolysis, or thrombectomy. Cases that the software was unable to analyse would be recorded.

If every reader assessed every case, using every AI software and none, this would be a full factorial design. But a pragmatic approach to optimise healthcare resource usage is a 'split plot' design where each reader assesses a subset of cases, but each case is assessed using each AI software and none, enabling a comparison. The sample size (number of patient cases, and number of readers) should account for the following factors, and patient cases will be randomly allocated to readers to ensure each factor is fairly represented:

The factors above would be controlled for in the design of the study, and the influence of other, explanatory, variables could be considered in cases of discordance. These variables may account for particular groups of interest, such as people aged over 80, people with cerebrovascular disease, specifications of different scanners used, type of stroke centre, and types of CT imaging included in the case.

Comparison between AI-assisted (intervention), and unassisted (comparator) readings, and the expert panel (reference standard) would allow assessment of concordance for each AI software, used as intended. By including 2 staff groups (radiologists and physicians), concordance could be measured within each group, and between groups. Discordant cases could be further explored to identify common characteristics, and reasons for discordance could be suggested by the expert panel.

A secondary outcome of this study would be the time spent assessing images to reach a treatment decision in each case, with and without AI assistance. Although this timing activity lacks some real-world validity by not reflecting all steps in the care pathway, this element of the proposed study may highlight differences in read times between different staff groups, and different experience levels.

Other concordance study designs are possible, which could better represent real-world use of the software, but may have less internal validity or need greater data collection.

Evaluation of SSNAP data

The evidence gaps that this will address are:

The work supported by Health Innovation Oxford & Thames Valley is using SSNAP data to capture use of e-Stroke across 26 sites in England. Because of variation in how time to outcome data was collected before April 2021, use of a 'before-and-after' study design is limited. But because sites implemented AI software at different time points, time between CT scan and treatment decision could be monitored over time to determine whether this decreased as use of AI increased. Although the work of Health Innovation Oxford & Thames Valley is restricted to e-Stroke, its methodology could be applied to studies of other software, including RapidAI and Viz.

A random selection of cases could be extracted from SSNAP. Completion of the data field relating to the use of AI software could be used as a factor in subsequent analysis. Time since AI implementation and the AI software in use at each site (taken from the user survey) could also be treated as factors. Additional data fields may be needed to address how often the software is unable to analyse CT brain scans, and the reasons why, which could be added to the Comprehensive Dataset for specific sites of interest.

Information to be collected by surveying all stroke centres using AI software

The following data items should be collected for each site:

Concordance study

Evaluation of SSNAP data

The following data items should be extracted, or calculated from the SSNAP dataset, for periods before and after AI implementation:

A subgroup analysis could consider the difference in time to treatment for those referred to another site for thrombectomy, or shorter or longer time after stroke symptom onset. For outcomes lacking comparable data from a 'before' phase, analysis could be repeated using quarterly extracts from SSNAP to determine whether increased use led to changes over time.

Data collection should follow a predefined protocol and quality assurance processes should be put in place to ensure the integrity and consistency of data collection. See NICE's real-world evidence framework, which provides guidance on the planning, conduct, and reporting of real-world evidence studies.