Advice
Clinical and technical evidence
Clinical and technical evidence
A literature search was carried out for this briefing in accordance with the interim process and methods statement for medtech innovation briefings. This briefing includes the most relevant or best available published evidence relating to the clinical effectiveness of the technology. Further information about how the evidence for this briefing was selected is available on request by contacting mibs@nice.org.uk.
Published evidence
There are 6 studies summarised in this briefing, selected as the most relevant and best quality evidence relating to the technology. The studies comprise evidence on a range of people, including 160 with relapsing–remitting MS. Four studies did not clearly state the number of people with relapsing–remitting MS and 1 did not state the number of people at all. The selected studies report on an earlier version and the current version of icobrain ms (previously MsMetrix).
The clinical evidence and its strengths and limitations is summarised in the overall assessment of the evidence.
Overall assessment of the evidence
Overall, the quantity and the quality of the technical evidence for the technology is moderate, but limited in terms of the clinical benefits, particularly for patients. There is no evidence in the NHS. The comparator in the technical validity studies is a reference (gold) standard, which is suitable. But in some cases, a different comparator could be more appropriate, for example, patient outcomes with and without the technology. The evidence comes from 5 papers and 1 conference abstract that report on multiple outcomes and often include people with different MS phenotypes. It was unclear how many people had relapsing–remitting MS in 4 studies. More prospective comparative studies are needed to evaluate the patient benefits of the technology. These should include assessment of icobrain ms in the NHS.
Van Hecke et al. (2021)
Intervention and comparator
icobrain ms lesion segmentations assessed by a non-specialised neurologist compared with experienced neuroradiologists; MRI results assessed by a radiologist twice, once with a computer-aided report with icobrain ms and once without.
Key outcomes
Performance of radiologists with different levels of experience. Intra-rater test-retest lesion count agreement on scan and rescan images was significantly improved for non-specialised neurologists, but the effect was not observed for experienced neuroradiologists. Inter-rater lesion count agreement was significantly better when icobrain ms was used. The mean time for conventional radiological reporting was 7 minutes 28 seconds (standard deviation [SD] 3 minutes 6 seconds). The median computer-aided reporting time was 5 minutes 49 seconds (SD 2 minutes 15 seconds). People with relapsing–remitting MS and people with clinically-isolated syndrome showed significantly higher whole brain and grey matter volumes and lower ventricular volumes compared with other groups.
Strengths and limitations
The study provides direct evidence for the potential system benefits of the technology. Not all people in the study had relapsing–remitting MS. All authors were affiliated with the company. Some of the results were also reported in abstract form (Sima et al. 2020).
Beadnall et al. (2019)
Intervention and comparator
Automated assessment with icobrain ms compared with Structural Image Evaluation using Normalisation of Atrophy (SIENA) analysis (gold standard).
Key outcomes
Agreement of PBVC measurements between the 2 technologies and assessment of correlation between lesion volume and PBVC. Annualised PBVC as measured by icobrain ms correlated strongly with SIENA measurements (Pearson's correlation coefficient, r=0.805), with excellent consistency (Intraclass Correlation Coefficient, ICC=0.801) and agreement (ICC=0.800). Agreement between non-annualised PBVC measurements was very similar (correlation coefficient, r=0.797; consistency ICC=0.793; and agreement ICC=0.793). Change in total FLAIR lesion volume weakly correlated with annualised PBVC for icobrain only (Kendall Tau rank correlation analysis, tau=0.134; p=0.046).
Strengths and limitations
The technology evaluated was compared with the current gold-standard method for the measurement of PBVC (SIENA). A limitation is that not all people in the study had relapsing–remitting MS and that brain atrophy is currently not used as an indicator in the NHS. Three of the authors were affiliated with the company.
Smeets et al. (2016)
Study size, design and location
A technical validation study of 167 separate scan pairs (from 10 people with MA in dataset 1, 3 healthy subjects in dataset 2, and 20 people in dataset 3) evaluating the measurement error, robustness toward physiological processes and consistency of icobrain ms.
Intervention and comparator
Automated assessment with icobrain ms compared with SIENA analysis (gold standard).
Key outcomes
Measurement error, robustness and consistency of the technology, measured by PBVC. icobrain ms showed measurement errors comparable to, or lower than, SIENA. The test–retest PBVC computed by icobrain ms differed in absolute value from the expected 0% by a median of 0.13% (interquartile range [IQR] 0.09% to 0.29%). Those of SIENA differed from 0% in absolute value by 0.17% (IQR 0.08% to 0.22%). However, the difference between the technologies was not significant (p=0.54, paired t‑test; p=0.60, Wilcoxon signed-rank test). icobrain ms showed robustness toward daily physiological processes. icobrain's overall error for whole-brain atrophy was significantly smaller than SIENA's (median absolute value of 0.19% for icobrain ms and of 0.31% for SIENA). The longitudinal consistency of the technology was assessed by the correlation between whole-brain atrophy measurements obtained with icobrain ms and SIENA. A Pearson correlation coefficient r=0.91 and an intraclass correlation coefficient ICC=0.90 suggested relatively high consistency. No significant differences were seen between the technologies in terms of the consistency index for 6‑month intervals compared with the 1‑year interval of the whole-brain atrophy.
Strengths and limitations
The technology evaluated was compared with the current gold-standard method for the measurement of PBVC (SIENA). However, this technical validity study gives little evidence for the clinical validity of the technology. A further limitation is that it is unclear which people in the study had relapsing–remitting MS. Also, the data used is heterogenous and comes from 3 countries (Belgium, the US and the Czech Republic). Eight of the authors were affiliated with the company.
Wang et al. (2016)
Study size, design and location
A technical validation study of 63 MRI scans (61 from people with relapsing–remitting MS, 2 from people with clinically-isolated syndrome) evaluating cross-sectional whole-brain volume (WBV) in people with MS in Australia.
Intervention and comparator
Automated cross-sectional assessment with icobrain ms compared with SIENAX (structural image evaluation using normalisation of atrophy-cross-sectional) analysis (version for cross-sectional analysis; gold standard).
Key outcomes
Precision and accuracy of WBV for icobrain ms compared with SIENAX. The statistical precision and accuracy of WBV estimation for icobrain ms compared with SIENAX were 0.992 (p<0.001) and 0.994, respectively. There was statistical agreement between the methods. icobrain ms showed a 1.0% volume overestimation compared with SIENAX.
Steenwijk et al. (2017)
Study size, design and location
A technical validation study on 20 simulated brain images, as well as in vivo data from 100 people with MS and 20 people without MS to act as controls. The study evaluated cross-sectional and longitudinal WBV and grey matter volume in the Netherlands.
Intervention and comparator
Automated cross-sectional assessment with icobrain ms compared with SIENAX analysis (gold standard) and longitudinal assessment with icobrain ms compared with SIENA analysis (gold standard).
Key outcomes
icobrain ms displayed bigger average deviation than SIENAX when compared with the reference WBV in the simulated data and in vivo. In longitudinal assessment, mean difference percentage brain volume change was low between icobrain ms and SIENA. icobrain ms suffered proportional errors in some cross-sectional and longitudinal methods.
Strengths and limitations
The sample size for longitudinal assessment was smaller than that for cross-sectional assessment. Another limitation is that the hardware for MRI was upgraded between baseline and follow up. The literature suggests that this may lead to substantial reduction in the reliability of the results.
Van Hecke et al. (2019)
Intervention and comparator
New and/or enlarging lesion count was assessed between baseline and 1‑year follow-up MRI scans and summarised in reports. Standard clinical radiological reports were compared with structured neuroradiologist review and automated software reports produced with icobrain ms.
Key outcomes
Both clinical radiological and neurologist review reports indicated the same 33% of people having new or enlarging lesions, yet incomplete separate counts. The proportion was 45% when using icobrain ms. Clinical radiological and neurologist review reports specified a presence of enlarging lesions in only 5% of people, while icobrain ms detected such lesions in 39% of people.
Strengths and limitations
The study suggests that automated techniques agree well with manual lesion counts in people with MS having MRI scans. However, the setting of the intervention and the number of people with relapsing–remitting MS in the study were unclear. This study is reported in conference abstract form only so is limited in detail. Three of the authors were affiliated with the company.
Recent and ongoing studies
Clinical disease activity with long-term natalizumab treatment: a retrospective observational study of 2 cohorts investigating the proportion of people free of new or enlarging fluid-attenuated inversion recovery (FLAIR) lesions assessed by semiautomatic lesion count by icobrain ms. ClinicalTrials.gov identifier: NCT02677077. Status: completed, no results published. Indication: new or enlarging FLAIR lesions. Medicine: natalizumab, Biogen. Date: 4 January 2019. Countries: Belgium and Czech Republic.