Tests in secondary care to identify people at high risk of ovarian cancer

Assessment of test accuracy

Table 4 Comparative accuracy of RMI I at thresholds of 200 and 250

Risk of ovarian malignancy algorithm (ROMA)

4.7 Fourteen studies (in 22 publications) reported diagnostic accuracy data for the ROMA using either Abbott ARCHITECT assays (9 studies) or Roche Elecsys assays (5 studies). No studies were identified that used the Fujirebio Lumipulse G automated CLEIA system.

Table 5 Comparative accuracy of ROMA (using Abbott ARCHITECT assays) and RMI I (threshold of 200)

4.10 Further identified studies assessed the performance of the ROMA score (using the Abbott ARCHITECT assays and at the company's suggested thresholds) without comparison with RMI I, across a range of target conditions. These included epithelial ovarian malignancies (both including and excluding borderline tumours). One study reported that the sensitivity of the ROMA was higher when the target condition was stage III or IV epithelial ovarian cancer, rather than stage I or II. Also, accuracy data at ROMA thresholds different from those suggested by the manufacturer were identified, but the EAG commented that no alternative threshold offered a clear performance advantage.

Table 6 Diagnostic accuracy of ROMA (using Roche Elecsys assays and manufacturer's suggested thresholds)

4.13 Two studies assessed the performance of the ROMA score (using the Roche Elecsys assays and at the company's suggested thresholds) without comparison with RMI I and with a target condition of ovarian malignancies excluding borderline tumours. The sensitivity estimates from these studies were very different (95.5% and 53.8%) and no summary estimate was calculated. Also, accuracy data at ROMA thresholds different from those suggested by the manufacturer were identified, but the EAG commented that no alternative threshold offered a clear performance advantage.

Simple Rules

4.15 Seventeen published studies had data on the diagnostic accuracy of Simple Rules. Eleven of these studies were done in Europe, including 3 in the UK. Two studies were multinational and included UK participants, 2 studies were done in Thailand, 1 was done in Brazil and 1 study did not provide detail on location. Also, the provided interim report (academic in confidence) had diagnostic accuracy results for Simple Rules. In studies included in summary estimates of sensitivity and specificity, Simple Rules was done by a level 2 or 3 examiner as defined by the European Federation of Societies for Ultrasound in Medicine and Biology (EFSUMB) classification system; 1 study also reported data from level 1 examiners.

4.16 Four published studies and the unpublished interim report provided direct comparison of the accuracy of Simple Rules and RMI I at a threshold of 200. The summary estimate of sensitivity was statistically significantly higher for Simple Rules (93.9%; 95% CI 92.8 to 94.9%) when compared with RMI I (threshold of 200; 66.9%; 95% CI 64.8 to 68.9%); however the summary specificity estimate was statistically significantly lower (74.2% [95% CI 72.6 to 75.8%] compared with 90.1% [95% CI 88.9 to 91.2%]). All these studies included all participants in analysis, regardless of their final histopathological diagnosis (target condition all malignant tumours including borderline). The unpublished interim report also directly compared Simple Rules and RMI I (threshold of 250; academic in confidence).

4.17 A further 4 studies had data on the accuracy of Simple Rules for the same target condition but without a direct comparison with RMI I. There was no statistically significant change in sensitivity (94.2%; 95% CI 93.3 to 95.1%) or specificity (76.1%; 95% CI 74.9 to 77.3%) when data from these studies were included in the summary estimates of Simple Rules accuracy (a total of 8 published studies and the unpublished interim work).

4.18 Three studies directly compared Simple Rules and RMI I (threshold of 200) stratified by menopausal status. There was no statistically significant difference between the sensitivity and specificity estimates for Simple Rules produced for the pre- and postmenopausal subgroups. However if data from a further study (which did not report a direct comparison with RMI I) were added, the summary estimate for specificity was statistically significantly higher for people who are premenopausal (79.3%; 95% CI 77.0 to 81.5%), when compared with people who are postmenopausal (67.3%; 95% CI 63.5 to 70.9%).

4.19 In the above estimates of accuracy for Simple Rules, inconclusive results were treated as malignancy positive. Test accuracy data were also available from some studies in which inconclusive results were instead classified by expert subjective assessment of the ultrasound images. Assessment of inconclusive results from Simple Rules using expert subjective assessment (rather than assuming them to be malignant) statistically significantly increased the specificity of the test, but statistically significantly lowered sensitivity.

The ADNEX model

4.20 Six published studies had data on the diagnostic accuracy of the ADNEX model. One was done entirely in the UK and 2 were multicentre studies that included UK participants. The remaining 3 studies were done elsewhere in Europe. A further unpublished interim report (provided as academic in confidence) also had data on the diagnostic accuracy of the ADNEX model. Four of the studies did not report details about the people doing the ultrasound scans. In 1 study, ultrasound scans were done by EFSUMB level 2 ultrasound examiners (non-consultant gynaecology specialists, gynaecology trainee doctors and gynaecology sonographers) and in another study they were done by EFSUMB level 2 or 3 practitioners with 8 to 20 years' experience in gynaecological sonography.

4.21 The EAG focused on test accuracy at the 10% threshold. One published study and the unpublished interim report made a direct comparison between the ADNEX model and RMI I (threshold of 200). Sensitivity was statistically significantly higher for ADNEX (96.0%; 95% CI 94.5 to 97.1%) than RMI I (threshold 200; 66.0%; 95% CI 62.9 to 69.0%), but specificity was statistically significantly lower (67.0% [95% CI 64.2 to 69.6%] compared with 89.0% [95% CI 87.0 to 90.7%]). Also, a further 2 studies reported on the accuracy of the ADNEX model in the same target population (all malignant tumours including borderline) but without direct comparison with RMI I. Inclusion of data from these studies in summary estimates did not cause a statistically significant change to sensitivity (96.3%; 95% CI 95.3 to 97.1%) or specificity (69.1%; 95% CI 67.4 to 70.8%) of the ADNEX model. The unpublished interim report also directly compared the ADNEX model and RMI I (threshold of 250; academic in confidence).

4.22 Two further studies had data on the accuracy of the ADNEX model without comparison with RMI I. These studies excluded people with histopathological diagnoses other than primary ovarian cancer from analysis (target condition ovarian malignancies including borderline). The summary estimate of sensitivity from these studies did not differ significantly from that of studies that included all participants in analysis; however the summary estimate of specificity (77.6%; 95% CI 73.6 to 81.2%) was statistically significantly higher.

4.23 Data stratified by menopausal status was available from 1 study. No statistically significant effect on sensitivity was reported, but specificity was statistically significantly higher for people who were premenopausal than for people who were postmenopausal.

4.24 One published study and the unpublished interim analysis directly compared the ADNEX model and Simple Rules (inconclusive results assumed to be malignant). The summary estimate of sensitivity was statistically significantly higher for ADNEX (96.0%; 95% CI 94.5 to 97.1%) than Simple Rules (92.8%; 95% CI 90.9 to 94.3%). Summary estimates of specificity were similar.

Overa (MIA2G)

4.25 Three studies (in 4 publications) had data on the diagnostic performance of Overa (MIA2G). All the studies were done in the USA and used a score of 5 units as a threshold. No studies were identified that directly compared Overa (MIA2G) with RMI I (at any threshold). However, 1 study assessed the accuracy of the Overa (MIA2G) and ROMA (using Roche Elecsys assays and manufacturer suggested thresholds for ROMA) in the same population with a target condition of all malignancies including borderline. Overa (MIA2G) had a statistically significantly higher sensitivity (91.0% [95% CI 86.8 to 94.0%] compared with 79.2% [73.7 to 83.8%]) and statistically significantly lower specificity (65.5% [95% CI 62.0 to 68.8%] compared with 78.9% [75.8 to 81.7%]) than the ROMA in this study.

4.26 Two further studies reported the diagnostic accuracy of Overa (MIA2G) without comparison with other risk scores. The summary estimate of sensitivity was 90.2% (95% CI 84.6 to 94.3%), and specificity was 65.8% (95% CI 61.9 to 69.5%). One of these studies assessed subgroups of people who were pre- and postmenopausal; there was no statistically significant difference between these groups.

Systematic review of cost-effectiveness evidence

4.27 The EAG did a systematic review to identify existing studies that assessed the cost effectiveness of the included tests and risk scores to help identify people with ovarian cancer. Five studies were identified, however 2 of these related to the use of tests in screening so were not applicable to the scope of this assessment. One of the studies (Havrilesky et al. 2015) included the ROMA and the Multivariate Index Assay algorithm (MIA; from Vermillion who also produce the Overa [MIA2G; multivariate index assay 2nd generation]). Both were dominated (that is, they cost more and produced less life years) by the use of CA125 alone or by a strategy of referring all people for specialist care (without testing). Conversely, in Forde et al. (2016) MIA dominated the use of CA125 alone (that is, it was cost saving and produced more quality-adjusted life years [QALYs]). No identified studies assessed the cost effectiveness of all the tests and risk scores included in this assessment.

Modelling approach

4.28 The EAG developed a de novo economic model designed to assess the cost effectiveness of the following tests and risk scores when used in secondary care to help decide whether to refer people with suspected ovarian cancer to a specialist multidisciplinary team (MDT):

4.29 The model did not include assessment of the ROMA using Fujirebio Diagnostics' Lumipulse G HE4 assay because no studies were identified that provided data on the accuracy of the ROMA using this assay. In the base-case analysis the starting cohort was assumed to be 40 years old, consistent with the modelling produced for the NICE guideline on ovarian cancer. All costs and effects included in the model were discounted by 3.5%.

Table 7 Diagnostic accuracy estimates used in the model

Table 8 Risk score costs used in modelling

Table 9 Utility scores used in modelling

Base-case results

4.35 The following assumptions were applied in the base-case analysis:

4.36 In the base-case model analysis, the EAG did a pairwise analysis comparing the costs and QALYs resulting from using the included tests and risk scores with RMI I (threshold of 250), and also a fully incremental analysis (table 10). Use of Simple Rules (inconclusive assumed to be malignant) was the cheapest and second most effective, and dominated RMI I (at a threshold of 200 and 250). Use of the ADNEX model was most effective (that is, produced the most QALYs) and when compared with Simple Rules produced an incremental cost-effectiveness ratio (ICER) of £15,304 per QALY gained. Use of the ROMA and Overa (MIA2G) were dominated.

Table 10 Base-case analysis results

4.37 At a maximum acceptable ICER of £20,000 per QALY gained, the ADNEX model and Simple Rules had a probability of being cost effective of 60% and 39% respectively. At a maximum acceptable ICER of £30,000 per QALY gained, these probabilities were 75% (ADNEX) and 23% (Simple Rules). The probability of RMI I (threshold of 250) being cost effective at both thresholds was about 1%, and the probabilities of the other tests and risk scores was less than 1%.

Alternative scenario analyses

4.39 The EAG did several scenario analyses to test assumptions made about parameter values used in the base-case model analysis. Use of the ADNEX model remained cost effective in most scenario analysis. However, in some scenarios Simple Rules (inconclusive results assumed to be malignant) was cost effective. These included when a disutility (the value of which was arbitrary) was applied for people with a false positive diagnosis for 1 year and when the benefit of treatment in specialist care was reduced.

4.40 In a scenario analysis in which a higher cost of surgery done by a specialist MDT was used, RMI I (threshold of 250) was cost effective at a maximum acceptable ICER of £20,000 per QALY gained and Simple Rules was cost effective at a maximum acceptable ICER of £30,000 per QALY gained. In this scenario, an additional cost of £2,500 was added to the average cost of surgery done by a specialist MDT, to reflect expert opinion that some patients referred to a specialist MDT will have extensive surgery for ovarian cancer.

Subgroup analyses

4.41 Results from subgroup analyses were similar to the base-case analyses when the starting age of the cohort was 50 years and also when only early stage cancer was considered. However, when the analysis was run for advanced stage cancer, Simple Rules (rather than ADNEX) was cost effective at maximum acceptable ICERs of £20,000 and £30,000 per QALY gained. No changes to sensitivity or specificity values for tests were made in these subgroup analyses (because of a lack of data on test performance in these populations).

4.42 The EAG also did subgroup analyses for populations who were pre- and postmenopausal. Sensitivity and specificity estimates for tests or risk scores in these subgroups were taken from the clinical-effectiveness review; but relatively few studies were available to inform these estimates. A different starting age of the cohort and prevalence of malignancy (compared with the base-case analysis) was also used for these subgroups. The ADNEX model was cost effective at thresholds of £20,000 and £30,000 per QALY gained for both these subgroup analyses.