The diagnostics advisory committee (section 9) considered evidence from a number of sources (section 10).
The assessment was performed by an external assessment group and consisted of a systematic review of the evidence on test performance and clinical effectiveness data for the RD‑100i OSNA system and the Metasin test compared with postoperative histopathology.
The outcome measures relevant to test performance included diagnostic test accuracy, test failure rate, discordant test results, and time to test result.
The outcome measures relevant to clinical effectiveness included: patient anxiety associated with the waiting time for results and with not knowing what the extent of surgery would be before the operation; number of repeat operations (excluding those for re-excision of positive margins); time to start and nature of adjuvant therapy; morbidity and mortality from biopsies, axillary dissections, first and second operations and treatment of cancer; and adverse events from false test results, including patient distress and sequelae.
No study was excluded on the basis of intervention, population, comparator or outcome, provided it appeared relevant to the scope of the evaluation.
The external assessment group included 16 studies in the systematic review that investigated the performance of the RD‑100i OSNA system in detecting metastases in the sentinel or axillary lymph nodes, although 2 of these studies reported the same trial. Fourteen of these 16 studies reported test accuracy as an outcome and of these 14 studies, 2 also reported time to analysis. An additional observational study reported time to analysis as an outcome alone. One other study reported time in operating theatre, days in hospital, costs and postoperative complications. No data were found for the clinical outcomes of patient anxiety and number of repeat operations.
Eleven of the 16 studies were single-gate studies in which people with unknown disease status were assessed using both the intraoperative test and the reference standard (histopathology) to compare the results of the 2 tests and confirm diagnosis. The remaining studies comprised 4 cohort studies and 1 observational study. In the cohort studies, different patient samples were used for each test, which enabled whole-node analysis with the RD‑100i OSNA system.
There was heterogeneity across studies in their definitions of histopathology. Three studies reported using five-level histopathological analysis to detect metastases in the node. Two other studies used five-level histopathological analysis during the validation phase of the study and one-level histopathological analysis during routine use. Five studies reported using three-level histopathological analysis and 1 other study reported using one-level histopathological analysis. The level of histopathological analysis used in the remaining studies is unclear. These varying levels of histopathological analyses may impact on the accuracy of histopathology because five-level analysis may be more likely to detect micrometastases than one-level analysis. In addition, depending on the level of analysis used some studies did not reflect current NHS practice for histopathological analysis.
The external assessment group found that, in all of the studies, there was a lack of detail on patient recruitment and patient characteristics so the risk of bias in the studies is unclear. Spectrum bias may arise if the severity of the cancer is greater in one study population than another because it is more likely that a test will detect metastases in people with severe disease and it will therefore appear to have a higher sensitivity. In addition, study populations may vary depending on the upstream diagnostic pathway because some clinics may be better at detecting metastases with fine needle aspiration cytology so sentinel lymph node biopsy would not be needed in some cases. This could result in a patient population with higher levels of micrometastases than macrometastases receiving sentinel lymph node biopsy and so the sensitivity of the intraoperative test or histopathology may appear lower in this population. There was also a lack of information on sampling methods, so there was no evidence of sample replicates and reproducibility for molecular analysis.
The assessment of test accuracy for the RD‑100i OSNA system was hindered by tissue allocation bias and by comparison with an inconsistent reference standard (different levels of histopathology). In some studies, discordant samples were further analysed (by extensive histopathology, molecular analysis or Western blotting) and attributed to tissue allocation bias. In these cases, the test accuracy analyses could be adjusted by excluding the data from the discordant samples. No adjustment could be made for the varying levels of histopathology used across the studies.
The range of estimates for sensitivity and specificity by patient before adjustment for tissue allocation bias from the studies were 77.8% to 80.0% and 88.0% to 97.2% respectively. The range of estimates for sensitivity and specificity by patient after adjustment for tissue allocation bias from the studies were 89.8% to 100% and 93.3% to 97.2% respectively.
The external assessment group performed a meta-analysis (bivariate method) of diagnostic test accuracy from studies that reported the numbers of true positives, true negatives, false negatives and false positives in the text (or sufficient data for these test statistics to be calculated). Five studies were included that did not adjust for tissue allocation bias and 3 studies were included that did adjust for tissue allocation bias. The sensitivity and specificity by patient without adjustment for tissue allocation bias were 84.5% (95% confidence interval [CI] 74.7% to 91.0%) and 91.8% (95% CI 87.8% to 94.6%) respectively. The sensitivity and specificity by patient with adjustment for tissue allocation bias were 91.3% (95% CI 83.6% to 95.6%) and 94.2% (95% CI 91.2% to 96.2%) respectively.
Four studies reported the time to analysis by the RD‑100i OSNA system. The estimates for time to analysis from the studies ranged from less than 30 minutes to 39.6 minutes for 1 node and increased by 5 to 10 minutes per additional node analysed. One study reported that the longest and most variable time period corresponded to transporting the node from the operating room to the pathology department. The least variable time period corresponded to the homogenisation of tissue, preparation of the diluted sample and gene amplification by RT‑LAMP.
One other study compared the number of postoperative complications between the use of histopathology and the use of RD‑100i OSNA to analyse lymph node samples. The aim of this study was to analyse the economic costs of intraoperative testing with the RD‑100i OSNA system compared with postoperative histopathology. Overall, the patients having intraoperative lymph node testing using the RD‑100i OSNA system experienced fewer postoperative complications than patients having postoperative histopathology analysis, although the only major complication reported occurred in the OSNA group (no further details of the major complication were reported). This study was conducted in Spain and the assessment group stated that it was uncertain to what extent the study findings might be generalisable to the UK owing to possible differences in clinical practice.
Two draft unpublished non-peer reviewed studies that investigated the performance of the Metasin test in detecting metastases in the lymph nodes of patients with breast cancer were included in the systematic review. The results of 1 of the studies are considered academic in confidence.
The other study, by Sundaresan et al. (unpublished), was designed as a single-gate study comparing the Metasin test with histopathology, in which people were assessed by both methods. It reported test accuracy and time to analysis as outcomes. It used three-level histopathological analysis to detect metastases in the nodes but did not report details of patient recruitment or patient characteristics, so the risk of bias is unclear. There was also a lack of information about sample replicates and reproducibility for molecular analysis, and the external assessment group did not consider the study to meet the STAndards for the Reporting of Diagnostic accuracy studies (STARD) criteria.
As with the studies assessing the RD‑100i OSNA system, one of the main issues with assessing the accuracy of the Metasin test was tissue allocation bias. No discordant analyses were performed in the study but discordant results were observed in 56 out of 1,265 patients (4.4%): 36 patients had a positive Metasin test and negative histopathology, and 20 patients had a negative Metasin test and positive histopathology. The authors considered that tissue allocation bias was responsible for the discordant results, although no evidence or analysis was presented in the study to support this.
The estimates for test accuracy by patient without adjustment for tissue allocation bias in Sundaresan et al. were 92% sensitivity (95% CI 89% to 95%) and 97% specificity (95% CI 95% to 97%). No meta-analysis was performed by the external assessment group for the 2 studies assessing the Metasin test because at least 4 studies are needed to use the bivariate method of meta-analysis. The accuracy values from Sundaresan et al. were used in the cost-effectiveness analyses.
The external assessment group identified 2 studies that were considered relevant for the systematic review on the cost effectiveness of intraoperative tests for the detection of sentinel lymph node metastases. Both studies were single-centre observational studies that compared an intraoperative test with histopathology for assessing sentinel lymph node biopsy. One study was based in the UK and assessed the GeneSearch BLN assay and the other study was conducted in Spain and evaluated the RD‑100i OSNA system. Both studies found their respective intraoperative tests to be cost effective compared with histopathology, with both assays being cost saving while reducing theatre time and length of hospital stay. Neither study considered outcomes beyond the diagnostic phase. The UK study provided evidence on resource use and costs of intraoperative testing in the UK but evaluated the GeneSearch BLN assay, which has been withdrawn from the market. The Metasin test uses the same markers as the GeneSearch BLN assay, CK19 and Mammaglobin, but different primer-probe combinations, so it is expected to perform differently from the GeneSearch BLN assay. Therefore, this UK study is not directly relevant to this evaluation. The study conducted in Spain also provided evidence on resource use and costs but was limited in the extent to which it was generalisable to the UK. Therefore, the external assessment group did not consider the study directly relevant to this evaluation.
The external assessment group performed an economic analysis to assess the cost effectiveness of using intraoperative tests to detect sentinel node metastases compared with using histopathology. The economic model was divided into 2 separate sections (diagnostic and management) to encompass both the short-term and long-term outcomes of intraoperative testing. The costs were evaluated from the perspective of the NHS and personal social services. Outcomes were expressed as quality-adjusted life years (QALYs). Both costs and outcomes were discounted using a 3.5% annual discount rate.
The external assessment group developed a decision tree to model the short-term diagnostic outcomes outlined in the decision problem. People enter the model as patients who have sentinel lymph node biopsy performed during their initial tumour removal. The model then splits into 3 different diagnostic strategies: postoperative histopathology (current practice) alone, intraoperative testing alone and intraoperative testing combined with postoperative histopathology confirmation. In the model, patients who are diagnosed with sentinel lymph node metastases receive axillary lymph node dissection, either during the same operation as their sentinel lymph node biopsy if intraoperative testing is used or during a second operation if postoperative histopathology is used.
Once the diagnostic subgroups have been identified (true positive sentinel lymph node, false positive sentinel lymph node, true negative sentinel lymph node and false negative sentinel lymph node), the model moves into the management pathway and the subgroups are separated based on whether or not patients receive an axillary lymph node dissection. This section of the model calculates the long-term outcomes for each subgroup and at this point, a discrete event simulation model, previously developed at the University of Sheffield (ScHARR-TAG), was used to model the natural disease history of the patients once their outcome from the diagnostic decision tree had been determined. In the model, after surgery, patients receive adjuvant therapy comprising chemotherapy and hormonal therapy (where appropriate) for patients diagnosed with metastases and hormonal therapy alone (where appropriate) for patients diagnosed without. After adjuvant therapy, patients can move into a disease-free state, or experience locoregional or metastatic relapse. Patients can also move between these states.
The meta-analysed accuracy values without adjustment for tissue allocation bias for the RD‑100i OSNA system and the accuracy values from one of the unpublished papers for the Metasin test were used in the base case. The node-positive prevalence was set at 20% in the base case, in line with the studies in the clinical systematic review.
Unit costs for the intraoperative tests were taken from the sponsors of the technologies and the cost of histopathology was based on data provided by the NHS Technology Adoption Centre. The unit cost of the RD‑100i OSNA system was £350 and the unit cost of histopathology was £472. The surgery costs were mainly based on NHS reference costs and the costs of short-term adverse events were taken from Jeruss et al. (2006). The costs associated with lymphoedema were obtained from the Sheffield Lymphoedema Service and the length of additional hospital stay was calculated from a study by the York Health Economics Consortium. The costs of additional time in surgery were estimated from a study by Ng et al. (2011) and from the report by York Health Economics Consortium. All costs were updated to 2010 levels.
The QALY decrement associated with a 2‑week wait for histopathology results was calculated by the external assessment group to be 0.019 (undiscounted). For patients having a separate second operation, the disutility was estimated as 0.03.
The external assessment group considered the results of the cost-effectiveness analyses for the Metasin test to be illustrative because of the high levels of uncertainty associated with the unpublished evidence base relating to the diagnostic accuracy of the test.
The cost-effectiveness analyses of the short-term outcomes examined the diagnostic accuracy of the intraoperative tests compared with postoperative histopathology, and the disutility of waiting for histopathology results and having a second operation. For strategies that did not involve histopathology, the utility was 1 because there was no wait for test results or any second operations directly resulting from the intraoperative test. Only short-term QALY gains were included in these analyses.
Using the NHS reference costs in the short-term model, whole-node OSNA analysis and half-node OSNA analysis dominated histopathology analysis because they were less costly and more effective. Whole-node OSNA analysis also dominated half-node OSNA analysis. It was estimated that 4.1% of the 76.5% of patients who received a negative test result from half-node OSNA analysis would end up with a positive result while waiting for confirmation by histopathology analysis, compared with 20% of patients who would receive a positive result using postoperative histopathology analysis alone.
In the short-term model, whole-node and half-node Metasin analyses dominated histopathology analysis because they were less costly and more effective. Whole-node Metasin also dominated half-node Metasin analysis. It was estimated that, of the 78.5% of patients who received a negative result by half-node Metasin analysis, 1.9% would receive a positive result while waiting for confirmation by histopathology analysis, compared with 20% of patients who would receive a positive result using postoperative histopathology analysis alone.
The cost-effectiveness analyses of the long-term outcomes examined all the costs and benefits from accurate diagnosis through to improved patient management. In these analyses, the diagnostic strategies were ordered by the number of QALYs associated with them, with whole-node OSNA analysis producing the least QALYs (9.22) and postoperative histopathology producing the most QALYs (9.32). The QALY difference is equal to 0.1 (that is, equivalent to 5 weeks of full-health life) and this difference occurs because the higher accuracy of histopathology assumed in the model leads to more correct diagnoses and appropriate subsequent treatment.
Using the NHS reference costs in the long-term model, the incremental cost-effectiveness ratio (ICER) was £4,324 saved per QALY lost for whole-node OSNA analysis compared with histopathology analysis and £24,863 saved per QALY lost for whole-node Metasin analysis compared with histopathology analysis. The ICERs for whole-node analysis compared with histopathology analysis suggest that the intraoperative testing strategies save money but that there is a loss of approximately 0.1 QALY, compared with histopathology analysis.
In the modelling, histopathology analysis was assumed to be the 'gold standard' and was given an accuracy of 100% sensitivity and 100% specificity. The level of uncertainty in this assumption is unclear and the estimated ICERs may change depending on the assumed absolute accuracy of histopathology.
The sensitivity and specificity of OSNA analysis were changed to use values from studies that had been adjusted for tissue allocation bias (Frere Belda et al. 2012, Snook et al. 2011 and Khaddage et al. 2011). Using NHS reference costs, the ICER was £9,493 saved per QALY lost for whole-node OSNA analysis compared with histopathology analysis using accuracy values from the Frere Belda et al. study (91.4% sensitivity and 93.3% specificity) and £8,840 saved per QALY lost for whole-node OSNA analysis compared with histopathology analysis using values from the Snook et al. study (89.8% sensitivity and 94.5% specificity). However, using the higher accuracy values from the Khaddage et al. study (100% sensitivity and 97.2% specificity) resulted in ICERs for whole-node OSNA analysis that dominated both half-node OSNA analysis and histopathology analysis.
The change in ICERs when accuracy values adjusted for tissue allocation bias were used showed that test accuracy has a direct impact on the cost effectiveness of the tests. Threshold analysis was used to investigate sensitivity by increasing sensitivity over a range of 70% to 100% while specificity was held constant. The opposite was also performed to investigate specificity. These analyses were conducted on the results for the whole-node OSNA analysis. Short-term utility results were not reported as the utility of OSNA was not affected by the accuracy of the test.
The results of the threshold analysis for the long-term results showed that, when the sensitivity of OSNA increased (and specificity was kept at the 91.8% base-case value), the saving per QALY lost increased when comparing whole-node OSNA with histopathology. When comparing OSNA with histopathology, the ICERs for OSNA ranged from £2,119 saved per QALY lost when OSNA had a sensitivity of 70% to £14,193 saved per QALY lost when OSNA had 95% sensitivity. At 100% sensitivity, OSNA dominated histopathology, having more QALYs gained and lower costs.
For specificity, the long-term cost of OSNA decreased and the QALY gain increased as the specificity increased. At a specificity of 70%, whole-node OSNA analysis was dominated by histopathology analysis because it was more expensive and had fewer QALYs. The largest ICER for OSNA was £8,430 saved per QALY lost compared with histopathology when whole-node OSNA analysis had 100% specificity (and sensitivity was kept at the base-case value of 94.5%).
Overall, the threshold analyses suggested that, if the true values of sensitivity and specificity for whole-node OSNA analysis lie within the range of 90% to 100%, the cost effectiveness of whole-node OSNA analysis may increase. The results also imply that changes to specificity may have more of an impact in the short-term than the long-term, but that changes to sensitivity may have a much greater impact on the long-term cost effectiveness.
Sensitivity analysis was also conducted on the effect of prevalence of sentinel lymph node metastases in the patient population. When the prevalence was reduced to 10%, histopathology analysis dominated half-node OSNA analysis and had an ICER of £2,626 per QALY gained compared with whole-node OSNA analysis. When the prevalence was increased to 40%, half-node OSNA analysis dominated histopathology analysis and had an ICER of £2,208 per QALY gained compared with whole-node OSNA analysis.
Changing individual costs and utility parameter values in the short-term or long-term sections of the model had very little impact on the overall cost-effectiveness results. This highlighted the importance of the diagnostic accuracy of the tests because this was the most influential parameter.