Interreader scoring variability in an observer study using dual-modality imaging for breast cancer detection in women with dense breasts

Abstract

Rationale and objectives: To evaluate variability in the clinical assessment of breast images, we evaluated scoring behavior of radiologists in a retrospective reader study combining x-ray mammography (XRM) and three-dimensional automated breast ultrasound (ABUS) for breast cancer detection in women with dense breasts.

Methods: The study involved 17 breast radiologists in a sequential study design with readers first interpreting XRM-alone followed by an interpretation of combined XRM + ABUS. Each interpretation included a forced Breast Imaging Reporting and Data System scale and a likelihood that the woman had breast cancer. The analysis included 164 asymptomatic patients, including 31 breast cancer patients, with dense breasts and a negative screening XRM. Of interest were interreader scoring variability for XRM-alone, XRM + ABUS, and the sequential effect. In addition, a simulated double reading by pairs of readers of XRM + ABUS was investigated. Performance analysis included receiver operating characteristic analysis, percentile analysis, and κ statistics. Bootstrapping was used to determine statistical significance.

Results: The median change in area under the receiver operating characteristic curve after ABUS interpretation was 0.12 (range 0.04-0.19). Reader agreement was fair with the median interreader κ being 0.26 (0.05-0.48) for XRM-alone and 0.34 (0.11-0.55) for XRM + ABUS (95% confidence interval for the difference in κ, 0.06-0.11). Simulated double reading of XRM + ABUS demonstrated tradeoffs in sensitivity and specificity, but conservative simulated double reading resulted in a significant improvement in both sensitivity (16.7%) and specificity (7.6%) with respect to XRM-alone.

Conclusion: A modest, but statistically significant, increase in interreader agreement was observed after interpretation of ABUS.

Conflict of interest statement

Conflict of interest statement: M.L.G. is a stockholder in R2 Technology/Hologic, co-founder and equity holder of Quantitative Insights, Inc., and receives royalties from Hologic, GE Medical Systems, MEDIAN Technologies, Riverain Medical, Mitsubishi and Toshiba. K.D. received royalties from Hologic. It is the University of Chicago Conflict of Interest Policy that investigators disclose publicly actual or potential significant financial interest that would reasonably appear to be directly and significantly affected by the research activities.

Copyright © 2013 AUR. Published by Elsevier Inc. All rights reserved.

Figures

Figure 1

Receiver operating characteristic (ROC) performance assessment of the 17 readers for the conditions x-ray mammography (XRM)-alone and XRM + three-dimensional automated breast ultrasound (ABUS), as (a) histogram of the area under the ROC curve (AUC) values, (b) the AUC values per reader, and (c) histogram of the change in AUC, Δ(AUC). When referred to by number, readers are ordered throughout this report by increasing AUCXRM value. The error bars are ± standard error as given by the proper binormal ROC model.

Figure 2

The change in reader-assigned likelihood of malignancy, Δ(LOM), between the x-ray mammography (XRM)-alone and XRM + three-dimensional automated breast ultrasound (ABUS) conditions for the (a) actually normal cases and (b) actually cancerous cases. In all box plots in this report, the bottom and top of each box denote the 25th and 75th percentiles, respectively, while the horizontal line within denotes the median value. Whiskers extend to mark the range in values not considered outliers, while individual outliers are marked with a “+.”

Figure 3

The impact of three-dimensional automated breast ultrasound (ABUS) on the identification of breast cancer cases as (a) box plot of the change in the number of cases assigned a Breast Imaging Reporting and Data System (BI-RADS) category 4a or higher by a reader, (b) the change in number of cases assigned a BI-RADS 4a or higher per reader, and (c) box plot of the change in the number of readers assessing a case as BI-RADS 4a or higher.

Figure 4

The impact of three-dimensional automated breast ultrasound (ABUS) on the identification of breast cancer illustrated by color-coded Breast Imaging Reporting and Data System (BI-RADS) assessment categories for all cases and all readers. BI-RADS categories vary as indicated from blue (category 1), through shades of yellow and orange, to red (category 5). Cases are divided into actually normal cases and actually cancerous cases and then ordered by increasing change in reader-assigned likelihood of malignancy (LOM). Readers are again ordered by increasing AUCXRM.

Figure 5

Interreader agreement as indicated by the interreader κ. The box plots are of the NR(NR − 1)/2, with NR = 17 readers, κ values for the x-ray mammography (XRM)-alone and XRM + three-dimensional automated breast ultrasound (ABUS) conditions for (a) all cases, (b) the normal cases, and (c) the cancer cases.