Evaluation of an Artificial Intelligence Model for Detection of Pneumothorax and Tension Pneumothorax in Chest Radiographs

James M Hillis, Bernardo C Bizzo, Sarah Mercaldo, John K Chin, Isabella Newbury-Chaet, Subba R Digumarthy, Matthew D Gilman, Victorine V Muse, Georgie Bottrell, Jarrel C Y Seah, Catherine M Jones, Mannudeep K Kalra, Keith J Dreyer, James M Hillis, Bernardo C Bizzo, Sarah Mercaldo, John K Chin, Isabella Newbury-Chaet, Subba R Digumarthy, Matthew D Gilman, Victorine V Muse, Georgie Bottrell, Jarrel C Y Seah, Catherine M Jones, Mannudeep K Kalra, Keith J Dreyer

Abstract

Importance: Early detection of pneumothorax, most often via chest radiography, can help determine need for emergent clinical intervention. The ability to accurately detect and rapidly triage pneumothorax with an artificial intelligence (AI) model could assist with earlier identification and improve care.

Objective: To compare the accuracy of an AI model vs consensus thoracic radiologist interpretations in detecting any pneumothorax (incorporating both nontension and tension pneumothorax) and tension pneumothorax.

Design, setting, and participants: This diagnostic study was a retrospective standalone performance assessment using a data set of 1000 chest radiographs captured between June 1, 2015, and May 31, 2021. The radiographs were obtained from patients aged at least 18 years at 4 hospitals in the Mass General Brigham hospital network in the United States. Included radiographs were selected using 2 strategies from all chest radiography performed at the hospitals, including inpatient and outpatient. The first strategy identified consecutive radiographs with pneumothorax through a manual review of radiology reports, and the second strategy identified consecutive radiographs with tension pneumothorax using natural language processing. For both strategies, negative radiographs were selected by taking the next negative radiograph acquired from the same radiography machine as each positive radiograph. The final data set was an amalgamation of these processes. Each radiograph was interpreted independently by up to 3 radiologists to establish consensus ground-truth interpretations. Each radiograph was then interpreted by the AI model for the presence of pneumothorax and tension pneumothorax. This study was conducted between July and October 2021, with the primary analysis performed between October and November 2021.

Main outcomes and measures: The primary end points were the areas under the receiver operating characteristic curves (AUCs) for the detection of pneumothorax and tension pneumothorax. The secondary end points were the sensitivities and specificities for the detection of pneumothorax and tension pneumothorax.

Results: The final analysis included radiographs from 985 patients (mean [SD] age, 60.8 [19.0] years; 436 [44.3%] female patients), including 307 patients with nontension pneumothorax, 128 patients with tension pneumothorax, and 550 patients without pneumothorax. The AI model detected any pneumothorax with an AUC of 0.979 (95% CI, 0.970-0.987), sensitivity of 94.3% (95% CI, 92.0%-96.3%), and specificity of 92.0% (95% CI, 89.6%-94.2%) and tension pneumothorax with an AUC of 0.987 (95% CI, 0.980-0.992), sensitivity of 94.5% (95% CI, 90.6%-97.7%), and specificity of 95.3% (95% CI, 93.9%-96.6%).

Conclusions and relevance: These findings suggest that the assessed AI model accurately detected pneumothorax and tension pneumothorax in this chest radiograph data set. The model's use in the clinical workflow could lead to earlier identification and improved care for patients with pneumothorax.

Conflict of interest statement

Conflict of Interest Disclosures: Dr Digumarthy reported receiving grants from GE, Vuno, and QureAI; personal fees from Siemens and Elsevier; and serving as a consultant for Merck, Pfizer, Bristol Myers Squibb, Novartis, Roche, Polaris, Cascadian, AbbVie, Gradalis, Bayer, Zai Laboratories, Biengen, Riverain, Resonance, Biengen, Janssen outside the submitted work. Dr Bottrell reported owning stock in Annalise AI outside the submitted work. Dr Seah reported receiving personal fees from Harrison.ai and having a patent for Annalise CXR Enterprise pending during the conduct of the study. Dr Kalra reported receiving grants from Coreline, Riverain Tech, and Siemens. No other disclosures were reported.

Figures

Figure.. Areas Under the Receiver Operating Characteristic…
Figure.. Areas Under the Receiver Operating Characteristic Curve (AUC) for Pneumothorax and Tension Pneumothorax Detection and Example Images
The shaded region reflects the bootstrapped 95% CI. The selected point on each graph reflects the model operating point. All 3 radiographs with pneumothorax were correctly interpreted by the artificial intelligence model to be negative for tension pneumothorax and all 3 radiographs with tension pneumothorax were correctly interpreted by the AI model to be positive for pneumothorax. Solid boxes cover text annotations on images.

References

    1. Sahn SA, Heffner JE. Spontaneous pneumothorax. N Engl J Med. 2000;342(12):868-874. doi:10.1056/NEJM200003233421207
    1. Ebrahimian S, Kalra MK, Agarwal S, et al. . FDA-regulated AI algorithms: trends, strengths, and gaps of validation studies. Acad Radiol. 2022;29(4):559-566. doi:10.1016/j.acra.2021.09.002
    1. Weisberg EM, Chu LC, Fishman EK. The first use of artificial intelligence (AI) in the ER: triage not diagnosis. Emerg Radiol. 2020;27(4):361-366. doi:10.1007/s10140-020-01773-6
    1. Kitamura G, Deible C. Retraining an open-source pneumothorax detecting machine learning algorithm for improved performance to medical images. Clin Imaging. 2020;61:15-19. doi:10.1016/j.clinimag.2020.01.008
    1. Wang H, Gu H, Qin P, Wang J. CheXLocNet: automatic localization of pneumothorax in chest radiographs using deep convolutional neural networks. PLoS One. 2020;15(11):e0242013. doi:10.1371/journal.pone.0242013
    1. Taylor AG, Mielke C, Mongan J. Automated detection of moderate and large pneumothorax on frontal chest X-rays using deep convolutional neural networks: a retrospective study. PLoS Med. 2018;15(11):e1002697. doi:10.1371/journal.pmed.1002697
    1. Cho Y, Kim JS, Lim TH, Lee I, Choi J. Detection of the location of pneumothorax in chest X-rays using small artificial neural networks and a simple training process. Sci Rep. 2021;11(1):13054. doi:10.1038/s41598-021-92523-2
    1. Feng S, Liu Q, Patel A, et al. . Automated pneumothorax triaging in chest X-rays in the New Zealand population using deep-learning algorithms. J Med Imaging Radiat Oncol. 2022. doi:10.1111/1754-9485.13393
    1. Park S, Lee SM, Kim N, et al. . Application of deep learning-based computer-aided detection system: detecting pneumothorax on chest radiograph after biopsy. Eur Radiol. 2019;29(10):5341-5348. doi:10.1007/s00330-019-06130-x
    1. US Food and Drug Administration . HealthPNX approval letter (K190362). Accessed June 19 2022.
    1. US Food and Drug Administration . Critical Care Suite approval letter (K183182). Accessed June 19 2022.
    1. US Food and Drug Administration . red dot Device approval letter (K191556). Accessed June 19 2022.
    1. US Food and Drug Administration . AIMI-Triage CXR PTX approval letter (K193300). Accessed June 19 2022.
    1. US Food and Drug Administration . Lunit INSIGHT CXR Triage approval letter (K211733). Accessed June 19 2022.
    1. US Food and Drug Administration . Annalise Enterprise CXR Triage Pneumothorax approval letter (K213941). Accessed June 19 2022.
    1. US Food and Drug Administration . ClearRead Xray Pneumothorax approval letter (K213566). Accessed June 19 2022.
    1. US Food and Drug Administration . BriefCase approval letter (K214043). Accessed June 19 2022.
    1. US Food and Drug Administration . Product classification: radiological computer-assisted prioritization software for lesions. Accessed June 19 2022.
    1. Seah JCY, Tang CHM, Buchlak QD, et al. . Effect of a comprehensive deep-learning model on the accuracy of chest x-ray interpretation by radiologists: a retrospective, multireader multicase study. Lancet Digit Health. 2021;3(8):e496-e506. doi:10.1016/S2589-7500(21)00106-0

Source: PubMed

Sponsors and Collaborators

Medical Conditions

Drug Interventions

3
Subscribe