Assessment of the effect of a comprehensive chest radiograph deep learning model on radiologist reports and patient outcomes: a real-world observational study

Catherine M Jones, Luke Danaher, Michael R Milne, Cyril Tang, Jarrel Seah, Luke Oakden-Rayner, Andrew Johnson, Quinlan D Buchlak, Nazanin Esmaili, Catherine M Jones, Luke Danaher, Michael R Milne, Cyril Tang, Jarrel Seah, Luke Oakden-Rayner, Andrew Johnson, Quinlan D Buchlak, Nazanin Esmaili

Abstract

Objectives: Artificial intelligence (AI) algorithms have been developed to detect imaging features on chest X-ray (CXR) with a comprehensive AI model capable of detecting 124 CXR findings being recently developed. The aim of this study was to evaluate the real-world usefulness of the model as a diagnostic assistance device for radiologists.

Design: This prospective real-world multicentre study involved a group of radiologists using the model in their daily reporting workflow to report consecutive CXRs and recording their feedback on level of agreement with the model findings and whether this significantly affected their reporting.

Setting: The study took place at radiology clinics and hospitals within a large radiology network in Australia between November and December 2020.

Participants: Eleven consultant diagnostic radiologists of varying levels of experience participated in this study.

Primary and secondary outcome measures: Proportion of CXR cases where use of the AI model led to significant material changes to the radiologist report, to patient management, or to imaging recommendations. Additionally, level of agreement between radiologists and the model findings, and radiologist attitudes towards the model were assessed.

Results: Of 2972 cases reviewed with the model, 92 cases (3.1%) had significant report changes, 43 cases (1.4%) had changed patient management and 29 cases (1.0%) had further imaging recommendations. In terms of agreement with the model, 2569 cases showed complete agreement (86.5%). 390 (13%) cases had one or more findings rejected by the radiologist. There were 16 findings across 13 cases (0.5%) deemed to be missed by the model. Nine out of 10 radiologists felt their accuracy was improved with the model and were more positive towards AI poststudy.

Conclusions: Use of an AI model in a real-world reporting environment significantly improved radiologist reporting and showed good agreement with radiologists, highlighting the potential for AI diagnostic support to improve clinical practice.

Keywords: deep learning; machine learning chest X-ray.

Conflict of interest statement

Competing interests: CMJ is a radiologist employed by the radiology practice and a clinical consultant for Annalise-AI. LD, LO-R and NE are independent of Annalise-AI and have no interests to declare. MRM, JS, CT, AJ and QDB are employed by or seconded to Annalise-AI. Study conception, study design, ethics approval and data security were conducted independent of Annalise-AI.

© Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Figures

Figure 1
Figure 1
Flow diagram illustrating the AI-assisted reporting process described in this study. AI, artificial intelligence; CXR, chest X-ray; PACS, picture archiving and communication system; RIS, radiological information system.
Figure 2
Figure 2
Example of the modified user interface used by the participating radiologists in this study. The red box highlights the feedback options added to the interface for this study.
Figure 3
Figure 3
Counts of numbers of critical findings for the cases seen by the radiologist, defined as the number of critical findings agreed + the number of critical findings added. The number of cases which returned zero findings was 1513.
Figure 4
Figure 4
Diverging stacked bar chart depicting the first set of radiologist survey responses. CXR, chest X-ray.
Figure 5
Figure 5
Diverging stacked bar chart visualising the second set of survey responses of the radiologists. AI, artificial intelligence; CXR, chest X-ray.

References

    1. Gillies RJ, Kinahan PE, Hricak H. Radiomics: images are more than pictures, they are data. Radiology 2016;278:563–77. 10.1148/radiol.2015151169
    1. Greene R, Williams FH. Francis H. Williams, MD: father of chest radiology in North America. Radiographics 1991;11:325–32. 10.1148/radiographics.11.2.2028067
    1. Schaefer-Prokop C, Neitzel U, Venema HW, et al. . Digital chest radiography: an update on modern technology, dose containment and control of image quality. Eur Radiol 2008;18:1818–30. 10.1007/s00330-008-0948-3
    1. Lee CS, Nagy PG, Weaver SJ, et al. . Cognitive and system factors contributing to diagnostic errors in radiology. AJR Am J Roentgenol 2013;201:611–7. 10.2214/AJR.12.10375
    1. Chotas HG, Ravin CE. Chest radiography: estimated lung volume and projected area obscured by the heart, mediastinum, and diaphragm. Radiology 1994;193:403–4. 10.1148/radiology.193.2.7972752
    1. Berlin L. Accuracy of diagnostic procedures: has it improved over the past five decades? AJR Am J Roentgenol 2007;188:1173–8. 10.2214/AJR.06.1270
    1. Zaorsky NG, Churilla TM, Egleston BL, et al. . Causes of death among cancer patients. Ann Oncol 2017;28:400–7. 10.1093/annonc/mdw604
    1. del Ciello A, Franchi P, Contegiacomo A. Missed lung cancer: when, where, and why? Diagn Interv Radiol 2017;23:118–26.
    1. Fazal MI, Patel ME, Tye J, et al. . The past, present and future role of artificial intelligence in imaging. Eur J Radiol 2018;105:246–50. 10.1016/j.ejrad.2018.06.020
    1. Jordan MI, Mitchell TM. Machine learning: trends, perspectives, and prospects. Science 2015;349:255–60. 10.1126/science.aaa8415
    1. Hosny A, Parmar C, Quackenbush J, et al. . Artificial intelligence in radiology. Nat Rev Cancer 2018;18:500–10. 10.1038/s41568-018-0016-5
    1. Erickson BJ, Korfiatis P, Akkus Z, et al. . Machine learning for medical imaging. Radiographics 2017;37:505–15. 10.1148/rg.2017160130
    1. Esteva A, Chou K, Yeung S, et al. . Deep learning-enabled medical computer vision. NPJ Digit Med 2021;4:1–9. 10.1038/s41746-020-00376-2
    1. Jang S, Song H, Shin YJ, et al. . Deep Learning-based automatic detection algorithm for reducing overlooked lung cancers on chest radiographs. Radiology 2020;296:652–61. 10.1148/radiol.2020200165
    1. Liang C-H, Liu Y-C, Wu M-T, et al. . Identifying pulmonary nodules or masses on chest radiography using deep learning: external validation and strategies to improve clinical practice. Clin Radiol 2020;75:38–45. 10.1016/j.crad.2019.08.005
    1. Hurt B, Kligerman S, Hsiao A. Deep learning localization of pneumonia: 2019 coronavirus (COVID-19) outbreak. J Thorac Imaging 2020;35:W87–9. 10.1097/RTI.0000000000000512
    1. Kim JY, Choe PG, Oh Y, et al. . The first case of 2019 novel coronavirus pneumonia imported into Korea from Wuhan, China: implication for infection prevention and control measures. J Korean Med Sci 2020;35:e61. 10.3346/jkms.2020.35.e61
    1. PRAS B, Attux R. A deep Convolutional neural network for COVID-19 detection using chest x-rays. Available: [Accessed 23 Mar 2021].
    1. Rueckel J, Trappmann L, Schachtner B, et al. . Impact of confounding thoracic tubes and pleural dehiscence extent on artificial intelligence pneumothorax detection in chest radiographs. Invest Radiol 2020;55:792–8. 10.1097/RLI.0000000000000707
    1. Sze-To A, Wang Z. tCheXNet: Detecting Pneumothorax on Chest X-Ray Images Using Deep Transfer Learning. In: Karray F, Campilho A, Yu A, eds. Image analysis and recognition. Cham: Springer International Publishing, 2019: 325–32.
    1. Hwang EJ, Hong JH, Lee KH, et al. . Deep learning algorithm for surveillance of pneumothorax after lung biopsy: a multicenter diagnostic cohort study. Eur Radiol 2020;30:3660–71. 10.1007/s00330-020-06771-3
    1. Park S, Lee SM, Kim N, et al. . Application of deep learning-based computer-aided detection system: detecting pneumothorax on chest radiograph after biopsy. Eur Radiol 2019;29:5341–8. 10.1007/s00330-019-06130-x
    1. Wang X, Yu J, Zhu Q, et al. . Potential of deep learning in assessing pneumoconiosis depicted on digital chest radiography. Occup Environ Med 2020;77:597–602. 10.1136/oemed-2019-106386
    1. Zhou S, Zhang X, Zhang R,. Identifying cardiomegaly in ChestX-ray8 using transfer learning. Stud Health Technol Inform 2019;264:482–6. 10.3233/SHTI190268
    1. Zou X-L, Ren Y, Feng D-Y, et al. . A promising approach for screening pulmonary hypertension based on frontal chest radiographs using deep learning: a retrospective study. PLoS One 2020;15:e0236378. 10.1371/journal.pone.0236378
    1. Pasa F, Golkov V, Pfeiffer F, et al. . Efficient deep network architectures for fast chest X-ray tuberculosis screening and visualization. Sci Rep 2019;9:6268. 10.1038/s41598-019-42557-4
    1. Nash M, Kadavigere R, Andrade J, et al. . Deep learning, computer-aided radiography reading for tuberculosis: a diagnostic accuracy study from a tertiary hospital in India. Sci Rep 2020;10:210. 10.1038/s41598-019-56589-3
    1. Heo S-J, Kim Y, Yun S, et al. . Deep Learning Algorithms with Demographic Information Help to Detect Tuberculosis in Chest Radiographs in Annual Workers’ Health Examination Data. Int J Environ Res Public Health 2019;16:250. 10.3390/ijerph16020250
    1. Qin ZZ, Sander MS, Rai B, et al. . Using artificial intelligence to read chest radiographs for tuberculosis detection: a multi-site evaluation of the diagnostic accuracy of three deep learning systems. Sci Rep 2019;9:15000. 10.1038/s41598-019-51503-3
    1. Lakhani P, Sundaram B. Deep learning at chest radiography: automated classification of pulmonary tuberculosis by using Convolutional neural networks. Radiology 2017;284:574–82. 10.1148/radiol.2017162326
    1. Seah JCY, Tang CHM, Buchlak QD, et al. . Effect of a comprehensive deep-learning model on the accuracy of chest X-ray interpretation by radiologists: a retrospective, multireader multicase study. Lancet Digit Health 2021;3:e496–506. 10.1016/S2589-7500(21)00106-0
    1. Annalise CXR comprehensive medical imaging AI. . Available: [Accessed 23 Mar 2021].
    1. Tan M, QV L. EfficientNet: rethinking model scaling for Convolutional neural networks. Available: [Accessed 30 Mar 2021].
    1. Ronneberger O, Fischer P, Brox T. U-Net: Convolutional networks for biomedical image segmentation. Available: [Accessed 30 Mar 2021].
    1. Annalise-AI Pty Ltd - Radiology DICOM image processing application software. Available: [Accessed 25 Aug 2021].
    1. Improving diagnostic pathways for patients with suspected lung cancer. Available: [Accessed 31 Aug 2021].
    1. Benjamini Y, Hochberg Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B 1995;57:289–300. 10.1111/j.2517-6161.1995.tb02031.x
    1. Mckinney W. Pandas: a foundational python library for data analysis and statistics. Python High Performance Science Computer 2011.
    1. Harris CR, Millman KJ, van der Walt SJ, et al. . Array programming with NumPy. Nature 2020;585:357–62. 10.1038/s41586-020-2649-2
    1. Jones E, Oliphant T, Peterson P. SciPy: open source scientific tools for python 2001.
    1. Pedregosa F, Varoquaux G, Gramfort A. Scikit-learn: machine learning in python. Journal of Machine Learning Research 2021.
    1. Jolly E. Pymer4: connecting R and python for linear mixed modeling. Journal of Open Source Software 2018;3:862. 10.21105/joss.00862
    1. InSeabold S, Perktold J. Statsmodels: Econometric and statistical modeling with python. Austin, Texas, 2010: 92–6.
    1. Seah J, Tang C, Buchlak QD. Radiologist chest X-ray diagnostic accuracy performance improvements when augmented by a comprehensive deep learning model. The Lancet Digital Health 2021.
    1. Hwang EJ, Park S, Jin K-N, et al. . Development and validation of a deep Learning-Based automated detection algorithm for major thoracic diseases on chest radiographs. JAMA Netw Open 2019;2:e191095. 10.1001/jamanetworkopen.2019.1095
    1. Hwang EJ, Nam JG, Lim WH, et al. . Deep learning for chest radiograph diagnosis in the emergency department. Radiology 2019;293:573–80. 10.1148/radiol.2019191225
    1. Singh R, Kalra MK, Nitiwarangkul C, et al. . Deep learning in chest radiography: detection of findings and presence of change. PLoS One 2018;13:e0204155. 10.1371/journal.pone.0204155
    1. Khan FA, Majidulla A, Tavaziva G, et al. . Chest X-ray analysis with deep learning-based software as a triage test for pulmonary tuberculosis: a prospective study of diagnostic accuracy for culture-confirmed disease. Lancet Digit Health 2020;2:e573–81. 10.1016/S2589-7500(20)30221-1
    1. Dellios N, Teichgraeber U, Chelaru R, et al. . Computer-Aided detection fidelity of pulmonary nodules in chest radiograph. J Clin Imaging Sci 2017;7:8. 10.4103/jcis.JCIS_75_16
    1. Sim Y, Chung MJ, Kotter E. Deep Convolutional neural Network–based software improves radiologist detection of malignant lung nodules on chest radiographs. Radiology. [Epub ahead of print: 12 Nov 2019]. 10.1148/radiol.2019182465
    1. Waymel Q, Badr S, Demondion X, et al. . Impact of the rise of artificial intelligence in radiology: what do radiologists think? Diagn Interv Imaging 2019;100:327–36. 10.1016/j.diii.2019.03.015
    1. Collado-Mesa F, Alvarez E, Arheart K. The role of artificial intelligence in diagnostic radiology: a survey at a single radiology residency training program. J Am Coll Radiol 2018;15:1753–7. 10.1016/j.jacr.2017.12.021

Source: PubMed

Спонсоры и соавторы

Заболевания

Медикаментозные вмешательства

Подписаться