COVID-19 pneumonia on chest X-rays: Performance of a deep learning-based computer-aided detection system

Abstract

Chest X-rays (CXRs) can help triage for Coronavirus disease (COVID-19) patients in resource-constrained environments, and a computer-aided detection system (CAD) that can identify pneumonia on CXR may help the triage of patients in those environment where expert radiologists are not available. However, the performance of existing CAD for identifying COVID-19 and associated pneumonia on CXRs has been scarcely investigated. In this study, CXRs of patients with and without COVID-19 confirmed by reverse transcriptase polymerase chain reaction (RT-PCR) were retrospectively collected from four and one institution, respectively, and a commercialized, regulatory-approved CAD that can identify various abnormalities including pneumonia was used to analyze each CXR. Performance of the CAD was evaluated using area under the receiver operating characteristic curves (AUCs), with reference standards of the RT-PCR results and the presence of findings of pneumonia on chest CTs obtained within 24 hours from the CXR. For comparison, 5 thoracic radiologists and 5 non-radiologist physicians independently interpreted the CXRs. Afterward, they re-interpreted the CXRs with corresponding CAD results. The performance of CAD (AUCs, 0.714 and 0.790 against RT-PCR and chest CT, respectively hereinafter) were similar with those of thoracic radiologists (AUCs, 0.701 and 0.784), and higher than those of non-radiologist physicians (AUCs, 0.584 and 0.650). Non-radiologist physicians showed significantly improved performance when assisted with the CAD (AUCs, 0.584 to 0.664 and 0.650 to 0.738). In addition, inter-reader agreement among physicians was also improved in the CAD-assisted interpretation (Fleiss' kappa coefficient, 0.209 to 0.322). In conclusion, radiologist-level performance of the CAD in identifying COVID-19 and associated pneumonia on CXR and enhanced performance of non-radiologist physicians with the CAD assistance suggest that the CAD can support physicians in interpreting CXRs and helping image-based triage of COVID-19 patients in resource-constrained environment.

Conflict of interest statement

I have read the journal’s policy and the authors of this manuscript have the following competing interests: Chang Min Park holds stock of Promedius and stock options of Lunit Inc. and Coreline Soft. Eui Jin Hwang, Hyungjin Kim, and Chang Min Park received research grants from Lunit Inc, outside the present paper. We checked all authors’ competing interests and confirmed that our statement does not alter the adherence to PLOS ONE policy.

Figures

Fig 1. Flow diagram for patients’ inclusion.

A total of 80 patients with COVID-19 were retrospectively included from four institutions, while 92 patients without COVID-19 were included from single institution. Among patients with COVID-19, 67 exhibited findings of pneumonia on chest CT obtained within 24 hours from CXR, while the other 13 patients did not exhibited any findings of pneumonia on chest CT.

Fig 2. Representative case with COVID-19 pneumonia.

A CXR (A) of a patient with confirmed COVID-19 shows patchy infiltrates in both lower lung fields (arrows). The corresponding chest CT image (B) obtained in the same day with the CXR shows multifocal patchy ground-glass opacities in both lower lobes of the lung. The CT severity score of the patient was 13. The CAD system correctly detected pulmonary infiltrates with a probability score of 56% (C). In the reader-alone interpretation, four thoracic radiologists correctly identified the abnormality while none of the non-radiologist physicians identified the abnormality. In the CAD-assisted interpretation, all five thoracic radiologists and four non-radiologist physicians identified the abnormality.

Fig 3. Representative case without COVID-19 pneumonia.

A CXR (A) and corresponding chest CT (B) of a patient with fever and dyspnea but negative RT-PCR result for COVID-19 show no pulmonary abnormality suggestive of pneumonia. The CAD system did not detect any abnormalities in the CXR and the probability score was 13% (C). In the reader-alone interpretation, four thoracic radiologists and four non-radiologist physicians misclassified the CXR as having findings of pneumonia. In the CAD-assisted interpretation, only one thoracic radiologist and two non-radiologist physicians made false-positive classification of the CXR. Mediastinal window CT image (D) show pulmonary embolism in the right descending pulmonary artery (arrow), presumed cause of patients’ symptom.

Fig 4. Performance of the CAD versus reader-alone interpretations.

For identification of RT-PCR-positive COVID-19 patients (A), the CAD exhibited AUC of 0.714 (black line), which did not significantly differ from that of thoracic radiologists (0.701, blue line) but significantly higher than that of non-radiologist physicians (0.584, red line). For identification of pneumonia defined on chest CT (B), the CAD exhibited AUC of 0.790 (black line), which was not significantly different from that of thoracic radiologists (0.784, blue line), but significantly higher than that of non-radiologist physicians (0.650, red line).

Fig 5. Performance of physician alone versus CAD-assisted interpretations.

For identification of RT-PCR positive COVID-19 patients, the AUCs of thoracic radiologists did not significantly differ between reader-alone (red line) and CAD-assisted interpretations (blue line) (0.701 vs. 0.699; P = .815) (A), while the AUC non-radiologist physicians was significantly improved in the CAD-assisted interpretation (blue line) compared to the reader-alone interpretation (red line) (0.584 vs. 0.664; P = .006) (B). For identification of pneumonia defined on chest CT, the AUCs of thoracic radiologists also did not significantly differ between reader-alone (red line) and CAD-assisted interpretations (blue line) (0.784 vs. 0.789; P = .524) (C), while the AUC non-radiologist physicians was significantly improved in the CAD-assisted interpretation (blue line) compared to the reader-alone interpretation (red line) (0.650 vs. 0.738; P = .003) (D).