Development and Validation of an Automated Radiomic CT Signature for Detecting COVID-19

Abstract

The coronavirus disease 2019 (COVID-19) outbreak has reached pandemic status. Drastic measures of social distancing are enforced in society and healthcare systems are being pushed to and beyond their limits. To help in the fight against this threat on human health, a fully automated AI framework was developed to extract radiomics features from volumetric chest computed tomography (CT) exams. The detection model was developed on a dataset of 1381 patients (181 COVID-19 patients plus 1200 non COVID control patients). A second, independent dataset of 197 RT-PCR confirmed COVID-19 patients and 500 control patients was used to assess the performance of the model. Diagnostic performance was assessed by the area under the receiver operating characteristic curve (AUC). The model had an AUC of 0.882 (95% CI: 0.851-0.913) in the independent test dataset (641 patients). The optimal decision threshold, considering the cost of false negatives twice as high as the cost of false positives, resulted in an accuracy of 85.18%, a sensitivity of 69.52%, a specificity of 91.63%, a negative predictive value (NPV) of 94.46% and a positive predictive value (PPV) of 59.44%. Benchmarked against RT-PCR confirmed cases of COVID-19, our AI framework can accurately differentiate COVID-19 from routine clinical conditions in a fully automated fashion. Thus, providing rapid accurate diagnosis in patients suspected of COVID-19 infection, facilitating the timely implementation of isolation procedures and early intervention.

Keywords: COVID-19; artificial intelligence; computed tomography; machine learning; radiomics.

Conflict of interest statement

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: A.V., F.Z., B.M., N.T. are salaried employees ofOncoradiomicsSA. P.L. (Philippe Lambin) reports, within and outside the submitted work, grants/sponsored research agreements from Varian medical, Oncoradiomics, ptTheragnostic/DNAmito, Health Innovation Ventures. He received an advisor/presenter fee and/or reimbursement of travel costs/external grant writing fee and/orin kindmanpower contribution fromOncoradiomics, BHV, Merck, Varian, Elekta, ptTheragnosticand Convert pharmaceuticals. P.L. (Philippe Lambin) has shares in the companyOncoradiomicsSA, Convert pharmaceuticals SA and The Medical Cloud Company SPRL and is co-inventor of two issued patents with royalties on radiomics (PCT/NL2014/050248, PCT/NL2014/050728) licensed toOncoradiomicsand one issue patent onmtDNA(PCT/EP2014/059089) licensed toptTheragnostic/DNAmito, three non-patented invention (softwares) licensed toptTheragnostic/DNAmito, Oncoradiomicsand Health Innovation Ventures and three non-issues, non licensedpatents on Deep Learning-Radiomics and LSRT (N2024482, N2024889, N2024889. R.T.H.L. has shares in the companyOncoradiomicsand is co-inventor of an issued patent with royalties on radiomics (PCT/NL2014/050728) licensed toOncoradiomics. S.W. and W.V. have shares in the companyOncoradiomics. The rest of the co-authors have no known competing financial interests or personal relationships to declare.

Figures

Figure 1

Schematic representation of the radiomics analysis steps: Imaging: chest CT scans of healthy and COVID-19 infected patients were collected and divided between training and testing cohort. Segment: the scans were automatically segmented to delineate the region of interest in the lung. Feature extraction: hand-crafted radiomics feature were extracted from the region of interest. Modelling: the radiomics features were used to train the AI model and the performances were validated in the test set. Actionable insight: the model discrimination performances were assessed in term of accuracy, sensitivity, specificity, negative predictive value (NPV) and positive predictive value (PPV).

Figure 2

Axial and coronal slices with accompanying segmentation masks. (A) Typical aspect of COVID-19 pneumonia characterized by bilateral multilobe ground-glass opacities of peripheral/subpleural distribution, with intralesional reticulations, presenting a “crazy paving” aspect. Subpleural atelectasis and retraction bronchiectasis, typical of organizing pneumonia can also be found; (B) a typical aspect of COVID-19 pneumonia, with posterior right lower lobe condensation and retraction of the ipsilateral diaphragm. Central and peripherical ground-glass opacities in right lower lobe, right upper lobe and left upper lobe; (C) typical chronic obstructive pulmonary disease (COPD) chest computed tomography (CT) characterized by severe centrilobular and para-septal emphysema, associated with cylindrical bronchiectasis and bronchial walls thickening. Right peripherical upper lobe tree in bud pattern seen in bronchiolitis. Middle lobe crescent-shaped atelectasis condensation; (D) normal chest CT.

Figure 3

Flow diagram: Training and validation data were collected, the COVID and Control cohorts were combined. Lungs were segmented from both the training and validations datasets, respectively, and radiomics features were extracted. The independent validation data was used to test the performance of coronavirus intelligence artificielle (COVIA) with unseen patient CTs.

Figure 4

(A) Features with a non-zero regression coefficient in the model and their importance, based on their absolute regression coefficient, and scaled between 0 and 100; (B) ROC plot illustrating the performance (black curve) of the AI framework to discriminate between COVID-19 positive and negative cases in the independent test data set with an area under the receiver operating characteristic curve (AUC) of 0.882 (95% CI: 0.851–0.913).

Figure 5

Classification performance plot. The classification performance in the test dataset, assuming a disease prevalence of 15%, in terms of accuracy (red line), sensitivity (blue line), specificity (green line), NPV (orange line) and PPV (purple line) for different decision thresholds.

Figure 6

Chest CTs of a typical COVID-19 positive patient (A): original scan—left; heat-map—right) with evident reticulation, ground glass opacities and condensations compared to a healthy patient CT scan (B): original scan—left; heat-map—right). Heat-maps underline the more relevant areas for model prediction. Box plots comparing the distribution of the top 5 features among COVID and non-COVID cases ((C)—NGTDM_Complexity; (D)—GLCM_MaxCorr; (E)—NGTDM_Strenght; (F)—GLDZM_LDE; (G)—Stats_Median).