External Validation of a Deep Learning Based Model for Pulmonary Embolism Detection on Chest CT Scans

The scope of this study is the external validation of an explainable deep learning-based classifier for the diagnosis and detection of pulmonary embolism in computed tomography pulmonary angiography (CTPA) and contrast enhanced CT scans.

Study Overview

• Status: Completed
• Conditions: Pulmonary Embolism

Detailed Description

Pulmonary embolism (PE) is a potentially fatal disease if not promptly diagnosed and treated. Chest CTPA remains the gold standard for diagnosis nowadays, but PE can also be incidentally found on enhanced CT scans. Most CTPA exams are performed in clinics in case of suspicion of PE in urgent conditions, whereas a minority is performed for conditions of suspicious or validated chronic pulmonary thromboembolism, a disease frequently overlooked on CT scans but affected by high morbidity and poor prognosis if left untreated. Thus methods to expedite and automatize the recognition of emboli within pulmonary vessels have the potential of becoming an important support in clinical practice, enabling the better triage of urgent cases of PE and an increased sensitivity in the identification of patients with chronic pulmonary thromboembolism. Based on these clinical needs, a deep learning-based model for the detection of pulmonary embolism has been developed on CTPA scans. The model was based on 2D ResNext50 architecture and was trained and validated using a multicentric open source dataset composed of 7169 patients. From these retrospective data, 85,000 slices positive for PE and 123,428 negative for PE were extracted for training. For internal validation, 9,922 slices were used for each class. The model was initially externally validated at the patient-level using a dataset of 156 adult patients from 3 different public sources, with all emboli segmented by at least one experienced radiologist. To gain insight into the model predictions, activation maps were extracted using the Grad-CAM method. Comparing these maps with the ground truth (GT) segmentations, it was determined if the activated regions corresponded to regions of PE by computing the percentage of GT PE that was activated and the percentage of activated regions corresponding to GT PE. The PE classification model reached an area under the curve (AUC) of 0.86 [0.800-0.919], a sensitivity of 82.68 % [75.16 - 88.27] and a specificity of 79.31 % [61.61 - 90.15] on the external validation set. However, these results have been obtained in an unbalanced external validation cohort (127 PE positive against 29 PE negative patients), thus it is very important to assess the model performances also in a more balanced patients cohort, representing the real clinical incidence of PE (between 12 and 22%). For this reason the scope of the present study is to collect an external validation cohort representative of the real clinical reality, including both CTPA and enhanced CT scans, with a more balanced percentage of positive and negative PE cases. Moreover, the performances of the model will be compared between enhanced CT and CTPA scans.

• Study Type: Observational
• Enrollment (Actual): 5000

Contacts and Locations

Study Locations

• Belgium
  • Liège, Belgium, 4000
    • Hospital Center Universitaire De Liège

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: 18 years and older (ADULT, OLDER_ADULT)
• Accepts Healthy Volunteers: No
• Genders Eligible for Study: All

• Sampling Method: Non-Probability Sample

Study Population

The study population is composed of participants who underwent an enhanced chest CT scan for screening purposes for the suspicion of pulmonary embolism

Description

Inclusion Criteria:

• Any patient that has benefit from contrast enhanced CT scan for any clinical reason
• Availability of contrast enhanced images with standard reconstruction kernel and at mediastinal window

Exclusion Criteria:

• Opposition to participate to retrospective clinical trial
• Severe respiratory and hard beam artifacts
• Patients already included in a clinical trial

Study Plan

How is the study designed?

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
The detection performances of the deep learning model	Area under the curve (AUC), sensitivity and specificity for the deep learning model in the identification of pulmonary embolism on enhanced chest CT scan	baseline

Secondary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Comparison of performances of the deep learning model on CTPA and enhanced CT scans	Area under the curve (AUC), sensitivity and specificity for the deep learning model in the identification of pulmonary embolism on enhanced chest CT scan compared to the CTPA scans	baseline

Collaborators and Investigators

• Sponsor: OncoRadiomics
• Collaborators: Centre Hospitalier Universitaire de Liege
• Investigators: Principal Investigator: Paul Meunier, PhD, Chu of Liège

Study record dates

Study Major Dates

• Study Start (ACTUAL): April 1, 2022
• Primary Completion (ACTUAL): April 1, 2022
• Study Completion (ACTUAL): July 14, 2022

Study Registration Dates

• First Submitted: April 11, 2022
• First Submitted That Met QC Criteria: April 11, 2022
• First Posted (ACTUAL): April 18, 2022

Study Record Updates

• Last Update Posted (ACTUAL): September 21, 2022
• Last Update Submitted That Met QC Criteria: September 20, 2022
• Last Verified: April 1, 2022

More Information

Terms related to this study

• Keywords: AI, Deep learning, Pulmonary embolism, CT scan
• Additional Relevant MeSH Terms: Cardiovascular Diseases; Vascular Diseases; Respiratory Tract Diseases; Lung Diseases; Embolism and Thrombosis; Embolism; Pulmonary Embolism
• Other Study ID Numbers: 0156_PE detector validation

Plan for Individual participant data (IPD)

• Plan to Share Individual Participant Data (IPD)?: NO

Drug and device information, study documents

• Studies a U.S. FDA-regulated drug product: No
• Studies a U.S. FDA-regulated device product: No