Evaluation of Contextflow DETECT Lung CT Nodule Detection Software in Chest CT Scans

Evaluation of Computer-Aided Lung Nodule Detection Software in Chest CT Scans With an Assessment of Its Impact on Readers Decision-Making Process

contextflow DETECT Lung CT is a Artificial Intelligence (AI)-based computed-aided detection (CADe) system, intended to support radiologists in the detection of lung nodules in chest computed tomography (CT) scans. System is intended to be used as a second-reader, therefore results provided by the software are meant to complement the radiologist's findings and decisions.

Proposed study will be multi-reader, multi case (MRMC) retrospective reader study. The goal of the study is to evaluate the influence of CADe on the effectiveness of lung nodule detection. During the study, 10 radiologists will analyze 350 chest CT scans of adult patients, with and without the assistance of CADe. The study will be conducted remotely. CT scans will be uploaded to a web-based image submission and annotation platform, in which every participant of the study will be provided with individual account and assigned task list.

The primary objective of the study determine if the diagnostic accuracy of radiologists with CADe assistance is superior to the diagnostic accuracy of radiologists without CADe assistance in localizing the pulmonary nodules with enhanced area under the free-response operating characteristic curve (AUC of FROC).

The study will target approximately 350 asymptomatic adult patients, whose CT scans were acquired during routine CT examination. The patient population will include patients with and without lung nodules.

Study Overview

• Status: Completed
• Conditions: Lung Cancer
• Intervention / Treatment: Device: Aided read with contextflow DETECT Lung CT

Detailed Description

1. Background and rationale

   According to the American Cancer Society, lung cancer was the second commonly diagnosed cancer with an estimated 2.2 million new cases in 2020. Currently, the most effective way of reducing cancer mortality is early diagnosis of lung cancer. However, in routine clinical practice, many early-stage lung cancer were delayed in diagnosis, due to the fact that most patients are often asymptomatic. That's the main reason why fast and the least invasive lung disease screening is so important for the early detection of pathological changes in asymptomatic patients from high-risk cohorts. Computed tomography (CT) is the most recent dominated imagining tool used to capture lung images, and therefore also for nodule detection and characterization. Consequently, computer-aided detection and software systems based on artificial intelligence have become one of the most prominent and valuable tools for detection of pulmonary nodules.

   The aim of the proposed reader study is to evaluate one of such tools, contextflow DETECT Lung CT, and to assess to what extent supporting radiologists with a computer-aided detection system improves the detection of nodules and supports radiologists in the decision-making process.

   contextflow DETECT Lung CT is an AI-based web application for 3D medical imaging data. It's intended to be used for the detection and visualization of pulmonary nodules in chest CT scans of clinically asymptomatic adult patients.

   The system is intended to alert radiologists to specific locations in which nodules can be detected that may otherwise be overlooked. The system provides measurements of the nodules, such as average diameter, long-axis diameter, short-axis diameter as well as the total volume.

   contextflow DETECT Lung CT is intended to be used as a second reader, in addition to original reading performed by radiologists. Results provided by contextflow DETECT Lung CT are intended to complement the information that radiologists take into account during their decision-making process.

2. Study design

   This will be a retrospective, multi-reader multi-case (MRMC) study. The performance of radiologists alone will be compared with the performance of radiologists supported by the detection software (as a second reader). The study will include 10 readers, and a final dataset of 350 scans collected from at least 3 radiology centers and revised by truthers. During the study three following reads are planned: Read 1 (Truthing), Read 2 (Baseline, without CADe assistance), Read 3 (Intervention, with CADe assistance).

   The study will consist of 350 CT images collected from asymptomatic adult patients. At least 50% of the cases shall be US data, other cases will be collected from European radiology centers. At least 65% of the cases will include at least one lung nodule. The dataset will also include cases without nodules.

3. Selection of readers

   Readers qualification criteria:

   • US-board of Radiology or equivalent specialty certification
   • At least 5 years of chest CT interpretation experience
   • Fellowship-trained in thoracic/chest imaging,
   • Successful training on the use of study software.

4. Selection of patients and imaging data

   Patients, whose chest CT images were included into the dataset, must meet all the following criteria:

   • Adult patients (aged 22 or older) with the number of nodules ranged from 0 to 10;
   • For patients with at least one lung nodule: nodules detected and confirmed in the ground-truthing procedure;
   • Asymptomatic patients, who undergo a routine chest CT scan.

   Imaging data must meet all the following inclusion criteria to be in this study:

   • Images are CT images of the chest showing full left and right lung;
   • Images do not show any movement artifacts of the patient;
   • Images of the chest must be reconstructed in axial plane;
   • The number of nodules per image was 10 or fewer;
   • All nodules measuring between 4 - 30 mm in maximal long-axis diameter at CT scan.

5. Reader study procedures

   Ground truth process workflow: 3 US board-certified thoracic radiologists with at least 5 years experience after corresponding fellowship will be asked to read CT scans and identify actionable nodules in images from ground truthing dataset (370 CT scans randomly chosen from balanced candidate database) by marking them by manually drawing bounding boxes around them.

   Reading process workflow: For each CT scan, the reading process will take place in two steps: (1) reader's stand-alone reading, without contextflow DETECT Lung CT assistance and (2) second reader's reading with contextflow DETECT Lung CT as a second-reader. In each step, readers will analyze 350 CT scans dataset, with limited clinical information.

6. Blinding and randomization

   The identification of cases selected for the study, as well as individual analyzes between readers will be blinded to readers. Readers will not have access to the results from truthing procedure or to the information how many actionable nodules have been localized in individual CT images.

   Image submission and annotation platform will generate the random task list for each reader. The reader order will be different for all of the readers.

7. Study objectives and endpoints

   The primary and secondary objectives of the reader study is to demonstrate enhanced diagnostic accuracy of radiologists with CADe in comparison with the diagnostic accuracy of radiologists without CADe in

   Primary

   • Localizing the lesions with the enhanced area under the free-response receiver operating characteristic curve (AUC of FROC)

   Secondary

   • Detecting disease using (AUC of ROC)
   • Enhancing disease identification with AUC of LROC

   The following endpoints will also be evaluated:

   • The difference in sensitivities of lesion detection with and without CADe given specified average false positive numbers of lesions per image;
   • The difference in sensitivities of disease diagnostic with and without CADe given specified specificity;
   • Positive predictive value for disease diagnosis
   • Negative predictive value for disease diagnosis

8. Data management

   All data that is part of the study, both shared by our partners or downloaded from public sources, was anonymized following HIPAA and/or GDPR regulations and does not include any protected health information.

9. End of the study

The reader study will be considered completed when a sufficient number of images have been analyzed to achieve a satisfactory statistical test performance in the means of statistical power. To achieve that, at least 300 scans should be analyzed.

• Study Type: Observational
• Enrollment (Actual): 337

Contacts and Locations

Study Locations

• Austria
  • Vienna, Austria, 1050
    • contextflow GmbH

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: 22 years and older (ADULT, OLDER_ADULT)
• Accepts Healthy Volunteers: Yes
• Genders Eligible for Study: All

• Sampling Method: Probability Sample

Study Population

Asymptomatic adult patients (aged 22 or older), with or without pulmonary nodules detected during the routine chest CT scan.

CT images taken with CT scanners provided by 3 different vendors, with 50% of images acquired from the US medical centers.

Description

Inclusion Criteria:

• adult asymptomatic patients, who undergo a routine chest CT scan.

Exclusion Criteria:

• symptomatic patients.

Study Plan

How is the study designed?

Design Details

• Observational Models: Case-Control
• Time Perspectives: Retrospective

Number of groups / cohorts

1

Cohorts and Interventions

Group / Cohort	Intervention / Treatment
Asymptomatic adult patients	Device: Aided read with contextflow DETECT Lung CT; Radiologists read chest CT scabs with and without the aid of contextflow DETECT Lung CT as a second reader

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Diagnostic accuracy	The primary objective of the reader study is to determine if the diagnostic accuracy of radiologists with CADe assistance is superior to the diagnostic accuracy of radiologists without CADe assistance in localizing the pulmonary nodules with enhanced area under the free-response operating characteristic curve (AUC of FROC). The true positive rate (or sensitivity) is calculated as the identified positive lesion among the true positive divided by the total number of true positive lesions among all images. The number of false positive findings is collected per image.	20 hours

Secondary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Disease diagnosis capabilities	To evaluate the capabilities in disease diagnosis by radiologists with and without CADe assistance using the area under the receiver operating characteristics curve (AUC of ROC). The ROC curve is plotted for the true positive and false positive rates based on various confidence thresholds in the reader's diagnostic decision.	20 hours
Disease identification capabilities	To evaluate the capabilities in disease identification of radiologists with accurate localization with and without CADe assistance using the area under the localization receiver operating characteristics curve (AUC of LROC). The LROC curve is plotted for the true positive and false positive rates based on various confidence thresholds in readers.	20 hours

Collaborators and Investigators

• Sponsor: contextflow GmbH

Publications and helpful links

General Publications

• Sung H, Ferlay J, Siegel RL, Laversanne M, Soerjomataram I, Jemal A, Bray F. Global Cancer Statistics 2020: GLOBOCAN Estimates of Incidence and Mortality Worldwide for 36 Cancers in 185 Countries. CA Cancer J Clin. 2021 May;71(3):209-249. doi: 10.3322/caac.21660. Epub 2021 Feb 4.
• Hansell DM, Bankier AA, MacMahon H, McLoud TC, Muller NL, Remy J. Fleischner Society: glossary of terms for thoracic imaging. Radiology. 2008 Mar;246(3):697-722. doi: 10.1148/radiol.2462070712. Epub 2008 Jan 14.
• Qian F, Yang W, Chen Q, Zhang X, Han B. Screening for early stage lung cancer and its correlation with lung nodule detection. J Thorac Dis. 2018 Apr;10(Suppl 7):S846-S859. doi: 10.21037/jtd.2017.12.123.

Helpful Links

• American Cancer Society, Cancer Statistics Center

Study record dates

Study Major Dates

• Study Start (ACTUAL): April 1, 2022
• Primary Completion (ACTUAL): October 30, 2022
• Study Completion (ACTUAL): October 30, 2022

Study Registration Dates

• First Submitted: July 28, 2022
• First Submitted That Met QC Criteria: July 28, 2022
• First Posted (ACTUAL): August 1, 2022

Study Record Updates

• Last Update Posted (ACTUAL): February 8, 2023
• Last Update Submitted That Met QC Criteria: February 7, 2023
• Last Verified: February 1, 2023

More Information

Terms related to this study

• Keywords: artificial intelligence; pulmonary nodule; computer-aided detection
• Other Study ID Numbers: 0020220401

Plan for Individual participant data (IPD)

• Plan to Share Individual Participant Data (IPD)?: NO
• IPD Plan Description: In this study, anonymized cases are used. There is no clear connection to patients or IPD included. The identification of cases selected for the study, as well as individual analyzes between readers will be blinded to readers.

Drug and device information, study documents

• Studies a U.S. FDA-regulated drug product: No
• Studies a U.S. FDA-regulated device product: No