Standalone Observational Study Assessing the Performance of an AI/ML Tech-based SaMD on Chest LDCT Images (REALITY) (REALITY)

Multinational, Multicenter, Retrospective Study to Evaluate an AI/ML Technology-Based End-to-End CADe/CADx SaMD, Which Allows Detection, Localization and Characterization of Pulmonary Nodules (REALITY)

This is a Multinational, Multicenter, retrospective study for the evaluation of the standalone efficacy and safety of an Artificial Intelligence/Machine Learning (AI/ML) technology-based end-to-end Computer assisted Detection/Computer Assisted Diagnosis (CADe/CADx) Software as a Medical Device (SaMD) developed to detect, localize and characterize malignant, and suspicious for lung cancer nodules on Low Dose Computed Tomography (LDCT) scans taken as part of a Lung Cancer Screening (LCS) program.

LDCT Digital Imaging and Communications in Medicine (DICOM) images of patients who underwent lung cancer screening were selected and included into the study. Selected scans will then be analyzed by the CADe/CADx SaMD and compared to radiologist generated reference standards including lesions localization and lesion cancer diagnosis.

Figures of merit at patient level and lesion level detection and diagnostic efficacy will be calculated as well as sub-class analysis to ensure algorithm performance generalizability.

Study Overview

• Status: Completed
• Conditions: High Risk Cancer
• Intervention / Treatment: Device: Median LCS

• Study Type: Observational
• Enrollment (Actual): 1147

Contacts and Locations

Study Locations

• Spain
  • Madrid, Spain, 28040
    • Fundacion instituto de investigacion sanitaria de la fundacion jimenez diaz (FJD)
  • Pamplona, Spain, 31009
    • Universidad de Navarra
• United States
  • Pennsylvania
    • Philadelphia, Pennsylvania, United States, 19104
      • University of Pennsylvania - Penn Center for Innovation
  • Tennessee
    • Memphis, Tennessee, United States, 38120
      • Baptist Clinical Research Institute
  • Texas
    • Houston, Texas, United States, 77030
      • The University of Texas M.D. Anderson Cancer Center

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: Adult; Older Adult
• Accepts Healthy Volunteers: Yes

• Sampling Method: Probability Sample

Study Population

High risk lung cancer population from Radiology or Pneumology hospital departments.

Patients enrolled in this study were retrospectively collected from centers across the EU and USA where they were enlisted into lung cancer screening due to high risk of lung cancer according to established lung cancer screening guidelines.

The cohort used for testing the efficacy and safety of the device will be an "enriched cohort" with a 1:2 distribution of cancer positive and benign patients

Description

Inclusion Criteria:

• ≥50-80 Years of age;
• Current or ex-smoker (>=20 pack years);
• Patient screened and surveilled for lung cancer screening following lung cancer screening guidelines (equivalent to United States Preventive Services Task Force (USPSTF) 2021 Criteria);
• Received LDCT due to inclusion in high-risk category for lung cancer.

Exclusion Criteria:

• Prior lung resection;
• Pacemaker or other indwelling metallic medical devices in the thorax that interfere with CT acquisition;
• Patients/images used during AI model development;
• Patients with only hilar and/or mediastinal cancer(s);
• Patients with only ground glass cancer(s);
• Patients with nodules, solid or part-solid >30mm (masses);
• Patients that are not accompanied with the required clinical information;
• Patients with imaging with any of the following: missing slices, slice thickness >3mm;
• Partial cover of the lung.

Study Plan

How is the study designed?

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
AUROC (Area under ROC curve) at patient level	AUROC that measures Median LCS performance at patient level is strictly superior to 0.8. Support for Primary Endpoint: Derived from the patient level AUROC at the product fixed operating point : Sensitivity, Specificity, PPV, NPV.	12 months

Secondary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Sensitivity > 70% when Specificity=70%		12 months
Specificity > 70% when Sensitivity=70%		12 months
AUC of LROC > 0.75	In contrast to the receiver operating characteristic (ROC) assessment paradigm, localization ROC (LROC) analysis provides a means to jointly assess the accuracy of localization and detection in an observational study.	12 months
Detection sensitivity>0.8 with average FP rate per scan<1		12 months
ICC>0.8 for average diameter	Intraclass Correlation Coefficient (ICC), is a descriptive statistic that can be used when quantitative measurements are made on units that are organized into groups. It describes how strongly units in the same group resemble each other.	12 months
ICC>0.8 for long axis diameter		12 months
ICC>0.8 for short axis diameter		12 months
ICC>0.75 for Volume		12 months
DICE Coefficient >0.7		12 months

Collaborators and Investigators

• Sponsor: Median Technologies
• Investigators: Principal Investigator: Anil Vachani, MD, University of Pennsylvania

Study record dates

Study Major Dates

• Study Start (Actual): September 21, 2022
• Primary Completion (Actual): July 24, 2024
• Study Completion (Actual): August 21, 2024

Study Registration Dates

• First Submitted: August 23, 2024
• First Submitted That Met QC Criteria: August 26, 2024
• First Posted (Actual): August 28, 2024

Study Record Updates

• Last Update Posted (Actual): August 28, 2024
• Last Update Submitted That Met QC Criteria: August 26, 2024
• Last Verified: August 1, 2024

More Information

Terms related to this study

• Other Study ID Numbers: MT-LCS-002

Drug and device information, study documents

• Studies a U.S. FDA-regulated drug product: No
• Studies a U.S. FDA-regulated device product: Yes
• product manufactured in and exported from the U.S.: No