Evaluation of Computer-Aided Lung Nodule Detection Software in Thoracic CT for Riverain Technologies LLC

Computer-Aided System for the Improvement of Nodule Detection and Characterization in Thoracic CT Based on a Novel Vessel Suppressed CT Volume

STUDY DESIGN:

This is a retrospective, multi-reader, multi-case, (MRMC) randomized reader study.

OBJECTIVE:

Primary: The primary objective of this clinical study is to prove that a user aided with ClearRead CT InSight (CRCTI) is superior to the unaided reader for detecting actionable lung nodules.

Secondary: The secondary objective of this clinical study is to prove that the reader's reading time is not significantly increased when aided with CRCTI.

NUMBER OF SUBJECTS:

Retrospective CT studies from approximately 300 patients will be included in the study. Approximately 100 true positive cases and 200 normal cases.

NUMBER OF READERS:

A reader study with at least ten (10) participating radiologists (US Board Certified) will be conducted.

PRIMARY ENDPOINTS:

Scores given by the radiologists with and without ClearRead CT Insight will be recorded and compared to the true status of the study-cases. The frequency of the scores for each method (Unaided, Aided) will be tabulated and LROC curves constructed along with sensitivity, specificity, PPV, NPV and clinical actions. Additionally, machine nodule detection rate and false positives per patient on normal cases will be measured.

PATIENT POPULATION :

The study will target approximately one hundred (100) patients whose CT nodules were shown to be cancer and two hundred (200) normal patients. The patient population will be consistent with the national lung cancer screening protocols.

Study Overview

• Status: Completed
• Conditions: Lung Cancer
• Intervention / Treatment: Device: ClearRead CT Insight

Detailed Description

A reader study with at least ten (10) participating radiologists will be conducted. A localized receiver operating characteristic curve will be used to evaluate radiologists' diagnostic performance (in terms of the trade-off between the sensitivity and specificity when the decision criteria changes) in the detection of lung nodules on lung CTs with and without the usage of the ClearRead CT Insight (CRCTI) system. The time needed for the review and interpretation of each case will also be recorded.

An initial (baseline) interpretation will be made by each of the radiologists based on the Lung CT in its original form. At a minimum of one month later, each radiologists will again interpret the same images viewing the pair of CRCTI CT series: Two sets of CT images (standard with CADe marks and processed with vessel suppression will be presented on either one large monitor or two adjacent monitors.

During the baseline reading the radiologist will mark the location of the actionable nodules and assign a score. The radiologist will also indicate the recommended method of follow-up (Contrast CT, PET-CT, CT Follow-up, Biopsy).

During the second reading session (concurrent read), the radiologist will be presented with a standard appearing CT with computer-aided detection (CADe) marks placed and the vessel suppressed same slice with the vessel suppressed view (right image). The second image, vessel suppressed, will be grayed out until the radiologist move the mouse to the second panel. The radiologist will mark locations. These may or may not correspond to the locations of the CAD markers. As before, the radiologist will assign a level of suspicious to each mark and indicate the need, if any, of an additional diagnostic action (CT Follow-up, Contrast CT, PET-CT, or Biopsy).

Based on the levels of suspicion for each nodule and the associated likelihood ratings, LROC curves will be constructed for both the baseline and the concurrent reads and the significance of any difference will be calculated. The recommendations for further action (CT Follow-up, Contrast CT, PET-CT, or Biopsy) will be used to calculate sensitivity and specificity, PPV and NPV.

Number and types of cases:

Retrospective lung CT image series from approximately 300 patients will be included in the study. Approximately one hundred (100) of the patients will have pathology confirmed cancers and approximately two hundred (200) of the patients will be CTs associated with normal patients. Also included as nodule images are those where the actionable nodule was not acted on at that time, but was detected and acted on based on a subsequent CT. These are the prior images where the nodule can be identified and its location is the same as on the "current" image confirmed by the radiologist expert panel (using a majority of three as the decision criterion).

The selected sample, randomly selected from a larger pool of CT cases will be enriched in the following way:

1. Lung nodules (cancers in this study) will be tumor size T1a (20 mm or less). The proportion of nodules 20 mm or less may be increased since this is where the investigators expect the major impact of this software to be.
2. Non-Solid (ground glass) nodules will be added to the sample (based on availability) to determine the performance of the system on non-solid nodules. For this group, to have sufficient cases, the investigators may have to include benign (non-malignant) non-solid nodules.
3. In this project, the investigators will perform a Machine Test of the ClearRead CT Insight algorithm followed by a reader performance evaluation study. Riverain will provide a system configured with the operating point set to be used for the reader studies and a configuration for an "open" system to be used for machine testing and FROC generation.

Arm 1: a baseline read (no secondary content) and Arm 2: concurrent, CAD augmented read.

1. st Arm: Do baseline (measure time, readers score regions according to action and suspiciousness) - mark all locations of concern
2. nd Arm: Concurrent read (measure time, readers score regions according to action and suspiciousness) - mark all locations of concern

The primary study hypothesis is that the adjunctive use of ClearRead CT Insight is superior to use of standard lung CT images alone, as measured by the area under the LROC curve.

STATISTICAL ANALYSES Accuracy

To evaluate the hypothesis of superiority in terms of improvement in accuracy for ClearRead CT Insight vs. unaided, a mixed effects model (DBM) will be implemented (similar to the model outlined in Dorfman, Berbaum and Metz, 1997), where variance components will be included to account for reader, case, reader by case, reader by modality, case by modality and reader by case by modality. However, it is anticipated that the three-way interaction will be inestimable and will subsequently be dropped from the statistical model. Specifically, the hypothesis to test for superiority of ClearRead CT Insight vs. unaided is:

H0: AUCunaided - AUCClearRead CT Insight ≥ 0.0 vs. HA: AUCunaided - AUCClearRead CT Insight < 0

The AUC of the LROC is the primary endpoint to evaluate accuracy and the test of interest will be a two-sided 95% confidence interval on the effect of modality (i.e. ClearRead CT Insight minus unaided). Significance will be concluded if the upper bound of the two-sided 95% confidence interval does not include zero. If the null hypothesis (H0) is rejected, the alternative hypothesis (HA) is accepted and the superiority of using the ClearRead CT Insight system will be established.

Time

The second co-primary objective is to evaluate reduction in time spent per image for ClearRead CT Insight vs. unaided. Specifically, the hypothesis to test for superiority of ClearRead CT Insight vs. unaided is:

H0= Tunaided - TClearRead CT Insight ≤ 0 vs. HA= Tunaided - TClearRead CT Insight > 0

To evaluate the hypothesis of non-inferiority in terms of improved read time for ClearRead CT Insight vs. unaided, a mixed effects model will be implemented (similar to the model outlined in Dorfman, Berbaum and Metz, 1992), where variance components will be included to account for reader, case, reader by case, reader by modality, case by modality and reader by case by modality. However, it is anticipated that the three-way interaction will be inestimable and will subsequently be dropped from the statistical model.

The read times will be tested using a two-sided 95% confidence interval on the effect of modality (i.e. ClearRead minus unaided). Significance will be concluded if the upper bound of the two-sided 95% confidence interval does not include zero.

With either analysis, the use of the mixed model could be modified to employ a bootstrap sampling approach if the model assumptions of the DBM method have been violated. The upper 95 % confidence limit for the difference in the areas under the curves would be calculated using 10,000 bootstraps in the MultiReader MultiCase (MRMC) ROC method.

POWER AND SAMPLE SIZE

The power to detect differences in the AUC of the LROC curve for the proposed statistical analysis using the current design baseline, i.e. 300 cases, each corresponding image read by 10 readers, was assessed through a simulation study. Specifically, the model outlined above was used to simulate 500 datasets across a range of effect sizes, where power was defined as the proportion of datasets that yielded a significant p-value for testing the fixed effect of modality. The method of Dorfman, Berbaum and Metz, 1992 (DBM) was implemented utilizing statistical mixed model theory with jackknife estimates.

These simulations required assumptions regarding the magnitude of the variance components associated with the different random effects. As a pilot study to obtain estimates of variance components was not conducted, variance component estimates from Riverain study SoftView 510(k) (Record # BSSI-PR-09-00006) that uses a similar technology as the proposed ClearRead CT Insight device were used in all power simulations. Please note that the variance components were scaled by the total variance to represent the proportion of total variance explained by each component. All power estimates are dependent upon the appropriateness of the assumed variance components.

• Study Type: Interventional
• Enrollment (Actual): 12
• Phase: Not Applicable

Contacts and Locations

Study Locations

• United States
  • Virginia
    • Arlington, Virginia, United States, 22203
      • Arlington Innovation Center: Health Resaerch

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: 55 years to 77 years (Adult, Older Adult)
• Accepts Healthy Volunteers: No
• Genders Eligible for Study: All

Description

Inclusion Criteria:

A thoracic CT case that is associated or with a feature of the following:

1. Asymptomatic patients age 55 - 77 with history of smoking
2. Primary Lung Cancer
3. Biopsy Proven (LuRADS 5) with radiology report
4. Screen detected event plus 1 prior CT. 2 year of prior CTs is preferred, if available.
5. <= 3mm slice spacing, no gaps
6. Use standard reconstruction kernels
7. Maximum of 5 nodules per image
8. Nodules must be 5-20mm in size
9. With or without contrast

Exclusion Criteria:

Thoracic CT cases that meet the following exclusion criteria will not be collected for use during clinical testing.

1. No Acute pneumothorax
2. Both lungs must be fully visible within the field of view
3. Apices cannot be cropped
4. No excessive motion artifacts
5. Symptomatic patients with co-morbidities

Study Plan

How is the study designed?

Design Details

• Primary Purpose: Diagnostic
• Allocation: Randomized
• Interventional Model: Parallel Assignment
• Masking: Double

Number of Arms

2

Arms and Interventions

Participant Group / Arm	Intervention / Treatment
No Intervention: Arm 1; Participating radiologists will perform clinical reading on ~300 cases. They will mark all locations of concern (clinically actionable nodules) on conventional thoracic CT images. Computer will measure time, readers score regions according to action and suspiciousness.
Active Comparator: Arm 2; Participating radiologists will perform clinical reading on ~300 cases. They will mark all locations of concern (clinically actionable nodules) on thoracic CT images aided by ClearRead CT Insight software as the intervention. Computer will measure time, readers score regions according to action and suspiciousness.	Device: ClearRead CT Insight; During the second reading session (concurrent read), the radiologist will be presented with a standard appearing CT with CADe marks placed and the vessel suppressed same slice with the vessel suppressed view as the intervention. The second image, vessel suppressed, will be grayed out until the radiologist move the mouse to the second panel. The radiologist will mark locations. These may or may not correspond to the locations of the CAD markers. As before, the radiologist will assign a level of suspicious to each mark and indicate the need, if any, of an additional diagnostic action (CT Follow-up, Contrast CT, PET-CT, or Biopsy).

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Comparing the Detection Accuracy, Sensitivity, and Specificity of Clinically Actionable Lung Nodules	The detection rates in accuracy, sensitivity, and specificity on clinically actionable lung nodules were computed and compared for Aim 1 (baseline) and Arm 2 (study participants were aided with ClearRead CT software) studies.	4 months
Comparing LROC Curve in the Detection of Clinically Actionable Lung Nodules Among Normal	The areas under location-specific Receiver Operating Characteristic (LROC) curves were computed and compared for Aim 1 (baseline) and Aim 2 (study participants were aided with ClearRead CT software).	4 months

Secondary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Average Reading Time Per Case	The average reading times per reader per case were computed and comapred in both Aim 1 and Arm 2 (study participants were aided with ClearRead CT software) studies.	4 months

Collaborators and Investigators

• Sponsor: Virginia Polytechnic Institute and State University
• Collaborators: Riverain Technologies
• Investigators: Study Director: Matthew T Freedman, M.D., Virginia Polytechnic Institute and State University

Publications and helpful links

General Publications

• Beyer F, Zierott L, Fallenberg EM, Juergens KU, Stoeckel J, Heindel W, Wormanns D. Comparison of sensitivity and reading time for the use of computer-aided detection (CAD) of pulmonary nodules at MDCT as concurrent or second reader. Eur Radiol. 2007 Nov;17(11):2941-7. doi: 10.1007/s00330-007-0667-1. Epub 2007 May 22.
• Dorfman DD, Berbaum KS, Metz CE. Receiver operating characteristic rating analysis. Generalization to the population of readers and patients with the jackknife method. Invest Radiol. 1992 Sep;27(9):723-31. No abstract available.
• Foti G, Faccioli N, D'Onofrio M, Contro A, Milazzo T, Pozzi Mucelli R. Evaluation of a method of computer-aided detection (CAD) of pulmonary nodules with computed tomography. Radiol Med. 2010 Sep;115(6):950-61. doi: 10.1007/s11547-010-0556-6. Epub 2010 Jun 23. English, Italian.
• Matsumoto S, Ohno Y, Aoki T, Yamagata H, Nogami M, Matsumoto K, Yamashita Y, Sugimura K. Computer-aided detection of lung nodules on multidetector CT in concurrent-reader and second-reader modes: a comparative study. Eur J Radiol. 2013 Aug;82(8):1332-7. doi: 10.1016/j.ejrad.2013.02.005. Epub 2013 Mar 6.
• Obuchowski NA. New methodological tools for multiple-reader ROC studies. Radiology. 2007 Apr;243(1):10-2. doi: 10.1148/radiol.2432060387. No abstract available.
• Obuchowski NA, McClish DK. Sample size determination for diagnostic accuracy studies involving binormal ROC curve indices. Stat Med. 1997 Jul 15;16(13):1529-42. doi: 10.1002/(sici)1097-0258(19970715)16:133.0.co;2-h.

Study record dates

Study Major Dates

• Study Start: April 1, 2015
• Primary Completion (Actual): September 1, 2017
• Study Completion (Actual): March 1, 2018

Study Registration Dates

• First Submitted: April 15, 2015
• First Submitted That Met QC Criteria: May 6, 2015
• First Posted (Estimate): May 12, 2015

Study Record Updates

• Last Update Posted (Actual): November 4, 2021
• Last Update Submitted That Met QC Criteria: November 2, 2021
• Last Verified: November 1, 2021

More Information

Terms related to this study

• Other Study ID Numbers: 458420

Plan for Individual participant data (IPD)

• Plan to Share Individual Participant Data (IPD)?: No

Drug and device information, study documents

• Studies a U.S. FDA-regulated drug product: No
• Studies a U.S. FDA-regulated device product: No
• product manufactured in and exported from the U.S.: No