Validation of an Artificial Intelligence-based Algorithm for Skeletal Age Assessment

May 24, 2021 updated by: Safwan S Halabi, MD, Stanford University

Prospective, Multi-Center, Randomized Controlled Trial for Skeletal Age Assessment AI Model

The purpose of this study is to understand the effects of using an Artificial Intelligence algorithm for skeletal age estimation as a computer-aided diagnosis (CADx) system. In this prospective real-time study, the investigators will send de-identified hand radiographs to the Artificial Intelligence algorithm and surface the output of this algorithm to the radiologist, who will incorporate this information with their normal workflows to make an estimation of the bone age. All radiologists involved in the study will be trained to recognize the surfaced prediction to be the output of the Artificial Intelligence algorithm. The radiologists' diagnosis will be final and considered independent to the output of the algorithm.

Study Overview

Status

Completed

Conditions

Intervention / Treatment

Detailed Description

The investigators are targeting to study the effect of their Artificial Intelligence algorithm on the radiologists' estimation of skeletal age. Currently, radiologists make the estimation using only the radiographic images and health records. As part of this study, the radiologists will estimate skeletal age from radiographic images, health records, and the output of the CADx algorithm. The investigators wish to understand how radiologists using the Artificial Intelligence algorithm compare to radiologists who do not for the specific task of estimating skeletal age.

This study is organized as a multi-institutional randomized control trial with two arms - experiment (receiving the Artificial Intelligence algorithm's output) and control (no intervention). Both of these arms will be compared to a clinical reference standard ("gold standard") composed of a panel of radiologists. The metric of comparison will be Mean Absolute Distance (MAD). The investigators plan to use statistical tests such as the t-test to determine any statistically-significant difference in skeletal age estimation between the two groups.

The investigators have recruited and analyzed data from a sample size of 1600 exams. Patients getting these exams will not undergo any research procedures that deviate from the current standard practices.

Study Type

Interventional

Enrollment (Actual)

1903

Phase

  • Not Applicable

Contacts and Locations

This section provides the contact details for those conducting the study, and information on where this study is being conducted.

Study Locations

  • United States
    • California
      • Stanford, California, United States, 94305
        • Stanford University
    • Connecticut
      • New Haven, Connecticut, United States, 06519
        • Yale New Haven Hospital
    • Massachusetts
      • Boston, Massachusetts, United States, 02115
        • Boston Children's Hospital
    • New York
      • New York, New York, United States, 10016
        • New York University
    • Ohio
      • Cincinnati, Ohio, United States, 45229
        • Cincinnati Children's Hospital Medical Center
    • Pennsylvania
      • Philadelphia, Pennsylvania, United States, 19104
        • Children's Hospital of Philadelphia

Participation Criteria

Researchers look for people who fit a certain description, called eligibility criteria. Some examples of these criteria are a person's general health condition or prior treatments.

Eligibility Criteria

Ages Eligible for Study

  • Child
  • Adult
  • Older Adult

Accepts Healthy Volunteers

No

Genders Eligible for Study

All

Description

Exams that meet the following inclusion criteria will be included: (1) exams read by radiologists who interpret pediatric skeletal age exams and verbally consent to participate (2) exams that contain a procedure code or study description indicative of a skeletal age exam.

Exams containing more than one radiograph will not be included. Exams for which a trainee provides a preliminary interpretation will be excluded. No further exclusion criteria will be applied on the basis of image quality metrics or manufacturers. No exclusion criteria will be applied on the basis of patient chronological age.

Study Plan

This section provides details of the study plan, including how the study is designed and what the study is measuring.

How is the study designed?

Design Details

  • Primary Purpose: Diagnostic
  • Allocation: Randomized
  • Interventional Model: Parallel Assignment
  • Masking: None (Open Label)

Arms and Interventions

Participant Group / Arm
Intervention / Treatment
No Intervention: Control (Without-AI)
This is the control arm where no intervention is provided; represents current standard of care.
Experimental: Experiment (With-AI)
This is the experiment arm where the intervention, "BoneAgeModel", is provided. The participating radiologists in this arm will receive the output of the Artificial Intelligence algorithm. They will be asked to incorporate this new information with their normal workflows to make a diagnosis. The radiologists' diagnosis will be considered final.
Device: BoneAgeModel
BoneAgeModel is an Artificial Intelligence tool that takes in a hand radiograph and gender, and outputs the skeletal (bone) age. The intervention involves using this tool as a factor in the clinical decision making process of the participating radiologists. The radiologist's decision will be considered final.

What is the study measuring?

Primary Outcome Measures

Outcome Measure
Measure Description
Time Frame
Paired Difference of Skeletal Age Estimate
Time Frame: Up to 10 minutes to acquire the scan; up to 2 days to complete diagnosis review
Mean absolute difference between dictated final impressions (baseline measure by Radiologist) and the consensus determination of a panel of radiologists following review.
Up to 10 minutes to acquire the scan; up to 2 days to complete diagnosis review

Secondary Outcome Measures

Outcome Measure
Measure Description
Time Frame
Time for Diagnosis
Time Frame: Up to approximately 4 minutes
Amount of time taken by radiologists when using the BoneAgeModel as compared to when they are not.
Up to approximately 4 minutes

Collaborators and Investigators

This is where you will find people and organizations involved with this study.

Sponsor

Stanford University

Investigators

  • Study Chair: Curtis Langlotz, M.D. Ph.D., Stanford University
  • Study Director: David Eng, B.S., Stanford University
  • Study Director: Nishith Khandwala, B.S., Stanford University
  • Principal Investigator: Safwan Halabi, M.D., Stanford University

Study record dates

These dates track the progress of study record and summary results submissions to ClinicalTrials.gov. Study records and reported results are reviewed by the National Library of Medicine (NLM) to make sure they meet specific quality control standards before being posted on the public website.

Study Major Dates

Study Start (Actual)

July 12, 2018

Primary Completion (Actual)

August 31, 2019

Study Completion (Actual)

August 31, 2019

Study Registration Dates

First Submitted

May 7, 2018

First Submitted That Met QC Criteria

May 17, 2018

First Posted (Actual)

May 21, 2018

Study Record Updates

Last Update Posted (Actual)

June 9, 2021

Last Update Submitted That Met QC Criteria

May 24, 2021

Last Verified

May 1, 2021

More Information

Terms related to this study

Keywords

Other Study ID Numbers

  • IRB #44764

Plan for Individual participant data (IPD)

Plan to Share Individual Participant Data (IPD)?

Yes

IPD Plan Description

Individual participant data that underlie the results reported in this article after deidentification (text, tables, figures and appendices).

IPD Sharing Time Frame

Beginning 3 months and ending 5 years following article publication.

IPD Sharing Access Criteria

Researchers who provide a methodologically sound proposal.

IPD Sharing Supporting Information Type

  • Study Protocol
  • Statistical Analysis Plan (SAP)
  • Analytic Code

Drug and device information, study documents

Studies a U.S. FDA-regulated drug product

No

Studies a U.S. FDA-regulated device product

Yes

product manufactured in and exported from the U.S.

Yes

This information was retrieved directly from the website clinicaltrials.gov without any changes. If you have any requests to change, remove or update your study details, please contact register@clinicaltrials.gov. As soon as a change is implemented on clinicaltrials.gov, this will be updated automatically on our website as well.

Subscribe