Human-AI Collaboration for Ultrasound Diagnosis of Thyroid Nodules - a Clinical Trial

This is an experimental study wherein groups of medical students and physicians of varying degrees of experience in head-and-neck ultrasound were asked to scan the same five patients each with a thyroid nodule.

The study participants did their own ultrasound assessment of the thyroid nodules, as well as using an AI-based ultrasound diagnostics system.

The researchers intended to study two primary outcomes: 1) how varying degrees of experience in ultrasound by the operator might affect the diagnostic performance of the AI-based system, and 2) how the AI-based system influenced the diagnostic performance of the ultrasound operator.

Study Overview

• Status: Completed
• Conditions: Thyroid Nodule
• Intervention / Treatment: Diagnostic test: S-Detect for Thyroid

Detailed Description

This is a prospective clinical study aiming to test how the experience of the ultrasound operator influences the performance of AI-based (artificial intelligence-based) diagnostics when analysing thyroid nodules on ultrasound scans. The investigators set up an experiment with five stations, each with a patient with a thyroid nodule and an ultrasound machine with the deep learning based system S-Detect for Thyroid installed. 20 study participants where recruited: 8 medical students of novice ultrasound skill, 3 junior ENT (ear-nose-throat) registrars of intermediate ultrasound skill, and 9 senior ENT registrars experienced in ultrasound. The participants scanned all the patients and recorded their analyses of the nodules using the EUTIRADS (European thyroid imagining reporting and data system) system in three different ways: a analysis of their own, S-Detect's analysis, and an analysis combining the two previous.

The hypothesis was that the AI system would perform equally well when between the participant groups. In addition, it was expected that the experienced participants would perform better than the students without AI help, and that the doctors would gain little from AI input, but that the students would have their performance improved by AI input.

• Study Type: Interventional
• Enrollment (Actual): 20
• Phase: Not Applicable

Contacts and Locations

Study Locations

• Denmark
  • Copenhagen, Denmark, 2100
    • Rigshospitalet

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: Child; Adult; Older Adult
• Accepts Healthy Volunteers: Yes

Description

Medical students

Inclusion Criteria:

• Last year student

Exclusion Criteria:

• Experience with ultrasound beyond that which is taught at the University of Copenhagen

Junior ENT registrar doctors

Inclusion Criteria:

• Doctor enrolled in introductory training as ENT physician.

Senior ENT registrar doctors

Inclusion Criteria:

• Doctor enrolled in ENT training.

Study Plan

How is the study designed?

Design Details

• Primary Purpose: Diagnostic
• Allocation: N/A
• Interventional Model: Single Group Assignment
• Masking: None (Open Label)

Number of Arms

1

Arms and Interventions

Participant Group / Arm

Intervention / Treatment

Experimental: Experiment; 20 participants ultrasound scan five patients with thyroid nodules, and assess these nodules themselves, then with the AI-program, and at last they give a combined assessment.

Diagnostic test: S-Detect for Thyroid; Deep learning based program on Samsung ultrasound machines designed to do real-time semi-automated analysis of thyroid nodules. The ultrasound operator freezes a transverse image of the patient's thyroid nodule and activates S-Detect. The operator selects the nodule on the screen, and the program automatically draws a region of interest. Then S-Detect gives a dichotomous diagnosis of either "Possibly benign" and "Possibly malignant". In addition, it measures the nodule and characterises it with a lexicon based on EUTIRADS.

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Accuracy of S-Detect diagnosis	Number of correct thyroid nodule malignancy diagnoses out of total malignancy diagnoses by the AI-based ultrasound diagnostic system "S-Detect" on the five patients' thyroid nodules. Gold standard is cytology and histology of the nodules.	1 day (day of experiment)

Secondary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Accuracy of biopsy recommendation	Number of correct biopsy recommendations on the five patients' thyroid nodules. Recommendation is derived from EUTIRADS analyses done by participants with and without AI assistance. Gold standard is biopsy recommendation derived from expert consensus EUTIRADS assessment of the nodules.	1 day (day of experiment)
Nodule measurement	Measurement of the five patients' thyroid nodules done by participants and S-Detect. Gold standard are measurements from expert consensus assessment analysis of the nodules.	1 day (day of experiment)
OSAUS score	Mean OSAUS (objective structured assessment of ultrasound skills) scores of participants as assessed from their ultrasound scans of the five patients. Assessment are independently by two head-and-neck ultrasound experts.	1 day (day of experiment)

Collaborators and Investigators

• Sponsor: Rigshospitalet, Denmark
• Investigators: Study Director: Tobias Todsen, Ph.d, Rigshospitalet, Denmark

Study record dates

Study Major Dates

• Study Start (Actual): September 1, 2023
• Primary Completion (Actual): November 4, 2023
• Study Completion (Actual): November 4, 2023

Study Registration Dates

• First Submitted: February 23, 2024
• First Submitted That Met QC Criteria: March 5, 2024
• First Posted (Actual): March 12, 2024

Study Record Updates

• Last Update Posted (Estimated): November 25, 2024
• Last Update Submitted That Met QC Criteria: November 20, 2024
• Last Verified: November 1, 2024

More Information

Terms related to this study

• Keywords: Ultrasound; AI; Thyroid cancer
• Additional Relevant MeSH Terms: Endocrine System Diseases; Neoplasms by Site; Neoplasms; Thyroid Neoplasms; Endocrine Gland Neoplasms; Head and Neck Neoplasms; Thyroid Diseases; Thyroid Nodule
• Other Study ID Numbers: Thyroid AI US operator exp

Plan for Individual participant data (IPD)

• Plan to Share Individual Participant Data (IPD)?: NO

Drug and device information, study documents

• Studies a U.S. FDA-regulated drug product: No
• Studies a U.S. FDA-regulated device product: No