Evaluating Decision-making Using ChatGPT-4 Among Trainees in Surgery (EDuCATe)

Evaluating ChatGPT-4 as a Decision-Making Support Tool for Surgical Trainees

This study aims to assess whether ChatGPT-4 can support surgical trainees in clinical decision-making. By comparing the performance of ChatGPT-4 with junior residents, senior residents, and attending surgeons on standardized clinical scenarios, the study seeks to understand the potential role of large language models in surgical education. The ultimate goal is to evaluate whether ChatGPT-4 can be safely integrated as a supplementary educational tool to aid junior residents in developing critical thinking and surgical judgment.

Study Overview

• Status: Not yet recruiting
• Conditions: Surgery; Training; Artificial Intelligence (AI)
• Intervention / Treatment: Other: Clinical Cases

Detailed Description

Background:

Artificial Intelligence (AI) is rapidly transforming the medical landscape, offering new possibilities in education, diagnostics, and decision support. In surgery, clinical decision-making is a core competency developed progressively through training. ChatGPT-4, a state-of-the-art large language model developed by OpenAI, has demonstrated competence in handling medical queries and clinical reasoning tasks. However, its performance in complex surgical decision-making compared to human trainees remains largely unexplored.

Objective:

The EDuCATe study aims to evaluate the accuracy and reliability of ChatGPT-4's responses to clinical scenarios involving general surgery cases. Specifically, the study compares the model's performance to that of junior residents, senior residents, and attending surgeons to understand if ChatGPT-4 can serve as a safe and effective educational tool for surgical trainees.

Methods:

Seven clinical scenarios will be constructed using real anonymized patient data representing common general surgery conditions. Each case will be presented step-by-step, mimicking the clinical decision-making process. Participants will answer a question related to treatment choice.

Participants will include junior residents (PGY1-2), senior residents (PGY3+), and attending surgeons from a single surgical department. ChatGPT-4 will be prompted with the same scenarios. All participants will be instructed to complete the cases without using external resources such as AI tools or internet searches, relying solely on their clinical knowledge.

Statistical analysis will compare performance across groups using non-parametric tests (e.g., Wilcoxon rank sum).

Expected Outcomes:

The study hypothesizes that ChatGPT-4 will perform at a level comparable to senior residents or attending surgeons and outperform junior residents in decision-making. If confirmed, these results could support the safe use of ChatGPT-4 as a training aid for junior surgical residents, potentially improving educational outcomes and clinical reasoning skills.

Significance:

This study will provide novel insight into the role of AI in surgical education. By rigorously comparing ChatGPT-4's decision-making capabilities to that of human surgeons at various levels, the study hopes to define its utility, limitations, and appropriate use in residency training programs.

• Study Type: Observational
• Enrollment (Estimated): 35

Contacts and Locations

• Study Contact: Name: Manuela Mastronardi; Phone Number: 0403994152; Email: manuela.mastronardi@gmail.com

Study Locations

• Italy
  • Trieste, Italy
    • University of Trieste
    • Contact: Manuela Mastronardi; Phone Number: 0403994152; Email: manuela.mastronardi@gmail.com
    • Principal Investigator: Silvia Palmisano
    • Principal Investigator: Manuela Mastronardi
    • Sub-Investigator: Paola Germani
    • Sub-Investigator: Margherita Sandano

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: Child; Adult; Older Adult
• Accepts Healthy Volunteers: No

• Sampling Method: Non-Probability Sample

Study Population

The study population will consist of general surgery trainees and faculty members from a single academic surgical department. Participants will be stratified into three groups based on their level of training and experience: Junior Residents: Postgraduate Year (PGY) 1-2; Senior Residents: PGY 3 and above; Attending Surgeons: Board-certified general surgeons with independent clinical practice

Description

Inclusion Criteria:

• Actively enrolled or employed in the general surgery residency or department at the participating institution
• Willingness to participate and complete all clinical case scenarios
• Consent to participate in the study

Exclusion Criteria:

• Incomplete responses
• Use of external assistance (e.g., internet search, AI tools) when answering scenarios, as self-reported in instructions

Study Plan

How is the study designed?

Number of groups / cohorts

3

Cohorts and Interventions

Group / Cohort	Intervention / Treatment
Residents; Junior and Senior Residents in General Surgery	Other: Clinical Cases; Seven clinical cases that have to be analysed
Consultants; Senior General Surgeons	Other: Clinical Cases; Seven clinical cases that have to be analysed
ChatGPT 4; Artificial Intelligence System	Other: Clinical Cases; Seven clinical cases that have to be analysed

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Proportion of correct responses	Binary outcome (correct vs. incorrect decision)	Baseline

Secondary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Comparison of accuracy across experience levels	Proportion of correct responses by group: Junior residents, Senior residents, Attending surgeons, ChatGPT-4	Baseline
Confidence level	Participants and ChatGPT are asked to rate how confident they feel in their answer (1-5 Likert scale, where 1 means no confident and 5 very confident)	Baseline
Percentage of use of AI for clinical cases evaluation	Participants are asked if they use or not ChatGPT in their clinical activity	Baseline

Collaborators and Investigators

• Sponsor: Ospedali Riuniti Trieste

Study record dates

Study Major Dates

• Study Start (Estimated): April 10, 2025
• Primary Completion (Estimated): April 25, 2025
• Study Completion (Estimated): April 30, 2025

Study Registration Dates

• First Submitted: April 2, 2025
• First Submitted That Met QC Criteria: April 8, 2025
• First Posted (Actual): April 10, 2025

Study Record Updates

• Last Update Posted (Actual): April 10, 2025
• Last Update Submitted That Met QC Criteria: April 8, 2025
• Last Verified: April 1, 2025

More Information

Terms related to this study

• Keywords: oncology; surgery; emergency; artificial intelligence; training
• Other Study ID Numbers: AI-Surgical training

Plan for Individual participant data (IPD)

• Plan to Share Individual Participant Data (IPD)?: NO

Drug and device information, study documents

• Studies a U.S. FDA-regulated drug product: No
• Studies a U.S. FDA-regulated device product: No