Physician Reasoning on Management Cases With Large Language Models

Management Reasoning With AI Chat Bots

This study will evaluate the effect of providing access to GPT-4, a large language model, compared to traditional management decision support tools on performance on case-based management reasoning tasks.

Study Overview

• Status: Completed
• Conditions: Clinical Decision-making
• Intervention / Treatment: Other: GPT-4

Detailed Description

Artificial intelligence (AI) technologies, specifically advanced large language models like OpenAI's ChatGPT, have the potential to improve medical decision-making. Although ChatGPT-4 was not developed for its use in medical-specific applications, it has demonstrated promise in various healthcare contexts, including medical note-writing, addressing patient inquiries, and facilitating medical consultation. However, little is known about how ChatGPT augments the clinical reasoning abilities of clinicians.

Clinical reasoning is a complex process involving pattern recognition, knowledge application, and probabilistic reasoning. Integrating AI tools like ChatGPT-4 into physician workflows could potentially help reduce clinician workload and decrease the likelihood of mismanagement. However, ChatGPT-4 was not developed for clinical reasoning nor has it been validated for this purpose. Further, it may be subject to disinformation, including convincing confabulations that may mislead clinicians. If clinicians misuse this tool, it may not improve reasoning and could even cause harm. Therefore, it is important to study how clinicians use large language models to augment clinical reasoning prior to routine incorporation into patient care.

In this study, participants will be randomized to answer clinical management cases with or without access to ChatGPT-4. Each case has multiple components, and the participants will be asked to discuss their reasoning for each component. Answers will be graded by independent reviewers blinded to treatment assignment. A grading rubric was developed for each case by a panel of 4-7 expert discussants. Discussants independently developed a rubric for each case, and then any discrepancies were resolved through multiple rounds of discussions.

• Study Type: Interventional
• Enrollment (Actual): 92
• Phase: Not Applicable

Contacts and Locations

Study Locations

• United States
  • California
    • Palo Alto, California, United States, 94304
      • Stanford University

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: Child; Adult; Older Adult
• Accepts Healthy Volunteers: Yes

Description

Inclusion Criteria:

• Participants must be licensed physicians and have completed at least post-graduate year 2 (PGY2) of medical training.
• Training in Internal medicine, family medicine, or emergency medicine.

Exclusion Criteria:

• Not currently practicing clinically.

Study Plan

How is the study designed?

Design Details

• Primary Purpose: Treatment
• Allocation: Randomized
• Interventional Model: Parallel Assignment
• Masking: Single

Number of Arms

2

Arms and Interventions

Participant Group / Arm	Intervention / Treatment
Active Comparator: GPT-4; Group will be given access to GPT-4	Other: GPT-4; OpenAI's GPT-4 large language model with chat interface.
No Intervention: Usual Resources; Group will not be given access to GPT-4 but will be encouraged to use any resources they wish besides large language models (UpToDate, Dynamed, google, etc).

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Management Reasoning	Percent correct (range: 0 to 100) for each case.	Within one-hour study

Secondary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Time Spent on Management	Time (in minutes) participants spend per case between the two study arms.	Within one-hour study

Collaborators and Investigators

• Sponsor: Stanford University
• Collaborators: Beth Israel Deaconess Medical Center; University of Minnesota
• Investigators: Principal Investigator: Jonathan H Chen, MD, PhD, Stanford University; Principal Investigator: Adam Rodman, MD, Beth Israel Deaconess Medical Center; Principal Investigator: Andrew Olson, MD, University of Minnesota

Study record dates

Study Major Dates

• Study Start (Actual): December 28, 2023
• Primary Completion (Actual): April 19, 2024
• Study Completion (Actual): April 19, 2024

Study Registration Dates

• First Submitted: December 20, 2023
• First Submitted That Met QC Criteria: January 5, 2024
• First Posted (Actual): January 17, 2024

Study Record Updates

• Last Update Posted (Actual): September 27, 2024
• Last Update Submitted That Met QC Criteria: September 26, 2024
• Last Verified: September 1, 2024

More Information

Terms related to this study

• Keywords: clinical decision support
• Other Study ID Numbers: 71319b

Plan for Individual participant data (IPD)

• Plan to Share Individual Participant Data (IPD)?: NO

Drug and device information, study documents

• Studies a U.S. FDA-regulated drug product: No
• Studies a U.S. FDA-regulated device product: No