The Effects of a Large Language Model on Clinical Questioning Skills

November 19, 2024 updated by: Haotian Lin, Sun Yat-sen University

A Randomized Controlled Trial of the Effects of a Large Language Model on Medical Students' Clinical Questioning Skills

The researchers have used the ophthalmology textbook, clinical guideline consensus, the Internet conversation data and knowledge base of Zhongshan Ophthalmology Center in the early stage, combined with artificial feedback reinforcement learning and other techniques to fine-tune and train the LLM, and developed "Digital Twin Patient", a localized large language model that has the ability to answer ophthalmology-related medical questions, and also constructed a combination of automated model evaluation and manual evaluation by medical experts. The evaluation system combining automated model evaluation and manual evaluation by medical experts was constructed at the same time.

This project intends to integrate "Digital Twin Patient" into undergraduate ophthalmology apprenticeship, simulate the consultation process of real patients through the online interaction between students and "Digital Twin Patient", explore the effect of "Digital Twin Patient" consultation teaching, provide emerging technology tools for guiding medical students to actively learn a variety of ophthalmology cases, cultivate clinical thinking, and provide the possibility of creating a new mode of intelligent teaching.

Study Overview

Status

Completed

Conditions

Intervention / Treatment

Detailed Description

At present, the main form of clinical questioning skills teaching is to let undergraduates who participate in the apprenticeship first learn the characteristics and diagnostic points of cases, and then practice questioning on real patients in the wards. However, due to the large number of trainee students, it is difficult to meet the teaching demand in terms of the number of cases available for questioning and the richness of disease types under the current teaching mode. Therefore, it is necessary to utilize new intelligent technologies and create a new model of questioning skills teaching to improve teaching efficiency and enhance students' clinical thinking.

Large-scale language modeling (LLM) is a deep learning technology that can learn knowledge from a large amount of text, and AI chatbots such as ChatGPT are a typical example of its application. AI chatbots are characterized by anthropomorphic comprehension and diversified natural language generation abilities in different contexts, and have been initially applied in the medical field, such as passing the U.S. Medical Licensing Examination, assisting in ophthalmic history documentation and answering ophthalmic questions. However, it has been found that although LLM has fair modeling performance in general medical knowledge, it still needs to be improved in the area of specialty diseases. Based on this, the researcher's team has used the ophthalmology textbook, clinical guideline consensus, the Internet conversation data and knowledge base of Zhongshan Ophthalmology Center in the early stage, combined with artificial feedback reinforcement learning and other techniques to fine-tune and train the LLM, and developed "Digital Twin Patient", a localized large language model that has the ability to answer ophthalmology-related medical questions, and also constructed a combination of automated model evaluation and manual evaluation by medical experts. The evaluation system combining automated model evaluation and manual evaluation by medical experts was constructed at the same time.

This project intends to integrate "Digital Twin Patient" into undergraduate ophthalmology apprenticeship, simulate the consultation process of real patients through the online interaction between students and "Digital Twin Patient", explore the effect of "Digital Twin Patient" consultation teaching, provide emerging technology tools for guiding medical students to actively learn a variety of ophthalmology cases, cultivate clinical thinking, and provide the possibility of creating a new mode of intelligent teaching.

Study Type

Interventional

Enrollment (Actual)

84

Phase

  • Not Applicable

Contacts and Locations

This section provides the contact details for those conducting the study, and information on where this study is being conducted.

Study Locations

  • China
    • Guangdong
      • Guangzhou, Guangdong, China, 510060
        • Zhongshan Ophthalmic Center, Sun Yat-sen Univerisity

Participation Criteria

Researchers look for people who fit a certain description, called eligibility criteria. Some examples of these criteria are a person's general health condition or prior treatments.

Eligibility Criteria

Ages Eligible for Study

  • Adult

Accepts Healthy Volunteers

Yes

Description

Inclusion Criteria:

  • All undergraduate students from Sun Yat-sen University who participate in the ophthalmological internship.

Exclusion Criteria:

  • Students who refuse to sign informed consent.

Study Plan

This section provides details of the study plan, including how the study is designed and what the study is measuring.

How is the study designed?

Design Details

  • Primary Purpose: Other
  • Allocation: Randomized
  • Interventional Model: Parallel Assignment
  • Masking: None (Open Label)

Arms and Interventions

Participant Group / Arm
Intervention / Treatment
Experimental: "Digital twin patient"
The students in the "Digital twin patient" group were trained in history taking using a "digital twin patient" on the first day of a training program, and then took a 15-minute clinical questioning exam using the "digital twin patient" on the second day.
Device: "Digital twin patient"
"Digital twin patient" can serve as patients with specific diseases for medical students to acquire disease history and thus practice clinical questioning skills.
Other: Real patient
The students in the Real patient group were trained in history taking using a real patient on the first day of a training program, and then took a 15-minute clinical questioning exam using the "digital twin patient" on the second day.
Device: "Digital twin patient"
"Digital twin patient" can serve as patients with specific diseases for medical students to acquire disease history and thus practice clinical questioning skills.
Behavioral: Interaction with real patients
As in traditional medical education, medical students need to interact with real patients to practice history collection skills.

What is the study measuring?

Primary Outcome Measures

Outcome Measure
Time Frame
Students' scores in the medical history acquisition exam
Time Frame: Weekly during this study (up to 10 months)
Weekly during this study (up to 10 months)

Collaborators and Investigators

This is where you will find people and organizations involved with this study.

Sponsor

Sun Yat-sen University

Study record dates

These dates track the progress of study record and summary results submissions to ClinicalTrials.gov. Study records and reported results are reviewed by the National Library of Medicine (NLM) to make sure they meet specific quality control standards before being posted on the public website.

Study Major Dates

Study Start (Actual)

November 13, 2023

Primary Completion (Actual)

May 10, 2024

Study Completion (Actual)

August 7, 2024

Study Registration Dates

First Submitted

January 3, 2024

First Submitted That Met QC Criteria

January 19, 2024

First Posted (Actual)

January 29, 2024

Study Record Updates

Last Update Posted (Estimated)

November 22, 2024

Last Update Submitted That Met QC Criteria

November 19, 2024

Last Verified

February 1, 2024

More Information

Terms related to this study

Keywords

Additional Relevant MeSH Terms

Other Study ID Numbers

  • 2023KYPJ283

Plan for Individual participant data (IPD)

Plan to Share Individual Participant Data (IPD)?

NO

Drug and device information, study documents

Studies a U.S. FDA-regulated drug product

No

Studies a U.S. FDA-regulated device product

No

product manufactured in and exported from the U.S.

No

This information was retrieved directly from the website clinicaltrials.gov without any changes. If you have any requests to change, remove or update your study details, please contact register@clinicaltrials.gov. As soon as a change is implemented on clinicaltrials.gov, this will be updated automatically on our website as well.

Clinical Trials on Glaucoma

Clinical Trials on "Digital twin patient"

Search Similar Trials

Sponsors and Collaborators

Medical Conditions

Drug Interventions

CROs by country

CROs in Ukraine

Conditions

Rare Diseases

Drug Interventions

Dietary Supplements

Sponsor/Collaborators

Locations

Subscribe