Generating Fast and Slow for Entree Level Medical Knowledge

The generative artificial intelligence tool, ChatGPT, has garnered widespread interest since its launch. This innovative multimedia platform has the potential to enhance medical communication and health education, thereby improving medical accessibility and reducing the burden on healthcare professionals. Some studies have indicated that ChatGPT achieves higher levels of satisfaction in counseling compared to human healthcare professionals. Additionally, research has shown that ChatGPT's performance in answering objective structured clinical examination questions is comparable to that of typical medical students. However, in both scenarios, it still requires editing by professionals before being used. Moreover, a recent meta-analysis evaluating ChatGPT's ability in various types of medical exams revealed inconsistent results. Before its application and actual integration into clinical practice, the investigators need to comprehend the advantages, disadvantages, and relevant limitations of ChatGPT in the field of medical communication. This study aims to simulate virtual consultations between ChatGPT, acting as a health professional, and study participants, serving as patients. It will evaluate the participants' satisfaction with the virtual consultation questions, categorized by different levels of cognition through Bloom's Taxonomy. This study plans to recruit medical professionals, healthcare-related professionals, medical students from National Taiwan University Hospital, and the general public. Two researchers will select 20 questions from the USMLE step 3 practice tests and the second stage of the Taiwan Medical Licensing Examination. These questions will be categorized into different levels of cognition based on knowledge, comprehension, application, analysis, synthesis, or evaluation. ChatGPT 3.5 and 4.0 will answer these 20 questions without specifying the question's level of cognition. Each participant will review the answers to these 20 questions and assign a satisfaction score based on the appropriateness of each answer.

Study Overview

• Status: Not yet recruiting
• Conditions: Patient Satisfaction
• Intervention / Treatment: Other: Answers generated by ChatGPT

• Study Type: Observational
• Enrollment (Estimated): 120

Contacts and Locations

Study Locations

• Taiwan
  • Taipei, Taiwan, 100
    • National Taiwan University
    • Contact: Kuanju Huang, MS; Phone Number: 63413 +886-5-5323911; Email: Y06058@ms1.ylh.gov.tw
    • Contact: Kuanju Huang; Phone Number: +886-972655731; Email: restroomer@icloud.com

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: Adult; Older Adult
• Accepts Healthy Volunteers: Yes

• Sampling Method: Non-Probability Sample

Study Population

The medical professionals, healthcare-related professionals, and medical students refer to attending physician, medical radiation technologist from National Taiwan University Hospital, and medical students from National Taiwan University College of Medicine, respectively. The general public refer to volunteers from Taiwan Emergency Medical Technician Association.

Description

Inclusion Criteria:

1. National Taiwan University Hospital

   • Medical professionals
   • Healthcare-related professionals
   • Medical students

2. Taiwan Emergency Medical Technician Association.

   • Volunteers

Exclusion Criteria:

• N/A.

Study Plan

How is the study designed?

Number of groups / cohorts

4

Cohorts and Interventions

Group / Cohort	Intervention / Treatment
Medical professionals; Attending physicians of National Taiwan University Hospital	Other: Answers generated by ChatGPT; Participants will assign a satisfaction score based on the appropriateness of each answer.
Healthcare-related professionals; Medical Radiation Technologist of National Taiwan University Hospital	Other: Answers generated by ChatGPT; Participants will assign a satisfaction score based on the appropriateness of each answer.
Medical students; Medical students of National Taiwan University College of Medicine who have passed the first stage of Taiwan Medical Licencing Exam.	Other: Answers generated by ChatGPT; Participants will assign a satisfaction score based on the appropriateness of each answer.
General public; Volunteers from Taiwan Emergency Medical Technician Association	Other: Answers generated by ChatGPT; Participants will assign a satisfaction score based on the appropriateness of each answer.

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Correlation between satisfaction and Bloom's taxonomy	Divided by the four participant groups	baseline (at completion of study questionnaire)

Secondary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Correctness of answers	Answers from ChatGPT 3.5/4.0 and the answers provided by Medical Licencing Exam institutions	baseline (at completion of study questionnaire)

Collaborators and Investigators

• Sponsor: National Taiwan University Hospital
• Collaborators: Taiwan Emergency Medical Technician Association

Study record dates

Study Major Dates

• Study Start (Estimated): February 1, 2024
• Primary Completion (Estimated): June 1, 2024
• Study Completion (Estimated): June 1, 2024

Study Registration Dates

• First Submitted: January 29, 2024
• First Submitted That Met QC Criteria: February 6, 2024
• First Posted (Actual): February 8, 2024

Study Record Updates

• Last Update Posted (Actual): February 16, 2024
• Last Update Submitted That Met QC Criteria: February 14, 2024
• Last Verified: January 1, 2024

More Information

Terms related to this study

• Keywords: Artificial Intelligence; Decision Making, Shared
• Other Study ID Numbers: 202308059RINB

Plan for Individual participant data (IPD)

• Plan to Share Individual Participant Data (IPD)?: NO
• IPD Plan Description: Data would be shared by request.

Drug and device information, study documents

• Studies a U.S. FDA-regulated drug product: No
• Studies a U.S. FDA-regulated device product: No