Improving AI-Assisted Medical Diagnosis and Triage by the General Public

This study is a randomized controlled trial (RCT) investigating whether access to a new LLM interface can improve medical triage and diagnostic accuracy for laypeople compared to access to a standard LLM interface. It addresses previous findings where laypeople using standard LLMs performed worse than those using conventional methods (e.g., web search) due to incomplete symptom sharing and poor interpretation of AI advice. To address this, the research tests a structured LLM system that proactively asks clinical history questions before providing a standardized, easy-to-read diagnostic output.

Study Overview

• Status: Recruiting
• Conditions: Large Language Models; AI-Assisted Diagnosis
• Intervention / Treatment: Device: new GPT-4o Interface; Other: Control

Detailed Description

As large language models (LLMs) become widely accessible, a growing share of the public already turns to AI-powered chatbots for health-related information: surveys suggest that one in six American adults consults AI chatbots for health queries at least once a month. At the same time, diagnostic errors and misplaced triage decisions represent a persistent source of preventable patient harm globally [4]. This has prompted considerable interest in whether LLMs can serve as a reliable "front door" to the healthcare system for patients who lack immediate access to a clinician.

However, incidents involving misguided medical suggestions from LLMs to the general public such as fatal overdose and erroneous diagnosis leading to life-threatening treatment delay have been reported. A recent controlled study shows LLMs perform significantly worse for medical assistance in real-world settings compared to their performance on controlled benchmarks. Bean et al. [1] conducted a large preregistered study with 1,298 UK participants in which laypeople were assigned to receive assistance from one of three popular LLMs (GPT-4o, Llama 3, or Command R+) or to use a source of their choice when assessing ten standardized medical scenarios. Although the LLMs alone correctly identified relevant conditions in up to 94.9% of cases, participants using those same LLMs did so in fewer than 34.5% of cases, significantly worse than the group using their typical home resources (∼60%). Similarly, triage accuracy showed no signifcant difference between LLM users and controls, with an overall correct response rate of 43.0% across all groups. This poor performance can be linked to two main failure modes: users providing incomplete symptom information to the AI, and users failing to correctly interpret or act upon the LLM's output.

To address these failure modes, the study tests a new GPT-4o interface wrapped with a fixed system prompt. Rather than passively responding to whatever the user volunteers, the AI is instructed to ask clarifying questions to gather a proper clinical history before offering any medical suggestions. Once the model judges that enough information is gathered, it provides a structured, easy-to-read response detailing possible conditions, their likelihoods, and a clear triage recommendation.

The trial is structured as a two-arm, single-blind RCT. Both arms retain access to whatever assistance methods participants would normally use at home (e.g., web search), and both arms additionally get a GPT-4o-based LLM interface. The difference lies in which interface: the treatment arm receives the new interface described above, while the control arm receives a standard, unmodified GPT-4o interface with no special prompting. Participants are restricted to using the provided LLM interface (GPT-4o) only, and not other LLMs. AI-overview in web searches will be disabled through an extension. The trial utilizes ten previously validated clinical vignettes covering various medical urgencies, ranging from self-care routines up to ambulance-level emergencies, with each participant randomly assigned two of the ten. To achieve sufficient statistical power (accounting for within-participant clustering across those two responses), the target sample size is calculated at 220 total participants, split evenly with 110 individuals per arm.

The participant pool is drawn from the enrolled students and administrative staff at the Lahore University of Management Sciences (LUMS) in Pakistan. To ensure the sample consists strictly of laypeople, anyone currently enrolled in or who has completed a medical, nursing, or allied health professional degree is explicitly excluded from participating.

The study is driven by two co-primary hypotheses: participants with access to the new LLM interface will identify relevant medical conditions at a higher rate than participants using the standard LLM interface, and they will correctly assess the urgency of the medical scenarios at a higher rate. Accuracy is measured by comparing participant responses to a physician-generated gold-standard list using fuzzy matching, then analyzed via mixed-effects logistic regression with participant-level covariates and a Bonferroni-corrected significance threshold. The study also tracks secondary outcomes - self-reported confidence and time spent per scenario - and several exploratory analyses, including an LLM-alone benchmark, moderation by LLM experience, internet usage patterns, digital literacy, and scenario-level variation in accuracy.

• Study Type: Interventional
• Enrollment (Estimated): 220
• Phase: Not Applicable

Contacts and Locations

• Study Contact: Name: Ihsan Ayyub Qazi, PhD; Phone Number: +923233333766; Email: ihsan.qazi@lums.edu.pk
• Study Contact Backup: Name: Ayesha Ali, PhD; Phone Number: 04235608368; Email: ayeshaali@lums.edu.pk

Study Locations

• Pakistan
  • Punjab Province
    • Lahore, Punjab Province, Pakistan, 54000
      • Recruiting
      • Lahore University of Management Sciences
      • Contact: Ihsan Ayyub Qazi, PhD; Phone Number: +923233333766; Email: ihsan.qazi@lums.edu.pk
      • Contact: Ayesha Ali, PhD; Phone Number: 04235608368; Email: ayeshaali@lums.edu.pk

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: Adult; Older Adult
• Accepts Healthy Volunteers: Yes

Description

Inclusion Criteria:

• Enrolled student or employed administrative staff at LUMS.
• 18 years of age or older.
• Able to read and understand English.

Exclusion Criteria:

• Individuals with any formal education or professional training in medicine, nursing, or any other healthcare profession.

Study Plan

How is the study designed?

Design Details

• Primary Purpose: Diagnostic
• Allocation: Randomized
• Interventional Model: Parallel Assignment
• Masking: Single

Number of Arms

2

Arms and Interventions

Participant Group / Arm	Intervention / Treatment
Experimental: Treatment Arm (new GPT-4o Interface); Participants can access any assistance methods they would typically employ (e.g., web search or health portals) in addition to a new LLM interface (based on GPT-4o) to complete medical scenarios. The new LLM interface uses a fixed system prompt that (a) instructs the model to ask targeted clarifying questions before providing any diagnostic or triage suggestions, and (b) requires all final responses to follow a structured template listing: possible conditions, approximate likelihood of each, and a recommended triage with brief reasoning.	Device: new GPT-4o Interface; Participants can access any assistance methods they would typically employ (e.g., web search or health portals) in addition to a new LLM (GPT-4o) interface to complete medical scenarios. The new LLM interface uses a fixed system prompt that (a) instructs the model to ask targeted clarifying questions before providing any diagnostic or triage suggestions, and (b) requires all final responses to follow a structured template listing: possible conditions, approximate likelihood of each, and a recommended triage with brief reasoning.
Placebo Comparator: Control; Participants can use any assistance methods they would typically employ (e.g., web search or health portals) in addition to a standard LLM (GPT-4o) to complete medical scenarios. AI-overview in web searches will be disabled via an extension. They would not be allowed to access any LLMs other than the standard LLM interface.	Other: Control; Participants use any assistance methods they would typically employ at home (e.g., Google or health portals) in addition to a standard LLM (GPT-4o) to complete medical scenarios. AI-overview in web searches will be disabled via an extension. They would not be allowed to access any LLMs other than the standard LLM interface.

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Urgency Assessment Accuracy	The primary outcome will be the percentage of correct urgency assessments, ranging from 0 to 100%, where higher scores indicate better urgency assessment (or triage) performance. The rating which participants give for the urgency of each case, will be measured on a five-point scale: Self-care, Routine GP, Urgent Primary Care, Accident & Emergency, and Ambulance. Responses will be compared against the gold-standard answers to produce an accuracy measure. The primary outcome will be compared at the case-level between the randomized groups.	Assessed at a single time point for each case, during the scheduled diagnostic evaluation session, which takes place between 0-5 days after participant enrollment.
Condition Identification Accuracy	The co-primary outcome is the Condition Identification Accuracy, which is the percentage of cases in which the condition was correctly identified, ranging from 0 to 100%., where higher scores indicate better medical condition identification performance. Participants name all medical conditions they considered relevant to their decision. A response is scored as correct for that scenario if at least one named condition matches the physician-generated gold-standard list of relevant conditions. The co-primary outcome will be compared at the case-level between the randomized groups.	Assessed at a single time point for each case, during the scheduled diagnostic evaluation session, which takes place between 0-5 days after participant enrollment.

Secondary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Self-Reported Confidence	Participants' self-reported confidence in their urgency assessments, measured using a scale ranging from 0 to 100.	Assessed at a single time point for each case, during the scheduled diagnostic evaluation session, which takes place between 0-5 days after participant enrollment.
Time Spent	The amount of time participants spend completing each medical scenario in milliseconds.	Assessed at a single time point for each case, during the scheduled diagnostic evaluation session, which takes place between 0-5 days after participant enrollment.

Collaborators and Investigators

• Sponsor: Lahore University of Management Sciences
• Investigators: Principal Investigator: Ayesha Ali, PhD, Lahore University of Management Sciences (LUMS); Study Chair: Ihsan Ayyub Qazi, PhD, Lahore University of Management Sciences (LUMS); Principal Investigator: Zafar Ayyub Qazi, PhD, Lahore University of Management Sciences (LUMS)

Study record dates

Study Major Dates

• Study Start (Estimated): July 25, 2026
• Primary Completion (Estimated): July 1, 2027
• Study Completion (Estimated): July 1, 2027

Study Registration Dates

• First Submitted: July 22, 2026
• First Submitted That Met QC Criteria: July 22, 2026
• First Posted (Actual): July 27, 2026

Study Record Updates

• Last Update Posted (Actual): July 27, 2026
• Last Update Submitted That Met QC Criteria: July 22, 2026
• Last Verified: July 1, 2026

More Information

Terms related to this study

• Keywords: Diagnosis; Triage; LLMs
• Additional Relevant MeSH Terms: Pathologic Processes; Pathological Conditions, Signs and Symptoms; Disease
• Other Study ID Numbers: AI-Assisted-Diag-Public789

Plan for Individual participant data (IPD)

• Plan to Share Individual Participant Data (IPD)?: NO

Drug and device information, study documents

• Studies a U.S. FDA-regulated drug product: No
• Studies a U.S. FDA-regulated device product: No