The Diagnostic and Triage Capacity of Laypeople-large Language Model Collaboration in China

The Diagnostic and Triage Capacity of Laypeople-large Language Model Collaboration: a National Pretest-posttest Randomized Controlled Experiment in China

The goal of this randomized controlled trial is to evaluate the role of large language models in enhancing laypeople's ability to self-diagnose and triage common diseases. The main questions it aims to answer are:

• Does using an LLM help participants make more accurate self-diagnoses and care decisions for common illnesses, compared to their first guess without any help?
• How much better is it when people work together with an LLM, compared to using a regular search engine, using the LLM alone, or how doctors would decide? Researchers will compare participants who were randomly assigned to either the LLM group (using DeepSeek) or the search engine group to see if the LLM-assisted approach leads to better clinical judgments.

Participants will:

• Read one of 48 short, realistic health vignettes;
• Make an initial guess about what might be wrong by listing up to three possible causes, ranked from most to least likely, and choose a care level: seek immediate care, see a doctor within one day, see a doctor within one week, or manage at home without medical care.
• Use their assigned tool (either DeepSeek or a standard search engine) to look up information and update their guess and care decision;
• Submit their final diagnosis and care choice after using the tool. In addition, the study team evaluated the performance of four other AI models (GPT-4o, GPT-o1, DeepSeek-v3, and DeepSeek-r1) and 33 experienced general physicians on the same vignettes.

Study Overview

• Status: Completed
• Conditions: Vignette Based Intervention; LLM-based AI Dialogue Bot
• Intervention / Treatment: Behavioral: AI-assisted health information seeking; Behavioral: Conventional internet search for health information

• Study Type: Interventional
• Enrollment (Actual): 6360
• Phase: Not Applicable

Contacts and Locations

Study Locations

• China
  • Hubei
    • Wuhan, Hubei, China
      • Tongji Medical College of Huazhong University of Science & Technology School of Medicine and Health Management

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: Adult; Older Adult
• Accepts Healthy Volunteers: No

Description

Inclusion Criteria:

• Age 18 years or older
• Current resident of mainland China
• History of high-quality participation in online surveys on Credamo platform (historical survey acceptance rate ≥ 80% and personal credit score ≥ 70)

Exclusion Criteria:

• Incomplete survey responses
• Failure on embedded quality-check items
• Implausibly short completion time (<180 seconds for search engine group; <360 seconds for LLM group)
• Provision of non-diagnostic or irrelevant responses (e.g., "unknown", "don't know")
• Consistent pattern of identical responses across all items

Study Plan

How is the study designed?

Design Details

• Primary Purpose: Health Services Research
• Allocation: Randomized
• Interventional Model: Parallel Assignment
• Masking: Single

Number of Arms

2

Arms and Interventions

Participant Group / Arm	Intervention / Treatment
Experimental: layperson-LLM integrated group; After initially answering a clinical diagnosis and triage question without the aid of tools, the participants were asked to use a large language model (Deepseek v3 or r1) to retrieve health information and then answer the same question again	Behavioral: AI-assisted health information seeking; Participants in this group used a large language model (DeepSeek) to search for medical information related to a clinical vignette after providing initial diagnostic and triage decisions. They were instructed to interact freely with the model to gather insights and then update their diagnoses and triage recommendations. The intervention simulates real-world use of AI tools for personal health decision-making
Active Comparator: layperson-search engine group; After initially answering a clinical diagnosis and triage question without the use of tools, the participants were required to use a search engine to retrieve health information and then answer the same question again	Behavioral: Conventional internet search for health information; Participants in this group used mainstream internet search engines (e.g., Baidu, Google, Bing) to look up information about the clinical vignette after making initial diagnostic and triage decisions. They were allowed to search freely but were not permitted to use any named AI chatbot or large language model platform. This group represents typical self-directed online health information seeking behavior.

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Top-3 Diagnostic Accuracy	The primary diagnostic outcome was defined as the proportion of participants who included the correct diagnosis in their top three differential diagnoses after using the assigned tool (LLM or search engine). Accuracy was assessed for each of the 48 clinical vignettes and aggregated across all participants in each group.	Immediately after intervention (within the same survey session)
Triage Accuracy (4-class exact match)	Triage accuracy was defined as the proportion of participants who selected the correct triage level (emergent care, within one day, within one week, or self-care) that matched the reference standard. There were 12 vignettes per triage category.	Immediately after intervention (within the same survey session)

Secondary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Top-1 Diagnostic Accuracy	The proportion of participants who selected the correct diagnosis as their top (first) diagnosis after using the assigned tool. This measures the precision of laypeople's final diagnostic judgment.	Immediately after intervention (within the same survey session)
Triage Accuracy (2-class binary match)		Immediately after intervention (within the same survey session)

Collaborators and Investigators

• Sponsor: Huazhong University of Science and Technology
• Investigators: Principal Investigator: Chenxi Liu, Huazhong University of Science and Technology

Study record dates

Study Major Dates

• Study Start (Actual): April 27, 2025
• Primary Completion (Actual): July 1, 2025
• Study Completion (Actual): July 1, 2025

Study Registration Dates

• First Submitted: November 17, 2025
• First Submitted That Met QC Criteria: November 25, 2025
• First Posted (Actual): November 26, 2025

Study Record Updates

• Last Update Posted (Actual): November 26, 2025
• Last Update Submitted That Met QC Criteria: November 25, 2025
• Last Verified: October 1, 2025

More Information

Terms related to this study

• Other Study ID Numbers: JCYJ20240813115806009

Plan for Individual participant data (IPD)

• Plan to Share Individual Participant Data (IPD)?: NO

Drug and device information, study documents

• Studies a U.S. FDA-regulated drug product: No
• Studies a U.S. FDA-regulated device product: No