Physician Diagnosis of Neurologic Cases With Large Language Models

Physician Diagnosis of Neurologic Cases Assisted by Large Language Models: A Randomized Controlled Trial

This study is a prospective randomized controlled trial that aims to evaluate whether large language model (LLM) assistance improves physicians' diagnostic performance compared with conventional assistance.

Study Overview

• Status: Completed
• Conditions: Neurology
• Intervention / Treatment: Other: LLM assistance

Detailed Description

The study population consists of licensed physicians who have completed at least two years of neurology training. Participants are randomly assigned in a 1:1 ratio to the LLM assistance group or the conventional assistance group. In the LLM assistance group, physicians are assisted by an in-house LLM and are allowed to use conventional resources (e.g., search engines, UpToDate, or clinical guidelines) before submitting a final diagnosis. Access to any other LLMs is not permitted. In the conventional assistance group, physicians use only conventional resources and do not have access to the in-house LLM or any other LLMs at any point during the study. The primary outcome is top-1 diagnostic accuracy. Secondary outcomes include top-3 diagnostic accuracy, the time required to complete each case, and physicians' diagnostic confidence.

• Study Type: Interventional
• Enrollment (Actual): 40
• Phase: Not Applicable

Contacts and Locations

Study Locations

• China
  • Beijing Municipality
    • Beijing, Beijing Municipality, China, 100053
      • Xuanwu Hospital Capital Medical University
    • Beijing, Beijing Municipality, China, 100000
      • Beijing Luhe Hospital Affiliated to Capital Medical University
  • Henan
    • Nanyang, Henan, China, 473000
      • Nanyang Central Hospital

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: Child; Adult; Older Adult
• Accepts Healthy Volunteers: Yes

Description

Inclusion Criteria:

• Licensed physicians who have completed at least two years of neurology training.
• Willing to participate in the study and able to provide written informed consent.

Exclusion Criteria:

• Physicians who have previously participated in the development of the in-house LLM or the construction of datasets used in this study.
• Physicians who were involved in the clinical care of patients whose cases were used to construct the case dataset for this study.
• Physicians who are known to have previously reviewed or been exposed to the specific cases used in this study.

Study Plan

How is the study designed?

Design Details

• Primary Purpose: Diagnostic
• Allocation: Randomized
• Interventional Model: Parallel Assignment
• Masking: Single

Number of Arms

2

Arms and Interventions

Participant Group / Arm	Intervention / Treatment
Experimental: LLM assistance group; Physicians are assisted by an in-house LLM and are allowed to use conventional resources. Access to any other LLMs is not permitted.	Other: LLM assistance; Physicians are assisted by an in-house large language model and are allowed to use conventional resources. Access to any other LLMs is not permitted.
No Intervention: Conventional assistance group; Physicians use only conventional resources and do not have access to the in-house LLM or any other LLMs at any point during the study.

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Top-1 diagnostic accuracy	Top-1 diagnosis is considered correct if the physician's final top-ranked diagnosis matches the reference diagnosis or a clinically similar diagnosis judged as equivalent by neurologists.	Baseline

Secondary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Top-3 diagnostic accuracy	Top-3 diagnosis is considered correct if the reference diagnosis or a clinically similar diagnosis judged as equivalent by neurologists appears among the physician's top three final diagnoses.	Baseline
Time spent on diagnosis	Time participants spend per case.	Baseline
Diagnostic confidence	Physicians' diagnostic confidence will be measured using a five-point Likert scale (1=very unsure, 5=very sure).	Baseline

Collaborators and Investigators

• Sponsor: Capital Medical University

Study record dates

Study Major Dates

• Study Start (Actual): January 17, 2026
• Primary Completion (Actual): January 28, 2026
• Study Completion (Actual): January 28, 2026

Study Registration Dates

• First Submitted: December 1, 2025
• First Submitted That Met QC Criteria: January 8, 2026
• First Posted (Actual): January 15, 2026

Study Record Updates

• Last Update Posted (Actual): January 29, 2026
• Last Update Submitted That Met QC Criteria: January 28, 2026
• Last Verified: January 1, 2026

More Information

Terms related to this study

• Other Study ID Numbers: LLM-001

Drug and device information, study documents

• Studies a U.S. FDA-regulated drug product: No
• Studies a U.S. FDA-regulated device product: No