Diese Seite wurde automatisch übersetzt und die Genauigkeit der Übersetzung wird nicht garantiert. Bitte wende dich an die englische Version für einen Quelltext.

Diagnostic Accuracy of GPT-4o and Claude 4.6 Sonnet in Turkish ED Anamnesis Notes (LLM-ED-DX-TR)

3. Juni 2026 aktualisiert von: Emir Ünal, Marmara University Pendik Training and Research Hospital

Diagnostic Accuracy of Large Language Models From Emergency Department Anamnesis Notes: A Comparison of GPT-4o and Claude 4.6 Sonnet With Emergency Medicine Specialists

This retrospective diagnostic accuracy study evaluates the ability of two large language models (LLMs) - GPT-4o (gpt-4o-2024-11-20; OpenAI) and Claude 4.6 Sonnet (claude-sonnet-4-6; Anthropic) - to generate correct diagnoses from anonymized Turkish-language emergency department (ED) anamnesis notes, and compares their performance with the diagnosis entered by the treating emergency physician. A consensus gold standard is established by three independent board-certified emergency medicine specialists who blindly review each note and vote on the primary diagnosis using ICD-10 three-character codes; the majority vote (at least 2 of 3 specialists agreeing) constitutes the reference standard. Both LLMs are evaluated using a standardized zero-shot direct prompting strategy (temperature=0, stateless API sessions). The primary outcome is diagnostic accuracy (proportion of ICD-10 chapter-level matches) and Cohen's kappa for each LLM against the gold standard. Secondary outcomes include top-3 accuracy, treating physician accuracy, inter-model agreement, and subgroup analyses by ESI triage level and ICD-10 chapter. Inter-rater reliability among the three specialists is quantified using Fleiss' kappa. Analyses are performed in Jamovi. This study represents the first evaluation of LLM diagnostic accuracy using Turkish-language clinical notes and the first to benchmark LLM performance against an independent three-specialist majority-vote gold standard rather than against the treating physician's own diagnosis.

Studienübersicht

Status

Noch keine Rekrutierung

Bedingungen

Detaillierte Beschreibung

STUDY DESIGN: Retrospective diagnostic accuracy study, STARD-AI 2025 reporting, single center, cohort design.

AI INDEX TESTS: (1) GPT-4o (model version gpt-4o-2024-11-20; OpenAI API). (2) Claude 4.6 Sonnet (model version claude-sonnet-4-6; Anthropic API). Both accessed via Python (Google Colab). Temperature=0 for reproducibility. Zero-shot, stateless sessions - no cross-case context. No task-specific fine-tuning or additional training applied; models used as-is via API.

MODEL INTERPRETABILITY: Model interpretability analyses (such as SHAP, Grad-CAM, or layer-attribute visualizations) are not applicable to this study. Because GPT-4o and Claude 4.6 Sonnet are accessed as black-box models through proprietary, closed-source commercial APIs, internal model weights, gradients, and attention architectures are structurally inaccessible for post-hoc interpretability computations.

REFERENCE STANDARD: Three board-certified emergency medicine specialists independently evaluate each anonymized note, blinded to the original physician diagnosis and to each other. Primary diagnosis assigned by at least 2/3 specialists (majority vote) constitutes the gold standard. A 5-case calibration session precedes the main evaluation.

DATA PRIVACY: All anamnesis notes are fully de-identified (name, ID number, date of birth, physician name removed) prior to processing. De-identified notes are stored in a password-protected encrypted database. Only de-identified text is transmitted to LLM APIs - no personal health data. Compliant with Turkish Personal Data Protection Law (KVKK No. 6698).

PATIENT AND PUBLIC INVOLVEMENT: Not applicable. This retrospective study uses fully anonymized existing records; no patient or public involvement in design or conduct.

DATA SHARING: Anonymized dataset will be shared via Zenodo upon article acceptance. Statistical analysis code (Jamovi project files and Python prompt scripts) will be available on GitHub.

Studientyp

Beobachtungs

Einschreibung (Geschätzt)

600

Kontakte und Standorte

Dieser Abschnitt enthält die Kontaktdaten derjenigen, die die Studie durchführen, und Informationen darüber, wo diese Studie durchgeführt wird.

Studienkontakt

  • Name: Emir Ünal, Assistant Professor
  • Telefonnummer: +905327766010
  • E-Mail: emirunal@gmail.com

Studieren Sie die Kontaktsicherung

Studienorte

Teilnahmekriterien

Forscher suchen nach Personen, die einer bestimmten Beschreibung entsprechen, die als Auswahlkriterien bezeichnet werden. Einige Beispiele für diese Kriterien sind der allgemeine Gesundheitszustand einer Person oder frühere Behandlungen.

Zulassungskriterien

Studienberechtigtes Alter

  • Erwachsene
  • Älterer Erwachsener

Akzeptiert gesunde Freiwillige

Nein

Probenahmeverfahren

Nicht-Wahrscheinlichkeitsprobe

Studienpopulation

The study population comprises consecutive adult patients (aged 18 years and older) who presented to the emergency department of a tertiary care training and research hospital and had their encounters fully documented in the hospital information system (HBYS). Eligible individuals must have a complete electronic anamnesis note containing the chief complaint, history of present illness, and clinical presentation, alongside a definitive primary ICD-10 diagnosis finalized by the treating emergency physician at file closure. The population excludes pediatric cases, patients triaged to high-acuity resuscitation areas (ESI level 1), and clinical notes with fewer than 50 words or insufficient clinical content.

Beschreibung

INCLUSION CRITERIA:

  • Adult patients (aged 18 years and older) presenting to the emergency department.
  • Complete electronic health record available in the hospital information system (HBYS) containing a detailed anamnesis note with chief complaint, symptom duration, associated symptoms, and relevant medical history.
  • A definitive primary diagnosis recorded by the treating emergency physician using ICD-10 codes at the time of patient file closure.

EXCLUSION CRITERIA:

  • Emergency department anamnesis notes containing fewer than 50 words or completely lacking substantive clinical content[cite: 1].
  • Pediatric cases (age under 18 years)[cite: 1].
  • Patients critically ill and triaged to high-acuity resuscitation areas (Emergency Severity Index [ESI] level 1)[cite: 1].
  • Clinical notes containing residual identifying information that cannot be fully de-identified, preventing compliance with data privacy regulations[cite: 1].
  • Non-independent clinical notes consisting solely of a brief cross-reference to a prior hospital visit without a new history entry[cite: 1].

Studienplan

Dieser Abschnitt enthält Einzelheiten zum Studienplan, einschließlich des Studiendesigns und der Messung der Studieninhalte.

Wie ist die Studie aufgebaut?

Designdetails

Kohorten und Interventionen

Gruppe / Kohorte
Emergency Department Patient Cohort
Consecutive adult patients presenting to the emergency department with a fully documented electronic anamnesis note and a definitive primary ICD-10 diagnosis

Was misst die Studie?

Primäre Ergebnismessungen

Ergebnis Maßnahme
Maßnahmenbeschreibung
Zeitfenster
Diagnostic Accuracy of GPT-4o for ICD-10 Chapter-Level Diagnosis
Zeitfenster: At the time of single-session algorithmic evaluation (each case evaluated once following data extraction in June 2026).
Proportion of cases in which GPT-4o primary (rank 1) diagnosis matches the 3-specialist majority-vote gold standard at the ICD-10 chapter level (22 categories). Range: 0 to 1.00.
At the time of single-session algorithmic evaluation (each case evaluated once following data extraction in June 2026).
Diagnostic Accuracy of Claude 4.6 Sonnet for ICD-10 Chapter-Level Diagnosis
Zeitfenster: At the time of single-session algorithmic evaluation (each case evaluated once following data extraction in June 2026).
Proportion of cases in which Claude 4.6 Sonnet primary (rank 1) diagnosis matches the 3-specialist majority-vote gold standard at the ICD-10 chapter level (22 categories). Range: 0 to 1.00.
At the time of single-session algorithmic evaluation (each case evaluated once following data extraction in June 2026).

Sekundäre Ergebnismessungen

Ergebnis Maßnahme
Maßnahmenbeschreibung
Zeitfenster
Cohen's Kappa Between GPT-4o Primary Diagnosis and Gold Standard
Zeitfenster: At the time of algorithmic evaluation (June-July 2026)
Kappa coefficient measuring agreement between GPT-4o rank-1 ICD-10 chapter and the 3-specialist gold standard . Interpreted per Landis & Koch (1977): <=0.20 slight; 0.21-0.40 fair; 0.41-0.60 moderate; 0.61-0.80 substantial; >0.80 almost perfect . Range: -1.00 to 1.00 .
At the time of algorithmic evaluation (June-July 2026)
Cohen's Kappa Between Claude 4.6 Sonnet Primary Diagnosis and Gold Standard
Zeitfenster: At the time of algorithmic evaluation (June-July 2026)
appa coefficient measuring agreement between Claude 4.6 Sonnet rank-1 ICD-10 chapter and the 3-specialist gold standard . Interpreted per Landis & Koch (1977): <=0.20 slight; 0.21-0.40 fair; 0.41-0.60 moderate; 0.61-0.80 substantial; >0.80 almost perfect . Range: -1.00 to 1.00
At the time of algorithmic evaluation (June-July 2026)
Top-3 Diagnostic Accuracy of GPT-4o
Zeitfenster: At the time of algorithmic evaluation (June-July 2026)
Proportion of cases in which the 3-specialist gold standard diagnosis appears within GPT-4o's ranked list of three differential diagnoses . Range: 0 to 1.00
At the time of algorithmic evaluation (June-July 2026)
Top-3 Diagnostic Accuracy of Claude 4.6 Sonnet
Zeitfenster: At the time of algorithmic evaluation (June-July 2026)
Proportion of cases in which the 3-specialist gold standard diagnosis appears within Claude 4.6 Sonnet's ranked list of three differential diagnoses[cite: 1]. Range: 0 to 1.00
At the time of algorithmic evaluation (June-July 2026)
Treating Physician Diagnostic Accuracy Against Gold Standard
Zeitfenster: At the time of the original clinical encounter (retrospective data spanning August-December 2025)
Proportion of cases in which the ICD-10 code entered by the treating emergency physician at file closure matches the 3-specialist majority-vote gold standard at the chapter level[cite: 1]. Range: 0 to 1.00
At the time of the original clinical encounter (retrospective data spanning August-December 2025)

Mitarbeiter und Ermittler

Hier finden Sie Personen und Organisationen, die an dieser Studie beteiligt sind.

Sponsor

Marmara University Pendik Training and Research Hospital

Ermittler

  • Hauptermittler: Emir Ünal, Marmara University

Publikationen und hilfreiche Links

Die Bereitstellung dieser Publikationen erfolgt freiwillig durch die für die Eingabe von Informationen über die Studie verantwortliche Person. Diese können sich auf alles beziehen, was mit dem Studium zu tun hat.

Allgemeine Veröffentlichungen

Studienaufzeichnungsdaten

Diese Daten verfolgen den Fortschritt der Übermittlung von Studienaufzeichnungen und zusammenfassenden Ergebnissen an ClinicalTrials.gov. Studienaufzeichnungen und gemeldete Ergebnisse werden von der National Library of Medicine (NLM) überprüft, um sicherzustellen, dass sie bestimmten Qualitätskontrollstandards entsprechen, bevor sie auf der öffentlichen Website veröffentlicht werden.

Haupttermine studieren

Studienbeginn (Geschätzt)

1. Juni 2026

Primärer Abschluss (Geschätzt)

1. Juli 2026

Studienabschluss (Geschätzt)

1. Oktober 2026

Studienanmeldedaten

Zuerst eingereicht

3. Juni 2026

Zuerst eingereicht, das die QC-Kriterien erfüllt hat

3. Juni 2026

Zuerst gepostet (Tatsächlich)

8. Juni 2026

Studienaufzeichnungsaktualisierungen

Letztes Update gepostet (Tatsächlich)

8. Juni 2026

Letztes eingereichtes Update, das die QC-Kriterien erfüllt

3. Juni 2026

Zuletzt verifiziert

1. Juni 2026

Mehr Informationen

Begriffe im Zusammenhang mit dieser Studie

Schlüsselwörter

Zusätzliche relevante MeSH-Bedingungen

Andere Studien-ID-Nummern

  • 09.2026.26-0514

Arzneimittel- und Geräteinformationen, Studienunterlagen

Studiert ein von der US-amerikanischen FDA reguliertes Arzneimittelprodukt

Nein

Studiert ein von der US-amerikanischen FDA reguliertes Geräteprodukt

Nein

Diese Informationen wurden ohne Änderungen direkt von der Website clinicaltrials.gov abgerufen. Wenn Sie Ihre Studiendaten ändern, entfernen oder aktualisieren möchten, wenden Sie sich bitte an register@clinicaltrials.gov. Sobald eine Änderung auf clinicaltrials.gov implementiert wird, wird diese automatisch auch auf unserer Website aktualisiert .

Klinische Studien zur Notfallmedizin

Suchen Sie nach ähnlichen Studien

Sponsoren und Mitarbeiter

Krankheiten

Drogeninterventionen

CROs by country

CROs in Togo

Bedingungen

Seltene Krankheiten

Drogeninterventionen

Nahrungsergänzungsmittel

Sponsor / Mitarbeiter

Standorte

Abonnieren