Large Language Models Versus Anesthesiologists for ASA Physical Status Classification (ASA-LLM)

July 6, 2026 updated by: dilara gocmen, Marmara University Pendik Training and Research Hospital

Comparison of Clinical Assessment and Large Language Models in Preoperative Risk Classification: A Retrospective Analysis of ChatGPT, DeepSeek, Gemini, and Claude in ASA Physical Status Classification

The American Society of Anesthesiologists Physical Status (ASA-PS) classification is a cornerstone of preoperative risk assessment, yet interrater variability among clinicians is well documented. Large language models (LLMs) have recently demonstrated expert-level performance in several clinical classification tasks, including ASA-PS assignment.

This retrospective observational study evaluates whether four widely used LLMs - ChatGPT, DeepSeek, Gemini, and Claude - can accurately and consistently assign ASA-PS classes from structured, fully anonymized clinical vignettes derived from real preoperative anesthesia evaluations, using a consensus of senior anesthesiologists as the reference standard.

No patient data will be transmitted to third-party platforms. Clinical information will be converted by the investigators into de-identified structured vignettes containing only age range, sex, body mass index range, presence or absence of systemic diseases, functional capacity, and the major/minor nature of the planned surgery, in full compliance with national data protection legislation (KVKK).

Study Overview

Status

Not yet recruiting

Conditions

Detailed Description

Adult patients who underwent preoperative anesthesia evaluation before elective surgery at Marmara University Pendik Training and Research Hospital will be included retrospectively. For each patient, demographic data (age, sex, body mass index), systemic comorbidities (hypertension, diabetes mellitus, coronary artery disease, chronic obstructive pulmonary disease, and others), functional capacity (metabolic equivalents, MET), type of planned surgery (major/minor), and the ASA-PS class assigned by the attending anesthesiologist will be recorded.

Clinical data will be anonymized and converted into structured clinical vignettes by the investigators. Vignettes will contain no identifiers, dates, protocol numbers, or rare diagnostic combinations that could directly or indirectly identify a patient.

Standardization of the LLM assessment process: To ensure independence between assessments, each vignette will be evaluated in a separate, history-free session. A new conversation will be initiated in the relevant model for every patient vignette, thereby eliminating the possibility that the model is influenced by its responses to previous vignettes (context anchoring). The ASA-PS class assigned to one vignette will not be carried over as context into the evaluation of any subsequent vignette. Each vignette will be presented to all four models using an identical, standardized prompt requesting only an ASA-PS class (I-VI) with a brief rationale, in a strictly defined output format. Model outputs will play no role in clinical decision-making. External information retrieval by the models will be disabled, and all queries will be completed within a narrow time window to minimize variability in model versions.

Each vignette will be submitted to each model once (single querying). Consequently, the intra-model test-retest reliability of the LLMs will not be assessed; this is acknowledged as a study limitation, consistent with the probabilistic nature of large language models, which may produce between-session variability in their outputs.

Model versions: The current version of each model available at the time of data collection will be used - ChatGPT (GPT-5.5, OpenAI), Gemini (Gemini 3.5, Google DeepMind), DeepSeek (DeepSeek V4, DeepSeek AI), and Claude (Claude Opus 4.8, Anthropic). These versions reflect the versions current at the time of protocol submission; the most recent stable version of each model accessible during data collection will be used, and the exact version and access date will be recorded. Because publicly available chat interfaces may perform automatic background routing to different model tiers, this is acknowledged as a reproducibility limitation.

The reference standard ASA-PS class will be determined by an independent, blinded panel of at least three senior anesthesiologists; consensus or majority vote will define the reference classification.

Statistical analysis: The primary (confirmatory) analysis will quantify the agreement between each LLM and the reference standard using quadratic weighted Cohen's kappa, respecting the ordinal structure of ASA-PS. Multi-rater agreement across the four models and the human raters will be assessed with Fleiss' kappa. Pairwise accuracy comparisons among the four models (six pairwise contrasts) will be treated as secondary/exploratory analyses and compared with McNemar or permutation tests for paired data, applying correction for multiple comparisons (e.g., Bonferroni or Holm); 95% confidence intervals will be estimated by bootstrap methods. Prespecified subgroup analyses include ASA III-IV boundary cases, multimorbidity burden, major versus minor surgery, and rater experience.

Primary hypothesis: The ASA-PS assignments of the LLMs (ChatGPT, DeepSeek, Gemini, and Claude) will show at least good agreement with the reference standard (weighted kappa ≥ 0.60). Secondary hypothesis: LLM errors will cluster in specific subgroups (e.g., the ASA III-IV boundary, multimorbid patients).

Study Type

Observational

Enrollment (Estimated)

350

Contacts and Locations

This section provides the contact details for those conducting the study, and information on where this study is being conducted.

Study Contact

Participation Criteria

Researchers look for people who fit a certain description, called eligibility criteria. Some examples of these criteria are a person's general health condition or prior treatments.

Eligibility Criteria

Ages Eligible for Study

  • Adult
  • Older Adult

Accepts Healthy Volunteers

No

Sampling Method

Probability Sample

Study Population

Adult patients who underwent preoperative anesthesia evaluation before elective surgery at a tertiary university hospital in Istanbul, Turkey.

Description

Inclusion Criteria:

  • Age 18 years or older
  • Planned elective surgery
  • Completed preoperative anesthesia evaluation

Exclusion Criteria:

  • Emergency surgical procedures
  • ASA VI (brain death)
  • Incomplete clinical records

Study Plan

This section provides details of the study plan, including how the study is designed and what the study is measuring.

How is the study designed?

Design Details

Cohorts and Interventions

Group / Cohort
elective surgery patients
Adult patients (≥18 years) who underwent preoperative anesthesia evaluation before elective surgery. Anonymized structured vignettes derived from their records will be classified by four LLMs (ChatGPT, DeepSeek, Gemini, Claude) and by a blinded senior anesthesiologist panel serving as the reference standard.

What is the study measuring?

Primary Outcome Measures

Outcome Measure
Measure Description
Time Frame
Agreement between LLM-assigned and reference-standard ASA-PS class
Time Frame: Through study completion, an average of 3 months
Quadratic weighted Cohen's kappa between each large language model's ASA-PS assignment (ChatGPT, DeepSeek, Gemini, Claude) and the reference standard defined by consensus of a blinded panel of at least three senior anesthesiologists. Agreement of at least "good" level (weighted kappa ≥ 0.60) is hypothesized.
Through study completion, an average of 3 months

Secondary Outcome Measures

Outcome Measure
Measure Description
Time Frame
Overall classification accuracy of each LLM
Time Frame: Through study completion, an average of 3 months
Proportion of vignettes in which the LLM-assigned ASA-PS class exactly matches the reference standard, with exploratory pairwise comparisons among the four models (McNemar/permutation tests, corrected for multiple comparisons)
Through study completion, an average of 3 months
Subgroup error patterns
Time Frame: Through study completion, an average of 3 months
Frequency and direction (over- vs. under-classification) of LLM misclassifications in prespecified subgroups: ASA III-IV boundary, multimorbidity, major vs. minor surgery
Through study completion, an average of 3 months

Collaborators and Investigators

This is where you will find people and organizations involved with this study.

Sponsor

Marmara University Pendik Training and Research Hospital

Publications and helpful links

The person responsible for entering information about the study voluntarily provides these publications. These may be about anything related to the study.

General Publications

Study record dates

These dates track the progress of study record and summary results submissions to ClinicalTrials.gov. Study records and reported results are reviewed by the National Library of Medicine (NLM) to make sure they meet specific quality control standards before being posted on the public website.

Study Major Dates

Study Start (Estimated)

July 21, 2026

Primary Completion (Estimated)

August 21, 2026

Study Completion (Estimated)

October 21, 2026

Study Registration Dates

First Submitted

July 6, 2026

First Submitted That Met QC Criteria

July 6, 2026

First Posted (Actual)

July 10, 2026

Study Record Updates

Last Update Posted (Actual)

July 10, 2026

Last Update Submitted That Met QC Criteria

July 6, 2026

Last Verified

July 1, 2026

More Information

Terms related to this study

Keywords

Additional Relevant MeSH Terms

Other Study ID Numbers

  • ASA-LLM-2026

Plan for Individual participant data (IPD)

Plan to Share Individual Participant Data (IPD)?

NO

Drug and device information, study documents

Studies a U.S. FDA-regulated drug product

No

Studies a U.S. FDA-regulated device product

No

This information was retrieved directly from the website clinicaltrials.gov without any changes. If you have any requests to change, remove or update your study details, please contact register@clinicaltrials.gov. As soon as a change is implemented on clinicaltrials.gov, this will be updated automatically on our website as well.

Clinical Trials on Anesthesia

Search Similar Trials

Sponsors and Collaborators

Medical Conditions

Drug Interventions

CROs by country

CROs in Singapore

Conditions

Rare Diseases

Drug Interventions

Dietary Supplements

Sponsor/Collaborators

Locations

Subscribe