Expert-Validated ChatGPT Support for Arterial Blood Gas Interpretation (ABG-AI RCT)

Participants completed the 11 ABG cases without receiving any AI response, explanation, or recommendation

Before each of the 11 ABG cases, participants viewed the expert-validated ChatGPT response before providing their own answers.

Before each of the 11 arterial blood gas (ABG) cases, participants viewed a standardized ChatGPT response that had been independently validated against the expert reference key by three physician experts. The response addressed five scored domains and included a brief rationale. Participants then provided their own answers to each case.

Total accuracy across 55 scored items, comprising five scored domains for each of 11 ABG cases. Each item was scored as correct (1) or incorrect (0), and total accuracy was calculated as the number of correct responses divided by 55 × 100. The primary analysis compared posttest total accuracy between the AI-supported and control groups, adjusted for baseline total accuracy.

Change in total ABG interpretation accuracy from baseline (pretest) to the one-week posttest was assessed within each study group. Total accuracy was calculated across 55 scored items as the number of correct responses divided by 55 × 100.

The change in total ABG interpretation accuracy from baseline to posttest was compared between the AI-supported and control groups. Change scores were calculated as posttest accuracy minus baseline accuracy.

Accuracy in identifying the primary acid-base disorder was calculated as the number of correct responses across 11 arterial blood gas cases divided by 11 and multiplied by 100. Each response was scored as correct (1) or incorrect (0). The outcome was the change in accuracy from baseline to posttest, calculated as posttest accuracy minus baseline accuracy and expressed in percentage points. Possible change scores ranged from -100 to +100 percentage points, with positive values indicating improved accuracy.

For the acute asthma exacerbation case, each participant received one point for each correct response across five predefined assessment domains, yielding a case-specific score ranging from 0 to 5. The posttest score was compared with the baseline score for the same case. Participants were classified into three mutually exclusive categories: improvement (posttest score greater than baseline score), no change (posttest score equal to baseline score), or deterioration (posttest score lower than baseline score). The number of participants in each category was recorded separately for the AI-supported and control groups.

Item-level performance was evaluated across the 55 scored items, comprising five scored domains for each of the 11 ABG cases. The probability of a correct response for individual items was compared between the AI-supported and control groups.

Accuracy in assessing acid-base compensation was calculated as the number of correct responses across 11 arterial blood gas cases divided by 11 and multiplied by 100. Each response was scored as correct (1) or incorrect (0). The outcome was the change in accuracy from baseline to posttest, calculated as posttest accuracy minus baseline accuracy and expressed in percentage points. Possible change scores ranged from -100 to +100 percentage points, with positive values indicating improved accuracy.

Accuracy in assessing electrolyte and anion gap findings was calculated as the number of correct responses across 11 arterial blood gas cases divided by 11 and multiplied by 100. Each response was scored as correct (1) or incorrect (0). The outcome was the change in accuracy from baseline to posttest, calculated as posttest accuracy minus baseline accuracy and expressed in percentage points. Possible change scores ranged from -100 to +100 percentage points, with positive values indicating improved accuracy.

Accuracy in identifying the most likely clinical diagnosis was calculated as the number of correct responses across 11 arterial blood gas cases divided by 11 and multiplied by 100. Each response was scored as correct (1) or incorrect (0). The outcome was the change in accuracy from baseline to posttest, calculated as posttest accuracy minus baseline accuracy and expressed in percentage points. Possible change scores ranged from -100 to +100 percentage points, with positive values indicating improved accuracy.

Accuracy in selecting appropriate clinical management was calculated as the number of correct responses across 11 arterial blood gas cases divided by 11 and multiplied by 100. Each response was scored as correct (1) or incorrect (0). The outcome was the change in accuracy from baseline to posttest, calculated as posttest accuracy minus baseline accuracy and expressed in percentage points. Possible change scores ranged from -100 to +100 percentage points, with positive values indicating improved accuracy.

For the acute COPD exacerbation case, each participant received one point for each correct response across five predefined assessment domains, yielding a case-specific score ranging from 0 to 5. The posttest score was compared with the baseline score for the same case. Participants were classified into three mutually exclusive categories: improvement (posttest score greater than baseline score), no change (posttest score equal to baseline score), or deterioration (posttest score lower than baseline score). The number of participants in each category was recorded separately for the AI-supported and control groups.

For the methanol intoxication case, each participant received one point for each correct response across five predefined assessment domains, yielding a case-specific score ranging from 0 to 5. The posttest score was compared with the baseline score for the same case. Participants were classified into three mutually exclusive categories: improvement (posttest score greater than baseline score), no change (posttest score equal to baseline score), or deterioration (posttest score lower than baseline score). The number of participants in each category was recorded separately for the AI-supported and control groups.

For the diabetic ketoacidosis case, each participant received one point for each correct response across five predefined assessment domains, yielding a case-specific score ranging from 0 to 5. The posttest score was compared with the baseline score for the same case. Participants were classified into three mutually exclusive categories: improvement (posttest score greater than baseline score), no change (posttest score equal to baseline score), or deterioration (posttest score lower than baseline score). The number of participants in each category was recorded separately for the AI-supported and control groups.

For the carbon monoxide intoxication case, each participant received one point for each correct response across five predefined assessment domains, yielding a case-specific score ranging from 0 to 5. The posttest score was compared with the baseline score for the same case. Participants were classified into three mutually exclusive categories: improvement (posttest score greater than baseline score), no change (posttest score equal to baseline score), or deterioration (posttest score lower than baseline score). The number of participants in each category was recorded separately for the AI-supported and control groups.

For the acute kidney injury on chronic kidney disease with hyperkalemia case, each participant received one point for each correct response across five predefined assessment domains, yielding a case-specific score ranging from 0 to 5. The posttest score was compared with the baseline score for the same case. Participants were classified into three mutually exclusive categories: improvement (posttest score greater than baseline score), no change (posttest score equal to baseline score), or deterioration (posttest score lower than baseline score). The number of participants in each category was recorded separately for the AI-supported and control groups.

For the pulmonary embolism case, each participant received one point for each correct response across five predefined assessment domains, yielding a case-specific score ranging from 0 to 5. The posttest score was compared with the baseline score for the same case. Participants were classified into three mutually exclusive categories: improvement (posttest score greater than baseline score), no change (posttest score equal to baseline score), or deterioration (posttest score lower than baseline score). The number of participants in each category was recorded separately for the AI-supported and control groups.

For the panic attack with hyperventilation case, each participant received one point for each correct response across five predefined assessment domains, yielding a case-specific score ranging from 0 to 5. The posttest score was compared with the baseline score for the same case. Participants were classified into three mutually exclusive categories: improvement (posttest score greater than baseline score), no change (posttest score equal to baseline score), or deterioration (posttest score lower than baseline score). The number of participants in each category was recorded separately for the AI-supported and control groups.

For the diuretic-associated metabolic alkalosis case, each participant received one point for each correct response across five predefined assessment domains, yielding a case-specific score ranging from 0 to 5. The posttest score was compared with the baseline score for the same case. Participants were classified into three mutually exclusive categories: improvement (posttest score greater than baseline score), no change (posttest score equal to baseline score), or deterioration (posttest score lower than baseline score). The number of participants in each category was recorded separately for the AI-supported and control groups.

For the hypoglycemia case, each participant received one point for each correct response across five predefined assessment domains, yielding a case-specific score ranging from 0 to 5. The posttest score was compared with the baseline score for the same case. Participants were classified into three mutually exclusive categories: improvement (posttest score greater than baseline score), no change (posttest score equal to baseline score), or deterioration (posttest score lower than baseline score). The number of participants in each category was recorded separately for the AI-supported and control groups.

For the upper gastrointestinal bleeding with hypovolemic shock case, each participant received one point for each correct response across five predefined assessment domains, yielding a case-specific score ranging from 0 to 5. The posttest score was compared with the baseline score for the same case. Participants were classified into three mutually exclusive categories: improvement (posttest score greater than baseline score), no change (posttest score equal to baseline score), or deterioration (posttest score lower than baseline score). The number of participants in each category was recorded separately for the AI-supported and control groups.