Comparison of Early Warning Scoring Systems for Hospitalized Patients With and Without Infection at Risk for In-Hospital Mortality and Transfer to the Intensive Care Unit

Abstract

Importance: Risk scores used in early warning systems exist for general inpatients and patients with suspected infection outside the intensive care unit (ICU), but their relative performance is incompletely characterized.

Objective: To compare the performance of tools used to determine points-based risk scores among all hospitalized patients, including those with and without suspected infection, for identifying those at risk for death and/or ICU transfer.

Design, setting, and participants: In a cohort design, a retrospective analysis of prospectively collected data was conducted in 21 California and 7 Illinois hospitals between 2006 and 2018 among adult inpatients outside the ICU using points-based scores from 5 commonly used tools: National Early Warning Score (NEWS), Modified Early Warning Score (MEWS), Between the Flags (BTF), Quick Sequential Sepsis-Related Organ Failure Assessment (qSOFA), and Systemic Inflammatory Response Syndrome (SIRS). Data analysis was conducted from February 2019 to January 2020.

Main outcomes and measures: Risk model discrimination was assessed in each state for predicting in-hospital mortality and the combined outcome of ICU transfer or mortality with area under the receiver operating characteristic curves (AUCs). Stratified analyses were also conducted based on suspected infection.

Results: The study included 773 477 hospitalized patients in California (mean [SD] age, 65.1 [17.6] years; 416 605 women [53.9%]) and 713 786 hospitalized patients in Illinois (mean [SD] age, 61.3 [19.9] years; 384 830 women [53.9%]). The NEWS exhibited the highest discrimination for mortality (AUC, 0.87; 95% CI, 0.87-0.87 in California vs AUC, 0.86; 95% CI, 0.85-0.86 in Illinois), followed by the MEWS (AUC, 0.83; 95% CI, 0.83-0.84 in California vs AUC, 0.84; 95% CI, 0.84-0.85 in Illinois), qSOFA (AUC, 0.78; 95% CI, 0.78-0.79 in California vs AUC, 0.78; 95% CI, 0.77-0.78 in Illinois), SIRS (AUC, 0.76; 95% CI, 0.76-0.76 in California vs AUC, 0.76; 95% CI, 0.75-0.76 in Illinois), and BTF (AUC, 0.73; 95% CI, 0.73-0.73 in California vs AUC, 0.74; 95% CI, 0.73-0.74 in Illinois). At specific decision thresholds, the NEWS outperformed the SIRS and qSOFA at all 28 hospitals either by reducing the percentage of at-risk patients who need to be screened by 5% to 20% or increasing the percentage of adverse outcomes identified by 3% to 25%.

Conclusions and relevance: In all hospitalized patients evaluated in this study, including those meeting criteria for suspected infection, the NEWS appeared to display the highest discrimination. Our results suggest that, among commonly used points-based scoring systems, determining the NEWS for inpatient risk stratification could identify patients with and without infection at high risk of mortality.

Conflict of interest statement

Conflict of Interest Disclosures: Dr Liu reported receiving grants from the National Institutes of Health (NIH) during the conduct of the study and was supported by National Institute of General Medical Sciences (NIGMS) grant R35GM128672 and the Permanente Medical Group. Dr Edelson reported receiving grants from the NIGMS (R01 GM123193) during the conduct of the study; research support from the American Heart Association, Laerdal Medical, and EarlySense; and research support and honoraria from Philips Healthcare. In addition, she has ownership interest in Quant HC and AgileMD outside the submitted work and had a patent to ARCD.P0535US.P2 pending. Dr Escobar reported receiving grants from Merck outside the submitted work. Dr Churpek reported receiving grants from the NIH (NIGMS R01 GM123193) during the conduct of the study and grants from EarlySense outside the submitted work; in addition, Dr Churpek had a patent pending (ARCD. P0535US.P2). No other disclosures were reported.

Figures

Figure 1.. Discrimination of Risk Scores for In-Hospital Mortality and the Combined Outcome of Intensive Care Unit (ICU) Transfer or Mortality

Area under the receiver operating characteristic curve (AUC) for in-hospital mortality (A) and the combined outcome of ICU transfer or mortality (B) for all patients, patients with suspected infection, and patients without suspected infection. BTF indicates Between the Flags; MEWS, Modified Early Warning Score; NEWS, National Early Warning Score; qSOFA, Quick Sepsis-Related Organ Failure Assessment; and SIRS, Systemic Inflammatory Response Syndrome. Error bars indicate 95% CIs, and values without error bars indicate that the minimum and maximum error values equaled the point estimate.

Figure 2.. Early Warning Score Efficiency Curves for In-Hospital Mortality in All Patients

Proportion of patients who reach each score threshold against the risk score sensitivity in evaluation of in-hospital mortality. The National Early Warning Score (NEWS) was associated with the lowest percentage of patients who crossed the alert threshold and would require screening. BTF indicates Between the Flags; MEWS, Modified Early Warning Score; qSOFA, Quick Sepsis-Related Organ Failure Assessment; and SIRS, Systemic Inflammatory Response Syndrome.

Figure 3.. Early Warning Score Efficiency Curves for Patients With Suspected Infection

Proportion of patients who reach each score threshold against the risk score sensitivity for suspected infection. Across any sensitivity threshold, the National Early Warning Score (NEWS) was associated with the lowest percentage of patients who crossed the alert threshold and would require screening. BTF indicates Between the Flags; MEWS, Modified Early Warning Score; qSOFA, Quick Sepsis-Related Organ Failure Assessment; and SIRS, Systemic Inflammatory Response Syndrome.

Figure 4.. Reduction in Clinical Workload With National Early Warning Score (NEWS) vs Systemic Inflammatory Response Syndrome (SIRS) and Quick Sepsis-Related Organ Failure Assessment (qSOFA) Scoring Systems

Each point represents one of the study hospitals and displays the estimated decrease in the proportion of patients screened at a similar sensitivity threshold (y-axis) and the increase in sensitivity as a similar specificity threshold (x-axis).