Multicenter derivation and validation of an early warning score for acute respiratory failure or death in the hospital

Mikhail A Dziadzko, Paul J Novotny, Jeff Sloan, Ognjen Gajic, Vitaly Herasevich, Parsa Mirhaji, Yiyuan Wu, Michelle Ng Gong, Mikhail A Dziadzko, Paul J Novotny, Jeff Sloan, Ognjen Gajic, Vitaly Herasevich, Parsa Mirhaji, Yiyuan Wu, Michelle Ng Gong

Abstract

Background: Acute respiratory failure occurs frequently in hospitalized patients and often starts before ICU admission. A risk stratification tool to predict mortality and risk for mechanical ventilation (MV) may allow for earlier evaluation and intervention. We developed and validated an automated electronic health record (EHR)-based model-Accurate Prediction of Prolonged Ventilation (APPROVE)-to identify patients at risk of death or respiratory failure requiring >= 48 h of MV.

Methods: This was an observational study of adults admitted to four hospitals in 2013 or a fifth hospital in 2017. Clinical data were extracted from the EHRs. The 2013 patients were randomly split 50:50 into a derivation/validation cohort. The qualifying event was death or intubation leading to MV >= 48 h. Random forest method was used in model derivation. APPROVE was calculated retrospectively whenever data were available in 2013, and prospectively every 4 h after hospital admission in 2017. The Modified Early Warning Score (MEWS) and National Early Warning Score (NEWS) were calculated at the same times as APPROVE. Clinicians were not alerted except for APPROVE in 2017cohort.

Results: There were 68,775 admissions in 2013 and 2258 in 2017. APPROVE had an area under the receiver operator curve of 0.87 (95% CI 0.85-0.88) in 2013 and 0.90 (95% CI 0.84-0.95) in 2017, which is significantly better than the MEWS and NEWS in 2013 but similar to the MEWS and NEWS in 2017. At a threshold of > 0.25, APPROVE had similar sensitivity and positive predictive value (PPV) (sensitivity 63% and PPV 21% in 2013 vs 64% and 16%, respectively, in 2017). Compared to APPROVE in 2013, at a threshold to achieve comparable PPV (19% at MEWS > 4 and 22% at NEWS > 6), the MEWS and NEWS had lower sensitivity (16% for MEWS and NEWS). Similarly in 2017, at a comparable sensitivity threshold (64% for APPROVE > 0.25 and 67% for MEWS and NEWS > 4), more patients who triggered an alert developed the event with APPROVE (PPV 16%) while achieving a lower false positive rate (FPR 5%) compared to the MEWS (PPV 7%, FPR 14%) and NEWS (PPV 4%, FPR 25%).

Conclusions: An automated EHR model to identify patients at high risk of MV or death was validated retrospectively and prospectively, and was determined to be feasible for real-time risk identification.

Trial registration: ClinicalTrials.gov, NCT02488174 . Registered on 18 March 2015.

Keywords: Acute respiratory failure; Early warning scores; Electronic health records; Prediction; Random forest.

Conflict of interest statement

Ethics approval and consent to participate

The study protocol was reviewed and approved by the institutional review boards of Albert Einstein College of Medicine/Montefiore Health System and Mayo Clinic. Informed consent was waived.

Consent for publication

All parties consented to publication of results.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Figures

Fig. 1
Fig. 1
Relative importance of included variables. Tree depth is average depth at which variable is first used in a tree split, assuming that the most discriminatory variables will split the dataset earlier in trees at lower depths. Variable importance (VIMP) is a measure of how changes in the variable impact prediction error. The bigger the VIMP, the more impact the variable has on prediction. Variables having smaller depth are more discriminatory, and those with bigger importance have a greater impact on prediction. BMI body mass index, BUN blood urea nitrogen, DBP diastolic blood pressure, FiO2 fraction of inspired oxygen, Hb hemoglobin, HCT hematocrit, PaCO2 partial pressure of carbon dioxide, PaO2 partial pressure of oxygen, PLT platelet count, POC glucose point of care, RASS Richmond Agitation-Sedation Scale, SBP systolic blood pressure, SpO2 peripheral capillary oxygen saturation, WBC white blood cell count
Fig. 2
Fig. 2
Area under the curve (AUCROC) for APPROVE, MEWS, and NEWS to predict for hospital mortality or intubation leading to mechanical ventilation > 48 h in retrospective 2013 validation cohort (a) and prospective 2017 validation hospital (b). APPROVE, MEWS and NEWS calculated at multiple random time points for each patient and evaluated for a qualifying event after score calculation. APPROVE Accurate Prediction of Prolonged Ventilation, CI confidence interval, MEWS Modified Early Warning Score, NEWS National Early Warning Score
Fig. 3
Fig. 3
Positive predictive value (Fig. 3a for 2013 and Fig. 3c for 2017) and number of patients needed to be evaluated to identify one event (Fig. 3b for 2013 and Fig. 3d for 2017) as a function of sensitivity for APPROVE, MEWS and NEWS for the retrospective 2013 cohort (Fig. 3a and b) and the prospective 2017 cohort (Fig. 3c and d). Qualifying event is defined as hospital mortality or mechanical ventilation > 48 h

References

    1. Stefan MS, Shieh MS, Pekow PS, Rothberg MB, Steingrub JS, Lagu T, Lindenauer PK. Epidemiology and outcomes of acute respiratory failure in the United States, 2001 to 2009: a national survey. J Hosp Med. 2013;8(2):76–82. doi: 10.1002/jhm.2004.
    1. Wunsch H, Guerra C, Barnato AE, Angus DC, Li G, Linde-Zwirble WT. Three-year outcomes for Medicare beneficiaries who survive intensive care. JAMA. 2010;303(9):849–856. doi: 10.1001/jama.2010.216.
    1. Barrett ML, Smith MW, Elixhauser A, Honigman LS, Pines JM. Utilization of Intensive Care Services, 2011. HCUP Statistical Brief #185. Rockville: Agency for Healthcare Research and Quality; 2014. p. 14.
    1. Wilcox ME, Herridge MS. Long-term outcomes in patients surviving acute respiratory distress syndrome. Semin Respir Crit Care Med. 2010;31(1):55–65. doi: 10.1055/s-0029-1246285.
    1. Schein RM, Hazday N, Pena M, Ruben BH, Sprung CL. Clinical antecedents to in-hospital cardiopulmonary arrest. Chest. 1990;98(6):1388–1392. doi: 10.1378/chest.98.6.1388.
    1. Chen J, Bellomo R, Flabouris A, Hillman K, Assareh H, Ou L. Delayed emergency team calls and associated hospital mortality: a multicenter study. Crit Care Med. 2015;43(10):2059–2065. doi: 10.1097/CCM.0000000000001192.
    1. Thornlow DK, Oddone E, Anderson R. Cascade iatrogenesis: a case-control study to detect postoperative respiratory failure in hospitalized older adults. Res Gerontol Nurs. 2014;7(2):66–77. doi: 10.3928/19404921-20131126-01.
    1. Barwise A, Thongprayoon C, Gajic O, Jensen J, Herasevich V, Pickering BW. Delayed rapid response team activation is associated with increased hospital mortality, morbidity, and length of stay in a tertiary care institution. Crit Care Med. 2016;44(1):54–63. doi: 10.1097/CCM.0000000000001346.
    1. Morgan RJM, Williams F, MM W. An early warning scoring system for detecting developing critical illness. Clin Intensive Care. 1997;8:100.
    1. Smith ME, Chiovaro JC, O'Neil M, Kansagara D, Quinones AR, Freeman M, Motu'apuaka ML, Slatore CG. Early warning system scores for clinical deterioration in hospitalized patients: a systematic review. Ann Am Thorac Soc. 2014;11(9):1454–1465. doi: 10.1513/AnnalsATS.201403-102OC.
    1. Nates JL, Nunnally M, Kleinpell R, Blosser S, Goldner J, Birriel B, Fowler CS, Byrum D, Miles WS, Bailey H, et al. ICU admission, discharge, and triage guidelines: a framework to enhance clinical operations, development of institutional policies, and further research. Crit Care Med. 2016;44(8):1553–1602. doi: 10.1097/CCM.0000000000001856.
    1. Admon AJ, Seymour CW, Gershengorn HB, Wunsch H, Cooke CR. Hospital-level variation in ICU admission and critical care procedures for patients hospitalized for pulmonary embolism. Chest. 2014;146(6):1452–1461. doi: 10.1378/chest.14-0059.
    1. Gershengorn HB, Iwashyna TJ, Cooke CR, Scales DC, Kahn JM, Wunsch H. Variation in use of intensive care for adults with diabetic ketoacidosis. Crit Care Med. 2012;40(7):2009–2015. doi: 10.1097/CCM.0b013e31824e9eae.
    1. Safavi KC, Dharmarajan K, Kim N, Strait KM, Li SX, Chen SI, Lagu T, Krumholz HM. Variation exists in rates of admission to intensive care units for heart failure patients across hospitals in the United States. Circulation. 2013;127(8):923–929. doi: 10.1161/CIRCULATIONAHA.112.001088.
    1. Admon AJ, Wunsch H, Iwashyna TJ, Cooke CR. Hospital contributions to variability in the use of ICUs among elderly medicare recipients. Crit Care Med. 2017;45(1):75–84. doi: 10.1097/CCM.0000000000002025.
    1. Chen LM, Render M, Sales A, Kennedy EH, Wiitala W, Hofer TP. Intensive care unit admitting patterns in the Veterans Affairs health care system. Arch Intern Med. 2012;172(16):1220–1226. doi: 10.1001/archinternmed.2012.2606.
    1. Chen R, Strait KM, Dharmarajan K, Li SX, Ranasinghe I, Martin J, Fazel R, Masoudi FA, Cooke CR, Nallamothu BK, et al. Hospital variation in admission to intensive care units for patients with acute myocardial infarction. Am Heart J. 2015;170(6):1161–1169. doi: 10.1016/j.ahj.2015.09.003.
    1. Wunsch H, Linde-Zwirble WT, Angus DC, Hartman ME, Milbrandt EB, Kahn JM. The epidemiology of mechanical ventilation use in the United States. Crit Care Med. 2010;38(10):1947–1953. doi: 10.1097/CCM.0b013e3181ef4460.
    1. Churpek MM, Yuen TC, Winslow C, Robicsek AA, Meltzer DO, Gibbons RD, Edelson DP. Multicenter development and validation of a risk stratification tool for ward patients. Am J Respir Crit Care Med. 2014;190(6):649–655. doi: 10.1164/rccm.201406-1022OC.
    1. Churpek MM, Yuen TC, Winslow C, Meltzer DO, Kattan MW, Edelson DP. Multicenter comparison of machine learning methods and conventional regression for predicting clinical deterioration on the wards. Crit Care Med. 2016;44(2):368–374. doi: 10.1097/CCM.0000000000001571.
    1. Escobar GJ, LaGuardia JC, Turk BJ, Ragins A, Kipnis P, Draper D. Early detection of impending physiologic deterioration among patients who are not in intensive care: development of predictive models using data from an automated electronic medical record. J Hosp Med. 2012;7(5):388–395. doi: 10.1002/jhm.1929.
    1. Churpek MM, Snyder A, Han X, Sokol S, Pettit N, Howell MD, Edelson DP. Quick sepsis-related organ failure assessment, systemic inflammatory response syndrome, and early warning scores for detecting clinical deterioration in infected patients outside the intensive care unit. Am J Respir Crit Care Med. 2017;195(7):906–911. doi: 10.1164/rccm.201604-0854OC.
    1. Yu S, Leung S, Heo M, Soto GJ, Shah RT, Gunda S, Gong MN. Comparison of risk prediction scoring systems for ward patients: a retrospective nested case-control study. Crit Care. 2014;18(3):R132. doi: 10.1186/cc13947.
    1. Goulden R, Hoyle MC, Monis J, Railton D, Riley V, Martin P, Martina R, Nsutebu E. qSOFA, SIRS and NEWS for predicting inhospital mortality and ICU admission in emergency admissions treated as sepsis. Emerg Med J. 2018;35(6):345–349. doi: 10.1136/emermed-2017-207120.
    1. Brabrand M, Hallas P, Hansen SN, Jensen KM, Madsen JLB, Posth S. Using scores to identify patients at risk of short term mortality at arrival to the acute medical unit: a validation study of six existing scores. Eur J Intern Med. 2017;45:32–36. doi: 10.1016/j.ejim.2017.09.042.
    1. Escobar GJ, Gardner MN, Greene JD, Draper D, Kipnis P. Risk-adjusting hospital mortality using a comprehensive electronic record in an integrated health care delivery system. Med Care. 2013;51(5):446–453. doi: 10.1097/MLR.0b013e3182881c8e.
    1. Kang MA, Churpek MM, Zadravecz FJ, Adhikari R, Twu NM, Edelson DP. Real-time risk prediction on the wards: a feasibility study. Crit Care Med. 2016;44(8):1468–1473. doi: 10.1097/CCM.0000000000001716.
    1. Finlay GD, Rothman MJ, Smith RA. Measuring the modified early warning score and the Rothman index: advantages of utilizing the electronic medical record in an early warning system. J Hosp Med. 2014;9(2):116–119. doi: 10.1002/jhm.2132.
    1. Needham DM, Yang T, Dinglas VD, Mendez-Tellez PA, Shanholtz C, Sevransky JE, Brower RG, Pronovost PJ, Colantuoni E. Timing of low tidal volume ventilation and intensive care unit mortality in acute respiratory distress syndrome. A prospective cohort study. Am J Respir Crit Care Med. 2015;191(2):177–185. doi: 10.1164/rccm.201409-1598OC.
    1. Shehabi Y, Bellomo R, Reade MC, Bailey M, Bass F, Howe B, McArthur C, Seppelt IM, Webb S, Weisbrodt L, et al. Early intensive care sedation predicts long-term mortality in ventilated critically ill patients. Am J Respir Crit Care Med. 2012;186(8):724–731. doi: 10.1164/rccm.201203-0522OC.
    1. Strom T, Martinussen T, Toft P. A protocol of no sedation for critically ill patients receiving mechanical ventilation: a randomised trial. Lancet. 2010;375(9713):475–480. doi: 10.1016/S0140-6736(09)62072-9.
    1. Jaderling G, Bell M, Martling CR, Ekbom A, Konrad D. Limitations of medical treatment among patients attended by the rapid response team. Acta Anaesthesiol Scand. 2013;57(10):1268–1274. doi: 10.1111/aas.12202.
    1. Cassel CK, Guest JA. Choosing wisely: helping physicians and patients make smart decisions about their care. JAMA. 2012;307(17):1801–1802. doi: 10.1001/jama.2012.476.
    1. Quenot JP, Binquet C, Kara F, Martinet O, Ganster F, Navellou JC, Castelain V, Barraud D, Cousson J, Louis G, et al. The epidemiology of septic shock in French intensive care units: the prospective multicenter cohort EPISS study. Crit Care. 2013;17(2):R65. doi: 10.1186/cc12598.
    1. Rhee C, Dantes R, Epstein L, Murphy DJ, Seymour CW, Iwashyna TJ, Kadri SS, Angus DC, Danner RL, Fiore AE, et al. Incidence and trends of sepsis in US hospitals using clinical vs claims data, 2009-2014. JAMA. 2017;318(13):1241–1249. doi: 10.1001/jama.2017.13836.
    1. Shah AD, Bartlett JW, Carpenter J, Nicholas O, Hemingway H. Comparison of random forest and parametric imputation models for imputing missing data using MICE: a CALIBER study. Am J Epidemiol. 2014;179(6):764–774. doi: 10.1093/aje/kwt312.

Source: PubMed

Sponsorer og samarbeidspartnere

Medisinsk tilstand

Legemiddelintervensjoner

3
Abonnere