Effect of a sepsis prediction algorithm on patient mortality, length of stay and readmission: a prospective multicentre clinical outcomes evaluation of real-world patient data from US hospitals

Hoyt Burdick, Eduardo Pino, Denise Gabel-Comeau, Andrea McCoy, Carol Gu, Jonathan Roberts, Sidney Le, Joseph Slote, Emily Pellegrini, Abigail Green-Saxena, Jana Hoffman, Ritankar Das, Hoyt Burdick, Eduardo Pino, Denise Gabel-Comeau, Andrea McCoy, Carol Gu, Jonathan Roberts, Sidney Le, Joseph Slote, Emily Pellegrini, Abigail Green-Saxena, Jana Hoffman, Ritankar Das

Abstract

Background: Severe sepsis and septic shock are among the leading causes of death in the USA. While early prediction of severe sepsis can reduce adverse patient outcomes, sepsis remains one of the most expensive conditions to diagnose and treat.

Objective: The purpose of this study was to evaluate the effect of a machine learning algorithm for severe sepsis prediction on in-hospital mortality, hospital length of stay and 30-day readmission.

Design: Prospective clinical outcomes evaluation.

Setting: Evaluation was performed on a multiyear, multicentre clinical data set of real-world data containing 75 147 patient encounters from nine hospitals across the continental USA, ranging from community hospitals to large academic medical centres.

Participants: Analyses were performed for 17 758 adult patients who met two or more systemic inflammatory response syndrome criteria at any point during their stay ('sepsis-related' patients).

Interventions: Machine learning algorithm for severe sepsis prediction.

Outcome measures: In-hospital mortality, length of stay and 30-day readmission rates.

Results: Hospitals saw an average 39.5% reduction of in-hospital mortality, a 32.3% reduction in hospital length of stay and a 22.7% reduction in 30-day readmission rate for sepsis-related patient stays when using the machine learning algorithm in clinical outcomes analysis.

Conclusions: Reductions of in-hospital mortality, hospital length of stay and 30-day readmissions were observed in real-world clinical use of the machine learning-based algorithm. The predictive algorithm may be successfully used to improve sepsis-related outcomes in live clinical settings.

Trial registration number: NCT03960203.

Keywords: computer methodologies; healthcare; information science; medical informatics.

Conflict of interest statement

Competing interests: All authors who have affiliations listed with Dascena (Oakland, California, USA) are employees or contractors of Dascena.

© Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Figures

Figure 1
Figure 1
Patientoutcomes——differences in (A) in-hospital mortality, (B) hospital length of stay and (C) 30-day readmissions in the baseline period and the MLA period for sepsis-related patients. Use of the MLA was associated with a 39.5% reduction of in-hospital mortality (p

References

    1. Angus DC, Linde-Zwirble WT, Lidicker J, et al. . Epidemiology of severe sepsis in the United States: analysis of incidence, outcome, and associated costs of care. Crit Care Med 2001;29:1303–10. 10.1097/00003246-200107000-00002
    1. Stevenson EK, Rubenstein AR, Radin GT, et al. . Two decades of mortality trends among patients with severe sepsis: a comparative meta-analysis*. Crit Care Med 2014;42:625–31. 10.1097/CCM.0000000000000026
    1. Torio CM, Celeste M, Andrews RM. "National inpatient hospital costs: the most expensive conditions by payer 2011", 2013.
    1. Torio C, Moore B. National Inpatient Hospital Costs: The Most Expensive Conditions by Payer, 2013. HCUP Statistical Brief #204. Agency for Healthcare Research and Quality, Rockville, MD, 2016. Available:
    1. Dellinger RP, Levy MM, Rhodes A, et al. . Surviving sepsis campaign: international guidelines for management of severe sepsis and septic shock, 2012. Intensive Care Med 2013;39:165–228. 10.1007/s00134-012-2769-8
    1. Lagu T, Rothberg MB, Shieh M-S, et al. . Hospitalizations, costs, and outcomes of severe sepsis in the United States 2003 to 2007. Crit Care Med 2012;40:754–61. 10.1097/CCM.0b013e318232db65
    1. Gaieski DF, Edwards JM, Kallan MJ, et al. . Benchmarking the incidence and mortality of severe sepsis in the United States. Crit Care Med 2013;41:1167–74. 10.1097/CCM.0b013e31827c09f8
    1. Levy MM, Fink MP, Marshall JC, et al. . 2001 sccm/esicm/accp/ats/sis international sepsis definitions conference. Crit Care Med 2003;31:1250–6. 10.1097/01.CCM.0000050454.01978.3B
    1. Singer M, Deutschman CS, Seymour CW, et al. . The third International consensus definitions for sepsis and septic shock (sepsis-3). JAMA 2016;315:801–10. 10.1001/jama.2016.0287
    1. Shankar-Hari M, Phillips GS, Levy ML, et al. . Developing a new definition and assessing new clinical criteria for septic shock: for the third International consensus definitions for sepsis and septic shock (Sepsis-3). JAMA 2016;315:775–87. 10.1001/jama.2016.0289
    1. Damiani E, Donati A, Serafini G, et al. . Effect of performance improvement programs on compliance with sepsis bundles and mortality: a systematic review and meta-analysis of observational studies. PLoS One 2015;10:e0125827–24. 10.1371/journal.pone.0125827
    1. Moore LJ, Moore FA. Early diagnosis and evidence-based care of surgical sepsis. J Intensive Care Med 2013;28:107–17. 10.1177/0885066611408690
    1. Kenzaka T, Okayama M, Kuroki S, et al. . Importance of vital signs to the early diagnosis and severity of sepsis: association between vital signs and sequential organ failure assessment score in patients with sepsis. Intern Med 2012;51:871–6. 10.2169/internalmedicine.51.6951
    1. Vincent JL, Moreno R, Takala J, et al. . The SOFA (sepsis-related organ failure assessment) score to describe organ dysfunction/failure. on behalf of the Working group on sepsis-related problems of the European Society of intensive care medicine. Intensive Care Med 1996;22:707–10. 10.1007/bf01709751
    1. Jaimes F, Garcés J, Cuervo J, et al. . The systemic inflammatory response syndrome (SIRS) to identify infected patients in the emergency room. Intensive Care Med 2003;29:1368–71. 10.1007/s00134-003-1874-0
    1. Subbe CP, Slater A, Menon D, et al. . Validation of physiological scoring systems in the accident and emergency department. Emerg Med J 2006;23:841–5. 10.1136/emj.2006.035816
    1. McLymont N, Glover GW. Scoring systems for the characterization of sepsis and associated outcomes. Ann Transl Med 2016;4:527. 10.21037/atm.2016.12.53
    1. Churpek MM, Snyder A, Han X, et al. . Quick sepsis-related organ failure assessment, systemic inflammatory response syndrome, and early warning scores for detecting clinical deterioration in infected patients outside the intensive care unit. Am J Respir Crit Care Med 2017;195:906–11. 10.1164/rccm.201604-0854OC
    1. Usman OA, Usman AA, Ward MA. Comparison of SIRS, qSOFA, and news for the early identification of sepsis in the emergency department. Am J Emerg Med 2019;37:1490–7. 10.1016/j.ajem.2018.10.058
    1. Johnson AEW, Aboab J, Raffa JD, et al. . A comparative analysis of sepsis identification methods in an electronic Database*. Crit Care Med 2018;46:494–9. 10.1097/CCM.0000000000002965
    1. Bhattacharjee P, Edelson DP, Churpek MM. Identifying patients with sepsis on the hospital wards. Chest 2017;151:898–907. 10.1016/j.chest.2016.06.020
    1. van der Woude SW, van Doormaal FF, Hutten BA, et al. . Classifying sepsis patients in the emergency department using SIRS, qSOFA or MEWS. Neth J Med 2018;76:158–66.
    1. Churpek MM, Zadravecz FJ, Winslow C, et al. . Incidence and prognostic value of the systemic inflammatory response syndrome and organ dysfunctions in ward patients. Am J Respir Crit Care Med 2015;192:958–64. 10.1164/rccm.201502-0275OC
    1. Kaukonen K-M, Bailey M, Pilcher D, et al. . Systemic inflammatory response syndrome criteria in defining severe sepsis. N Engl J Med 2015;372:1629–38. 10.1056/NEJMoa1415236
    1. Finkelsztein EJ, Jones DS, Ma KC, et al. . Comparison of qSOFA and SIRS for predicting adverse outcomes of patients with suspicion of sepsis outside the intensive care unit. Crit Care 2017;21:73. 10.1186/s13054-017-1658-5
    1. Roney JK, Whitley BE, Maples JC, et al. . Modified early warning scoring (MEWS): evaluating the evidence for tool inclusion of sepsis screening criteria and impact on mortality and failure to rescue. J Clin Nurs 2015;24:3343–54. 10.1111/jocn.12952
    1. Lie KC, Lau C-Y, Van Vinh Chau N, et al. . Utility of SOFA score, management and outcomes of sepsis in Southeast Asia: a multinational multicenter prospective observational study. J Intensive Care 2018;6:9. 10.1186/s40560-018-0279-7
    1. Innocenti F, Tozzi C, Donnini C, et al. . SOFA score in septic patients: incremental prognostic value over age, comorbidities, and parameters of sepsis severity. Intern Emerg Med 2018;13:405–12. 10.1007/s11739-017-1629-5
    1. Office of the National Coordinator for Health Information Technology 'Office-based Physician Electronic Health Record Adoption,' Health IT Quick-Stat #50, 2016. Available:
    1. EMR Incentives & Certification, 2013. Available:
    1. Available:
    1. Available:
    1. Available:
    1. Phua J, Ngerng W, See K, et al. . Characteristics and outcomes of culture-negative versus culture-positive severe sepsis. Crit Care 2013;17:R202. 10.1186/cc12896
    1. Lamantia MA, Stewart PW, Platts-Mills TF, et al. . Predictive value of initial triage vital signs for critically ill older adults. West J Emerg Med 2013;14:453–60. 10.5811/westjem.2013.5.13411
    1. Girard TD, Opal SM, Ely EW. Insights into severe sepsis in older patients: from epidemiology to evidence-based management. Clin Infect Dis 2005;40:719–27. 10.1086/427876
    1. Delahanty RJ, Alvarez J, Flynn LM, et al. . Development and evaluation of a machine learning model for the early identification of patients at risk for sepsis. Ann Emerg Med 2019;73:334–44. 10.1016/j.annemergmed.2018.11.036
    1. Horng S, Sontag DA, Halpern Y, et al. . Creating an automated trigger for sepsis clinical decision support at emergency department triage using machine learning. PLoS One 2017;12:e0174708. 10.1371/journal.pone.0174708
    1. Mao Q, Jay M, Hoffman JL, et al. . Multicentre validation of a sepsis prediction algorithm using only vital sign data in the emergency department, general ward and ICU. BMJ Open 2018;8:e017833. 10.1136/bmjopen-2017-017833
    1. Shimabukuro DW, Barton CW, Feldman MD, et al. . Effect of a machine learning-based severe sepsis prediction algorithm on patient survival and hospital length of stay: a randomised clinical trial. BMJ Open Respir Res 2017;4:e000234. 10.1136/bmjresp-2017-000234
    1. McCoy A, Das R. Reducing patient mortality, length of stay and readmissions through machine learning-based sepsis prediction in the emergency department, intensive care unit and hospital floor units. BMJ Open Qual 2017;6:e000158. 10.1136/bmjoq-2017-000158
    1. Burdick H, Pino E, Gabel-Comeau D, et al. . Evaluating a sepsis prediction machine learning algorithm using minimal electronic health record data in the emergency department and intensive care unit. bioRxiv:224014.
    1. Calvert JS, Price DA, Chettipally UK, et al. . A computational approach to early sepsis detection. Comput Biol Med 2016;74:69–73. 10.1016/j.compbiomed.2016.05.003
    1. Desautels T, Calvert J, Hoffman J, et al. . Prediction of sepsis in the intensive care unit with minimal electronic health record data: a machine learning approach. JMIR Med Inform 2016;4:e28. 10.2196/medinform.5909
    1. Calvert J, Desautels T, Chettipally U, et al. . High-Performance detection and early prediction of septic shock for alcohol-use disorder patients. Ann Med Surg 2016;8:50–5. 10.1016/j.amsu.2016.04.023
    1. Chen T, Guestrin C. XGBoost: a scalable tree boosting system. Paper presented at the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016.
    1. Shao J, Zhong B. Last observation carry-forward and last observation analysis. Stat Med 2003;22:2429–41. 10.1002/sim.1519
    1. Ali MW, Talukder E. Analysis of longitudinal binary data with missing data due to dropouts. J Biopharm Stat 2005;15:993–1007. 10.1080/10543400500266692
    1. Mohamadlou H, Lynn-Palevsky A, Barton C, et al. . Prediction of acute kidney injury with a machine learning algorithm using electronic health record data. Can J Kidney Health Dis 2018;5:205435811877632. 10.1177/2054358118776326
    1. Fang X, Wang Z, Yang J, et al. . Clinical evaluation of Sepsis-1 and Sepsis-3 in the ICU. Chest 2018;153:1169–76. 10.1016/j.chest.2017.06.037
    1. Burdick H, Pino E, Gabel-Comeau D, et al. . Effect of a sepsis prediction algorithm on patient mortality, length of stay and readmission. bioRxiv 2018;1:457465.
    1. Mayr FB, Yende S, Angus DC. Epidemiology of severe sepsis. Virulence 2014;5:4–11. 10.4161/viru.27372
    1. Umscheid CA, Betesh J, VanZandbergen C, et al. . Development, implementation, and impact of an automated early warning and response system for sepsis. J Hosp Med 2015;10:26–31. 10.1002/jhm.2259
    1. Kumar A, Roberts D, Wood KE, et al. . Duration of hypotension before initiation of effective antimicrobial therapy is the critical determinant of survival in human septic shock. Crit Care Med 2006;34:1589–96. 10.1097/01.CCM.0000217961.75225.E9
    1. Rivers E, Nguyen B, Havstad S, et al. . Early goal-directed therapy in the treatment of severe sepsis and septic shock. N Engl J Med 2001;345:1368–77. 10.1056/NEJMoa010307
    1. Yealy DM, Kellum JA, ProCESS Investigators, et al. . A randomized trial of protocol-based care for early septic shock. N Engl J Med 2014;370:1683–93. 10.1056/NEJMoa1401602
    1. Giannini HM, Ginestra JC, Chivers C, et al. . A machine learning algorithm to predict severe sepsis and septic shock: development, implementation, and impact on clinical practice. Crit Care Med 2019;47:1485–92. 10.1097/CCM.0000000000003891
    1. Taylor RA, Pare JR, Venkatesh AK, et al. . Prediction of in-hospital mortality in emergency department patients with sepsis: a local big data-driven, machine learning approach. Acad Emerg Med 2016;23:269–78. 10.1111/acem.12876
    1. Wulff A, Montag S, Marschollek M, et al. . Clinical decision-support systems for detection of systemic inflammatory response syndrome, sepsis, and septic shock in critically ill patients: a systematic review. Methods Inf Med 2019:1–15.
    1. Nachimuthu SK, Haug PJ. Early detection of sepsis in the emergency department using dynamic Bayesian networks. AMIA Annu Symp Proc 2012;2012:653–62.
    1. Henry KE, Hager DN, Pronovost PJ, et al. . A targeted real-time early warning score (TREWScore) for septic shock. Sci Transl Med 2015;7:299ra122–299. 10.1126/scitranslmed.aab3719
    1. Nemati S, Holder A, Razmi F, et al. . An interpretable machine learning model for accurate prediction of sepsis in the ICU. Crit Care Med 2018;46:547–53. 10.1097/CCM.0000000000002936
    1. Dyagilev K, Saria S. Learning (predictive) risk scores in the presence of censoring due to interventions. Mach Learn 2016;102:323–48. 10.1007/s10994-015-5527-7
    1. Stanculescu I, Williams CK, Freer Y, editors . A hierarchical switching linear dynamical system applied to the detection of sepsis in neonatal condition monitoring. UAI, 2014.
    1. Stanculescu I, Williams CKI, Freer Y. Autoregressive hidden Markov models for the early detection of neonatal sepsis. IEEE J Biomed Health Inform 2014;18:1560–70. 10.1109/JBHI.2013.2294692
    1. Harrison AM, Thongprayoon C, Kashyap R, et al. . Developing the surveillance algorithm for detection of failure to recognize and treat severe sepsis. Mayo Clin Proc 2015;90:166–75. 10.1016/j.mayocp.2014.11.014
    1. Nelson JL, Smith BL, Jared JD, et al. . Prospective trial of real-time electronic surveillance to expedite early care of severe sepsis. Ann Emerg Med 2011;57:500–4. 10.1016/j.annemergmed.2010.12.008
    1. Umscheid CA, Betesh J, VanZandbergen C, et al. . Development, implementation, and impact of an automated early warning and response system for sepsis. J Hosp Med 2015;10:26–31. 10.1002/jhm.2259
    1. Austrian JS, Jamin CT, Doty GR, et al. . Impact of an emergency department electronic sepsis surveillance system on patient mortality and length of stay. J Am Med Inform Assoc 2018;25:523–9. 10.1093/jamia/ocx072
    1. Tiru B, DiNino EK, Orenstein A, et al. . The economic and humanistic burden of severe sepsis. Pharmacoeconomics 2015;33:925–37. 10.1007/s40273-015-0282-y
    1. Ferrer R, Martin-Loeches I, Phillips G, et al. . Empiric antibiotic treatment reduces mortality in severe sepsis and septic shock from the first hour: results from a guideline-based performance improvement program. Crit Care Med 2014;42:1749–55. 10.1097/CCM.0000000000000330
    1. Calvert J, Hoffman J, Barton C, et al. . Cost and mortality impact of an algorithm-driven sepsis prediction system. J Med Econ 2017;20:646–51. 10.1080/13696998.2017.1307203
    1. Cabitza F, Rasoini R, Gensini GF. Unintended consequences of machine learning in medicine. JAMA 2017;318:517–8. 10.1001/jama.2017.7797

Source: PubMed

Sponsorer og samarbejdspartnere

Medicinske tilstande

Narkotikainterventioner

3
Abonner