A validation of machine learning-based risk scores in the prehospital setting

Douglas Spangler, Thomas Hermansson, David Smekal, Hans Blomberg, Douglas Spangler, Thomas Hermansson, David Smekal, Hans Blomberg

Abstract

Background: The triage of patients in prehospital care is a difficult task, and improved risk assessment tools are needed both at the dispatch center and on the ambulance to differentiate between low- and high-risk patients. This study validates a machine learning-based approach to generating risk scores based on hospital outcomes using routinely collected prehospital data.

Methods: Dispatch, ambulance, and hospital data were collected in one Swedish region from 2016-2017. Dispatch center and ambulance records were used to develop gradient boosting models predicting hospital admission, critical care (defined as admission to an intensive care unit or in-hospital mortality), and two-day mortality. Composite risk scores were generated based on the models and compared to National Early Warning Scores (NEWS) and actual dispatched priorities in a prospectively gathered dataset from 2018.

Results: A total of 38203 patients were included from 2016-2018. Concordance indexes (or areas under the receiver operating characteristics curve) for dispatched priorities ranged from 0.51-0.66, while those for NEWS ranged from 0.66-0.85. Concordance ranged from 0.70-0.79 for risk scores based only on dispatch data, and 0.79-0.89 for risk scores including ambulance data. Dispatch data-based risk scores consistently outperformed dispatched priorities in predicting hospital outcomes, while models including ambulance data also consistently outperformed NEWS. Model performance in the prospective test dataset was similar to that found using cross-validation, and calibration was comparable to that of NEWS.

Conclusions: Machine learning-based risk scores outperformed a widely-used rule-based triage algorithm and human prioritization decisions in predicting hospital outcomes. Performance was robust in a prospectively gathered dataset, and scores demonstrated adequate calibration. Future research should explore the robustness of these methods when applied to other settings, establish appropriate outcome measures for use in determining the need for prehospital care, and investigate the clinical impact of interventions based on these methods.

Conflict of interest statement

The authors have declared that no competing interests exist.

Figures

Fig 1. Receiver operating characteristics in predicting…
Fig 1. Receiver operating characteristics in predicting hospital outcomes.
Dotted line corresponds to 95% sensitivity.
Fig 2. Importance of variables in predicting…
Fig 2. Importance of variables in predicting hospital outcomes in Ambulance models.
Variables are arranged in order of descending mean gain across the models predicting the outcomes included in the ambulance data-based risk score.

References

    1. Platts-Mills TF, Leacock B, Cabañas JG, Shofer FS, McLean SA. Emergency Medical Services Use by the Elderly: Analysis of a Statewide Database. Prehosp Emerg Care. 2010;14: 329–333. 10.3109/10903127.2010.481759
    1. Lowthian JA, Jolley DJ, Curtis AJ, Currell A, Cameron PA, Stoelwinder JU, et al. The challenges of population ageing: accelerating demand for emergency ambulance services by older patients, 1995–2015. Med J Aust. 2011;194 Available:
    1. Hwang U, Shah MN, Han JH, Carpenter CR, Siu AL, Adams JG. Transforming Emergency Care For Older Adults. Health Aff (Millwood). 2013;32: 2116–2121. 10.1377/hlthaff.2013.0670
    1. Pines JM, Mullins PM, Cooper JK, Feng LB, Roth KE. National Trends in Emergency Department Use, Care Patterns, and Quality of Care of Older Adults in the United States. J Am Geriatr Soc. 2013;61: 12–17. 10.1111/jgs.12072
    1. Dale J, Williams S, Foster T, Higgins J, Snooks H, Crouch R, et al. Safety of telephone consultation for “non-serious” emergency ambulance service patients. Qual Saf Health Care. 2004;13: 363–373. 10.1136/qshc.2003.008003
    1. Haines CJ, Lutes RE, Blaser M, Christopher NC. Paramedic Initiated Non-Transport of Pediatric Patients. Prehosp Emerg Care. 2006;10: 213–219. 10.1080/10903120500541308
    1. Gray JT, Wardrope J. Introduction of non‐transport guidelines into an ambulance service: a retrospective review. Emerg Med J EMJ. 2007;24: 727–729. 10.1136/emj.2007.048850
    1. Magnusson C, Källenius C, Knutsson S, Herlitz J, Axelsson C. Pre-hospital assessment by a single responder: The Swedish ambulance nurse in a new role: A pilot study. Int Emerg Nurs. 2015. [cited 7 Dec 2015]. 10.1016/j.ienj.2015.09.001
    1. Krumperman K, Weiss S, Fullerton L. Two Types of Prehospital Systems Interventions that Triage Low-Acuity Patients to Alternative Sites of Care. South Med J. 2015;108: 381–386. 10.14423/SMJ.0000000000000303
    1. Eastwood K, Morgans A, Smith K, Hodgkinson A, Becker G, Stoelwinder J. A novel approach for managing the growing demand for ambulance services by low-acuity patients. Aust Health Rev Publ Aust Hosp Assoc. 2015. 10.1071/AH15134
    1. Höglund E, Schröder A, Möller M, Andersson‐Hagiwara M, Ohlsson‐Nevo E. The ambulance nurse experiences of non-conveying patients. J Clin Nurs. 2019;28: 235–244. 10.1111/jocn.14626
    1. Kirkland SW, Soleimani A, Rowe BH, Newton AS. A systematic review examining the impact of redirecting low-acuity patients seeking emergency department care: is the juice worth the squeeze? Emerg Med J. 2019;36: 97–106. 10.1136/emermed-2017-207045
    1. Heward A, Damiani M, Hartley-Sharpe C. Does the use of the Advanced Medical Priority Dispatch System affect cardiac arrest detection? Emerg Med J. 2004;21: 115–118. 10.1136/emj.2003.006940
    1. Bolorunduro OB, Villegas C, Oyetunji TA, Haut ER, Stevens KA, Chang DC, et al. Validating the Injury Severity Score (ISS) in different populations: ISS predicts mortality better among Hispanics and females. J Surg Res. 2011;166: 40–44. 10.1016/j.jss.2010.04.012
    1. Maddali A, Razack FA, Cattamanchi S, Ramakrishnan TV. Validation of the Cincinnati Prehospital Stroke Scale. J Emerg Trauma Shock. 2018;11: 111–114. 10.4103/JETS.JETS_8_17
    1. Silcock DJ, Corfield AR, Gowens PA, Rooney KD. Validation of the National Early Warning Score in the prehospital setting. Resuscitation. 2015;89: 31–35. 10.1016/j.resuscitation.2014.12.029
    1. Seymour CW, Kahn JM, Cooke CR, Watkins TR, Heckbert SR, Rea TD. Prediction of Critical Illness During Out-of-Hospital Emergency Care. JAMA. 2010;304: 747–754. 10.1001/jama.2010.1140
    1. Lane DJ, Wunsch H, Saskin R, Cheskes S, Lin S, Morrison LJ, et al. Assessing Severity of Illness in Patients Transported to Hospital by Paramedics: External Validation of 3 Prognostic Scores. Prehosp Emerg Care. 2019;0: 1–9. 10.1080/10903127.2019.1632998
    1. Pirneskoski J, Kuisma M, Olkkola KT, Nurmi J. Prehospital National Early Warning Score predicts early mortality. Acta Anaesthesiol Scand. 2019;63: 676–683. 10.1111/aas.13310
    1. Patel R, Nugawela MD, Edwards HB, Richards A, Le Roux H, Pullyblank A, et al. Can early warning scores identify deteriorating patients in pre-hospital settings? A systematic review. Resuscitation. 2018;132: 101–111. 10.1016/j.resuscitation.2018.08.028
    1. Hettinger AZ, Cushman JT, Shah MN, Noyes K. Emergency Medical Dispatch Codes Association with Emergency Department Outcomes. Prehosp Emerg Care. 2013;17: 29–37. 10.3109/10903127.2012.710716
    1. Veen M van, Steyerberg EW, Ruige M, Meurs AHJ van, Roukema J, Lei J van der, et al. Manchester triage system in paediatric emergency care: prospective observational study. BMJ. 2008;337: a1501 10.1136/bmj.a1501
    1. Khorram-Manesh A, Montán KL, Hedelin A, Kihlgren M, Örtenwall P. Prehospital triage, discrepancy in priority-setting between emergency medical dispatch centre and ambulance crews. Eur J Trauma Emerg Surg. 2010;37: 73–78. 10.1007/s00068-010-0022-0
    1. Dami F, Golay C, Pasquier M, Fuchs V, Carron P-N, Hugli O. Prehospital triage accuracy in a criteria based dispatch centre. BMC Emerg Med. 2015;15 10.1186/s12873-015-0041-6
    1. Newgard CD, Yang Z, Nishijima D, McConnell KJ, Trent SA, Holmes JF, et al. Cost-Effectiveness of Field Trauma Triage among Injured Adults Served by Emergency Medical Services. J Am Coll Surg. 2016;222: 1125–1137. 10.1016/j.jamcollsurg.2016.02.014
    1. Levin S, Toerper M, Hamrock E, Hinson JS, Barnes S, Gardner H, et al. Machine-Learning-Based Electronic Triage More Accurately Differentiates Patients With Respect to Clinical Outcomes Compared With the Emergency Severity Index. Ann Emerg Med. 2018;71: 565–574.e2. 10.1016/j.annemergmed.2017.08.005
    1. Hong WS, Haimovich AD, Taylor RA. Predicting hospital admission at emergency department triage using machine learning. PLOS ONE. 2018;13: e0201016 10.1371/journal.pone.0201016
    1. Raita Y, Goto T, Faridi MK, Brown DFM, Camargo CA, Hasegawa K. Emergency department triage prediction of clinical outcomes using machine learning models. Crit Care. 2019;23: 64 10.1186/s13054-019-2351-7
    1. Rajkomar A, Oren E, Chen K, Dai AM, Hajaj N, Hardt M, et al. Scalable and accurate deep learning with electronic health records. Npj Digit Med. 2018;1: 18 10.1038/s41746-018-0029-1
    1. Blomberg SN, Folke F, Ersbøll AK, Christensen HC, Torp-Pedersen C, Sayre MR, et al. Machine learning as a supportive tool to recognize cardiac arrest in emergency calls. Resuscitation. 2019;138: 322–329. 10.1016/j.resuscitation.2019.01.015
    1. Gianfrancesco MA, Tamang S, Yazdany J, Schmajuk G. Potential Biases in Machine Learning Algorithms Using Electronic Health Record Data. JAMA Intern Med. 2018;178: 1544–1547. 10.1001/jamainternmed.2018.3763
    1. Dugas AF, Kirsch TD, Toerper M, Korley F, Yenokyan G, France D, et al. An Electronic Emergency Triage System to Improve Patient Distribution by Critical Outcomes. J Emerg Med. 2016;50: 910–918. 10.1016/j.jemermed.2016.02.026
    1. Guttmann A, Schull MJ, Vermeulen MJ, Stukel TA. Association between waiting times and short term mortality and hospital admission after departure from emergency department: population based cohort study from Ontario, Canada. BMJ. 2011;342: d2983 10.1136/bmj.d2983
    1. Di Somma S, Paladino L, Vaughan L, Lalle I, Magrini L, Magnanti M. Overcrowding in emergency department: an international issue. Intern Emerg Med. 2015;10: 171–175. 10.1007/s11739-014-1154-8
    1. Berg LM, Ehrenberg A, Florin J, Östergren J, Discacciati A, Göransson KE. Associations Between Crowding and Ten-Day Mortality Among Patients Allocated Lower Triage Acuity Levels Without Need of Acute Hospital Care on Departure From the Emergency Department. Ann Emerg Med. 2019;74: 345–356. 10.1016/j.annemergmed.2019.04.012
    1. Newgard CD. The Validity of Using Multiple Imputation for Missing Out-of-hospital Data in a State Trauma Registry. Acad Emerg Med. 2006;13: 314–324. 10.1197/j.aem.2005.09.011
    1. Laudermilch DJ, Schiff MA, Nathens AB, Rosengart MR. Lack of Emergency Medical Services Documentation Is Associated with Poor Patient Outcomes: A Validation of Audit Filters for Prehospital Trauma Care. J Am Coll Surg. 2010;210: 220–227. 10.1016/j.jamcollsurg.2009.10.008
    1. Buuren S van, Groothuis-Oudshoorn K. Multivariate Imputation by Chained Equations in R. J Stat Softw. 2011;45 Available:
    1. Chen T, Guestrin C. XGBoost: A Scalable Tree Boosting System. Proceedings of the 22Nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York, NY, USA: ACM; 2016. pp. 785–794. 10.1145/2939672.2939785
    1. Friedman J, Hastie T, Tibshirani R. The elements of statistical learning. Springer series in statistics New York; 2001.
    1. Davison AC, Hinkley DV. Bootstrap Methods and Their Applications. Cambridge: Cambridge University Press; 1997. Available:
    1. Harrell FE. rms: Regression Modeling Strategies. 2017. Available:
    1. R Core Team. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing; 2017. Available:
    1. American College of Surgeons. Resources for optimal care of the injured patient. 6th ed. chicago, IL; 2014.
    1. Cox S, Smith K, Currell A, Harriss L, Barger B, Cameron P. Differentiation of confirmed major trauma patients and potential major trauma patients using pre-hospital trauma triage criteria. Injury. 2011;42: 889–895. 10.1016/j.injury.2010.03.035
    1. Fosbøl EL, Granger CB, Peterson ED, Lin L, Lytle BL, Shofer FS, et al. Prehospital system delay in ST-segment elevation myocardial infarction care: A novel linkage of emergency medicine services and inhospital registry data. Am Heart J. 2013;165: 363–370. 10.1016/j.ahj.2012.11.003
    1. Crilly JL, O’Dwyer JA, O’Dwyer MA, Lind JF, Peters JAL, Tippett VC, et al. Linking ambulance, emergency department and hospital admissions data: understanding the emergency journey. Med J Aust. 2011;194: S34–S37. 10.5694/j.1326-5377.2011.tb02941.x
    1. Birk HO, Henriksen LO. Prehospital Interventions: On-scene-Time and Ambulance-Technicians’ Experience. Prehospital Disaster Med. 2002;17: 167–169. 10.1017/s1049023x00000406
    1. Hale KE, Gavin C, O’Driscoll BR. Audit of oxygen use in emergency ambulances and in a hospital emergency department. Emerg Med J. 2008;25: 773–776. 10.1136/emj.2008.059287
    1. Collins GS, Reitsma JB, Altman DG, Moons KGM. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD statement. Ann Intern Med. 2015;162: 55–63. 10.7326/M14-0697
    1. Swaminathan S, Qirko K, Smith T, Corcoran E, Wysham NG, Bazaz G, et al. A machine learning approach to triaging patients with chronic obstructive pulmonary disease. PLOS ONE. 2017;12: e0188532 10.1371/journal.pone.0188532
    1. Goto T, Camargo CA, Faridi MK, Yun BJ, Hasegawa K. Machine learning approaches for predicting disposition of asthma and COPD exacerbations in the ED. Am J Emerg Med. 2018;36: 1650–1654. 10.1016/j.ajem.2018.06.062
    1. Hong WS, Haimovich AD, Taylor RA. Predicting 72-hour and 9-day return to the emergency department using machine learning. JAMIA Open. 2019. [cited 28 Aug 2019]. 10.1093/jamiaopen/ooz019
    1. Marinovich A, Afilalo J, Afilalo M, Colacone A, Unger B, Giguère C, et al. Impact of Ambulance Transportation on Resource Use in the Emergency Department. Acad Emerg Med. 2004;11: 312–315. 10.1111/j.1553-2712.2004.tb02218.x
    1. Ruger JP, Richter CJ, Lewis LM. Clinical and Economic Factors Associated with Ambulance Use to the Emergency Department. Acad Emerg Med. 2006;13: 879–885. 10.1197/j.aem.2006.04.006
    1. Squire BT, Tamayo A, Tamayo-Sarver JH. At-Risk Populations and the Critically Ill Rely Disproportionately on Ambulance Transport to Emergency Departments. Ann Emerg Med. 2010;56: 341–347. 10.1016/j.annemergmed.2010.04.014
    1. Bohm K, Kurland L. The accuracy of medical dispatch—a systematic review. Scand J Trauma Resusc Emerg Med. 2018;26: 94 10.1186/s13049-018-0528-8
    1. Brangan E, Banks J, Brant H, Pullyblank A, Roux HL, Redwood S. Using the National Early Warning Score (NEWS) outside acute hospital settings: a qualitative study of staff experiences in the West of England. BMJ Open. 2018;8: e022528 10.1136/bmjopen-2018-022528
    1. Spangler D. openTriage prehospital risk score demo. Uppsala: Uppsala University; 2019. Available:

Source: PubMed

Sponsoren und Mitarbeiter

Krankheiten

Drogeninterventionen

3
Abonnieren