Determinants of the calibration of SAPS II and SAPS 3 mortality scores in intensive care: a European multicenter study
Antoine Poncet, Thomas V Perneger, Paolo Merlani, Maurizia Capuzzo, Christophe Combescure, Antoine Poncet, Thomas V Perneger, Paolo Merlani, Maurizia Capuzzo, Christophe Combescure
Abstract
Background: The aim of the Simplified Acute Physiology Score (SAPS) II and SAPS 3 is to predict the mortality of patients admitted to intensive care units (ICUs). Previous studies have suggested that the calibration of these scores may vary across countries, centers, and/or characteristics of patients. In the present study, we aimed to assess determinants of the calibration of these scores.
Methods: We assessed the calibration of the SAPS II and SAPS 3 scores among 5266 patients admitted to ICUs during a 4-week period at 120 centers in 17 European countries. We obtained calibration curves, Brier scores, and standardized mortality ratios. Points attributed to SAPS items were reevaluated and compared with those of the original scores. Finally, we tested associations between the calibration and center characteristics.
Results: The mortality was overestimated by both scores: The standardized mortality ratios were 0.75 (95% CI 0.71-0.79) for the SAPS II score and 0.91 (95% CI 0.86-0.96) for the SAPS 3 score. This overestimation was partially explained by changes in associations between some items of the scores and mortality, especially the heart rate, Glasgow Coma Scale score, and diagnosis of AIDS for SAPS II. The calibration of both scores was better in countries with low health expenditures. The between-center variability in calibration curves was much greater than expected by chance.
Conclusions: Both scores overestimate current mortality among European ICU patients. The magnitude of the miscalibration of SAPS II and SAPS 3 scores depends not only on patient characteristics but also on center characteristics. Furthermore, much between-center variability in calibration remains unexplained by these factors.
Trial registration: ClinicalTrials.gov identifier: NCT01422070 . Registered 19 August 2011.
Keywords: Calibration; Determinants; SAPS 3; SAPS II.
Figures
References
- de Vos M, Graafmans W, Keesman E, Westert G, van der Voort PH. Quality measurement at intensive care units: which indicators should we use? J Crit Care. 2007;22:267–74. doi: 10.1016/j.jcrc.2007.01.002.
- Glance LG, Osler TM, Dick A. Rating the quality of intensive care units: is it a function of the intensive care unit scoring system? Crit Care Med. 2002;30:1976–82. doi: 10.1097/00003246-200209000-00005.
- Vincent JL, Moreno R. Clinical review: scoring systems in the critically ill. Crit Care. 2010;14:207. doi: 10.1186/cc8204.
- Vosylius S, Sipylaite J, Ivaskevicius J. Evaluation of intensive care unit performance in Lithuania using the SAPS II system. Eur J Anaesthesiol. 2004;21:619–24. doi: 10.1097/00003643-200408000-00006.
- Le Gall JR, Lemeshow S, Saulnier F. A new Simplified Acute Physiology Score (SAPS II) based on a European/North American multicenter study. JAMA. 1993;270:2957–63. doi: 10.1001/jama.1993.03510240069035.
- Moreno RP, Metnitz PG, Almeida E, Jordan B, Bauer P, Campos RA, et al. SAPS 3—from evaluation of the patient to evaluation of the intensive care unit. Part 2: Development of a prognostic model for hospital mortality at ICU admission. Intensive Care Med. 2005;31:1345–55. doi: 10.1007/s00134-005-2763-5.
- Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010;21:128–38. doi: 10.1097/EDE.0b013e3181c30fb2.
- Metnitz PGH, Valentin A, Vesely H, Alberti C, Lang T, Lenz K, et al. Prognostic performance and customization of the SAPS II: results of a multicenter Austrian study. Intensive Care Med. 1999;25:192–7. doi: 10.1007/s001340050815.
- Poole D, Rossi C, Latronico N, Rossi G, Finazzi S, Bertolini G. Comparison between SAPS II and SAPS 3 in predicting hospital mortality in a cohort of 103 Italian ICUs: is new always better? Intensive Care Med. 2012;38:1280–8. doi: 10.1007/s00134-012-2578-0.
- Strand K, Soreide E, Aardal S, Flaatten H. A comparison of SAPS II and SAPS 3 in a Norwegian intensive care unit population. Acta Anaesthesiol Scand. 2009;53:595–600. doi: 10.1111/j.1399-6576.2009.01948.x.
- Apolone G, Bertolini G, D’Amico R, Iapichino G, Cattaneo A, De Salvo G, et al. The performance of SAPS II in a cohort of patients admitted to 99 Italian ICUs: results from GiViTI. Intensive Care Med. 1996;22:1368–78. doi: 10.1007/BF01709553.
- Villers D, Fulgencio JP, Gouzes C, Hémery F, Blériot JP, Garrigues B, et al. ICU performance: results of a French study involving 80,000 ICU stays [in French] Ann Fr Anesth Reanim. 2006;25:1111–8. doi: 10.1016/j.annfar.2006.04.015.
- Capuzzo M, Moreno RP, Le Gall JR. Outcome prediction in critical care: the Simplified Acute Physiology Score models. Curr Opin Crit Care. 2008;14:485–90. doi: 10.1097/MCC.0b013e32830864d7.
- Sprung CL, Cohen SL, Sjokvist P, Baras M, Bulow HH, Hovilehto S, et al. End-of-life practices in European intensive care units: the Ethicus Study. JAMA. 2003;290:790–7. doi: 10.1001/jama.290.6.790.
- Capuzzo M, Volta C, Tassinati T, Moreno R, Valentin A, Guidet B, et al. Hospital mortality of adults admitted to intensive care units in hospitals with and without intermediate care units: a multicentre European cohort study. Crit Care. 2014;18:551. doi: 10.1186/s13054-014-0551-8.
- Copas JB. Plotting p against x. J R Stat Soc: Ser C: Appl Stat. 1983;32:25–31.
- Brier GW. Verification of forecasts expressed in terms of probability. Mon Weather Rev. 1950;78:1–3. doi: 10.1175/1520-0493(1950)078<0001:VOFEIT>;2.
- Finazzi S, Poole D, Luciani D, Cogo PE, Bertolini G. Calibration belt for quality-of-care assessment based on dichotomous outcomes. PLoS One. 2011;6:16110. doi: 10.1371/journal.pone.0016110.
- Capuzzo M, Scaramuzza A, Vaccarini B, Gilli G, Zannoli S, Farabegoli L, et al. Validation of SAPS 3 admission score and comparison with SAPS II. Acta Anaesthesiol Scand. 2009;53:589–94. doi: 10.1111/j.1399-6576.2009.01929.x.
- Desa K, Peric M, Husedzinovic I, Sustic A, Korusic A, Karadza V, et al. Prognostic performance of the Simplified Acute Physiology Score II in major Croatian hospitals: a prospective multicenter study. Croat Med J. 2012;53:442–9. doi: 10.3325/cmj.2012.53.442.
- Haaland OA, Lindemark F, Flaatten H, Kvale R, Johansson KA. A calibration study of SAPS II with Norwegian intensive care registry data. Acta Anaesthesiol Scand. 2014;58:701–8. doi: 10.1111/aas.12327.
- Nassar AP, Malbouisson LM, Moreno R. Evaluation of Simplified Acute Physiology Score 3 performance: a systematic review of external validation studies. Crit Care. 2014;18:117. doi: 10.1186/cc13911.
- Suistomaa M, Kari A, Ruokonen E, Takala J. Sampling rate causes bias in APACHE II and SAPS II scores. Intensive Care Med. 2000;26:1773–8. doi: 10.1007/s001340000677.
- Green SM. Cheerio, laddie! Bidding farewell to the Glasgow Coma Scale. Ann Emerg Med. 2011;58:427–30. doi: 10.1016/j.annemergmed.2011.06.009.
Source: PubMed