Determinants of the calibration of SAPS II and SAPS 3 mortality scores in intensive care: a European multicenter study

Antoine Poncet, Thomas V Perneger, Paolo Merlani, Maurizia Capuzzo, Christophe Combescure, Antoine Poncet, Thomas V Perneger, Paolo Merlani, Maurizia Capuzzo, Christophe Combescure

Abstract

Background: The aim of the Simplified Acute Physiology Score (SAPS) II and SAPS 3 is to predict the mortality of patients admitted to intensive care units (ICUs). Previous studies have suggested that the calibration of these scores may vary across countries, centers, and/or characteristics of patients. In the present study, we aimed to assess determinants of the calibration of these scores.

Methods: We assessed the calibration of the SAPS II and SAPS 3 scores among 5266 patients admitted to ICUs during a 4-week period at 120 centers in 17 European countries. We obtained calibration curves, Brier scores, and standardized mortality ratios. Points attributed to SAPS items were reevaluated and compared with those of the original scores. Finally, we tested associations between the calibration and center characteristics.

Results: The mortality was overestimated by both scores: The standardized mortality ratios were 0.75 (95% CI 0.71-0.79) for the SAPS II score and 0.91 (95% CI 0.86-0.96) for the SAPS 3 score. This overestimation was partially explained by changes in associations between some items of the scores and mortality, especially the heart rate, Glasgow Coma Scale score, and diagnosis of AIDS for SAPS II. The calibration of both scores was better in countries with low health expenditures. The between-center variability in calibration curves was much greater than expected by chance.

Conclusions: Both scores overestimate current mortality among European ICU patients. The magnitude of the miscalibration of SAPS II and SAPS 3 scores depends not only on patient characteristics but also on center characteristics. Furthermore, much between-center variability in calibration remains unexplained by these factors.

Trial registration: ClinicalTrials.gov identifier: NCT01422070 . Registered 19 August 2011.

Keywords: Calibration; Determinants; SAPS 3; SAPS II.

Figures

Fig. 1
Fig. 1
Calibration curves for the Simplified Acute Physiology Score (SAPS) II (dark line) and the SAPS 3 (gray line) obtained with a kernel function. The calibration curve represents the relationship between the mortality predicted by the score (x-axis) and the observed mortality (y-axis). The identity line (dashed line) represents a perfect calibration. A calibration curve below the identity line indicates that the score overestimates the mortality. The black and gray circles represent the estimates of the observed mortality in sample, stratified by levels of predicted mortality (by step of 0.01 up to a predicted mortality of 0.20, by step of 0.025 for a predicted mortality from 0.20 to 0.35, and by step of 0.05 for a predicted mortality greater than 0.35). The size of the circles is proportional to the number of patients in categories of predicted mortality
Fig. 2
Fig. 2
Calibration curves of the Simplified Acute Physiology Score (SAPS) II and SAPS 3 obtained by kernel function by reason for admission to the intensive care unit. a Basic observation. b Severe trauma. c Respiratory reason. d Cardiovascular reason. e Digestive reason. f Neurological reason
Fig. 3
Fig. 3
Observed and expected calibration curves for the Simplified Acute Physiology Score (SAPS) II score (top) and SAPS 3 score (bottom) in 120 centers and between-center variability in standardized mortality ratio (SMR). a Calibration curves of SAPS II in each of the 120 centers fitted with a logistic regression model. The black line represents the overall calibration curve. b Expected calibration curves of SAPS II under the assumption that the calibration is the same in all centers. The represented between-center variability is the random (sampling) variability. c Distribution of the SD of the center-specific SMRs under the assumption that the calibration of SAPS II is the same in all centers. The vertical line represents the observed value of the SD. d–f are the same figures shown in a–c repeated for the SAPS 3 score
Fig. 4
Fig. 4
Calibration curves of the Simplified Acute Physiology Score (SAPS) II score obtained by kernel function according to (a) health expenditure expressed in percentage of gross domestic product (GDP) and (b) number of hospital beds, and calibration curves of the SAPS 3 score by (c) health expenditure expressed as a percentage of GDP and (d) daytime nurse/patient ratio

References

    1. de Vos M, Graafmans W, Keesman E, Westert G, van der Voort PH. Quality measurement at intensive care units: which indicators should we use? J Crit Care. 2007;22:267–74. doi: 10.1016/j.jcrc.2007.01.002.
    1. Glance LG, Osler TM, Dick A. Rating the quality of intensive care units: is it a function of the intensive care unit scoring system? Crit Care Med. 2002;30:1976–82. doi: 10.1097/00003246-200209000-00005.
    1. Vincent JL, Moreno R. Clinical review: scoring systems in the critically ill. Crit Care. 2010;14:207. doi: 10.1186/cc8204.
    1. Vosylius S, Sipylaite J, Ivaskevicius J. Evaluation of intensive care unit performance in Lithuania using the SAPS II system. Eur J Anaesthesiol. 2004;21:619–24. doi: 10.1097/00003643-200408000-00006.
    1. Le Gall JR, Lemeshow S, Saulnier F. A new Simplified Acute Physiology Score (SAPS II) based on a European/North American multicenter study. JAMA. 1993;270:2957–63. doi: 10.1001/jama.1993.03510240069035.
    1. Moreno RP, Metnitz PG, Almeida E, Jordan B, Bauer P, Campos RA, et al. SAPS 3—from evaluation of the patient to evaluation of the intensive care unit. Part 2: Development of a prognostic model for hospital mortality at ICU admission. Intensive Care Med. 2005;31:1345–55. doi: 10.1007/s00134-005-2763-5.
    1. Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010;21:128–38. doi: 10.1097/EDE.0b013e3181c30fb2.
    1. Metnitz PGH, Valentin A, Vesely H, Alberti C, Lang T, Lenz K, et al. Prognostic performance and customization of the SAPS II: results of a multicenter Austrian study. Intensive Care Med. 1999;25:192–7. doi: 10.1007/s001340050815.
    1. Poole D, Rossi C, Latronico N, Rossi G, Finazzi S, Bertolini G. Comparison between SAPS II and SAPS 3 in predicting hospital mortality in a cohort of 103 Italian ICUs: is new always better? Intensive Care Med. 2012;38:1280–8. doi: 10.1007/s00134-012-2578-0.
    1. Strand K, Soreide E, Aardal S, Flaatten H. A comparison of SAPS II and SAPS 3 in a Norwegian intensive care unit population. Acta Anaesthesiol Scand. 2009;53:595–600. doi: 10.1111/j.1399-6576.2009.01948.x.
    1. Apolone G, Bertolini G, D’Amico R, Iapichino G, Cattaneo A, De Salvo G, et al. The performance of SAPS II in a cohort of patients admitted to 99 Italian ICUs: results from GiViTI. Intensive Care Med. 1996;22:1368–78. doi: 10.1007/BF01709553.
    1. Villers D, Fulgencio JP, Gouzes C, Hémery F, Blériot JP, Garrigues B, et al. ICU performance: results of a French study involving 80,000 ICU stays [in French] Ann Fr Anesth Reanim. 2006;25:1111–8. doi: 10.1016/j.annfar.2006.04.015.
    1. Capuzzo M, Moreno RP, Le Gall JR. Outcome prediction in critical care: the Simplified Acute Physiology Score models. Curr Opin Crit Care. 2008;14:485–90. doi: 10.1097/MCC.0b013e32830864d7.
    1. Sprung CL, Cohen SL, Sjokvist P, Baras M, Bulow HH, Hovilehto S, et al. End-of-life practices in European intensive care units: the Ethicus Study. JAMA. 2003;290:790–7. doi: 10.1001/jama.290.6.790.
    1. Capuzzo M, Volta C, Tassinati T, Moreno R, Valentin A, Guidet B, et al. Hospital mortality of adults admitted to intensive care units in hospitals with and without intermediate care units: a multicentre European cohort study. Crit Care. 2014;18:551. doi: 10.1186/s13054-014-0551-8.
    1. Copas JB. Plotting p against x. J R Stat Soc: Ser C: Appl Stat. 1983;32:25–31.
    1. Brier GW. Verification of forecasts expressed in terms of probability. Mon Weather Rev. 1950;78:1–3. doi: 10.1175/1520-0493(1950)078<0001:VOFEIT>;2.
    1. Finazzi S, Poole D, Luciani D, Cogo PE, Bertolini G. Calibration belt for quality-of-care assessment based on dichotomous outcomes. PLoS One. 2011;6:16110. doi: 10.1371/journal.pone.0016110.
    1. Capuzzo M, Scaramuzza A, Vaccarini B, Gilli G, Zannoli S, Farabegoli L, et al. Validation of SAPS 3 admission score and comparison with SAPS II. Acta Anaesthesiol Scand. 2009;53:589–94. doi: 10.1111/j.1399-6576.2009.01929.x.
    1. Desa K, Peric M, Husedzinovic I, Sustic A, Korusic A, Karadza V, et al. Prognostic performance of the Simplified Acute Physiology Score II in major Croatian hospitals: a prospective multicenter study. Croat Med J. 2012;53:442–9. doi: 10.3325/cmj.2012.53.442.
    1. Haaland OA, Lindemark F, Flaatten H, Kvale R, Johansson KA. A calibration study of SAPS II with Norwegian intensive care registry data. Acta Anaesthesiol Scand. 2014;58:701–8. doi: 10.1111/aas.12327.
    1. Nassar AP, Malbouisson LM, Moreno R. Evaluation of Simplified Acute Physiology Score 3 performance: a systematic review of external validation studies. Crit Care. 2014;18:117. doi: 10.1186/cc13911.
    1. Suistomaa M, Kari A, Ruokonen E, Takala J. Sampling rate causes bias in APACHE II and SAPS II scores. Intensive Care Med. 2000;26:1773–8. doi: 10.1007/s001340000677.
    1. Green SM. Cheerio, laddie! Bidding farewell to the Glasgow Coma Scale. Ann Emerg Med. 2011;58:427–30. doi: 10.1016/j.annemergmed.2011.06.009.

Source: PubMed

Sponzoři a spolupracovníci

Zdravotní podmínky

Drogové intervence

3
Předplatit