Head-to-head comparison of 14 prediction models for postoperative delirium in elderly non-ICU patients: an external validation study

Abstract

Objectives: Delirium is associated with increased morbidity, mortality, prolonged hospitalisation and increased healthcare costs. The number of clinical prediction models (CPM) to predict postoperative delirium has increased exponentially. Our goal is to perform a head-to-head comparison of CPMs predicting postoperative delirium in non-intensive care unit (non-ICU) elderly patients to identify the best performing models.

Setting: Single-site university hospital.

Design: Secondary analysis of prospective cohort study.

Participants and inclusion: CPMs published within the timeframe of 1 January 1990 to 1 May 2020 were checked for eligibility (Preferred Reporting Items for Systematic Reviews and Meta-Analyses). For the time period of 1 January 1990 to 1 January 2017, included CPMs were identified in systematic reviews based on prespecified inclusion and exclusion criteria. An extended literature search for original studies was performed independently by two authors, including CPMs published between 1 January 2017 and 1 May 2020. External validation was performed using a surgical cohort consisting of 292 elderly non-ICU patients.

Primary outcome measures: Discrimination, calibration and clinical usefulness.

Results: 14 CPMs were eligible for analysis out of 366 full texts reviewed. External validation was previously published for 8/14 (57%) CPMs. C-indices ranged from 0.52 to 0.74, intercepts from -0.02 to 0.34, slopes from -0.74 to 1.96 and scaled Brier from -1.29 to 0.088. Based on predefined criteria, the two best performing models were those of Dai et al (c-index: 0.739; (95% CI: 0.664 to 0.813); intercept: -0.018; slope: 1.96; scaled Brier: 0.049) and Litaker et al (c-index: 0.706 (95% CI: 0.590 to 0.823); intercept: -0.015; slope: 0.995; scaled Brier: 0.088). For the remaining CPMs, model discrimination was considered poor with corresponding c-indices <0.70.

Conclusion: Our head-to-head analysis identified 2 out of 14 CPMs as best-performing models with a fair discrimination and acceptable calibration. Based on our findings, these models might assist physicians in postoperative delirium risk estimation and patient selection for preventive measures.

Keywords: delirium & cognitive disorders; geriatric medicine; internal medicine; risk management; surgery.

Conflict of interest statement

Competing interests: None declared.

© Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Figures

Figure 1

Flow chart indicating the selection process of included delirium prediction models.

Figure 2

Head-to-head comparison of discriminative power of delirium prediction models. Discriminative power of externally validated delirium prediction models is reported as c-indices with associated 95% CIs, ranked from low to high. A c-index of 0.5 resembles a situation in which the model has no discriminative power, that is, the model predicts no better than flipping a coin. Only 2 out of 14 validated models showed fair discrimination with c-indices >0.70 (0.71 and 0.74 for the models developed by Litaker et al and Dai et al, respectively) and 95% CIs with lower bounds >0.50. Discriminative power of the remaining 12 models was considered poor.

Figure 3

Discrimination, calibration and clinical utility of best performing models. Panels A and B show the receiver operating characteristic (ROC) curve of the delirium prediction models by Litaker et al and Dai et al, respectively, with the area under the ROC curve (c-index) indicating the discriminative power of the model. A graphical representation of the calibration of both models is shown in panels C and D, plotting the predicted probability (x-axis) with corresponding 95% CI against the actually observed occurrence of delirium in the validation cohort (y-axis). The model by Litaker et al showed adequate calibration (panel C), correctly differentiating patients at low risk of delirium (20%). The model by Dai et al correctly identified patients at low risk (20%). Panels E and F show decision curve analyses as a measure of clinical utility of both models. For the models by Litaker et al and Dai et al, a positive net benefit was observed in the 10%–35% threshold probability range (panel E) and the 5%–20% threshold probability range (panel F), respectively.