Disease-Course Adapting Machine Learning Prognostication Models in Elderly Patients Critically Ill With COVID-19: Multicenter Cohort Study With External Validation

Abstract

Background: The COVID-19 pandemic caused by SARS-CoV-2 is challenging health care systems globally. The disease disproportionately affects the elderly population, both in terms of disease severity and mortality risk.

Objective: The aim of this study was to evaluate machine learning-based prognostication models for critically ill elderly COVID-19 patients, which dynamically incorporated multifaceted clinical information on evolution of the disease.

Methods: This multicenter cohort study (COVIP study) obtained patient data from 151 intensive care units (ICUs) from 26 countries. Different models based on the Sequential Organ Failure Assessment (SOFA) score, logistic regression (LR), random forest (RF), and extreme gradient boosting (XGB) were derived as baseline models that included admission variables only. We subsequently included clinical events and time-to-event as additional variables to derive the final models using the same algorithms and compared their performance with that of the baseline group. Furthermore, we derived baseline and final models on a European patient cohort, which were externally validated on a non-European cohort that included Asian, African, and US patients.

Results: In total, 1432 elderly (≥70 years old) COVID-19-positive patients admitted to an ICU were included for analysis. Of these, 809 (56.49%) patients survived up to 30 days after admission. The average length of stay was 21.6 (SD 18.2) days. Final models that incorporated clinical events and time-to-event information provided superior performance (area under the receiver operating characteristic curve of 0.81; 95% CI 0.804-0.811), with respect to both the baseline models that used admission variables only and conventional ICU prediction models (SOFA score, P<.001). The average precision increased from 0.65 (95% CI 0.650-0.655) to 0.77 (95% CI 0.759-0.770).

Conclusions: Integrating important clinical events and time-to-event information led to a superior accuracy of 30-day mortality prediction compared with models based on the admission information and conventional ICU prediction models. This study shows that machine-learning models provide additional information and may support complex decision-making in critically ill elderly COVID-19 patients.

Trial registration: ClinicalTrials.gov NCT04321265; https://ichgcp.net/clinical-trials-registry/NCT04321265.

Keywords: COVID-19; clinical informatics; elderly population; machine learning; machine-based learning; outcome prediction; pandemic; patient data; prediction models.

Conflict of interest statement

Conflicts of Interest: None declared.

©Christian Jung, Behrooz Mamandipoor, Jesper Fjølner, Raphael Romano Bruno, Bernhard Wernly, Antonio Artigas, Bernardo Bollen Pinto, Joerg C Schefold, Georg Wolff, Malte Kelm, Michael Beil, Sigal Sviri, Peter V van Heerden, Wojciech Szczeklik, Miroslaw Czuczwar, Muhammed Elhadi, Michael Joannidis, Sandra Oeyen, Tilemachos Zafeiridis, Brian Marsh, Finn H Andersen, Rui Moreno, Maurizio Cecconi, Susannah Leaver, Dylan W De Lange, Bertrand Guidet, Hans Flaatten, Venet Osmani. Originally published in JMIR Medical Informatics (https://medinform.jmir.org), 31.03.2022.

Figures

Figure 1

Graphical methods. (1) Study design, from admission to derivation and validation of baseline setup. (2) Derivation and validation of six models incorporating clinical events individually.Performance of individual models is shown in Multimedia Appendix 2-5. (3) Derivation of the final model, including baseline variables as well as clinical events. (4) Evaluation of the final model in predicting 30-day outcomes. SOFA: Sequential Organ Failure Assessment; ICU: intensive care unit.

Figure 2

Performance of the baseline model (top) and improved performance in the final model (bottom) in response to clinical events with respect to the area under the receiver operating characteristic (ROC) curve (AUC) and area under the precision-recall curve (PRC). The PRC shows the relationship between the positive predictive value (precision) and sensitivity (recall) at all thresholds. XGB: extreme gradient boosting; RF: random forest; LR: logistic regression; SOFA: Sequential Organ Failure Assessment.

Figure 3

Performance of the final model derived using the EU patient cohort and externally validated on a non-EU patient cohort, comprising Asian, African, and US patients. Model performance is measured using area under the receiver operating characteristic (ROC) curve (AUC) and area under the precision-recall curve (PRC). XGB: extreme gradient boosting; RF: random forest; LR: logistic regression.

Figure 4

Ranking of input variables of the final setup derived from the extreme gradient boost algorithm, using the shapely additive explanation (SHAP) method.

Figure 5

Calibration curves for each model and individual algorithms used to derive the model. XGB, extreme gradient boosting; RF: random forest; LR: logistic regression.