Early prediction of noninvasive ventilation failure after extubation: development and validation of a machine-learning model

Abstract

Background: Noninvasive ventilation (NIV) has been widely used in critically ill patients after extubation. However, NIV failure is associated with poor outcomes. This study aimed to determine early predictors of NIV failure and to construct an accurate machine-learning model to identify patients at risks of NIV failure after extubation in intensive care units (ICUs).

Methods: Patients who underwent NIV after extubation in the eICU Collaborative Research Database (eICU-CRD) were included. NIV failure was defined as need for invasive ventilatory support (reintubation or tracheotomy) or death after NIV initiation. A total of 93 clinical and laboratory variables were assessed, and the recursive feature elimination algorithm was used to select key features. Hyperparameter optimization was conducted with an automated machine-learning toolkit called Neural Network Intelligence. A machine-learning model called Categorical Boosting (CatBoost) was developed and compared with nine other models. The model was then prospectively validated among patients enrolled in the Cardiac Surgical ICU of Zhongshan Hospital, Fudan University.

Results: Of 929 patients included in the eICU-CRD cohort, 248 (26.7%) had NIV failure. The time from extubation to NIV, age, Glasgow Coma Scale (GCS) score, heart rate, respiratory rate, mean blood pressure (MBP), saturation of pulse oxygen (SpO2), temperature, glucose, pH, pressure of oxygen in blood (PaO2), urine output, input volume, ventilation duration, and mean airway pressure were selected. After hyperparameter optimization, our model showed the greatest accuracy in predicting NIV failure (AUROC: 0.872 [95% CI 0.82-0.92]) among all predictive methods in an internal validation. In the prospective validation cohort, our model was also superior (AUROC: 0.846 [95% CI 0.80-0.89]). The sensitivity and specificity in the prediction group is 89% and 75%, while in the validation group they are 90% and 70%. MV duration and respiratory rate were the most important features. Additionally, we developed a web-based tool to help clinicians use our model.

Conclusions: This study developed and prospectively validated the CatBoost model, which can be used to identify patients who are at risk of NIV failure. Thus, those patients might benefit from early triage and more intensive monitoring.

Trial registration: NCT03704324. Registered 1 September 2018, https://register.

Clinicaltrials: gov .

Keywords: Categorical Boosting; Hyperparameter optimization; Non-invasive mechanical ventilation failure; Prospective validation; Recursive feature elimination.

Conflict of interest statement

The authors have no conflict of interests to declare.

© 2022. The Author(s).

Figures

Fig. 1

Flow chart of patient selection. eICU-CRD, eICU Collaborative Research Database; ICU, intensive care unit

Fig. 2

Schematic illustration of the study design. A Patients with NIV initiated within 48 h after extubation in the eICU Collaborative Research Database were included in the study, and 93 variables were extracted. The dataset was divided into a training set (70%) and internal validation set (30%). B The recursive feature elimination algorithms were performed on the training set, and key features were selected. C Hyperparameters was optimized by using an automated machine learning toolkit on the training set. D The developed CatBoost model outperformed other models in both the internal validation and prospective validation sets

Fig. 3

Hyperparameter optimization process with an automated machine learning toolkit. A The blue point represents the result of a trail, and the dark orange line represents the best area under the receiver operating characteristic curve (AUROC). B Each line represents a trail, and the green to red color line represents its AUROC

Fig. 4

Comparison of the full and compact CatBoost models. The full model was developed on the basis of all available features, whereas the compact model was derived on the basis of key features selected by the recursive feature elimination algorithm. Both models had optimized hyperparameters. A Receiver operating characteristic curves (ROCs) of the full and the compact models. Distribution of the effects of each feature on the output of B the full model or C the compact model, estimated using the SHapley Additive exPlanations (SHAP) values. The plot sorts features by the sum of SHAP value magnitudes over all samples. The blue to red color represents the feature value (red high, blue low). The x-axis measures the effects on model output (right positive, left negative)

Fig. 5

Comparison of model performance with other predictive tools and in the internal validation set. A Receiver operating characteristic curves (ROCs) of CatBoost and other predictive tools/factors. B Receiver operating characteristic curves (ROCs) of different models

Fig. 6

Application of the CatBoost model. A Receiver operating characteristic curves of different models in the prospective validation set. B Influence of the SHAP value on model output. C An example of the web-based tool. D The prediction results of CatBoost model and the top ten importance features are summarized. A green bar indicates a protective factor, whereas a red bar represents a risk factor. Bar length corresponds to the magnitude of protection or risk