Use of Machine Learning to Develop and Evaluate Models Using Preoperative and Intraoperative Data to Identify Risks of Postoperative Complications

Abstract

Importance: Postoperative complications can significantly impact perioperative care management and planning.

Objectives: To assess machine learning (ML) models for predicting postoperative complications using independent and combined preoperative and intraoperative data and their clinically meaningful model-agnostic interpretations.

Design, setting, and participants: This retrospective cohort study assessed 111 888 operations performed on adults at a single academic medical center from June 1, 2012, to August 31, 2016, with a mean duration of follow-up based on the length of postoperative hospital stay less than 7 days. Data analysis was performed from February 1 to September 31, 2020.

Main outcomes and measures: Outcomes included 5 postoperative complications: acute kidney injury (AKI), delirium, deep vein thrombosis (DVT), pulmonary embolism (PE), and pneumonia. Patient and clinical characteristics available preoperatively, intraoperatively, and a combination of both were used as inputs for 5 candidate ML models: logistic regression, support vector machine, random forest, gradient boosting tree (GBT), and deep neural network (DNN). Model performance was compared using the area under the receiver operating characteristic curve (AUROC). Model interpretations were generated using Shapley Additive Explanations by transforming model features into clinical variables and representing them as patient-specific visualizations.

Results: A total of 111 888 patients (mean [SD] age, 54.4 [16.8] years; 56 915 [50.9%] female; 82 533 [73.8%] White) were included in this study. The best-performing model for each complication combined the preoperative and intraoperative data with the following AUROCs: pneumonia (GBT), 0.905 (95% CI, 0.903-0.907); AKI (GBT), 0.848 (95% CI, 0.846-0.851); DVT (GBT), 0.881 (95% CI, 0.878-0.884); PE (DNN), 0.831 (95% CI, 0.824-0.839); and delirium (GBT), 0.762 (95% CI, 0.759-0.765). Performance of models that used only preoperative data or only intraoperative data was marginally lower than that of models that used combined data. When adding variables with missing data as input, AUROCs increased from 0.588 to 0.905 for pneumonia, 0.579 to 0.848 for AKI, 0.574 to 0.881 for DVT, 0.5 to 0.831 for PE, and 0.6 to 0.762 for delirium. The Shapley Additive Explanations analysis generated model-agnostic interpretation that illustrated significant clinical contributors associated with risks of postoperative complications.

Conclusions and relevance: The ML models for predicting postoperative complications with model-agnostic interpretation offer opportunities for integrating risk predictions for clinical decision support. Such real-time clinical decision support can mitigate patient risks and help in anticipatory management for perioperative contingency planning.

Conflict of interest statement

Conflict of Interest Disclosures: Dr King reported receiving grants from the National Institute of General Medical Sciences during the conduct of the study. Dr Wildes reported receiving grants from the National Institute of Nursing Research during the conduct of the study. Dr Kannampallil reported receiving personal fees from Pfizer Inc outside the submitted work. No other disclosures were reported.

Figures

Figure 1.. Flowchart of Complication Analysis and Cohort Split

AKI indicates acute kidney injury, DVT, deep vein thrombosis; PE, pulmonary embolism.

Figure 2.. Results of Machine Learning Models

A, Areas under the receiver operating characteristic curve (AUROCs) of best-performing learning models. B, AUROCs when using only preoperative data, intraoperative data, and combined data. C, AUROCs with added features in ascending order of missing rate. D, AUROCs with varied number of features. AKI indicates acute kidney injury; DVT, deep vein thrombosis; PE, pulmonary embolism. The error bars indicate 95% CIs.

Figure 3.. Complication-Specific Model Interpretation

A, Evolvement of risks (from top to bottom) contributed by each variable (magnitude of contribution decreasing from left to right) compared with a group of patients who did not have pneumonia. B, Characterization of significant intraoperative time series (in this case, it is the noninvasive mean blood pressure [MBP]) by its statistical features. Each statistical feature is normalized to zero mean and unit variance; therefore, the magnitude reflects its deviation from the historical mean of patients. BMI indicates body mass index (calculated as weight in kilograms divided by height in meters squared); ETCO2, end-tidal carbon dioxide; SHAP, Shapley Additive Explanations.