Machine learning-based mortality prediction of patients undergoing cardiac resynchronization therapy: the SEMMELWEIS-CRT score

Abstract

Aims: Our aim was to develop a machine learning (ML)-based risk stratification system to predict 1-, 2-, 3-, 4-, and 5-year all-cause mortality from pre-implant parameters of patients undergoing cardiac resynchronization therapy (CRT).

Methods and results: Multiple ML models were trained on a retrospective database of 1510 patients undergoing CRT implantation to predict 1- to 5-year all-cause mortality. Thirty-three pre-implant clinical features were selected to train the models. The best performing model [SEMMELWEIS-CRT score (perSonalizEd assessMent of estiMatEd risk of mortaLity With machinE learnIng in patientS undergoing CRT implantation)], along with pre-existing scores (Seattle Heart Failure Model, VALID-CRT, EAARN, ScREEN, and CRT-score), was tested on an independent cohort of 158 patients. There were 805 (53%) deaths in the training cohort and 80 (51%) deaths in the test cohort during the 5-year follow-up period. Among the trained classifiers, random forest demonstrated the best performance. For the prediction of 1-, 2-, 3-, 4-, and 5-year mortality, the areas under the receiver operating characteristic curves of the SEMMELWEIS-CRT score were 0.768 (95% CI: 0.674-0.861; P < 0.001), 0.793 (95% CI: 0.718-0.867; P < 0.001), 0.785 (95% CI: 0.711-0.859; P < 0.001), 0.776 (95% CI: 0.703-0.849; P < 0.001), and 0.803 (95% CI: 0.733-0.872; P < 0.001), respectively. The discriminative ability of our model was superior to other evaluated scores.

Conclusion: The SEMMELWEIS-CRT score (available at semmelweiscrtscore.com) exhibited good discriminative capabilities for the prediction of all-cause death in CRT patients and outperformed the already existing risk scores. By capturing the non-linear association of predictors, the utilization of ML approaches may facilitate optimal candidate selection and prognostication of patients undergoing CRT implantation.

Keywords: Cardiac resynchronization therapy; Heart failure; Machine learning; Mortality prediction; Precision medicine; Risk stratification.

© The Author(s) 2020. Published by Oxford University Press on behalf of the European Society of Cardiology.

Figures

Figure 1

Computation of survival probabilities: illustrating the methodology through an example case. (A) The outputs of each model were series of class membership probabilities. (B) Cumulative probabilities were calculated by summing these values until the given year of follow-up. (C) To calibrate the cumulative probabilities, Platt’s scaling was performed. Using these calibrated cumulative probabilities, the survival curve could be plotted for each patient. (D) Then, the expected survival time of each patient was estimated from the annual survival probabilities. Pi, the calibrated cumulative probability of all-cause mortality at year i.

Figure 2

Receiver operating characteristic curve analysis of the evaluated risk scores. Calibrated cumulative probabilities were used in the receiver operating characteristic curve analysis.

Figure 3

The 12 most important predictors of all-cause mortality as assessed by the SEMMELWEIS-CRT score. The importance of each feature was quantified by calculating the decrease in the model’s performance (area under the receiver operating characteristic curve) after permuting its values (permutation feature importances method). The higher its value, the more important the feature is. As the values of feature importances were spread over a wide range (more orders of magnitude), base-10 logarithmic transformation was performed to facilitate plotting. CRT, cardiac resynchronization therapy; LVEF, left ventricular ejection fraction; NYHA, New York Heart Failure Association functional class.

Figure 4

Effect of the nine most important continuous features on the calibrated cumulative probability of mortality in the test cohort. Annual probabilities of each patient are marked with different colours (five dots per patient on each plot): 1-year (blue), 2-year (orange), 3-year (green), 4-year (red), and 5-year (purple) calibrated cumulative probabilities. Second order polynomial trendlines are fitted to each year’s probabilities. EF, ejection fraction; GFR, glomerular filtration rate.

Figure 5

Survival analysis of the quartiles. Based on the predicted probability of death, patients were split into four quartiles at each year of follow-up. The survival of the quartiles was visualized on Kaplan–Meier curves and log-rank test was performed for comparison.

Take home figure

Using commonly available pre-implant clinical variables, the machine learning-based SEMMELWEIS-CRT score (available at semmelweiscrtscore.com) can effectively predict all-cause mortality of patients undergoing cardiac resynchronization therapy. AUC, area under the receiver operating characteristic curve; CRT, cardiac resynchronization therapy; ECG, electrocardiogram.