Development and External Validation of a Deep Learning Algorithm for Prognostication of Cardiovascular Outcomes

In Jeong Cho, Ji Min Sung, Hyeon Chang Kim, Sang Eun Lee, Myeong Hun Chae, Maryam Kavousi, Oscar L Rueda-Ochoa, M Arfan Ikram, Oscar H Franco, James K Min, Hyuk Jae Chang, In Jeong Cho, Ji Min Sung, Hyeon Chang Kim, Sang Eun Lee, Myeong Hun Chae, Maryam Kavousi, Oscar L Rueda-Ochoa, M Arfan Ikram, Oscar H Franco, James K Min, Hyuk Jae Chang

Abstract

Background and objectives: We aim to explore the additional discriminative accuracy of a deep learning (DL) algorithm using repeated-measures data for identifying people at high risk for cardiovascular disease (CVD), compared to Cox hazard regression.

Methods: Two CVD prediction models were developed from National Health Insurance Service-Health Screening Cohort (NHIS-HEALS): a Cox regression model and a DL model. Performance of each model was assessed in the internal and 2 external validation cohorts in Koreans (National Health Insurance Service-National Sample Cohort; NHIS-NSC) and in Europeans (Rotterdam Study). A total of 412,030 adults in the NHIS-HEALS; 178,875 adults in the NHIS-NSC; and the 4,296 adults in Rotterdam Study were included.

Results: Mean ages was 52 years (46% women) and there were 25,777 events (6.3%) in NHIS-HEALS during the follow-up. In internal validation, the DL approach demonstrated a C-statistic of 0.896 (95% confidence interval, 0.886-0.907) in men and 0.921 (0.908-0.934) in women and improved reclassification compared with Cox regression (net reclassification index [NRI], 24.8% in men, 29.0% in women). In external validation with NHIS-NSC, DL demonstrated a C-statistic of 0.868 (0.860-0.876) in men and 0.889 (0.876-0.898) in women, and improved reclassification compared with Cox regression (NRI, 24.9% in men, 26.2% in women). In external validation applied to the Rotterdam Study, DL demonstrated a C-statistic of 0.860 (0.824-0.897) in men and 0.867 (0.830-0.903) in women, and improved reclassification compared with Cox regression (NRI, 36.9% in men, 31.8% in women).

Conclusions: A DL algorithm exhibited greater discriminative accuracy than Cox model approaches.

Trial registration: ClinicalTrials.gov Identifier: NCT02931500.

Keywords: Artificial intelligence; Cardiovascular diseases.

Conflict of interest statement

Myeong-Hun Chae is an employee of Selvas AI Inc, which contributed to the development of deep learning models described in the study. All other authors have reported that they have no relationships relevant to the contents of this paper to disclose.

Copyright © 2020. The Korean Society of Cardiology.

Figures

Figure 1. Formation of the development, internal…
Figure 1. Formation of the development, internal validation, and external validation cohorts.
CVD = cardiovascular disease; NHIS-HEALS = National Health Insurance Service-Health Screening Cohort; NHIS-NSC = National Health Insurance Service-National Sample Cohort. *Individuals with CVD and non-CVD were defined according to CVD occurrence during the mean 9.8 ± 2.2 years follow-up period to 2013; †123,601 out of 412,030 individuals were randomly selected as the final dataset to deal with the imbalanced data between CVD and non-CVD. During the matching process for the development and validation datasets, 288,429 non-CVD were excluded.
Figure 2. Predicted vs. observed probability of…
Figure 2. Predicted vs. observed probability of cardiovascular disease by deep learning in the internal validation and external validation cohorts.

References

    1. Conroy RM, Pyörälä K, Fitzgerald AP, et al. Estimation of ten-year risk of fatal cardiovascular disease in Europe: the SCORE project. Eur Heart J. 2003;24:987–1003.
    1. Hippisley-Cox J, Coupland C, Vinogradova Y, Robson J, May M, Brindle P. Derivation and validation of QRISK, a new cardiovascular disease risk score for the United Kingdom: prospective open cohort study. BMJ. 2007;335:136.
    1. D'Agostino RB, Sr, Grundy S, Sullivan LM, Wilson P; CHD Risk Prediction Group Validation of the Framingham coronary heart disease prediction scores: results of a multiple ethnic groups investigation. JAMA. 2001;286:180–187.
    1. Goldstein BA, Navar AM, Carter RE. Moving beyond regression techniques in cardiovascular risk prediction: applying machine learning to address analytic challenges. Eur Heart J. 2017;38:1805–1814.
    1. Deo RC. Machine learning in medicine. Circulation. 2015;132:1920–1930.
    1. Narain R, Saxena S, Goyal AK. Cardiovascular risk prediction: a comparative study of Framingham and quantum neural network based approach. Patient Prefer Adherence. 2016;10:1259–1270.
    1. Khatibi V, Montazer GA. A fuzzy-evidential hybrid inference engine for coronary heart disease risk assessment. Expert Syst Appl. 2010;37:8536–8542.
    1. Kukar M, Kononenko I, Grošelj C, Kralj K, Fettich J. Analysing and improving the diagnosis of ischaemic heart disease with machine learning. Artif Intell Med. 1999;16:25–50.
    1. Seong SC, Kim YY, Park SK, et al. Cohort profile: the National Health Insurance Service-National Health Screening Cohort (NHIS-HEALS) in Korea. BMJ Open. 2017;7:e016640
    1. Seong SC, Kim YY, Khang YH, et al. Data resource profile: the National Health Information Database of the National Health Insurance Service in South Korea. Int J Epidemiol. 2017;46:799–800.
    1. Hofman A, Brusselle GG, Darwish Murad S, et al. The Rotterdam Study: 2016 objectives and design update. Eur J Epidemiol. 2015;30:661–708.
    1. National Health Insurance Service. National Health Screening Statistical Yearbook 2014. Wonju: National Health Insurance Service; 2015.
    1. López V, Fernández A, García S, Palade V, Herrera F. An insight into classification with imbalanced data: Empirical results and current trends on using data intrinsic characteristics. Inf Sci. 2013;250:113–141.
    1. Cho IJ, Sung JM, Chang HJ, Chung N, Kim HC. Incremental value of repeated risk factor measurements for cardiovascular disease prediction in middle-aged Korean adults: results from the NHIS-HEALS (National Health Insurance System-National Health Screening Cohort) Circ Cardiovasc Qual Outcomes. 2017;10:004197
    1. Hochreiter S, Schmidhuber J. Long short-term memory. Neural Comput. 1997;9:1735–1780.
    1. Liao L, Ahn HI. Combining deep learning and survival analysis for asset health management. Int J Progn Health Manag. 2016;7:020.
    1. Lloyd-Jones DM, Leip EP, Larson MG, et al. Prediction of lifetime risk for cardiovascular disease by risk factor burden at 50 years of age. Circulation. 2006;113:791–798.
    1. Pencina MJ, D'Agostino RB, Sr, Larson MG, Massaro JM, Vasan RS. Predicting the 30-year risk of cardiovascular disease: the Framingham heart study. Circulation. 2009;119:3078–3084.
    1. Ramsay SE, Morris RW, Whincup PH, Papacosta AO, Thomas MC, Wannamethee SG. Prediction of coronary heart disease risk by Framingham and SCORE risk assessments varies by socioeconomic position: results from a study in British men. Eur J Cardiovasc Prev Rehabil. 2011;18:186–193.
    1. Murphy TP, Dhangana R, Pencina MJ, D'Agostino RB., Sr Ankle-brachial index and cardiovascular risk prediction: an analysis of 11,594 individuals with 10-year follow-up. Atherosclerosis. 2012;220:160–167.
    1. Paynter NP, Chasman DI, Buring JE, Shiffman D, Cook NR, Ridker PM. Cardiovascular disease risk prediction with and without knowledge of genetic variation at chromosome 9p21.3. Ann Intern Med. 2009;150:65–72.
    1. Polonsky TS, McClelland RL, Jorgensen NW, et al. Coronary artery calcium score and risk classification for coronary heart disease prediction. JAMA. 2010;303:1610–1616.
    1. Hinton G, Deng L, Yu D, et al. Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Process Mag. 2012;29:82–97.
    1. Kennedy EH, Wiitala WL, Hayward RA, Sussman JB. Improved cardiovascular risk prediction using nonparametric regression and electronic health record data. Med Care. 2013;51:251–258.
    1. Waljee AK, Higgins PD. Machine learning in medicine: a primer for physicians. Am J Gastroenterol. 2010;105:1224–1226.
    1. Jung K, Shah NH. Implications of non-stationarity on predictive modeling using EHRs. J Biomed Inform. 2015;58:168–174.
    1. Ross EG, Shah NH, Dalman RL, Nead KT, Cooke JP, Leeper NJ. The use of machine learning for the identification of peripheral artery disease and future mortality risk. J Vasc Surg. 2016;64:1515–1522.e3.
    1. Antman EM, Loscalzo J. Precision medicine in cardiology. Nat Rev Cardiol. 2016;13:591–602.

Source: PubMed

Sponsorer og samarbejdspartnere

Medicinske tilstande

Narkotikainterventioner

3
Abonner