Multimodal learning of clinically accessible tests to aid diagnosis of neurodegenerative disorders: a scoping review

Guan Huang, Renjie Li, Quan Bai, Jane Alty, Guan Huang, Renjie Li, Quan Bai, Jane Alty

Abstract

With ageing populations around the world, there is a rapid rise in the number of people with Alzheimer's disease (AD) and Parkinson's disease (PD), the two most common types of neurodegenerative disorders. There is an urgent need to find new ways of aiding early diagnosis of these conditions. Multimodal learning of clinically accessible data is a relatively new approach that holds great potential to support early precise diagnosis. This scoping review follows the PRSIMA guidelines and we analysed 46 papers, comprising 11,750 participants, 3569 with AD, 978 with PD, and 2482 healthy controls; the recency of this topic was highlighted by nearly all papers being published in the last 5 years. It highlights the effectiveness of combining different types of data, such as brain scans, cognitive scores, speech and language, gait, hand and eye movements, and genetic assessments for the early detection of AD and PD. The review also outlines the AI methods and the model used in each study, which includes feature extraction, feature selection, feature fusion, and using multi-source discriminative features for classification. The review identifies knowledge gaps around the need to validate findings and address limitations such as small sample sizes. Applying multimodal learning of clinically accessible tests holds strong potential to aid the development of low-cost, reliable, and non-invasive methods for early detection of AD and PD.

Keywords: Age-related diseases; Alzheimer’s disease; Artificial intelligence; Diagnosis; Multimodal learning; Multiple biomarkers; Parkinson’s disease; Pre-clinical.

© The Author(s) 2023.

Figures

Fig. 1
Fig. 1
Model of the cognitive function decline trajectory of Alzheimer’s disease (AD) vs normal ageing. The stage of preclinical AD precedes with mild cognitive impairment (MCI), graph adapted from [10]
Fig. 2
Fig. 2
PRSIMA flow chart of the scoping review. This diagram shows the processing of scoping review, including identification, screening and the number of papers included in our study
Fig. 3
Fig. 3
Meta-data from the review process

References

    1. Hebert LE, Weuve J, Scherr PA, Evans DA. Alzheimer disease in the united states (2010–2050) estimated using the 2010 census. Neurology. 2013;80(19):1778–1783. doi: 10.1212/WNL.0b013e31828726f5.
    1. De Rijk M, Breteler M, Graveland G, Ott A, Grobbee D, Van der Meche F, Hofman A. Prevalence of Parkinson’s disease in the elderly: the Rotterdam study. Neurology. 1995;45(12):2143–2146. doi: 10.1212/WNL.45.12.2143.
    1. Parkinson disease. World Health Organization. (2022).
    1. Livingston G, Huntley J, Sommerlad A, Ames D, Ballard C, Banerjee S, Brayne C, Burns A, Cohen-Mansfield J, Cooper C. Dementia prevention, intervention, and care: 2020 report of the lancet commission. Lancet. 2020;396(10248):413–446. doi: 10.1016/S0140-6736(20)30367-6.
    1. Association A 2019 Alzheimer’s disease facts and figures. Alzheimer’s Dementia. 2019;15(3):321–387. doi: 10.1016/j.jalz.2019.01.010.
    1. Livingston G, Sommerlad A, Orgeta V, Costafreda SG, Huntley J, Ames D, Ballard C, Banerjee S, Burns A, Cohen-Mansfield J. Dementia prevention, intervention, and care. Lancet. 2017;390(10113):2673–2734. doi: 10.1016/S0140-6736(17)31363-6.
    1. Thabtah F, Spencer R, Ye Y. The correlation of everyday cognition test scores and the progression of Alzheimer’s disease: a data analytics study. Health Inf Sci Syst. 2020;8:1–11. doi: 10.1007/s13755-020-00114-8.
    1. Dubois B, Feldman HH, Jacova C, DeKosky ST, Barberger-Gateau P, Cummings J, Delacourte A, Galasko D, Gauthier S, Jicha G. Research criteria for the diagnosis of Alzheimer’s disease: revising the Nincds–Adrda criteria. Lancet Neurol. 2007;6(8):734–746. doi: 10.1016/S1474-4422(07)70178-3.
    1. Dubois B, Hampel H, Feldman HH, Scheltens P, Aisen P, Andrieu S, Bakardjian H, Benali H, Bertram L, Blennow K. Preclinical Alzheimer’s disease: definition, natural history, and diagnostic criteria. Alzheimer’s Dementia. 2016;12(3):292–323. doi: 10.1016/j.jalz.2016.02.002.
    1. Sperling RA, Aisen PS, Beckett LA, Bennett DA, Craft S, Fagan AM, Iwatsubo T, Jack CR, Jr, Kaye J, Montine TJ. Toward defining the preclinical stages of Alzheimer’s disease: recommendations from the national institute on aging-Alzheimer’s association workgroups on diagnostic guidelines for Alzheimer’s disease. Alzheimer’s Dementia. 2011;7(3):280–292. doi: 10.1016/j.jalz.2011.03.003.
    1. Zetterberg H. Blood-based biomarkers for Alzheimer’s disease-an update. J Neurosci Methods. 2019;319:2–6. doi: 10.1016/j.jneumeth.2018.10.025.
    1. Carrette O, Demalte I, Scherl A, Yalkinoglu O, Corthals G, Burkhard P, Hochstrasser DF, Sanchez J-C. A panel of cerebrospinal fluid potential biomarkers for the diagnosis of Alzheimer’s disease. Proteomics. 2003;3(8):1486–1494. doi: 10.1002/pmic.200300470.
    1. Obrien J. Role of imaging techniques in the diagnosis of dementia. Br J Radiol. 2007;80:71–77. doi: 10.1259/bjr/33117326.
    1. Cohen JA, Verghese J. Gait and dementia. Handb Clin Neurol. 2019;167:419–427. doi: 10.1016/B978-0-12-804766-8.00022-4.
    1. Bucks RS, Singh S, Cuerden JM, Wilcock GK. Analysis of spontaneous, conversational speech in dementia of Alzheimer type: Evaluation of an objective technique for analysing lexical performance. Aphasiology. 2000;14(1):71–91. doi: 10.1080/026870300401603.
    1. De Stefano C, Fontanella F, Impedovo D, Pirlo G, di Freca AS. Handwriting analysis to support neurodegenerative diseases diagnosis: a review. Pattern Recogn Lett. 2019;121:37–45. doi: 10.1016/j.patrec.2018.05.013.
    1. Suk H-I, Lee S-W, Shen D, Initiative ADN. Hierarchical feature representation and multimodal fusion with deep learning for AD/MCI diagnosis. Neuroimage. 2014;101:569–582. doi: 10.1016/j.neuroimage.2014.06.077.
    1. Wang X, St George RJ, Bai Q, Tran S, Alty J. New horizons in late-onset essential tremor: a pre-cognitive biomarker of dementia? Age Ageing. 2022;51(7):135. doi: 10.1093/ageing/afac135.
    1. Marras C, Beck J, Bower J, Roberts E, Ritz B, Ross G, Abbott R, Savica R, Van Den Eeden S, Willis A. Parkinson’s foundation p4 group. Prevalence of Parkinson’s disease across North America. NPJ Parkinsons Dis. 2018;4(1):21. doi: 10.1038/s41531-018-0058-0.
    1. Hughes A, Daniel S, Kilford L, Ajjon L. Neurosurgery, psychiatry, accuracy of clinical diagnosis of idiopathic Parkinson’s disease: a clinico-pathological study of 100 cases. J Neural Transm Neurosurg Psych. 1992;55(3):181–4. doi: 10.1136/jnnp.55.3.181.
    1. Li R, St George RJ, Wang X, Lawler K, Hill E, Garg S, Williams S, Relton S, Hogg D, Bai Q. Moving towards intelligent telemedicine: computer vision measurement of human movement. Comput Biol Med. 2022;147:105776. doi: 10.1016/j.compbiomed.2022.105776.
    1. Li R, Wang X, Lawler K, Garg S, Bai Q, Alty J. Applications of artificial intelligence to aid detection of dementia: a scoping review on current capabilities and future directions. J Biomed Inf. 2022;127:104030. doi: 10.1016/j.jbi.2022.104030.
    1. Baltrušaitis T, Ahuja C, Morency L-P. Multimodal machine learning: a survey and taxonomy. IEEE Trans Pattern Anal Mach Intell. 2018;41(2):423–443. doi: 10.1109/TPAMI.2018.2798607.
    1. Palliya Guruge C, Oviatt S, Delir Haghighi P, Pritchard E. Advances in multimodal behavioral analytics for early dementia diagnosis: a review. In: Proceedings of the 2021 International Conference on Multimodal Interaction. 2021. pp. 328–340.
    1. Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, Moher D, Peters MD, Horsley T, Weeks L. Prisma extension for scoping reviews (PRISMA-SCR): checklist and explanation. Ann Intern Med. 2018;169(7):467–473. doi: 10.7326/M18-0850.
    1. Prashanth R, Roy SD, Mandal PK, Ghosh S. High-accuracy detection of early Parkinson’s disease through multimodal features and machine learning. Int J Med Inf. 2016;90:13–21. doi: 10.1016/j.ijmedinf.2016.03.001.
    1. Shi J, Zheng X, Li Y, Zhang Q, Ying S. Multimodal neuroimaging feature learning with multimodal stacked deep polynomial networks for diagnosis of Alzheimer’s disease. IEEE J Biomed Health Inf. 2017;22(1):173–183. doi: 10.1109/JBHI.2017.2655720.
    1. Pahuja G, Nagabhushan T, Prasad B, Pushkarna R. Early detection of Parkinson’s disease through multimodal features using machine learning approaches. Int J Signal Imaging Syst Eng. 2018;11(1):31–43. doi: 10.1504/IJSISE.2018.090605.
    1. Garcia N, Váisquez Correa J.C, Orozco-Arroyave JR, Nöth E. Multimodal I-vectors to detect and evaluate Parkinson’s disease. In: Proc. Interspeech 2018. 2018; pp. 2349–2353. 10.21437/Interspeech.2018-2295
    1. Vásquez-Correa JC, Arias-Vergara T, Orozco-Arroyave JR, Eskofier B, Klucken J, Nöth E. Multimodal assessment of Parkinson’s disease: a deep learning approach. IEEE J Biomed Health Inf. 2018;23(4):1618–1630. doi: 10.1109/JBHI.2018.2866873.
    1. Pham HN, Do TT, Chan KYJ, Sen G, Han AY, Lim P, Cheng TSL, Nguyen QH, Nguyen BP, Chua MC. Multimodal detection of Parkinson disease based on vocal and improved spiral test. In: 2019 International Conference on System Science and Engineering (ICSSE). 2019;pp. 279–284. IEEE.
    1. Noella RN, Gupta D, Priyadarshini J. Diagnosis of Parkinson’s disease using gait dynamics and images. Proc Comput Sci. 2019;165:428–434. doi: 10.1016/j.procs.2020.01.002.
    1. Zhou T, Liu M, Thung K-H, Shen D. Latent representation learning for Alzheimer’s disease diagnosis with incomplete multi-modality neuroimaging and genetic data. IEEE Trans Med Imaging. 2019;38(10):2411–2422. doi: 10.1109/TMI.2019.2913158.
    1. Taleb C, Likforman-Sulem L, Mokbel C, Khachab M. Detection of Parkinson’s disease from handwriting using deep learning: a comparative study. Evol Intell. 2020 doi: 10.1007/s12065-020-00470-0.
    1. Dachena C, Casu S, Lodi MB, Fanti A, Mazzarella G. Application of MRI, fMRI and cognitive data for Alzheimer’s disease detection. In: 2020 14th European Conference on Antennas and Propagation (EuCAP). 2020;pp. 1–4. IEEE.
    1. Koo J, Lee JH, Pyo J, Jo Y, Lee K. Exploiting multi-modal features from pre-trained networks for Alzheimer’s dementia recognition. 2020.
    1. Martinc M, Pollak S. Tackling the address challenge: a multimodal approach to the automated recognition of Alzheimer’s dementia. In: INTERSPEECH. 2020; pp. 2157–2161.
    1. Pompili A, Rolland T, Abad A. The INESC-ID multi-modal system for the address 2020 challenge. 2020.
    1. Sánchez-Reyna AG, Celaya-Padilla JM, Galván-Tejada CE, Luna-García H, Gamboa-Rosales H, Ramirez-Morales A, Galván-Tejada JI, Initiative ADN. Multimodal early Alzheimer’s detection, a genetic algorithm approach with support vector machines. In: Healthcare, vol. 9, 2021; p. 971. MDPI.
    1. Song X, Mao M, Qian X. Auto-metric graph neural network based on a meta-learning strategy for the diagnosis of Alzheimer’s disease. IEEE J Biomed Health Inform. 2021;25(8):3141–3152. doi: 10.1109/JBHI.2021.3053568.
    1. Pandey SK, Shekhawat HS, Bhasin S, Jasuja R, Prasanna S. Alzheimer’s dementia recognition using multimodal fusion of speech and text embeddings. In: International conference on intelligent human computer interaction. 2021; pp. 718–728. Springer.
    1. Arco JE, Ramírez J, Górriz JM, Ruz M, Initiative ADN. Data fusion based on searchlight analysis for the prediction of Alzheimer’s disease. Expert Syst Appl. 2021;185:115549. doi: 10.1016/j.eswa.2021.115549.
    1. Dong A, Li Z, Wang M, Shen D, Liu M. High-order Laplacian regularized low-rank representation for multimodal dementia diagnosis. Front Neurosci. 2021;15:634124. doi: 10.3389/fnins.2021.634124.
    1. Bi X-A, Hu X, Xie Y, Wu H. A novel cernne approach for predicting Parkinson’s disease-associated genes and brain regions based on multimodal imaging genetics data. Med Image Anal. 2021;67:101830. doi: 10.1016/j.media.2020.101830.
    1. Yang Y, Wei L, Hu Y, Wu Y, Hu L, Nie S. Classification of Parkinson’s disease based on multi-modal features and stacking ensemble learning. J Neurosci Methods. 2021;350:109019. doi: 10.1016/j.jneumeth.2020.109019.
    1. Nasreen S, Rohanian M, Hough J, Purver M. Alzheimer’s dementia recognition from spontaneous speech using disfluency and interactional features. Front Comput Sci. 2021;3:640669. doi: 10.3389/fcomp.2021.640669.
    1. Yamada Y, Shinkawa K, Kobayashi M, Caggiano V, Nemoto M, Nemoto K, Arai T. Combining multimodal behavioral data of gait, speech, and drawing for classification of Alzheimer’s disease and mild cognitive impairment. J Alzheimers Dis. 2021;84(1):315–327. doi: 10.3233/JAD-210684.
    1. Nasreen S, Hough J, Purver M Detecting alzheimer’s disease using interactional and acoustic features from spontaneous speech. 2021; Interspeech.
    1. Fukushima A, Morooka R, Tanaka H, Kentaro H, Tugawa A, Hanyu H. Classification of dementia type using the brain–computer interface. Artif Life Robot. 2021;26(2):216–221. doi: 10.1007/s10015-020-00673-9.
    1. Rohanian M, Hough J, Purver M. Multi-modal fusion with gating using audio, lexical and disfluency features for Alzheimer’s dementia recognition from spontaneous speech. 2021.
    1. Jang H, Soroski T, Rizzo M, Barral O, Harisinghani A, Newton-Mason S, Granby S, Stutz da Cunha Vasco TM, Lewis C. Tutt P Classification of Alzheimer’s disease leveraging multi-task machine learning analysis of speech and eye-movement data. Front Hum Neurosci. 2021;15:716670. doi: 10.3389/fnhum.2021.716670.
    1. Sheng J, Xin Y, Zhang Q, Wang L, Yang Z, Yin J. Predictive classification of Alzheimer’s disease using brain imaging and genetic data. Sci Rep. 2022;12(1):1–9. doi: 10.1038/s41598-022-06444-9.
    1. Jiao Z, Chen S, Shi H, Xu J. Multi-modal feature selection with feature correlation and feature structure fusion for mci and ad classification. Brain Sci. 2022;12(1):80. doi: 10.3390/brainsci12010080.
    1. Ilias L, Askounis D. Multimodal deep learning models for detecting dementia from speech and transcripts. Front Aging Neurosci. 2022;14:830943. doi: 10.3389/fnagi.2022.830943.
    1. Min J-Y, Ha S-W, Lee K, Min K-B. Use of electroencephalogram, gait, and their combined signals for classifying cognitive impairment and normal cognition. Front Aging Neurosci. 2022;14:927295. doi: 10.3389/fnagi.2022.927295.
    1. Dolci G, Rahaman MA, Chen J, Duan K, Fu Z, Abrol A, Menegaz G, Calhoun VD. A deep generative multimodal imaging genomics framework for Alzheimer’s disease prediction. In: 2022 IEEE 22nd International Conference on Bioinformatics and Bioengineering (BIBE). 2022; pp. 41–44. IEEE.
    1. Hansen S, Keune J, Küfner K, Meister R, Habich J, Koska J, Förster S, Oschmann P, Keune PM. The congruency of neuropsychological and F18-FDG brain PET/CT diagnostics of Alzheimer’s disease (AD) in routine clinical practice: insights from a mixed neurological patient cohort. BMC Neurol. 2022;22(1):83. doi: 10.1186/s12883-022-02614-4.
    1. El-Sappagh S, Ali F, Abuhmed T, Singh J, Alonso JM. Automatic detection of Alzheimer’s disease progression: an efficient information fusion approach with heterogeneous ensemble classifiers. Neurocomputing. 2022;512:203–224. doi: 10.1016/j.neucom.2022.09.009.
    1. Ying Y, Yang T, Zhou H. Multimodal fusion for Alzheimer’s disease recognition. Appl Intell. 2022;53:1–12.
    1. Moguilner S, Birba A, Fittipaldi S, Gonzalez-Campo C, Tagliazucchi E, Reyes P, Matallana D, Parra MA, Slachevsky A, Farías G. Multi-feature computational framework for combined signatures of dementia in underrepresented settings. J Neural Eng. 2022;19(4):046048. doi: 10.1088/1741-2552/ac87d0.
    1. Dwivedi S, Goel T, Tanveer M, Murugan R, Sharma R. Multimodal fusion-based deep learning network for effective diagnosis of Alzheimer’s disease. IEEE Multimedia. 2022;29(2):45–55. doi: 10.1109/MMUL.2022.3156471.
    1. Habuza T, Zaki N, Mohamed EA, Statsenko Y. Deviation from model of normal aging in Alzheimer’s disease: application of deep learning to structural MRI data and cognitive tests. IEEE Access. 2022;10:53234–53249. doi: 10.1109/ACCESS.2022.3174601.
    1. Velazquez M, Lee Y. Multimodal ensemble model for Alzheimer’s disease conversion prediction from early mild cognitive impairment subjects. Comput Biol Med. 2022;151:106201. doi: 10.1016/j.compbiomed.2022.106201.
    1. Shi Y, Zu C, Hong M, Zhou L, Wang L, Wu X, Zhou J, Zhang D, Wang Y. Asmfs: adaptive-similarity-based multi-modality feature selection for classification of Alzheimer’s disease. Pattern Recogn. 2022;126:108566. doi: 10.1016/j.patcog.2022.108566.
    1. Safai A, Vakharia N, Prasad S, Saini J, Shah A, Lenka A, Pal PK, Ingalhalikar M. Multimodal brain connectomics-based prediction of Parkinson’s disease using graph attention networks. Front Neurosci. 2022;15:1903. doi: 10.3389/fnins.2021.741489.
    1. Zhang W, Yang Z, Li H, Huang D, Wang L, Wei Y, Zhang L, Ma L, Feng H, Pan J. Multimodal data for the detection of freezing of gait in Parkinson’s disease. Sci Data. 2022;9(1):606. doi: 10.1038/s41597-022-01713-8.
    1. Ilias L, Askounis D, Psarras J. Detecting dementia from speech and transcripts using transformers. Comput Speech Lang. 2023;79:101485. doi: 10.1016/j.csl.2023.101485.
    1. Rallabandi VS, Seetharaman K. Deep learning-based classification of healthy aging controls, mild cognitive impairment and Alzheimer’s disease using fusion of MRI-PET imaging. Biomed Signal Process Control. 2023;80:104312. doi: 10.1016/j.bspc.2022.104312.
    1. Goel T, Sharma R, Tanveer M, Suganthan P, Maji K, Pilli R. Multimodal neuroimaging based Alzheimer’s disease diagnosis using evolutionary RVFL classifier. IEEE J Biomed Health Inf. 2023 doi: 10.1109/JBHI.2023.3242354.
    1. Chai J, Wu R, Li A, Xue C, Qiang Y, Zhao J, Zhao Q, Yang Q. Classification of mild cognitive impairment based on handwriting dynamics and GEEG. Comput Biol Med. 2023;152:106418. doi: 10.1016/j.compbiomed.2022.106418.
    1. Lei B, Yang P, Zhuo Y, Zhou F, Ni D, Chen S, Xiao X, Wang T. Neuroimaging retrieval via adaptive ensemble manifold learning for brain disease diagnosis. IEEE J Biomed Health Inform. 2019;23(4):1661–1673. doi: 10.1109/JBHI.2018.2872581.
    1. Asgari M, Kaye J, Dodge H. Predicting mild cognitive impairment from spontaneous spoken utterances. Alzheimer’s Dementia. 2017;3(2):219–228.
    1. Postuma RB, Bertrand J-A, Montplaisir J, Desjardins C, Vendette M, Rios Romenets S, Panisset M, Gagnon J-F. Rapid eye movement sleep behavior disorder and risk of dementia in Parkinson’s disease: a prospective study. Mov Disord. 2012;27(6):720–726. doi: 10.1002/mds.24939.
    1. Wang X, Garg S, Tran SN, Bai Q, Alty J. Hand tremor detection in videos with cluttered background using neural network based approaches. Health Inf Sci Syst. 2021;9:1–14. doi: 10.1007/s13755-021-00159-3.
    1. Thomas M, Lenka A, Kumar Pal P. Handwriting analysis in Parkinson’s disease: current status and future directions. Mov Disorders Clin Pract. 2017;4(6):806–818. doi: 10.1002/mdc3.12552.
    1. Pasha A, Latha P. Bio-inspired dimensionality reduction for Parkinson’s disease (PD) classification. Health Inf Sci Syst. 2020;8:1–22. doi: 10.1007/s13755-020-00104-w.
    1. Shi Y, Ma Q, Feng C, Wang M, Wang H, Li B, Fang J, Ma S, Guo X, Li T. Microstate feature fusion for distinguishing AD from MCI. Health Inf Sci Syst. 2022;10(1):16. doi: 10.1007/s13755-022-00186-8.

Source: PubMed

Sponsors and Collaborators

Medical Conditions

Drug Interventions

3
Subscribe