Multimodal Deep Learning for the Diagnosis and Assessment of Alzheimer's Disease

October 7, 2023 updated by: Liu Huayan, First Hospital of China Medical University

Alzheimer's disease (AD) is the most common dementia and has been one of the most expensive diseases with the highest lethality. With the rapid increase of the aging population, more and more burdens will be posed on society and economics. The manifestations of AD are the progressive loss of memory, language and visuospatial function, executive and daily living abilities, and so forth. The Pathophysiological changes of AD occur 10-20 years before the clinical symptoms, while there is still a lack of effective strategy for early diagnosis. Mild cognitive impairment (MCI) is considered to be a transitional state between healthy aging and the clinical diagnosis of dementia and has received increasing attention as a separate diagnostic entity.

To make the diagnosis, doctors ought to compressively consider the multimodal medical information including clinical symptoms, neuroimages, neuropsychological tests, laboratory examinations, etc. Multimodal deep learning has risen to this challenge， which could integrate the various modalities of biological information and capture the relationships among them contributing to higher accuracy and efficiency. It has been widely applied in imaging, tumor pathology, genomics, etc. Recently, the studies on AD based on deep learning still mainly focused on multimodal neuroimaging, while multimodal medical information requires comprehensive integration and intellectual analysis. Moreover, studies reveal that some imperceptible symptoms in MCI and the early stage of AD may also play an effective role in diagnosis and assessment, such as gait disorder, facial expression identification dysfunction, and speech and language impairment. However, doctors could hardly detect the slight and complex changes, which could rely on the full mining of the video and audio information by multimodal deep learning.

In conclusion, we aim to explore the features of gait disorder, facial expression identification dysfunction, and speech and language impairment in MCI and AD, and analyze their diagnostic efficiency. We would identify the different degrees of dependency on multimodal medical information in diagnosis and finally build an optimal multimodal diagnostic method utilizing the most convenient and economical information. Besides, based on follow-up observations on the changes in multimodal medical information with the progress of AD and MCI, we expect to establish an effective and convenient diagnostic strategy.

Study Overview

Status

Not yet recruiting

Conditions

Intervention / Treatment

Detailed Description

Our objective is to make the early diagnosis and assessment of AD and MCI based on multimodal deep learning. Initially, Gait disorder, facial expression identification dysfunction, and speech and language impairment are of great significance in the occurrence and development of AD and MCI. However, due to the high complexity and stealthiness of these clinical symptoms, no uniform conclusions have been made. Hence, we attempt to apply machine learning methods to recognize the video and audio information. In this way, we will explore the changing characteristics of gait, expression, and language in AD and MCI, and analyze their diagnostic effectiveness as diagnostic markers, finally providing new ideas and experimental data for the diagnosis of AD and MCI. Secondly, multimodal medical information needs to be integrated and comprehensively analyzed. We aim to propose an optimal diagnostic strategy referring to the different degrees of dependency on multimodal medical information in diagnosis. Moreover, observing the changes in multimodal medical information with the progress of AD and MCI, we expect to build a predicting model of AD diagnosis and prognosis.

The methods are as follows:

  1. Collecting multimodal medical information A variety of multimodal medical information would be carefully collected including the baseline demographic data, chief complaint and medical history, peripheral organ function assessment, laboratory examination, imaging examination, neuroelectrophysiological examination, neurocognitive and psychological examination, information on gait, expression, and language, and biological samples, etc.

  2. Revealing the changes of gait, expression, and language in patients with AD and MCI, and verifying their diagnostic efficacy.

    For multimodal medical information on gait, OpenPose model was used to extract human key points and construct a human skeleton structure diagram. Based on graph neural networks and convolutional neural networks, instantaneous action analysis of single-frame images is carried out. And then utilizing the Transformer model, gait sequence analysis is carried out by integrating multi-frame video.For multimodal medical information on facial expression, the Dlib algorithm will be used to extract facial key points, combined with facial expression images, and the spatiotemporal Transformer model will be used for facial expression analysis. For multimodal medical information on language, ASRT model will be used for speech recognition and text content extraction. Simultaneously, the frequency domain Fourier transform and wavelet transform will be applied to extract frequency domain information and analyze the speech features by integrating language content, voice intonation, speech speed, and other information. Based on the attention model, the gait, expression, and language analysis results of AD and MCI will be compared with those of the control group to reveal the features of AD and MCI and provide evidence for disease diagnosis.

  3. Analyzing the different degrees of dependency on multimodal information in the diagnosis of AD and MCI diseases, and establishing an optimal diagnosis strategy In the supervised learning process, the attention mechanism-based method will be used to analyze the influence of multimodal information on the final results. At the same time, based on the knowledge map, the patient's blood biochemical indicators, genomic information and other fields of knowledge would be added to the model. Based on Bayesian probability inference and causal inference theory, the causal programming method will be used to model the causal analysis of information and diagnosis results of different modes. Based on AutoML method, multimodal information will be combined and optimized, and a reliable optimal diagnosis strategy will be established according to experimental results.
  4. Exploring the changes of multimodal medical information with the progression of the disease, and build a predicting model for early diagnosis and disease progression of AD.

Viewing multimodal medical information as the control condition, the Transformer model will be used to model time sequence information, and the conditional diffusion model will be used to generate patients' MRI image changes and other disease progression-related information, providing the basis for disease progression prediction. Based on the large multimodal model technology, the output of the model will be interfered with and adjusted referring to the judgment and description of professional doctors, to generate the prediction in line with the judgment of professional doctors, and finally construct the interpretable early diagnosis and disease progression prediction model.

Study Type

Observational

Enrollment (Estimated)

300

Contacts and Locations

This section provides the contact details for those conducting the study, and information on where this study is being conducted.

Study Contact

  • Name: Huayan Liu, PhD.
  • Phone Number: +86 13609831417
  • Email: liuhy@cmu1h.com

Study Contact Backup

Participation Criteria

Researchers look for people who fit a certain description, called eligibility criteria. Some examples of these criteria are a person's general health condition or prior treatments.

Eligibility Criteria

Ages Eligible for Study

  • Adult
  • Older Adult

Accepts Healthy Volunteers

Yes

Sampling Method

Non-Probability Sample

Study Population

We plan to include 100 AD, 100 MCI, and 100 control participants. These participants are those who visit our hospital from 1st October 2023 to 1st March 2024.

Description

Inclusion Criteria:

  1. . Participants' age is between 50 and 85 years old, male or female;
  2. . Participants graduated from primary school or above, with normal hearing, vision, and pronunciation, using Chinese as their mother tongue and Mandarin as their daily language；
  3. . The diagnosis of AD and MCI participants conform to the corresponding diagnostic criteria mentioned above;
  4. . The scores of MMSE are between 10 and 28, and the scores of CDR are no more than 2.
  5. . Patients or family members agree to sign informed consent.

Exclusion Criteria:

  1. . Participants suffer from neurological disorders that could cause dysfunction of the brain, such as depression, tumors, Parkinson's disease, metabolic encephalopathy, encephalitis, multiple sclerosis, epilepsy, brain trauma, normal cranial pressure hydrocephalus, and so forth;
  2. . Participants suffer from systematic diseases that could cause cognitive impairment, such as liver insufficiency, renal insufficiency, thyroid dysfunction, severe anemia, folic acid or vitamin B12 deficiency, syphilis, HIV infection, alcohol and drug abuse, and so forth;
  3. . Participants suffer from diseases that are unable to cooperate with the examinations;
  4. . Participants cannot take magnetic resonance imaging;
  5. . Participants suffer from mental and neurodevelopmental retardation;
  6. . Participants refuse to sign informed consent.

Study Plan

This section provides details of the study plan, including how the study is designed and what the study is measuring.

How is the study designed?

Design Details

Cohorts and Interventions

Group / Cohort
Intervention / Treatment
Alzheimer's disease
the diagnosis of AD is according to the recommendations from the National Institute on Aging-Alzheimer's Association workgroups on diagnostic guidelines for AD.
Diagnostic test: gait video; speech video; facial expression video;
The videos of participants' gait, facial expression, and speech will be recorded and analyzed further. Other routine diagnostic tests will also be performed such as imaging of MRI, cognitive scales, etc.
Other Names:
  • Other routine diagnostic tests such as imaging, cognitive scales, etc.
Mild cognitive impairment
the diagnosis of MCI refers to the criteria defined by Peterson in 2004.
Diagnostic test: gait video; speech video; facial expression video;
The videos of participants' gait, facial expression, and speech will be recorded and analyzed further. Other routine diagnostic tests will also be performed such as imaging of MRI, cognitive scales, etc.
Other Names:
  • Other routine diagnostic tests such as imaging, cognitive scales, etc.
Control
participants who are age-matched with AD and MCI participants, without cognitive impairment.
Diagnostic test: gait video; speech video; facial expression video;
The videos of participants' gait, facial expression, and speech will be recorded and analyzed further. Other routine diagnostic tests will also be performed such as imaging of MRI, cognitive scales, etc.
Other Names:
  • Other routine diagnostic tests such as imaging, cognitive scales, etc.

What is the study measuring?

Primary Outcome Measures

Outcome Measure
Measure Description
Time Frame
The diagnostic efficiency of multimodal deep learning diagnostic strategy
Time Frame: The outcome will be measured and analyzed once all the baseline multimodal medical information has been collected.
The diagnostic efficiency will be measured by the area under curve（AUC）of receiver operating characteristic（ROC）curve.
The outcome will be measured and analyzed once all the baseline multimodal medical information has been collected.

Secondary Outcome Measures

Outcome Measure
Measure Description
Time Frame
The prognostic efficiency of multimodal deep learning prognostic strategy
Time Frame: The outcome will be measured and analyzed once all two-year follow-up multimodal medical information has been collected.
The prognostic efficiency will be measured by the area under curve（AUC）of receiver operating characteristic（ROC）curve.
The outcome will be measured and analyzed once all two-year follow-up multimodal medical information has been collected.

Collaborators and Investigators

This is where you will find people and organizations involved with this study.

Sponsor

First Hospital of China Medical University

Investigators

  • Study Chair: Huayan Liu, the first affiliated hospital of China medical university, neurology department

Publications and helpful links

The person responsible for entering information about the study voluntarily provides these publications. These may be about anything related to the study.

General Publications

Study record dates

These dates track the progress of study record and summary results submissions to ClinicalTrials.gov. Study records and reported results are reviewed by the National Library of Medicine (NLM) to make sure they meet specific quality control standards before being posted on the public website.

Study Major Dates

Study Start (Estimated)

October 15, 2023

Primary Completion (Estimated)

October 15, 2024

Study Completion (Estimated)

October 15, 2026

Study Registration Dates

First Submitted

October 7, 2023

First Submitted That Met QC Criteria

October 7, 2023

First Posted (Actual)

October 13, 2023

Study Record Updates

Last Update Posted (Actual)

October 13, 2023

Last Update Submitted That Met QC Criteria

October 7, 2023

Last Verified

October 1, 2023

More Information

Terms related to this study

Keywords

Additional Relevant MeSH Terms

Other Study ID Numbers

  • LHuayan

Plan for Individual participant data (IPD)

Plan to Share Individual Participant Data (IPD)?

YES

IPD Plan Description

There is a plan to make IPD and related data dictionaries available.

IPD Sharing Time Frame

starting 12 months after publication

IPD Sharing Access Criteria

the IPD and any additional supporting information will be shared with the researchers who follow our idea and theory, and concern on the AD and MCI diagnosis. Huayan Liu will review the requests.

IPD Sharing Supporting Information Type

  • STUDY_PROTOCOL
  • SAP
  • ICF
  • CSR

Drug and device information, study documents

Studies a U.S. FDA-regulated drug product

No

Studies a U.S. FDA-regulated device product

No

This information was retrieved directly from the website clinicaltrials.gov without any changes. If you have any requests to change, remove or update your study details, please contact register@clinicaltrials.gov. As soon as a change is implemented on clinicaltrials.gov, this will be updated automatically on our website as well.

Clinical Trials on Alzheimer Disease

Clinical Trials on gait video; speech video; facial expression video;

Search Similar Trials

Sponsors and Collaborators

Medical Conditions

Drug Interventions

CROs by country

CROs in Saudi Arabia

Conditions

Rare Diseases

Drug Interventions

Dietary Supplements

Sponsor/Collaborators

Locations

Subscribe