A hierarchical attention-based multimodal fusion framework for predicting the progression of Alzheimer’s disease
Open accessThe DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Abstract
Early detection and treatment can slow the progression of Alzheimer's Disease (AD), one of the most common neurodegenerative diseases. Recent studies have demonstrated the value of multimodal fusion in early AD detection. However, most approaches to this have failed to consider data modality domains, their relationships, and variations in their relative importance. To address these challenges, we propose a Hierarchical Attention-Based Multimodal Fusion framework (HAMF) that utilizes imaging, genetic and clinical data for early AD detection. In the HAMF model, attention mechanisms are utilized to learn the appropriate weights for each modality and to understand the interaction between modalities through hierarchical attention. HAMF performs better than state-of-the-art methods, achieving an accuracy of 87.2% and an AUC of 0.913, which are superior to unimodal models. By comparing the results of different unimodal and multimodal models, we find that multimodal fusion can improve model performance more than unimodal models and clinical data is the most important modality. Our ablation experiment confirmed the effectiveness of HAMF. Finally, we used SHapley Additive exPlanations (SHAP) to improve the model's interpretability. We provide the model as a guide for future research in the field, and as a framework for generating actional advice and decision support system for clinical practitioners.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
NIA‐AA Research Framework: Toward a biological definition of Alzheimer's disease
Jack, C. R., Bennett, D. A., Blennow, K., Carrillo, M. C., Dunn, B., Haeberlein, S. B., Holtzman, D. M., Jagust, W., Jessen, F., Karlawish, J., et al. (2018) · Alzheimer's & Dementia · doi:10.1016/j.jalz.2018.02.018 · Cited by 10,537
Review of deep learning: concepts, CNN architectures, challenges, applications, future directions
Alzubaidi, L., Zhang, J., Humaidi, A. J., Al-Dujaili, A., Duan, Y., Al-Shamma, O., Santamaría, J., Fadhel, M. A., Al-Amidie, M., & Farhan, L. (2021) · Journal of Big Data · doi:10.1186/s40537-021-00444-8 · Cited by 7,906
Dementia prevention, intervention, and care
Livingston, G., Sommerlad, A., Orgeta, V., Costafreda, S. G., Huntley, J., Ames, D., Ballard, C., Banerjee, S., Burns, A., Cohen-Mansfield, J., et al. (2017) · The Lancet · doi:10.1016/s0140-6736(17)31363-6 · Cited by 6,072
The Alzheimer’s disease neuroimaging initiative (ADNI): MRI methods
Jack, C. R., Bernstein, M. A., Fox, N. C., Thompson, P., Alexander, G., Harvey, D., Borowski, B., Britson, P. J., L. Whitwell, J., Ward, C., et al. (2008) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.21049 · Cited by 4,347
2019 Alzheimer’s disease facts and figures.
Alzheimer's Association (2019) · Alzheimer's & Dementia · doi:10.1016/j.jalz.2019.01.010 · Cited by 2,461
Multimodal Transformer for Unaligned Multimodal Language Sequences
Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1656 · Cited by 2,259 · cited 3×
“While many existing methods adopt a straightforward approach to integrate multimodal information by simply concatenating features extracted from different modalities, there is a growing interest in leveraging attention mechanisms to fuse multimodal data [36 – 38].”
Multimodal deep learning models for early detection of Alzheimer’s disease stage
Venugopalan, J., Tong, L., Hassanzadeh, H. R., & Wang, M. D. (2021) · Scientific Reports · doi:10.1038/s41598-020-74399-w · Cited by 732
Rathore, S., Habes, M., Iftikhar, M. A., Shacklett, A., & Davatzikos, C. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2017.03.057 · Cited by 622
Palmqvist, S., Tideman, P., Cullen, N., Zetterberg, H., Blennow, K., Initiative, T. A. D. N., Dage, J. L., Stomrud, E., Janelidze, S., Mattsson-Carlgren, N., et al. (2021) · Nature Medicine · doi:10.1038/s41591-021-01348-z · Cited by 524
Predicting alzheimer’s disease progression using multi-modal deep learning approach
Lee, G., Nho, K., Kang, B., Sohn, K.-A., Kim, D., Initiative, F. A. D. N., Weiner, M. W., Aisen, P., Petersen, R., Jack, C. R., et al. (2019) · Scientific Reports · doi:10.1038/s41598-018-37769-z · Cited by 406 · cited 1×
“The model performance is improved by improving the expression of important information and suppressing redundant information to obtain a more accurate representation of fusion features [63].”
Cited by
Li, Y., El Habib Daho, M., Conze, P.-H., Zeghlache, R., Le Boité, H., Tadayoni, R., Cochener, B., Lamard, M., & Quellec, G. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108635 · Cited by 274
Zhang, H., Ni, M., Yang, Y., Xie, F., Wang, W., He, Y., Chen, W., & Chen, Z. (2025) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.107085 · Cited by 34
A Review of Multimodal Fusion–Based Deep Learning for Alzheimer’s Disease
Zhang, R., Sheng, J., Zhang, Q., Wang, J., & Wang, B. (2025) · Neuroscience · doi:10.1016/j.neuroscience.2025.04.035 · Cited by 32
Missing-modality enabled multi-modal fusion architecture for medical data
Wang, M., Fan, S., Li, Y., Xie, Z., & Chen, H. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104796 · Cited by 31
Xiao, H., Wang, J., & Wan, S. (2024) · doi:10.1101/2024.09.25.614862 · Cited by 6
Metadata
- Authors
- Lu, P., Hu, L., Mitelpunkt, A., Bhatnagar, S., Lu, L., & Liang, H.
- Venue
- Biomedical Signal Processing and Control
- Publisher
- Elsevier BV
- Published
- 2024-02
- Volume
- 88
- Pages
- 105669
- ISSN
- 1746-8094 (ISSN-L)
- OpenAlex
- W4388473567
- Cited by
- 68
- References
- 18
- FWCI
- 12.17
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A hierarchical attention-based multimodal fusion framework for predicting the progression of Alzheimer’s disease. https://app.cochat.ai/papers/10.1016/j.bspc.2023.105669
- The paper
- Lu, P., Hu, L., Mitelpunkt, A., Bhatnagar, S., Lu, L., & Liang, H. (2024). A hierarchical attention-based multimodal fusion framework for predicting the progression of Alzheimer’s disease. Biomedical Signal Processing and Control. https://doi.org/10.1016/j.bspc.2023.105669
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.