The DOI matches its Crossref record; 56 of the 58 references it cites were found in a registry or an index.
Cascaded Multi-Modal Mixing Transformers for Alzheimer’s Disease Classification with Incomplete Data
Open accessAbstract
Accurate medical classification requires a large number of multi-modal data, and in many cases, different feature types. Previous studies have shown promising results when using multi-modal data, outperforming single-modality models when classifying diseases such as Alzheimer's Disease (AD). However, those models are usually not flexible enough to handle missing modalities. Currently, the most common workaround is discarding samples with missing modalities which leads to considerable data under-utilisation. Adding to the fact that labelled medical images are already scarce, the performance of data-driven methods like deep learning can be severely hampered. Therefore, a multi-modal method that can handle missing data in various clinical settings is highly desirable. In this paper, we present Multi-Modal Mixing Transformer (3MT), a disease classification transformer that not only leverages multi-modal data but also handles missing data scenarios. In this work, we test 3MT for AD and Cognitively normal (CN) classification and mild cognitive impairment (MCI) conversion prediction to progressive MCI (pMCI) or stable MCI (sMCI) using clinical and neuroimaging data. The model uses a novel Cascaded Modality Transformers architecture with cross-attention to incorporate multi-modal information for more informed predictions. We propose a novel modality dropout mechanism to ensure an unprecedented level of modality independence and robustness to handle missing data scenarios. The result is a versatile network that enables the mixing of arbitrary numbers of modalities with different feature types and also ensures full data utilization in missing data scenarios. The model is trained and evaluated on the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset with the state-of-the-art performance and further evaluated with The Australian Imaging Biomarker & Lifestyle Flagship Study of Ageing (AIBL) dataset with missing data.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
McKhann, G., Drachman, D., Folstein, M., Katzman, R., Price, D., & Stadlan, E. M. (1984) · Neurology · doi:10.1212/wnl.34.7.939 · Cited by 27,915
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
McKhann, G. M., Knopman, D. S., Chertkow, H., Hyman, B. T., Jack, C. R., Kawas, C. H., Klunk, W. E., Koroshetz, W. J., Manly, J. J., Mayeux, R., et al. (2011) · Alzheimer's & Dementia · doi:10.1016/j.jalz.2011.03.005 · Cited by 18,657
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition
(1993) · doi:10.1109/cvpr.1993 · Cited by 17,989
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,881
The Alzheimer’s disease neuroimaging initiative (ADNI): MRI methods
Jack, C. R., Bernstein, M. A., Fox, N. C., Thompson, P., Alexander, G., Harvey, D., Borowski, B., Britson, P. J., L. Whitwell, J., Ward, C., et al. (2008) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.21049 · Cited by 4,347
TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation
Chen, J., Lu, Y., Yu, Q., Luo, X., Adeli, E., Wang, Y., Lu, L., Yuille, A. L., & Zhou, Y. (2021) · arXiv · doi:10.48550/arxiv.2102.04306 · Cited by 4,206
Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers
Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., Torr, P. H. S., et al. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00681 · Cited by 4,008
Self-citationUNETR: Transformers for 3D Medical Image Segmentation
Hatamizadeh, A., Tang, Y., Nath, V., Yang, D., Myronenko, A., Landman, B., Roth, H. R., & Xu, D. (2022) · 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv51458.2022.00181 · Cited by 3,586
Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1406.1078 · Cited by 3,317
2019 Alzheimer’s disease facts and figures.
Alzheimer's Association (2019) · Alzheimer's & Dementia · doi:10.1016/j.jalz.2019.01.010 · Cited by 2,461
Medical Transformer: Gated Axial-Attention for Medical Image Segmentation
Valanarasu, J. M. J., Oza, P., Hacihaliloglu, I., & Patel, V. M. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87193-2_4 · Cited by 1,605
Automatic classification of MR scans in Alzheimer’s disease.
Kloppel, S., Stonnington, C. M., Chu, C., Draganski, B., Scahill, R. I., Rohrer, J. D., Fox, N. C., Jack, C. R., Ashburner, J., & Frackowiak, R. S. J. (2008) · Brain · doi:10.1093/brain/awm319 · Cited by 1,170
Beach, T. G., Monsell, S. E., Phillips, L. E., & Kukull, W. (2012) · Journal of Neuropathology & Experimental Neurology · doi:10.1097/nen.0b013e31824b211b · Cited by 1,103
Blood-based biomarkers for Alzheimer's disease: towards clinical implementation
Teunissen, C. E., Verberk, I. M. W., Thijssen, E. H., Vermunt, L., Hansson, O., Zetterberg, H., van der Flier, W. M., Mielke, M. M., & del Campo, M. (2022) · The Lancet Neurology · doi:10.1016/s1474-4422(21)00361-6 · Cited by 1,045
Ellis, K. A., Bush, A. I., Darby, D., Fazio, D. D., Foster, J. K., Hudson, P. J., Lautenschlager, N. T., Lenzo, N. P., Martins, R. N., Maruff, P. T., et al. (2009) · International Psychogeriatrics · doi:10.1017/s1041610209009405 · Cited by 1,005
Association of Amyloid and Tau With Cognition in Preclinical Alzheimer Disease: A Longitudinal Study
Hanseeuw, B. J., Betensky, R. A., Jacobs, H. I. L., Schultz, A. P., Sepulcre, J., Becker, J. A., Cosio, D. M. O., Farrell, M., Quiroz, Y. T., Mormino, E. C., et al. (2019) · JAMA Neurology · doi:10.1001/jamaneurol.2019.1424 · Cited by 865
Wen, J., Thibeau-Sutre, E., Diaz-Melo, M., Samper-González, J., Routier, A., Bottani, S., Dormont, D., Durrleman, S., Burgos, N., & Colliot, O. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101694 · Cited by 809
Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation
Wang, H., Zhu, Y., Green, B., Adam, H., Yuille, A., & Chen, L.-C. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58548-8_7 · Cited by 809
PET of Brain Amyloid and Tau in Mild Cognitive Impairment
Small, G. W., Kepe, V., Ercoli, L. M., Siddarth, P., Bookheimer, S. Y., Miller, K. J., Lavretsky, H., Burggren, A. C., Cole, G. M., Vinters, H. V., et al. (2006) · New England Journal of Medicine · doi:10.1056/nejmoa054625 · Cited by 731
A comparison between voxel-based cortical thickness and voxel-based morphometry in normal aging
Hutton, C., Draganski, B., Ashburner, J., & Weiskopf, N. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2009.06.043 · Cited by 582
Tau biomarkers in Alzheimer's disease: towards implementation in clinical practice and trials
Ossenkoppele, R., van der Kant, R., & Hansson, O. (2022) · The Lancet Neurology · doi:10.1016/s1474-4422(22)00168-5 · Cited by 557
Lian, C., Liu, M., Zhang, J., & Shen, D. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2889096 · Cited by 544
Identifying and validating biomarkers for Alzheimer's disease
Humpel, C. (2011) · Trends in Biotechnology · doi:10.1016/j.tibtech.2010.09.007 · Cited by 484
Qiu, S., Joshi, P. S., Miller, M. I., Xue, C., Zhou, X., Karjadi, C., Chang, G. H., Joshi, A. S., Dwyer, B., Zhu, S., et al. (2020) · Brain · doi:10.1093/brain/awaa137 · Cited by 472
Deep learning to detect Alzheimer's disease from neuroimaging: A systematic literature review
Ghahnavieh, A. E., Luo, S., & Chiong, R. (2019) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2019.105242 · Cited by 456
2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI)
(2021) · doi:10.1109/isbi48211.2021 · Cited by 437
Perceiver: General Perception with Iterative Attention
Jaegle, A., Gimeno, F., Brock, A., Zisserman, A., Vinyals, O., & Carreira, J. (2021) · arXiv · doi:10.48550/arxiv.2103.03206 · Cited by 414
Lin, W.-M., Tong, T., Gao, Q., Guo, D., Du, X., Yang, Y., Guo, G., Xiao, M., Du, M., Qu, X., et al. (2018) · Frontiers in Neuroscience · doi:10.3389/fnins.2018.00777 · Cited by 406
Lebedev, A. V., Westman, E., van Westen, G. J. P., Kramberger, M. G., Lundervold, A. J., Aarsland, D., Soininen, H. S., Kłoszewska, I., Mecocci, P., Tsolaki, M., et al. (2014) · NeuroImage Clinical · doi:10.1016/j.nicl.2014.08.023 · Cited by 305
Zhang, Y., Dong, Z., Phillips, P., Wang, S., Ji, G., Yang, J., & Yuan, T. (2015) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2015.00066 · Cited by 285
Characterization of Novel CSF Tau and ptau Biomarkers for Alzheimer’s Disease
Meredith Jr., J. E., Sankaranarayanan, S., Guss, V., Lanzetti, A. J., Berisha, F., Neely, R. J., Slemmon, J. R., Portelius, E., Zetterberg, H., Blennow, K., et al. (2013) · PLoS ONE · doi:10.1371/journal.pone.0076523 · Cited by 209
An efficient 3D deep convolutional network for Alzheimer's disease diagnosis using MR images
Bäckström, K., Nazari, M., Gu, I. Y., & Jakola, A. S. (2018) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2018.8363543 · Cited by 143
Qiu, S., Chang, G. H., Panagia, M., Gopal, D. M., Au, R., & Kolachalama, V. B. (2018) · Alzheimer's & Dementia: Diagnosis, Assessment & Disease Monitoring · doi:10.1016/j.dadm.2018.08.013 · Cited by 110
Detecting Neurodegenerative Disease from MRI: A Brief Review on a Deep Learning Perspective
Noor, M. B. T., Zenia, N. Z., Kaiser, M. S., Mahmud, M., & Mamun, S. A. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-37078-7_12 · Cited by 105
Pan, Y., Liu, M.-X., Lian, C., Xia, Y., & Shen, D. (2020) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2020.2983085 · Cited by 87
Addformer: Alzheimer’s Disease Detection from Structural Mri Using Fusion Transformer
Kushol, R., Masoumzadeh, A., Dong, H., Kalra, S., & Yang, Y. (2022) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi52829.2022.9761421 · Cited by 75
Deep Residual Nets for Improved Alzheimer's Diagnosis
Valliani, A. A.-A., & Soni, A. B. (2017) · ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics (ACM-BCB) · doi:10.1145/3107411.3108224 · Cited by 74
MRI-based Alzheimer’s disease prediction via distilling the knowledge in multi-modal data
Guan, H., Wang, C., & Tao, D. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2021.118586 · Cited by 58
Trans-ResNet: Integrating Transformers and CNNs for Alzheimer’s disease classification
Li, C., Cui, Y. J., Luo, N., Liu, Y., Bourgeat, P., Fripp, J., & Jiang, T. (2022) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi52829.2022.9761549 · Cited by 55
Servick, K. (2019) · Science · doi:10.1126/science.aax4236 · Cited by 32
Ji, Y., & Gao, S. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 10
MRI Field Strength Predicts Alzheimer’s Disease: a Case Example of Bias in the ADNI Data Set
Thibeau–Sutre, E., Couvy‐Duchesne, B., Dormont, D., Colliot, O., & Burgos, N. (2022) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi52829.2022.9761504 · Cited by 9
Research works printed but not linked: 1
- Radford, 2019, Language models are unsupervised multitask learners, OpenAI blog, 1, 9
Cited by
Sorted by citations
Li, Y., El Habib Daho, M., Conze, P.-H., Zeghlache, R., Le Boité, H., Tadayoni, R., Cochener, B., Lamard, M., & Quellec, G. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108635 · Cited by 274
Schouten, D., Nicoletti, G., Dille, B., Chia, C., Vendittelli, P., Schuurmans, M., Litjens, G., & Khalili, N. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103621 · Cited by 162
Simon, B. D., Ozyoruk, K. B., Gelikman, D. G., Harmon, S. A., & Türkbey, B. (2024) · Diagnostic and Interventional Radiology · doi:10.4274/dir.2024.242631 · Cited by 73
Multi-scale multimodal deep learning framework for Alzheimer's disease diagnosis
Abdel-Aziz, M. H., Wang, T., Anwaar, W., & Elazab, A. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.109438 · Cited by 71
Comprehensive review of Transformer‐based models in neuroscience, neurology, and psychiatry
Cong, S., Wang, H., Zhou, Y., Wang, Z., Yao, X., & Yang, C. (2024) · Brain‐X · doi:10.1002/brx2.57 · Cited by 75
Metadata
- Authors
- Liu, L., Liu, S., Zhang, L., To, X. V., Nasrallah, F., & Chandra, S. S.
- Affiliations
- The University of Queensland (AU)
- Corresponding author
- Liu, Linfeng
- Venue
- NeuroImage
- Publisher
- Elsevier BV
- Published
- 2023-08
- Volume
- 277
- Pages
- 120267
- ISSN
- 1053-8119 (ISSN-L)
- OpenAlex
- W4383315874
- Cited by
- 101
- References
- 58
- Citations by year
- 2026: 23 · 2025: 38 · 2024: 32 · 2023: 7
- FWCI
- 20.27
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- NeuroImage (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- U.S. Department of Defense (U01 AG024904, W81XWH, W81XWH-12-2-, W81XWH-12, AG024904)
- Alzheimer's Association (U01 AG024904, W81XWH-12-2-0012)
- Bristol-Myers Squibb (U01 AG024904, AG024904, W81XWH-12-2-0012)
- Eli Lilly and Company (U01 AG024904, AG024904, W81XWH-12-2-0012)
- Pfizer (AG024904, W81XWH-12-2-0012, U01 AG024904)
- Biogen (W81XWH-12-2-0012, U01 AG024904, AG024904)
- University of Southern California (W81XWH-12-2-0012, U01 AG024904)
- F. Hoffmann-La Roche (W81XWH-12-2-0012, AG024904, U01 AG024904)
- Meso Scale Diagnostics (U01 AG024904, AG024904, W81XWH-12-2-0012)
- Alzheimer's Disease Neuroimaging Initiative (W81XWH-12-2–0012, U01AG024904, AG024904, W81XWH-12-2-)
- BioClinica (U01 AG024904, W81XWH-12-2-0012, AG024904)
- Novartis Pharmaceuticals Corporation (W81XWH-12-2-0012, U01 AG024904, AG024904)
- Northern California Institute for Research and Education (U01 AG024904, W81XWH-12-2-0012)
- Queensland Government (2014000857)
- Eisai (W81XWH-12-2-0012, U01 AG024904, AG024904)
- Servier (AG024904, W81XWH-12-2-0012, U01 AG024904)
- H. Lundbeck A/S (U01 AG024904, AG024904, W81XWH-12-2-0012)
- Motor Accident Insurance Commission (2014000857, AG024904)
- National Institutes of Health (5u01ag024904-10, W81XWH, W81XWH-12-2-0012)
- Genentech (W81XWH-12-2-0012, AG024904, U01 AG024904)
- Canadian Institutes of Health Research (U01 AG024904, W81XWH-12-2-0012)
- National Institute on Aging (U01-AG-024904)
- Topics
- Health Sciences › Medicine › Psychiatry and Mental health › Dementia and Cognitive Impairment Research
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- Alzheimer's disease classification, multimodal fusion, missing data, modality dropout, cross-attention, neuroimaging data, clinical data, ADNI, AIBL
- MeSH terms
- Alzheimer Disease, Australia, Humans, Magnetic Resonance Imaging, Biomarkers, Neuroimaging, Cognitive Dysfunction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
