Deep Adversarial Learning for Multi-Modality Missing Data Completion
Abstract
Multi-modality data are widely used in clinical applications, such as tumor detection and brain disease diagnosis. Different modalities can usually provide complementary information, which commonly leads to improved performance. However, some modalities are commonly missing for some subjects due to various technical and practical reasons. As a result, multi-modality data are usually incomplete, raising the multi-modality missing data completion problem. In this work, we formulate the problem as a conditional image generation task and propose an encoder-decoder deep neural network to tackle this problem. Specifically, the model takes the existing modality as input and generates the missing modality. By employing an auxiliary adversarial loss, our model is able to generate high-quality missing modality images. At the same time, we propose to incorporate the available category information of subjects in training to enable the model to generate more informative images. We evaluate our method on the Alzheimer's Disease Neuroimaging Initiative~(ADNI) database, where positron emission tomography~(PET) modalities are missing. Experimental results show that the trained network can generate high-quality PET modalities based on existing magnetic resonance imaging~(MRI) modalities, and provide complementary information to improve the detection and tracking of the Alzheimer's disease. Our results also show that the proposed methods generate higher quality images than baseline methods as m…
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Existing multimodal fusion models trained on full-modality samples usually fail when partial modalities are missing.
“Existing multimodal fusion models trained on full-modality samples usually fail when partial modalities are missing (Aguilar et al., 2019; Pham et al., 2019; Cai et al., 2018; Parthasarathy and Sundaram, 2020).”
From Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities (2021) · cites this paper as Cai et al., 2018
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404 · see citing works
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892 · see citing works
Self-citationUnpaired image-to-image translation using cycle-consistent adversarial networks
Zhu, J.-Y., Park, T., Isola, P., & Efros, A. A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.244 · Cited by 23,602 · see citing works
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation
Badrinarayanan, V., Kendall, A., & Cipolla, R. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2644615 · Cited by 19,631 · see citing works
Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network
Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., et al. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.19 · Cited by 12,696 · see citing works
Self-citationImage Super-Resolution Using Deep Convolutional Networks
Dong, C., Loy, C. C., He, K., & Tang, X. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2439281 · Cited by 10,105 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities
Zhao, J., Li, R., & Jin, Q. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.203 · Cited by 225 · see citing works
“Cai et al. (Cai et al., 2018) propose an encoder-decoder deep neural network to generate the missing modality (Positron Emission To-mography, PET) given the available modality (Magnetic Resonance Imaging, MRI), and the generated PET can provide complementary information to improve the detection and tracking of Alzheim…”
BackgroundMethodInfluentialDeep learning for Alzheimer's disease diagnosis: A survey
Khojaste-Sarakhsi, M., Haghighi, S. S., Ghomi, S. M. T. F., & Marchiori, E. (2022) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2022.102332 · Cited by 194 · see citing works
Incomplete multi-modal representation learning for Alzheimer’s disease diagnosis
Liu, Y., Fan, L., Zhang, C., Zhou, T., Xiao, Z., Geng, L., & Shen, D. (2021) · Medical Image Analysis · doi:10.1016/j.media.2020.101953 · Cited by 102 · see citing works
Self-citationLearning Representations from Imperfect Time Series Data via Tensor Rank Regularization
Liang, P. P., Liu, Z., Tsai, Y.-H. H., Zhao, Q., Salakhutdinov, R., & Morency, L.-P. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1152 · Cited by 95 · see citing works
Liu, R., Zuo, H., Lian, Z., Schuller, B. W., & Li, H. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3378570 · Cited by 96 · see citing works
Tag-assisted Multimodal Sentiment Analysis under Uncertain Missing Modalities
Zeng, J., Liu, T., & Zhou, J. (2022) · Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3477495.3532064 · Cited by 94 · see citing works
Multimodal Prompt Learning with Missing Modalities for Sentiment Analysis and Emotion Recognition
Guo, Z., Jin, T., & Zhao, Z. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.94 · Cited by 77 · see citing works
Yuan, H., Yu, K., Xie, F., Liu, M., & Sun, S. (2024) · Medicine Advances · doi:10.1002/med4.75 · Cited by 61 · see citing works
Rajendran, S., Pan, W., Sabuncu, M. R., Chen, Y., Zhou, J., & Wang, F. (2024) · Patterns · doi:10.1016/j.patter.2023.100913 · Cited by 47 · see citing works
Metric Learning on Healthcare Data with Incomplete Modalities
Suo, Q., Zhong, W., Ma, F., Yuan, Y., Gao, J., & Zhang, A. (2019) · Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/490 · Cited by 46 · see citing works
Unbiased Missing-Modality Multimodal Learning
Dai, R., Li, C., Yan, Y., Mo, L., Qin, K., & He, T. (2025) · 2025 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.02272 · Cited by 11 · see citing works
- Authors
- Cai, L., Wang, Z., Gao, H., Shen, D., & Ji, S.
- Affiliations
- Washington State University (US)
- University of North Carolina at Chapel Hill (US)
- Venue
- Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining
- Publisher
- ACM
- Published
- 2018-07-19
- Pages
- 1158-1166
- OpenAlex
- W2809410879
- Cited by
- 180
- References
- 8
- Citations by year
- 2026: 17 · 2025: 31 · 2024: 38 · 2023: 33 · 2022: 24 · 2021: 18 · 2020: 10 · 2019: 9
- FWCI
- 2.75
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) (publishedVersion)
- Funders
- Defense Advanced Research Projects Agency (N66001-17-2-4031)
- National Science Foundation (DBI-1641223, IIS-1615035, IIS-1633359)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Medical Imaging Techniques and Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- conditional image generation, adversarial loss, encoder-decoder neural network, Alzheimer's disease, ADNI, multimodal medical imaging
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
