Reducing annotation burden through multimodal learning
Open accessAbstract
Choosing an optimal data fusion technique is essential when performing machine learning with multimodal data. In this study, we examined deep learning-based multimodal fusion techniques for the combined classification of radiological images and associated text reports. In our analysis, we (1) compared the classification performance of three prototypical multimodal fusion techniques: Early, Late, and Model fusion, (2) assessed the performance of multimodal compared to unimodal learning; and finally (3) investigated the amount of labeled data needed by multimodal versus unimodal models to yield comparable classification performance. Our experiments demonstrate the potential of multimodal fusion methods to yield competitive results using less training data (labeled data) than their unimodal counterparts. This was more pronounced using the Early and less so using the Model and Late fusion approaches. With increasing amount of training data, unimodal models achieved comparable results to multimodal models. Overall, our results suggest the potential of multimodal learning to decrease the need for labeled training data resulting in a lower annotation burden for domain experts.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · see citing works
(2014) · Cited by 33,419 · see citing works
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · see citing works
Convolutional Neural Networks for Sentence Classification
Kim, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1181 · Cited by 14,831 · see citing works
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018 · see citing works
Multimodal fusion for multimedia analysis: a survey
Atrey, P. K., Hossain, M. A., El Saddik, A., & Kankanhalli, M. S. (2010) · Multimedia Systems · doi:10.1007/s00530-010-0182-0 · Cited by 1,279 · see citing works
Preparing a collection of radiology examinations for distribution and retrieval
Demner-Fushman, D., Kohli, M. D., Rosenman, M. B., Shooshan, S. E., Rodriguez, L., Antani, S., Thoma, G. R., & McDonald, C. J. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv080 · Cited by 1,164 · see citing works
PadChest: A large chest x-ray image dataset with multi-label annotated reports
Bustos, A., Pertusa, A., Salinas, J.-M., & de la Iglesia-Vayá, M. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101797 · Cited by 691 · see citing works
Structured crowdsourcing enables convolutional segmentation of histology images
Amgad, M., Elfandy, H., Hussein, H., Atteya, L. A., Elsebaie, M. A. T., Abo Elnasr, L. S., Sakr, R. A., Salem, H. S. E., Ismail, A. F., Saad, A. M., et al. (2019) · Bioinformatics · doi:10.1093/bioinformatics/btz083 · Cited by 315 · see citing works
Lin, M., Chen, Q., & Yan, S. (2013) · arXiv · doi:10.48550/arxiv.1312.4400 · Cited by 252 · see citing works
Learn to combine modalities in multimodal deep learning
Liu, K., Li, Y., Xu, N., & Natarajan, P. (2018) · arXiv · doi:10.48550/arxiv.1805.11730 · Cited by 136 · see citing works
Egleston, B. L., Miller, S. M., & Meropol, N. J. (2011) · Statistics in Medicine · doi:10.1002/sim.4377 · Cited by 133 · see citing works
Word Emdeddings through Hellinger PCAPreprint
Lebret, R., & Collobert, R. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.5542 · Cited by 65 · see citing works
Learning and Fusing Multimodal Deep Features for Acoustic Scene Categorization
Yin, Y., Shah, R. R., & Zimmermann, R. (2018) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3240508.3240631 · Cited by 37 · see citing works
A Deep Learning Approach for Multimodal Deception DetectionPreprint
Krishnamurthy, G., Majumder, N., Poria, S., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.00344 · Cited by 35 · see citing works
Complementary ensemble clustering of biomedical data
Fodeh, S. J., Brandt, C., Luong, T. B., Haddad, A., Schultz, M. H., Murphy, T. E., & Krauthammer, M. O. (2013) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2013.02.001 · Cited by 30 · see citing works
Self-citationDeep Multimodal Classification of Image Types in Biomedical Journal Figures
Andrearczyk, V., & Müller, H. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-98932-7_1 · Cited by 15 · see citing works
Lateral Chest X-Ray for Physicians
Gaber, K. A., McGavin, C. R., & Wells, I. P. (2005) · Journal of the Royal Society of Medicine · doi:10.1177/014107680509800705 · Cited by 11 · see citing works
Do Lateral Views Help Automated Chest X-ray Predictions?Preprint
Bertrand, H., Hashir, M., & Cohen, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.08534 · Cited by 10 · see citing works
Lateral chest X-ray for physicians
Gaber, K. A. (2005) · Journal of the Royal Society of Medicine · doi:10.1258/jrsm.98.7.310 · Cited by 8 · see citing works
Kluthke, R., Kickuth, R., Bansmann, P. M., Tüshaus, C., Adams, S., Liermann, D., & Kirchner, J. (2016) · British Journal of Radiology · doi:10.1259/bjr.20160394 · Cited by 8 · see citing works
Enhancing Clustering by Exploiting Complementary Data Modalities in the Medical Domain
Fodeh, S. J., Haddad, A., Brandt, C., Schultz, M. H., & Krauthammer, M. O. (2012) · IFIP International Federation for Information Processing/IFIP · doi:10.1007/978-3-642-33409-2_37 · Cited by 3 · see citing works
Self-citationLocalization with Limited Annotation for Chest X-raysPreprint
Rozenberg, E., Freedman, D. Z., & Bronstein, A. M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.08842 · Cited by 3 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- Huang G. Liu Z. van der Maaten L. Weinberger K. Q. (2018). Densely connected convolutional networks. arXiv [Preprint]. arXiv:1608.06993. Available online at: http://arxiv.org/abs/1608.06993 (accessed May 11, 2020). · arXiv
- Vaswani A. Shazeer N. Parmar N. Uszkoreit J. Jones L. Gomez A. N.. (2017). “Attention is all you need,” in Advances in Neural Information Processing Systems 30, eds I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Curran Associates, Inc.), 5998– 6008. Available online at: http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf (accessed May 11, 2020).
Cited by
Sorted by citations
Harnessing multimodal data integration to advance precision oncology
Boehm, K. M., Khosravi, P., Vanguri, R., Gao, J., & Shah, S. P. (2022) · Nature Reviews Cancer · doi:10.1038/s41568-021-00408-3 · Cited by 662 · see citing works
Aung, Y. Y. M., Wong, D. C. S., & Ting, D. S. W. (2021) · British Medical Bulletin · doi:10.1093/bmb/ldab016 · Cited by 511 · see citing works
Benefits and Risks of AI in Health Care: Narrative Review
Chustecki, M. (2024) · Interactive Journal of Medical Research · doi:10.2196/53616 · Cited by 208 · see citing works
“The continuous availability of data for ongoing improvement of ML-based systems can be difficult due to organizational resistance [57].”
Multi-modality machine learning predicting Parkinson’s disease
Makarious, M. B., Leonard, H. L., Vitale, D., Iwaki, H., Sargent, L. J., Dadu, A., Violich, I., Hutchins, E. D., Saffo, D., Bandrés‐Ciga, S., et al. (2022) · npj Parkinson s Disease · doi:10.1038/s41531-022-00288-w · Cited by 188 · see citing works
Khosravi, P., Lysandrou, M., Eljalby, M., Li, Q., Kazemi, E. M., Zisimopoulos, P., Sigaras, A., Brendel, M., Barnes, J., Ricketts, C., et al. (2021) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.27599 · Cited by 121 · see citing works
Schouten, D., Nicoletti, G., Dille, B., Chia, C., Vendittelli, P., Schuurmans, M., Litjens, G., & Khalili, N. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103621 · Cited by 108 · see citing works
Chen, M. (2024) · American Journal of Translational Research · doi:10.62347/myhe3488 · Cited by 48 · see citing works
Federated learning for multi-omics: A performance evaluation in Parkinson’s disease
Danek, B. P., Makarious, M. B., Dadu, A., Vitale, D., Lee, P. S., Singleton, A., Nalls, M. A., Sun, J., & Faghri, F. (2024) · Patterns · doi:10.1016/j.patter.2024.100945 · Cited by 34 · see citing works
Shetty, S., Ananthanarayana, V. S., & Mahale, A. R. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-14940-x · Cited by 30 · see citing works
Comprehensive Review of Multimodal Medical Data Analysis: Open Issues and Future Research Directions
Shetty, S., Ananthanarayana, V. S., & Mahale, A. R. (2022) · Acta Informatica Pragensia · doi:10.18267/j.aip.202 · Cited by 23 · see citing works
Pan, C., Dong, X., Lança, C., Wang, Y., Saw, S., He, X., Hu, D., Fan, Q., Grzybowski, A. E., & Ohno‐Matsui, K. (2025) · Progress in Retinal and Eye Research · doi:10.1016/j.preteyeres.2025.101415 · Cited by 18 · see citing works
Shetty, S., Ananthanarayana, V. S., & Mahale, A. R. (2022) · Mathematics · doi:10.3390/math10193646 · Cited by 10 · see citing works
Revolutionizing Drug Discovery: The Role of Artificial Intelligence
Shah, B. (2023) · International Journal of Science and Research (IJSR) · doi:10.21275/sr231219092956 · Cited by 3 · see citing works
Improving the Efficiency of Multimodal Approach for Chest X-Ray
Upadhya, J., Vargas, J., Poudel, K. N., & Ranganathan, J. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-56950-0_5 · Cited by 3 · see citing works
Artificial intelligence in diagnostics and medical decision making
Levi, M. M. (2026) · European Journal of Internal Medicine · doi:10.1016/j.ejim.2026.107056 · Cited by 2 · see citing works
Early Gastric Cancer Screening Framework Based on Multimodal Fusion
Lu, M., & Chai, Y. (2023) · Lecture notes in electrical engineering · doi:10.1007/978-981-99-6882-4_5
Yadav, D. K., Rathee, S., & Patil, U. K. (2024) · Current Cancer Therapy Reviews · doi:10.2174/0115733947301672240528061642 · Cited by 1 · see citing works
Evaluating multiple large language models on orbital diseases
Yang, Q., Zeng, Y.-M., Wei, H., Chen, C., Ling, Q., Wang, X., Chen, X., & Shao, Y. (2025) · Frontiers in Cell and Developmental Biology · doi:10.3389/fcell.2025.1574378
Zhan, G. (2026) · American Journal of Clinical and Experimental Medicine · doi:10.11648/j.ajcem.20261401.11
Weakly Supervised Learning for Facial Affective Behavior Analysis: A Review
Praveen, R. G., Cardinal, P., & Granger, É. (2026) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2026.3677128 · Cited by 4 · see citing works
Kumar, R., Sharma, N., Deshmukh, A., Nair, A., & Pillai, M. (2022) · Journal of Artificial Intelligence for Healthcare Systems · doi:10.68159/z613962324
- Authors
- Lopez, K., Fodeh, S. J., Allam, A., Brandt, C. A., & Krauthammer, M.
- Affiliations
- Yale University (US)
- University of Zurich (CH)
- Corresponding author
- Lopez, Kevin
- Venue
- Frontiers in Big Data
- Publisher
- Frontiers Media SA
- Published
- 2020-06-02
- Volume
- 3
- Pages
- 19-19
- ISSN
- 2624-909X (ISSN-L)
- PMC
- PMC7931886
- OpenAlex
- W3033171300
- Cited by
- 21
- References
- 19
- Citations by year
- 2026: 3 · 2025: 3 · 2024: 5 · 2023: 3 · 2022: 4 · 2021: 3
- FWCI
- 0.97
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Frontiers in Big Data (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- U.S. National Library of Medicine (T15LM007056)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Physical Sciences › Engineering › Media Technology › Advanced Image Fusion Techniques
- Keywords
- multimodal fusion, radiology reports, training data efficiency, feature-level fusion, decision fusion, model fusion, deep learning, medical image classification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
