Multimodal Deep Learning for Integrating Chest Radiographs and Clinical Parameters: A Case for Transformers
Abstract
See also the editorial by Kitamura and Topol in this issue.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Attention mechanisms such as gated attention, cross-attention, or transformer encoders are increasingly adopted in multimodal AI.
“attention mechanisms such as gated attention [40,41], cross-attention [42–45], or transformer encoders [20,36,46–49] are increasingly adopted.”
From Multimodal Artificial Intelligence for Precision Critical Care: A Scoping Review (2026) · cites this paper as [20]
Prior studies have shown that multimodal models outperform single-modality models in predictive performance.
“All of these studies have shown that multimodal models outperform single-modality models in predictive performance.”
From Multimodal Artificial Intelligence for Precision Critical Care: A Scoping Review (2026) · cites this paper as [20]
Only 6 studies addressed the issue of missing modalities.
“In this review, only 6 studies addressed the issue of missing modalities [20,48,69,110–112].”
From Multimodal Artificial Intelligence for Precision Critical Care: A Scoping Review (2026) · cites this paper as [20]
Several multimodal AI systems have been introduced, including models that analyze radiology images together with clinical data or integrate histopathology with genomic or text-based information.
“Recently, several such AI systems have been introduced6. Examples include models that analyze radiology images together with clinical data7 or integrate information from histopathology with genomic8 or text-based information9.”
From Development and validation of an autonomous artificial intelligence agent for clinical decision-making in oncology (2025) · cites this paper as [7]
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988) · Biometrics · doi:10.2307/2531595 · Cited by 22,649 · see citing works
Goldberger, A. L., Amaral, L. A. N., Glass, L., Hausdorff, J. M., Ivanov, P. C., Mark, R. G., Mietus, J. E., Moody, G. B., Peng, C.-K., & Stanley, H. E. (2000) · Circulation · doi:10.1161/01.cir.101.23.e215 · Cited by 14,809 · see citing works
Proceedings of the 24th international conference on Machine learning
(2007) · doi:10.1145/1273496 · Cited by 11,869 · see citing works
Scientists rise up against statistical significance
Amrhein, V., Greenland, S., & McShane, B. (2019) · Nature · doi:10.1038/d41586-019-00857-9 · Cited by 2,983 · see citing works
Clinically applicable deep learning for diagnosis and referral in retinal disease
De Fauw, J., Ledsam, J. R., Romera-Paredes, B., Nikolov, S., Tomasev, N., Blackwell, S., Askham, H., Glorot, X., O’Donoghue, B., Visentin, D., et al. (2018) · Nature Medicine · doi:10.1038/s41591-018-0107-6 · Cited by 2,639 · see citing works
Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images
Hatamizadeh, A., Nath, V., Tang, Y., Yang, D., Roth, H. R., & Xu, D. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-08999-2_22 · Cited by 2,273 · see citing works
Advances in Neural Information Processing Systems 19 (NIPS 2006)
Simonotto, E., Whalley, H. C., Lawrie, S. M., Murray, L. M., McGonigle, D. J., & Storkey, A. (2006) · Cited by 1,651 · see citing works
Acosta, J. N., Falcone, G. J., Rajpurkar, P., & Topol, E. J. (2022) · Nature Medicine · doi:10.1038/s41591-022-01981-2 · Cited by 1,365 · see citing works
Huang, S.-C., Pareek, A., Seyyedi, S., Banerjee, I., & Lungren, M. P. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00341-z · Cited by 934 · see citing works
A deep learning system for differential diagnosis of skin diseases
Liu, Y., Jain, A., Eng, C., Way, D. H., Lee, K., Bui, P., Kanada, K., de Oliveira Marinho, G., Gallegos, J., Gabriele, S., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-0842-3 · Cited by 853 · see citing works
Artificial intelligence for multimodal data integration in oncology
Lipkova, J., Chen, R. J., Chen, B., Lu, M. Y., Barbieri, M., Shao, D., Vaidya, A. J., Chen, C., Zhuang, L., Williamson, D. F. K., et al. (2022) · Cancer Cell · doi:10.1016/j.ccell.2022.09.012 · Cited by 777 · see citing works
Artificial intelligence in histopathology: enhancing cancer research and clinical oncology
Shmatko, A., Ghaffari Laleh, N., Gerstung, M., & Kather, J. N. (2022) · Nature Cancer · doi:10.1038/s43018-022-00436-4 · Cited by 446 · see citing works
Self-citationBoehm, K. M., Aherne, E. A., Ellenson, L., Nikolovski, I., Alghamdi, M., Vázquez-García, I., Zamarin, D., Long Roche, K., Liu, Y., Patel, D., et al. (2022) · Nature Cancer · doi:10.1038/s43018-022-00388-9 · Cited by 357 · see citing works
Truhn, D., Schrading, S., Haarburger, C., Schneider, H., Merhof, D., & Kuhl, C. (2019) · Radiology · doi:10.1148/radiol.2018181352 · Cited by 273 · see citing works
Self-citationSelf-evolving vision transformer for chest X-ray diagnosis through knowledge distillation
Park, S. J., Kim, G., Oh, Y., Seo, J. B., Lee, S. M., Kim, J. H., Moon, S., Lim, J., Park, C. M., & Ye, J. C. (2022) · Nature Communications · doi:10.1038/s41467-022-31514-x · Cited by 82 · see citing works
Schrammen, P. L., Laleh, N. G., Echle, A., Truhn, D., Schulz, V., Brinker, T. J., Brenner, H., Chang-Claude, J. C., Alwers, E., Brobeil, A., et al. (2021) · The Journal of Pathology · doi:10.1002/path.5800 · Cited by 81 · see citing works
Self-citationAssessment of Radiology Artificial Intelligence Software: A Validation and Evaluation Framework
Tanguay, W., Acar, P., Fine, B., Abdolell, M., Gong, B., Cadrin-Chênevert, A., Chartrand‐Lefebvre, C., Chalaoui, J., Gorgos, A. B., Chin, A. S.-L., et al. (2022) · Canadian Association of Radiologists Journal · doi:10.1177/08465371221135760 · Cited by 59 · see citing works
Artificial Intelligence for Clinical Interpretation of Bedside Chest Radiographs
Khader, F., Han, T., Müller‐Franzes, G. A., Huck, L. C., Schad, P., Keil, S., Barzakova, E., Schulze-Hagen, M. F., Pedersoli, F., Schulz, V., et al. (2022) · Radiology · doi:10.1148/radiol.220510 · Cited by 33 · see citing works
Self-citationImage prediction of disease progression for osteoarthritis by style-based manifold extrapolation
Han, T., Kather, J. N., Pedersoli, F., Zimmermann, M. N., Keil, S., Schulze-Hagen, M. F., Terwoelbeck, M., Isfort, P., Haarburger, C., Kiessling, F. M., et al. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00560-x · Cited by 29 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Ferber, D., El Nahhas, O. S. M., Wölflein, G., Wiest, I. C., Clusmann, J., Leßmann, M.-E., Foersch, S., Lammert, J., Tschochohei, M., Jäger, D., et al. (2025) · Nature Cancer · doi:10.1038/s43018-025-00991-6 · Cited by 175 · see citing works
Self-citationMultimodal Deep Learning Approaches for Precision Oncology: A Comprehensive Review
Yang, H., Yang, M., Chen, J., Yao, G., Zou, Q., & Jia, L. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae699 · Cited by 115 · see citing works
Schouten, D., Nicoletti, G., Dille, B., Chia, C., Vendittelli, P., Schuurmans, M., Litjens, G., & Khalili, N. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103621 · Cited by 108 · see citing works
Truhn, D., Weber, C. D., Braun, B. J., Bressem, K., Kather, J. N., Kuhl, C., & Nebelung, S. (2023) · Scientific Reports · doi:10.1038/s41598-023-47500-2 · Cited by 71 · see citing works
Self-citationSimon, B. D., Ozyoruk, K. B., Gelikman, D. G., Harmon, S. A., & Türkbey, B. (2024) · Diagnostic and Interventional Radiology · doi:10.4274/dir.2024.242631 · Cited by 73 · see citing works
Multimodal Artificial Intelligence for Precision Critical Care: A Scoping Review
Yang, M., Shi, N., Chen, H., Po, E. Y. P., Ng, P. Y., Lai, P. C. K., Wu, Y.-C., & Fong, D. Y. T. (2026) · Health Data Science · doi:10.34133/hds.0356 · Cited by 3 · see citing works
“In this review, only 6 studies addressed the issue of missing modalities [ 20 , 48 , 69 , 110 – 112 ].”
Influential
- Authors
- Khader, F., Müller-Franzes, G., Wang, T., Han, T., Tayebi Arasteh, S., Haarburger, C., Stegmaier, J., Bressem, K., Kuhl, C., Nebelung, S., Kather, J. N., & Truhn, D.
- Affiliations
- University of Leeds (GB)
- Corresponding author
- Kather, Jakob Nikolas
- Truhn, Daniel
- ORCID
- Venue
- Radiology
- Published
- 2023-10-01
- Volume
- 309
- Issue
- 1
- Pages
- e230806-e230806
- ISSN
- 0033-8419 (ISSN-L)
- 1527-1315
- OpenAlex
- W4387296225
- Semantic Scholar
- 6355482485057273f6eeda8aedb9bb3501557205
- Cited by
- 107
- References
- 20
- Citations by year
- 2026: 46 · 2025: 35 · 2024: 23 · 2023: 3
- FWCI
- 9.71
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Radiology (publishedVersion)
- PubMed (publishedVersion)
- RWTH Publications (RWTH Aachen) (publishedVersion)
- Funders
- German Federal Ministry of Health (ZMVI1-2520DAT111)
- German Cancer Aid (70113864)
- Deutsche Forschungsgemeinschaft (NE 2136/3-1)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- multimodal deep learning, chest radiograph, clinical features, transformer models, multimodal fusion, chest imaging
- MeSH terms
- Deep Learning, Aged, Humans, Inpatients, Male, Middle Aged, Radiography, Retrospective Studies, Databases, Factual
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
