The DOI matches its Crossref record; 37 of the 38 references it cites were found in a registry or an index.
Multimodal fusion with deep neural networks for leveraging CT imaging and electronic health record: a case-study in pulmonary embolism detection
Open accessAbstract
Recent advancements in deep learning have led to a resurgence of medical imaging and Electronic Medical Record (EMR) models for a variety of applications, including clinical decision support, automated workflow triage, clinical prediction and more. However, very few models have been developed to integrate both clinical and imaging data, despite that in routine practice clinicians rely on EMR to provide context in medical imaging interpretation. In this study, we developed and compared different multimodal fusion model architectures that are capable of utilizing both pixel data from volumetric Computed Tomography Pulmonary Angiography scans and clinical patient data from the EMR to automatically classify Pulmonary Embolism (PE) cases. The best performing multimodality model is a late fusion model that achieves an AUROC of 0.947 [95% CI: 0.946–0.948] on the entire held-out test set, outperforming imaging-only and EMR-only single modality models.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Huang et al. [83] found that averaging-based late fusion with regularized DNNs as sub-models outperforms early, intermediate and other late fusion strategies on the prediction of pulmonary embolism detection utilizing CT scans and EHR data.”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [83]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Huang et al. created seven different fusion architectures. These included, early, joint, and late fusion. The architecture that performed the best was the late elastic average fusion for the diagnosis of pulmonary embolism using computed tomography and EHR data159.”
From Multimodal machine learning in precision health: A scoping review (2022) · cites this paper as [159]
The cited source says what the claim attributes to it.
In the citing paper
“Imaging and EHR-based data have been fused to improve detection of pulmonary embolism, outperforming single-modality models 99”
From Multimodal biomedical AI (2022) · cites this paper as [99]
The cited source says what the claim attributes to it.
In the citing paper
“Huang et al. [83] applied an FCNN for fusing sub-model predictions”
From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [83]
The cited source says what the claim attributes to it.
In the citing paper
“recent work has demonstrated that the combination of imaging and EHR data outperforms each of these modalities alone to identify pulmonary embolism 99”
From Multimodal biomedical AI (2022) · cites this paper as [99]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Focal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.324 · Cited by 30,581
Regularization and Variable Selection Via the Elastic Net
Zou, H., & Hastie, T. (2005) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2005.00503.x · Cited by 20,957
CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning
Rajpurkar, P., Irvin, J., Zhu, K., Yang, B., Mehta, H., Duan, T., Ding, D. Y., Bagul, A., Langlotz, C. P., Shpanskaya, K. S., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.05225 · Cited by 1,219
Self-citationA Deep Learning Mammography-based Model for Improved Breast Cancer Risk Prediction
Yala, A., Lehman, C., Schuster, T., Portnoi, T., & Barzilay, R. (2019) · Radiology · doi:10.1148/radiol.2019182716 · Cited by 758
Bien, N., Rajpurkar, P., Ball, R. L., Irvin, J., Park, A., Jones, E., Bereket, M., Patel, B. N., Yeom, K. W., Shpanskaya, K., et al. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002699 · Cited by 734
Self-citationHorlander, K. T., Mannino, D. M., & Leeper, K. V. (2003) · Archives of Internal Medicine · doi:10.1001/archinte.163.14.1711 · Cited by 657
Seven-point checklist and skin lesion classification using multitask multimodal neural nets
Kawahara, J., Daneshvar, S., Argenziano, G., & Hamarneh, G. (2019) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2018.2824327 · Cited by 584
CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison
Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.3301590 · Cited by 392
Self-citationCheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison
Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al. (2019) · arXiv · doi:10.48550/arxiv.1901.07031 · Cited by 392
Self-citationMultimodal skin lesion classification using deep learning
Yap, J., Yolland, W., & Tschandl, P. (2018) · Experimental Dermatology · doi:10.1111/exd.13777 · Cited by 326
Research works printed but not linked: 1
- Rajan, D., Beymer, D., Abedin, S. & Dehghan, E. Pi-PE: A pipeline for pulmonary embolism detection using sparsely annotated 3D CT images. Mach. Learn. Health NeurIPS Workshop 116, 220–232 (2020).
Cited by
Acosta, J. N., Falcone, G. J., Rajpurkar, P., & Topol, E. J. (2022) · Nature Medicine · doi:10.1038/s41591-022-01981-2 · Cited by 1,365
Artificial intelligence for multimodal data integration in oncology
Lipkova, J., Chen, R. J., Chen, B., Lu, M. Y., Barbieri, M., Shao, D., Vaidya, A. J., Chen, C., Zhuang, L., Williamson, D. F. K., et al. (2022) · Cancer Cell · doi:10.1016/j.ccell.2022.09.012 · Cited by 777
Multimodal deep learning for biomedical data fusion: a review
Stahlschmidt, S. R., Ulfenborg, B., & Synnergren, J. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbab569 · Cited by 766
“[83] applied an FCNN for fusing sub-model predictions, while Reda et al.”
BackgroundMethodMultimodal machine learning in precision health: A scoping review
Kline, A., Wang, H., Li, Y., Dennis, S., Hutch, M., Xu, Z., Wang, F., Cheng, F., & Luo, Y. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00712-8 · Cited by 528
The current and future state of AI interpretation of medical images
Rajpurkar, P., & Lungren, M. P. (2023) · New England Journal of Medicine · doi:10.1056/nejmra2301725 · Cited by 443
Self-citation
Metadata
- Authors
- Huang, S.-C., Pareek, A., Zamanian, R., Banerjee, I., & Lungren, M. P.
- Affiliations
- Stanford University (US)
- Stanford Medicine (US)
- Emory University (US)
- Corresponding author
- Huang, Shih-Cheng
- Venue
- Scientific Reports
- Published
- 2020-12-17
- Volume
- 10
- Issue
- 1
- Pages
- 22147-22147
- ISSN
- 2045-2322 (ISSN-L)
- PMC
- PMC7746687
- OpenAlex
- W3112990870
- Cited by
- 246
- References
- 38
- Citations by year
- 2026: 38 · 2025: 56 · 2024: 62 · 2023: 49 · 2022: 31 · 2021: 9
- FWCI
- 8.75
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Scientific Reports (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- U.S. National Library of Medicine (R01LM012966)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- multimodal fusion, pulmonary embolism detection, CT pulmonary angiography, electronic health records, decision fusion, automated diagnosis, deep neural network
- MeSH terms
- Clinical Decision-Making, Machine Learning, Humans, Image Interpretation, Computer-Assisted, Image Processing, Computer-Assisted, Pulmonary Embolism, Sensitivity and Specificity, Tomography, X-Ray Computed, Neural Networks, Computer, Disease Management, Workflow, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Multimodal fusion with deep neural networks for leveraging CT imaging and electronic health record: a case-study in pulmonary embolism detection. https://app.cochat.ai/papers/10.1038/s41598-020-78888-w
- The paper
- Huang, S.-C., Pareek, A., Zamanian, R., Banerjee, I., & Lungren, M. P. (2020). Multimodal fusion with deep neural networks for leveraging CT imaging and electronic health record: a case-study in pulmonary embolism detection. Scientific Reports. https://doi.org/10.1038/s41598-020-78888-w
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.