Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 37 of the 38 references it cites were found in a registry or an index.

Multimodal fusion with deep neural networks for leveraging CT imaging and electronic health record: a case-study in pulmonary embolism detection

Open access

Abstract

Recent advancements in deep learning have led to a resurgence of medical imaging and Electronic Medical Record (EMR) models for a variety of applications, including clinical decision support, automated workflow triage, clinical prediction and more. However, very few models have been developed to integrate both clinical and imaging data, despite that in routine practice clinicians rely on EMR to provide context in medical imaging interpretation. In this study, we developed and compared different multimodal fusion model architectures that are capable of utilizing both pixel data from volumetric Computed Tomography Pulmonary Angiography scans and clinical patient data from the EMR to automatically classify Pulmonary Embolism (PE) cases. The best performing multimodality model is a late fusion model that achieves an AUROC of 0.947 [95% CI: 0.946–0.948] on the entire held-out test set, outperforming imaging-only and EMR-only single modality models.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41598-020-78888-w

Cites

37 of 37 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Overview of the workflow for this study. We extracted a total of 108,991 studies from Stanford University Medical Center (A) and sampled a subset (B) for manual review (C). 1837 studies remained after screening by two radiologists and were used to train and evaluate our models. Single modality models were created (D) both as baselines for comparisons as well as components for the fusion models (E).
Figure 1 Overview of the workflow for this study. We extracted a total of 108,991 studies from Stanford University Medical Center (A) and sampled a subset (B) for manual review (C). 1837 studies remained after screening by two radiologists and were used to train and evaluate our models. Single modality models were created (D) both as baselines for comparisons as well as components for the fusion models (E).CC BY · PubMed Central
Figure 2 Fusion model architectures. The 7 different fusion architectures used in this study, including (A) Early Fusion, (B) Joint All Fusion, (C) Joint Separate Fusion, (D) Late NN Average, (E) Late Elastic Average, (F) Late Separate Average and (G) Late Meta. Each input feature modality is color coded. Detailed definition of each model architecture is described in the Methods.
Figure 2 Fusion model architectures. The 7 different fusion architectures used in this study, including (A) Early Fusion, (B) Joint All Fusion, (C) Joint Separate Fusion, (D) Late NN Average, (E) Late Elastic Average, (F) Late Separate Average and (G) Late Meta. Each input feature modality is color coded. Detailed definition of each model architecture is described in the Methods.CC BY · PubMed Central
Figure 3 Two selected axial CT images of the chest from two separate patients with positive diagnosis of PE. The left CT scan demonstrates a left lower lobe posterolateral basal segmental artery filling defect consistent with a pulmonary embolism. The CT scan on the right panel demonstrates a small elongated filling defect bridging across the segmental arteries of the right lower lobe consistent with a segmental pulmonary embolism, in addition to surrounding collapse of the right lower lobe. The vision-only model yielded false-negative predictions for both cases, but the fusion model correctly predicted both as positive.
Figure 3 Two selected axial CT images of the chest from two separate patients with positive diagnosis of PE. The left CT scan demonstrates a left lower lobe posterolateral basal segmental artery filling defect consistent with a pulmonary embolism. The CT scan on the right panel demonstrates a small elongated filling defect bridging across the segmental arteries of the right lower lobe consistent with a segmental pulmonary embolism, in addition to surrounding collapse of the right lower lobe. The vision-only model yielded false-negative predictions for both cases, but the fusion model correctly predicted both as positive.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 38 references

Research works printed but not linked: 1

  1. Rajan, D., Beymer, D., Abedin, S. & Dehghan, E. Pi-PE: A pipeline for pulmonary embolism detection using sparsely annotated 3D CT images. Mach. Learn. Health NeurIPS Workshop 116, 220–232 (2020).

Cited by

See all 263

Metadata

Authors
Huang, S.-C., Pareek, A., Zamanian, R., Banerjee, I., & Lungren, M. P.
Affiliations
  • Stanford University (US)
  • Stanford Medicine (US)
  • Emory University (US)
Corresponding author
Huang, Shih-Cheng
Venue
Scientific Reports
Published
2020-12-17
Volume
10
Issue
1
Pages
22147-22147
ISSN
2045-2322 (ISSN-L)
OpenAlex
W3112990870
Cited by
246
References
38
Citations by year
2026: 38 · 2025: 56 · 2024: 62 · 2023: 49 · 2022: 31 · 2021: 9
FWCI
8.75
Type
journal article
Language
en
Open access
Licence
Locations
  • Scientific Reports (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
U.S. National Library of Medicine (R01LM012966)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
multimodal fusion, pulmonary embolism detection, CT pulmonary angiography, electronic health records, decision fusion, automated diagnosis, deep neural network
MeSH terms
Clinical Decision-Making, Machine Learning, Humans, Image Interpretation, Computer-Assisted, Image Processing, Computer-Assisted, Pulmonary Embolism, Sensitivity and Specificity, Tomography, X-Ray Computed, Neural Networks, Computer, Disease Management, Workflow, Electronic Health Records

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Multimodal fusion with deep neural networks for leveraging CT imaging and electronic health record: a case-study in pulmonary embolism detection. https://app.cochat.ai/papers/10.1038/s41598-020-78888-w
The paper
Huang, S.-C., Pareek, A., Zamanian, R., Banerjee, I., & Lungren, M. P. (2020). Multimodal fusion with deep neural networks for leveraging CT imaging and electronic health record: a case-study in pulmonary embolism detection. Scientific Reports. https://doi.org/10.1038/s41598-020-78888-w

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.