The DOI matches its Crossref record; 62 of the 62 references it cites were found in a registry or an index; 15 of the 19 sources its claims cite were checked for support.
K-STAMM: a knowledge-enhanced spatial - temporal attention model with multimodal fusion for pneumonia prediction
Open accessAbstract
We present K-STAMM, a knowledge-augmented spatiotemporal attention model for multimodal fusion. Different from traditional methods, K-STAMM brings together biomedical knowledge sourced from the Unified Medical Language System through embedding-based representations, which allow for semantically enriched feature learning. On top of that, it uses attention-based spatial modeling of structured EHR data without direct graph construction along with temporal sequence modeling to effectively capture disease progression at irregular time intervals. Besides, a cross-modal fusion mechanism that harmonizes chest X-ray images, clinical text, and knowledge embeddings is used to build a single and interpretable patient representation. The experimental results on MIMIC-IV and MIMIC-CXR datasets exhibit that K-STAMM surpasses strong unimodal and multimodal baselines, obtaining an AUROC of 0.953, an AUPRC of 0.962, and an F1-score of 0.910.
Overview
CoChat Summary
## TL;DR
K-STAMM is a multimodal pneumonia prediction model that combines structured EHR, chest X-ray images, clinical text, and UMLS-based knowledge embeddings with spatial, temporal, and cross-modal attention. On MIMIC-IV and MIMIC-CXR, it outperformed baseline models, reaching AUROC 0.953, AUPRC 0.962, and F1-score 0.910.
## Key findings
- On the held-out test set, K-STAMM achieved AUROC 0.953, AUPRC 0.962, F1-score 0.910, accuracy 0.912, precision 0.889, and recall 0.934.
- It outperformed unimodal and multimodal baselines, including DenseNet-121, ClinicalBERT, an EHR Transformer, and early/late fusion models.
- The paper reports that K-STAMM improved AUROC by 11.1% over the image-only baseline and 7.2% over EHR-only models.
- Ablation results showed that removing knowledge embeddings reduced AUROC by 1.5%; the paper also states that temporal attention and cross-modal fusion were important contributors.
## Methods
- The model uses a knowledge-enhanced multimodal architecture: structured EHR features are encoded with self-attention, longitudinal patient sequences with a time-aware LSTM/Bi-LSTM, chest X-rays with a DenseNet-121 image encoder, and UMLS concepts with knowledge embeddings (TransE-based, 128-dimensional).
- Data came from MIMIC-IV and MIMIC-CXR, with patient-level splitting into 70% training, 15% validation, and 15% test; the paper also says results were evaluated on the held-out test set.
- Main analyses included comparison against baseline models, ablation studies, and bootstrapped 95% confidence intervals with 1000 resamples; significance was assessed with paired bootstrap testing (p < 0.05).
## Limitations
- The paper explicitly notes that the datasets are from a single healthcare institution, and says external validation on independent datasets from different healthcare systems is crucial for generalizability.
- It also states that future work should test the model in prospective validation and across different clinical environments.
- Evident from the text: the reported performance is based on public, deidentified MIMIC data and carefully selected cohorts, so real-world deployment performance may differ from the reported results.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
13 Valid · 0 Not valid · 3 Not checked · 46 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“A meta-analysis of fusion strategies for combining imaging and EHR data show that multimodal integration leads to better predictive performance in various clinical tasks41.”
§ K-STAMM: a knowledge-enhanced spatial – temporal attention model with multimodal fusion for pneumonia prediction > Methodology > Multimodal deep learning frameworks for clinical prediction
What the sources say
- Partly supportedcites [41] Huang, S. C., Pareek, A., Seyyedi, S., Banerjee, I. & Lungren, M. P. Fusion of medical imaging and electronic health records using deep learning: A systematic review and implementation guidelines. npj Digit. Med. 3 (1), 136 (2020). sourceExperimental — accuracy not yet measured
The Discussion reports that imaging–EHR fusion models generally improved accuracy and AUROC over single-modality models across clinical applications. The source describes itself as a systematic review of 17 studies, however, not a meta-analysis; the citing sentence misidentifies the type of analysis. (LLM, conf=0.84)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“With the use of vision language models, it is possible to perform joint embedding of radiographs and clinical notes, thus enabling them to learn the features of different modes effectively48.”
§ K-STAMM: a knowledge-enhanced spatial – temporal attention model with multimodal fusion for pneumonia prediction > Methodology > Multimodal fusion of imaging and structured clinical data
What the sources say
- Supportedcites [48] Yan, B. & Pei, M. Clinical-BERT: Vision–language pre-training for radiograph diagnosis and reports generation. In Proceedings of the AAAI Conference on Artificial Intelligence. 36(3), 2982–2990. (2022). source
The Clinical-BERT section (p. 2) says its encoder learns vision and language representations jointly in a unified semantic space. Input Representations (p. 3) describes combining features from a radiograph with word embeddings from its corresponding clinical report, supporting the claim's joint-embedding paraphrase. (LLM, conf=0.90)
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Usually, recurrent neural networks (RNNs), including time-aware versions and attention-based methods, allow models to handle irregularly sampled EHR sequences and decide which time points are most clinically meaningful46.”
§ K-STAMM: a knowledge-enhanced spatial – temporal attention model with multimodal fusion for pneumonia prediction > Methodology > Temporal modeling with recurrent and attention-based networks
What the sources say
- Partly supportedcites [46] Al Olaimat, M., Bozdag, S. & Alzheimer’s Disease Neuroimaging Initiative. TA-RNN: An attention-based time-aware recurrent neural network architecture for electronic health records. Bioinformatics 40 (Supplement_1), i169–i179 (2024). sourceExperimental — accuracy not yet measured
The Introduction says time-aware RNN extensions can address irregular intervals and attention can identify significant time points. But it also says ordinary RNNs assume equal gaps between visits, so the claim is too broad in suggesting RNNs generally handle irregular sampling. (LLM, conf=0.87)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“K-STAMM shows an improvement of 11.1% in AUROC compared to the image-only baseline while also being 7.2% better than EHR-only models”
§ K-STAMM: a knowledge-enhanced spatial – temporal attention model with multimodal fusion for pneumonia prediction > Results and analysis
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The experimental results on MIMIC-IV and MIMIC-CXR datasets exhibit that K-STAMM surpasses strong unimodal and multimodal baselines, obtaining an AUROC of 0.953, an AUPRC of 0.962, and an F1-score of 0.910.”
- Also stated in: § Dataset description (EHR, CXR, Ontology Sources) · boosted
“The pipeline in this case visualizes the performance upgrades that have been achieved from the combined application of multimodal data and knowledge-informed spatial-temporal learning, which have resulted in discrimination and robustness that surpass even baseline methods.”
§ K-STAMM: a knowledge-enhanced spatial – temporal attention model with multimodal fusion for pneumonia prediction > Abstract
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Dataset description (EHR, CXR, Ontology Sources) · boosted
19 more claims
Create a free account to see all 62 claims and every source verdict.
- Probably not a substantial claim. 18
- It may state two separate claims as one. 24
- The quote only partly states this claim. 1
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ K-STAMM: a knowledge-enhanced spatial – temporal attention model with multimodal fusion for pneumonia prediction > Results and analysis
“Training accuracy increased from 71% to 91%, while validation accuracy stabilized at 90.3%, indicating good generalization.”
§ K-STAMM: a knowledge-enhanced spatial – temporal attention model with multimodal fusion for pneumonia prediction > Results and analysis
“Both curves exhibit a smooth and almost parallel rise, with the validation accuracy levelling off at about 0.91, which agrees with the final test accuracy of 0.912 shown in Table 1.”
Citations
Cites
Saito, T., & Rehmsmeier, M. (2015) · PLOS ONE · doi:10.1371/journal.pone.0118432 · Cited by 5,309 · cited 1×
“…was chosen as a threshold, independent measure of discriminative ability: where As pneumonia prediction was a problem with class imbalance, the main focus was on the Area Under the Precision Recall Curve (AUPRC): which is a more informative measure than ROC analysis in imbalanced scenarios 59 .”
Calibration: the Achilles heel of predictive analytics
On behalf of Topic Group ‘Evaluating diagnostic tests and prediction models’ of the STRATOS initiative, Van Calster, B., McLernon, D. J., van Smeden, M., Wynants, L., & Steyerberg, E. W. (2019) · BMC Medicine · doi:10.1186/s12916-019-1466-7 · Cited by 2,650 · cited 1×
“Besides that, model calibration was also taken into account to determine the accuracy of the predicted probabilities 60 .”
Scalable and accurate deep learning with electronic health records
Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., Liu, P. J., Liu, X., Marcus, J., Sun, M., et al. (2018) · npj Digital Medicine · doi:10.1038/s41746-018-0029-1 · Cited by 2,648 · cited 1×
“For instance, Li et al. 36 combined chest X-rays and EHR features over time to make a prediction of in-hospital mortality for heart failure patients.”
Huang, S.-C., Pareek, A., Seyyedi, S., Banerjee, I., & Lungren, M. P. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00341-z · Cited by 934 · cited 2×
“A meta-analysis of fusion strategies for combining imaging and EHR data show that multimodal integration leads to better predictive performance in various clinical tasks 41 .”
Rasmy, L., Xiang, Y., Xie, Z., Tao, C., & Zhi, D. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00455-y · Cited by 886 · cited 2×
“Structured EHR baselines implemented embedding, based and transformer, based architectures for disease prediction from longitudinal clinical records 56 .”
Comparison of Deep Learning Approaches for Multi-Label Chest X-Ray Classification
Baltruschat, I. M., Nickisch, H., Grass, M., Knopp, T., & Saalbach, A. (2019) · Scientific Reports · doi:10.1038/s41598-019-42294-8 · Cited by 455 · cited 2×
“The use of chest radiograph images as the major imaging method is justified by their general clinical applicability and confirmed effectiveness in pneumonia and thoracic disease detection 52,58 .”
Hidden stratification causes clinically meaningful failures in machine learning for medical imaging
Oakden-Rayner, L., Dunnmon, J., Carneiro, G., & Re, C. (2020) · Proceedings of the ACM Conference on Health, Inference, and Learning · doi:10.1145/3368555.3384468 · Cited by 411 · cited 1×
“Chest X ray images were subjected to standardized preprocessing and augmentation to help class imbalance and hidden stratification effects which are known for causing model failures that need not be considered further 55 .”
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook.
AlSaad, R., Abd-alrazaq, A., Boughorbel, S., Ahmed, A., Renault, M.-A., Damseh, R., & Sheikh, J. (2024) · Journal of medical Internet research · doi:10.2196/59505 · Cited by 349 · cited 1×
“This advancement supports not only generalized clinical reasoning but also the extension beyond unimodal networks 33 .”
Transformers and large language models in healthcare: A review
Nerella, S., Bandyopadhyay, S., Zhang, J., Contreras, M., Siegel, S., Bumin, A., Silva, B., Sena, J., Shickel, B., Bihorac, A., et al. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102900 · Cited by 285 · cited 1×
“Models that are built on a large scale and are meant for the healthcare sector, in addition, can be seen as the future of multimodal architecture which can be used to provide diagnostic support, risk prediction, and decision assistance in different clinical domains 3 .”
Time Series Prediction Using Deep Learning Methods in Healthcare
Morid, M. A., Sheng, O. R. L., & Dunbar, J. (2022) · ACM Transactions on Management Information Systems · doi:10.1145/3531326 · Cited by 283 · cited 1×
“X p = { x p, 1 , x p, 2 , . . . , x p,T } , which kept temporal dependencies necessary for modeling disease progression, i.e., they were consistent with irregular clinical time, series modeling practices 54,62 .”
Research works printed but not linked: 3
- Poulain, R. & Beheshti, R. Graph transformers on EHRs: Better representation improves downstream performance. In The Twelfth International Conference on Learning Representations. (2024).
- Subramanian, S. Recurrent multitask-learning for irregular clinical time series forecasting (Doctoral dissertation, University of Pittsburgh). (2020).
- Pavlou, M., Omar, R. Z. & Ambler, G. Sample size calculations for the development of risk prediction models that account for performance variability (arXiv preprint, 2025).
Cited by
Sruthy, K. G., & Navamani, T. M. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2026.1949796
Metadata
- Authors
- Anbukkarasi, S., Hemalatha, S., Balakrishnan, A., Varadhaganapathy, S., & Easwaramoorthy, S. V.
- Affiliations
- Manipal Academy of Higher Education (IN)
- Sunway University (MY)
- Corresponding author
- Anbukkarasi, S.
- Venue
- Scientific reports
- Published
- 2026-04-09
- Volume
- 16
- Issue
- 1
- ISSN
- 2045-2322 (ISSN-L)
- PMC
- PMC13223296
- OpenAlex
- W7152670117
- Cited by
- 1
- References
- 60
- Citations by year
- 2026: 1
- FWCI
- 3.37
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor, 53 d)
- Locations
- Scientific Reports (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Manipal Academy of Higher Education, Manipal
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- pneumonia prediction, multimodal fusion, spatio-temporal attention, knowledge improvement, electronic health records, chest X-ray images, clinical text, temporal sequence modeling, irregular time intervals, disease progression, MIMIC-IV, MIMIC-CXR
- MeSH terms
- Humans, Pneumonia, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: K-STAMM: a knowledge-enhanced spatial - temporal attention model with multimodal fusion for pneumonia prediction. https://app.cochat.ai/papers/10.1038/s41598-026-47146-w
- The paper
- Anbukkarasi, S., Hemalatha, S., Balakrishnan, A., Varadhaganapathy, S., & Easwaramoorthy, S. V. (2026). K-STAMM: a knowledge-enhanced spatial - temporal attention model with multimodal fusion for pneumonia prediction. Scientific reports. https://doi.org/10.1038/s41598-026-47146-w
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.