Multimodal Data Hybrid Fusion and Natural Language Processing for Clinical Prediction Models
Open accessAbstract
Objective: To propose a novel approach for enhancing clinical prediction models by combining structured and unstructured data with multimodal data fusion. Methods: We presented a comprehensive framework that integrated multimodal data sources, including textual clinical notes, structured electronic health records (EHRs), and relevant clinical data from National Electronic Injury Surveillance System (NEISS) datasets. We proposed a novel hybrid fusion method, which incorporated state-of-the-art pre-trained language model, to integrate unstructured clinical text with structured EHR data and other multimodal sources, thereby capturing a more comprehensive representation of patient information. Results: The experimental results demonstrated that the hybrid fusion approach significantly improved the performance of clinical prediction models compared to traditional fusion frameworks and unimodal models that rely solely on structured data or text information alone. The proposed hybrid fusion system with RoBERTa language encoder achieved the best prediction of the Top 1 injury with an accuracy of 75.00% and Top 3 injuries with an accuracy of 93.54%. Conclusion: Our study highlights the potential of integrating natural language processing (NLP) techniques with multimodal data fusion for enhancing clinical prediction models’ performances. By leveraging the rich information present in clinical text and combining it with structured EHR data, the proposed approach can improve the accuracy and robustness of predictive models. The approach has the potential to advance clinical decision support systems, enable personalized medicine, and facilitate evidence-based health care practices. Future research can further explore the application of this hybrid fusion approach in real-world clinical settings and investigate its impact on improving patient outcomes.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Hybrid fusion integrates unstructured and structured data along with other multimodal sources via a pretrained language model, capturing a more comprehensive patient representation.
“Hybrid fusion [144] integrates unstructured and structured data along with other multimodal sources via a pretrained language model, capturing a more comprehensive patient representation.”
From Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook. (2024) · cites this paper as [144]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Focal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.324 · Cited by 30,581 · see citing works
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works
Focal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2858826 · Cited by 9,556 · see citing works
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · see citing works
Image Captioning with Semantic Attention
You, Q., Jin, H., Wang, Z., Chen, F., & Luo, J. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.503 · Cited by 1,859 · see citing works
Self-citationMultimodal Data Fusion: An Overview of Methods, Challenges, and Prospects
Lahat, D., Adali, T., & Jutten, C. (2015) · Proceedings of the IEEE · doi:10.1109/jproc.2015.2460697 · Cited by 1,296 · see citing works
A survey on deep learning for multimodal data fusion
Gao, J., Li, P., Chen, Z., & Zhang, J. (2020) · Neural Computation · doi:10.1162/neco_a_01273 · Cited by 821 · see citing works
Multimodal deep learning for biomedical data fusion: a review
Stahlschmidt, S. R., Ulfenborg, B., & Synnergren, J. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbab569 · Cited by 766 · see citing works
ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission
Huang, K., Altosaar, J., & Ranganath, R. (2019) · arXiv · doi:10.48550/arxiv.1904.05342 · Cited by 618 · see citing works
Liao, K. P., Cai, T. A., Savova, G., Murphy, S. N., Karlson, E. W., Ananthakrishnan, A. N., Gainer, V. S., Shaw, S. Y., Xia, Z., Peter Szolovits, B., et al. (2015) · BMJ · doi:10.1136/bmj.h1885 · Cited by 284 · see citing works
Early vs Late Fusion in Multimodal Convolutional Neural Networks
Gadzicki, K., Khamsehashari, R., & Zetzsche, C. (2020) · International Conference on Information Fusion (FUSION) · doi:10.23919/fusion45008.2020.9190246 · Cited by 256 · see citing works
Boulahia, S. Y., Amamra, A., Madi, M. R., & Daikh, S. (2021) · Machine Vision and Applications · doi:10.1007/s00138-021-01249-8 · Cited by 236 · see citing works
MEANS: A medical question-answering system combining NLP techniques and semantic Web technologies
Abacha, A. B., & Zweigenbaum, P. (2015) · Information Processing & Management · doi:10.1016/j.ipm.2015.04.006 · Cited by 232 · see citing works
Ye, J. (2020) · JMIR Medical Informatics · doi:10.2196/19866 · Cited by 194 · see citing works
Self-citationCohen, D. J., Keller, S., Hayes, G. R., Dorr, D. A., Ash, J. S., & Sittig, D. F. (2016) · JMIR Human Factors · doi:10.2196/humanfactors.5919 · Cited by 153 · see citing works
Ye, J., He, L., & Beestrum, M. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00908-6 · Cited by 145 · see citing works
Self-citationExtraction of Adverse Drug Effects from Clinical Records
Aramaki, E., Miura, Y., Tonoike, M., Ohkuma, T., Masuichi, H., Waki, K., & Ohe, K. (2010) · Studies in health technology and informatics · doi:10.3233/978-1-60750-588-4-739 · Cited by 130 · see citing works
Ye, J., Yao, L., Shen, J., Janarthanam, R., & Luo, Y. (2020) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-020-01318-4 · Cited by 105 · see citing works
Self-citationNielson, J. L., Cooper, S. R., Yue, J. K., Sorani, M. D., Inoue, T., Yuh, E. L., Mukherjee, P., Petrossian, T. C., Paquette, J., Lum, P. Y., et al. (2017) · PLoS ONE · doi:10.1371/journal.pone.0169490 · Cited by 100 · see citing works
Abdulsalam, A. A., & Meystre, S. M. (2015) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2015.08.002 · Cited by 87 · see citing works
The impact of electronic health record–integrated patient-generated health data on clinician burnout
Ye, J. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab017 · Cited by 80 · see citing works
Self-citationClinical-Longformer and Clinical-BigBird: Transformers for long clinical sequencesPreprint
Li, Y., Wehbe, R. M., Ahmad, F. S., Wang, H., & Luo, Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2201.11838 · Cited by 65 · see citing works
Identifying Practice Facilitation Delays and Barriers in Primary Care Quality Improvement
Ye, J., Zhang, R., Bannon, J. E., Wang, A. A., Walunas, T. L., Kho, A., & Soulakis, N. D. (2020) · The Journal of the American Board of Family Medicine · doi:10.3122/jabfm.2020.05.200058 · Cited by 57 · see citing works
Self-citationNagamine, T., Gillette, B. M., Pakhomov, A., Kahoun, J., Mayer, H., Burghaus, R., Lippert, J., & Saxena, M. (2020) · Scientific Reports · doi:10.1038/s41598-020-77286-6 · Cited by 39 · see citing works
Ye, J. (2021) · Applied Clinical Informatics · doi:10.1055/s-0041-1728770 · Cited by 33 · see citing works
Self-citationYe, J., Hai, J., Wang, Z., Wei, C., & Song, J. (2023) · JAMIA Open · doi:10.1093/jamiaopen/ooad023 · Cited by 32 · see citing works
Self-citationYe, J. (2023) · JMIR Perioperative Medicine · doi:10.2196/34453 · Cited by 32 · see citing works
Self-citationNatural language processing of lifestyle modification documentation
Shoenbill, K. A., Song, Y., Gress, L., Johnson, H. M., Smith, M. A., & Mendonça, E. A. (2019) · Health Informatics Journal · doi:10.1177/1460458218824742 · Cited by 28 · see citing works
Ye, J., Wang, Z., & Hai, J. (2022) · Journal of Medical Internet Research · doi:10.2196/30898 · Cited by 28 · see citing works
Self-citationYe, J., Woods, D. M., Bannon, J. E., Bilaver, L., Kricke, G. S., McHugh, M., Kho, A., & Walunas, T. L. (2022) · JMIR Human Factors · doi:10.2196/32174 · Cited by 28 · see citing works
Self-citationYe, J., & Sanchez-Pinto, L. N. (2020) · PubMed · Cited by 24 · see citing works
Self-citationKhan, N. U., Khan, U. R., Ahmed, N., Ali, A., Raheem, A., Soomar, S. M., Waheed, S., Kerai, S. M., Baig, M. A., Salman, S., et al. (2023) · BMC Emergency Medicine · doi:10.1186/s12873-022-00768-5 · Cited by 18 · see citing works
Ye, J., Xiong, S., Wang, T., Li, J., Cheng, N., Tian, M., & Yang, Y. (2023) · JMIR Medical Informatics · doi:10.2196/47052 · Cited by 17 · see citing works
Self-citationNunes, A. P., Seeger, J. D., Stewart, A., Gupta, A., & McGraw, T. (2021) · The Journal of Sexual Medicine · doi:10.1016/j.jsxm.2021.10.012 · Cited by 13 · see citing works
Ye, J. (2021) · PubMed Central · Cited by 12 · see citing works
Self-citationLeveraging Clinical BERT in Multimodal Mortality Prediction Models for COVID-19
Pawar, Y., Henriksson, A., Hedberg, P., & Nauclér, P. (2022) · Proceedings - IEEE Symposium on Computer-Based Medical Systems · doi:10.1109/cbms55023.2022.00042 · Cited by 12 · see citing works
Patient Portal Perceptions in an Urban Community Health Center Setting: Insights for Telehealth
Sakumoto, M., Ye, J., Kalu, R., Jackson, K., Rittner, S. S., Long, T., Goel, M. S., & Walunas, T. L. (2022) · Telehealth and Medicine Today · doi:10.30953/thmt.v7.373 · Cited by 11 · see citing works
Self-citationSound Event Detection Guided by Semantic Contexts of Scenes
Tonami, N., Imoto, K., Nagase, R., Okamoto, Y., Fukumori, T., & Yamashita, Y. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747588 · Cited by 7 · see citing works
Rosner, B. I., Kvedar, J. C., & Adler‐Milstein, J. (2021) · JAMIA Open · doi:10.1093/jamiaopen/ooab097 · Cited by 4 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
Cited by
Sorted by citations
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook.
AlSaad, R., Abd-alrazaq, A., Boughorbel, S., Ahmed, A., Renault, M.-A., Damseh, R., & Sheikh, J. (2024) · Journal of medical Internet research · doi:10.2196/59505 · Cited by 349 · see citing works
“Hybrid fusion [144] integrates unstructured and structured data along with other multimodal sources via a pretrained language model, capturing a more comprehensive patient representation.”
Ye, J., He, L., & Beestrum, M. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00908-6 · Cited by 145 · see citing works
Self-citationMultimodal Interaction, Interfaces, and Communication: A Survey
Dritsas, E., Τρίγκα, Μ., Troussas, C., & Phivos Mylonas, B. (2025) · Multimodal Technologies and Interaction · doi:10.3390/mti9010006 · Cited by 108 · see citing works
Ye, J., Hai, J., Song, J., & Wang, Z. (2024) · medRxiv · doi:10.1101/2024.05.01.24306690 · Cited by 33 · see citing works
Self-citationIbrahim, S., Li, M., Patel, J., & Katapally, T. (2025) · Comput. Biol. Medicine · doi:10.1016/j.compbiomed.2025.109859 · Cited by 26 · see citing works
Clinical applications of large language models in medicine and surgery: A scoping review
Liang, E. N., Pei, S., Staibano, P., & van der Woerd, B. D. (2025) · Journal of International Medical Research · doi:10.1177/03000605251347556 · Cited by 26 · see citing works
Teles, A. S., de Moura, I. R., Silva, F., Roberts, A., & Stahl, D. R. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.102981 · Cited by 22 · see citing works
Siam, K., Faruk, M. J. H., He, B., Cheng, J. Q., & Gu, H. (2025) · Information · doi:10.3390/info16110971 · Cited by 18 · see citing works
Ye, J., Xiong, S., Wang, T., Li, J., Cheng, N., Tian, M., & Yang, Y. (2023) · JMIR Medical Informatics · doi:10.2196/47052 · Cited by 17 · see citing works
Self-citationUsing large language model (LLM) to identify high-burden informal caregivers in long-term care
Chien, S.-C., Yen, C.-M., Chang, Y.-H., Chen, Y., Liu, C.-C., Hsiao, Y.-P., Yang, P.-Y., Lin, H.-M., Yang, T.-E., Lu, X., et al. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2024.108329 · Cited by 15 · see citing works
Zhang, S., Ding, S., Xu, Z., & Ye, J. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1686378 · Cited by 10 · see citing works
Self-citationYe, J., He, L., Hai, J., Xu, C., Ding, S., & Beestrum, M. (2024) · medRxiv · doi:10.1101/2024.02.27.24303468 · Cited by 10 · see citing works
Self-citationJaradat, S., Elhenawy, M. M. Z., Nayak, R., Paz, A., Ashqar, H. I., & Glaser, S. (2025) · AI · doi:10.3390/ai6040072 · Cited by 9 · see citing works
Zhang, Z., & Ye, J. (2025) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2025.1568907 · Cited by 7 · see citing works
Self-citationArtificial Intelligence in Anesthesiology Drug Administration
Ye, J., & Bronstein, S. (2025) · Anesthesiology Clinics · doi:10.1016/j.anclin.2025.05.002 · Cited by 7 · see citing works
Self-citationZhang, W., Luo, L., He, M., Hai, J., & Ye, J. (2026) · Scientific Reports · doi:10.1038/s41598-025-33843-5 · Cited by 5 · see citing works
Self-citationSingh, R. K., Chen, C., Bekena, S., Brown, D. C., Taylor, K., Blake, M., Zhu, Y., Beyene, K. A., Carr, D. B., & Babulal, G. M. (2025) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-025-03332-w · Cited by 4 · see citing works
Artificial intelligence and digital health in Anesthesiology
Ye, J., & Bronstein, S. (2025) · Elsevier eBooks · doi:10.1016/b978-0-443-15728-8.00008-2 · Cited by 4 · see citing works
Self-citationComprehensive linkage between molecular biology and imaging radiomics for thyroid nodules
He, Z.-X., Zhang, X., & Yang, J. (2025) · World Journal of Radiology · doi:10.4329/wjr.v17.i9.111005 · Cited by 1 · see citing works
Babatunde, A. N., Oyinloye, D. P., Ogundokun, R. O., Aluko, F. A., Babatunde, A. J., Kadri, A. F., Mohammed, S. B., Popoola, D. D., Awotunde, J. B., & Bello, R.-W. (2026) · Scientific Reports · doi:10.1038/s41598-026-54871-9 · Cited by 1 · see citing works
Chen, H. C., & Ye, J. (2025) · medRxiv · doi:10.64898/2025.12.25.25343007 · Cited by 1 · see citing works
Self-citationKulagina, T. Y., Yelantsev, S. V., Zakatov, M. D., Егоров, А. В., & Va, S. (2025) · Problems of Balneology Physiotherapy and Exercise Therapy · doi:10.17116/kurort202510205266
Liu, D., Chen, Z., Ji, S., Zheng, X., Zhang, X., Zhou, Z., Zhang, Y., & Sun, G. (2025) · International Conference on Applied Statistics, Modeling and Advanced Algorithms (ASMA) · doi:10.1109/asma67812.2025.11553036
Zhang, W., Luo, L., He, M., Beestrum, M., & Ye, J. (2026) · BMC Medical Imaging · doi:10.1186/s12880-026-02600-z
Self-citation
- Authors
- Ye, J., Hai, J., Song, J., & Wang, Z.
- Affiliations
- Northwestern University (US)
- Johns Hopkins University (US)
- University of Hong Kong (HK)
- Corresponding author
- Ye, Jiancheng
- Venue
- medRxiv
- Publisher
- openRxiv
- Published
- 2023-08-25
- Volume
- 2024
- Pages
- 191-200
- PMC
- PMC11141806
- OpenAlex
- W4386157633
- Cited by
- 24
- References
- 37
- Citations by year
- 2026: 3 · 2025: 15 · 2024: 4 · 2023: 2
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Licence
- cc-by-nc
- Locations
- medRxiv (acceptedVersion, cc-by-nc)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- multimodal fusion, hybrid fusion, clinical prediction model, electronic health records, clinical notes, National Electronic Injury Surveillance System, injury prediction, RoBERTa, pre-trained language models, clinical decision support
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
