Design and Development of a Multimodal Biomedical Information Retrieval System
Open accessAbstract
The search for relevant and actionable information is a key to achieving clinical and research goals in biomedicine. Biomedical information exists in different forms: as text and illustrations in journal articles and other documents, in images stored in databases, and as patients' cases in electronic health records. This paper presents ways to move beyond conventional text-based searching of these resources, by combining text and visual features in search queries and document representation. A combination of techniques and tools from the fields of natural language processing, information retrieval, and content-based image retrieval allows the development of building blocks for advanced information services. Such services enable searching by textual as well as visual queries, and retrieving documents enriched by relevant images, charts, and other illustrations from the journal literature, patient records and image databases.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The model was tested on held-out subsets of MIMIC-CXR, CheXpert (n=61), Open-I (n=2163), and US-CXR (n=1757).
“the model is tested not only on a held-out subset of the MIMIC-CXR dataset, but also on a held-out subset of the CheXpert (n = 61)59, Open-I (n = 2163)35, and US-CXR (n = 1757) datasets.”
From A clinically accessible small multimodal radiology model and evaluation metric for chest X-ray findings (2025) · cites this paper as [35]
Cites
Automatically Identifying Health Outcome Information in MEDLINE Records
Demner-Fushman, D., Few, B., Hauser, S. E., & Thoma, G. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1911 · Cited by 35 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Preparing a collection of radiology examinations for distribution and retrieval
Demner-Fushman, D., Kohli, M. D., Rosenman, M. B., Shooshan, S. E., Rodriguez, L., Antani, S., Thoma, G. R., & McDonald, C. J. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv080 · Cited by 1,164 · see citing works
Self-citationImproving Factual Completeness and Consistency of Image-to-Text Radiology Report Generation
Miura, Y., Zhang, Y., Tsai, E., Langlotz, C., & Jurafsky, D. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.416 · Cited by 149 · see citing works
Clinical Text Summarization: Adapting Large Language Models Can Outperform Human Experts
Veen, D. V., Uden, C. V., Blankemeier, L., Delbrouck, J.-B., Aali, A., Bluethgen, C., Pareek, A., Polacin, M., Reis, E. P., Seehofnerova, A., et al. (2023) · doi:10.21203/rs.3.rs-3483777/v1 · Cited by 892 · see citing works
“From the initial set of 4K studies, Demner-Fushman et al. [21] selected a final set of 3.4K reports based on the quality of imaging views and diagnostic content.”
BackgroundLi, Y., Wang, H., & Luo, Y. (2020) · 2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm49941.2020.9313289 · Cited by 81 · see citing works
Biomedical text mining for research rigor and integrity: tasks, challenges, directions
Kilicoglu, H. (2017) · Briefings in Bioinformatics · doi:10.1093/bib/bbx057 · Cited by 61 · see citing works
Zambrano Chaves, J. M., Huang, S.-C., Xu, Y., Xu, H., Usuyama, N., Zhang, S., Wang, F., Xie, Y., Khademi, M., Yang, Z., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-58344-x · Cited by 45 · see citing works
“The image also reveals a left-sided pneumothorax with a chest tube in situ, meaning that a tube has been placed in the pleural space to help remove the air and allow the lung to re-expand.”
Kaur, N., & Mittal, A. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104220 · Cited by 37 · see citing works
Improving the Factual Correctness of Radiology Report Generation with Semantic Rewards
Delbrouck, J.-B., Chambon, P., Bluethgen, C., Tsai, E., Almusa, O., & Langlotz, C. P. (2022) · arXiv · doi:10.48550/arxiv.2210.12186 · Cited by 63 · see citing works
“To generate language, we use a one-layered BERT (Vaswani et al., 2017; Devlin et al., 2019) with cross-attention over the visual features.”
Method
- Authors
- Demner-Fushman, D., Antani, S., Simpson, M., & Thoma, G. R.
- Affiliations
- United States National Library of Medicine (US)
- ORCID
- Antani, Sameer
- Venue
- Journal of Computing Science and Engineering
- Published
- 2012-06-30
- Volume
- 6
- Issue
- 2
- Pages
- 168-177
- ISSN
- 1976-4677 (ISSN-L)
- OpenAlex
- W2073982987
- Cited by
- 161
- References
- 1
- Citations by year
- 2026: 4 · 2025: 7 · 2024: 35 · 2023: 22 · 2022: 13 · 2021: 10 · 2020: 8 · 2019: 11 · 2018: 3 · 2017: 14 · 2016: 7 · 2015: 7 · 2014: 4 · 2013: 8 · 2012: 6
- FWCI
- 8.24
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- other-oa
- Locations
- Journal of Computing Science and Engineering (publishedVersion, other-oa)
- Funders
- National Institutes of Health
- U.S. National Library of Medicine
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- biomedical information retrieval, multimodal retrieval, content-based image retrieval, document representation, visual query, electronic health records, journal articles, image database, information services
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
