RadioBERT: A deep learning-based system for medical report generation from chest X-ray images using contextual embeddings
Open accessAbstract
Background Increasing number of chest X-ray (CXR) examinations in radiodiagnosis departments burdens radiologists' and makes the timely generation of accurate radiological reports highly challenging. An automatic radiological report generation (ARRG) system is envisaged to generate radiographic reports with minimal human intervention, ease radiologists' burden, and smoothen the clinical workflow. The success of an ARRG system depends on two critical factors: i) quality of the features extracted by the ARRG system from the CXR images, and ii) quality of the linguistic expression generated by the ARRG system describing the normalities and abnormalities as indicated by the extracted features. Most of the existing ARRG systems miserably fail due to the latter factor and do not generate clinically acceptable reports because they ignore the contextual importance of the medical terms.Objectives The advent of contextual word embeddings, like ELMo and BERT, has revolutionized several natural language processing (NLP) tasks. A contextual embedding represents a word based on its context. The main objective of this work is to develop an ARRG system that uses contextual word embeddings to generate clinically accurate reports from CXR images.Methods We present an end-to-end deep neural network that uses contextual word representations for generating clinically useful radiological reports from CXR images. The proposed network, termed as RadioBERT, uses DistilBERT for contextual word representation and leverages transfer learning. Additionally, due to the importance of abnormal observations over the normal ones, the network reorders the generated sentences by applying sentiment analysis to keep abnormal descriptions on the top of the generated report.Results The empirical study consisting of several experiments performed on the OpenI dataset indicates that CNN+Hierarchical LSTM with DistilBERT embedding improves the benchmark performance. We have been able to achieve the following performance scores: BLEU-1 = 0.772, BLEU-2 = 0.770, BLEU-3 = 0.768, BLEU-4 = 0.767, CIDEr = 0.5563, and ROUGE = 0.897.Conclusion The proposed method improves the state-of-the-art performance scores by a substantial margin. It is concluded that the use of word embeddings generated by DistilBERT enhances the performance of hierarchical LSTM for producing clinical reports by significant margin.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Existing studies on automatic FFA interpretation suffer from insufficient model performance and inadequate evaluation metrics.
“Secondly, existing studies are also suffering from insufficient model performance and inadequate evaluation metrics10,13.”
From Generative artificial intelligence for fundus fluorescein angiography interpretation and human expert evaluation (2025) · cites this paper as [13]
Section 3.2 explains that a commonly used evaluation metric can score reports highly even when they miss abnormalities. Sections 4.3 and 5 describe errors in chest X-ray report generation and scope for better evaluation of clinical efficacy. The paper concerns chest X-rays, not automatic FFA interpretation, so it does not establish these shortcomings for FFA studies. Page numbers are unavailable in our copy. (LLM, conf=0.83)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141 · see citing works
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · see citing works
CIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845 · see citing works
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019) · arXiv · doi:10.48550/arxiv.1910.01108 · Cited by 4,653 · see citing works
Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., & Fidler, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.11 · Cited by 2,264 · see citing works
Proceedings of the 40th Annual Meeting on Association for Computational Linguistics
Isabelle, P. (2002) · Cited by 1,633 · see citing works
Publicly Available Clinical BERT EmbeddingsPreprint
Alsentzer, E., Murphy, J. R., Boag, W., Weng, W., Jin, D., Naumann, T., & McDermott, M. B. A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.03323 · Cited by 1,040 · see citing works
2010 IEEE International Conference on Data Mining
(2010) · doi:10.1109/icdm17440.2010 · Cited by 779 · see citing works
Generating Radiology Reports via Memory-driven Transformer
Chen, Z., Song, Y., Chang, T.-H., & Wan, X. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.112 · Cited by 693 · see citing works
Wang, X., Peng, Y., Lü, L., Lu, Z., & Summers, R. M. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00943 · Cited by 574 · see citing works
A Hierarchical Approach for Generating Descriptive Image Paragraphs
Krause, J., Johnson, J., Krishna, R., & Fei-Fei, L. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.356 · Cited by 374 · see citing works
When Radiology Report Generation Meets Knowledge Graph
Zhang, Y., Wang, X., Xu, Z., Yu, Q., Yuille, A., & Xu, D. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6989 · Cited by 348 · see citing works
Knowledge-Driven Encode, Retrieve, Paraphrase for Medical Image Report Generation
Li, C. Y., Liang, X., Hu, Z., & Xing, E. P. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016666 · Cited by 316 · see citing works
Clinically Accurate Chest X-Ray Report GenerationPreprint
Liu, G., Hsu, T.-M. H., McDermott, M. B. A., Boag, W., Weng, W., Peter Szolovits, B., & Ghassemi, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.02633 · Cited by 212 · see citing works
Automated radiology report generation using conditioned transformers
Alfarghaly, O., Khaled, R., Elkorany, A., Helal, M., & Fahmy, A. (2021) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2021.100557 · Cited by 183 · see citing works
Design and Development of a Multimodal Biomedical Information Retrieval System
Demner-Fushman, D., Antani, S., Simpson, M., & Thoma, G. R. (2012) · Journal of Computing Science and Engineering · doi:10.5626/jcse.2012.6.2.168 · Cited by 161 · see citing works
Progressive Transformer-Based Generation of Radiology Reports
Nooralahzadeh, F., Gonzalez, N. A. P., Frauenfelder, T., Fujimoto, K., & Krauthammer, M. O. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.241 · Cited by 121 · see citing works
Learning to Generate Clinically Coherent Chest X-Ray Reports
Lovelace, J., & Mortazavi, B. J. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.110 · Cited by 66 · see citing works
Huang, X., Yan, F., Xu, W., & Li, M. (2019) · IEEE Access · doi:10.1109/access.2019.2947134 · Cited by 65 · see citing works
Waiting for Radiology Test Results: Patient Expectations and Emotional Disutility
Woolen, S. A., Kazerooni, E. A., Wall, A., et al. (2017) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2017.09.017 · Cited by 44 · see citing works
Kaur, N., Mittal, A., & Singh, G. (2021) · Multimedia Tools and Applications · doi:10.1007/s11042-021-11272-6 · Cited by 43 · see citing works
Self-citationAddressing Data Bias Problems for Chest X-ray Image Report GenerationPreprint
Harzig, P., Chen, Y., Chen, F., & Lienhart, R. W. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.02123 · Cited by 43 · see citing works
Vispi: Automatic Visual Perception and Interpretation of Chest X-raysPreprint
Li, X. S., Cao, R., & Zhu, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.05190 · Cited by 20 · see citing works
Waiting times for x ray results in England are increasing, figures show
Mayor, S. (2015) · BMJ · doi:10.1136/bmj.h1598 · Cited by 9 · see citing works
Flagged first, then by citations
Research works printed but not linked: 9
- Hoo-Chang Shin, Kirk Roberts, Le Lu, Dina Demner-Fushman, Jianhua Yao, Ronald M Summers, Learning to read chest x-rays: Recurrent neural cascade model for automated image annotation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 2497-2506.
- Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, Yoshua Bengio, Show, attend and tell: Neural image caption generation with visual attention, in: International Conference on Machine Learning, PMLR, 2015, pp. 2048-2057.
- Baoyu Jing, Pengtao Xie, Eric Xing, On the automatic generation of medical imaging reports, 2017, arXiv preprint arXiv:1711.08195. · arXiv
- Jianbo Yuan, Haofu Liao, Rui Luo, Jiebo Luo, Automatic radiology report generation based on multi-view image fusion and medical concept enrichment, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2019, pp. 721-729.
- Yuan Li, Xiaodan Liang, Zhiting Hu, Eric P. Xing, Hybrid retrieval-generation reinforced agent for medical image report generation, in: Advances in Neural Information Processing Systems, 2018, pp. 1530-1540.
- Yuxuan Xiong, Bo Du, Pingkun Yan, Reinforced transformer for medical image captioning, in: International Workshop on Machine Learning in Medical Imaging, Springer, 2019, pp. 673-680.
- Chin-Yew Lin, ROUGE: A package for automatic evaluation of summaries, in: Text Summarization Branches Out, 2004, pp. 74-81.
- Jacob Devlin Ming-Wei Chang Kenton, Lee Kristina Toutanova, BERT: Pre- training of deep bidirectional transformers for language understanding, in: Proceedings of NAACL-HLT, 2019, pp. 4171-4186.
- Baoyu Jing, Zeya Wang, Eric Xing, Show, describe and conclude: On exploiting the structure information of chest X-Ray reports, 2020, arXiv preprint arXiv: 2004.12274. · arXiv
Cited by
Sorted by citations
Deep learning for report generation on chest X-ray images
Ouis, M. Y., & A. Akhloufi, M. (2024) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2023.102320 · Cited by 25 · see citing works
Shao, A., Liu, X., Shen, W., Li, Y., Wu, H., Pan, X., Yang, Z., Xu, Y., Zhu, T., Wang, Y., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01759-z · Cited by 16 · see citing works
“Secondly, existing studies are also suffering from insuf fi cient model performance and inadequate evaluation metrics 10,13 .”
Li, Y., Kong, C., Zhao, G., & Zhao, Z. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11337-0 · Cited by 7 · see citing works
- Authors
- Kaur, N., & Mittal, A.
- Affiliations
- Panjab University (IN)
- Corresponding author
- Mittal, Ajay
- Venue
- Journal of Biomedical Informatics
- Publisher
- Elsevier BV
- Published
- 2022-11
- Volume
- 135
- Pages
- 104220
- ISSN
- 1532-0464 (ISSN-L)
- OpenAlex
- W4304084022
- Cited by
- 37
- References
- 27
- Citations by year
- 2026: 6 · 2025: 15 · 2024: 11 · 2023: 5
- FWCI
- 2.54
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- www.elsevier.com/legal/tdmrep-license (tdm)
- www.elsevier.com/open-access/userlicense/1.0/ (vor, 365 d)
- doi.org/10.15223/policy-017 (stm-asf)
- doi.org/10.15223/policy-037 (stm-asf)
- doi.org/10.15223/policy-012 (stm-asf)
- doi.org/10.15223/policy-029 (stm-asf)
- doi.org/10.15223/policy-004 (stm-asf)
- Locations
- Journal of Biomedical Informatics (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Department of Science and Technology, Ministry of Science and Technology, India (TDP/BDTD/45/2021)
- Ministry of Electronics and Information technology
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- chest radiography, contextual word embeddings, DistilBERT, hierarchical LSTM, clinical text generation, anomaly detection, sentiment analysis, transfer learning
- MeSH terms
- Deep Learning, Humans, Linguistics, Natural Language Processing, X-Rays, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
