Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation
Open accessAbstract
Generating long and coherent reports to describe medical images poses challenges to bridging visual patterns with informative human linguistic descriptions. We propose a novel Hybrid Retrieval-Generation Reinforced Agent (HRGR-Agent) which reconciles traditional retrieval-based approaches populated with human prior knowledge, with modern learning-based approaches to achieve structured, robust, and diverse report generation. HRGR-Agent employs a hierarchical decision-making procedure. For each sentence, a high-level retrieval policy module chooses to either retrieve a template sentence from an off-the-shelf template database, or invoke a low-level generation module to generate a new sentence. HRGR-Agent is updated via reinforcement learning, guided by sentence-level and word-level rewards. Experiments show that our approach achieves the state-of-the-art results on two medical report datasets, generating well-balanced structured sentences with robust coverage of heterogeneous medical report contents. In addition, our model achieves the highest detection accuracy of medical terminologies, and improved human evaluation performance.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Deep learning methods have been used for automated generation of radiology reports, typically incorporating LSTM network models to generate textual paragraphs, and to identify findings in radiology reports.
“Recently, deep learning methods have also been used for automated generation of radiology reports , typically incorporating long-short-term-memory (LSTM) network models to generate the textual paragraphs [314, 315, 316, 317], and also to identify findings in radiology reports [318, 319, 320].”
From An overview of deep learning in medical imaging focusing on MRI (2019) · cites this paper as [315]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
Feature Pyramid Networks for Object Detection
Lin, T.-Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.106 · Cited by 30,939 · see citing works
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · see citing works
(2004) · Cited by 9,718 · see citing works
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · see citing works
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 7,936 · see citing works
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318 · see citing works
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · see citing works
Long-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228 · see citing works
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902 · see citing works
CIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845 · see citing works
Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., & Summers, R. M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.369 · Cited by 4,616 · see citing works
(2005) · Cited by 4,152 · see citing works
Get To The Point: Summarization with Pointer-Generator Networks
See, A., Liu, P. J., & Manning, C. D. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1099 · Cited by 3,775 · see citing works
Self-Critical Sequence Training for Image Captioning
Rennie, S. J., Marcheret, E., Mroueh, Y., Ross, J., & Goel, V. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.131 · Cited by 2,177 · see citing works
Generating Sentences from a Continuous Space
Bowman, S. R., Vilnis, L., Vinyals, O., Dai, A., Jozefowicz, R., & Bengio, S. (2016) · Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning · doi:10.18653/v1/k16-1002 · Cited by 2,061 · see citing works
Image Captioning with Semantic Attention
You, Q., Jin, H., Wang, Z., Chen, F., & Luo, J. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.503 · Cited by 1,859 · see citing works
Knowing When to Look: Adaptive Attention via a Visual Sentinel for Image Captioning
Lu, J., Xiong, C., Parikh, D., & Socher, R. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.345 · Cited by 1,724 · see citing works
From captions to visual concepts and back
Fang, H., Gupta, S., Iandola, F., Srivastava, R. K., Deng, L., Dollár, P., Gao, J., He, X., Mitchell, M., Platt, J. C., et al. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298754 · Cited by 1,305 · see citing works
A Deep Reinforced Model for Abstractive Summarization
Paulus, R., Xiong, C., & Socher, R. (2017) · arXiv · doi:10.48550/arxiv.1705.04304 · Cited by 1,182 · see citing works
Preparing a collection of radiology examinations for distribution and retrieval
Demner-Fushman, D., Kohli, M. D., Rosenman, M. B., Shooshan, S. E., Rodriguez, L., Antani, S., Thoma, G. R., & McDonald, C. J. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv080 · Cited by 1,164 · see citing works
Long-term Recurrent Convolutional Networks for Visual Recognition and Description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., & Darrell, T. (2014) · doi:10.21236/ada623249 · Cited by 1,081 · see citing works
Toward Controlled Generation of TextPreprint
Hu, Z., Yang, Z., Liang, X., Salakhutdinov, R., & Xing, E. P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.00955 · Cited by 802 · see citing works
Self-citationOn the Automatic Generation of Medical Imaging Reports
Jing, B., Xie, P., & Xing, E. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1240 · Cited by 657 · see citing works
Self-citationSequence Level Training with Recurrent Neural Networks
Ranzato, M., Chopra, S., Auli, M., & Zaremba, W. (2016) · International Conference on Learning Representations · Cited by 572 · see citing works
Challenges in Data-to-Document Generation
Wiseman, S., Shieber, S., & Rush, A. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1239 · Cited by 524 · see citing works
Dayan, P., & Hinton, G. E. (1992) · Cited by 488 · see citing works
Improved Image Captioning via Policy Gradient optimization of SPIDEr
Liu, S., Zhu, Z., Ye, N., Guadarrama, S., & Murphy, K. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.100 · Cited by 430 · see citing works
An Actor-Critic Algorithm for Sequence Prediction
Bahdanau, D., Brakel, P., Xu, K., Goyal, A., Lowe, R., Pineau, J., Courville, A., & Bengio, Y. (2016) · arXiv · doi:10.48550/arxiv.1607.07086 · Cited by 423 · see citing works
Video Captioning via Hierarchical Reinforcement Learning
Wang, X. E., Chen, W., Wu, J., Wang, Y.-F., & Wang, W. Y. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00443 · Cited by 269 · see citing works
Bottom-Up and Top-Down Attention for Image Captioning and VQA.Preprint
Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S. J., & Zhang, L. (2017) · arXiv (Cornell University) · Cited by 252 · see citing works
Bosmans, J. M. L., Weyler, J., de Schepper, A. M., & Parizel, P. M. (2011) · Radiology · doi:10.1148/radiol.10101045 · Cited by 238 · see citing works
Recurrent Topic-Transition GAN for Visual Paragraph Generation
Liang, X., Hu, Z., Zhang, H., Gan, C., & Xing, E. P. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.364 · Cited by 209 · see citing works
Self-citationRetrieve, Rerank and Rewrite: Soft Template Based Neural Summarization
Cao, Z., Li, W., Li, S., & Wei, F. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1015 · Cited by 202 · see citing works
No Metrics Are Perfect: Adversarial Reward Learning for Visual Storytelling
Wang, X. E., Chen, W., Wang, Y.-F., & Wang, W. Y. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1083 · Cited by 161 · see citing works
End-to-End Video Captioning With Multitask Reinforcement Learning
Li, L., & Gong, B. (2019) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2019.00042 · Cited by 70 · see citing works
Encode, Review, and Decode: Reviewer Module for Caption Generation.Preprint
Yang, Z., Yuan, Y., Wu, Y., Salakhutdinov, R., & Cohen, W. W. (2016) · arXiv (Cornell University) · Cited by 65 · see citing works
Liu, Y., Fu, J., Mei, T., & Chen, C. W. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10760 · Cited by 59 · see citing works
Hierarchical Text Generation and Planning for Strategic DialoguePreprint
Yarats, D., & Lewis, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.05846 · Cited by 44 · see citing works
Content Analysis of Reporting Templates and Free-Text Radiology Reports
Hong, Y., & Kahn, C. E. (2013) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-013-9597-4 · Cited by 36 · see citing works
Neural Baby TalkPreprint
Lu, J., Yang, J., Batra, D., & Parikh, D. (2018) · arXiv (Cornell University) · Cited by 23 · see citing works
Texar: A Modularized, Versatile, and Extensible Toolkit for Text GenerationPreprint
Hu, Z., Shi, H., Tan, B., Wang, W., Yang, Z., Zhao, T., He, J., Qin, L., Wang, D., Ma, X., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.00794 · Cited by 23 · see citing works
Self-citationGenerative Bridging Network in Neural Sequence PredictionPreprint
Chen, W., Li, G., Ren, S., Liu, S., Zhang, Z., Li, M., & Zhou, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.09152 · Cited by 1 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
An overview of deep learning in medical imaging focusing on MRI
Lundervold, A. S., & Lundervold, A. (2019) · Zeitschrift für Medizinische Physik · doi:10.1016/j.zemedi.2018.11.002 · Cited by 2,058 · see citing works
An overview of deep learning in medical imaging focusing on MRI
Lundervold, A. S., & Lundervold, A. (2018) · NORA - Norwegian Open Research Archives · Cited by 1,012 · see citing works
Generating Radiology Reports via Memory-driven Transformer
Chen, Z., Song, Y., Chang, T.-H., & Wan, X. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.112 · Cited by 693 · see citing works
Solving inverse problems using data-driven models
Arridge, S., Maaß, P., Öktem, O., & Schönlieb, C. (2019) · Acta Numerica · doi:10.1017/s0962492919000059 · Cited by 691 · see citing works
Advances in medical image analysis with vision Transformers: A comprehensive review
Azad, R., Kazerouni, A., Heidari, M., Aghdam, E. K., Molaei, A., Jia, Y., Jose, A., Roy, R., & Merhof, D. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.103000 · Cited by 523 · see citing works
Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation
Liu, F., Wu, X., Ge, S., Fan, W., & Zou, Y. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01354 · Cited by 396 · see citing works
Cross-modal Memory Networks for Radiology Report Generation
Chen, Z., Shen, Y., Song, Y., & Wan, X. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.459 · Cited by 369 · see citing works
Knowledge-Driven Encode, Retrieve, Paraphrase for Medical Image Report Generation
Li, C. Y., Liang, X., Hu, Z., & Xing, E. P. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016666 · Cited by 316 · see citing works
Self-citationA survey on incorporating domain knowledge into deep learning for medical image analysis
Xie, X., Niu, J., Liu, X., Chen, Z., Tang, S., & Yu, S. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.101985 · Cited by 316 · see citing works
Li, C. (2019) · Chapman & Hall/CRC eBooks · doi:10.1201/9781351006620-6 · Cited by 251 · see citing works
Self-citationKnowledge matters: Chest radiology report generation with general and specific knowledge
Yang, S., Wu, X., Ge, S., Zhou, S. K., & Xiao, L. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102510 · Cited by 225 · see citing works
Interactive and Explainable Region-guided Radiology Report Generation
Tanida, T., Müller, P., Kaissis, G. A., & Rueckert, D. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00718 · Cited by 219 · see citing works
Clinically Accurate Chest X-Ray Report GenerationPreprint
Liu, G., Hsu, T.-M. H., McDermott, M. B. A., Boag, W., Weng, W., Peter Szolovits, B., & Ghassemi, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.02633 · Cited by 212 · see citing works
Dynamic Graph Enhanced Contrastive Learning for Chest X-Ray Report Generation
Li, M., Lin, B., Chen, Z., Lin, H., Liang, X., & Chang, X. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00325 · Cited by 206 · see citing works
Self-citationRadiology report generation with a learned knowledge base and multi-modal alignment
Yang, S., Wu, X., Ge, S., Zheng, Z., Zhou, S. K., & Xiao, L. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102798 · Cited by 199 · see citing works
VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning
Chen, J., Guo, H., Yi, K., Li, B., & Elhoseiny, M. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01750 · Cited by 205 · see citing works
Deep learning in generating radiology reports: A survey
Monshi, M. M. A., Poon, J., & Chung, Y. Y. (2020) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2020.101878 · Cited by 194 · see citing works
METransformer: Radiology Report Generation by Transformer with Multiple Learnable Expert Tokens
Wang, Z., Liu, L., Wang, L., & Zhou, L. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01112 · Cited by 184 · see citing works
“CoAtt [12] and HGRNAgent [17] only report results on IU-Xray in their original paper.”
BackgroundMethodInfluentialContrastive Attention for Automatic Chest X-ray Report Generation
Liu, F., Yin, C., Wu, X., Ge, S., Zhang, P., & Sun, X. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.23 · Cited by 175 · see citing works
Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports
Zhang, Y., Merck, D., Tsai, E., Manning, C. D., & Langlotz, C. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.458 · Cited by 171 · see citing works
Improving chest X-ray report generation by leveraging warm starting
Nicolson, A., Dowling, J., & Koopman, B. (2023) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2023.102633 · Cited by 161 · see citing works
KiUT: Knowledge-injected U-Transformer for Radiology Report Generation
Huang, Z., Zhang, X., & Zhang, S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01897 · Cited by 149 · see citing works
Triple attention learning for classification of 14 thoracic diseases using chest radiography
Wang, H., Wang, S., Qin, Z., Zhang, Y., Li, R., & Xia, Y. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101846 · Cited by 143 · see citing works
Competence-based Multimodal Curriculum Learning for Medical Report Generation
Liu, F., Ge, S., & Wu, X. (2021) · doi:10.18653/v1/2021.acl-long.234 · Cited by 145 · see citing works
Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generation
Miura, Y., Zhang, Y., Tsai, E., Langlotz, C., & Jurafsky, D. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.416 · Cited by 149 · see citing works
Reinforced Cross-modal Alignment for Radiology Report Generation
Qin, H., & Song, Y. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.38 · Cited by 137 · see citing works
Jiang, H., Shen, F., Gao, F., & Han, W. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.107825 · Cited by 120 · see citing works
Progressive Transformer-Based Generation of Radiology Reports
Nooralahzadeh, F., Gonzalez, N. A. P., Frauenfelder, T., Fujimoto, K., & Krauthammer, M. O. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.241 · Cited by 121 · see citing works
Automatic Generation of Medical Imaging Diagnostic Report with Hierarchical Recurrent Neural Network
Yin, C., Qian, B., Wei, J., Li, X., Zhang, X., Li, Y., & Zheng, Q. (2019) · Proceedings · doi:10.1109/icdm.2019.00083 · Cited by 116 · see citing works
Automated Radiographic Report Generation Purely on Transformer: A Multicriteria Supervised Approach
Wang, Z., Han, H., Wang, L., Li, X., & Zhou, L. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3171661 · Cited by 112 · see citing works
Reinforced Transformer for Medical Image Captioning
Xiong, Y., Du, B., & Yan, P. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32692-0_77 · Cited by 94 · see citing works
A Self-boosting Framework for Automated Radiographic Report Generation
Wang, Z., Zhou, L., Wang, L., & Li, X. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00246 · Cited by 92 · see citing works
Automated Radiology Report Generation: A Review of Recent Advances
Sloan, P., Clatworthy, P. L., Simpson, E., & Mirmehdi, M. (2024) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2024.3408456 · Cited by 98 · see citing works
PRIOR: Prototype Representation Joint Learning from Medical Images and Reports
Cheng, P., Lin, L., Lyu, J., Huang, Y., Luo, W., & Tang, X. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01953 · Cited by 83 · see citing works
Liu, G., Liao, Y., Wang, F., Zhang, B., Zhang, L., Liang, X., Wan, X., Li, S., Li, Z., Zhang, S., et al. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3099165 · Cited by 79 · see citing works
Self-citationA survey on automatic generation of medical imaging reports based on deep learning
Pang, T., Li, P., & Zhao, L. (2023) · BioMedical Engineering OnLine · doi:10.1186/s12938-023-01113-y · Cited by 80 · see citing works
Attentive and ensemble 3D dual path networks for pulmonary nodules classification
Jiang, H., Gao, F., Xu, X., Huang, F., & Zhu, S. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.03.103 · Cited by 77 · see citing works
Bootstrapping Large Language Models for Radiology Report Generation
Liu, C., Tian, Y., Chen, W., Song, Y., & Zhang, Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29826 · Cited by 78 · see citing works
Joint Embedding of Deep Visual and Semantic Features for Medical Image Report Generation
Yang, Y. Q., Yu, J., Zhang, J., Han, W., Jiang, H., & Huang, Q. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3122542 · Cited by 73 · see citing works
ChestXRayBERT: A Pretrained Language Model for Chest Radiology Report Summarization
Cai, X., Liu, S., Han, J., Yang, L., Liu, Z., & Liu, T. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3132724 · Cited by 72 · see citing works
Weakly Supervised Contrastive Learning for Chest X-Ray Report Generation
Yan, A., He, Z., Lü, X., Jiang, D., Chang, E., Gentili, A., McAuley, J., & Hsu, C. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.336 · Cited by 79 · see citing works
Reale-Nosei, G., Amador-Domínguez, E., & Serrano, E. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103264 · Cited by 68 · see citing works
Learning to Generate Clinically Coherent Chest X-Ray Reports
Lovelace, J., & Mortazavi, B. J. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.110 · Cited by 66 · see citing works
Huang, X., Yan, F., Xu, W., & Li, M. (2019) · IEEE Access · doi:10.1109/access.2019.2947134 · Cited by 65 · see citing works
Unifying neural learning and symbolic reasoning for spinal medical report generation
Han, Z., Wei, B., Xi, X., Chen, B., Yin, Y., & Li, S. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101872 · Cited by 64 · see citing works
CAMANet: Class Activation Map Guided Attention Network for Radiology Report Generation
Wang, J., Bhalerao, A. H., Yin, T., See, S. C. W., & He, Y. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3354712 · Cited by 66 · see citing works
Semi-Supervised Medical Report Generation via Graph-Guided Hybrid Feature Consistency
Zhang, K., Jiang, H., Zhang, J., Huang, Q., Jianping Fan, B., Yu, J., & Han, W. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3273390 · Cited by 62 · see citing works
TSGET: Two-Stage Global Enhanced Transformer for Automatic Radiology Report Generation
Yi, X., Fu, Y., Liu, R., Zhang, H., & Hua, R. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3350077 · Cited by 61 · see citing works
MMTN: Multi-Modal Memory Transformer Network for Image-Report Consistent Medical Report Generation
Cao, Y., Cui, L., Zhang, L., Yu, F., Li, Z., & Xu, Y. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i1.25100 · Cited by 59 · see citing works
Chest X-Ray Report Generation Through Fine-Grained Label Learning
Syeda-Mahmood, T. F., Wong, K. C. L., Gur, Y., Wu, J. T.-Y., Jadhav, A., Kashyap, S., Karargyris, A., Pillai, A., Sharma, A., Syed, A., et al. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-59713-9_54 · Cited by 57 · see citing works
Memory Guided Transformer With Spatio-Semantic Visual Extractor for Medical Report Generation
Divya, P., Yenduri, S., Vishnu, C., Mohan, C. K., & Chen, Y. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3371894 · Cited by 61 · see citing works
Automated Generation of Accurate & Fluent Medical X-ray Reports
Nguyen, H. N., Nie, D., Badamdorj, T., Liu, Y., Zhu, Y., Truong, J., & Li, C. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.288 · Cited by 60 · see citing works
Self-citationContrast-Attentive Thoracic Disease Recognition With Dual-Weighting Graph Reasoning
Zhou, Y., Zhou, T., Zhou, T., Fu, H., Liu, J., & Shao, L. (2021) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2021.3049498 · Cited by 54 · see citing works
Multi-Grained Radiology Report Generation With Sentence-Level Image-Language Contrastive Learning
Liu, A., Guo, Y., Yong, J., & Xu, F. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3372638 · Cited by 55 · see citing works
Generating Radiology Report Via Auxiliary Signal Guidance and Memory-Driven Network
Xue, Y., Tan, Y., Tan, L., Qin, J., & Xiang, X. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121260 · Cited by 55 · see citing works
Multi-task paired masking with alignment modeling for medical vision-language pre-training
Zhang, K., Yang, Y., Yu, J., Jiang, H., Fan, J., Huang, Q., & Han, W. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3325965 · Cited by 51 · see citing works
Park, H., Kim, K. M., Park, S., & Choi, J. (2021) · IEEE Access · doi:10.1109/access.2021.3124564 · Cited by 52 · see citing works
Attention based automated radiology report generation using CNN and LSTM
Sirshar, M., Paracha, M. F. K., Akram, M. U., Alghamdi, N. S., Zaidi, S. Z. Y., & Fatima, T. (2022) · PLoS ONE · doi:10.1371/journal.pone.0262209 · Cited by 52 · see citing works
Abbass, M. A. B., & Kang, H. (2023) · Drones · doi:10.3390/drones7040225 · Cited by 52 · see citing works
ORGAN: Observation-Guided Radiology Report Generation via Tree Reasoning
Hou, W., Xu, K., Cheng, Y., Li, W., & Liu, J. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.451 · Cited by 55 · see citing works
SODA: Detecting COVID-19 in Chest X-Rays With Semi-Supervised Open Set Domain Adaptation
Zhou, J., Jing, B., Wang, Z., Xin, H., & Tong, H. (2021) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2021.3066331 · Cited by 49 · see citing works
Automatic ultrasound image report generation with adaptive multimodal attention mechanism
Yang, S., Niu, J., Wu, J., Wang, Y., Liu, X., & Li, Q. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.09.084 · Cited by 47 · see citing works
MKCL: Medical Knowledge with Contrastive Learning model for radiology report generation
Hou, X., Liu, Z., Li, X., Li, X., Sang, S., & Zhang, Y. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104496 · Cited by 47 · see citing works
Reinforcement Learning ApplicationsPreprint
Li, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.06973 · Cited by 58 · see citing works
Writing by Memorizing: Hierarchical Retrieval-based Medical Report Generation
Yang, X., Ye, M., You, Q., & Ma, F. (2021) · doi:10.18653/v1/2021.acl-long.387 · Cited by 49 · see citing works
Kaur, N., Mittal, A., & Singh, G. (2021) · Multimedia Tools and Applications · doi:10.1007/s11042-021-11272-6 · Cited by 43 · see citing works
Nam, W. K., & Jang, B. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121168 · Cited by 45 · see citing works
Unify, Align and Refine: Multi-Level Semantic Alignment for Radiology Report Generation
Li, Y., Yang, B., Cheng, X., Zhu, Z., Li, H., & Zou, Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00268 · Cited by 43 · see citing works
AnaXNet: Anatomy Aware Multi-label Finding Classification in Chest X-Ray
Agu, N. N., Wu, J. T.-Y., Chao, H., Lourentzou, I., Sharma, A., Moradi, M., Yan, P., & Hendler, J. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87240-3_77 · Cited by 42 · see citing works
Unifying Relational Sentence Generation and Retrieval for Medical Image Report Composition
Wang, F., Liang, X., Xu, L. H., & Lin, L. (2020) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2020.3026098 · Cited by 41 · see citing works
Self-citationPrior Guided Transformer for Accurate Radiology Reports Generation
Yan, B., Pei, M., Zhao, M., Shan, C., & Tian, Z. (2022) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2022.3197162 · Cited by 42 · see citing works
Uncertainty-aware report generation for chest X-rays by variational topic inference
Najdenkoska, I., Zhen, X., Worring, M., & Shao, L. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102603 · Cited by 41 · see citing works
Deep learning approaches to automatic radiology report generation: A systematic review
Liao, Y., Liu, H., & Spasić, . (2023) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2023.101273 · Cited by 43 · see citing works
Reinforcement learning approaches for efficient and secure blockchain-powered smart health systems
Al-Marridi, A. Z., Mohamed, A. M. S., & Erbad, A. (2021) · Computer Networks · doi:10.1016/j.comnet.2021.108279 · Cited by 44 · see citing works
Lin, Z., Zhang, D., Shi, D., Xu, R., Tao, Q., Wu, L., He, M., & Ge, Z. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104281 · Cited by 39 · see citing works
Learning Visual-Semantic Embeddings for Reporting Abnormal Findings on Chest X-rays
Ni, J., Hsu, C., Gentili, A., & McAuley, J. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.176 · Cited by 41 · see citing works
Evaluating diagnostic content of AI-generated radiology reports of chest X-rays
Babar, Z., van Laarhoven, T., Zanzotto, F. M., & Marchiori, E. (2021) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2021.102075 · Cited by 37 · see citing works
Al-Marridi, A. Z., Mohamed, A. M. S., & Erbad, A. (2024) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2024.103834 · Cited by 37 · see citing works
Utilizing Longitudinal Chest X-Rays and Reports to Pre-fill Radiology Reports
Zhu, Q., Mathai, T. S., Mukherjee, P., Peng, Y., Summers, R. M., & Lu, Z. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43904-9_19 · Cited by 38 · see citing works
From Observation to Concept: A Flexible Multi-View Paradigm for Medical Report Generation
Liu, Z., Zhu, Z., Zheng, S., Zhao, Y., He, K., & Zhao, Y. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3342691 · Cited by 38 · see citing works
RepsNet: Combining Vision with Language for Automated Medical Reports
Tanwani, A. K., Barral, J. K., & Freedman, D. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16443-9_68 · Cited by 37 · see citing works
CADxReport: Chest x-ray report generation using co-attention mechanism and reinforcement learning
Kaur, N., & Mittal, A. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.105498 · Cited by 34 · see citing works
Attributed Abnormality Graph Embedding for Clinically Accurate X-Ray Report Generation
Yan, S., Cheung, W. K., Chiu, K. W.-H., Tong, T. M., Cheung, K. C., & See, S. C. W. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3245608 · Cited by 38 · see citing works
Diagnostic captioning: a survey
Pavlopoulos, J., Kougia, V., Androutsopoulos, I., & Papamichail, D. (2022) · Knowledge and Information Systems · doi:10.1007/s10115-022-01684-7 · Cited by 37 · see citing works
Variational Topic Inference for Chest X-Ray Report Generation
Najdenkoska, I., Zhen, X., Worring, M., & Shao, L. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87199-4_59 · Cited by 31 · see citing works
A Novel Deep Learning Model for Medical Report Generation by Inter-Intra Information Calibration
Zhang, J., Shen, X., Wan, S., Goudos, S. K., Wu, J., Cheng, M., & Zhang, W. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3236661 · Cited by 32 · see citing works
Visual-Textual Attentive Semantic Consistency for Medical Report Generation
Zhou, Y., Huang, L., Zhou, T., Fu, H., & Shao, L. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00395 · Cited by 32 · see citing works
A Self-guided Framework for Radiology Report Generation
Li, J., Li, S., Hu, Y., & Tao, H. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16452-1_56 · Cited by 31 · see citing works
Zhao, G., Zhao, Z., Gong, W., & Li, F. (2023) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2023.102714 · Cited by 30 · see citing works
A Self-training Framework for Automated Medical Report Generation
Wang, S., Liu, Z., & Peng, B. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.1024 · Cited by 31 · see citing works
CNX-B2: A Novel CNN-Transformer Approach For Chest X-Ray Medical Report Generation
Alqahtani, F. F., Mohsan, M. M., Alshamrani, K. A., Zeb, J., Alhamami, S. A., & Alqarni, D. (2024) · IEEE Access · doi:10.1109/access.2024.3367360 · Cited by 36 · see citing works
Hybrid Reinforced Medical Report Generation With M-Linear Attention and Repetition Penalty
Xu, Z., Xu, W., Wang, R., Chen, J., Chang, Q., & Lukasiewicz, T. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3343391 · Cited by 32 · see citing works
Tang, Y., Yang, H., Zhang, L., & Yuan, Y. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121442 · Cited by 30 · see citing works
Hierarchical X-Ray Report Generation via Pathology Tags and Multi Head Attention
Srinivasan, P., Thapar, D., Bhavsar, A. V., & Nigam, A. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-69541-5_36 · Cited by 29 · see citing works
CLARA: Clinical Report Auto-completion
Biswal, S., Xiao, C., Glass, L. M., Westover, M. B., & Sun, J. (2020) · International World Wide Web Conference (WWW) · doi:10.1145/3366423.3380137 · Cited by 28 · see citing works
Efficient Evolving Deep Ensemble Medical Image Captioning Network
Singh, D., Kaur, M., Alanazi, J. M., AlZubi, A. A., & Lee, H.-N. (2022) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2022.3223181 · Cited by 28 · see citing works
Contextualized Keyword Representations for Multi-modal Retinal Image Captioning
Huang, J.-H., Wu, T.-W., & Worring, M. (2021) · International Conference on Multimedia Retrieval · doi:10.1145/3460426.3463667 · Cited by 27 · see citing works
A Comprehensive Study of GPT-4V’s Multimodal Capabilities in Medical ImagingPreprint
Li, Y., Liu, Y., Wang, Z., Liang, X., Liu, L., Wang, L., Cui, L., Tu, Z., Wang, L., & Zhou, L. (2023) · medRxiv · doi:10.1101/2023.11.03.23298067 · Cited by 28 · see citing works
Self-citationDeep Context-Encoding Network For Retinal Image Captioning
Huang, J.-H., Wu, T.-W., Yang, C.-H. H., & Worring, M. (2021) · Proceedings - International Conference on Image Processing · doi:10.1109/icip42928.2021.9506803 · Cited by 27 · see citing works
Retracted: Deep Learning in Dynamic Modeling of Medical Imaging: A Review Study
Swarna, S. R., Boyapati, S., Dutt, V., & Bajaj, K. (2020) · International Conference on Intelligent Sustainable Systems (ICISS) · doi:10.1109/iciss49785.2020.9315990 · Cited by 26 · see citing works
Retracted
- Authors
- Li, C. Y., Liang, X., Hu, Z., & Xing, E. P.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2018
- Volume
- 31
- Pages
- 1530-1540
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1805.08298
- OpenAlex
- W2803411968
- Cited by
- 271
- References
- 45
- Citations by year
- 2026: 2 · 2025: 32 · 2024: 54 · 2023: 62 · 2022: 33 · 2021: 48 · 2020: 23 · 2019: 15 · 2018: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- medical report generation, retrieval-augmented generation, reinforcement learning, hierarchical decision-making, structured report generation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
