On the Automatic Generation of Medical Imaging Reports
Open accessAbstract
Medical imaging is widely used in clinical practice for diagnosis and treatment.Report-writing can be error-prone for unexperienced physicians, and timeconsuming and tedious for experienced physicians.To address these issues, we study the automatic generation of medical imaging reports.This task presents several challenges.First, a complete report contains multiple heterogeneous forms of information, including findings and tags.Second, abnormal regions in medical images are difficult to identify.Third, the reports are typically long, containing multiple sentences.To cope with these challenges, we (1) build a multi-task learning framework which jointly performs the prediction of tags and the generation of paragraphs, (2) propose a co-attention mechanism to localize regions containing abnormalities and generate narrations for them, (3) develop a hierarchical LSTM model to generate long paragraphs.We demonstrate the effectiveness of the proposed methods on two publicly available datasets.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Deep learning methods have been used for automated generation of radiology reports, typically incorporating LSTM network models to generate textual paragraphs, and to identify findings in radiology reports.
“Recently, deep learning methods have also been used for automated generation of radiology reports , typically incorporating long-short-term-memory (LSTM) network models to generate the textual paragraphs [314, 315, 316, 317], and also to identify findings in radiology reports [318, 319, 320].”
From An overview of deep learning in medical imaging focusing on MRI (2019) · cites this paper as [314]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1× · see citing works
“than adopting a single-layer LSTM (Hochreiter and Schmidhuber, 1997), which is less capable of modeling long word sequences, we leverage the compositional nature of the report and adopt a hierarchical LSTM to produce long texts.”
MethodAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 1× · see citing works
“We used the Adam (Kingma and Ba, 2014) optimizer for parameter learning.”
MethodVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · cited 5× · see citing works
“Results in Table 3 show that Ours-CoAttention and VGG-19 [17] network performs very similar for tag prediction.”
BackgroundMethodInfluentialBLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · see citing works
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · see citing works
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318 · cited 7× · see citing works
“Note that the only difference between Ours-NoAttention and CNN-RNN [18] in Table 1 is that Ours-NoAttention adopts a hierarchical LSTM decoder while CNNRNN [18] only adopts a single-layer LSTM.”
BackgroundMethodInfluentialLong-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228 · see citing works
CIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845 · cited 2× · see citing works
“We report the paragraph generation (upper part of Table 1) and one sentence generation (lower part of Table 1) results using the following captioning evaluation tools: BLEU [13], METEOR [4], ROUGE [12] and CIDER [17].”
MethodMeteor Universal: Language Specific Translation Evaluation for Any Target Language
Denkowski, M., & Lavie, A. (2014) · WMT@ACL · doi:10.3115/v1/w14-3348 · Cited by 2,108 · cited 2× · see citing works
“We report the paragraph generation (upper part of Table 1) and one sentence generation (lower part of Table 1) results using the following captioning evaluation tools: BLEU [13], METEOR [4], ROUGE [12] and CIDER [17].”
MethodImage Captioning with Semantic Attention
You, Q., Jin, H., Wang, Z., Chen, F., & Luo, J. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.503 · Cited by 1,859 · cited 6× · see citing works
“We first compare our method with several state-of-the-art image captioning models: CNN-RNN [19], LRCN [5], Soft ATT [20], ATT-RK [21].”
BackgroundMethodInfluentialFrom captions to visual concepts and back
Fang, H., Gupta, S., Iandola, F., Srivastava, R. K., Deng, L., Dollár, P., Gao, J., He, X., Mitchell, M., Platt, J. C., et al. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298754 · Cited by 1,305 · see citing works
Preparing a collection of radiology examinations for distribution and retrieval
Demner-Fushman, D., Kohli, M. D., Rosenman, M. B., Shooshan, S. E., Rodriguez, L., Antani, S., Thoma, G. R., & McDonald, C. J. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv080 · Cited by 1,164 · see citing works
DenseCap: Fully Convolutional Localization Networks for Dense Captioning
Johnson, J., Karpathy, A., & Fei-Fei, L. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.494 · Cited by 1,161 · cited 2× · see citing works
“Most of recent image captioning models are based on a CNN-RNN framework [18, 6, 9, 19, 20, 8, 11].”
BackgroundMethodDeep Visual-Semantic Alignments for Generating Image Descriptions
Karpathy, A., & Fei-Fei, L. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2598339 · Cited by 1,048 · see citing works
Kastner, S., & Buschman, T. J. (2014) · doi:10.1093/acrefore/9780190264086.013.79 · Cited by 820 · cited 1×
“Previous works have shown that visual attention alone can perform fairly well for localizing the objects [1] and aiding caption generation [19].”
TagProp: Discriminative metric learning in nearest neighbor models for image auto-annotation
Guillaumin, M., Mensink, T., Verbeek, J. J., & Schmid, C. (2009) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2009.5459266 · Cited by 678 · cited 1× · see citing works
“As for the training loss of the multi-label classification (MLC) task, since the number of tags for semantic attention is fixed as 10, we treat MLC as a multilabel retrieval task and adopt a softmax crossentropy loss (a multi-label ranking loss), similar to (Gong et al., 2013; Guillaumin et al., 2009).”
MethodHow to Construct Deep Recurrent Neural NetworksPreprint
Pascanu, R., Gülçehre, Ç., Cho, K., & Bengio, Y. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.6026 · Cited by 435 · cited 1× · see citing works
“Topic Generator We use a deep output layer [14] to strengthen the context information in topic vector t.”
MethodLearning to Read Chest X-Rays: Recurrent Neural Cascade Model for Automated Image Annotation
Shin, H.-C., Roberts, K. E., Lü, L., Demner‐Fushman, D., Yao, J., & Summers, R. M. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.274 · Cited by 390 · see citing works
A Hierarchical Approach for Generating Descriptive Image Paragraphs
Krause, J., Johnson, J., Krishna, R., & Fei-Fei, L. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.356 · Cited by 374 · cited 4× · see citing works
“Following [11], when generating words, topic vector t and the special START token are used as the first and the second input of the word LSTM, and the subsequent inputs are the word sequence.”
BackgroundMethodInfluentialMDNet: A Semantically and Visually Interpretable Medical Image Diagnosis Network
Zhang, Z., Xie, Y., Xing, F., McGough, M., & Yang, L. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.378 · Cited by 354 · see citing works
Deep Convolutional Ranking for Multilabel Image AnnotationPreprint
Gong, Y., Jia, Y., Leung, T., Toshev, A., & Ioffe, S. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.4894 · Cited by 293 · cited 1× · see citing works
“Note that the loss function of both models are softmax cross-entropy loss [7].”
BackgroundRecurrent Topic-Transition GAN for Visual Paragraph Generation
Liang, X., Hu, Z., Zhang, H., Gan, C., & Xing, E. P. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.364 · Cited by 209 · see citing works
Self-citationParallelizing Linear Recurrent Neural Nets Over Sequence LengthPreprint
Martin, E., & Cundy, C. (2017) · International Conference on Learning Representations · doi:10.48550/arxiv.1709.04057 · Cited by 191 · cited 1× · see citing works
“irectly demonstrates the effectiveness of the hierarchical LSTM. This result is not surprising since it is well-known that a single-layer LSTM cannot effectively model long sequences (Liu et al.,2015;Martin and Cundy, 2018). Additionally, employing semantic attention alone (Ours-Semantic-only) or visual attention alon…”
BackgroundMulti-Timescale Long Short-Term Memory Neural Network for Modelling Sentences and Documents
Liu, P., Qiu, X., Chen, X., Wu, S., & Huang, X. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1280 · Cited by 182 · cited 1× · see citing works
“This result is not surprising since it is well-known that a single-layer LSTM cannot effectively model long sequences (Liu et al., 2015; Martin and Cundy, 2018).”
BackgroundDifficulties in the Interpretation of Chest Radiography
Delrue, L., Gosselin, R., Ilsen, B., et al. (2011) · Medical radiology · doi:10.1007/978-3-540-79942-9_2 · Cited by 76 · see citing works
From medical image to automatic medical report generation
Kisilev, P., Walach, E., Barkan, E., et al. (2015) · IBM Journal of Research and Development · doi:10.1147/jrd.2015.2393193 · Cited by 52
BMJ Publishing Group Ltd, & Royal College of Psychiatrists and British Psychological Society (2008) · Evidence-Based Mental Health · doi:10.1136/ebmh.11.4.102 · cited 2×
“Most of recent image captioning models are based on a CNN-RNN framework [18, 6, 9, 19, 20, 8, 11].”
Method
Flagged first, then by citations
1 of the 28 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- Jimmy Ba, Volodymyr Mnih, and Koray Kavukcuoglu. 2015. Multiple object recognition with visual atten- tion. ICLR.
- Chin-Yew Lin. 2004. Rouge: A package for auto- matic evaluation of summaries. In Text summariza- tion branches out: Proceedings of the ACL-04 work- shop, volume 8. Barcelona, Spain.
Cited by
Sorted by citations
An overview of deep learning in medical imaging focusing on MRI
Lundervold, A. S., & Lundervold, A. (2019) · Zeitschrift für Medizinische Physik · doi:10.1016/j.zemedi.2018.11.002 · Cited by 2,058 · see citing works
Generative adversarial network in medical imaging: A review
Xin, Y., Walia, E., & Babyn, P. (2019) · Medical Image Analysis · doi:10.1016/j.media.2019.101552 · Cited by 1,883 · see citing works
“Tang et al. (2018) SGAN L1, 2, 8 – M32 [ √ ] Denoising, contrast enhance Shan et al.”
Explainable artificial intelligence (XAI) in deep learning-based medical image analysis
van der Velden, B. H. M., Kuijf, H. J., Gilhuijs, K. G. A., & Viergever, M. A. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102470 · Cited by 1,328 · see citing works
Zhou, S. K., Greenspan, H. K., Davatzikos, C. A., Duncan, J. S., van Ginneken, B., Madabhushi, A., Prince, J. L., Rueckert, D., & Summers, R. M. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3054390 · Cited by 1,292 · see citing works
“6) Others technologies include landmark detection [18], image or view recognition [19], automatic report generation [20], and so on.”
BackgroundA General Survey on Attention Mechanisms in Deep Learning
Brauwers, G., & Frăsincar, F. (2021) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3126456 · Cited by 722 · see citing works
Generating Radiology Reports via Memory-driven Transformer
Chen, Z., Song, Y., Chang, T.-H., & Wan, X. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.112 · Cited by 693 · see citing works
From Show to Tell: A Survey on Deep Learning-Based Image Captioning
Stefanini, M., Cornia, M., Baraldi, L., Cascianelli, S., Fiameni, G., & Cucchiara, R. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3148210 · Cited by 373 · see citing works
“imitate the style of real medical reports [227], [228], [229].”
BackgroundExploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation
Liu, F., Wu, X., Ge, S., Fan, W., & Zou, Y. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01354 · Cited by 396 · see citing works
A generalist vision-language foundation model for diverse biomedical tasks
Zhang, K., Zhou, R., Adhikarla, E., Yan, Z., Liu, Y., Yu, J., Liu, Z., Chen, X., Davison, B. D., Ren, H., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03185-2 · Cited by 333 · see citing works
“On the Peir Gross dataset, our BiomedGPT model surpassed the existing SOTA benchmark 36 , demonstrating improvements of 8.1 percentage points in ROUGE-L and 0.5 points in METEOR, and a substantial gain of 89.8 points in the CIDEr metric.”
InfluentialA survey on incorporating domain knowledge into deep learning for medical image analysis
Xie, X., Niu, J., Liu, X., Chen, Z., Tang, S., & Yu, S. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.101985 · Cited by 316 · see citing works
Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation
Li, C. Y., Liang, X., Hu, Z., & Xing, E. P. (2018) · arXiv · doi:10.48550/arxiv.1805.08298 · Cited by 271 · see citing works
Self-citationInteractive and Explainable Region-guided Radiology Report Generation
Tanida, T., Müller, P., Kaissis, G. A., & Rueckert, D. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00718 · Cited by 219 · see citing works
“While early works [16, 17, 22, 48, 56] adopted CNN-RNN architectures, more recent studies have shifted towards utilizing the effectiveness of the Transformer [44].”
MethodRadiology report generation with a learned knowledge base and multi-modal alignment
Yang, S., Wu, X., Ge, S., Zheng, Z., Zhou, S. K., & Xiao, L. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102798 · Cited by 199 · see citing works
VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning
Chen, J., Guo, H., Yi, K., Li, B., & Elhoseiny, M. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01750 · Cited by 205 · see citing works
“We compared VisualGPT against state-of-the-art medical report generation models including Att2in [61], CoAtt [26], HRGR [35], CMAS-RL [25] and the model from Chen et al. [10].”
MethodMETransformer: Radiology Report Generation by Transformer with Multiple Learnable Expert Tokens
Wang, Z., Liu, L., Wang, L., & Zhou, L. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01112 · Cited by 184 · see citing works
“Moreover, eight SOTA medical report generation models are involved in the comparison, including CoAtt [12], HGRNAgent [17], R2Gen [5], R2GenCMN [4], PPKED [21], GSK [41] and MSAT [34].”
BackgroundMethodResultInfluentialYuan, J., Liao, H., Luo, R., & Luo, J. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32226-7_80 · Cited by 215 · see citing works
“ral state-of-the-art baselines: (1) a visual attention based image captioning model (Vis-Att) [13]; (2) radiology report generation models, including a hierarchical decoder with co-attention (Co-Att) [6], multimodal generative model with visual attention (MM-Att) [14], and knowledge-drive retrieval based report genera…”
BackgroundInfluentialImproving chest X-ray report generation by leveraging warm starting
Nicolson, A., Dowling, J., & Koopman, B. (2023) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2023.102633 · Cited by 161 · see citing works
Explainable Deep Learning Methods in Medical Image Classification: A Survey
Patrício, C., Neves, J. C., & Teixeira, L. F. (2023) · ACM Computing Surveys · doi:10.1145/3625287 · Cited by 190 · see citing works
“…Retina Images 2019 5,590 Multiclass LIDC-IDRI [7] CT scans 2011 1,018 Includes annotations COV-CTR [74] CT scans 2023 728 Includes reports PEIR [54] Photographs 2017 33,648 Includes reports ROCO [106] Multimodal 2018 81,825 Includes reports/UMLS Concepts The datasets marked with “ ∗ ” have…”
BackgroundMethodA medical multimodal large language model for future pandemics
Liu, F., Zhu, T., Wu, X., Yang, B., You, C., Wang, C., Lu, L., Liu, Z., Zheng, Y., Sun, X., et al. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00952-2 · Cited by 155 · see citing works
“…(EMR), Transformer based frameworks 5 have been be applied to estimate the mortality or re-hospitalisation probabilities 6 ; For multi-modal data such as radiology image-report pairs, the encoder-decoder based frameworks 7 – 11 have been applied to generate textual reports from medical images.”
InfluentialHe, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.102963 · Cited by 147 · see citing works
Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generation
Miura, Y., Zhang, Y., Tsai, E., Langlotz, C., & Jurafsky, D. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.416 · Cited by 149 · see citing works
You, D., Liu, F., Ge, S., Xie, X., Zhang, J., & Wu, X. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87199-4_7 · Cited by 170 · see citing works
“However, due to the data bias and the very long sequence, some errors occur in the generated reports of the existing methods, like duplicate reports, inexact descriptions, etc [42,44,17].”
BackgroundInfluentialAuxiliary signal-guided knowledge encoder-decoder for medical report generation
Li, M., Liu, R., Wang, F., Chang, X., & Liang, X. (2022) · World Wide Web · doi:10.1007/s11280-022-01013-6 · Cited by 149 · see citing works
Automated Radiographic Report Generation Purely on Transformer: A Multicriteria Supervised Approach
Wang, Z., Han, H., Wang, L., Li, X., & Zhou, L. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3171661 · Cited by 112 · see citing works
“Following [12], [17], 10% of reports are randomly picked to form the test set.”
BackgroundMethodResultInfluentialA Review on Machine Learning Strategies for Real-World Engineering Applications
Jhaveri, R. H., Revathi, A. R., Ramana, K., Raut, R., & Dhanaraj, R. K. (2022) · Mobile Information Systems · doi:10.1155/2022/1833507 · Cited by 124 · see citing works
Explainable AI for Medical Data: Current Methods, Limitations, and Future Directions
Hossain, M. I., Zamzmi, G., Mouton, P. R., Salekin, M. S., Sun, Y., & Goldgof, D. B. (2023) · ACM Computing Surveys · doi:10.1145/3637487 · Cited by 120 · see citing works
Medical image captioning via generative pretrained transformers
Selivanov, A., Rogov, O. Y., Chesakov, D., Shelmanov, A., Fedulova, I., & Dylov, D. V. (2023) · Scientific Reports · doi:10.1038/s41598-023-31223-5 · Cited by 119 · see citing works
Automatic captioning for medical imaging (MIC): a rapid review of literature
Beddiar, D. R., Oussalah, M., & Seppänen, T. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10270-w · Cited by 103 · see citing works
“Example from the IU X-Ray dataset, retrieved from (Jing et al. 2017)”
MethodDeep Neural Architectures for Medical Image Semantic Segmentation: Review
Khan, M. Z., Gajendran, M. K., Lee, Y., & Khan, M. A. (2021) · IEEE Access · doi:10.1109/access.2021.3086530 · Cited by 101 · see citing works
“The prominent deep learning applications for medical image analysis includes image construction [21], enhancement [22], [23], segmentation [24], registration [25], computer-aided detection [26], view detection [27], landmark detection [28], and automated report generation [29].”
BackgroundAutomatic Generation of Medical Imaging Diagnostic Report with Hierarchical Recurrent Neural Network
Yin, C., Qian, B., Wei, J., Li, X., Zhang, X., Li, Y., & Zheng, Q. (2019) · Proceedings · doi:10.1109/icdm.2019.00083 · Cited by 116 · see citing works
“[8] propose to leverage CNN to detect tags, and then combine the hierarchical recurrent network and attention mechanism to generate detailed medical reports.”
BackgroundMethodCross-Modal Prototype Driven Network for Radiology Report Generation
Wang, J., Bhalerao, A. H., & He, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19833-5_33 · Cited by 108 · see citing works
Clinical-BERT: Vision-Language Pre-training for Radiograph Diagnosis and Reports Generation
Yan, B., & Pei, M. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20204 · Cited by 94 · see citing works
Introduction to Artificial Intelligence and Machine Learning for Pathology
Harrison, J. H., Gilbertson, J. R., Hanna, M. G., Olson, N. H., Seheult, J. N., Sorace, J. M., & Stram, M. N. (2021) · Archives of Pathology & Laboratory Medicine · doi:10.5858/arpa.2020-0541-cp · Cited by 83 · see citing works
“Recurrent neural networks have been used for interpretation, translation, and creation of speech and text, including automated report generation in radiology.(88) They are also capable of time series analysis and are useful for anomaly detection in quality control data.”
MethodA Self-boosting Framework for Automated Radiographic Report Generation
Wang, Z., Zhou, L., Wang, L., & Li, X. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00246 · Cited by 92 · see citing works
“Jing et al [10] proposed a multi-task hierarchical model with co-attention to automatically predict keywords and generate long paragraphs.”
BackgroundMethodResultInfluentialHasani, A. M., Singh, S. M., Zahergivar, A., Ryan, B., Nethala, D., Bravomontenegro, G., Mendhiratta, N., Ball, M. W., Farhadi, F., & Malayeri, A. A. (2023) · European Radiology · doi:10.1007/s00330-023-10384-x · Cited by 82 · see citing works
“Studies by Jing et al [15] and Alfarghaly et al [9] achieved BLEU scores of 0.247 and 0.111, respectively.”
InfluentialAutomated Radiology Report Generation: A Review of Recent Advances
Sloan, P., Clatworthy, P. L., Simpson, E., & Mirmehdi, M. (2024) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2024.3408456 · Cited by 98 · see citing works
“In 2018, research into ARRG started gaining popularity [43], [44], however the generated reportsfromtheseCNN-RNNmethodologiesareoverlyrigid[9] andoftenrepeatphrasesfromthetrainingsetatinappropriate times,leadingtosummariesthatappearlesshuman-like.”
Liu, G., Liao, Y., Wang, F., Zhang, B., Zhang, L., Liang, X., Wan, X., Li, S., Li, Z., Zhang, S., et al. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3099165 · Cited by 79 · see citing works
“Models CIDEr-D ROUGE-L BLEU@1 BLEU@2 BLEU@3 BLEU@4 CoAtt [2] 273.”
BackgroundMethodInfluentialReinforced Transformer for Medical Image Captioning
Xiong, Y., Du, B., & Yan, P. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32692-0_77 · Cited by 94 · see citing works
A survey on automatic generation of medical imaging reports based on deep learning
Pang, T., Li, P., & Zhao, L. (2023) · BioMedical Engineering OnLine · doi:10.1186/s12938-023-01113-y · Cited by 80 · see citing works
“2017 [34] IU X-Ray (7470, 7470) CNN–HLSTM– ATT 0.”
Deep image captioning: A review of methods, trends and future challenges
Xu, L., Tang, Q., Lv, J., Zheng, B., Zeng, X., & Li, W. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126287 · Cited by 78 · see citing works
Joint Embedding of Deep Visual and Semantic Features for Medical Image Report Generation
Yang, Y. Q., Yu, J., Zhang, J., Han, W., Jiang, H., & Huang, Q. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3122542 · Cited by 73 · see citing works
“Inspiredbythegreatsuccessofimagecaption-ing,mostexistingMeIRGmethodsfollowtheencoder-decoder pipeline[6]–[8]togenerateunstructuredfree-textreports.”
BackgroundMethodInfluentialGeneration and evaluation of artificial mental health records for Natural Language Processing
Ive, J., Viani, N., Kam, J., Yin, L., Verma, S., Puntis, S., Cardinal, R. N., Roberts, A., Stewart, R. J., & Velupillai, S. U. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0267-x · Cited by 73 · see citing works
“For instance, Jing et al. 4 tackle the generation of medical imaging reports (up to 50 words), using a hierarchical recurrent neural network decoder.”
BackgroundMethodHe, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2024) · doi:10.2139/ssrn.4809363 · Cited by 69 · see citing works
Dyrba, M., Hanzig, M., Altenstein, S., Bader, S., Ballarini, T., Brosseron, F., Büerger, K., Cantré, D., Dechent, P., Dobisch, L., et al. (2021) · Alzheimer s Research & Therapy · doi:10.1186/s13195-021-00924-2 · Cited by 66 · see citing works
“Another aspect is the automated generation of textual descriptions and diagnostic explanations from images (Jing et al., 2018; Lucieri et al., 2020; Zhang et al., 2017).”
BackgroundDeepOpht: Medical Report Generation for Retinal Images via Deep Models and Visual Explanation
Huang, J.-H., Yang, C.-H. H., Liu, F., Tian, M., Liu, Y.-C., Wu, T.-W., Lin, I.-H., Wang, K., Morikawa, H., Chang, H.-H., et al. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00249 · Cited by 65 · see citing works
“[30]” also has the best performance among all the models.”
BackgroundMethodInfluentialHas multimodal learning delivered universal intelligence in healthcare? A comprehensive survey
Lin, Q., Zhu, Y., Mei, X., Huang, L., Ma, J., He, K., Peng, Z., Cambria, E., & Feng, M. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102795 · Cited by 76 · see citing works
Huang, X., Yan, F., Xu, W., & Li, M. (2019) · IEEE Access · doi:10.1109/access.2019.2947134 · Cited by 65 · see citing works
“According to Vinyals’s[10] method, the first and second inputs of the Word RNN are topic vectors and a special <START> token.”
BackgroundMethodResultInfluentialA Survey on Attention Mechanisms for Medical Applications: are we Moving Toward Better Algorithms?
Gonçalves, T., Rio-Torto, I., Teixeira, L. F., & Cardoso, J. S. (2022) · IEEE Access · doi:10.1109/access.2022.3206449 · Cited by 81 · see citing works
A Medical Semantic-Assisted Transformer for Radiographic Report Generation
Wang, Z., Tang, M., Wang, L., Li, X., & Zhou, L. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16437-8_63 · Cited by 81 · see citing works
CAMANet: Class Activation Map Guided Attention Network for Radiology Report Generation
Wang, J., Bhalerao, A. H., Yin, T., See, S. C. W., & He, Y. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3354712 · Cited by 66 · see citing works
“For example, Jing et al. [17] proposed a hierarchical LSTM model to mitigate the long sentence generation problem and developed a co-attention mechanism to localize abnormal regions.”
Secure and robust machine learning for healthcare: A survey
Qayyum, A., Qadir, J., Bilal, M., & Al-Fuqaha, A. (2021) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2020.3013489 · Cited by 59 · see citing works
“A multi-task ML based framework is proposed for automatic tagging and description of medical images [63].”
BackgroundSemi-Supervised Medical Report Generation via Graph-Guided Hybrid Feature Consistency
Zhang, K., Jiang, H., Zhang, J., Huang, Q., Jianping Fan, B., Yu, J., & Han, W. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3273390 · Cited by 62 · see citing works
“A hierarchical model with Co-Attention [18] was also presented to generate reports.”
MethodCross-modal Clinical Graph Transformer for Ophthalmic Report Generation
Li, M., Cai, W., Verspoor, K., Pan, S., Liang, X., & Chang, X. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.02000 · Cited by 60 · see citing works
“Inspired by work on image captioning, researchers have adopted a hierarchical recurrent network (HRNN) to describe medical images at the beginning [16,40].”
TSGET: Two-Stage Global Enhanced Transformer for Automatic Radiology Report Generation
Yi, X., Fu, Y., Liu, R., Zhang, H., & Hua, R. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3350077 · Cited by 61 · see citing works
Contrast-Attentive Thoracic Disease Recognition With Dual-Weighting Graph Reasoning
Zhou, Y., Zhou, T., Zhou, T., Fu, H., Liu, J., & Shao, L. (2021) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2021.3049498 · Cited by 54 · see citing works
“Besides, the automatic generation of medical image reports has also been investigated; for example, [23] exploited a co-attention mechanism to identify abnormal regions and used a hierarchical LSTM to generate sentences.”
MethodMemory Guided Transformer With Spatio-Semantic Visual Extractor for Medical Report Generation
Divya, P., Yenduri, S., Vishnu, C., Mohan, C. K., & Chen, Y. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3371894 · Cited by 61 · see citing works
“We compare our proposed methodology with other methods like ST [2], ATT2IN [58], ADAATT [6], TOPDOWN [7], and models designed specifically for the medical field, such as COATT [5], HRGR [44], CMAS-RL [43], R2Gen [9], CMN [53], and PPKED [10].”
BackgroundMethodInfluentialChest X-Ray Report Generation Through Fine-Grained Label Learning
Syeda-Mahmood, T. F., Wong, K. C. L., Gur, Y., Wu, J. T.-Y., Jadhav, A., Kashyap, S., Karargyris, A., Pillai, A., Sharma, A., Syed, A., et al. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-59713-9_54 · Cited by 57 · see citing works
“Although we have tested our algorithm for very large number of images from the combined MIMIC-NIH data, for purposes of comparison, we show the results on the same Indiana test dataset that has been used most commonly by other algorithms as reported in [10].”
MethodResultJoint Modeling of Chest Radiographs and Radiology Reports for Pulmonary Edema Assessment
Chauhan, G., Liao, R., Wells, W. M., Andreas, J., Wang, X., Berkowitz, S. J., Horng, S., Szolovits, P., & Golland, P. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-59713-9_51 · Cited by 59 · see citing works
“This is in contrast to visual question answering [2,3,19], where inference is performed on an image-text pair, and image/video captioning [16,22,24,28], where the model generates text from the input image.”
BackgroundPark, H., Kim, K. M., Park, S., & Choi, J. (2021) · IEEE Access · doi:10.1109/access.2021.3124564 · Cited by 52 · see citing works
“The model developed by Jing et al. [28] applies VGG-19 as a CNN to extract a visual feature vector from an image and uses a multi-layer perceptron for tag classi-fication.”
MethodA Skin Cancer Classification Approach using GAN and RoI-Based Attention Mechanism
Teodoro, A. A. M., Silva, D. H., Rosa, R. L., et al. (2022) · Journal of Signal Processing Systems · doi:10.1007/s11265-022-01757-4 · Cited by 50 · see citing works
“In [54], a co-attention mechanism was proposed to locate regions that contain abnormalities in medical images of radiology and pathology.”
BackgroundImproving Medical Vision-Language Contrastive Pretraining With Semantics-Aware Triage
Liu, B., Lu, D., Wei, D., Wu, X., Wang, Y., Zhang, Y., & Zheng, Y. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3294980 · Cited by 52 · see citing works
“Although several efforts [22], [41] have been made to apply the models pretrained on large natural image datasets such as ImageNet [35] to medical domain, the effectiveness of such transfer learning is often unsatisfactory [33] due to the large difference between medical and natural images.”
BackgroundSODA: Detecting COVID-19 in Chest X-Rays With Semi-Supervised Open Set Domain Adaptation
Zhou, J., Jing, B., Wang, Z., Xin, H., & Tong, H. (2021) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2021.3066331 · Cited by 49 · see citing works
“The problems of disease detection [4], [5], [6] and report generation using chest x-rays [7], [8], [9], [10] are investigated recently and have achieved much-improved results upon recently.”
BackgroundSelf-citationCT2Rep: Automated Radiology Report Generation for 3D Medical Imaging
Hamamci, I. E., Er, S., & Menze, B. (2024) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-72390-2_45 · Cited by 47 · see citing works
Blagec, K., Kraiger, J., Frühwirt, W., & Samwald, M. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104274 · Cited by 47 · see citing works
MKCL: Medical Knowledge with Contrastive Learning model for radiology report generation
Hou, X., Liu, Z., Li, X., Li, X., Sang, S., & Zhang, Y. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104496 · Cited by 47 · see citing works
Cai, J., Lü, L., Harrison, A. P., et al. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-00934-2_66 · Cited by 44 · see citing works
“Other work investigate methods to automatically generate X-ray reports [4,10].”
BackgroundMethodKaur, N., Mittal, A., & Singh, G. (2021) · Multimedia Tools and Applications · doi:10.1007/s11042-021-11272-6 · Cited by 43 · see citing works
“Jing et al. [31] used the co-attention model (see Fig.”
BackgroundMethodUnifying Relational Sentence Generation and Retrieval for Medical Image Report Composition
Wang, F., Liang, X., Xu, L. H., & Lin, L. (2020) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2020.3026098 · Cited by 41 · see citing works
“Due to relatively rare and remarkably diverse abnormal findings, [2] fails to detect abnormalities and tends to generate trivial descriptions.”
InfluentialPrior Guided Transformer for Accurate Radiology Reports Generation
Yan, B., Pei, M., Zhao, M., Shan, C., & Tian, Z. (2022) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2022.3197162 · Cited by 42 · see citing works
“Recently, many methods are proposed to solve the automatic radiology report generation problem [1], [2], [3], [4].”
MethodNatural Language Processing in Pathology
Úbeda, P. L., Martín‐Noguerol, T., Aneiros‐Fernández, J., & Luna, A. (2022) · American Journal Of Pathology · doi:10.1016/j.ajpath.2022.07.012 · Cited by 41 · see citing works
“The American Journal of Pathology - ajp.amjpathol.org model on two publicly available data sets, one containing pathologic images [Pathology Education Informational Resource (PEIR) gross data set] 73 and one composed of radiological images.”
MethodAI musculoskeletal clinical applications: how can AI increase my day-to-day efficiency?
Shin, Y., Kim, S., & Lee, Y. (2021) · Skeletal Radiology · doi:10.1007/s00256-021-03876-8 · Cited by 36 · see citing works
“Regarding inter-modal connections between language and visuals, radiologic images are converted to radiologic information-containing text descriptions (i.e., radiologic reports) through DL [39, 40].”
BackgroundLin, Z., Zhang, D., Shi, D., Xu, R., Tao, Q., Wu, L., He, M., & Ge, Z. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104281 · Cited by 39 · see citing works
Appositeness of Optimized and Reliable Machine Learning for Healthcare: A Survey
Swain, S., Bhushan, B., Dhiman, G., & Viriyasitavat, W. (2022) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-022-09733-8 · Cited by 38 · see citing works
Towards Visual Dialog for Radiology
Kovaleva, O., Shivade, C., Kashyap, S., Kanjaria, K., Wu, J., Ballah, D., Coy, A., Karargyris, A., Guo, Y., Beymer, D. B., et al. (2020) · Proceedings of the 19th SIGBioMed Workshop on Biomedical Language Processing · doi:10.18653/v1/2020.bionlp-1.6 · Cited by 38 · see citing works
Unify, Align and Refine: Multi-Level Semantic Alignment for Radiology Report Generation
Li, Y., Yang, B., Cheng, X., Zhu, Z., Li, H., & Zou, Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00268 · Cited by 43 · see citing works
“We compare our UAR model with state-of-the-art (SOTA) methods of different backbones, including CNN-RNN [48, 33, 1, 18] and CNN-Transformer [8, 7, 29, 56].”
BackgroundMethodInfluentialTextRay: Mining Clinical Reports to Gain a Broad Understanding of Chest X-Rays
Laserson, J., Lantsman, C. D., Cohen-Sfady, M., et al. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-00934-2_62 · Cited by 36 · see citing works
“he ability to achieve expert-level prediction for the presence of pneumonia after training upon a DenseNet121 CNN. Utilizing several public datasets with image labels and reports provided, Jing et al.[10] built a system that can generate a natural appearing radiology report using a hierarchical RNN. The high-level RNN…”
BackgroundA multimodal multidomain multilingual medical foundation model for zero shot clinical diagnosis
Liu, F., Zheng, L., Yin, Q., Huang, J., Luo, J., Thakur, A., Branson, K., Schwab, P., Yin, B., Wu, X., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-024-01339-7 · Cited by 43 · see citing works
Attributed Abnormality Graph Embedding for Clinically Accurate X-Ray Report Generation
Yan, S., Cheung, W. K., Chiu, K. W.-H., Tong, T. M., Cheung, K. C., & See, S. C. W. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3245608 · Cited by 38 · see citing works
From Observation to Concept: A Flexible Multi-View Paradigm for Medical Report Generation
Liu, Z., Zhu, Z., Zheng, S., Zhao, Y., He, K., & Zhao, Y. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3342691 · Cited by 38 · see citing works
“…the data bias and long report problems in MRG, a number of efforts have been made, by designing spatial attention for improving the representation of visual features [12], [15], [16], introducing extra medical knowledge [14], [17], [18] or developing memory mechanism on decoder for generation [19].”
BackgroundMethodInfluentialDiagnostic captioning: a survey
Pavlopoulos, J., Kougia, V., Androutsopoulos, I., & Papamichail, D. (2022) · Knowledge and Information Systems · doi:10.1007/s10115-022-01684-7 · Cited by 37 · see citing works
“3) that is used to generate diagnostic captions, both images (Li et al., 2018; Wang et al., 2018; Gale et al., 2018; Xue et al., 2018; Liu et al., 2019a; Huang et al., 2019) and images combined with text (Shin et al., 2016b; Jing et al., 2018; Yin et al., 2019; Yuan et al., 2019) have been explored.”
BackgroundMethodResultInfluentialAutomatic Generation of Chest X-Ray Medical Imaging Reports using LSTM-CNN
Tiwari, V., Bapat, K., Shrimali, K. R., et al. (2021) · doi:10.1145/3484824.3484918 · Cited by 32 · see citing works
“Multiview CNN and a word and sentence level LSTM [11] comprise the framework.”
MethodUtilizing Longitudinal Chest X-Rays and Reports to Pre-fill Radiology Reports
Zhu, Q., Mathai, T. S., Mukherjee, P., Peng, Y., Summers, R. M., & Lu, Z. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43904-9_19 · Cited by 38 · see citing works
“A number of methods to generate radiology reports have been proposed previously with a significant focus on CXR images [3,4,5,6,7,8,9,10].”
Generative Models in Medical Visual Question Answering: A Survey
Dong, W., Shen, S., Han, Y., Tao, T., Wu, J., & Xu, H. (2025) · Applied Sciences · doi:10.3390/app15062983 · Cited by 35 · see citing works
Caption-Aware Medical VQA via Semantic Focusing and Progressive Cross-Modality Comprehension
Cong, F., Xu, S., Guo, L., & Tian, Y. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548122 · Cited by 33 · see citing works
“In this work, similar to [13], we employ the Show Attend and Tell model [39] to generate the caption y which is encoded as a sequence of 1-of-K words for the images in the medical VQA task.”
MethodA Novel Deep Learning Model for Medical Report Generation by Inter-Intra Information Calibration
Zhang, J., Shen, X., Wan, S., Goudos, S. K., Wu, J., Cheng, M., & Zhang, W. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3236661 · Cited by 32 · see citing works
“From top tobottominTableIII,CoAtt[2]isaclassicframeworkinthefield ofautomaticgenerationofmedicalimagereports,whichincor-poratesco-attentionintoahierarchicalLSTMnetwork.”
Mapping medical image-text to a joint space via masked modeling
Chen, Z., Du, Y., Hu, J., Liu, Y., Li, G., Wan, X., & Chang, T. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.103018 · Cited by 30 · see citing works
Memory-Based Cross-Modal Semantic Alignment Network for Radiology Report Generation
Tao, Y., Ma, L., Yu, J., & Zhang, H. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3393018 · Cited by 31 · see citing works
“…[38] and IU-Xray [39] datasets between our method and previous state-of-the-art methods, including CNN-RNN based model (i.e., ADAATT [45], COATT [46], ATT2IN [47], CMAS [48]), CNN-Transformer models (i.e., R2Gen [21], Self-boost [20], JPG [49], PGT [50]), and methods using techniques such as…”
MethodHybrid Reinforced Medical Report Generation With M-Linear Attention and Repetition Penalty
Xu, Z., Xu, W., Wang, R., Chen, J., Chang, Q., & Lukasiewicz, T. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3343391 · Cited by 32 · see citing works
“To attend to visual and semantic domains simultaneously, [2] proposes a co-attention mechanism that builds attention distributions separately for visual and semantic domains and ignores multimodal interactions while repetitions tend to be found in the generated reports for the ignorance of contextual coherence.”
BackgroundMethodInfluentialWsiCaption: Multiple Instance Generation of Pathology Reports for Gigapixel Whole-Slide Images
Chen, P., Li, H., Zhu, C., Zheng, S., Shui, Z., & Yang, L. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72083-3_51 · Cited by 35 · see citing works
“And in the medical area, the generation of radiology reports has been explored by several works [14,7,18].”
BackgroundImproved Disease Classification in Chest X-Rays with Transferred Features from Report Generation
Xue, Y., & Huang, X. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-20351-1_10 · Cited by 29 · see citing works
“To provide some insights to the performance of our report generator, we report BLEU [18], METEOR [5], ROUGE [16] and CIDEr [21] scores as for image captioning tasks and in previous works [11, 14, 26].”
BackgroundMethodInfluentialHierarchical X-Ray Report Generation via Pathology Tags and Multi Head Attention
Srinivasan, P., Thapar, D., Bhavsar, A. V., & Nigam, A. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-69541-5_36 · Cited by 29 · see citing works
“To aid radiologists, many researchers are investigating the generation of automatic reports from X-Ray images [2,3] by formulating the problem as image captioning [4].”
BackgroundShetty, S., Ananthanarayana, V. S., & Mahale, A. R. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-14940-x · Cited by 30 · see citing works
“The majority of the existing work on the Indiana University dataset focuses on cross-modal retrieval of radiology reports given CXR as input [35, 45, 73].”
BackgroundLeveraging Summary Guidance on Medical Report Summarization
Zhu, Y., Yang, X., Wu, Y., & Zhang, W. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3304376 · Cited by 33 · see citing works
“In recent years, there has been a growing interest in the research of free-form natural language generation in the medical domain, including medical text generation and medical image captioning [7]–[12].”
BackgroundVisual-Textual Attentive Semantic Consistency for Medical Report Generation
Zhou, Y., Huang, L., Zhou, T., Fu, H., & Shao, L. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00395 · Cited by 32 · see citing works
“[20] presented a basic co-attention model to implement automatic report generation without enough thoughtful design.”
BackgroundMethodAttention-Based Abnormal-Aware Fusion Network for Radiology Report Generation
Xie, X., Xiong, Y., Yu, P. S., et al. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-18590-9_64 · Cited by 28 · see citing works
Shentu, J., & Moubayed, N. A. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00513 · Cited by 29 · see citing works
“[17] leveraged a CNN-RNN structure with a hierarchical LSTM [22] being the text decoder to generate corresponding descriptions and localize sub-regions.”
MethodEfficient Evolving Deep Ensemble Medical Image Captioning Network
Singh, D., Kaur, M., Alanazi, J. M., AlZubi, A. A., & Lee, H.-N. (2022) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2022.3223181 · Cited by 28 · see citing works
“Jing et al. [32] designed a multitask learning model that predicted tags and paragraphs simultaneously.”
BackgroundGao, D., Kong, M. C., Zhao, Y., Huang, J.-Z. J., Huang, Z., Kuang, K., Wu, F. Q., & Zhu, Q. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102982 · Cited by 28 · see citing works
Medical report generation based on multimodal federated learning
Chen, J., & Pan, R. (2024) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2024.102342 · Cited by 29 · see citing works
Tripathi, S., Augustin, A. I., Sukumaran, R., Dheer, S., & Kim, E. (2022) · Artificial Intelligence in the Life Sciences · doi:10.1016/j.ailsci.2022.100043 · Cited by 30 · see citing works
- Authors
- Jing, B., Xie, P., & Xing, E.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2018
- Pages
- 2577-2586
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2770165365
- Cited by
- 657
- References
- 28
- Citations by year
- 2026: 57 · 2025: 123 · 2024: 115 · 2023: 111 · 2022: 80 · 2021: 81 · 2020: 45 · 2019: 34 · 2018: 11
- FWCI
- 12.69
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Keywords
- multi-task learning, co-attention, hierarchical LSTM, tag prediction, clinical report writing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
