Cross-modal Memory Networks for Radiology Report Generation
Open accessAbstract
Zhihong Chen, Yaling Shen, Yan Song, Xiang Wan. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The paper's system using NLL training performs similarly on NLG metrics (better in BLEU but worse in ROUGE-L compared to Chen et al. 2021) but outperforms previous works on factually-oriented metrics.
“We can see that our simple approach, referred to as”
From Improving the Factual Correctness of Radiology Report Generation with Semantic Rewards (2022) · cites this paper as Chen et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · see citing works
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · see citing works
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · see citing works
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318 · cited 5× · see citing works
“…studies adopt a conventional encoder-decoder architecture, with convolutional neural networks (CNNs) as the encoder and recurrent (e.g., LSTM/GRU) or non-recurrent networks (e.g., Transformer) as the decoder following the image captioning paradigm (Vinyals et al., 2015; Anderson et al., 2018).”
BackgroundMethodInfluentialGenerating Radiology Reports via Memory-driven Transformer
Chen, Z., Song, Y., Chang, T.-H., & Wan, X. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.112 · Cited by 693 · see citing works
Self-citationHybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation
Li, C. Y., Liang, X., Hu, Z., & Xing, E. P. (2018) · arXiv · doi:10.48550/arxiv.1805.08298 · Cited by 271 · see citing works
Flagged first, then by citations
Research works printed but not linked: 38
- Emily Alsentzer, John Murphy, William Boag, Wei- Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. 2019. Publicly Available Clinical BERT Embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Work- shop, pages 72-78.
- Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answer- ing. In Proceedings of the IEEE conference on com- puter vision and pattern recognition, pages 6077- 6086.
- Guimin Chen, Yuanhe Tian, Yan Song, and Xiang Wan. 2021. Relation Extraction with Type-aware Map Memories of Word Dependencies. In Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021.
- Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara. 2019. M 2 : Meshed-Memory Transformer for Image Captioning. arXiv preprint arXiv:1912.08226. · arXiv
- Dina Demner-Fushman, Marc D Kohli, Marc B Rosen- man, Sonya E Shooshan, Laritza Rodriguez, Sameer Antani, George R Thoma, and Clement J McDon- ald. 2016. Preparing a collection of radiology ex- aminations for distribution and retrieval. Journal of the American Medical Informatics Association, 23(2):304-310.
- Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A large-scale hier- archical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255.
- Michael Denkowski and Alon Lavie. 2011. Meteor 1.3: Automatic Metric for Reliable Optimization and Evaluation of Machine Translation Systems. In Pro- ceedings of the sixth workshop on statistical ma- chine translation, pages 85-91.
- Shizhe Diao, Yan Song, and Tong Zhang. 2020. Keyphrase Generation with Cross-Document Atten- tion. arXiv preprint arXiv:2004.09800. · arXiv
- Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Mark- lund, Behzad Haghgoo, Robyn Ball, Katie Shpan- skaya, et al. 2019. CheXpert: A Large Chest Ra- diograph Dataset with Uncertainty Labels and Ex- pert Comparison. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 33, pages 590-597.
- Baoyu Jing, Zeya Wang, and Eric Xing. 2019. Show, Describe and Conclude: On Exploiting the Structure Information of Chest X-ray Reports. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6570-6580.
- Baoyu Jing, Pengtao Xie, and Eric Xing. 2018. On the Automatic Generation of Medical Imaging Reports. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2577-2586.
- Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew P Lungren, Chih- ying Deng, Roger G Mark, and Steven Horng. 2019. MIMIC-CXR: A large publicly available database of labeled chest radiographs. arXiv preprint arXiv:1901.07042. · arXiv
- Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. BioBERT: a pre- trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234-1240.
- Chin-Yew Lin. 2004. ROUGE: A Package for Auto- matic Evaluation of Summaries. In Text Summariza- tion Branches Out, pages 74-81.
- Guanxiong Liu, Tzu-Ming Harry Hsu, Matthew Mc- Dermott, Willie Boag, Wei-Hung Weng, Peter Szolovits, and Marzyeh Ghassemi. 2019. Clinically Accurate Chest X-Ray Report Generation. In Ma- chine Learning for Healthcare Conference, pages 249-269.
- Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher. 2017. Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Caption- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 375- 383.
- Yuyang Nie, Yuanhe Tian, Yan Song, Xiang Ao, and Xiang Wan. 2020. Improving Named Entity Recog- nition with Attentive Ensemble of Syntactic Infor- mation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 4231-4245.
- Anusri Pampari, Preethi Raghavan, Jennifer Liang, and Jian Peng. 2018. emrQA: A Large Corpus for Ques- tion Answering on Electronic Medical Records. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2357-2368.
- Jeffrey Pennington, Richard Socher, and Christo- pher Manning. 2014. GloVe: Global Vectors for Word Representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar.
- Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep Contextualized Word Rep- resentations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227- 2237, New Orleans, Louisiana.
- Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. 2017. Self-critical Sequence Training for Image Captioning. In Pro- ceedings of the IEEE Conference on Computer Vi- sion and Pattern Recognition, pages 7008-7024.
- Adam Santoro, Ryan Faulkner, David Raposo, Jack Rae, Mike Chrzanowski, Theophane Weber, Daan Wierstra, Oriol Vinyals, Razvan Pascanu, and Timo- thy Lillicrap. 2018. Relational recurrent neural net- works. In Advances in neural information process- ing systems, pages 7299-7310.
- Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Im- age Recognition. CoRR, abs/1409.1556. · arXiv
- Yan Song, Chia-Jung Lee, and Fei Xia. 2017. Learn- ing Word Representations with Regularization from Prior Knowledge. In Proceedings of the 21st Confer- ence on Computational Natural Language Learning (CoNLL 2017), pages 143-152.
- Yan Song and Shuming Shi. 2018. Complementary Learning of Word Embeddings. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, pages 4368- 4374.
- Yan Song, Shuming Shi, Jing Li, and Haisong Zhang. 2018. Directional Skip-Gram: Explicitly Distin- guishing Left and Right Context for Word Embed- dings. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, Volume 2 (Short Papers), pages 175-180.
- Yan Song, Yuanhe Tian, Nan Wang, and Fei Xia. 2020. Summarizing Medical Conversations via Identifying Important Utterances. In Proceedings of the 28th International Conference on Computational Linguis- tics, pages 717-729.
- Yan Song, Tong Zhang, Yonggang Wang, and Kai-Fu Lee. 2021. ZEN 2.0: Continue Training and Adap- tion for N-gram Enhanced Text Encoders. arXiv preprint arXiv:2105.01279. · arXiv
- Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. 2015. End-To-End Memory Networks. In Ad- vances in neural information processing systems, pages 2440-2448.
- Yuanhe Tian, Guimin Chen, and Yan Song. 2021. En- hancing Aspect-level Sentiment Analysis with Word Dependencies. In Proceedings of the 16th Confer- ence of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3726-3739, Online.
- Yuanhe Tian, Weicheng Ma, Fei Xia, and Yan Song. 2019. ChiMed: A Chinese Medical Corpus for Question Answering. In Proceedings of the 18th BioNLP Workshop and Shared Task, pages 250-260.
- Yuanhe Tian, Yan Song, Fei Xia, Tong Zhang, and Yonggang Wang. 2020b. Improving Chinese Word Segmentation with Wordhood Memory Networks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8274-8285.
- Nan Wang, Yan Song, and Fei Xia. 2018. Coding Struc- tures and Actions with the COSTA Scheme in Med- ical Conversations. In Proceedings of the BioNLP 2018 workshop, pages 76-86.
- Nan Wang, Yan Song, and Fei Xia. 2020. Studying Challenges in Medical Conversation with Structured Annotation. In Proceedings of the First Workshop on Natural Language Processing for Medical Con- versations, pages 12-21, Online.
- Weixuan Wang, Zhihong Chen, and Haifeng Hu. 2019. Hierarchical Attention Network for Image Caption- ing. In Proceedings of the AAAI Conference on Arti- ficial Intelligence, volume 33, pages 8957-8964.
- Jason Weston, Sumit Chopra, and Antoine Bordes. 2015. Memory Networks. CoRR, abs/1410.3916. · arXiv
- Jichuan Zeng, Jing Li, Yan Song, Cuiyun Gao, Michael R Lyu, and Irwin King. 2018. Topic Mem- ory Networks for Short Text Classification. In Pro- ceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing, pages 3120- 3131.
- Yuhao Zhang, Daisy Yi Ding, Tianpei Qian, Christo- pher D Manning, and Curtis P Langlotz. 2018. Learning to Summarize Radiology Findings. In Pro- ceedings of the Ninth International Workshop on Health Text Mining and Information Analysis, pages 204-213.
Cited by
Sorted by citations
Dynamic Graph Enhanced Contrastive Learning for Chest X-Ray Report Generation
Li, M., Lin, B., Chen, Z., Lin, H., Liang, X., & Chang, X. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00325 · Cited by 206 · see citing works
Self-citationMETransformer: Radiology Report Generation by Transformer with Multiple Learnable Expert Tokens
Wang, Z., Liu, L., Wang, L., & Zhou, L. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01112 · Cited by 184 · see citing works
Improving chest X-ray report generation by leveraging warm starting
Nicolson, A., Dowling, J., & Koopman, B. (2023) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2023.102633 · Cited by 161 · see citing works
Clinical-BERT: Vision-Language Pre-training for Radiograph Diagnosis and Reports Generation
Yan, B., & Pei, M. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20204 · Cited by 94 · see citing works
Multi-task paired masking with alignment modeling for medical vision-language pre-training
Zhang, K., Yang, Y., Yu, J., Jiang, H., Fan, J., Huang, Q., & Han, W. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3325965 · Cited by 51 · see citing works
Image captioning by diffusion models: a survey
Daneshfar, F., Bartani, A., & Lotfi, P. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109288 · Cited by 49 · see citing works
ICGA-GPT: report generation and question answering for indocyanine green angiography images
Chen, X., Zhang, W., Zhao, Z., Xu, P., Zheng, Y., Shi, D., & He, M. (2024) · British Journal of Ophthalmology · doi:10.1136/bjo-2023-324446 · Cited by 36 · see citing works
Deep learning for report generation on chest X-ray images
Ouis, M. Y., & A. Akhloufi, M. (2024) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2023.102320 · Cited by 25 · see citing works
Dalla Serra, F., Clackett, W., MacKinnon, H., Wang, C., Deligianni, F., Dalton, J., & O’Neil, A. Q. (2022) · Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2022.aacl-main.47 · Cited by 12 · see citing works
Li, Y., Kong, C., Zhao, G., & Zhao, Z. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11337-0 · Cited by 7 · see citing works
Improving the Factual Correctness of Radiology Report Generation with Semantic Rewards
Delbrouck, J.-B., Chambon, P., Bluethgen, C., Tsai, E., Almusa, O., & Langlotz, C. P. (2022) · arXiv · doi:10.48550/arxiv.2210.12186 · Cited by 63 · see citing works
Allaberdiev, S., Chen, X., Mamatov, D., Abdusalomov, A., Ubaydullaev, U., & Khan, A. (2026) · IET Software · doi:10.1049/sfw2/5529107 · Cited by 1 · see citing works
Abdaoui, H., Barbaria, S., Al Kuwaiti, A., Raouafi, N., Ben Salah, R., & Boussi Rahmouni, H. (2026) · Frontiers in Radiology · doi:10.3389/fradi.2026.1875905
- Authors
- Chen, Z., Shen, Y., Song, Y., & Wan, X.
- Affiliations
- Shenzhen Research Institute of Big Data (CN)
- Chinese University of Hong Kong, Shenzhen (CN)
- Venue
- Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
- Published
- 2021
- Pages
- 5904-5914
- OpenAlex
- W3173688449
- Cited by
- 369
- References
- 9
- Citations by year
- 2026: 69 · 2025: 131 · 2024: 99 · 2023: 53 · 2022: 16 · 2021: 1
- FWCI
- 13.55
- Type
- conference paper
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Funders
- National Natural Science Foundation of China (12026610)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- radiology report generation, medical report summarization, multimodal memory, clinical text generation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
