Multimodal Generation of Radiology Reports using Knowledge-Grounded Extraction of Entities and Relations
Open accessAbstract
Francesco Dalla Serra, William Clackett, Hamish MacKinnon, Chaoyang Wang, Fani Deligianni, Jeff Dalton, Alison Q. O’Neil. Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2022.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 28
- Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 6077-6086.
- Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved cor- relation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summariza- tion, pages 65-72.
- Care Quality Commission. 2018. A national review of radiology reporting within the nhs in england. pages 1-26.
- Kyunghyun Cho, Bart Van Merriënboer, Caglar Gul- cehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078. · arXiv
- Dina Demner-Fushman, Marc D. Kohli, Marc B. Rosen- man, Sonya E. Shooshan, Laritza M. Rodriguez, S. Antani, George R. Thoma, and Clement J. Mc- Donald. 2016. Preparing a collection of radiology examinations for distribution and retrieval. Journal of the American Medical Informatics Association : JAMIA, 23 2:304-10.
- Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hier- archical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255.
- Ary L Goldberger, Luis AN Amaral, Leon Glass, Jef- frey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus, George B Moody, Chung-Kang Peng, and H Eugene Stanley. 2000. Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals. circulation, 101(23):e215-e220.
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778.
- Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735- 1780.
- Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. 2019. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 590-597.
- Grzegorz Jacenków, Alison Q O'Neil, and Sotirios A Tsaftaris. 2022. Indication as prior knowledge for multimodal disease classification in chest radio- graphs with transformers. In 2022 IEEE 19th Inter- national Symposium on Biomedical Imaging (ISBI), pages 1-5. IEEE.
- Saahil Jain, Ashwin Agrawal, Adriel Saporta, Steven QH Truong, Du Nguyen Duong, Tan Bui, Pierre Chambon, Yuhao Zhang, Matthew P Lungren, Andrew Y Ng, et al. 2021. Radgraph: Extracting clinical entities and relations from radiology reports. arXiv preprint arXiv:2106.14463. · arXiv
- Alistair EW Johnson, Tom J Pollard, Nathaniel R Green- baum, Matthew P Lungren, Chih-ying Deng, Yifan Peng, Zhiyong Lu, Roger G Mark, Seth J Berkowitz, and Steven Horng. 2019b. MIMIC-CXR-JPG, a large publicly available database of labeled chest radio- graphs. arXiv preprint arXiv:1901.07042. · arXiv
- Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. · arXiv
- Mingjie Li, Wenjia Cai, Karin Verspoor, Shirui Pan, Xi- aodan Liang, and Xiaojun Chang. 2022. Cross-modal clinical graph transformer for ophthalmic report gen- eration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20656-20665.
- Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
- Fenglin Liu, Shen Ge, and Xian Wu. 2021a. Competence-based multimodal curriculum learning for medical report generation. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol- ume 1: Long Papers), pages 3001-3012.
- Fenglin Liu, Xian Wu, Shen Ge, Wei Fan, and Yuexian Zou. 2021b. Exploring and distilling posterior and prior knowledge for radiology report generation. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 13753- 13762.
- Fenglin Liu, Changchang Yin, Xian Wu, Shen Ge, Ping Zhang, and Xu Sun. 2021c. Contrastive attention for automatic chest x-ray report generation. arXiv preprint arXiv:2106.06965. · arXiv
- Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher. 2017. Knowing when to look: Adaptive attention via a visual sentinel for image captioning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 375-383.
- NHS England and NHS improvement. 2022. Diagnostic imaging dataset statistical release. pages 1-17.
- Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computa- tional Linguistics, pages 311-318.
- Leslie E Quint, Douglas J Quint, and James D Myles. 2008. Frequency and spectrum of errors in final radiology reports generated with automatic speech recognition technology. Journal of the American College of Radiology, 5(12):1196-1199.
- DS Rana, G Hurst, L Shepstone, J Pilling, J Cockburn, and M Crawford. 2005. Voice recognition for radiol- ogy reporting: is it good enough? Clinical radiology, 60(11):1205-1212.
- Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. 2017. Self-critical sequence training for image captioning. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 7008-7024.
- Michael D Ringler, Brian C Goss, and Brian J Barthol- mai. 2017. Syntactic and semantic errors in radiology reports associated with speech recognition software. Health informatics journal, 23(1):3-13.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
- Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pages 3156-3164.
Cited by
Sorted by citations
Automated Radiology Report Generation: A Review of Recent Advances
Sloan, P., Clatworthy, P. L., Simpson, E., & Mirmehdi, M. (2024) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2024.3408456 · Cited by 98 · see citing works
Current and future state of evaluation of large language models for medical summarization tasks
Croxford, E., Gao, Y., Pellegrino, N., Wong, K. K. Y., Wills, G. J., First, E., Liao, F., Goswami, C., Patterson, B. W., & Afshar, M. (2025) · npj Health Systems · doi:10.1038/s44401-024-00011-2 · Cited by 70 · see citing works
A survey of deep-learning-based radiology report generation using multimodal inputs
Wang, X., Figueredo, G. P., Li, R., Zhang, W. E., Chen, W., & Chen, X. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103627 · Cited by 50 · see citing works
Controllable Chest X-Ray Report Generation from Longitudinal Representations
Serra, F., Wang, C., Deligianni, F., Dalton, J., & O’Neil, A. Q. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.325 · Cited by 20 · see citing works
Self-citationMAIRA-2: Grounded Radiology Report Generation
Bannur, S., Bouzid, K., Castro, D. C., Schwaighofer, A., Thieme, A., Bond-Taylor, S., Ilse, M., Pérez-García, F., Salvatelli, V., Sharma, H., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.04449 · Cited by 19 · see citing works
“Empirically, providing the Indication to the model improves the quality of generated reports (Dalla Serra et al., 2022; Hyland et al., 2023) and has become more commonplace (Tu et al., 2024; Chaves et al., 2024; Yang et al., 2024).”
BackgroundFinding-Aware Anatomical Tokens for Chest X-Ray Automated Reporting
Serra, F. D., Wang, C., Deligianni, F., Dalton, J., & O’Neil, A. Q. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-45673-2_41 · Cited by 11 · see citing works
Self-citationHe, Z., Wong, A. N. N., & Yoo, J. S. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110625 · Cited by 7 · see citing works
Enhancing Radiology Report Generation via Multi-Phased Supervision
Chen, Z., Li, Y., Wang, Z., Gao, P., Barthélemy, J., Zhou, L., & Wang, L. (2025) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2025.3580659 · Cited by 5 · see citing works
Xu, Q., Xu, X., Zhou, C., Liu, Z., Huang, F., Li, S., Zhu, L., Bai, Z., Xu, Y., & Hu, W. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.126585 · Cited by 4 · see citing works
Feature Decomposition via Shared Low-Rank Matrix Recovery for CT Report Generation
Tian, Y., & Song, Y. (2025) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2025.3628159 · Cited by 3 · see citing works
Radiology report generation based on adaptive enhanced fusion of multi features
Cao, Y., Ding, H., Zhang, Y., & Hei, Y. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110494
Shen, T., Teng, Z., & Zhang, D. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04555-3_13
Optimizing Efficiency and Visual-Textual Alignment for LLM-Based Radiology Report Generation
Chen, Z., Gao, P., Lee, Y., Barthélemy, J., Zhou, L., & Wang, L. (2025) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme59968.2025.11210172
- Authors
- Dalla Serra, F., Clackett, W., MacKinnon, H., Wang, C., Deligianni, F., Dalton, J., & O’Neil, A. Q.
- Affiliations
- Canon Medical Research Europe (United Kingdom) (GB)
- University of Glasgow (GB)
- Venue
- Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
- Published
- 2022
- Pages
- 615-624
- OpenAlex
- W4404752295
- Cited by
- 12
- References
- 0
- Citations by year
- 2025: 9 · 2024: 1 · 2023: 2
- FWCI
- 1.09
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing (AACL-IJCNLP) (publishedVersion, cc-by)
- ENLIGHTEN (Jurnal Bimbingan dan Konseling Islam) (acceptedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
- Keywords
- radiology report generation, multimodal generation, named entity recognition, relation extraction, clinical text generation, multimodal fusion
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
