CoChat
Sign Up
PreprintCited by 1Multimodal Machine Learning ApplicationsArtificial Intelligence in Healthcare and EducationRadiology practices and education

Pragmatic Radiology Report Generation

Open access
Nguyen, Dang, Chen, Chacha, He, He, Tan, Chenhao·2023·arXiv
DOI 10.48550/arxiv.2311.17154arXiv 2311.17154
Open this paper in CoChat — free accountOpen PDF

Abstract

When pneumonia is not found on a chest X-ray, should the report describe this negative observation or omit it? We argue that this question cannot be answered from the X-ray alone and requires a pragmatic perspective, which captures the communicative goal that radiology reports serve between radiologists and patients. However, the standard image-to-text formulation for radiology report generation fails to incorporate such pragmatic intents. Following this pragmatic perspective, we demonstrate that the indication, which describes why a patient comes for an X-ray, drives the mentions of negative observations and introduce indications as additional input to report generation. With respect to the output, we develop a framework to identify uninferable information from the image as a source of model hallucinations, and limit them by cleaning groundtruth reports. Finally, we use indications and cleaned groundtruth reports to develop pragmatic models, and show that they outperform existing methods not only in new pragmatics-inspired metrics (+4.3 Negative F1) but also in standard metrics (+6.3 Positive F1 and +11.0 BLEU-2).

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2311.17154

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Comparison to previous imaging studies is crucial for tracking the development of disease or impact of treatment.

    “Comparison to previous imaging studies is crucial for tracking the development of disease or impact of treatment, and references to prior studies are frequent in radiology reporting. 16,17”

    From MAIRA-2: Grounded Radiology Report Generation (2024) · cites this paper as [16]

    The Concluding Discussion (page unavailable) says comparisons with prior studies describe how conditions change over time, supporting their use in tracking disease progression. Table 1 gives an example of a smaller pleural effusion after tube removal, and Table 7 shows prior-study references are common in the reports examined. The paper does not establish that such comparisons are crucial for assessing treatment impact. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • References to prior studies are frequent in radiology reporting.

    “Comparison to previous imaging studies is crucial for tracking the development of disease or impact of treatment, and references to prior studies are frequent in radiology reporting. 16,17”

    From MAIRA-2: Grounded Radiology Report Generation (2024) · cites this paper as [16]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 26

  1. William Boag, Tzu-Ming Harry Hsu, Matthew Mc- Dermott, Gabriela Berner, Emily Alesentzer, and Peter Szolovits. Baselines for chest x-ray report generation. In Machine learning for health work- shop, pages 126-140. PMLR, 2020. URL http: //proceedings.mlr.press/v116/boag20a.
  2. Zhihong Chen, Yan Song, Tsung-Hui Chang, and Xiang Wan. Generating radiology reports via memory-driven transformer. In Proceedings of the 2020 Conference on Empirical Methods in Natu- ral Language Processing (EMNLP), pages 1439- 1449, Online, nov 2020. Association for Com- putational Linguistics. doi: 10.18653/v1/2020. emnlp-main.112. URL https://aclanthology. org/2020.emnlp-main.112. · DOI · Paper page
  3. Dina Demner-Fushman, Marc D Kohli, Marc B Rosenman, Sonya E Shooshan, Laritza Rodriguez, Sameer Antani, George R Thoma, and Clement J McDonald. Preparing a collection of radiology ex- aminations for distribution and retrieval. Journal of the American Medical Informatics Association, 23(2):304-310, 2016. URL https://academic. oup.com/jamia/article/23/2/304/2572395.
  4. Mark Endo, Rayan Krishnan, Viswesh Krishna, An- drew Y Ng, and Pranav Rajpurkar. Retrieval- based chest x-ray report generation using a pre- trained contrastive language-image model. In Ma- chine Learning for Health, pages 209-219. PMLR, 2021. URL https://proceedings.mlr.press/ v158/endo21a.html.
  5. Michael P Hartung, Ian C Bickle, Frank Gaillard, and Jeffrey P Kanne. How to create a great ra- diology report. RadioGraphics, 40(6):1658-1670, 2020. URL https://pubs.rsna.org/doi/full/ 10.1148/rg.2020200020. · DOI
  6. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of the IEEE Conference on Computer Vision and Pat- tern Recognition (CVPR), June 2016. URL https://openaccess.thecvf.com/content_ cvpr_2016/html/He_Deep_Residual_Learning_ CVPR_2016_paper.html.
  7. Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yi- fan Yu, Silviana Ciurea-Ilcus, Chris Chute, Hen- rik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. Chexpert: A large chest radio- graph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 590-597, 2019. URL https://ojs.aaai.org/index.php/ AAAI/article/view/3834.
  8. Saahil Jain, Ashwin Agrawal, Adriel Saporta, Steven QH Truong, Du Nguyen Duong, Tan Bui, Pierre Chambon, Yuhao Zhang, Matthew P Lun- gren, Andrew Y Ng, et al. Radgraph: Extract- ing clinical entities and relations from radiology reports. arXiv preprint arXiv:2106.14463, 2021. URL https://arxiv.org/abs/2106.14463. · arXiv
  9. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucina- tion in natural language generation. ACM Com- put. Surv., 55(12), mar 2023. ISSN 0360-0300. doi: 10.1145/3571730. URL https://doi.org/ 10.1145/3571730. · DOI · Paper page
  10. Baoyu Jing, Pengtao Xie, and Eric Xing. On the au- tomatic generation of medical imaging reports. In Proceedings of the 56th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 2577-2586, Melbourne, Australia, jul 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18-1240. URL https://aclanthology.org/P18-1240. · DOI · Paper page
  11. Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Roger G Mark, and Steven Horng. MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data, 6(1):317, 2019. URL https://www.nature.com/articles/ s41597-019-0322-0.
  12. Hyungyung Lee, Wonjae Kim, Jin-Hwa Kim, Tackeun Kim, Jihang Kim, Leonard Sunwoo, and Edward Choi. Unified chest x-ray and radiology report generation model with multi-view chest x- rays. arXiv preprint arXiv:2302.12172, 2023. URL https://arxiv.org/abs/2302.12172. · arXiv
  13. Guanxiong Liu, Tzu-Ming Harry Hsu, Matthew Mc- Dermott, Willie Boag, Wei-Hung Weng, Peter Szolovits, and Marzyeh Ghassemi. Clinically ac- curate chest x-ray report generation. In Machine Learning for Healthcare Conference, pages 249- 269. PMLR, 2019. URL http://proceedings. mlr.press/v106/liu19a.html.
  14. Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, and Adam Roberts. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301.13688, 2023. URL https://arxiv. org/abs/2301.13688. · arXiv
  15. Yasuhide Miura, Yuhao Zhang, Emily Tsai, Curtis Langlotz, and Dan Jurafsky. Improving factual completeness and consistency of image-to-text ra- diology report generation. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguis- tics: Human Language Technologies, pages 5288- 5304, Online, jun 2021. Association for Com- putational Linguistics. doi: 10.18653/v1/2021. naacl-main.416. URL https://aclanthology. org/2021.naacl-main.416. · DOI · Paper page
  16. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: A method for automatic eval- uation of machine translation. In Proceedings of the 40th annual meeting of the Association for Com- putational Linguistics, pages 311-318, 2002. URL https://aclanthology.org/P02-1040/.
  17. Vignav Ramesh, Nathan A Chi, and Pranav Rajpurkar. Improving radiology report gen- eration systems by removing hallucinated ref- erences to non-existent priors. In Machine Learning for Health, pages 456-473. PMLR, 2022. URL https://proceedings.mlr.press/ v193/ramesh22a.html.
  18. Akshay Smit, Saahil Jain, Pranav Rajpurkar, Anuj Pareek, Andrew Ng, and Matthew Lungren. Com- bining automatic labelers and expert annotations for accurate radiology report labeling using BERT. In Proceedings of the 2020 Conference on Em- pirical Methods in Natural Language Processing (EMNLP), pages 1500-1519, Online, nov 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.117. URL https: //aclanthology.org/2020.emnlp-main.117. · DOI · Paper page
  19. Omkar Thawkar, Abdelrahman Shaker, Sahal Shaji Mullappilly, Hisham Cholakkal, Rao Muhammad Anwer, Salman Khan, Jorma Laaksonen, and Fa- had Shahbaz Khan. Xraygpt: Chest radiographs summarization using medical vision-language mod- els. arXiv preprint arXiv:2306.07971, 2023. URL https://arxiv.org/abs/2306.07971. · arXiv
  20. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and ef- ficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. URL https://arxiv. org/abs/2302.13971. · arXiv
  21. Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaek- ermann, Mohamed Amin, Pi-Chuan Chang, An- drew Carroll, Chuck Lau, Ryutaro Tanno, Ira Ktena, et al. Towards generalist biomedical ai. arXiv preprint arXiv:2307.14334, 2023. URL https://arxiv.org/abs/2307.14334. · arXiv
  22. Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, and Ronald M Summers. Tienet: Text-image embedding network for common thorax disease classification and reporting in chest x-rays. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9049-9058, 2018. URL https://openaccess.thecvf. com/content_cvpr_2018/html/Wang_TieNet_ Text-Image_Embedding_CVPR_2018_paper.html.
  23. Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. Medclip: Contrastive learning from unpaired medical images and text. arXiv preprint arXiv:2210.10163, 2022. URL https://arxiv. org/abs/2210.10163. · arXiv
  24. Feiyang Yu, Mark Endo, Rayan Krishnan, Ian Pan, Andy Tsai, Eduardo Pontes Reis, Eduardo Kaiser Ururahy Nunes Fonseca, Henrique Min Ho Lee, Zahra Shakeri Hossein Abad, Andrew Y Ng, et al. Evaluating progress in automatic chest x-ray ra- diology report generation. medRxiv, pages 2022- 08, 2022. URL https://www.ncbi.nlm.nih.gov/ pmc/articles/PMC10499844/.
  25. Jianbo Yuan, Haofu Liao, Rui Luo, and Jiebo Luo. Automatic radiology report generation based on multi-view image fusion and med- ical concept enrichment. In Medical Image Computing and Computer Assisted Intervention- MICCAI 2019: 22nd International Conference, Shenzhen, China, October 13-17, 2019, Proceed- ings, Part VI 22, pages 721-729. Springer, 2019. URL https://link.springer.com/chapter/10. 1007/978-3-030-32226-7_80. · DOI
  26. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evalu- ating text generation with bert. arXiv preprint arXiv:1904.09675, 2019. URL https://arxiv. org/abs/1904.09675. · arXiv

Cited by

Sorted by citations

  • MAIRA-2: Grounded Radiology Report Generation

    Bannur, S., Bouzid, K., Castro, D. C., Schwaighofer, A., Thieme, A., Bond-Taylor, S., Ilse, M., Pérez-García, F., Salvatelli, V., Sharma, H., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.04449 · Cited by 19 · see citing works

    “Hence, in this work we generate chest X-ray (CXR) reports using: the current frontal image, the current lateral image, the prior frontal image and prior report, and the Indication , Technique , and Comparison sections of the current study.”
    BackgroundMethodInfluential

Authors
Nguyen, D., Chen, C., He, H., & Tan, C.
Affiliations
University of Chicago (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2311.17154
arXiv
2311.17154
OpenAlex
W4389217204
Cited by
1
References
0
Citations by year
2024: 1
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
National Science Foundation (2126602, 2040989)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiology practices and education
Keywords
radiology report generation, pragmatics, negative results, chest radiography, indications, model hallucination, image-to-text generation, communicative purpose

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review