A Pilot Study in Surveying Clinical Judgments to Evaluate Radiology Report Generation
Open accessAbstract
The recent release of many Chest X-Ray datasets has prompted a lot of interest in radiology report generation. To date, this has been framed as an image captioning task, where the machine takes an RGB image as input and generates a 2-3 sentence summary of findings as output. The quality of these reports has been canonically measured using metrics from the NLP community for language generation such as Machine Translation and Summarization. However, the evaluation metrics (e.g. BLEU, CIDEr) are inappropriate for the medical domain, where clinical correctness is critical. To address this, our team brought together machine learning experts with radiologists for a pilot study in co-designing a better metric for evaluating the quality of an algorithmically-generated radiology report. The interdisciplinary collaborative process involved multiple interviews, outreach, and preliminary annotation to design a larger scale study - which is now underway - to build a more meaningful evaluation tool.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Even metrics designed to evaluate the correctness of radiology information by capturing domain-specific concepts do not align with radiologists.
“Even metrics that were designed to evaluate the correctness of radiology information by capturing domain-specific concepts do not align with radiologists.23”
From Evaluating progress in automatic chest X-ray radiology report generation (2023) · cites this paper as [23]
Prior works have investigated the alignment between metrics and human judgment.
“Prior works have investigated the alignment between metrics and human judgment.23,33”
From Evaluating progress in automatic chest X-ray radiology report generation (2023) · cites this paper as [23]
Related works have instrumented quantitative (completeness, correctness, conciseness) and qualitative (content, language, structure) measurements to distinguish synthesized clinical reports from human reports.
“Therefore, related works have instrumented quantitative (completeness, correctness, conciseness) and qualitative (content, language, structure) measurements to dissect the riveting characters that distinguish synthesized clinical reports from human reports46–49.”
From Towards a holistic framework for multimodal LLM in 3D brain CT radiology report generation (2025) · cites this paper as [48]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · see citing works
Flesch, R. (1948) · Journal of Applied Psychology · doi:10.1037/h0057532 · Cited by 5,391 · see citing works
CIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845 · see citing works
(2005) · Cited by 4,152 · see citing works
BERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865 · see citing works
Artificial intelligence in healthcare
Yu, K.-H., Beam, A. L., & Kohane, I. S. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0305-z · Cited by 2,817 · see citing works
CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep LearningPreprint
Rajpurkar, P., Irvin, J., Zhu, K., Yang, B., Mehta, H., Duan, T., Ding, D. Y., Bagul, A., Langlotz, C. P., Shpanskaya, K. S., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.05225 · Cited by 2,283 · see citing works
The Technique of Clear Writing.
Gunning, R. C. (1968) · CERN Document Server (European Organization for Nuclear Research) · Cited by 1,675 · see citing works
Proceedings of the 40th Annual Meeting on Association for Computational Linguistics
Isabelle, P. (2002) · Cited by 1,633 · see citing works
Deep Learning Applications in Medical Image Analysis
Ker, J., Wang, L., Rao, J. P., & Lim, C. C. T. (2017) · IEEE Access · doi:10.1109/access.2017.2788044 · Cited by 1,521 · see citing works
Preparing a collection of radiology examinations for distribution and retrieval
Demner-Fushman, D., Kohli, M. D., Rosenman, M. B., Shooshan, S. E., Rodriguez, L., Antani, S., Thoma, G. R., & McDonald, C. J. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv080 · Cited by 1,164 · see citing works
A Deep Learning Mammography-based Model for Improved Breast Cancer Risk Prediction
Yala, A., Lehman, C., Schuster, T., Portnoi, T., & Barzilay, R. (2019) · Radiology · doi:10.1148/radiol.2019182716 · Cited by 758 · see citing works
Murray, G., Renals, S. J., Carletta, J., & Moore, J. D. (2005) · Meeting of the Association for Computational Linguistics · Cited by 716 · see citing works
(1995) · Medical Entomology and Zoology · Cited by 706 · see citing works
Deep learning in medical image registration: a survey
Haskins, G., Kruger, U., & Yan, P. (2020) · Machine Vision and Applications · doi:10.1007/s00138-020-01060-x · Cited by 704 · see citing works
A SICK cure for the evaluation of compositional distributional semantic models
Marelli, M., Menini, S., Baroni, M., Bentivogli, L., Raffaella Bernardi, B., & Zamparelli, R. (2014) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/39qdhuevzbqa · Cited by 680 · see citing works
Re-evaluating the Role of Bleu in Machine Translation Research
Callison-Burch, C., Osborne, M., & Koehn, P. (2006) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 632 · see citing works
CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison
Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.3301590 · Cited by 392 · see citing works
Learning to Read Chest X-Rays: Recurrent Neural Cascade Model for Automated Image Annotation
Shin, H.-C., Roberts, K. E., Lü, L., Demner‐Fushman, D., Yao, J., & Summers, R. M. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.274 · Cited by 390 · see citing works
A Structured Review of the Validity of BLEU
Reiter, E. (2018) · Computational Linguistics · doi:10.1162/coli_a_00322 · Cited by 386 · see citing works
MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs
Johnson, A. E. W., Pollard, T. J., Greenbaum, N. R., Lungren, M. P., Deng, C.-Y., Peng, Y., Lu, Z., Mark, R. G., Berkowitz, S. J., & Horng, S. (2019) · arXiv · doi:10.48550/arxiv.1901.07042 · Cited by 269 · see citing works
Clinically Accurate Chest X-Ray Report GenerationPreprint
Liu, G., Hsu, T.-M. H., McDermott, M. B. A., Boag, W., Weng, W., Peter Szolovits, B., & Ghassemi, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.02633 · Cited by 212 · see citing works
Self-citationSentence Mover’s Similarity: Automatic Evaluation for Multi-Sentence Texts
Clark, E., Çelikyılmaz, A., & Smith, N. A. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1264 · Cited by 138 · see citing works
Lo, C.-K. (2019) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w19-5358 · Cited by 114 · see citing works
Deep Learning for Medical Image SegmentationPreprint
Lai, M. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.02000 · Cited by 96 · see citing works
BEER: BEtter Evaluation as Ranking
Stanojević, M., & Sima’an, K. (2014) · Conference on Machine Translation (WMT) · doi:10.3115/v1/w14-3354 · Cited by 91 · see citing works
RUSE: Regressor Using Sentence Embeddings for Automatic Machine Translation Evaluation
Shimanaka, H., Kajiwara, T., & Komachi, M. (2018) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w18-6456 · Cited by 89 · see citing works
11th Conference of the European Chapter of the Association for Computational Linguistics
McCarthy, D., & Wintner, S. (2006) · Conference of the European Chapter of the Association for Computational Linguistics · Cited by 75 · see citing works
MEANT 2.0: Accurate semantic MT evaluation for any output language
Lo, C.-K. (2017) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w17-4767 · Cited by 52 · see citing works
Ma, Q., Graham, Y., Wang, S., & Liu, Q. (2017) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w17-4768 · Cited by 45 · see citing works
Unsupervised Multimodal Representation Learning across Medical Images and ReportsPreprint
Hsu, T.-M. H., Weng, W., Boag, W., McDermott, M. B. A., & Peter Szolovits, B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.08615 · Cited by 33 · see citing works
Self-citationProducing radiologist-quality reports for interpretable artificial intelligencePreprint
Gale, W., Oakden‐Rayner, L., Carneiro, G., Bradley, A. P., & Palmer, L. J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.00340 · Cited by 27 · see citing works
Bimodal Network Architectures for Automatic Generation of Image Annotation from Text
Moradi, M., Madani, A., Gur, Y., Guo, Y., & Syeda-Mahmood, T. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-00928-1_51 · Cited by 20 · see citing works
Towards Neural Similarity Evaluator
Kané, H., Kocyigit, M. Y., Ajanoh, P., Abdalla, A. A. A., & Coulibali, M. (2019) · Cited by 6 · see citing works
Self-citationClinical Collabsheets: 53 Questions to Guide a Clinical Collaboration.
Saleh, S., Boag, W., Erdman, L., & Naumann, T. (2020) · Machine Learning for Healthcare Conference · Cited by 6 · see citing works
Self-citationMUTT: Metric Unit TesTing for Language Generation Tasks
Boag, W., Campos, R., Saenko, K., & Rumshisky, A. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1182 · Cited by 3 · see citing works
Self-citationBLEU is Not Suitable for the Evaluation of Text SimplificationPreprint
Sulem, E., Abend, O., & Rappoport, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.05995 · Cited by 3 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
Research works printed but not linked: 4
- William Boag, Tzu-Ming Harry Hsu, Matthew McDermott, Gabriela Berner, Emily Alsentzer, Wei-Hung Weng, Peter Szolovits, and Marzyeh Ghassemi. 2019. Baselines for Chest X-Ray Report Generation. Machine Learning for Health workshop at NeurIPS.
- Aurelia Bustos, Antonio Pertusa, Jose-Maria Salinas, and Maria de la Iglesia-Vayá. 2019. Padchest: A large chest x-ray image dataset with multi-label annotated reports. arXiv preprint arXiv:1901.07441 (2019). · arXiv
- Richard H. Thaler and howpublished = Penguin Books year = 2009 isbn = 9780143115267
- Cass R. Sunstein, title = Nudge: Improving Decisions About Health, Wealth, and Happiness Revised Expanded Edition. [n.d.]. .
Cited by
Sorted by citations
Evaluating progress in automatic chest X-ray radiology report generation
Yu, F., Endo, M., Krishnan, R., Pan, I., Tsai, A., Reis, E. P., Fonseca, E. K. U. N., Lee, H. M. H., Abad, Z. S. H., Ng, A. Y., et al. (2023) · Patterns · doi:10.1016/j.patter.2023.100802 · Cited by 200 · see citing works
Towards a holistic framework for multimodal LLM in 3D brain CT radiology report generation
Li, C.-Y., Chang, K.-J., Yang, C.-F., Wu, H.-Y., Chen, W., Bansal, H., Chen, L., Yang, Y.-P., Chen, Y.-C., Chen, S.-P., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-57426-0 · Cited by 75 · see citing works
“To 35-37 By adopting similar designs with objective linguistic criteria, we found that both reviewer success rate and answer alternation reasons (“Suspicious wording” and “Both not mentioning critical features”) were associated with the writing style (“Familiarity and voice” and "Specificity or…”
Ziegelmayer, S., Marka, A. W., Lenhart, N. S., Nehls, N., Reischl, S., Harder, F., Sauter, A. P., Makowski, M., Graf, M. M., & Gawlitza, J. F. (2023) · Journal of Medical Internet Research · doi:10.2196/50865 · Cited by 26 · see citing works
NLP for Maternal Healthcare: Perspectives and Guiding Principles in the Age of LLMs
Antoniak, M., Naik, A., Alvarado, C. S., Wang, L. L., & Chen, I. Y. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658982 · Cited by 23 · see citing works
MAIRA-2: Grounded Radiology Report Generation
Bannur, S., Bouzid, K., Castro, D. C., Schwaighofer, A., Thieme, A., Bond-Taylor, S., Ilse, M., Pérez-García, F., Salvatelli, V., Sharma, H., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.04449 · Cited by 19 · see citing works
The Consensus Trap: Dissecting Subjectivity and the “Ground Truth” Illusion in Data Annotation
Munir, S., Mah, B., Kalsi, K., Kapania, S., Posada, J., Law, E., Wang, D., & Ahmed, S. I. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3806510
- Authors
- Boag, W., Kané, H., Rawat, S., Wei, J., & Goehler, A.
- Affiliations
- Massachusetts Institute of Technology (US)
- WL Research, USA
- Beth Israel Deaconess Medical Center (US)
- Venue
- Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency
- Publisher
- ACM
- Published
- 2021-03
- Pages
- 458-465
- OpenAlex
- W3133569056
- Cited by
- 5
- References
- 30
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 1 · 2023: 2
- FWCI
- 0.25
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion, cc-by)
- DSpace@MIT (Massachusetts Institute of Technology) (acceptedVersion, cc-by)
- Funders
- NSF (1122374)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiology practices and education
- Keywords
- radiology report generation, clinical judgment, evaluation metrics, chest radiography, image captioning, co-design, annotation, natural language generation evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
