CoChat
Sign Up
Cited by 2

The DOI matches its DataCite record.

Radiology Report Expert Evaluation (ReXVal) Dataset

Yu, Feiyang, Endo, Mark, Krishnan, Rayan, Pan, Ian, Tsai, Andy, Reis, Eduardo Pontes, Kaiser Ururahy Nunes Fonseca, Eduardo, Lee, Henrique, Shakeri, Zahra, Ng, Andrew, Langlotz, Curtis, Venugopal, Vasantha Kumar, Rajpurkar, Pranav·2023·PhysioNet
DOI 10.13026/r7wq-9h91
Open this paper in CoChat — free account

Abstract

The Radiology Report Expert Evaluation (ReXVal) Dataset is a publicly available dataset of radiologist evaluations of errors in automatically generated radiology reports. The dataset contains annotations from 6 board certified radiologists on clinically significant and clinically insignificant errors under 6 error categories for candidate radiology reports with respect to ground-truth reports from the MIMIC-CXR dataset. There are 4 candidate reports generated for 50 studies, translating to 200 pairs of candidate and ground-truth reports on which radiologists provided annotations. The dataset has been used to evaluate the alignment between scoring of automated metrics and that of radiologists, investigate the failure modes of automated metrics, and build a composite automated metric, in a study on how to meaningfully measure progress in radiology report generation. It is also created to support additional medical AI research in radiology and other expert tasks.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.13026/r7wq-9h91

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • CheXprompt's validity was evaluated using the ReXval dataset, which is comprised of error annotations from 6 board-certified radiologists on 200 pairs of candidate and ground-truth reports.

    “We evaluate the validity of the proposed CheXprompt using the ReXval dataset39, which is comprised of error annotations from 6 board-certified radiologists on 200 pairs of candidate and ground-truth reports”

    From A clinically accessible small multimodal radiology model and evaluation metric for chest X-ray findings (2025) · cites this paper as [39]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 30

  1. Silver D, Huang A, Maddison CJ, Guez A, Sifre L, Van Den Driessche G, Schrittwieser J, Antonoglou I, Panneershelvam V, Lanctot M, Dieleman S. Mastering the game of Go with deep neural networks and tree search. nature. 2016 Jan;529(7587):484-9.
  2. Silver D, Schrittwieser J, Simonyan K, Antonoglou I, Huang A, Guez A, Hubert T, Baker L, Lai M, Bolton A, Chen Y. Mastering the game of go without human knowledge. nature. 2017 Oct;550(7676):354-9.
  3. Silver D, Hubert T, Schrittwieser J, Antonoglou I, Lai M, Guez A, Lanctot M, Sifre L, Kumaran D, Graepel T, Lillicrap T. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science. 2018 Dec 7;362(6419):1140-4.
  4. Schrittwieser J, Antonoglou I, Hubert T, Simonyan K, Sifre L, Schmitt S, Guez A, Lockhart E, Hassabis D, Graepel T, Lillicrap T. Mastering atari, go, chess and shogi by planning with a learned model. Nature. 2020 Dec 24;588(7839):604-9.
  5. Li Y, Choi D, Chung J, Kushman N, Schrittwieser J, Leblond R, Eccles T, Keeling J, Gimeno F, Dal Lago A, Hubert T. Competition-level code generation with alphacode. Science. 2022 Dec 9;378(6624):1092-7.
  6. Chen M, Tworek J, Jun H, Yuan Q, Pinto HP, Kaplan J, Edwards H, Burda Y, Joseph N, Brockman G, Ray A. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. 2021 Jul 7. · arXiv
  7. Bojarski M, Del Testa D, Dworakowski D, Firner B, Flepp B, Goyal P, Jackel LD, Monfort M, Muller U, Zhang J, Zhang X. End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316. 2016 Apr 25. · arXiv
  8. Fridman L, Brown DE, Glazer M, Angell W, Dodd S, Jenik B, Terwilliger J, Patsekin A, Kindelsberger J, Ding L, Seaman S. MIT advanced vehicle technology study: Large-scale naturalistic driving study of driver behavior and interaction with automation. IEEE Access. 2019 Jul 1;7:102021-38.
  9. Rajpurkar P, Chen E, Banerjee O, Topol EJ. AI in health and medicine. Nature medicine. 2022 Jan;28(1):31-8.
  10. Senior AW, Evans R, Jumper J, Kirkpatrick J, Sifre L, Green T, Qin C, Žídek A, Nelson AW, Bridgland A, Penedones H. Improved protein structure prediction using potentials from deep learning. Nature. 2020 Jan 30;577(7792):706-10.
  11. Jumper J, Evans R, Pritzel A, Green T, Figurnov M, Ronneberger O, Tunyasuvunakool K, Bates R, Žídek A, Potapenko A, Bridgland A. Highly accurate protein structure prediction with AlphaFold. Nature. 2021 Aug 26;596(7873):583-9.
  12. Irvin J, Rajpurkar P, Ko M, Yu Y, Ciurea-Ilcus S, Chute C, Marklund H, Haghgoo B, Ball R, Shpanskaya K, Seekins J. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. InProceedings of the AAAI conference on artificial intelligence 2019 Jul 17 (Vol. 33, No. 01, pp. 590-597).
  13. Smit A, Jain S, Rajpurkar P, Pareek A, Ng AY, Lungren MP. CheXbert: combining automatic labelers and expert annotations for accurate radiology report labeling using BERT. arXiv preprint arXiv:2004.09167. 2020 Apr 20. · arXiv
  14. Pino P, Parra D, Besa C, Lagos C. Clinically correct report generation from chest X-rays using templates. InMachine Learning in Medical Imaging: 12th International Workshop, MLMI 2021, Held in Conjunction with MICCAI 2021, Strasbourg, France, September 27, 2021, Proceedings 12 2021 (pp. 654-663). Springer International Publishing.
  15. Miura Y, Zhang Y, Tsai EB, Langlotz CP, Jurafsky D. Improving factual completeness and consistency of image-to-text radiology report generation. arXiv preprint arXiv:2010.10042. 2020 Oct 20. · arXiv
  16. Chen Z, Song Y, Chang TH, Wan X. Generating radiology reports via memory-driven transformer. arXiv preprint arXiv:2010.16056. 2020 Oct 30. · arXiv
  17. Endo M, Krishnan R, Krishna V, Ng AY, Rajpurkar P. Retrieval-based chest X-ray report generation using a pre-trained contrastive language-image model. InMachine Learning for Health 2021 Nov 28 (pp. 209-219). PMLR.
  18. Yan A, He Z, Lu X, Du J, Chang E, Gentili A, McAuley J, Hsu CN. Weakly supervised contrastive learning for chest x-ray report generation. arXiv preprint arXiv:2109.12242. 2021 Sep 25. · arXiv
  19. Nicolson A, Dowling J, Koopman B. Improving chest X-Ray report generation by leveraging warm-starting. arXiv preprint arXiv:2201.09405. 2022 Jan 24. · arXiv
  20. Zhou HY, Chen X, Zhang Y, Luo R, Wang L, Yu Y. Generalized radiograph representation learning via cross-supervision between images and free-text radiology reports. Nature Machine Intelligence. 2022 Jan;4(1):32-40.
  21. Hossain MZ, Sohel F, Shiratuddin MF, Laga H. A comprehensive survey of deep learning for image captioning. ACM Computing Surveys (CsUR). 2019 Feb 4;51(6):1-36.
  22. Boag W, Kané H, Rawat S, Wei J, Goehler A. A Pilot Study in Surveying Clinical Judgments to Evaluate Radiology Report Generation. InProceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency 2021 Mar 3 (pp. 458-465).
  23. Yu F, Endo M, Krishnan R, Pan I, Tsai A, Reis EP, Fonseca EK, Lee HM, Abad ZS, Ng AY, Langlotz CP. Evaluating Progress in Automatic Chest X-Ray Radiology Report Generation. medRxiv. 2022:2022-08.
  24. Johnson AE, Pollard TJ, Berkowitz SJ, Greenbaum NR, Lungren MP, Deng CY, Mark RG, Horng S. MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data. 2019 Dec 12;6(1):317.
  25. Goldberger AL, Amaral LA, Glass L, Hausdorff JM, Ivanov PC, Mark RG, Mietus JE, Moody GB, Peng CK, Stanley HE. PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals. circulation. 2000 Jun 13;101(23):e215-20.
  26. Jain S, Agrawal A, Saporta A, Truong SQ, Duong DN, Bui T, Chambon P, Zhang Y, Lungren MP, Ng AY, Langlotz CP. Radgraph: Extracting clinical entities and relations from radiology reports. arXiv preprint arXiv:2106.14463. 2021 Jun 28. https://github.com/rajpurkarlab/CXR-Report-Metric. Accessed on 6-17-2023. · arXiv
  27. Vision–language foundation models for multimodal medical image understanding and report generation: a narrative review with quantitative synthesis Shahid Mehmood, Farman Matloob Khan, Abeer A. K. Alharbi et al. Frontiers in Medicine, 2026.
  28. From single slices to whole studies: A scoping review of frontier LLM evaluation on CT and MRI and the case for study-level assessment Peter Nemeth, Enikő Orsi, Balázs Patczai et al. Open Academic Journal of Advanced Science and Technology, 2026.
  29. Concept-Aware Knowledge Grounding for Explainable Abstractive Radiology Report Summarization: A Functional Verification of KG-BioBART Esraa Najjar, Mohammed Hasan Abdulameer. VFAST Transactions on Software Engineering, 2026.
  30. A Disease-Memory Guided Framework for Radiology Report Generation from Multi-View Chest X-rays Prajakta Dhamanskar, Chintan Bhupeshbhai Thacker, Saurabh A Shah. Journal of Electronics Electromedical Engineering and Medical Informatics, 2026.

Cited by

Sorted by citations

Metadata

Authors
Yu, F., Endo, M., Krishnan, R., Pan, I., Tsai, A., Reis, E. P., Kaiser Ururahy Nunes Fonseca, E., Lee, H., Shakeri, Z., Ng, A., Langlotz, C., Venugopal, V. K., & Rajpurkar, P.
ORCID
Venue
PhysioNet
Publisher
PhysioNet
Published
2023
DOI
10.13026/r7wq-9h91
References
0

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review