GPT-4 as an Effective Zero-Shot Evaluator for Scientific Figure Captions
Open accessAbstract
There is growing interest in systems that generate captions for scientific figures.However, assessing these systems' output poses a significant challenge.Human evaluation requires academic expertise and is costly, while automatic evaluation depends on often low-quality author-written captions.This paper investigates using large language models (LLMs) as a cost-effective, reference-free method for evaluating figure captions.We first constructed SCICAP-EVAL, 1 a human evaluation dataset that contains human judgments for 3,600 scientific figure captions, both original and machinemade, for 600 arXiv figures.We then prompted LLMs like GPT-4 and GPT-3 to score (1-6) each caption based on its potential to aid reader understanding, given relevant context such as figure-mentioning paragraphs.Results show that GPT-4, used as a zero-shot evaluator, outperformed all other models and even surpassed assessments made by Computer Science and Informatics undergraduates, achieving a Kendall correlation score of 0.401 with Ph.D. students' rankings.1 SCICAP-EVAL is accessible at https://github.com/ Crowd-AI-Lab/SciCap-Eval.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLMs have been shown to perform well on diverse text-analysis tasks, often matching or exceeding state-of-the-art models and crowd workers, even with zero-shot prompts.
“LLMs have been shown to perform well on diverse text-analysis tasks 36 , often matching or exceeding state-of-the-art models and crowd workers 37,38 , even with zero-shot prompts 39 .”
From Quantifying the prevalence and impact of overreaching causal claims in social science (2026) · cites this paper as [39]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,695 · see citing works
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505 · see citing works
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · see citing works
SciBERT: A Pretrained Language Model for Scientific Text
Beltagy, I., Lo, K., & Cohan, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1371 · Cited by 3,218 · see citing works
BERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865 · see citing works
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · see citing works
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 2,523 · see citing works
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Zhang, J., Zhao, Y., Saleh, M., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1912.08777 · Cited by 1,206 · see citing works
BLEURT: Learning Robust Metrics for Text Generation
Sellam, T., Das, D., & Parikh, A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.704 · Cited by 1,005 · see citing works
TrOCR: Transformer-Based Optical Character Recognition with Pre-trained Models
Li, M., Lv, T., Chen, J., Cui, L., Lu, Y., Florencio, D. A. F., Zhang, C., Li, Z., & Wei, F. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26538 · Cited by 600 · see citing works
Can Large Language Models Be an Alternative to Human Evaluations?
Chiang, C.-H., & Lee, H.-Y. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.870 · Cited by 331 · see citing works
Can Large Language Models Be an Alternative to Human Evaluations?
Chiang, C.-H., & Lee, H.-Y. (2023) · arXiv · doi:10.48550/arxiv.2305.01937 · Cited by 331 · see citing works
Chart-to-Text: A Large-Scale Benchmark for Chart Summarization
Kantharaj, S., Leong, R. T. K., Lin, X., Masry, A., Thakkar, M., Hoque, E., & Joty, S. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.277 · Cited by 126 · see citing works
Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarization
Shen, C., Cheng, L., Nguyen, X.-P., You, Y., & Bing, L. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.278 · Cited by 71 · see citing works
VisText: A Benchmark for Semantically Rich Chart Captioning
Tang, B. J., Boggust, A., & Satyanarayan, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.401 · Cited by 68 · see citing works
SciCap: Generating Captions for Scientific Figures
Hsu, T.-Y., Giles, C. L., & Huang, T.-H. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.277 · Cited by 59 · see citing works
Self-citationLess is More for Long Document Summary Evaluation by LLMsPreprint
Wu, Y., Iso, H., Pezeshkpour, P., Bhutani, N., & Hruschka, E. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2309.07382 · Cited by 51 · see citing works
Sun, S., Shapira, O., Dagan, I., & Nenkova, A. (2019) · doi:10.18653/v1/w19-2303 · Cited by 40 · see citing works
Figure Captioning with Relation Maps for Reasoning
Chen, C., Zhang, R., Koh, E., Kim, S., Cohen, S. D., & Rossi, R. A. (2020) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv45572.2020.9093592 · Cited by 36 · see citing works
Self-citationSciGraphQA: A Large-Scale Synthetic Multi-Turn Question-Answering Dataset for Scientific GraphsPreprint
Li, S., & Tajbakhsh, N. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.03349 · Cited by 19 · see citing works
Neural caption generation over figures
Chen, C., Zhang, R., Kim, S., Cohen, S. D., Yu, T., Rossi, R. A., & Bunescu, R. (2019) · ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp) · doi:10.1145/3341162.3345601 · Cited by 18 · see citing works
Self-citationImage or Information? Examining the Nature and Impact of Visualization Perceptual Classification
Arunkumar, A., Padilla, L., Bae, G., & Bryan, C. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3326919 · Cited by 15 · see citing works
Huang, C.-Y., Hsu, T.-Y., Rossi, R. A., Nenkova, A., Kim, S., Chan, G. Y.-Y., Koh, E., Giles, C. L., & Huang, T.-H. (2023) · International Conference on Natural Language Generation · doi:10.18653/v1/2023.inlg-main.6 · Cited by 12 · see citing works
Self-citationSciCap+: A Knowledge Augmented Dataset to Study the Challenges of Scientific Figure Captioning
Yang, Z., Dabre, R., Tanaka, H., & Okazaki, N. (2024) · Journal of Natural Language Processing · doi:10.5715/jnlp.31.1140 · Cited by 12 · see citing works
Large Language Models are Not Yet Human-Level Evaluators for Abstractive SummarizationPreprint
Shen, C., Cheng, L., Nguyen, X.-P., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.13091 · Cited by 6 · see citing works
PatFig: Generating Short and Long Captions for Patent Figures
Aubakirova, D., Gerdes, K., & Liu, L. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00305 · Cited by 5 · see citing works
SciCap+: A Knowledge Augmented Dataset to Study the Challenges of Scientific Figure CaptioningPreprint
Yang, Z., Dabre, R., Tanaka, H., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.03491 · Cited by 2 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- PMLR.
Cited by
Sorted by citations
VLSBench: Unveiling Visual Leakage in Multimodal Safety
Hu, X., Liu, D., Li, Z., Huang, X., & Shao, J. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.405 · Cited by 18 · see citing works
General-Purpose Models for the Chemical Sciences: LLMs and Beyond
Alampara, N., Aneesh, A., Ríos-García, M., Mirza, A., Schilling-Wilhelmi, M., Aghajani, A. A., Sun, M., Prastalo, G., & Jablonka, K. M. (2026) · Chemical Reviews · doi:10.1021/acs.chemrev.5c00583 · Cited by 12 · see citing works
Hsu, T.-Y., Huang, C.-Y., Huang, S.-H., Rossi, R., Kim, S., Yu, T., Giles, C. L., & Huang, T.-H. K. (2024) · Extended Abstracts of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613905.3650738 · Cited by 12 · see citing works
Self-citationMulti-LLM Collaborative Caption Generation in Scientific Documents
Kim, J., Lee, J., Choi, H.-J., Hsu, T.-Y., Huang, C.-Y., Kim, S., Rossi, R. A., Yu, T., Giles, C. L., Huang, T.-H., et al. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-8912-5_6 · Cited by 8 · see citing works
Self-citationUnderstanding Writing Assistants for Scientific Figure Captions: A Thematic Analysis
Ng, H. Y. S., Hsu, T.-Y., Min, J., Kim, S., Rossi, R. A., Yu, T., Jung, H., & Huang, T.-H. (2025) · Workshop on Intelligent and Interactive Writing Assistants (In2Writing) · doi:10.18653/v1/2025.in2writing-1.1 · Cited by 2 · see citing works
Self-citationMQAD: A Large-Scale Question Answering Dataset for Training Music Large Language Models
Ouyang, Z., Wang, J.-C., Zhang, D., Chen, B., Li, S., & Lin, Q. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890561 · Cited by 2 · see citing works
Hsu, T.-Y., Hsu, Y.-L., Rohatgi, S., Huang, C.-Y., Ng, H. Y. S., Rossi, R., Kim, S., Yu, T., Ku, L., Giles, C. L., et al. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.653 · Cited by 2 · see citing works
Self-citationRaees, M., Iwamoto, Y., Papangelis, K., Heard, J., & Tigwell, G. W. (2026) · arXiv · doi:10.48550/arxiv.2602.08937 · Cited by 1 · see citing works
Understanding How Paper Writers Use AI-Generated Captions in Figure Caption Writing
Ng, H. Y., Hsu, T.-Y., Min, J., Kim, S., Rossi, R. A., Yu, T., Jung, H., & Huang, T.-H. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-8912-5_8 · Cited by 1 · see citing works
Self-citationQuantifying the prevalence and impact of overreaching causal claims in social science
Isch, C., Dörr, T., Fasching, N., Jennings, G., & Watts, D. J. (2026) · Nature Human Behaviour · doi:10.1038/s41562-026-02553-x
Hidden in Plain Sight: Reasoning in Underspecified and Misspecified Scenarios for Multimodal LLMs
Yan, Q., Li, H., Jiang, S., Zhao, Y., Guan, X., Kuo, C.-C., & Wang, X. E. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.1255
The influence of Generative AI on the transfer of algorithmic skills in students
Kerimbayev, N., Menlibay, Z., Shadiev, R., Mambetakunov, E. M., Kolumbayeva, S., Nurymova, A., & Adamova, K. (2025) · International Conference Automatics and Informatics (ICAI) · doi:10.1109/icai67591.2025.11324568
- Authors
- Hsu, T.-Y., Huang, C.-Y., Rossi, R., Kim, S., Giles, C., & Huang, T.-H.
- Affiliations
- Pennsylvania State University (US)
- Adobe Systems (United States) (US)
- Venue
- Findings of the Association for Computational Linguistics: EMNLP 2023
- Published
- 2023
- Pages
- 5464-5474
- OpenAlex
- W4389518939
- Cited by
- 8
- References
- 23
- Citations by year
- 2026: 3 · 2025: 4 · 2024: 1
- FWCI
- 0.85
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- zero-shot evaluation, GPT-4, large language models, reader comprehension, reference-free evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
