Summaries as Captions: Generating Figure Captions for Scientific Documents with Automated Text Summarization
Open accessAbstract
Good figure captions help paper readers understand complex scientific figures. Unfortunately, even published papers often have poorly written captions. Automatic caption generation could aid paper writers by providing good starting captions that can be refined for better quality. Prior work often treated figure caption generation as a vision-to-language task. In this paper, we show that it can be more effectively tackled as a text summarization task in scientific documents. We fine-tuned PEGASUS, a pre-trained abstractive summarization model, to specifically summarize figure-referencing paragraphs (e.g., "Figure 3 shows...") into figure captions. Experiments on large-scale arXiv figures show that our method outperforms prior vision methods in both automatic and human evaluations. We further conducted an in-depth investigation focused on two key challenges: (i) the common presence of low-quality author-written captions and (ii) the lack of clear standards for good captions. Our code and data are available at: https://github.com/Crowd-AI-Lab/Generating-Figure-Captions-as-a-Text-Summarization-Task.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Most published work lacks accurate alt text for visual content (e.g., scientific figures) and requires alt text remediation.
“Despite the widespread acknowledgment of these accessibility needs, most published work lacks accurate alt text for visual content (e.g., scientific figures) [14, 41] and requires alt text remediation [33].”
From How University Disability Services Professionals Write Image Descriptions for HCI Figures Using Generative AI (2026) · cites this paper as [33]
The Introduction and Quality of Real-World Data sections report that roughly half of the sampled author-written line-chart captions in arXiv cs.CL papers were unhelpful. That supports a concern about scientific figure descriptions, but the paper does not establish that most published work lacks accurate alt text across visual content. The remediation clause is attached to a different citation in the citing sentence. (LLM, conf=0.85)
Experimental — accuracy not yet measuredWriting alt text becomes a nontrivial task without domain expertise, even for experienced accessibility professionals.
“Therefore, writing alt text becomes a nontrivial task without domain expertise, even for experienced accessibility professionals [25, 33, 79].”
From How University Disability Services Professionals Write Image Descriptions for HCI Figures Using Generative AI (2026) · cites this paper as [33]
In “Main Human Evaluation with Domain Experts” (page unavailable), the paper says people without basic domain understanding have difficulty evaluating scientific figure captions. Its “Quality of Real-World Data” section also reports that many author-written captions were judged unhelpful. These passages support the general difficulty of producing useful captions, but the paper does not establish that experienced accessibility professionals struggle to write alt text without domain expertise. (LLM, conf=0.78)
Experimental — accuracy not yet measuredPeople find it hard to produce effective alt text for inaccessible figures.
“and people find it hard to produce effective alt text for those [33, 48, 53].”
From How University Disability Services Professionals Write Image Descriptions for HCI Figures Using Generative AI (2026) · cites this paper as [33]
The Introduction and Quality of Real-World Data sections report that many author-written captions for scientific line charts were unhelpful, supporting the general point that effective figure descriptions are difficult to produce. The paper evaluates captions, however, not alt text specifically or figures identified as inaccessible. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 53
- Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. 2022. BEit: BERT pre-training of image transform- ers. In International Conference on Learning Repre- sentations.
- Regina Barzilay and Mirella Lapata. 2005. Collective content selection for concept-to-text generation. In Proceedings of Human Language Technology Con- ference and Conference on Empirical Methods in Natural Language Processing, pages 331-338, Van- couver, British Columbia, Canada. Association for Computational Linguistics.
- Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciB- ERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP), pages 3615- 3620, Hong Kong, China. Association for Computa- tional Linguistics. · DOI · Paper page
- Constance M. Biegel and Prashant V. Kamat. 2019. Ten tips for capturing figures with captions. ACS Energy Letters, 4(3):637-638. · DOI
- Charles Chen, Ruiyi Zhang, Sungchul Kim, Scott Co- hen, Tong Yu, Ryan Rossi, and Razvan Bunescu. 2019. Neural caption generation over figures. In Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Com- puting and Proceedings of the 2019 ACM Inter- national Symposium on Wearable Computers, Ubi- Comp/ISWC '19 Adjunct, page 482-485, New York, NY, USA. Association for Computing Machinery. · DOI
- Charles Chen, Ruiyi Zhang, Eunyee Koh, Sungchul Kim, Scott Cohen, and Ryan Rossi. 2020a. Figure captioning with relation maps for reasoning. In 2020 IEEE Winter Conference on Applications of Com- puter Vision (WACV), pages 1526-1534. · DOI
- Wenhu Chen, Jianshu Chen, Yu Su, Zhiyu Chen, and William Yang Wang. 2020b. Logical natural lan- guage generation from open-domain tables. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 7929- 7942, Online. Association for Computational Lin- guistics. · DOI
- Zhiyu Chen, Wenhu Chen, Hanwen Zha, Xiyou Zhou, Yunkai Zhang, Sairam Sundaresan, and William Yang Wang. 2020c. Logic2Text: High-fidelity natural lan- guage generation from logical forms. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2096-2111, Online. Association for Computational Linguistics. · DOI
- Marc Corio and Guy Lapalme. 1999. Generation of texts for information graphics. In Proceedings of the 7th European Workshop on Natural Language Generation EWNLG'99, pages 49-58.
- Seniz Demir, Sandra Carberry, and Kathleen F McCoy. 2008. Generating textual summaries of bar charts. In Proceedings of the Fifth International Natural Language Generation Conference, pages 7-15.
- Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR.
- Zi-Yi Dou and Graham Neubig. 2021. Word alignment by fine-tuning embeddings on parallel corpora. In Conference of the European Chapter of the Associa- tion for Computational Linguistics (EACL).
- Mortaza Doulaty, Oscar Saz, and Thomas Hain. 2015. Data-selective transfer learning for multi-domain speech recognition. In INTERSPEECH 2015, 16th Annual Conference of the International Speech Com- munication Association, Dresden, Germany, Septem- ber 6-10, 2015, pages 2897-2901. ISCA. · DOI
- Charles G Durbin. 2004. Effective use of tables and figures in abstracts, presentations, and papers. Respi- ratory care, 49(10):1233-1237.
- Massimo Fasciano and Guy Lapalme. 1996. Postgraphe: a system for the generation of statistical graphics and text. In Eighth International Natural Language Generation Workshop.
- Andrew Gelman, Cristian Pasarica, and Rahul Dodhia. 2002. Let's practice what we preach: turning tables into graphs. The American Statistician, 56(2):121- 130.
- Li Gong, Josep Crego, and Jean Senellart. 2019. En- hanced transformer model for data-to-text generation. In Proceedings of the 3rd Workshop on Neural Gen- eration and Translation, pages 148-156, Hong Kong. Association for Computational Linguistics. · DOI
- James Hartley. 2003. Single authors are not alone: Col- leagues often help. Journal of Scholarly Publishing, 34(2):108-113.
- Ting-Yao Hsu, C Lee Giles, and Ting-Hao Huang. 2021. SciCap: Generating captions for scientific figures. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3258-3264, Punta Cana, Dominican Republic. Association for Compu- tational Linguistics. · DOI · Paper page
- Zhuoxuan Jiang, Ziming Huang, Dong Sheng Li, and Xian-Ling Mao. 2019. Dialogact2vec: Towards end- to-end dialogue agent by multi-task representation learning. CoRR, abs/1911.04088. · arXiv
- Kushal Kafle, Brian Price, Scott Cohen, and Christo- pher Kanan. 2018. Dvqa: Understanding data visual- izations via question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5648-5656.
- Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Ákos Kádár, Adam Trischler, and Yoshua Bengio. 2017. Figureqa: An annotated figure dataset for visual reasoning. Sixth International Conference on Learning Representations Workshop.
- Shankar Kantharaj, Rixie Tiffany Leong, Xiang Lin, Ahmed Masry, Megh Thakkar, Enamul Hoque, and Shafiq Joty. 2022. Chart-to-text: A large-scale bench- mark for chart summarization. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 4005-4023, Dublin, Ireland. Association for Compu- tational Linguistics. · DOI
- Minghao Li, Tengchao Lv, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, and Furu Wei. 2021. Trocr: Transformer-based optical character recognition with pre-trained models. arXiv preprint arXiv:2109.10282. · arXiv
- Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
- Yixin Liu and Pengfei Liu. 2021. SimCLS: A sim- ple framework for contrastive learning of abstractive summarization. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 1065-1072, Online. Association for Computational Linguistics. · DOI
- Alan Lundgard and Arvind Satyanarayan. 2021. Ac- cessible visualization via natural language descrip- tions: A four-level model of semantic content. IEEE transactions on visualization and computer graphics, 28(1):1073-1083.
- A. Mahinpei, Z. Kostic, and C. Tanner. 2022. Linecap: Line charts for data visualization captioning mod- els. In 2022 IEEE Visualization and Visual Analytics (VIS), pages 35-39, Los Alamitos, CA, USA. IEEE Computer Society. · DOI
- Nitesh Methani, Pritha Ganguly, Mitesh M. Khapra, and Pratyush Kumar. 2020. Plotqa: Reasoning over scientific plots. In Proceedings of the IEEE/CVF Win- ter Conference on Applications of Computer Vision (WACV). microsoft. 2022. Blingfire. https://github.com/mic rosoft/BlingFire.
- Vibhu O Mittal, Johanna D Moore, Giuseppe Carenini, and Steven Roth. 1998. Describing complex charts in natural language: A caption generation system. Computational Linguistics, 24(3):431-467.
- Priscilla Moraes, Gabriel Sina, Kathy McCoy, and San- dra Carberry. 2014. Generating summaries of line graphs. In Proceedings of the 8th International Natu- ral Language Generation Conference (INLG), pages 95-98.
- Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çaglar Gulçehre, and Bing Xiang. 2016. Abstrac- tive text summarization using sequence-to-sequence RNNs and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Lan- guage Learning, pages 280-290, Berlin, Germany. Association for Computational Linguistics. · DOI · Paper page
- Jason Obeid and Enamul Hoque. 2020. Chart-to-text: Generating natural language descriptions for charts by adapting the transformer model. In Proceedings of the 13th International Conference on Natural Lan- guage Generation, pages 138-147, Dublin, Ireland. Association for Computational Linguistics.
- Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann, Man- aal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipan- jan Das. 2020. ToTTo: A controlled table-to-text generation dataset. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 1173-1186, Online. As- sociation for Computational Linguistics. · DOI · Paper page
- Obioma Pelka, Asma Ben Abacha, Alba García Seco de Herrera, Janadhip Jacutprakart, Christoph M Friedrich, and Henning Müller. 2021. Overview of the ImageCLEFmed 2021 concept & caption pre- diction task. In CLEF2021 Working Notes, CEUR Workshop Proceedings, Bucharest, Romania. CEUR- WS.org.
- Xin Qian, Eunyee Koh, Fan Du, Sungchul Kim, Joel Chan, Ryan A Rossi, Sana Malik, and Tak Yeon Lee. 2021. Generating accurate caption units for figure captioning. In Proceedings of the Web Conference 2021, pages 2792-2804.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
- Ehud Reiter, Somayajulu Sripada, Jim Hunter, Jin Yu, and Ian Davy. 2005. Choosing words in computer- generated weather forecasts. Artificial Intelligence, 167(1-2):137-169.
- Joseph Lee Rodgers and W. Alan Nicewander. 1988. Thirteen ways to look at the correlation coefficient. The American Statistician, 42(1):59-66. · DOI
- Nicolas P. Rougier, Michael Droettboom, and Philip E. Bourne. 2014. Ten simple rules for better figures. PLOS Computational Biology, 10(9):1-7. · DOI
- Noah Siegel, Zachary Horvitz, Roie Levin, Santosh Divvala, and Ali Farhadi. 2016. Figureseer: Pars- ing result-figures in research papers. In European Conference on Computer Vision, pages 664-680. Springer.
- Chase Stokes and Marti Hearst. 2022. Why more text is (often) better: Themes from reader preferences for integration of charts and text. The NLVIZ workshop at IEEE Transaction on Visualization and Graphics conference.
- Yixuan Su, David Vandyke, Sihui Wang, Yimai Fang, and Nigel Collier. 2021. Plan-then-generate: Con- trolled data-to-text generation via planning. In Find- ings of the Association for Computational Linguis- tics: EMNLP 2021, pages 895-909, Punta Cana, Do- minican Republic. Association for Computational Linguistics. · DOI
- Simeng Sun, Ori Shapira, Ido Dagan, and Ani Nenkova. 2019. How to compare summarizers without target length? pitfalls, solutions and re-examination of the neural summarization literature. In Proceedings of the Workshop on Methods for Optimizing and Eval- uating Neural Language Generation, pages 21-29, Minneapolis, Minnesota. Association for Computa- tional Linguistics. · DOI
- Jie Wang and Xinyan Zhao. 2019. Theme-aware gen- eration model for chinese lyrics. arXiv preprint arXiv:1906.02134. · arXiv
- Sam Wiseman, Stuart Shieber, and Alexander Rush. 2017. Challenges in data-to-document generation. In Proceedings of the 2017 Conference on Empiri- cal Methods in Natural Language Processing, pages 2253-2263, Copenhagen, Denmark. Association for Computational Linguistics. · DOI · Paper page
- Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transform- ers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
- Jiacheng Xu, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Discourse-aware neural extractive text sum- marization. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 5021-5031, Online. Association for Computa- tional Linguistics. · DOI
- Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Pe- ter Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In In- ternational Conference on Machine Learning, pages 11328-11339.
- PMLR.
- Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Chris- tian M. Meyer, and Steffen Eger. 2019. MoverScore: Text generation evaluating with contextualized em- beddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Interna- tional Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP), pages 563-578, Hong Kong, China. Association for Computational Lin- guistics. · DOI · Paper page
- Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. 2020. Extractive summarization as text matching. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6197-6208, Online. Association for Computational Linguistics. · DOI
- Jiawen Zhu, Jinye Ran, Roy Ka-Wei Lee, Zhi Li, and Kenny Choo. 2021. AutoChart: A dataset for chart- to-text generation task. In Proceedings of the Inter- national Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 1636- 1644, Held Online. INCOMA Ltd.
Cited by
Sorted by citations
Raees, M., Iwamoto, Y., Papangelis, K., Heard, J., & Tigwell, G. W. (2026) · arXiv · doi:10.48550/arxiv.2602.08937 · Cited by 1 · see citing works
“Such findings suggest that, rather than automating alt text generation [32], calibrating DSO professionals’ trust and reliance on AI assistance is crucial, for instance, by enhancing interactions/conversations to improve and refine outputs to maintain their accuracy.”
Influential
- Authors
- Huang, C.-Y., Hsu, T.-Y., Rossi, R., Nenkova, A., Kim, S., Chan, G. Y.-Y., Koh, E., Giles, C. L., & Huang, T.-H. '.
- Affiliations
- Pennsylvania State University (US)
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2302.12324
- OpenAlex
- W4322718327
- Cited by
- 0
- References
- 0
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- automatic caption generation, text summarization, scientific documents, Pegasus, abstractive summarization, arXiv papers
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
