CoChat
Sign Up
Cited by 12Academic Writing and PublishingData Visualization and AnalyticsMultimodal Machine Learning Applications

SciCapenter: Supporting Caption Composition for Scientific Figures with Machine-Generated Captions and Ratings

Open access
Hsu, Ting-Yao, Huang, Chieh-Yang, Huang, Shih-Hong, Rossi, Ryan, Kim, Sungchul, Yu, Tong, Giles, C Lee, Huang, Ting-Hao Kenneth·2024·Extended Abstracts of the CHI Conference on Human Factors in Computing Systems
DOI 10.1145/3613905.3650738
Open this paper in CoChat — free accountOpen PDF

Abstract

Crafting effective captions for figures is important. Readers heavily depend on these captions to grasp the figure’s message. However, despite a well-developed set of AI technologies for figures and captions, these have rarely been tested for usefulness in aiding caption writing. This paper introduces SciCapenter, an interactive system that puts together cutting-edge AI technologies for scientific figure captions to aid caption composition. SciCapenter generates a variety of captions for each figure in a scholarly article, providing scores and a comprehensive checklist to assess caption quality across multiple critical aspects, such as helpfulness, OCR mention, key takeaways, and visual properties reference. Users can directly edit captions in SciCapenter, resubmit for revised evaluations, and iteratively refine them. A user study with Ph.D. students indicates that SciCapenter significantly lowers the cognitive load of caption writing. Participants’ feedback further offers valuable design insights for future systems aiming to enhance caption writing.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3613905.3650738

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Little is known about writing support for non-experts, such as DSO professionals, in figure accessibility.

    “While previous research in figure accessibility has mostly focused on readers [2, 32, 66], or authors of papers [31, 71], little is known about the writing support for non-experts , such as DSO professionals.”

    From How University Disability Services Professionals Write Image Descriptions for HCI Figures Using Generative AI (2026) · cites this paper as [31]

    The Introduction (page unavailable) says caption-writing technologies have rarely been studied in a writing context and describes support for authors of scholarly papers. That supports the cited paper as an example of author-focused work, but it does not assess writing support for non-experts or DSO professionals specifically. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Hsu et al. explored that automated alt text caption generation using an AI-based method is useful for writers to reduce their effort.

    “For instance, Hsu et al. [31] explored that automated alt text caption generation using an AI-based method is useful for writers to reduce their effort.”

    From How University Disability Services Professionals Write Image Descriptions for HCI Figures Using Generative AI (2026) · cites this paper as [31]

    The Findings section (page not provided) reports that SciCapenter reduced writers’ effort when composing scientific figure captions. The paper studies figure captions, not alt text, so the claim’s alt-text-specific application is unsupported. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 46

  1. Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciBERT: A pretrained language model for scientific text. arXiv preprint arXiv:1903.10676 (2019). · arXiv
  2. Yu-Ying Chang and John M Swales. 2014. Informal elements in English academic writing: threats or opportunities for advanced non-native speakers? In Writing: Texts, processes and practices. Routledge, 145-167.
  3. Charles Chen, Ruiyi Zhang, Eunyee Koh, Sungchul Kim, Scott Cohen, and Ryan Rossi. 2020. Figure captioning with relation maps for reasoning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 1537-1545.
  4. Jiwon Choi and Jaemin Jo. 2022. Intentable: A mixed-initiative system for intent- based chart captioning. In 2022 IEEE Visualization and Visual Analytics (VIS). IEEE, 40-44.
  5. Jinho Choi, Sanghun Jung, Deok Gun Park, Jaegul Choo, and Niklas Elmqvist. 2019. Visualizing for the non-visual: Enabling the visually impaired to use visualization. In Computer Graphics Forum, Vol. 38. Wiley Online Library, 249- 260.
  6. Christopher Clark and Santosh Divvala. 2016. Pdffigures 2.0: Mining figures from research papers. In Proceedings of the 16th ACM/IEEE-CS on Joint Conference on Digital Libraries. 143-152.
  7. Susan M Conrad. 1996. Investigating academic texts with corpus-based tech- niques: An example from biology. Linguistics and education 8, 3 (1996), 299-326.
  8. Ignacio Garcia and María Isabel Pena. 2011. Machine translation-assisted lan- guage learning: writing for beginners. Computer Assisted Language Learning 24, 5 (2011), 471-487.
  9. Yashesh Gaur, Walter S Lasecki, Florian Metze, and Jeffrey P Bigham. 2016. The effects of automatic speech recognition quality on human transcription latency. In Proceedings of the 13th International Web for All Conference. 1-8.
  10. Li Gong, Josep Crego, and Jean Senellart. 2019. Enhanced Transformer Model for Data-to-Text Generation. In Proceedings of the 3rd Workshop on Neural Gener- ation and Translation, Alexandra Birch, Andrew Finch, Hiroaki Hayashi, Ioan- nis Konstas, Thang Luong, Graham Neubig, Yusuke Oda, and Katsuhito Su- doh (Eds.). Association for Computational Linguistics, Hong Kong, 148-156. https://doi.org/10.18653/v1/D19-5615 · DOI
  11. Mary Hegarty and Marcel-Adam Just. 1993. Constructing mental models of machines from text and diagrams. Journal of memory and language 32, 6 (1993), 717-742.
  12. Sameera Horawalavithana, Sai Munikoti, Ian Stewart, and Henry Kvinge. 2023. Scitune: Aligning large language models with scientific multimodal instructions. arXiv preprint arXiv:2307.01139 (2023). · arXiv
  13. Ting-Yao Hsu, C Lee Giles, and Ting-Hao Huang. 2021. SciCap: Generating Captions for Scientific Figures. In Findings of the Association for Computational Linguistics: EMNLP 2021. Association for Computational Linguistics, Punta Cana, Dominican Republic, 3258-3264. https://doi.org/10.18653/v1/2021.findings- emnlp.277 · DOI · Paper page
  14. Ting-Yao Hsu, Chieh-Yang Huang, Ryan Rossi, Sungchul Kim, C. Giles, and Ting- Hao Huang. 2023. GPT-4 as an Effective Zero-Shot Evaluator for Scientific Figure Captions. In Findings of the Association for Computational Linguistics: EMNLP 2023,
  15. Chieh-Yang Huang, Ting-Yao Hsu, Ryan Rossi, Ani Nenkova, Sungchul Kim, Gromit Yeuk-Yin Chan, Eunyee Koh, Clyde Lee Giles, and Ting-Hao'Kenneth' Huang. 2023. Summaries as Captions: Generating Figure Captions for Scientific Documents with Automated Text Summarization. arXiv preprint arXiv:2302.12324 (2023). · arXiv
  16. KV Jobin, Ajoy Mondal, and CV Jawahar. 2019. Docfigure: A dataset for scientific document figure classification. In 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW), Vol. 1. IEEE, 74-79.
  17. Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. 2018. Dvqa: Understanding data visualizations via question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition. 5648-5656.
  18. Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Ákos Kádár, Adam Trischler, and Yoshua Bengio. 2017. Figureqa: An annotated figure dataset for visual reasoning. arXiv preprint arXiv:1710.07300 (2017). · arXiv
  19. Shankar Kantharaj, Xuan Long Do, Rixie Tiffany Leong, Jia Qing Tan, Enamul Hoque, and Shafiq Joty. 2022. OpenCQA: Open-ended Question Answering with Charts. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (Eds.). Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 11817-11837. https://doi.org/10.18653/v1/2022.emnlp-main.811 · DOI
  20. Shankar Kantharaj, Rixie Tiffany Leong, Xiang Lin, Ahmed Masry, Megh Thakkar, Enamul Hoque, and Shafiq Joty. 2022. Chart-to-Text: A Large-Scale Benchmark for Chart Summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (Eds.). Association for Computational Linguistics, Dublin, Ireland, 4005-4023. https://doi.org/10.18653/v1/2022.acl-long.277 · DOI
  21. Zeba Karishma, Shaurya Rohatgi, Kavya Shrinivas Puranik, Jian Wu, and C Lee Giles. 2023. ACL-Fig: A Dataset for Scientific Figure Classification. arXiv preprint arXiv:2301.12293 (2023). · arXiv
  22. Dae Hyun Kim, Vidya Setlur, and Maneesh Agrawala. 2021. Towards Under- standing How Readers Integrate Charts and Captions: A Case Study with Line Charts. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI '21).
  23. Dae Hyun Kim, Vidya Setlur, and Maneesh Agrawala. 2021. Towards understand- ing how readers integrate charts and captions: A case study with line charts. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1-11.
  24. Philipp Koehn and Barry Haddow. 2009. Interactive assistance to human trans- lators using statistical machine translation methods. In Proceedings of Machine Translation Summit XII: Papers.
  25. Andrew Large, Jamshid Beheshti, Alain Breuleux, and Andre Renaud. 1995. Multi- media and comprehension: The relationship among text, animation, and captions. Journal of the American society for information science 46, 5 (1995), 340-347.
  26. Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Mar- tin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. 2023. Pix2struct: Screenshot parsing as pretraining for visual lan- guage understanding. In International Conference on Machine Learning. PMLR, 18893-18912.
  27. Shengzhi Li and Nima Tajbakhsh. 2023. Scigraphqa: A large-scale synthetic multi-turn question-answering dataset for scientific graphs. arXiv preprint arXiv:2308.03349 (2023). · arXiv
  28. Yanna Lin, Haotian Li, Leni Yang, Aoyu Wu, and Huamin Qu. 2023. Inksight: Leveraging sketch interaction for documenting chart findings in computational notebooks. IEEE Transactions on Visualization and Computer Graphics (2023).
  29. Can Liu, Yuhan Guo, and Xiaoru Yuan. 2023. Autotitle: An interactive title generator for visualizations. IEEE Transactions on Visualization and Computer Graphics (2023).
  30. Fangyu Liu, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Yasemin Altun, Nigel Collier, and Julian Martin Eisenschlos. 2022. Matcha: Enhancing visual language pretraining with math reasoning and chart derendering. arXiv preprint arXiv:2212.09662 (2022). · arXiv
  31. Anita Mahinpei, Zona Kostic, and Chris Tanner. 2022. Linecap: Line charts for data visualization captioning models. In 2022 IEEE Visualization and Visual Analytics (VIS). IEEE, 35-39.
  32. Ahmed Masry, Parsa Kavehzadeh, Xuan Long Do, Enamul Hoque, and Shafiq Joty. 2023. UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning. arXiv preprint arXiv:2305.14761 (2023). · arXiv
  33. Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244 (2022). · arXiv
  34. Florian Mathis, John H Williamson, Kami Vaniea, and Mohamed Khamis. 2021. Fast and secure authentication in virtual reality using coordinated 3d manipula- tion and pointing. ACM Transactions on Computer-Human Interaction (ToCHI) 28, 1 (2021), 1-44.
  35. Gwen C Nugent. 1983. Deaf students' learning from captioned instruction: The relationship between the visual and caption display. The Journal of Special Education 17, 2 (1983), 227-234.
  36. Jason Obeid and Enamul Hoque. 2020. Chart-to-Text: Generating Natural Lan- guage Descriptions for Charts by Adapting the Transformer Model. In Proceedings of the 13th International Conference on Natural Language Generation, Brian Davis, Yvette Graham, John Kelleher, and Yaji Sripada (Eds.). Association for Computa- tional Linguistics, Dublin, Ireland, 138-147. https://doi.org/10.18653/v1/2020.inlg- 1.20 · DOI
  37. OpenAI. 2023. GPT-4V(ision) System Card. https://api.semanticscholar.org/ CorpusID:263218031
  38. William D Page. 1974. The author and the reader in writing and reading. Research in the Teaching of English 8, 2 (1974), 170-183.
  39. Xin Qian, Eunyee Koh, Fan Du, Sungchul Kim, Joel Chan, Ryan A Rossi, Sana Malik, and Tak Yeon Lee. 2021. Generating accurate caption units for figure captioning. In Proceedings of the Web Conference 2021. 2792-2804.
  40. Noah Siegel, Zachary Horvitz, Roie Levin, Santosh Divvala, and Ali Farhadi. 2016. Figureseer: Parsing result-figures in research papers. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VII 14. Springer, 664-680.
  41. Ashish Singh, Prateek Agarwal, Zixuan Huang, Arpita Singh, Tong Yu, Sungchul Kim, Victor Bursztyn, Nikos Vlassis, and Ryan A Rossi. 2023. FigCaps-HF: A Figure-to-Caption Generative Framework and Benchmark with Human Feedback. arXiv preprint arXiv:2307.10867 (2023). · arXiv
  42. Chase Stokes, Vidya Setlur, Bridget Cogley, Arvind Satyanarayan, and Marti A Hearst. 2022. Striking a balance: reader takeaways and preferences when inte- grating text and charts. IEEE Transactions on Visualization and Computer Graphics 29, 1 (2022), 1233-1243.
  43. Benny J Tang, Angie Boggust, and Arvind Satyanarayan. 2023. Vistext: A bench- mark for semantically rich chart captioning. arXiv preprint arXiv:2307.05356 (2023). · arXiv
  44. Yanan Wang and Yea-Seul Kim. 2023. Making data-driven articles more accessible: An active preference learning approach to data fact personalization. In Proceedings of the 2023 ACM Designing Interactive Systems Conference. 1353-1366.
  45. Zhishen Yang, Raj Dabre, Hideki Tanaka, and Naoaki Okazaki. 2023. SciCap+: A Knowledge Augmented Dataset to Study the Challenges of Scientific Figure Captioning. arXiv preprint arXiv:2306.03491 (2023). · arXiv
  46. Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Yuhao Dan, Chenlin Zhao, Guohai Xu, Chenliang Li, Junfeng Tian, et al. 2023. mplug-docowl: Modu- larized multimodal large language model for document understanding. arXiv preprint arXiv:2307.02499 (2023). · arXiv

Cited by

Sorted by citations

Authors
Hsu, T.-Y., Huang, C.-Y., Huang, S.-H., Rossi, R., Kim, S., Yu, T., Giles, C. L., & Huang, T.-H. K.
Affiliations
  • Pennsylvania State University (US)
  • Adobe Systems (United States) (US)
ORCID
Venue
Extended Abstracts of the CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2024-05-02
Pages
1-9
DOI
10.1145/3613905.3650738
OpenAlex
W4393247979
Cited by
12
References
3
Citations by year
2026: 4 · 2025: 6 · 2024: 2
FWCI
49.41
Type
conference paper
Language
en
Open access
Licence
Locations
  • CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
  • arXiv (Cornell University)
Topics
  • Social Sciences › Arts and Humanities › History and Philosophy of Science › Academic Writing and Publishing
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Data Visualization and Analytics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
cognitive load, visual features, journal articles

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review