CoChat
Sign Up
PreprintCited by 57Multimodal Machine Learning ApplicationsData Visualization and AnalyticsTopic Modeling

VisText: A Benchmark for Semantically Rich Chart Captioning

Open access
Tang, Benny J., Boggust, Angie, Satyanarayan, Arvind·2023·arXiv
DOI 10.48550/arxiv.2307.05356arXiv 2307.05356
Open this paper in CoChat — free accountOpen PDF

Abstract

Captions that describe or explain charts help improve recall and comprehension of the depicted data and provide a more accessible medium for people with visual disabilities. However, current approaches for automatically generating such captions struggle to articulate the perceptual or cognitive features that are the hallmark of charts (e.g., complex trends and patterns). In response, we introduce VisText: a dataset of 12,441 pairs of charts and captions that describe the charts' construction, report key statistics, and identify perceptual and cognitive phenomena. In VisText, a chart is available as three representations: a rasterized image, a backing data table, and a scene graph -- a hierarchical representation of a chart's visual elements akin to a web page's Document Object Model (DOM). To evaluate the impact of VisText, we fine-tune state-of-the-art language models on our chart captioning task and apply prefix-tuning to produce captions that vary the semantic content they convey. Our models generate coherent, semantically rich captions and perform on par with state-of-the-art chart captioning models across machine translation and text generation metrics. Through qualitative analysis, we identify six broad categories of errors that our models make that can inform future work.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2307.05356

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Research probes like VisText struggle to capture the visual meaning of complex figures.

    “Research probes like VisText [74] are explored, but they struggle to capture the visual meaning of complex figures.”

    From How University Disability Services Professionals Write Image Descriptions for HCI Figures Using Generative AI (2026) · cites this paper as [74]

    The paper’s “Chart Types and the Visualization Design Space” section says VisText covers only univariate bar, area, and line charts, leaving more complex forms for future work. Its “Chart images do not support captioning” section also reports that the image model fails to extract chart information adequately. These passages establish limitations, but the paper does not show that VisText struggles to capture the visual meaning of complex figures specifically. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 61 · Cited by 57

Research works printed but not linked: 65

  1. Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for im- age captioning and visual question answering. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6077-6086, Los Alamitos, CA, USA. IEEE Computer Society. · DOI
  2. Abhijit Balaji, Thuvaarakkesh Ramanathan, and Venkateshwarlu Sonathi. 2018. Chart-text: A fully automated chart image descriptor. ArXiv, abs/1812.10636. · arXiv
  3. Emily M Bender, Timnit Gebru, Angelina McMillan- Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the ACM Confer- ence on Fairness, Accountability, and Transparency (FAccT), pages 610-623.
  4. Cynthia L Bennett, Cole Gleason, Morgan Klaus Scheuerman, Jeffrey P Bigham, Anhong Guo, and Alexandra To. 2021. "it's complicated": Negotiat- ing accessibility and (mis) representation in image descriptions of race, gender, and disability. In Pro- ceedings of the Conference on Human Factors in Computing Systems (CHI), pages 1-19.
  5. Andrea J Bingham and Patricia Witkowsky. 2021. De- ductive and inductive approaches to qualitative data analysis. Analyzing and interpreting qualitative data: After the interview, pages 133-146.
  6. Michelle A Borkin, Zoya Bylinskii, Nam Wook Kim, Constance May Bainbridge, Chelsea S Yeh, Daniel Borkin, Hanspeter Pfister, and Aude Oliva. 2015. Be- yond memorability: Visualization recognition and recall. IEEE Transactions on Visualization and Com- puter Graphics (VIS), 22(1):519-528.
  7. Matthew Brehmer and Tamara Munzner. 2013. A multi- level typology of abstract visualization tasks. IEEE Transactions on Visualization and Computer Graph- ics (VIS), 19(12):2376-2385. · DOI
  8. Charles Chen, Ruiyi Zhang, Sungchul Kim, Scott Co- hen, Tong Yu, Ryan Rossi, and Razvan Bunescu. 2019a. Neural caption generation over figures. In Adjunct Proceedings of the ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the ACM International Sympo- sium on Wearable Computers (UbiComp/ISWC), Ubi- Comp/ISWC '19 Adjunct, page 482-485, New York, NY, USA. Association for Computing Machinery. · DOI
  9. Charles Chen, Ruiyi Zhang, Eunyee Koh, Sungchul Kim, Scott Cohen, and Ryan Rossi. 2020. Figure captioning with relation maps for reasoning. In IEEE Winter Conference on Applications of Computer Vi- sion (WACV), pages 1526-1534. · DOI
  10. Charles Chen, Ruiyi Zhang, Eunyee Koh, Sungchul Kim, Scott Cohen, Tong Yu, Ryan Rossi, and Raz- van Bunescu. 2019b. Figure captioning with rea- soning and sequence-level training. arXiv preprint arXiv:1906.02850. · arXiv
  11. Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. 2021. Unifying vision-and-language tasks via text genera- tion. In Proceedings of the 38th International Con- ference on Machine Learning, volume 139 of Pro- ceedings of Machine Learning Research, pages 1931- 1942. PMLR.
  12. Michael Correll. 2019. Ethical dimensions of visualiza- tion research. In Proceedings of the Conference on Human Factors in Computing Systems (CHI), pages 1-13.
  13. Zhe Cui, Sriram Karthik Badam, M Adil Yalçin, and Niklas Elmqvist. 2019. Datasite: Proactive vi- sual data exploration with computation of insight- based recommendations. Information Visualization, 18(2):251-267.
  14. Seniz Demir, Sandra Carberry, and Kathleen F. Mc- Coy. 2008. Generating textual summaries of bar charts. In Proceedings of the International Natu- ral Language Generation Conference (INLG), INLG '08, page 7-15, USA. Association for Computational Linguistics.
  15. Seniz Demir, Sandra Carberry, and Kathleen F McCoy. 2012. Summarizing information graphics textually. Computational Linguistics, 38(3):527-574.
  16. Seniz Demir, David Oliver, Edward Schwartz, Stephanie Elzer, Sandra Carberry, and Kathleen F McCoy. 2010. Interactive sight into information graphics. In Pro- ceedings of the International Cross Disciplinary Con- ference on Web Accessibility (W4A), pages 1-10.
  17. Zihao Fu, Wai Lam, Anthony Man-Cho So, and Bei Shi. 2021. A theoretical analysis of the repetition problem in text generation. In Proceedings of the Conference on Artificial Intelligence (AAAI).
  18. Margot Hanley, Solon Barocas, Karen Levy, Shiri Azenkot, and Helen Nissenbaum. 2021. Computer vision and conflicting values: Describing people with automated alt text. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pages 543- 554.
  19. Mary Hegarty and Marcel-Adam Just. 1993. Construct- ing mental models of machines from text and dia- grams. Journal of memory and language, 32(6):717- 742.
  20. Ting-Yao Hsu, C Lee Giles, and Ting-Hao Huang. 2021. SciCap: Generating captions for scientific figures. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3258-3264, Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
  21. Jessica Hullman and Nick Diakopoulos. 2011. Visu- alization rhetoric: Framing effects in narrative visu- alization. IEEE Transactions on Visualization and Computer Graphics (VIS), 17(12):2231-2240.
  22. Alessandra Helena Jandrey, Duncan Dubugras Alcoba Ruiz, and Milene Selbach Silveira. 2021. Image descriptions' limitations for people with visual im- pairments: Where are we and where are we going? In Proceedings of the Brazilian Symposium on Hu- man Factors in Computing Systems (IHC), IHC '21, New York, NY, USA. Association for Computing Machinery. · DOI
  23. Crescentia Jung, Shubham Mehta, Atharva Kulkarni, Yuhang Zhao, and Yea-Seul Kim. 2021. Communi- cating visualizations without visuals: Investigation of visualization alternative text for people with visual impairments. IEEE Transactions on Visualization and Computer Graphics (VIS), 28(1):1095-1105.
  24. Daekyoung Jung, Wonjae Kim, Hyunjoo Song, Jeong-in Hwang, Bongshin Lee, Bohyoung Kim, and Jinwook Seo. 2017. Chartsense: Interactive data extraction from chart images. In Proceedings of the Conference on Human Factors in Computing Systems (CHI), CHI '17, page 6706-6717, New York, NY, USA. Associa- tion for Computing Machinery. · DOI
  25. Ju Yeon Jung, Tom Steinberger, Junbeom Kim, and Mark S. Ackerman. 2022. "So What? What's That to Do With Me?" Expectations of People With Visual Im- pairments for Image Descriptions in Their Personal Photo Activities, page 1893-1906. Association for Computing Machinery, New York, NY, USA. · DOI
  26. Shankar Kantharaj, Rixie Tiffany Leong, Xiang Lin, Ahmed Masry, Megh Thakkar, Enamul Hoque, and Shafiq Joty. 2022. Chart-to-text: A large-scale bench- mark for chart summarization. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 4005-4023, Dublin, Ireland. Association for Computational Linguistics.
  27. Daniel Keim, Gennady Andrienko, Jean-Daniel Fekete, Carsten Görg, Jörn Kohlhammer, and Guy Melançon. 2008. Visual analytics: Definition, process, and chal- lenges. In Information visualization, pages 154-175. Springer.
  28. Helen Kennedy, Rosemary Lucy Hill, Giorgia Aiello, and William Allen. 2016. The work that visualisa- tion conventions do. Information, Communication & Society, 19(6):715-735.
  29. Dae Hyun Kim, Enamul Hoque, Juho Kim, and Ma- neesh Agrawala. 2018. Facilitating document read- ing by linking text and tables. In Proceedings of the Annual ACM Symposium on User Interface Software and Technology (UIST), UIST '18, page 423-434, New York, NY, USA. Association for Computing Machinery. · DOI
  30. Dae Hyun Kim, Vidya Setlur, and Maneesh Agrawala. 2021. Towards understanding how readers integrate charts and captions: A case study with line charts. In Proceedings of the Conference on Human Factors in Computing Systems (CHI), pages 1-11.
  31. Ha-Kyung Kong, Zhicheng Liu, and Karrie Karahalios. 2018. Frames and slants in titles of visualizations on controversial topics. In Proceedings of the Confer- ence on Human Factors in Computing Systems (CHI), pages 1-12.
  32. Ha-Kyung Kong, Zhicheng Liu, and Karrie Karahalios. 2019. Trust and recall of information across vary- ing degrees of title-visualization misalignment. In Proceedings of the Conference on Human Factors in Computing Systems (CHI), pages 1-13.
  33. Nicholas Kong and Maneesh Agrawala. 2012. Graph- ical overlays: Using layered elements to aid chart reading. IEEE Transactions on Visualization and Computer Graphics (VIS), 18(12):2631-2638.
  34. Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Wein- berger. 2015. From word embeddings to document distances. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Pro- ceedings of Machine Learning Research (PMLR), pages 957-966, Lille, France. PMLR.
  35. Andrew Large, Jamshid Beheshti, Alain Breuleux, and Andre Renaud. 1995. Multimedia and comprehen- sion: The relationship among text, animation, and captions. Journal of the American society for infor- mation science, 46(5):340-347.
  36. Shahid Latif, Zheng Zhou, Yoon Kim, Fabian Beck, and Nam Wook Kim. 2021. Kori: Interactive synthesis of text and charts in data documents. IEEE Transac- tions on Visualization and Computer Graphics (VIS), 28(1):184-194.
  37. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and com- prehension. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 7871-7880, Online. Association for Computa- tional Linguistics.
  38. Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
  39. Chin-Yew Lin and Franz Josef Och. 2004. ORANGE: a method for evaluating automatic evaluation met- rics for machine translation. In Proceedings of the International Conference on Computational Linguis- tics (COLING), pages 501-507, Geneva, Switzerland. COLING.
  40. Alan Lundgard and Arvind Satyanarayan. 2022. Ac- cessible visualization via natural language descrip- tions: A four-level model of semantic content. IEEE Transactions on Visualization and Computer Graph- ics (VIS), 28(1):1073-1083.
  41. Junyu Luo, Zekun Li, Jinpeng Wang, and Chin-Yew Lin. 2021. Chartocr: Data extraction from charts images via a deep hybrid framework. In 2021 IEEE Win- ter Conference on Applications of Computer Vision (WACV). The Computer Vision Foundation.
  42. Anita Mahinpei, Zona Kostic, and Chris Tanner. 2022. Linecap: Line charts for data visualization captioning models. In IEEE Visualization and Visual Analytics (VIS), pages 35-39. IEEE.
  43. Valerie S Morash, Yue-Ting Siu, Joshua A Miele, Lu- cia Hasty, and Steven Landau. 2015. Guiding novice web workers in making image descriptions using tem- plates. ACM Transactions on Accessible Computing (TACCESS), 7(4):1-21.
  44. Jason Obeid and Enamul Hoque. 2020. Chart-to-text: Generating natural language descriptions for charts by adapting the transformer model. In Proceedings of the International Conference on Natural Language Generation (INLG), pages 138-147, Dublin, Ireland. Association for Computational Linguistics.
  45. Lace M Padilla, Sarah H Creem-Regehr, Mary Hegarty, and Jeanine K Stefanucci. 2018. Decision making with visualizations: a cognitive framework across disciplines. Cognitive research: principles and im- plications, 3(1):1-25.
  46. Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic eval- uation of machine translation. In Proceedings of the Annual Meeting of the Association for Computa- tional Linguistics (ACL), pages 311-318, Philadel- phia, Pennsylvania, USA. Association for Computa- tional Linguistics. · DOI · Paper page
  47. Jorge Poco and Jeffrey Heer. 2017. Reverse- engineering visualizations: Recovering visual encod- ings from chart images. Computer Graphics Forum, 36(3):353-363. · DOI
  48. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Conference on Ma- chine Translation (WMT), pages 186-191, Belgium, Brussels. Association for Computational Linguistics.
  49. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. The Journal of Machine Learning Research, 21(1).
  50. Ehud Reiter and Robert Dale. 1997. Building applied natural language generation systems. Natural Lan- guage Engineering, 3(1):57-87.
  51. Arvind Satyanarayan, Dominik Moritz, Kanit Wong- suphasawat, and Jeffrey Heer. 2016. Vega-lite: A grammar of interactive graphics. IEEE Transac- tions on Visualization and Computer Graphics (VIS), 23(1):341-350.
  52. M Six Silberman, Bill Tomlinson, Rochelle LaPlante, Joel Ross, Lilly Irani, and Andrew Zaldivar. 2018. Responsible research with crowds: pay crowdwork- ers at least minimum wage. Communications of the ACM, 61(3):39-41.
  53. Matthew Snover, Bonnie Dorr, Rich Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of the Conference of the Association for Machine Translation in the Americas (AMTA), pages 223-231, Cambridge, Massachusetts, USA. Association for Machine Translation in the Americas.
  54. Andrea Spreafico and Giuseppe Carenini. 2020. Neural data-driven captioning of time-series line charts. In Proceedings of the International Conference on Ad- vanced Visual Interfaces (AVI), AVI '20, New York, NY, USA. Association for Computing Machinery. · DOI
  55. Arjun Srinivasan, Steven M. Drucker, Alex Endert, and John Stasko. 2019. Augmenting visualizations with interactive data facts to facilitate interpretation and communication. IEEE Transactions on Visualization and Computer Graphics (VIS), 25(1):672-681. · DOI
  56. Chase Stokes, Vidya Setlur, Bridget Cogley, Arvind Satyanarayan, and Marti A Hearst. 2022. Striking a balance: Reader takeaways and preferences when in- tegrating text and charts. IEEE Transactions on Visu- alization and Computer Graphics (VIS), 29(1):1233- 1243.
  57. Umut Sulubacak, Ozan Caglayan, Stig-Arne Grönroos, Aku Rouhe, Desmond Elliott, Lucia Specia, and Jörg Tiedemann. 2020. Multimodal machine translation through visuals and speech. Machine Translation, 34(2-3):97-147. · DOI
  58. Jenny Tang, Eleanor Birrell, and Ada Lerner. 2022. Replication: How well do my results generalize now? the external validity of online privacy and security surveys. In Proceedings of the Symposium on Usable Privacy and Security (SOUPS 2022), pages 367-385.
  59. Jacob VanderPlas, Brian Granger, Jeffrey Heer, Do- minik Moritz, Kanit Wongsuphasawat, Arvind Satya- narayan, Eitan Lees, Ilia Timofeev, Ben Welsh, and Scott Sievert. 2018. Altair: interactive statistical visualizations for python. Journal of Open Source Software, 3(32):1057.
  60. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the International Con- ference on Neural Information Processing Systems (NeurIPS), NIPS'17, page 6000-6010, Red Hook, NY, USA.
  61. O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. 2015. Show and tell: A neural image caption generator. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3156-3164, Los Alami- tos, CA, USA. IEEE Computer Society. · DOI · Paper page
  62. Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. 2021. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359. · arXiv
  63. Leland Wilkinson. 2012. The grammar of graphics. In Handbook of computational statistics, pages 375- 414. Springer.
  64. Sam Wiseman, Stuart Shieber, and Alexander Rush. 2017. Challenges in data-to-document generation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2253-2263, Copenhagen, Denmark. Association for Computational Linguistics.
  65. Linting Xue, Aditya Barua, Noah Constant, Rami Al- Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. ByT5: Towards a token-free future with pre-trained byte-to-byte models. Transac- tions of the Association for Computational Linguis- tics (ACL), 10:291-306.

Authors
Tang, B. J., Boggust, A., & Satyanarayan, A.
Affiliations
MIT Computer Science and Artificial Intelligence Laboratory (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
Pages
7268-7298
ISSN
0736-587X (ISSN-L)
DOI
10.48550/arxiv.2307.05356
arXiv
2307.05356
OpenAlex
W4385570745
Cited by
57
References
61
Citations by year
2026: 13 · 2025: 25 · 2024: 18 · 2023: 1
FWCI
6.07
Type
conference paper
Language
en
Open access
Licence
Locations
  • Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (1900991)
  • U.S. Air Force (FA8750‐19‐2‐1000)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Data Visualization and Analytics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
scene graph, visual accessibility, prefix tuning, language model fine-tuning, data visualization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review