Cited by 43Digital Accessibility for DisabilitiesData Visualization and AnalyticsAcademic Writing and Publishing

FigurA11y: AI Assistance for Writing Scientific Alt Text

Open access
Singh, Nikhil, Wang, Lucy Lu, Bragg, Jonathan·2024·Proceedings of the 29th International Conference on Intelligent User Interfaces

Abstract

High-quality alt text is crucial for making scientific figures accessible to blind and low-vision readers. Crafting complete, accurate alt text is challenging even for domain experts, as published figures often depict complex visual information and readers have varied informational needs. These challenges, along with high diversity in figure types and domain-specific details, also limit the usefulness of fully automated approaches. Consequently, the prevalence of high-quality alt text is very low in scientific papers today. We investigate whether and how human-AI collaborative editing systems can help address the difficulty of writing high-quality alt text for complex scientific figures. We present FigurA11y, an interactive system that generates draft alt text and provides suggestions for author revisions using a pipeline driven by extracted figure and paper metadata. We test two versions, motivated by prior work on visual accessibility and writing support. The base Draft+Revise version provides authors with an automatically generated draft description to revise, along with extracted figure metadata and figure-specific alt text guidelines to support the revision process. The full Interactive Assistance version further adds contextualized suggestions: text snippets to iteratively produce descriptions, and hypothetical user questions with possible answers to reveal potential ambiguities and resolutions. In a study of authors (N=14), we found the system assisted them in efficiently producing descriptive alt text. Generated drafts and interface elements enabled authors to quickly initiate and edit detailed descriptions. Additionally, interactive suggestions from the full system prompted more iteration and highlighted aspects for authors to consider, resulting in greater deviation from the drafts without increased average cognitive load or manual effort.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3640543.3645212

Cites

28 of 28 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Most research in this space has focused on content creators or domain experts who author their figures.

    “Most research in this space has focused on content creators or domain experts who author their figures [4, 26, 48, 71].”

    From How University Disability Services Professionals Write Image Descriptions for HCI Figures Using Generative AI (2026) · cites this paper as [71]

    The Introduction and §3.2 Formative Study describe research with authors writing alt text for their own figures, supporting the claim that this is a focus of the cited paper. The paper does not establish that most research in the space focuses on such authors; its Related Work also discusses automated and crowdsourced approaches. (LLM, conf=0.82)

    Experimental — accuracy not yet measured
  • Existing work shows the need for human subject matter validation for evaluating AI-augmented alt text quality to enhance the interpretation for screen reader users.

    “Existing work also shows the need for human subject matter validation for evaluating AI-augmented alt text quality to enhance the interpretation for screen reader users [71, 87].”

    From How University Disability Services Professionals Write Image Descriptions for HCI Figures Using Generative AI (2026) · cites this paper as [71]

    The Introduction calls for authors to critically review and refine AI-generated descriptions because model errors can produce misleading alt text. Section 7.2 says evaluation of whether collaboratively written alt text improves blind and low-vision readers’ comprehension remains future work. The paper supports human review and the need to evaluate reader outcomes, but does not establish that subject-matter validation enhances screen-reader users’ interpretation. (LLM, conf=0.83)

    Experimental — accuracy not yet measured
  • When LLMs are used as a collaborative tool, they show promise in assisting authors to write alt text.

    “Nevertheless, when LLMs are used as a collaborative tool, they show promise in assisting authors to write alt text [3, 71].”
  • This concern is often attributed to authors' challenges in writing effective alt text for science figures.

    “This concern is often attributed to authors' challenges in writing effective alt text for science figures [71, 79].”
  • Little is known about writing support for non-experts, such as DSO professionals, in figure accessibility.

    “While previous research in figure accessibility has mostly focused on readers [2, 32, 66], or authors of papers [31, 71], little is known about the writing support for non-experts , such as DSO professionals.”

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 28

  1. Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S Weld. 2020. Specter: Document-level representation learning using citation-informed transformers. arXiv preprint arXiv:2004.07180 (2020). · arXiv
  2. Katy Ilonka Gero, Vivian Liu, and Lydia Chilton. 2022. Sparks: Inspiration for science writing using language models. In Designing interactive systems conference. 1002-1019.
  3. Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022. News summarization and evaluation in the era of gpt-3. arXiv preprint arXiv:2209.12356 (2022). · arXiv
  4. Danna Gurari, Yinan Zhao, Meng Zhang, and Nilavra Bhattacharya. 2020. Cap- tioning images taken by people who are blind. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVII 16. Springer, 417-434.
  5. Ting-Yao Hsu, C Lee Giles, and Ting-Hao'Kenneth' Huang. 2021. SciCap: Gener- ating captions for scientific figures. arXiv preprint arXiv:2110.11624 (2021). · arXiv
  6. Kung-Hsiang Huang, Mingyang Zhou, Hou Pong Chan, Yi R Fung, Zhenhailong Wang, Lingyu Zhang, Shih-Fu Chang, and Heng Ji. 2023. Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioning. arXiv preprint arXiv:2312.10160 (2023). · arXiv
  7. Alyssa Hwang, Andrew Head, and Chris Callison-Burch. 2023. Grounded Intuition of GPT-Vision's Abilities with Scientific Images. arXiv preprint arXiv:2311.02069 (2023). · arXiv
  8. Vladimir I Levenshtein et al. 1966. Binary codes capable of correcting deletions, insertions, and reversals. In Soviet physics doklady, Vol. 10. Soviet Union, 707-710.
  9. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 (2023). · arXiv
  10. Fangyu Liu, Julian Martin Eisenschlos, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Wenhu Chen, Nigel Collier, and Yasemin Altun. 2022. DePlot: One-shot visual language reasoning by plot-to-table transla- tion. arXiv preprint arXiv:2212.10505 (2022). · arXiv
  11. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruc- tion tuning. arXiv preprint arXiv:2304.08485 (2023). · arXiv
  12. Ahmed Masry, Parsa Kavehzadeh, Xuan Long Do, Enamul Hoque, and Shafiq Joty. 2023. UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning. arXiv preprint arXiv:2305.14761 (2023). · arXiv
  13. Mickey Nguyen, Matthew Crane, John Romley, and Yannis M Paulus. 2023. Acces- sibility of Figures in Leading Biomedical and Ophthalmology Journals: Analysis of Alternative Text Use. Investigative Ophthalmology & Visual Science 64, 8 (2023), 2806-2806.
  14. OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL] · arXiv
  15. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084 (2019). · arXiv
  16. Jacob Schreiber, Jeffrey Bilmes, and William Stafford Noble. 2020. apricot: Sub- modular selection for data summarization in Python. The Journal of Machine Learning Research 21, 1 (2020), 6474-6479.
  17. Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint arXiv:2303.17580 (2023). · arXiv
  18. Zejiang Shen, Tal August, Pao Siangliulue, Kyle Lo, Jonathan Bragg, Jeff Ham- merbacher, Doug Downey, Joseph Chee Chang, and David Sontag. 2023. Beyond Summarization: Designing AI Support for Real-World Expository Writing Tasks. arXiv preprint arXiv:2304.02623 (2023). · arXiv
  19. Dídac Surís, Sachit Menon, and Carl Vondrick. 2023. Vipergpt: Visual inference via python execution for reasoning. arXiv preprint arXiv:2303.08128 (2023). · arXiv
  20. Benny J Tang, Angie Boggust, and Arvind Satyanarayan. 2023. VisText: A Bench- mark for Semantically Rich Chart Captioning. arXiv preprint arXiv:2307.05356 (2023). · arXiv
  21. Teng Wang, Jinrui Zhang, Junjie Fei, Yixiao Ge, Hao Zheng, Yunlong Tang, Zhe Li, Mingqi Gao, Shanshan Zhao, Ying Shan, et al. 2023. Caption anything: Interactive image description with diverse multimodal controls. arXiv preprint arXiv:2305.02677 (2023). · arXiv
  22. Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. 2023. Visual chatgpt: Talking, drawing and editing with visual foun- dation models. arXiv preprint arXiv:2303.04671 (2023). · arXiv
  23. Xianjun Yang, Yan Li, Xinlu Zhang, Haifeng Chen, and Wei Cheng. 2023. Explor- ing the limits of chatgpt for query or aspect-based text summarization. arXiv preprint arXiv:2302.08081 (2023). · arXiv
  24. Zhishen Yang, Raj Dabre, Hideki Tanaka, and Naoaki Okazaki. 2023. SciCap+: A Knowledge Augmented Dataset to Study the Challenges of Scientific Figure Captioning. arXiv preprint arXiv:2306.03491 (2023). · arXiv
  25. Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. 2023. Mm- react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303.11381 (2023). · arXiv
  26. Haoxuan You, Rui Sun, Zhecan Wang, Long Chen, Gengyu Wang, Hammad A Ayyubi, Kai-Wei Chang, and Shih-Fu Chang. 2023. IdealGPT: Iteratively De- composing Vision and Language Reasoning via Large Language Models. arXiv preprint arXiv:2305.14985 (2023). · arXiv
  27. Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B Hashimoto. 2023. Benchmarking large language models for news summarization. arXiv preprint arXiv:2301.13848 (2023). · arXiv
  28. Deyao Zhu, Jun Chen, Kilichbek Haydarov, Xiaoqian Shen, Wenxuan Zhang, and Mohamed Elhoseiny. 2023. Chatgpt asks, blip-2 answers: Automatic questioning towards enriched visual descriptions. arXiv preprint arXiv:2303.06594 (2023). · arXiv

Cited by

Sorted by citations

Authors
Singh, N., Wang, L. L., & Bragg, J.
Affiliations
  • MIT Media Lab (US)
  • University of Washington (US)
  • Allen Institute for Artificial Intelligence (US)
Venue
Proceedings of the 29th International Conference on Intelligent User Interfaces
Publisher
ACM
Published
2024-03-18
Pages
886-906
OpenAlex
W4393970562
Cited by
42
References
27
Citations by year
2026: 17 · 2025: 21 · 2024: 4
FWCI
22.94
Type
conference paper
Language
en
Open access
Locations
  • ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion, cc-by)
  • DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc)
Topics
  • Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Data Visualization and Analytics
  • Social Sciences › Arts and Humanities › History and Philosophy of Science › Academic Writing and Publishing
Keywords
alt text, scientific figures, accessibility, visual accessibility, human-AI collaboration, AI-assisted writing, cognitive load

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review