A Descriptive Review of Image Datasets for Accessible Alternative Descriptions in STEM Domains
Open accessAbstract
STEM disciplines rely heavily on visual content, such as charts, diagrams, formulas.These figures are often inaccessible to blind or visually impaired users due to the lack of meaningful alternative text.While automated image captioning has progressed, existing datasets are largely oriented toward general images and overlook the structural and semantic complexity of STEM visual contents.This paper presents a descriptive review of publicly available image datasets, evaluating their applicability for generating accessible descriptions of STEM images.Our analysis reveals major gaps: limited support for complex scientific content, shallow annotations, and little consideration for accessibility standards.We argue for the creation of a specialized dataset with rich, structured annotations aligned with accessibility goals.By identifying critical gaps, this work supports the development of AI tools and datasets that enhance inclusive access to STEM content.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Most existing evaluations rely on datasets and metrics originally developed for generic image captioning or diagram understanding, which do not capture whether a description provides meaningful and trustworthy access for blind and low-vision users.
“As noted by [6], most existing evaluations rely on datasets and metrics originally developed for generic image captioning or diagram understanding, which do not capture whether a description provides meaningful and trustworthy access for blind and low-vision users.”
From Human vs LLM-Generated Alt Text for STEM Image Accessibility: A Comparative Study (2026) · cites this paper as [6]
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · see citing works
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition
(1993) · doi:10.1109/cvpr.1993 · Cited by 17,989 · see citing works
Young, P., Lai, A., Hodosh, M., & Hockenmaier, J. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00166 · Cited by 2,439 · see citing works
Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics
Hodosh, M., Young, P., & Hockenmaier, J. (2013) · Journal of Artificial Intelligence Research · doi:10.1613/jair.3994 · Cited by 1,349 · see citing works
ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning
Masry, A., Long, X., Tan, J. Q., Joty, S., & Hoque, E. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.177 · Cited by 508 · see citing works
Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures
Raffaella Bernardi, B., Çakıcı, R., Elliott, D., Erdem, A., Erdem, E., İkizler-Cinbiş, N., Keller, F., Muscat, A., & Plank, B. (2016) · Journal of Artificial Intelligence Research · doi:10.1613/jair.4900 · Cited by 350 · see citing works
A Diagram is Worth a Dozen Images
Kembhavi, A., Salvato, M., Kolve, E., Seo, M., Hajishirzi, H., & Farhadi, A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_15 · Cited by 290 · see citing works
DVQA: Understanding Data Visualizations via Question Answering
Kafle, K., Price, B. L., Cohen, S. D., & Kanan, C. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00592 · Cited by 260 · see citing works
Canadian Conference on Artificial Intelligence
Klement, W., Flach, P. A., Japkowicz, N., & Matwin, S. (2009) · Cited by 204 · see citing works
Captioning Images Taken by People Who Are Blind
Gurari, D., Zhao, Y., Zhang, M., & Bhattacharya, N. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58520-4_25 · Cited by 182 · see citing works
PlotQA: Reasoning over Scientific Plots
Methani, N., Ganguly, P., Khapra, M. M., & Kumar, P. (2020) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv45572.2020.9093523 · Cited by 165 · see citing works
Accessible Visualization via Natural Language Descriptions: A Four-Level Model of Semantic Content
Lundgard, A., & Satyanarayan, A. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3114770 · Cited by 155 · see citing works
Chart-to-Text: A Large-Scale Benchmark for Chart Summarization
Kantharaj, S., Leong, R. T. K., Lin, X., Masry, A., Thakkar, M., Hoque, E., & Joty, S. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.277 · Cited by 126 · see citing works
Chart-to-Text: Generating Natural Language Descriptions for Charts by Adapting the Transformer Model
Obeid, J., & Hoque, E. (2020) · International Conference on Natural Language Generation · doi:10.18653/v1/2020.inlg-1.20 · Cited by 106 · see citing works
Accessible Visualization via Natural Language Descriptions: A Four-Level Model of Semantic Content
Lundgard, A., & Satyanarayan, A. (2022) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 97 · see citing works
SciCap: Generating Captions for Scientific Figures
Hsu, T.-Y., Giles, C. L., & Huang, T.-H. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.277 · Cited by 59 · see citing works
Designing Tools for High-Quality Alt Text Authoring
Mack, K., Cutrell, E., Lee, B., & Morris, M. R. (2021) · Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3441852.3471207 · Cited by 58 · cited 1× · see citing works
“Therefore, preparing alternative and narrative descriptions is not an easy task and requires competence and accuracy, especially in the STEM field [21, 22].”
Sharma, H., & Padha, D. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10488-2 · Cited by 51 · see citing works
Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models
Li, L., Wang, Y., Xu, R., Wang, P., Feng, X., Kong, L., & Liu, Q. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.775 · Cited by 37 · see citing works
Hayes, C., & Proulx, M. J. (2024) · British Journal of Visual Impairment · doi:10.1177/02646196221149561 · Cited by 32 · see citing works
Evaluating the effectiveness of automatic image captioning for web accessibility
Leotta, M., Mori, F., & Ribaudo, M. (2022) · Universal Access in the Information Society · doi:10.1007/s10209-022-00906-7 · Cited by 32 · see citing works
Toward supporting quality alt text in computing publications
Williams, C., de Greef, L., Harris, E., Findlater, L., Pavel, A., & Bennett, C. (2022) · Proceedings of the 19th International Web for All Conference · doi:10.1145/3493612.3520449 · Cited by 26 · see citing works
College Graduates with Visual Impairments: A Report on Seeking and Finding Employment
Antonelli, K. B., Steverson, A., & O’Mally, J. (2018) · Journal of Visual Impairment & Blindness · doi:10.1177/0145482x1811200104 · Cited by 25 · see citing works
mPLUG-PaperOwl: Scientific Diagram Analysis with the Multimodal Large Language Model
Hu, A., Shi, Y., Xu, H., Ye, J., Ye, Q., Yan, M., Li, C., Qian, Q., Zhang, J., & Huang, F. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681294 · Cited by 18 · see citing works
SciCap+: A Knowledge Augmented Dataset to Study the Challenges of Scientific Figure Captioning
Yang, Z., Dabre, R., Tanaka, H., & Okazaki, N. (2024) · Journal of Natural Language Processing · doi:10.5715/jnlp.31.1140 · Cited by 12 · see citing works
Teaching Cybersecurity to Students with Visual Impairments and Blindness
Hairston, J. R., Williams, T., Smith, D. W., Sabados, W. T., & Forney, S. (2020) · Journal of science education for students with disabilities · doi:10.14448/jsesd.12.0007 · Cited by 9 · see citing works
Splendiani, B., & Ribera, M. (2015) · Procedia Computer Science · doi:10.1016/j.procs.2015.09.250 · Cited by 4 · see citing works
(2014) · doi:10.5040/9798216023166.ch-003 · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 8
- Michael Grubinger, Paul Clough, Henning Müller, and Thomas Deselaers. 2006. The iapr tc-12 benchmark: A new evaluation resource for visual information systems. In International workshop ontoImage, Vol. 2.
- Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Ákos Kádár, Adam Trischler, and Yoshua Bengio. 2017. Figureqa: An annotated figure dataset for visual reasoning. arXiv preprint arXiv:1710.07300 (2017). · arXiv
- Z. Karishma, S. Rohatgi, K.S. Puranik, J. Wu, and C.L. Giles. 2023. Acl-fig: A dataset for scientific figure classification. https://arxiv.org/abs/2301.12293. · arXiv
- Jaeyoung Kim, Jongho Lee, Hong-Jun Choi, Ting-Yao Hsu, Chieh-Yang Huang, Sungchul Kim, Ryan Rossi, Tong Yu, Clyde Lee Giles, Ting-Hao'Kenneth' Huang, et al. 2025. Multi-LLM Collaborative Caption Generation in Scientific Documents. arXiv preprint arXiv:2501.02552 (2025). · arXiv
- Z. Li, X. Yang, K. Choi, W. Zhu, R. Hsieh, H. Kim, J.H. Lim, S. Ji, B. Lee, X. Yan, et al. 2024. MMSCI: A dataset for graduate-level multi-discipline multimodal scientific understanding. arXiv preprint arXiv:2407.04903 (2024). · arXiv
- Vicente Ordonez, Girish Kulkarni, and Tamara Berg. 2011. Im2text: Describing images using 1 million captioned photographs. Advances in neural information processing systems 24 (2011).
- M. Pates, an, A. Balagiu, and C. Alibec. 2018. Visual aids in language education. In International Conference Knowledge-Based Organization, Vol. 24. 356-361.
- J. Zhu, J. Ran, R.K.W. Lee, K. Choo, and Z. Li. 2021. AutoChart: A dataset for chart-to-text generation task. arXiv preprint arXiv:2108.06897 (2021). · arXiv
Cited by
Sorted by citations
Human vs LLM-Generated Alt Text for STEM Image Accessibility: A Comparative Study
Cardia, M., Angileri, L., Poggianti, C., & Leporini, B. (2026) · Proceedings of the 23rd International Web for All Conference · doi:10.1145/3800424.3800457 · Cited by 2 · see citing works
Self-citationA Systematic Survey on Image Description Techniques for STEM Domains
Cardia, M., Angileri, L., Buzzi, M., Galesi, G., & Leporini, B. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2026.2668031 · Cited by 1 · see citing works
Self-citationKimogol, S. S., Nachtnebel, D. A., Safari, M. C., & Sanderson, N. C. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29653-5_17
Evaluation of Human and AI-Generated Image Descriptions for STEM Charts by Varying Expertise Groups
Antonakopoulou, T., Riga, P., & Κουρουπέτρογλου, Γ. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-31285-3_19
Accessible Alt-Text Generation of 2D Graphs with Multimodal LLMs: Comparison and Evaluation
Cardia, M., Poggianti, C., Leporini, B., Pelagatti, S., & Chessa, S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38407-2_59
Self-citationHassan, S., Khalaila, A., Cashman, D., Sharif, A., & Faust, R. (2026) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3797867.3829030
- Authors
- Cardia, M., Buzzi, M., Galesi, G., & Leporini, B.
- Affiliations
- University of Pisa (IT)
- Institute of Informatics and Telematics (IT)
- Venue
- Proceedings of the 16th Biannual Conference of the Italian SIGCHI Chapter
- Publisher
- ACM
- Published
- 2025-10-06
- Pages
- 1-7
- OpenAlex
- W4415156682
- Cited by
- 5
- References
- 29
- Citations by year
- 2026: 5
- FWCI
- 3.78
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Biannual Conference of the Italian SIGCHI Chapter (CHItaly) (publishedVersion, cc-by)
- CINECA IRIS Institutial research information system (University of Pisa) (submittedVersion, cc-by)
- Funders
- STEMMA PRIN project thanks to the funding by the European Union - Next Generation EU (B53D23019500006)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Keywords
- alternative text, accessibility, scientific figures, blind and visually impaired users, structured annotation, accessibility standards
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
