Human vs LLM-Generated Alt Text for STEM Image Accessibility: A Comparative Study
Open accessAbstract
Image descriptions are a fundamental aspect of web accessibility, allowing blind and low-vision users to access visual information through assistive technologies such as screen readers. Despite their importance, high-quality alternative text is often missing or inadequate, especially for complex images like diagrams, graphs, and other STEM-related content. Recent advances in generative artificial intelligence and large language models (LLMs) have renewed interest in automatically generating image descriptions, raising the question of whether these systems can reliably support accessibility at scale.
AI snapshot
Written from the paper’s full text.
TL;DR
This paper compares human-written and LLM-generated alt text for 73 STEM automata diagrams under batch and single-image conditions, using automated metrics plus human-centered structural and hallucination analysis. It finds that LLMs often produce fluent descriptions but remain structurally unreliable, and that BLEU, METEOR, ROUGE-L, and CLIPScore only partially capture accessibility-critical errors.
Key findings
- In batch mode, GPT-OSS 120b achieved the highest State Coverage (100.0 ± 0.0) and Transition Coverage (97.6 ± 6.6), with 20 total hallucinations; Gemini had 98.8% SC, 92.2% TC, and 13 hallucinations but failed to provide an alternative description for 38 images; Claude had much lower TC (68.7 ± 30.8), 104 hallucinations, and 16 catastrophic failures.
- In single-image mode, Gemini achieved 100.0% SC, 99.2 ± 4.4% TC, and the lowest hallucination count (10); GPT-5.2 had 98.6 ± 4.3% TC, 90.4 ± 9.4% ISI, and 18 hallucinations; Claude had 88.2 ± 15.9% TC and 204 hallucinations.
- Automated metrics diverged from structural quality: in single-image mode Gemini had the highest BLEU-4 (0.051), METEOR (0.257), and ROUGE-L (0.369), while Claude had the highest CLIPScore (22.31) despite the worst TC and hallucination count. In batch mode GPT-OSS 120b had the highest BLEU-4 (0.033), METEOR (0.186), and ROUGE-L (0.340), while Gemini had the highest CLIPScore (23.41).
- The paper identifies two recurring failure modes: over-description of irrelevant physical details, and under-grounded abstraction, including Gemini batch outputs for at least 18 images that described a plausible but not visually present automaton.
Methods
- Dataset: 73 automata diagrams from undergraduate computer science lecture slides, each paired with a human-written, accessibility-oriented alternative description produced by the four authors following W3C and Diagram Center guidelines in compliance with WCAG 2.2 Level AA; descriptions were domain-level rather than geometric.
- Two experimental conditions: single-image interactive description with ChatGPT 5.2, Gemini 3 Pro, and Claude Sonnet 4.5, each image processed independently; and batch description with Gemini 3 Pro, Claude 3.5 Sonnet, and GPT-OSS 120b, all images processed in one structured output using the same prompt.
- Evaluation combined automated metrics (BLEU-4, METEOR, ROUGE-L, CLIPScore) with human-centered structural metrics: State Coverage, Transition Coverage, Initial State Identification, Accepting State Identification, Hallucination Count, and normalized Hallucination Count; four authors independently evaluated outputs and resolved disagreements by consensus.
Limitations
- Stated: the study focuses on a single class of structured STEM figures (automata diagrams) and expert-authored reference descriptions reflecting one accessibility-informed writing style; future work should expand to plots, circuits, tables, proofs/derivations, include blind and low-vision participants, and analyze prompt strategies and output constraints that reduce hallucinations.
- Stated: the human-written descriptions are not assumed to be a single canonical ground truth but represent accessibility-informed human judgments; the human-centered evaluation was conducted by the four authors, with no blind or low-vision participants directly assessing task success or perceived utility.
- Evident from the text: model naming/versioning is inconsistent across the narrative and tables (e.g., “Gemini 1” vs. “Gemini 3 Pro”; “Claude 3.5 Sonnet” vs. “Claude Sonnet 4.5”), and batch refusal/failure rates may affect comparability; the automated metrics used were not designed to capture accessibility-critical structural correctness.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
7 of 17 cited sources checked · 3 supported · 0 flagged · accuracy not yet measured
- Premisecitation§ Abstract · p. 2
Errors, omissions, and hallucinated elements remain frequent when models are asked to describe scientific images that rely on formal semantics, symbolic conventions, and precise structural relationships.
“In particular, errors, omissions, and hallucinated elements remain frequent when models are asked to describe scientific images that rely on formal semantics, symbolic conventions, and precise structural relationships.”
Quote in the paperSubstantial claim1 of 1 sources checked- cites [5] [5] Marina Buzzi, Giulio Galesi, Barbara Leporini, and Annalisa Nicotera. 2024. Is Generative AI Mature for Alternative Image Descriptions of STEM Content?. In International Conference on Web Information Systems and Technologies . 274-281. doi:10.5220/0012996800003825 - source
The Results (p. 5) report misinterpretations, failures to recognize the state diagram, and imprecise or superficial descriptions of STEM images. The Discussion (p. 6) explains that descriptions can omit structural details needed to understand a graph. But the paper does not report hallucinated elements or establish that all three problems are frequent across scientific images generally; its Conclusions (p. 7) caution that the study is too limited for a broad assessment. (LLM, conf=0.86)
Experimental — accuracy not yet measured
- Premisecitation§ 2 Related Work · p. 3
Most existing evaluations rely on datasets and metrics originally developed for generic image captioning or diagram understanding, which do not capture whether a description provides meaningful and trustworthy access for blind and low-vision users.
“As noted by [6], most existing evaluations rely on datasets and metrics originally developed for generic image captioning or diagram understanding, which do not capture whether a description provides meaningful and trustworthy access for blind and low-vision users.”
Quote in the paperSubstantial claim1 of 1 sources checked- cites [6] [6] Marco Cardia, Marina Buzzi, Giulio Galesi, and Barbara Leporini. 2025. A Descriptive Review of Image Datasets for Accessible Alternative Descriptions in STEM Domains. In Proceedings of the 16th Biannual Conference of the Italian SIGCHI Chapter . 1-7. - source
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measured
- Findingreasoning§ 4.2 Automated Metrics Analysis · p. 7
Claude's automated scores are comparable to GPT5.2's despite an order-of-magnitude difference in hallucination counts.
“Claude's automated scores are comparable to those of GPT5.2 despite the presence of an order-of-magnitude difference in hallucination counts.”
Quote in the paperSubstantial claim - Findingreasoning§ 4.2 Automated Metrics Analysis · p. 7
Across all models, absolute scores for n-gram-based metrics (BLEU-4 and ROUGE-L) remain low.
“Overall, absolute scores for n-gram-based metrics (BLEU-4 and ROUGE-L) remain low across all models.”
Quote in the paperSubstantial claim - Findingown data§ 5.2 Structural vs. Automated Metrics · p. 8
Gemini's dominance in structural accuracy (100% SC, 99.2% TC) is reflected in its highest scores for BLEU-4 (0.051), METEOR (0.257), and ROUGE-L (0.369).
“Comparing the structural results in Table 2 with the automated metrics in Table 3, we observe that Gemini 's dominance in structural accuracy (100% SC, 99.2% TC) is reflected in its highest scores for BLEU-4 (0.051), METEOR (0.257), and ROUGE-L (0.369).”
Quote in the paperSubstantial claim
- Findingown data§ 5.3 Irrelevant Physical Detail · p. 8
Across models and settings, models frequently include physically accurate but accessibility-irrelevant details.
“Across models and settings, we observe the frequent inclusion of physically accurate but accessibility-irrelevant details.”
Quote in the paperSubstantial claim - Findingown data§ 5.4 Failure in the Batch Setting · p. 9
For at least 18 images in the batch setting, Gemini outputs descriptions that are internally consistent and resemble a valid automaton specification but do not correspond to the image being described; they reflect a canonical or inferred automaton structure not visually present.
“Although this description is internally consistent and resembles a valid automaton specification, it does not correspond to the image being described. Instead, it reflects a canonical or inferred automaton structure that is not visually present in the diagram.”
Quote in the paperSubstantial claim - Findingown data§ Abstract · p. 1
Commonly used automated metrics only partially capture accessibility-relevant errors.
“We further observe that commonly used automated metrics only partially capture accessibility-relevant errors, underscoring the need for evaluation methodologies based on the needs of screen reader users.”
Quote in the paperSubstantial claim - Findingown data§ 4.1 Batch Results · p. 6
In the batch setting, Gemini failed to provide an alternative description for 38 images.
“the model failed to provide an alternative description for 38 images in the dataset, providing a description that cannot be considered alternative description.”
Quote in the paperNot a substantial claim - Findingown data§ 4.2 Automated Metrics Analysis · p. 7
In the single-image setting, Claude achieved the highest CLIPScore.
“Claude achieved the highest CLIPScore.”
- Also stated in: § 5.2 Structural vs. Automated Metrics · p. 8 · plain
“Claude achieved the highest CLIPScore (22.31).”
Quote in the paperSubstantial claim - Also stated in: § 5.2 Structural vs. Automated Metrics · p. 8 · plain
- Findingown data§ 4.1 Batch Results · p. 6
In the batch setting, Gemini produced 13 total hallucinations, compared to 20 for GPT.
“Notably, Gemini produced the fewest total hallucinations (13), compared to 20 for GPT.”
Quote in the paperNot a substantial claim - Findingown data§ 4.1 Batch Results · p. 7
In the single-image interactive setting, GPT5.2 achieved Transition Coverage of 98.6% ± 4.3.
“GPT5.2 showed poorer performance with TC of 98.6% ( ± 4.3).”
Quote in the paperNot a substantial claim - Findingown data§ 4.1 Batch Results · p. 6
In the batch setting, Claude experienced 16 cases of catastrophic failure where output was entirely unrelated to the source diagram.
“Beyond these metrics, Claude experienced 16 cases of catastrophic failure, where the output was entirely unrelated to the source diagram.”
Quote in the paperSubstantial claim - Premisecitation§ 2 Related Work · p. 3
Students with disabilities face substantial barriers when diagrams and symbolic content lack adequate textual representations or clear organization.
“Universal design principles and studies on inclusive educational practices [4] consistently show that students with disabilities face substantial barriers when diagrams and symbolic content lack adequate textual representations or clear organization.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [4] [4] Sheryl E Burgstahler and Rebecca C Cory. 2010. Universal design in higher education: From principles to practice . Harvard Education Press. - source
- Interpretationreasoning§ 2 Related Work · p. 3
Progress in generative approaches for accessible STEM image description is constrained not only by model capabilities but also by the lack of evaluation resources aligned with accessibility and educational requirements.
“Consequently, progress in generative approaches for accessible STEM image description is constrained not only by model capabilities but also by the lack of evaluation resources aligned with accessibility and educational requirements.”
Quote in the paperSubstantial claim - Premisecitation§ Abstract · p. 1
Vision-language models often fall short when generating reliable alt text for technical figures, where precision, clarity, and conceptual correctness are essential.
“While early work on vision-language models has shown promising results on everyday images, these approaches often fall short when generating reliable alt text for technical figures, where precision, clarity, and conceptual correctness are essential [1].”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [1] [1] Huda Diab Abdulgalil and Otman A. Basir. 2025. Next-generation image captioning: A survey of methodologies and emerging challenges from transformers to Multimodal Large Language Models. Natural Language Processing Journal 12 (2025), 100159. doi:10.1016/j.nlp.2025.100159 - source
- Premisecitation§ Abstract · p. 2
The lack of meaningful image descriptions impedes access to scholarly and educational materials, which increasingly rely on diagrams, graphs, and other STEM images.
“Recent studies emphasize that the lack of meaningful image descriptions impedes access not only to general web visuals but also to scholarly and educational materials, which increasingly rely on diagrams, graphs, and other STEM images [14].”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [14] [14] Maurizio Leotta, Fabrizio Mori, and Marina Ribaudo. 2023. Evaluating the effectiveness of automatic image captioning for web accessibility. Universal access in the information society 22, 4 (2023), 1293-1313. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Interpretationown data§ 5.2 Structural vs. Automated Metrics · p. 8
CLIPScore measures broad semantic image-text compatibility but is insensitive to the fine-grained structural details required for technical diagrams.
“This effectively illustrates that CLIPScore measures broad semantic image-text compatibility but is insensitive to the fine-grained structural details required for technical diagrams.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 4.2 Automated Metrics Analysis · p. 7
A description can be semantically compatible with an image while failing to convey the precise relationships required for understanding an automaton.
“a description can be semantically 'compatible' with an image while failing to convey the precise relationships required for understanding an automaton.”
Quote in the paperSubstantial claim - Interpretationown data§ 5.1 Performance Disparities Across Settings · p. 7
There is a significant performance disparity between batch and interactive modes, particularly for GPT and Claude.
“The results highlight a significant performance disparity between batch and interactive modes, particularly for GPT and Claude.”
Quote in the paperSubstantial claim - Premisereasoning§ Listing 1: Full system prompt used in the batch description experiments. · p. 5
Standard automated text-similarity metrics (BLEU, ROUGE, METEOR) measure lexical overlap and paraphrasing similarity but do not directly capture accessibility-relevant qualities such as factual correctness, completeness, or the presence of hallucinated content.
“These metrics measure lexical overlap and paraphrasing similarity but do not directly capture accessibility-relevant qualities such as factual correctness, completeness, or the presence of hallucinated content.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 5.3 Irrelevant Physical Detail · p. 9
Including such visually faithful but irrelevant details increases cognitive load for screen reader users by foregrounding low-level geometric properties instead of higher-level relational information.
“While such descriptions are visually faithful, they increase cognitive load for screen reader users by foregrounding low-level geometric properties instead of higher-level relational information.”
Quote in the paperSubstantial claim- Rests on an assumption: Including physically accurate but accessibility-irrelevant details increases cognitive load for screen reader users.
- Interpretationreasoning§ 5.3 Irrelevant Physical Detail · p. 9
Not all visible features should be treated as equally salient in alternative descriptions.
“highlighting that not all visible features should be treated as equally salient in alternative descriptions.”
Quote in the paperNot a substantial claim - Interpretationown data§ 5.2 Structural vs. Automated Metrics · p. 8
There is a divergence between human-computed structural metrics (SC, TC, HCN) and automated NLP metrics (BLEU, METEOR, ROUGE, CLIPScore).
“A key finding of this study is the divergence between humancomputed structural metrics (SC, TC, HCN) and automated NLP metrics (BLEU, METEOR, ROUGE, CLIPScore).”
Quote in the paperSubstantial claim - Premisereasoning§ Listing 1: Full system prompt used in the batch description experiments. · p. 5
In the context of STEM images and automata diagrams, automated metrics may be particularly misleading: a model can achieve a high score by mimicking the expected technical style and structure, yet fail on critical details such as transition labels or state names.
“On the other hand, in the xontext of STEM images and in the context of automata diagrams, these metrics may be particularly misleading: a model can achieve a high score by mimicking the expected technical style and structure, yet fail on critical details such as transition labels or state names.”
Quote in the paperSubstantial claim - Interpretationown data§ 4.2 Automated Metrics Analysis · p. 7
BLEU and ROUGE scores fail to penalize severe structural errors.
“Similarly, BLEU and ROUGE scores fail to penalize severe structural errors.”
Quote in the paperSubstantial claim - Interpretationown data§ 5.3 Irrelevant Physical Detail · p. 8
Such physically accurate but irrelevant details do not contribute meaningfully to understanding the automaton's structure or behavior.
“when these details do not contribute meaningfully to understanding the automaton's structure or behavior.”
Quote in the paperNot a substantial claim - Interpretationreasoning§ 5.2 Structural vs. Automated Metrics · p. 8
Automated metrics alone are insufficient for benchmarking multimodal LLMs on technical tasks where precision is paramount.
“Automated metrics alone are insufficient for benchmarking mutlimodal LLMs on technical tasks where precision is paramount.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 5.4 Failure in the Batch Setting · p. 9
These failures indicate a breakdown in Claude's ability to ground descriptions in the actual visual input under fully automated, dataset-level constraints.
“These failures go beyond isolated hallucinations and indicate a breakdown in the model's ability to ground descriptions in the actual visual input when operating under fully automated, dataset-level constraints.”
Quote in the paperSubstantial claim - Interpretationown data§ 5 Discussion · p. 7
There is a significant disconnect between the linguistic fluency of multimodal LLMs and their structural reliability for STEM accessibility.
“The results of this study highlight a significant disconnect between the linguistic fluency of multimodal LLMs and their structural reliability for STEM accessibility.”
Quote in the paperSubstantial claim- Rests on an assumption: The structural metrics (State Coverage, Transition Coverage, Initial State Identification, Accepting State Identification, Hallucination Count) adequately capture the accessibility-critical content of automata descriptions for screen reader users.
- Interpretationreasoning§ 4.2 Automated Metrics Analysis · p. 7
N-gram-based metrics can be satisfied by fluent, domain-appropriate language even when described states, transitions, or accepting conditions do not correspond to the actual diagram.
“This suggests that n-gram-based metrics can be satisfied by fluent, domain-appropriate language even when the described states, transitions, or accepting conditions do not correspond to the actual diagram.”
Quote in the paperSubstantial claim - Interpretationown data§ 5.4 Failure in the Batch Setting · p. 9
In the batch setting, Gemini often produces descriptions that are formally correct but still fail to function as alternative descriptions.
“Gemini, by contrast, often produces descriptions that are formally correct but still fail to function as alternative descriptions.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 4.2 Automated Metrics Analysis · p. 7
Without explicit structural validation, automated scores risk overestimating the practical accessibility of generated descriptions.
“Without explicit structural validation, automated scores risk overestimating the practical accessibility of generated descriptions.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 4.2 Automated Metrics Analysis · p. 7
Automated metrics systematically fail to capture the accuracy and completeness of formal structures essential for STEM comprehension.
“they systematically fail to capture the accuracy and completeness of formal structures that are essential for STEM comprehension.”
Quote in the paperSubstantial claim - Interpretationown data§ 5.4 Failure in the Batch Setting · p. 9
The batch-processing setup exposes more severe and systematic failures that are not apparent in single-image interactive use.
“The batch-processing setup exposes more severe and systematic failures that are not apparent in single-image interactive use.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 4.2 Automated Metrics Analysis · p. 7
For Gemini, automated metrics partially reflect genuine descriptive accuracy because its outputs closely align with the content and structure of the human-written references.
“In this case, automated metrics partially reflect genuine descriptive accuracy, as the model's outputs closely align with the content and structure of the human-written references.”
Quote in the paperSubstantial claim - Interpretationown data§ 5.4 Failure in the Batch Setting · p. 9
Gemini appears to substitute image-grounded description with a plausible abstract automaton, effectively ignoring the requirement to describe the specific visual instance.
“In these cases, Gemini appears to substitute image-grounded description with a plausible abstract automaton, effectively ignoring the requirement to describe the specific visual instance.”
Quote in the paperSubstantial claim - Interpretationown data§ 4.1 Batch Results · p. 7
In the single-image interactive setting, Claude systematically fails to accurately perceive or describe the automata structure.
“This represents a systematic failure to accurately perceive or describe the automata structure in this interactive mode.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 4.2 Automated Metrics Analysis · p. 7
Automated metrics alone are insufficient for evaluating the accessibility of descriptions of technical diagrams.
“Taken together, these findings demonstrate that automated metrics alone are insufficient for evaluating the accessibility of the descriptions of technical diagrams.”
Quote in the paperSubstantial claim- Rests on an assumption: The structural metrics (State Coverage, Transition Coverage, Initial State Identification, Accepting State Identification, Hallucination Count) adequately capture the accessibility-critical content of automata descriptions for screen reader users.
- Interpretationreasoning§ 4.2 Automated Metrics Analysis · p. 7
Hallucinations involving states, transitions, or labels often have minimal impact on automated scores despite representing severe accessibility failures.
“In particular, hallucinations involving states, transitions, or labels often have minimal impact on automated scores despite representing severe accessibility failures.”
Quote in the paperSubstantial claim - Interpretationown data§ 4.2 Automated Metrics Analysis · p. 7
CLIP-based metrics prioritize coarse image-text alignment over formal correctness.
“again highlighting that CLIP-based metrics prioritize coarse image-text alignment over formal correctness.”
Quote in the paperSubstantial claim - Interpretationown data§ 5.2 Structural vs. Automated Metrics · p. 8
The model that accurately identifies graph elements also produces text that lexically resembles the ground truth.
“This indicates a strong alignment: the model that accurately identifies graph elements also produces text that lexically resembles the ground truth.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 5.5 Accessibility Implications · p. 9
Current multimodal language models cannot be safely deployed for unsupervised, large-scale generation of alternative descriptions for complex technical diagrams.
“These findings further support the conclusion that current multimodal language models cannot be safely deployed for unsupervised, large-scale generation of alternative descriptions for complex technical diagrams.”
Quote in the paperSubstantial claim- Rests on an assumption: The human-authored reference descriptions used as ground truth are appropriate and correct for evaluating accessibility of automata descriptions.
- Rests on an assumption: The structural metrics (State Coverage, Transition Coverage, Initial State Identification, Accepting State Identification, Hallucination Count) adequately capture the accessibility-critical content of automata descriptions for screen reader users.
- Rests on an assumption: Including physically accurate but accessibility-irrelevant details increases cognitive load for screen reader users.
- Rests on an assumption: Under-grounded abstraction (replacing the actual image with a generic or inferred representation) leads to persistent misconceptions for blind users.
- Rests on an assumption: The tested models (Gemini, GPT, Claude) are representative of current state-of-the-art multimodal LLMs.
- Rests on an assumption: Blind users rely on generated descriptions to construct mental models of automata.
- Rests on an assumption: Alternative descriptions must provide access to what is actually shown, not to a mathematically plausible construct.
- Classified as probably not a substantial claim. 26
- May combine two claims. 43
- The quote only partly states this claim. 1
Classified as probably not a substantial claim. May combine two claims.
Premisecitation§ 2 Related Work · p. 3Conversational AI systems often provide richer and more context-aware descriptions than traditional mobile applications, but also reveal frequent inaccuracies, omissions, and inconsistencies, particularly when diagrams require explicit reasoning about structure, relationships, or domain-specific semantics.
“Their findings indicate that conversational AI systems often provide richer and more contextaware descriptions than traditional mobile applications; however, they also reveal frequent inaccuracies, omissions, and inconsistencies, particularly when diagrams require explicit reasoning about structure, relationships, or domain-specific semantics.”
Quote in the paperSubstantial claim1 of 1 sources checked- cites [5] [5] Marina Buzzi, Giulio Galesi, Barbara Leporini, and Annalisa Nicotera. 2024. Is Generative AI Mature for Alternative Image Descriptions of STEM Content?. In International Conference on Web Information Systems and Technologies . 274-281. doi:10.5220/0012996800003825 - source
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Premisecitation§ 2 Related Work · p. 3Prompt formulation-progressively specifying the user's visual impairment and explicitly requesting a graphic or educational description-significantly alters the level of detail and relevance of the output, yet does not guarantee correctness or pedagogical adequacy.
“The study also highlights the critical role of prompt formulation: progressively specifying the user's visual impairment and explicitly requesting a graphic or educational description significantly alters the level of detail and relevance of the output, yet does not guarantee correctness or pedagogical adequacy.”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites [5] [5] Marina Buzzi, Giulio Galesi, Barbara Leporini, and Annalisa Nicotera. 2024. Is Generative AI Mature for Alternative Image Descriptions of STEM Content?. In International Conference on Web Information Systems and Technologies . 274-281. doi:10.5220/0012996800003825 - source
Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measured
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Premisecitation§ 2 Related Work · p. 3LLM-based methods can produce explanations that are more detailed, coherent, and linguistically flexible than those from earlier vision-language models.
“Recent research shows that LLM-based methods can produce explanations that are more detailed, coherent, and linguistically flexible than those from earlier vision-language models [19].”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites [19] [19] João Daniel Silva, João Magalhães, Devis Tuia, and Bruno Martins. 2024. Large language models for captioning and retrieving remote sensing images. arXiv preprint arXiv:2402.06475 (2024). - source
The Image Captioning Results section (page unavailable) reports that the LLM-based RS-CapRet model outperforms an earlier captioning model on several metrics, and the Qualitative Analysis section shows that it generates image descriptions. Neither section establishes that its output is more detailed, coherent, or linguistically flexible than explanations from earlier vision-language models. (LLM, conf=0.82)
Experimental — accuracy not yet measured
May combine two claims.
Premisecitation§ Abstract · p. 2Generative systems show promising capabilities in producing fluent and seemingly informative descriptions, but also exhibit substantial limitations in conceptual accuracy, informational completeness, and alignment with the actual needs of screen reader users.
“These early analyses reveal a mixed picture: while generative systems show promising capabilities in producing fluent and seemingly informative descriptions, they also exhibit substantial limitations in conceptual accuracy, informational completeness, and alignment with the actual needs of screen reader users.”
Quote in the paperSubstantial claim1 of 1 sources checked- cites [5] [5] Marina Buzzi, Giulio Galesi, Barbara Leporini, and Annalisa Nicotera. 2024. Is Generative AI Mature for Alternative Image Descriptions of STEM Content?. In International Conference on Web Information Systems and Technologies . 274-281. doi:10.5220/0012996800003825 - source
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ Listing 1: Full system prompt used in the batch description experiments. · p. 5CLIPScore leverages a shared multimodal embedding space to assess the semantic alignment between the generated description and the original image, without relying on the reference text.
“CLIPScore : Uses the CLIP model to measure the semantic similarity and compatibility between the original image and the generated text, without relying on the reference text [10].”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites [10] [10] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021. Clipscore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 conference on empirical methods in natural language processing . 75147528. - source
May combine two claims.
Findingown data§ 4.1 Batch Results · p. 6In the batch setting, Claude achieved Transition Coverage of 68.7% ± 30.8 and Hallucination Count of 104.
“Claude exhibited significant difficulties in this mode, with a much lower Transition Coverage (68.7% ± 30.8), and a drastically higher total Hallucination Count (104).”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ 4.2 Automated Metrics Analysis · p. 7In the batch setting, Gemini attained the highest CLIPScore despite lower lexical similarity to references.
“In contrast, Gemini attains the highest CLIPScore despite lower lexical similarity to the references”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ 6 Conclusion · p. 9In the interactive single-image setting, Gemini achieved near-perfect coverage and the lowest hallucination counts, while GPT and especially Claude showed reduced reliability, including failures on fundamental structural elements (such as initial state identification) and increased hallucination frequency.
“In the interactive single-image setting, closer to optimal assistive use, performance patterns changed significantly. Gemini achieved near-perfect coverage and the lowest hallucination counts, while GPT and especially Claude showed reduced reliability, including failures on fundamental structural elements (such as initial state identification) and increased hallucination frequency.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ 4.1 Batch Results · p. 6In the batch setting, GPT achieved perfect State Coverage (100%) and Transition Coverage of 97.6% ± 6.6.
“GPT demonstrated superior performance in the batch setting, achieving perfect State Coverage. It correctly identified every state and label in the dataset, with a Transition Coverage of 97.6% ( ± 6.6).”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown data§ 6 Conclusion · p. 9Claude exhibited the most severe robustness issues in batch mode, including frequent hallucinations and several cases of catastrophic failure where outputs were unrelated to the source diagrams.
“Claude exhibited the most severe robustness issues in batch mode, including frequent hallucinations and several cases of catastrophic failure where outputs were unrelated to the source diagrams.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ 5.4 Failure in the Batch Setting · p. 9Claude exhibits a pronounced degradation in the batch setting, with outputs often unrelated to the source image, internally inconsistent, or structurally incoherent.
“Claude exhibits a pronounced degradation in this setting. In many cases, the generated outputs cannot be considered alternative descriptions at all: descriptions are either unrelated to the source image, internally inconsistent, or structurally incoherent.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ 5.2 Structural vs. Automated Metrics · p. 8Claude's BLEU and ROUGE scores were comparable to GPT's despite its massive structural hallucinations (204 vs 18).
“Similarly, Claude's BLEU and ROUGE scores were comparable to GPT's despite its massive structural hallucinations (204 vs 18)”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ 4.2 Automated Metrics Analysis · p. 7In the batch setting, GPT-OSS achieved the highest BLEU-4, METEOR, and ROUGE-L scores.
“GPT-OSS achieves the highest BLEU-4, METEOR, and ROUGE-L scores, aligning with its strong structural performance in batch mode.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown data§ 4.1 Batch Results · p. 7In the single-image interactive setting, Claude achieved Hallucination Count of 204 and Transition Coverage of 88.2%.
“Claude's performance drops with extremely high Hallucination Count of 204 and a Transition Coverage of 88.2%.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ Abstract · p. 1While LLMs often produce fluent and seemingly informative descriptions, substantial gaps remain compared to human-written alt text, particularly in conveying formal semantics and essential structural details required for accessibility.
“The results show that while LLMs often produce fluent and seemingly informative descriptions, substantial gaps remain compared to human-written alt text, particularly in conveying formal semantics and essential structural details required for accessibility.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ 4.1 Batch Results · p. 6In the batch setting, Gemini achieved State Coverage of 98.8% and Transition Coverage of 92.2% ± 18.2.
“Gemini followed closely, with a high State Coverage of 98.8%, though slightly lower Transition Coverage (92.2% ± 18.2).”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown data§ 4.1 Batch Results · p. 7In the single-image interactive setting, Gemini achieved State Coverage of 100%, Transition Coverage of 99.2% ± 4.4, and Hallucination Count of 10.
“Gemini achieves perfect scores in State Coverage (100%), and a near-perfect Transition Coverage of 99.2% ( ± 4.4). It also maintained the lowest Hallucination Count (10).”
Quote in the paperSubstantial claimThe quote only partly states this claim.
Findingown data§ 4.2 Automated Metrics Analysis · p. 7In the single-image setting, Gemini achieved the highest scores for BLEU-4, METEOR, and ROUGE-L.
“Among the evaluated models, Gemini achieves the highest scores for BLEU-4, METEOR, and ROUGE-L.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown data§ 5.2 Structural vs. Automated Metrics · p. 8Despite failing to identify some states and having the worst Transition Coverage (88.2%), Claude achieved the highest CLIPScore (22.31).
“Despite failing to identify some states, and having the worst score on TC (88.2%), Claude achieved the highest CLIPScore (22.31).”
Quote in the paperSubstantial claimClassified as probably not a substantial claim. May combine two claims.
Premisecitation§ Abstract · p. 2Missing or inadequate alt text remains one of the most frequently reported barriers for users with visual impairments, significantly affecting usability across devices and platforms.
“In web accessibility research and practice, missing or inadequate alt text remains one of the most frequently reported barriers for users with visual impairments, significantly affecting usability across devices and platforms [7].”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [7] [7] Maitraye Das, Alexander J Fiannaca, Meredith Ringel Morris, Shaun K Kane, and Cynthia L Bennett. 2024. From provenance to aberrations: Image creator and screen reader user perspectives on alt text for AI-generated images. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems . 1-21. -
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 2 Related Work · p. 2Spatial organization-relative positioning, alignment, and grouping-directly contributes to semantic meaning in mathematical notation.
“A similar perspective appears in studies on mathematical notation, such as those by Zanibbi and Blostein [24], which highlight how spatial organization -relative positioning, alignment, and grouping -directly contributes to semantic meaning.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [24] [24] Richard Zanibbi and Dorothea Blostein. 2012. Recognition and retrieval of mathematical expressions. International Journal on Document Analysis and Recognition (IJDAR) 15, 4 (2012), 331-357. source
Probably not a substantial claim: may not pass S3 (load-bearing).
Interpretationcitation§ 2 Related Work · p. 2Accurate diagram descriptions must encode both visual features and underlying logical structures.
“These findings indicate that accurate diagram descriptions must encode both visual features and underlying logical structures.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationreasoning§ 2 Related Work · p. 3The absence of dedicated benchmarks limits comparability across approaches and obscures accessibility-critical failure modes such as hallucination, over-abstraction, and omission of structurally relevant information.
“This absence of dedicated benchmarks limits comparability across approaches and obscures accessibility-critical failure modes such as hallucination, over-abstraction, and omission of structurally relevant information.”
Quote in the paperSubstantial claimMay combine two claims.
Study design and conductcitation§ 3.1 Dataset · p. 3The human-written alternative descriptions were produced following accessibility-oriented principles, in particular the W3C and the Diagram Center guidelines, which are in compliance with WCAG 2.2 Level AA.
“Each image is associated with a human-written alternative description produced following accessibility-oriented principles, in particular the W3C and the Diagram Center guidelines [8, 21] which are in compliance with the Web Content Accessibility Guidelines (WCAG) 2.2 Level AA [22].”
Quote in the paperSubstantial claim0 of 3 sources checked- cites [8] [8] DIAGRAM Center / Benetech. [n. d.]. Image Description Guidelines . DIAGRAM Center. http://diagramcenter.org/specific-guidelines-d.html Accessed: 18 Jan. 2026. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [21] [21] World Wide Web Consortium. Apri. Functional Images . Web Accessibility Initiative (WAI), W3C. https://www.w3.org/WAI/tutorials/images/functional/ Accessed: 21 Jan. 2026. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [22] [22] World Wide Web Consortium. 2023. Web Content Accessibility Guidelines (WCAG) 2.2 . Web Accessibility Initiative (WAI), W3C. https://www.w3.org/TR/WCAG22/ Accessed: 17 Jan. 2026. -
Not checked in the free run.
Classified as probably not a substantial claim.
Premisecitation§ 2 Related Work · p. 2Effective interpretation of diagrams requires identifying and reasoning about the relationships that connect symbols, shapes, and spatial groupings.
“Research on diagram understanding shows that effective interpretation goes beyond detecting individual visual elements. Comprehension requires identifying and reasoning about the relationships that connect symbols, shapes, and spatial groupings.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [12] [12] Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. 2016. A diagram is worth a dozen images. In European conference on computer vision . Springer, 235-251. - source
May combine two claims.
Premisecitation§ 1 Introduction · p. 1Manual alt text creation is time-consuming, labor-intensive, and often omitted or minimized in practice, resulting in the exclusion of screen reader users from important information conveyed by visual representations.
“Traditionally, authors or accessibility specialists with domain knowledge have manually created alt text; however, this process is time-consuming, labor-intensive, and often omitted or minimized in practice, resulting in the exclusion of screen reader users from important information conveyed by visual representations [7].”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [7] [7] Maitraye Das, Alexander J Fiannaca, Meredith Ringel Morris, Shaun K Kane, and Cynthia L Bennett. 2024. From provenance to aberrations: Image creator and screen reader user perspectives on alt text for AI-generated images. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems . 1-21. -
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Interpretationnone given§ 3.1 Dataset · p. 3Many images in the dataset are intentionally similar in visual appearance while representing distinct automata, making them particularly sensitive to descriptive errors.
“Many images are intentionally similar in visual appearance while representing distinct automata, making them particularly sensitive to descriptive errors.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationown data§ 6 Conclusion · p. 9Model behavior depends heavily on deployment conditions (batch versus interactive), and evaluations limited to a single interaction paradigm risk overestimating real-world reliability.
“These results indicate that model behavior depends heavily on deployment conditions (batch versus interactive) and that evaluations limited to a single interaction paradigm risk overestimating real-world reliability.”
Quote in the paperSubstantial claim- Rests on an assumption: The tested models (Gemini, GPT, Claude) are representative of current state-of-the-art multimodal LLMs.
May combine two claims.
Interpretationown data§ 6 Conclusion · p. 9Across both settings, LLMs often produce fluent and structurally plausible descriptions, but fluency did not reliably translate into accessibility usefulness.
“Across both settings, we observed that LLMs often produce fluent and structurally plausible descriptions, but fluency did not reliably translate into accessibility usefulness.”
Quote in the paperSubstantial claim- Rests on an assumption: The structural metrics (State Coverage, Transition Coverage, Initial State Identification, Accepting State Identification, Hallucination Count) adequately capture the accessibility-critical content of automata descriptions for screen reader users.
May combine two claims.
Interpretationown data§ 4.1 Batch Results · p. 6In the batch setting, Claude tends to fabricate non-existent graph elements and lacks robustness under realistic automation constraints.
“These results indicate a tendency to fabricate non-existent graph elements and a lack of robustness under realistic automation constraints.”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationreasoning§ 4.2 Automated Metrics Analysis · p. 7Low lexical overlap does not necessarily imply poor descriptive quality, nor does higher overlap guarantee accessibility correctness.
“Consequently, low lexical overlap does not necessarily imply poor descriptive quality, nor does higher overlap guarantee accessibility correctness.”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationreasoning§ 5 Discussion · p. 7Model outputs often appear technically authoritative but frequently lack the semantic precision required to support functional mental models for screen reader users.
“While model outputs often appear technically authoritative, they frequently lack the semantic precision required to support functional mental models for screen reader users.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S2 (checkable).
Interpretationreasoning§ 5.3 Irrelevant Physical Detail · p. 9This represents a form of overdescription: information that is correct but distracts from what is necessary to build a functional mental model of the automaton.
“From an accessibility perspective, this represents a form of overdescription: information that is correct but distracts from what is necessary to build a functional mental model of the automaton.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone given§ 3.3 Human-Centered Evaluation · p. 5Any disagreements in the human evaluation were discussed and resolved through consensus.
“Any disagreements were discussed and resolved through consensus.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Interpretationreasoning§ 4.2 Automated Metrics Analysis · p. 7Structural errors may lead blind users to construct an entirely incorrect mental model of the automaton.
“such errors are catastrophic, as they may lead blind users to construct an entirely incorrect mental model of the automaton.”
Quote in the paperSubstantial claim- Rests on an assumption: Blind users rely on generated descriptions to construct mental models of automata.
Classified as probably not a substantial claim. May combine two claims.
Study design and conductnone given§ 3.1 Dataset · p. 3The dataset is used exclusively for evaluation purposes and is not employed for training or fine-tuning any of the models.
“The dataset is used exclusively for evaluation purposes and is not employed for training or fine-tuning any of the models.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim. May combine two claims.
Interpretationreasoning§ 3.2 Models and Generative Setup · p. 4The batch setup introduces greater information density and potential contextual interference, allowing study of model robustness and failure modes under realistic automation constraints.
“Unlike the single-image setup, this configuration introduces greater information density and potential contextual interference, allowing us to study model robustness and failure modes under realistic automation constraints.”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationreasoning§ Listing 1: Full system prompt used in the batch description experiments. · p. 5Fine-grained errors such as incorrect transition labels or state names are catastrophic for accessibility but often have a marginal impact on n-gram-based scores.
“Such fine-grained errors are catastrophic for accessibility but often have a marginal impact on n-gram-based scores.”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationreasoning§ 5.5 Accessibility Implications · p. 9Both over-description and under-grounded abstraction are amplified in batch-processing scenarios and are only weakly reflected in automated evaluation metrics.
“Both phenomena are amplified in batch-processing scenarios and are only weakly reflected in automated evaluation metrics.”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationown data§ 5.1 Performance Disparities Across Settings · p. 8Gemini demonstrated remarkable stability and robustness, performing exceptionally well in both settings and dominating the single-image task.
“Gemini, by contrast, demonstrated remarkable stability and robustness, performing exceptionally well in both settings and dominating the single-image task.”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationreasoning§ 6 Conclusion · p. 9N-gram-based measures (e.g., BLEU) remained low overall and did not consistently reflect the severity of accessibility-critical errors; a description can match the expected style of the reference while still omitting or distorting key diagram semantics, such as transition labels or accepting states.
“Ngram-based measures (e.g., BLEU) remained low overall and did not consistently reflect the severity of accessibility-critical errors: a description can match the expected style of the reference while still omitting or distorting key diagram semantics, such as transition labels or accepting states.”
Quote in the paperSubstantial claim- Rests on an assumption: The structural metrics (State Coverage, Transition Coverage, Initial State Identification, Accepting State Identification, Hallucination Count) adequately capture the accessibility-critical content of automata descriptions for screen reader users.
Classified as probably not a substantial claim.
Interpretationreasoning§ 4.2 Automated Metrics Analysis · p. 7Alternative descriptions for complex STEM diagrams allow for substantial lexical and syntactic variation while still being correct and accessible.
“alternative descriptions for complex STEM diagrams allow for substantial lexical and syntactic variation while still being correct and accessible.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Study design and conductnone given§ 3.1 Dataset · p. 3The alternative descriptions used as a baseline were written by the four authors, all of whom have expertise in accessibility or computer vision.
“The alternative descriptions used as a baseline were written by the four authors, all of whom have expertise in accessibility or computer vision.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based).
Study design and conductnone given§ 3.2 Models and Generative Setup · p. 4All models are evaluated under identical experimental conditions to ensure comparability.
“All models are evaluated under identical experimental conditions to ensure comparability.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationown data§ 6 Conclusion · p. 9In the batch scenario, GPT achieved the strongest overall performance in state and transition coverage, while Gemini produced relatively few hallucinations but showed a high refusal rate that substantially limited its practical applicability.
“In the batch scenario, representative of large-scale remediation pipelines, GPT achieved the strongest overall performance in state and transition coverage, while Gemini produced relatively few hallucinations but showed a high refusal rate that substantially limited its practical applicability.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim. May combine two claims.
Study design and conductreasoning§ 3.1 Dataset · p. 3The reference descriptions are not assumed to be a single canonical ground truth; rather, they represent accessibility-informed human judgments.
“These descriptions serve as reference material for evaluation, but are not assumed to be a single canonical ground truth; rather, they represent accessibility-informed human judgments.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim. May combine two claims.
Premisereasoning§ Listing 1: Full system prompt used in the batch description experiments. · p. 5In the context of automata diagrams, two descriptions may differ substantially in wording while conveying equivalent information, or conversely may score highly while sharing the same incorrect inference.
“In the context of automata diagrams, two descriptions may differ substantially in wording while conveying equivalent information, or conversely may score highly while sharing the same incorrect inference.”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationreasoning§ 5.4 Failure in the Batch Setting · p. 9Such outputs are unusable and potentially harmful because they provide confident but misleading information to blind users.
“From an accessibility standpoint, such outputs are unusable and potentially harmful, as they provide confident but misleading information to blind users.”
Quote in the paperSubstantial claim- Rests on an assumption: Under-grounded abstraction (replacing the actual image with a generic or inferred representation) leads to persistent misconceptions for blind users.
May combine two claims.
Interpretationown data§ 5.1 Performance Disparities Across Settings · p. 7GPT excelled in batch processing but its performance dropped precipitously in the single-image interactive setting.
“While GPT excelled in batch processing its performance dropped precipitously in the single-image interactive setting (GPT5.2).”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone given§ 3.3 Human-Centered Evaluation · p. 5The descriptions generated under both setups were independently evaluated by the four authors of the paper.
“The descriptions generated under both setups were independently evaluated by the four authors of the paper.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Interpretationreasoning§ Listing 1: Full system prompt used in the batch description experiments. · p. 5CLIPScore captures the degree of visual grounding, providing an evaluation that remains independent of the specific phrasing used in human references.
“This allows us to capture the degree of visual grounding, providing an evaluation that remains independent of the specific phrasing used in human references”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
Interpretationreasoning§ 3.2 Models and Generative Setup · p. 4The single-image setup approximates best-case usage conditions for multimodal language models in assistive technologies.
“This setup aims to approximate best-case usage conditions for multimodal language models in assistive technologies.”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationreasoning§ 5.5 Accessibility Implications · p. 9Neither over-description nor under-grounded abstraction is easily detectable by blind users relying on the generated descriptions.
“Crucially, neither failure mode is easily detectable by blind users relying on the generated descriptions.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S4 (based).
Interpretationreasoning§ Listing 1: Full system prompt used in the batch description experiments. · p. 5The batch setup closely approximates realistic deployment scenarios in which accessibility descriptions are generated at scale through programmatic model calls.
“We remark that while such a setup does not reflect best case interactive usage, it closely approximates realistic deployment scenarios in which accessibility descriptions are generated at scale through programmatic model calls.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationreasoning§ 5.2 Structural vs. Automated Metrics · p. 8Automated metrics provide a rough proxy for linguistic fluency and general visual relevance but fail to capture the rigorous factual accuracy demanded by STEM accessibility applications.
“While they provide a rough proxy for linguistic fluency and general visual relevance, they fail to capture the rigorous factual accuracy demanded by STEM accessibility applications.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone given§ 3.1 Dataset · p. 3One of the authors is also an end user who regularly relies on screen readers.
“Importantly, one of the author is also an end user who regularly relies on screen readers.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationreasoning§ 5.4 Failure in the Batch Setting · p. 9Such substitutions are indistinguishable from correct descriptions for blind users and may lead to persistent misconceptions, especially in educational contexts.
“For blind users, such substitutions are indistinguishable from correct descriptions and may lead to persistent misconceptions, especially in educational contexts.”
Quote in the paperSubstantial claim- Rests on an assumption: Under-grounded abstraction (replacing the actual image with a generic or inferred representation) leads to persistent misconceptions for blind users.
Classified as probably not a substantial claim.
Study design and conductnone given§ 3.2 Models and Generative Setup · p. 4All models are queried using exactly the same prompt, without any model-specific adaptation.
“All models are queried using exactly the same prompt, without any model-specific adaptation.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
Study design and conductnone given§ 3.1 Dataset · p. 3The human annotations are constructed following a domain-level interpretation, describing the functional and semantic interpretation of the images, rather than a geometric level description.
“The human annotations are constructed following a domain-level interpretation, describing the functional and semantic interpretation (e.g. State Q1 transits to state Q2 on input 'a') of the images, rather than a geometric level description (e.g. A circle with an arrow).”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationown data§ 4.2 Automated Metrics Analysis · p. 7CLIPScore captures broad semantic compatibility between generated text and image but remains insensitive to fine-grained structural correctness.
“This indicates that CLIPScore captures broad semantic compatibility between the generated text and the image, such as recognizing that the image depicts a diagram with circles and arrows, but remains insensitive to fine-grained structural correctness.”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationreasoning§ 6 Conclusion · p. 9CLIPScore captured some aspects of visual grounding but still failed to guarantee correctness at the level required for STEM comprehension.
“Conversely, CLIPScore captured some aspects of visual grounding but still failed to guarantee correctness at the level required for STEM comprehension.”
Quote in the paperSubstantial claim- Rests on an assumption: The structural metrics (State Coverage, Transition Coverage, Initial State Identification, Accepting State Identification, Hallucination Count) adequately capture the accessibility-critical content of automata descriptions for screen reader users.
The paper relies on this without support.
LLMs still struggle with STEM diagrams, especially when descriptions require explicit reasoning about hierarchical structures, causal flows, or formal dependencies among elements.
“However, LLMs still struggle with STEM diagrams, especially when descriptions require explicit reasoning about hierarchical structures, causal flows, or formal dependencies among elements.”
The paper relies on this without support.
The main issue for STEM image description is the lack of training data that captures symbols, formal relationships, and domain-specific visual conventions.
“The main issue is the lack of training data that captures symbols, formal relationships, and domain-specific visual conventions, which are central to STEM imagery.”
The paper relies on this without support.
The exclusion of screen reader users from visual information is especially problematic in educational and scientific contexts, where figures and diagrams are essential for communicating complex concepts and supporting learning.
“This exclusion is especially problematic in educational and scientific contexts, where figures and diagrams are essential for communicating complex concepts and supporting learning.”
The paper relies on this without support.
The dataset images represent the visual complexity that students encounter in a standard curriculum.
“The images were extracted from undergraduate computer science lecture slides, representing the visual complexity that students encounter in a standard curriculum.”
The paper relies on this without support.
The human-authored reference descriptions used as ground truth are appropriate and correct for evaluating accessibility of automata descriptions.
“Because our reference descriptions are intentionally domain-level and reflect accessibility-informed human judgments rather than a single canonical ground truth”
The paper relies on this without support.
The structural metrics (State Coverage, Transition Coverage, Initial State Identification, Accepting State Identification, Hallucination Count) adequately capture the accessibility-critical content of automata descriptions for screen reader users.
“Our objective is to assess whether a generated description provides reliable cognitive access to the automaton's semantics (states, initial and accepting markers, and labeled transitions) rather than rewarding textual style or surface similarity.”
The paper relies on this without support.
Including physically accurate but accessibility-irrelevant details increases cognitive load for screen reader users.
“While such descriptions are visually faithful, they increase cognitive load for screen reader users by foregrounding low-level geometric properties instead of higher-level relational information.”
The paper relies on this without support.
Under-grounded abstraction (replacing the actual image with a generic or inferred representation) leads to persistent misconceptions for blind users.
“For blind users, such substitutions are indistinguishable from correct descriptions and may lead to persistent misconceptions, especially in educational contexts.”
The paper relies on this without support.
The tested models (Gemini, GPT, Claude) are representative of current state-of-the-art multimodal LLMs.
“current multimodal language models cannot be safely deployed for unsupervised, large-scale generation of alternative descriptions for complex technical diagrams.”
The paper relies on this without support.
Blind users rely on generated descriptions to construct mental models of automata.
“they may lead blind users to construct an entirely incorrect mental model of the automaton.”
The paper relies on this without support.
Alternative descriptions must provide access to what is actually shown, not to a mathematically plausible construct.
“An alternative description must provide access to what is actually shown, not to a mathematically plausible construct.”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Batch Claude model version changes between methods and results
MajorThe batch setup lists Claude 3.5 Sonnet, but the batch automated-metrics table identifies the evaluated model as Claude Sonnet 4.5. The text does not establish which version produced the batch results.
“- Claude 3.5 Sonnet”
“| Claude Sonnet 4.5 | 0.009 | 0.148 | 0.293 | 19.43 |”
GPT-5.2's initial-state standard deviation conflicts with its binary mean
MinorFor binary results with a mean of 90.4%, the standard deviation across images is approximately 29.4 percentage points, not 9.4. The reported mean is consistent with 66 of 73 images, but its accompanying standard deviation is not.
“| GPT-5.2 | 100.0 ± 0.0 | 98.6 ± 4.3 | 90.4 ± 9.4 | 1.00 ± 0.00 | 18 | 0.008 ± 0.003 |”
“Initial State Identification (ISI) : Binary metric indicating if the start state is correctly identified (1) or not (0).”
Gemini's batch transition-coverage deviation differs between table and text
MinorTable 1 reports a standard deviation of 8.2 percentage points for Gemini's batch Transition Coverage, while the results text reports 18.2.
“| Gemini 1 | 98.8 ± 6.9 | 92.2 ± 8.2 | 100.0 ± 0.0 | 0.95 ± 0.16 | 13 | 0.08 ± 0.02 |”
“Gemini followed closely, with a high State Coverage of 98.8%, though slightly lower Transition Coverage (92.2% ± 18.2).”
Cites
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · cited 1× · see citing works
“Over the past decade, progress in image captioning has been driven by the availability of large-scale datasets such as Flickr8k/30k [11], MS COCO [17], and VizWiz [9].”
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 1× · see citing works
“…grounding, providing an evaluation that remains independent of the specific phrasing used in human references To assess the quality of the generated descriptions, we employed the following metrics: • BLEU-4 : Measures n-gram precision overlap between the generated text and the reference [18].”
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Bourdev, L., Girshick, R., Hays, J., Perona, P., Ramanan, D., Zitnick, C. L., & Dollár, P. (2014) · arXiv · doi:10.48550/arxiv.1405.0312 · Cited by 2,579 · see citing works
Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics
Hodosh, M., Young, P., & Hockenmaier, J. (2013) · Journal of Artificial Intelligence Research · doi:10.1613/jair.3994 · Cited by 1,349 · cited 1× · see citing works
“Over the past decade, progress in image captioning has been driven by the availability of large-scale datasets such as Flickr8k/30k [11], MS COCO [17], and VizWiz [9].”
CLIPScore: A Reference-free Evaluation Metric for Image Captioning
Hessel, J., Holtzman, A., Forbes, M., Le Bras, R., & Choi, Y. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.595 · Cited by 1,232 · cited 1× · see citing works
“• CLIPScore : Uses the CLIP model to measure the semantic similarity and compatibility between the original image and the generated text, without relying on the reference text [10].”
VizWiz Grand Challenge: Answering Visual Questions from Blind People
Gurari, D., Li, Q., Stangl, A. J., Guo, A., Lin, C., Grauman, K., Luo, J., & Bigham, J. P. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00380 · Cited by 621 · cited 1× · see citing works
“Over the past decade, progress in image captioning has been driven by the availability of large-scale datasets such as Flickr8k/30k [11], MS COCO [17], and VizWiz [9].”
The Meteor metric for automatic evaluation of machine translation
Lavie, A., & Denkowski, M. (2009) · Machine Translation · doi:10.1007/s10590-009-9059-4 · Cited by 351 · cited 1× · see citing works
“• METEOR : Accounts for synonymy and stemming, providing better correlation with human judgment [3, 13].”
A Diagram is Worth a Dozen Images
Kembhavi, A., Salvato, M., Kolve, E., Seo, M., Hajishirzi, H., & Farhadi, A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_15 · Cited by 290 · cited 1× · see citing works
“Kemb-havi et al. formalized this concept through Diagram Parse Graphs (DPGs) [12], proposing a framework that integrates visual recognition with structural parsing and symbolic reasoning.”
Recognition and retrieval of mathematical expressions
Zanibbi, R., & Blostein, D. (2012) · International Journal on Document Analysis and Recognition (IJDAR) · doi:10.1007/s10032-011-0174-4 · Cited by 284 · cited 1× · see citing works
“A similar perspective appears in studies on mathematical notation, such as those by Zanibbi and Blostein [24], which highlight how spatial organization — relative positioning, alignment, and grouping — directly contributes to semantic meaning.”
CLIPScore: A Reference-free Evaluation Metric for Image CaptioningPreprint
Hessel, J., Holtzman, A., Forbes, M., Bras, R. J. L., & Choi, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.08718 · Cited by 174 · see citing works
Universal Design: Implications for Computing Education
Burgstahler, S. (2011) · ACM Transactions on Computing Education · doi:10.1145/2037276.2037283 · Cited by 68 · see citing works
Das, M., Fiannaca, A., Morris, M. R., Kane, S., & Bennett, C. L. (2024) · International Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642325 · Cited by 41 · see citing works
Evaluating the effectiveness of automatic image captioning for web accessibility
Leotta, M., Mori, F., & Ribaudo, M. (2022) · Universal Access in the Information Society · doi:10.1007/s10209-022-00906-7 · Cited by 32 · see citing works
Abdulgalil, H. D., & Basir, O. A. (2025) · Natural Language Processing Journal · doi:10.1016/j.nlp.2025.100159 · Cited by 20 · see citing works
Large Language Models for Captioning and Retrieving Remote Sensing Images
Silva, J. D., Magalhães, J., Tuia, D., & Martins, B. (2024) · arXiv · doi:10.48550/arxiv.2402.06475 · Cited by 20 · see citing works
A Diagram Is Worth A Dozen ImagesPreprint
Kembhavi, A., Salvato, M., Kolve, E., Seo, M., Hajishirzi, H., & Farhadi, A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.07396 · Cited by 12 · see citing works
Evaluating the Effectiveness of STEM Images Captioning
Leotta, M., & Ribaudo, M. (2024) · International Web for All Conference (W4A) · doi:10.1145/3677846.3677863 · Cited by 8 · see citing works
Agrawal, V., Kantipudi, M. P., & Jagtap, J. (2025) · Scientific Reports · doi:10.1038/s41598-025-01823-4 · Cited by 5 · cited 1× · see citing works
“Similarly, Agrawal et al. [2] introduced a dataset of hand-drawn diagrams that better reflect the variability, noise, and imprecision typical of real educational settings.”
Chart Accessibility: A Review of Current Alt Text Generation
Yan, C., Hutter, H.-P., Schmitt-Koopmann, F. M., & Darvishy, A. (2025) · IEEE Access · doi:10.1109/access.2025.3571626 · Cited by 5 · see citing works
A Descriptive Review of Image Datasets for Accessible Alternative Descriptions in STEM Domains
Cardia, M., Buzzi, M., Galesi, G., & Leporini, B. (2025) · Proceedings of the 16th Biannual Conference of the Italian SIGCHI Chapter · doi:10.1145/3750069.3750122 · Cited by 5 · see citing works
Self-citationIs Generative AI Mature for Alternative Image Descriptions of STEM Content?
Buzzi, M., Galesi, G., Leporini, B., & Nicotera, A. (2024) · Proceedings of the 20th International Conference on Web Information Systems and Technologies · doi:10.5220/0012996800003825 · Cited by 4 · see citing works
Self-citationRL-CSDia: Representation Learning of Computer Science DiagramsPreprint
Wang, S., Zhang, L., Luo, X., Yang, Y., Hu, X., & Liu, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.05900 · Cited by 1 · cited 1× · see citing works
“RL-CSDia [20], for example, provides thousands of computer science diagrams, including trees, graphs, and flowcharts, annotated to support the learning of abstract and relational representations.”
Flagged first, then by citations
Research works printed but not linked: 3
- Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization. 65–72.
- DIAGRAM Center / Benetech. [n. d.]. Image Description Guidelines. DIAGRAM Center. http://diagramcenter.org/specific-guidelines-d.html Accessed: 18 Jan. 2026.
- Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74–81.
Cited by
Sorted by citations
Amado-Salvatierra, H. R. (2026) · arXiv · doi:10.48550/arxiv.2608.00749
“Alt-text generation for complex technical or educational images, diagrams, charts, equations, remains prone to missing or low-quality descriptions [24].”
Accessible Alt-Text Generation of 2D Graphs with Multimodal LLMs: Comparison and Evaluation
Cardia, M., Poggianti, C., Leporini, B., Pelagatti, S., & Chessa, S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38407-2_59
Self-citationOladipo, T., Oladipo, F. O., Chakma, A., et al. (2026) · International Conference on Information Science and Technology · doi:10.1109/icist71072.2026.11727873
“Accessibility research has more often examined tasks such as generating or evaluating alternative text for STEM images [25], and broader reviews of LLM-based web-accessibility work emphasize generation, issue detection, and remediation [26].”
- Authors
- Cardia, M., Angileri, L., Poggianti, C., & Leporini, B.
- Affiliations
- University of Pisa (IT)
- Venue
- Proceedings of the 23rd International Web for All Conference
- Publisher
- ACM
- Published
- 2026-04-13
- Pages
- 23-32
- OpenAlex
- W7163129038
- Semantic Scholar
- ca3b411277a6f541d943d6dfaf11fca5a3e5e150
- Cited by
- 2
- References
- 13
- Citations by year
- 2026: 2
- FWCI
- 17.71
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- International Web for All Conference (W4A) (publishedVersion, cc-by)
- CINECA IRIS Institutial research information system (University of Pisa) (submittedVersion, cc-by-nc-nd)
- Funders
- European Union - Next Generation EU (B53D23019500006)
- Topics
- Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Keywords
- alt text, image accessibility, large language models, screen readers, blind and visually impaired users, web accessibility, assistive technology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-25.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-25.
Cited sources are checked by gpt-6-sol.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
