Turning manual web accessibility success criteria into automatic: an LLM-based approach
Open accessAbstract
Web accessibility evaluation is a costly process that usually requires manual intervention. Currently, large language model (LLM) based systems have gained popularity and shown promising capabilities to perform tasks that seemed impossible or required programming knowledge specific to a given area or were supposed to be impossible to be performed automatically. Our research explores whether an LLM-based system would be able to evaluate web accessibility success criteria that require manual evaluation. Three specific success criteria of the Web Content Accessibility Guidelines (WCAG) that currently require manual checks were tested: 1.1.1 Non-text Content, 2.4.4 Link Purpose (In Context), and 3.1.2 Language of Parts. LLM-based scripts were developed to evaluate the test cases. Results were compared against current web accessibility evaluators. While automated accessibility evaluators were unable to reliably test the three WCAG criteria, often missing or only warning about issues, the LLM-based scripts successfully identified accessibility issues the tools missed, achieving overall 87.18% detection across the test cases. Conclusion The results demonstrate LLMs can augment automated accessibility testing to catch issues that pure software testing misses today. Further research should expand evaluation across more test cases and types of content.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLMs can identify accessibility violations that automated checkers often miss.
“while López-Gil and Pereira [51] found that LLMs can identify accessibility violations that automated checkers often miss.”
From A Multi-Agent Framework for Accessible Web Code Generation (2026) · cites this paper as [51]
Accessibility is not an inherent capability of current LLMs but a specialized constraint that is frequently overridden by training biases toward common, and often inaccessible, web design patterns.
“This finding aligns with recent evidence that accessibility is not an inherent capability of current LLMs but a specialized constraint that is frequently overridden by training biases toward common, and often inaccessible, web design patterns [43, 51, 58].”
From A Multi-Agent Framework for Accessible Web Code Generation (2026) · cites this paper as [51]
The Abstract and §6 (Limitations; page unavailable) discuss LLM-based scripts for detecting accessibility errors and the need to choose prompts for each task. They do not assess whether accessibility is an inherent LLM capability or whether training biases cause LLMs to favor inaccessible web design patterns. Searches for those claims found no supporting passage. (LLM, conf=0.75)
Experimental — accuracy not yet measuredClaude and GPT-5 generated hundreds of contrast violations, indicating a lack of inherent visual grounding for accessibility-critical design decisions.
“Claude and GPT-5 generated hundreds of contrast violations, indicating a lack of inherent visual grounding for accessibility-critical design decisions [51, 75].”
From A Multi-Agent Framework for Accessible Web Code Generation (2026) · cites this paper as [51]
The Methods (section 3.2; page unavailable) list GPT-3.5, GPT-4, Claude, and Bard, but not GPT-5. The Discussion (section 5; page unavailable) describes tests of alternative text, link purpose, and language tags, not generated designs or counts of contrast violations. A search for hundreds of contrast violations and visual grounding found no passage addressing that claim. (LLM, conf=0.76)
Experimental — accuracy not yet measuredContrast adjustments account for a disproportionate share of maintenance effort in human-led accessibility remediation.
“This behavior highlights challenges similar to those reported in human-led accessibility remediation, where contrast adjustments account for a disproportionate share of maintenance effort [20, 51].”
From A Multi-Agent Framework for Accessible Web Code Generation (2026) · cites this paper as [51]
The Related work section discusses color-contrast false positives and the need for human verification, but does not compare the maintenance effort required for contrast adjustments with other remediation tasks. Searches for contrast remediation and maintenance workload found no passage addressing the claimed disproportionate share. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Stephanidis, C., Salvendy, G., Antona, M., Chen, J. Y. C., Dong, J., Duffy, V. G., Fang, X., Fidopiastis, C., Fragomeni, G., Fu, L. P., et al. (2019) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2019.1619259 · Cited by 524 · see citing works
Guidelines are only half of the story: accessibility problems encountered by blind users on the web
Power, C., Freire, A., Petrie, H., & Swallow, D. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2207736 · Cited by 327 · cited 2× · see citing works
“Key activities also include assistive technology user testing and expert heuristic reviews grounded in WCAG criteria [12].”
Vigo, M., Brown, J., & Conway, V. (2013) · Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility · doi:10.1145/2461121.2461124 · Cited by 215 · see citing works
Mankoff, J. C., Fait, H., & Tran, T. (2005) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1054972.1054979 · Cited by 206 · see citing works
A web accessibility report card for top international university web sites
Kane, S. K., Shulman, J. A., Shockley, T. J., & Ladner, R. E. (2007) · Proceedings of the 2007 international cross-disciplinary conference on Web accessibility (W4A) - W4A '07 · doi:10.1145/1243441.1243472 · Cited by 107 · see citing works
Naftali, M., & Findlater, L. (2014) · Proceedings of the 16th international ACM SIGACCESS conference on Computers & accessibility - ASSETS '14 · doi:10.1145/2661334.2661372 · Cited by 105 · see citing works
Comparing accessibility evaluation tools: a method for tool effectiveness
Brajnik, G. (2004) · Universal Access in the Information Society · doi:10.1007/s10209-004-0105-y · Cited by 101 · see citing works
The Expertise Effect on Web Accessibility Evaluation Methods
Brajnik, G., Yeşilada, Y., & Harper, S. (2011) · Human-Computer Interaction · doi:10.1080/07370024.2011.601670 · Cited by 85 · see citing works
Abou-Zahra, S. (2008) · Human-Computer Interaction Series · doi:10.1007/978-1-84800-050-6_7 · Cited by 73 · see citing works
Othman, A., Dhouib, A., & Nasser Al Jabor, A. (2023) · Proceedings of the 16th International Conference on PErvasive Technologies Related to Assistive Environments · doi:10.1145/3594806.3596542 · Cited by 59 · cited 2× · see citing works
“To the authors’ knowledge, [31] is the only work so far that suggests using LLMs for improving website accessibility.”
InfluentialAre separate interfaces inherently unequal?
Wentz, B., & Lazar, J. K. (2011) · iConference · doi:10.1145/1940761.1940774 · Cited by 59 · see citing works
Comparing ten WCAG tools for accessibility evaluation of websites
Kumar, S., Jeevithashree, D. V., & Biswas, P. (2021) · Technology and Disability · doi:10.3233/tad-210329 · Cited by 52 · see citing works
Is accessibility conformance an elusive property? A study of validity and reliability of WCAG 2.0
Brajnik, G., Yeşilada, Y., & Harper, S. (2012) · ACM Transactions on Accessible Computing · doi:10.1145/2141943.2141946 · Cited by 52 · see citing works
Evaluating a Large Language Model on Searching for GUI Layouts
Brie, P., Burny, N., Sluÿters, A., & Vanderdonckt, J. M. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3593230 · Cited by 41 · see citing works
From Web accessibility to Web adaptability
Kelly, B., Nevile, L., Sloan, D., Fanou, S., Ellison, R., & Herrod, L. (2009) · Disability and Rehabilitation Assistive Technology · doi:10.1080/17483100902903408 · Cited by 39 · see citing works
The Unified Web Evaluation Methodology (UWEM) 1.2 for WCAG 1.0
Nietzio, A., Strobbe, C., & Velleman, E. M. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-70540-6_57 · Cited by 28 · see citing works
Detecting Phishing Sites Using ChatGPTPreprint
Koide, T., Fukushi, N., Nakano, H., & Chiba, D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.05816 · Cited by 26 · see citing works
Evaluating accessibility-in-use
Vigo, M., & Harper, S. (2013) · International Cross-Disciplinary Conference on Web Accessibility (W4A) · doi:10.1145/2461121.2461136 · Cited by 23 · see citing works
Exploratory Analysis of Collaborative Web Accessibility Improvement
Sato, D., Takagi, H., Kobayashi, M., Kawanaka, S., & Asakawa, C. (2010) · ACM Transactions on Accessible Computing · doi:10.1145/1857920.1857922 · Cited by 20 · see citing works
Navarrete, R., & Luján‐Mora, S. (2015) · International Conference on eDemocracy & eGovernment · doi:10.1109/icedeg.2015.7114457 · Cited by 19 · see citing works
Researching Pedagogy in Digital Accessibility Education
Lewthwaite, S., Horton, S., & Coverdale, A. (2022) · ACM SIGACCESS Accessibility and Computing · doi:10.1145/3582298.3582300 · Cited by 10 · see citing works
Keith, S., Floratos, N., & Whitney, G. (2012) · International Cross-Disciplinary Conference on Web Accessibility (W4A) · doi:10.1145/2207016.2207029 · Cited by 8 · see citing works
Evaluating Web Accessibility of Spanish Universities
Máñez-Carvajal, C., Piqueras, R. F., & Cervera-Mérida, J. F. (2019) · Journal of Engineering and Applied Sciences · doi:10.36478/jeasci.2019.4876.4881 · Cited by 7 · see citing works
Understanding HTML with Large Language ModelsPreprint
Gür, İ., Nachum, O., Miao, Y., Safdari, M., Huang, A., Chowdhery, A., Narang, S., Fiedel, N., & Faust, A. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.03945 · Cited by 4 · see citing works
Flagged first, then by citations
2 of the 26 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
A Comprehensive Survey on Generative AI for Metaverse: Enabling Immersive Experience
Chamola, V., Sai, S., Bhargava, A., Sahu, A., Jiang, W., Xiong, Z.-H., Niyato, D., & Hussain, A. (2024) · Cognitive Computation · doi:10.1007/s12559-024-10342-9 · Cited by 36 · see citing works
CodeA11y: Making AI Coding Assistants Useful for Accessible Web Development
Mowar, P., Peng, Y.-H., Wu, J., Steinfeld, A., & Bigham, J. P. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713335 · Cited by 31 · see citing works
Towards accessible website design through artificial intelligence: A systematic literature review
Vera-Amaro, G., & Rojano-Cáceres, J. R. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107821 · Cited by 21 · see citing works
Automated evaluation of accessibility issues of webpage content: tool and evaluation
Ara, J., & Sik‐Lányi, C. (2025) · Scientific Reports · doi:10.1038/s41598-025-92192-5 · Cited by 17 · see citing works
Pereira, L. S., & Duarte, C. (2025) · Universal Access in the Information Society · doi:10.1007/s10209-025-01210-w · Cited by 15 · see citing works
AccessGuru: Leveraging LLMs to Detect and Correct Web Accessibility Violations in HTML Code
Fathallah, N., Hernández, D., & Staab, S. (2025) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3663547.3746360 · Cited by 14 · see citing works
Unmoderated Usability Studies Evolved: Can GPT Ask Useful Follow-up Questions?
Kuric, E., Demcak, P., & Krajcovic, M. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2427978 · Cited by 12 · see citing works
LLMs for Accessibility in Mobile Apps: Detection and Repair
Aljedaani, W., Aljohani, A., Eler, M. M., Habib, A., & Do, H. (2025) · International Web for All Conference (W4A) · doi:10.1145/3744257.3744270 · Cited by 11 · see citing works
Artificial intelligence in web accessibility: A systematic mapping study
Campoverde‐Molina, M., & Luján‐Mora, S. (2025) · Computer Standards & Interfaces · doi:10.1016/j.csi.2025.104055 · Cited by 13 · see citing works
Król, K. (2025) · Applied Sciences · doi:10.3390/app15052292 · Cited by 9 · see citing works
Patvardhan, N., Ranade, M., Vandana, & Khatwani, R. (2024) · The International Journal of Information and Learning Technology · doi:10.1108/ijilt-04-2024-0066 · Cited by 8 · see citing works
Duarte, C., da Costa, M. A. B., Pereira, L. S., & Guerreiro, J. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708557.3716329 · Cited by 8 · see citing works
The Tabular Accessibility Dataset: A Benchmark for LLM-Based Web Accessibility Auditing
Andruccioli, M., Bassi, B., Delnevo, G., & Salomoni, P. (2025) · Data · doi:10.3390/data10090149 · Cited by 6 · see citing works
Coverage of web accessibility guidelines provided by automated checking tools
Fischer, T., Lundell, B., & Gamalielsson, J. (2025) · Universal Access in the Information Society · doi:10.1007/s10209-025-01263-x · Cited by 10 · see citing works
Kumar, A., Padath, T., & Wang, L. L. (2025) · Proceedings of the 27th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3663547.3746380 · Cited by 5 · see citing works
GemmaWithLoRA: A New Approach to Click Fraud Detection
Dai, B., Wang, L., Zhao, X., Guo, Y., & Obaidat, M. S. (2024) · International Conference on Communications, Computing, Cybersecurity, and Informatics (CCCI) · doi:10.1109/ccci61916.2024.10736481 · Cited by 5 · see citing works
Iannuzzi, N., Manca, M., Paternò, F., & Santoro, C. (2025) · ACM Journal on Responsible Computing · doi:10.1145/3762815 · Cited by 4 · see citing works
Leporini, B., Buzzi, M., & Penna, G. D. (2025) · International Conference on PErvasive Technologies Related to Assistive Environments (PETRA) · doi:10.1145/3733155.3737910 · Cited by 4 · see citing works
Leedy, K., Preston, M., Feng, J. H., & Rezwana, J. (2025) · International Academic Mindtrek Conference (Mindtrek) · doi:10.1145/3757980.3757989 · Cited by 3 · see citing works
Measuring the Semantic Accessibility Gap in LLM-Generated Web UIs
Calò, T., Guriță, A.-E., & Russis, L. D. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3799364 · Cited by 3 · see citing works
Wang, S., & Shao, Y. (2025) · Data Science and Management · doi:10.1016/j.dsm.2025.06.005 · Cited by 5 · see citing works
Towards a Conceptual Model for Ai-Driven Web Accessibility Remediation: A Prompt-Based Approach
Vera-Amaro, G., & Rojano-Cáceres, J. R. (2025) · International Conference in Software Engineering Research and Innovation (CONISOFT) · doi:10.1109/conisoft66928.2025.00044 · Cited by 2 · see citing works
Vera-Amaro, G., Vera-Amaro, R., Mata-Rivera, M.-F., & Rojano-Cáceres, J. R. (2026) · Technologies · doi:10.3390/technologies14050287 · Cited by 2 · see citing works
A Conceptual Model for Born-Accessible Web Accessibility Regeneration Using Large Language Models
Vera-Amaro, G., & Rojano-Cáceres, J. R. (2026) · Computers · doi:10.3390/computers15060343 · Cited by 2 · see citing works
Large Language Models for Web Accessibility: A Systematic Literature Review
Aljedaani, W., & Mollik, R. H. (2026) · International Web for All Conference (W4A) · doi:10.1145/3800424.3800452 · Cited by 3 · see citing works
A Multi-Agent Framework for Accessible Web Code Generation
Aljedaani, W., Shaik, M. A., & Aljohani, A. (2026) · Proceedings of the 23rd International Web for All Conference · doi:10.1145/3800424.3800453 · Cited by 1 · see citing works
Generative AI as a New Assistive Technology for Web Interaction
Penna, G. D., Buzzi, M., & Leporini, B. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04999-5_6 · Cited by 1 · see citing works
User cum expert judgement model for accessibility using fuzzy approach
Sarita, K., Kaur, P., & Kaur, S. (2025) · Scientific Reports · doi:10.1038/s41598-025-00117-z · Cited by 1 · see citing works
Leewis, S., Smit, K., van de Hoef, A., Hartman, F., Kuiper, N., Todorova, J., & Gerritsen, T. (2025) · International Conference on Software and e-Business (ICSEB) · doi:10.1145/3789037.3789044 · Cited by 1 · see citing works
Leveraging Large Language Models for Alt-Text Evaluation in E-Commerce: A Data-Driven Study
Leonardi, N., Manca, M., & Paternò, F. (2026) · International Conference on Advanced Visual Interfaces (AVI) · doi:10.1145/3811427.3811431 · Cited by 1 · see citing works
Wegener, A., Kretzer, F., Nicolay, T. D., & Maedche, A. (2025) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3663547.3759747
de França, L. C. F., Andrade, L. M. M. D. S., & Viana, W. (2025) · Brazilian Symposium on Multimedia and the Web · doi:10.5753/webmedia.2025.16064
Xavier, L. E. O., Rabelo, D. M. F., Martins, R. D. S., & Viana, W. (2025) · Brazilian Symposium on Multimedia and the Web · doi:10.5753/webmedia.2025.15298 · Cited by 2 · see citing works
Huda, M., Setyanto, A., Kusrini, K., & Wahyuni, S. N. (2025) · International Conference on Information Technology and Digital Applications (ICITDA) · doi:10.1109/icitda68167.2025.11332348
Zhang, M., Yu, J., Xu, C., Li, J., Yin, X., & Liu, H. (2026) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3793673
An Accessible Real-Time Document Question Answering Extension for People with Visual Impairment
Anagha, V., & Kuppusamy, K. S. (2025) · International Conference on Data Science and Intelligent Network Computing (ICDSINC) · doi:10.1109/icdsinc66221.2025.11448265
Penna, G. D., Buzzi, M., & Leporini, B. (2026) · Research Square · doi:10.2139/ssrn.6643942
Enhancing Cognitive Accessibility in Mobile Apps via LLM-driven Redundant Entry Reduction
Zhang, M., Xiong, Y., Cheng, X., Liu, Y., Liu, R., & Chen, Z. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3816705
On the Limits for Web Accessibility Evaluation Using AI Agents
de Santana, V. F., Ferraz, R., & de Deus, J. V. (2026) · International Web for All Conference (W4A) · doi:10.1145/3800424.3800445
López-Gil, J.-M., & Pereira, J. (2026) · Information and Software Technology · doi:10.1016/j.infsof.2026.108215
Self-citationFitzpatrick, C., Liu, Y., Mai, S. T., Taheri, J., & Li, Z. (2026) · High-Confidence Computing · doi:10.1016/j.hcc.2026.100414
Gatto, S., & Gaggi, O. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29372-5_24
ReFLAIR: Detecting Responsive Layout Reflow Issues using Multimodal Generative AI
He, Y., He, Z., Huq, S. F., & Malek, S. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3808136 · Cited by 1 · see citing works
Evaluating the Use of LLMs for Automated DOM-Level Resolution of Web Performance Issues
Peters, G., Khatoonabadi, S., & Shihab, E. (2026) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3793302.3793361
Expanding WCAG Success Criteria Testing Coverage with Browser-Native AI
Keating, S., & Maike, V. (2026) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac69091.2026.00262
The Long Road to Web Accessibility: Three Decades of Retrospective Analysis
Afroz, M., Dar, M. D., & Chowdhury, O. (2026) · ACM Internet Measurement Conference (IMC) · doi:10.1145/3777912.3839806
LADar: an LLM-based Tool to Detect Web Accessibility Violations
Majidi, F., Ribaudo, M., & Leotta, M. (2026) · International Conference on Information Technology for Social Good (GoodIT) · doi:10.1145/3794786.3830742
A Hybrid Light-AI-based Framework for Accessibility Validation
Leonardi, N., Manca, M., & Paternò, F. (2026) · International Conference on Information Technology for Social Good (GoodIT) · doi:10.1145/3794786.3830785
An Intelligent Data-Driven Framework for Web Accessibility Diagnosis and Remediation
Acosta-Vargas, P., Díaz‐Bedoya, D., Salvador-Ullauri, L., Novillo-Villegas, S., Salvador-Acosta, B., & González, M. (2026) · Computers · doi:10.3390/computers15100690
From Reports to Resolutions: Accessibility in Practice Through Moodle’s Issue Tracker
He, Z., Huq, S. F., He, Y., Dam, H. K., & Malek, S. (2026) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3797867.3829062
- Authors
- López-Gil, J.-M., & Pereira, J.
- Affiliations
- University of the Basque Country (ES)
- Venue
- Universal Access in the Information Society
- Published
- 2024-03-16
- Volume
- 24
- Issue
- 1
- Pages
- 837-852
- ISSN
- 1615-5289 (ISSN-L)
- 1615-5297
- OpenAlex
- W4392881840
- Cited by
- 50
- References
- 26
- Citations by year
- 2026: 21 · 2025: 25 · 2024: 4
- FWCI
- 104.92
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Universal Access in the Information Society (publishedVersion, cc-by)
- Funders
- Universidad del País Vasco
- Topics
- Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- web accessibility, automated accessibility evaluation, large language models, accessibility testing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
