CoChat
Sign Up
PreprintCited by 15Digital Accessibility for DisabilitiesSoftware Engineering ResearchWeb Applications and Data Management

Human or LLM? A Comparative Study on Accessible Code Generation Capability

Open access
Suh, Hyunjae, Tafreshipour, Mahan, Malek, Sam, Ahmed, Iftekhar·2025·arXiv
DOI 10.48550/arxiv.2503.15885arXiv 2503.15885
Open this paper in CoChat — free accountOpen PDF

Abstract

Web accessibility is essential for inclusive digital experiences, yet the accessibility of LLM-generated code remains underexplored. This paper presents an empirical study comparing the accessibility of web code generated by GPT-4o and Qwen2.5-Coder-32B-Instruct-AWQ against human-written code. Results show that LLMs often produce more accessible code, especially for basic features like color contrast and alternative text, but struggle with complex issues such as ARIA attributes. We also assess advanced prompting strategies (Zero-Shot, Few-Shot, Self-Criticism), finding they offer some gains but are limited. To address these gaps, we introduce FeedA11y, a feedback-driven ReAct-based approach that significantly outperforms other methods in improving accessibility. Our work highlights the promise of LLMs for accessible code generation and emphasizes the need for feedback-based techniques to address persistent challenges.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2503.15885

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Prior work indicates that GPT-series models are effective at inferring implicit accessibility requirements from underspecified prompts.

    “Prior work further indicates that GPT-series models are effective at inferring implicit accessibility requirements from underspecified prompts, making them suitable for initializing accessibility-aware design constraints [68].”

    From A Multi-Agent Framework for Accessible Web Code Generation (2026) · cites this paper as [68]

    Section 3.4 (p. 6) says naive code-generation prompts did not mention accessibility, and Section 4.2 (pp. 8–9) reports that GPT-4o’s naive-generated code had fewer measured accessibility issues than human-written code. That supports accessibility performance without explicit instructions, but the paper does not establish that GPT-4o inferred implicit requirements from underspecified prompts. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 53

  1. [19][19] Wajdi Aljedaani, Abdulrahman Habib, Ahmed Aljohani, Marcelo Eler, and Yunhe Feng. 2024. Does ChatGPT Generate Accessible Code? Investigating Accessibility Challenges in LLM-Generated Source Code. In Proceedings of the 21st International Web for All Conference (Singapore, Singapore) (W4A '24) . Association for Computing Machinery, New York, NY, USA, 165-176. https: //doi.org/10.1145/3677846.3677854 - · DOI · Paper page
  2. [18][18] Hasan O Al-Sakran and Mohammed A Alsudairi. 2021. Usability and accessibility assessment of Saudi Arabia mobile e-government websites. IEEE Access 9 (2021), 48254-48275. -
  3. [20][20] Zainab AlMeraj, Fatima Boujarwah, Dari Alhuwail, and Rumana Qadri. 2021. Evaluating the accessibility of higher education institution websites in the State of Kuwait: empirical evidence. Universal Access in the Information Society 20, 1 (2021), 121-138. -
  4. [22][22] Kewalin Angkananon, Mike Wald, and Lester Gilbert. 2015. Technology enhanced interaction framework and method for accessibility in Thai museums. In 2015 3rd international conference on information and communication technology (icoict) . IEEE, 316-321. -
  5. [21][21] Abdulaziz Alshayban, Iftekhar Ahmed, and Sam Malek. 2020. Accessibility issues in Android apps: state of affairs, sentiments, and ways forward. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering (Seoul, South Korea) (ICSE '20) . Association for Computing Machinery, New York, NY, USA, 1323-1334. https://doi.org/10.1145/3377811.3380392 - · DOI
  6. [23][23] Patrick Bareiß, Beatriz Souza, Marcelo d'Amorim, and Michael Pradel. 2022. Code Generation Tools (Almost) for Free? A Study of Few-Shot, Pre-Trained Language Models on Code. arXiv:2206.01335 [cs.SE] https://arxiv.org/abs/2206.01335 - · arXiv
  7. [25][25] Milton Campoverde-Molina, Sergio Lujan-Mora, and Llorenc Valverde Garcia. 2020. Empirical studies on web accessibility of educational websites: A systematic literature review. IEEE Access 8 (2020), 91676-91700. -
  8. [24][24] Tingting Bi, Xin Xia, David Lo, and Aldeida Aleti. 2021. A First Look at Accessibility Issues in Popular GitHub Projects. In 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME) . 390-401. https: //doi.org/10.1109/ICSME52107.2021.00041 - · DOI
  9. [26][26] Nilesh Dhulshette, Sapan Shah, and Vinay Kulkarni. 2025. Hierarchical Repository-Level Code Summarization for Business Applications Using Local LLMs. arXiv:2501.07857 [cs.SE] https://arxiv.org/abs/2501.07857 - · arXiv
  10. [27][27] Mingzhe Du, Anh Tuan Luu, Bin Ji, and See-Kiong Ng. 2024. Mercury: An efficiency benchmark for llm code synthesis. arXiv preprint arXiv:2402.07844 (2024). , , - and from blind and visually impaired people. In Proceedings of the 12th ACM International Conference on PErvasive Technologies Related to Assistive Environments . 186-195. - · arXiv
  11. [30][30] Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M Zhang. 2023. Large language models for software engineering: Survey and open problems. In 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) . IEEE, 31-53. -
  12. [29][29] DeepSeek-AI et al. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948 [cs.CL] https://arxiv.org/ abs/2501.12948 - · arXiv
  13. [31][31] Mexhid Ferati, Bahtijar Vogel, Arianit Kurti, Bujar Raufi, and David Salvador Astals. 2016. Web accessibility for visually impaired people: requirements and design issues. In Usability-and Accessibility-Focused Requirements Engineering: First International Workshop, UsARE 2012, Held in Conjunction with ICSE 2012, Zurich, Switzerland, June 4, 2012 and Second International Workshop, UsARE 2014, Held in Conjunction with RE 2014, Karlskrona, Sweden, August 25, 2014, Revised Selected Papers 1 . Springer, 79-96. -
  14. [33][33] Mark G Friedman and Diane Nelson Bryen. 2007. Web accessibility design recommendations for people with cognitive disabilities. Technology and disability 19, 4 (2007), 205-212. -
  15. [32][32] Jane Forman and Laura Damschroder. 2007. Qualitative content analysis. In Empirical methods for bioethics: A primer . Emerald Group Publishing Limited, 39-62. -
  16. [34][34] Yujia Fu, Peng Liang, Amjed Tahir, Zengyang Li, Mojtaba Shahin, and Jiaxin Yu. 2023. Security weaknesses of copilot generated code in github. arXiv preprint arXiv:2310.02059 (2023). - · arXiv
  17. [36][36] Shawn Lawton Henry, Shadi Abou-Zahra, and Judy Brewer. 2014. The role of accessibility in a universal web. In Proceedings of the 11th Web for all Conference . 1-4. -
  18. [37][37] Michael Heron, Vicki L Hanson, and Ian W Ricketts. 2013. ACCESS: a technical framework for adaptive accessibility support. In Proceedings of the 5th ACM SIGCHI symposium on Engineering interactive computing systems . 33-42. -
  19. [39][39] Claudia S Idrobo, María I Idrobo, Gabriel E Chan-chí, and María I Vidal. 2017. Accessibility guide for the design and implementation of websites according to the NTC 5854 standard. In Colombian Conference on Computing . Springer, 572-585. -
  20. [38][38] Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, Yunlong Feng, Xingzhang Ren, Xuancheng Ren, Jingren Zhou, and Junyang Lin. 2024. Qwen2.5-Coder Technical Report. arXiv:2409.12186 [cs.CL] https://arxiv.org/ abs/2409.12186 - · arXiv
  21. [40][40] Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024. A Survey on Large Language Models for Code Generation. arXiv:2406.00515 [cs.CL] https://arxiv.org/abs/2406.00515 - · arXiv
  22. [42][42] Eirini Kalliamvakou, Georgios Gousios, Kelly Blincoe, Leif Singer, Daniel M German, and Daniela Damian. 2014. The promises and perils of mining github. In Proceedings of the 11th working conference on mining software repositories . 92-101. -
  23. [41][41] Kalpana Johari and Arvinder Kaur. 2013. Web Accessibility: Designing and Testing of Web Based Application for Persons with Disabilities. In Computer Networks & Communications (NetCom) Proceedings of the Fourth International Conference on Networks & Communications . Springer, 729-736. -
  24. [43][43] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems 35 (2022), 22199-22213. -
  25. [45][45] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. 2022. Competition-level code generation with alphacode. Science 378, 6624 (2022), 1092-1097. -
  26. [46][46] Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, et al. 2021. Fewshot learning with multilingual language models. arXiv preprint arXiv:2112.10668 (2021). - · arXiv
  27. [48][48] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2024. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems 36 (2024). -
  28. [47][47] Lin Ling. 2024. Evaluating Social Bias in Code Generation Models. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering . 695-697. -
  29. [49][49] Mingwei Liu, Tianyong Yang, Yiling Lou, Xueying Du, Ying Wang, and Xin Peng. 2023. CodeGen4Libs: A Two-Stage Approach for Library-Oriented Code Generation. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 434-445. , , -
  30. [52][52] Kelly Mack, Emma McDonnell, Dhruv Jain, Lucy Lu Wang, Jon E. Froehlich, and Leah Findlater. 2021. What do we mean by 'accessibility research'? A literature survey of accessibility papers in CHI and ASSETS from 1994 to 2019. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems . 1-18. -
  31. [51][51] Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568 (2023). - · arXiv
  32. [53][53] Carlos Máñez-Carvajal, Jose Francisco Cervera-Mérida, and Rocío FernándezPiqueras. 2021. Web accessibility evaluation of top-ranking university Web sites in Spain, Chile and Mexico. Universal Access in the Information Society 20 (2021), 179-184. -
  33. [55][55] Liang Niu, Shujaat Mirza, Zayd Maradni, and Christina Pöpper. 2023. { CodexLeaks } : Privacy leaks from code generation language models in { GitHub } copilot. In 32nd USENIX Security Symposium (USENIX Security 23) . 2133-2150. -
  34. [54][54] Darliane Miranda. 2021. A Web Accessibility Requirements Framework for Agile Development. In 2021 IEEE 29th International Requirements Engineering Conference (RE) . IEEE, 474-479. -
  35. [58][58] Surjit Paul and Saini Das. 2020. Accessibility and usability analysis of Indian e-government websites. Universal Access in the Information Society 19, 4 (2020), 949-957. -
  36. [57][57] Achraf Othman, Amira Dhouib, and Aljazi Nasser Al Jabor. 2023. Fostering websites accessibility: A case study on the use of the Large Language Models ChatGPT for automatic remediation. In Proceedings of the 16th International Conference on PErvasive Technologies Related to Assistive Environments . 707-713. -
  37. [59][59] Cara Peters and David A Bradbard. 2010. Web accessibility: an introduction and ethical implications. Journal of Information, Communication and Ethics in Society 8, 2 (2010), 206-232. -
  38. [60][60] Ruizhong Qiu, Weiliang Will Zeng, Hanghang Tong, James Ezick, and Christopher Lott. 2024. How Efficient is LLM-Generated Code? A Rigorous & High-Standard Benchmark. arXiv preprint arXiv:2406.06647 (2024). - · arXiv
  39. [62][62] Bujar Raufi, Mexhid Ferati, Xhemal Zenuni, Jaumin Ajdari, and Florije Ismaili. 2015. Methods and techniques of adaptive web accessibility for the blind and visually impaired. Procedia-Social and Behavioral Sciences 195 (2015), 1999-2007. -
  40. [64][64] X. Ren, X. Ye, D. Zhao, Z. Xing, and X. Yang. 2023. From Misuse to Mastery: Enhancing Code Generation with Knowledge-Driven AI Chaining. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE Computer Society, Los Alamitos, CA, USA, 976-987. https://doi.org/10. 1109/ASE56229.2023.00143 -
  41. [63][63] Xiaoxue Ren, Xinyuan Ye, Dehai Zhao, Zhenchang Xing, and Xiaohu Yang. 2023. From Misuse to Mastery: Enhancing Code Generation with Knowledge-Driven AI Chaining. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 976-987. -
  42. [65][65] Germania Rodríguez, Jennifer Pérez, Samanta Cueva, and Rommel Torres. 2017. A framework for improving web accessibility and usability of Open Course Ware sites. Computers & education 109 (2017), 197-215. -
  43. [67][67] MohammedLatif Siddiq and Joanna Santos. 2023. Generate and pray: Using sallms to evaluate the security of llm generated code. arXiv preprint arXiv:2311.00889 (2023). - · arXiv
  44. [68][68] Dominik Sobania, Martin Briesch, Carol Hanna, and Justyna Petke. 2023. An analysis of the automatic bug fixing performance of chatgpt. In 2023 IEEE/ACM International Workshop on Automated Program Repair (APR) . IEEE, 23-30. -
  45. [70][70] Xiaoyu Tan, Shaojie Shi, Xihe Qiu, Chao Qu, Zhenting Qi, Yinghui Xu, and Yuan Qi. 2023. Self-Criticism: Aligning Large Language Models with their Understanding of Helpfulness, Honesty, and Harmlessness. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track , Mingxuan Wang and Imed Zitouni (Eds.). Association for Computational Linguistics, Singapore, 650-662. https://doi.org/10.18653/v1/2023.emnlp-industry.62 - · DOI
  46. [73][73] Bolin Wei, Ge Li, Xin Xia, Zhiyi Fu, and Zhi Jin. 2019. Code generation as a dual task of code summarization. Advances in neural information processing systems 32 (2019). -
  47. [72][72] Chaozheng Wang, Zongjie Li, Cuiyun Gao, Wenxuan Wang, Ting Peng, Hailiang Huang, Yuetang Deng, Shuai Wang, and Michael R Lyu. 2024. Exploring Multi-Lingual Bias of Large Code Models in Code Generation. arXiv preprint arXiv:2404.19368 (2024). - · arXiv
  48. [74][74] Zhou Yang, Zhipeng Zhao, Chenyu Wang, Jieke Shi, Dongsum Kim, Donggyun Han, and David Lo. 2023. Gotcha! this model uses my code! evaluating membership leakage risks in code models. arXiv preprint arXiv:2310.01166 (2023). - · arXiv
  49. [76][76] Yong Jeong Yi. 2020. Web accessibility of healthcare Web sites of Korean government and public agencies: a user test for persons with visual impairment. Universal Access in the Information Society 19, 1 (2020), 41-56. -
  50. [75][75] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629 (2022). - · arXiv
  51. [77][77] Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li, Qianxiang Wang, and Tao Xie. 2024. Codereval: A benchmark of pragmatic code generation with generative pre-trained models. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering . 1-12. -
  52. [79][79] Quanjun Zhang, Tongke Zhang, Juan Zhai, Chunrong Fang, Bowen Yu, Weisong Sun, and Zhenyu Chen. 2023. A critical review of large language model on software engineering: An example from chatgpt and automated program repair. arXiv preprint arXiv:2310.08879 (2023). - · arXiv
  53. [78][78] Zhiyuan Yu, Yuhao Wu, Ning Zhang, Chenguang Wang, Yevgeniy Vorobeychik, and Chaowei Xiao. 2023. Codeipprompt: Intellectual property infringement assessment of code language models. In International Conference on Machine Learning . PMLR, 40373-40389.

Cited by

Sorted by citations

Authors
Suh, H., Tafreshipour, M., Malek, S., & Ahmed, I.
Affiliations
University of California, Irvine (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
1049-331X (ISSN-L)
DOI
10.48550/arxiv.2503.15885
arXiv
2503.15885
OpenAlex
W7164496665
Cited by
15
References
0
Citations by year
2026: 10 · 2025: 5
FWCI
134.44
Type
journal article
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • ACM Transactions on Software Engineering and Methodology (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Information Systems › Web Applications and Data Management
Keywords
web accessibility, LLM-generated code, color contrast, alternative text, prompting strategies, React, GPT-4o, Qwen2.5-Coder

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review