Ma11y: A Mutation Framework for Web Accessibility Testing
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Abstract
Despite the availability of numerous automatic accessibility testing solutions, web accessibility issues persist on many websites. Moreover, there is a lack of systematic evaluations of the efficacy of current accessibility testing tools. To address this gap, we present the first mutation analysis framework, called Ma11y, designed to assess web accessibility testing tools. Ma11y includes 25 mutation operators that intentionally violate various accessibility principles and an automated oracle to determine whether a mutant is detected by a testing tool. Evaluation on real-world websites demonstrates the practical applicability of the mutation operators and the framework’s capacity to assess tool performance. Our results demonstrate that the current tools cannot identify nearly 50% of the accessibility bugs injected by our framework, thus underscoring the need for the development of more effective accessibility testing tools. Finally, the framework’s accuracy and performance attest to its potential for seamless and automated application in practical settings.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An empirical analysis of flaky tests
Luo, Q., Hariri, F., Eloussi, L., & Marinov, D. (2014) · Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2635868.2635920 · Cited by 450 · cited 1×
“In essence, this is similar to the problem of running flaky tests [33] for the conventional mutation testing.”
MethodPUMA: programmable UI-automation for large-scale dynamic analysis of mobile apps
Hao, S., Liu, B., Nath, S., Halfond, W. G. J., & Govindan, R. (2014) · Proceedings of the 12th annual international conference on Mobile systems, applications, and services · doi:10.1145/2594368.2594390 · Cited by 340 · cited 1×
“Web accessibility testing/evaluation: Web accessibility guidelines have been instrumental in aiding numerous studies and tools to evaluate both web pages [1, 13, 24, 31, 39, 56] and mobile apps [7– 9, 16, 23, 28, 35, 36, 42].”
MethodVigo, M., Brown, J., & Conway, V. (2013) · Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility · doi:10.1145/2461121.2461124 · Cited by 215 · cited 3×
“As a result, many developers opt for automatic testing to evaluate accessibility [15, 48].”
BackgroundAutomated Accessibility Testing of Mobile Apps
Eler, M. M., Rojas, J. M., Ge, Y., & Fraser, G. (2018) · 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2018.00021 · Cited by 116 · cited 1×
“Web accessibility testing/evaluation: Web accessibility guidelines have been instrumental in aiding numerous studies and tools to evaluate both web pages [1, 13, 24, 31, 39, 56] and mobile apps [7– 9, 16, 23, 28, 35, 36, 42].”
MethodAlsaeedi, A. (2020) · Information · doi:10.3390/info11010040 · Cited by 101 · cited 1×
“Other research has focused on comparing tool features, usability, and web page evaluation [18, 30], or on evaluating tool performance on real-world websites with known accessibility issues [6, 26, 29, 41, 48].”
VoxLens: Making Online Data Visualizations Accessible with an Interactive JavaScript Plug-In
Sharif, A., Wang, O. H., Muongchan, A. T., Reinecke, K., & Wobbrock, J. O. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517431 · Cited by 86
Guided Mutation Testing for JavaScript Web Applications
Mirshokraie, S., Mesbah, A., & Pattabiraman, K. (2015) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2014.2371458 · Cited by 80 · cited 1×
“Another research [37] developed mutation operators tailored to JavaScript in web applications, along with generic JavaScript operators.”
MethodMehralian, F., Salehnamadi, N., & Malek, S. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3468604 · Cited by 63 · cited 1×
“Web accessibility testing/evaluation: Web accessibility guidelines have been instrumental in aiding numerous studies and tools to evaluate both web pages [1, 13, 24, 31, 39, 56] and mobile apps [7– 9, 16, 23, 28, 35, 36, 42].”
Self-citationComparing ten WCAG tools for accessibility evaluation of websites
Kumar, S., Jeevithashree, D. V., & Biswas, P. (2021) · Technology and Disability · doi:10.3233/tad-210329 · Cited by 55 · cited 1×
“Other research has focused on comparing tool features, usability, and web page evaluation [18, 30], or on evaluating tool performance on real-world websites with known accessibility issues [6, 26, 29, 41, 48].”
BackgroundWeb accessibility in Turkey: awareness, understanding and practices of user experience professionals
Inal, Y., Rızvanoğlu, K., & Yesilada, Y. (2019) · Universal Access in the Information Society · doi:10.1007/s10209-017-0603-3 · Cited by 50 · cited 1×
“Several studies have investigated the reasons behind such prevalence of accessibility issues and have identified developers’ lack of awareness of website accessibility guidelines and lack of reliable, automated tools as some of the primary reasons [2, 3, 25].”
Research works printed but not linked: 5
- Android. 2023. Accessibility Scanner - Apps on Google Play. https://play.google.com/store/apps/details?id=com.google.android.apps.accessibility.auditor&hl=en_US Last Accessed: December 7, 2023
- Kazuki Nishiura, Yuta Maezawa, Hironori Washizaki, and Shinichi Honiden. 2013. Mutation Analysis for JavaScript Web Applications Testing. Proceedings of the International Conference on Software Engineering and Knowledge Engineering, SEKE, 2013.
- Marian Pădure and Costin Pribeanu. 2019. Exploring the differences between five accessibility evaluation tools. In Proceedings of RoCHI 2019.
- Navid Salehnamadi, Forough Mehralian, and Sam Malek. 2022. GroundHog: An Automated Accessibility Crawler for Mobile Apps. In 2022 37th IEEE/ACM International Conference on Automated Software Engineering. ACM New York, NY, USA, Rochester, Michigan, USA. 13 pages.
- Claudia Timbi-Sisalima, Cecilia I. Martin Amor, Silvia Otón Tortosa, José R. Hilera, and Juan Aguado-Delgado. 2016. Comparative Analysis of Online Web Accessibility Evaluation Tools. In Information Systems Development: Complexity in Information Systems Development (ISD2016 Proceedings), Jerzy Gołuchowski, Malgorzata Pańkowska, Chris Barry, Michael Lang, Henry Linger, and Christoph Schneider (Eds.). University of Economics in Katowice, Katowice, Poland. isbn:978-83-7875-307-0 http://aisel.aisnet.org/isd2014/proceedings2016/CreativitySupport/3
Cited by
An Empirical Study on Evaluating Accessible Code Generation Capabilities of LLMs
Suh, H., Tafreshipour, M., Malek, S., & Ahmed, I. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3820782 · Cited by 24
Self-citationCoverage of web accessibility guidelines provided by automated checking tools
Fischer, T., Lundell, B., & Gamalielsson, J. (2025) · Universal Access in the Information Society · doi:10.1007/s10209-025-01263-x · Cited by 16
“The only publication in English we are aware of that discussed A11yWatch is [ 56] where this tool is praised for its ‘effectiveness in detecting Operability violations’.”
Human or LLM? A Comparative Study on Accessible Code Generation Capability
Suh, H., Tafreshipour, M., Malek, S., & Ahmed, I. (2025) · arXiv · doi:10.48550/arxiv.2503.15885 · Cited by 15
Self-citationLLMs for Accessibility in Mobile Apps: Detection and Repair
Aljedaani, W., Aljohani, A., Eler, M. M., Habib, A., & Do, H. (2025) · International Web for All Conference (W4A) · doi:10.1145/3744257.3744270 · Cited by 12
“To mitigate this, we identified the most common accessibility violations based on existing literature [15, 16, 20, 21, 23, 30, 45, 55, 65, 66, 69, 75], ensuring a diverse selection of violation types for a comprehensive evaluation.”
LLM-Driven Optimization of HTML Structure to Support Screen Reader NavigationPreprint
Yu, Y., Ryskeldiev, B., Tsutsui, A., Gillingham, M., & Wang, Y. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.18701 · Cited by 9
“Due to the time-intensive nature of manual remediation [2], automated approaches—such as ChatGPT-based accessibility repair [43], evaluation tools [51], and context-aware image description systems [19]—have gained traction.”
Metadata
- Authors
- Tafreshipour, M., Deshpande, A., Mehralian, F., Ahmed, I., & Malek, S.
- Affiliations
- University of California, Irvine (US)
- Venue
- Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis
- Publisher
- ACM
- Published
- 2024-09-11
- Pages
- 100-111
- OpenAlex
- W4402457672
- Cited by
- 10
- References
- 21
- Citations by year
- 2026: 5 · 2025: 5
- FWCI
- 2.73
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) (publishedVersion, cc-by)
- eScholarship (California Digital Library) (submittedVersion, cc-by)
- Funders
- National Science Foundation (2211790)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- web accessibility testing, mutation analysis, mutation operator, accessibility testing tools, tool evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Ma11y: A Mutation Framework for Web Accessibility Testing. https://app.cochat.ai/papers/10.1145/3650212.3652113
- The paper
- Tafreshipour, M., Deshpande, A., Mehralian, F., Ahmed, I., & Malek, S. (2024). Ma11y: A Mutation Framework for Web Accessibility Testing. Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. https://doi.org/10.1145/3650212.3652113
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.