CoChat

Ma11y: A Mutation Framework for Web Accessibility Testing

Open access
Tafreshipour, Mahan, Deshpande, Anmol, Mehralian, Forough, Ahmed, Iftekhar, Malek, Sam · 2024 · Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis

The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.

Abstract

Despite the availability of numerous automatic accessibility testing solutions, web accessibility issues persist on many websites. Moreover, there is a lack of systematic evaluations of the efficacy of current accessibility testing tools. To address this gap, we present the first mutation analysis framework, called Ma11y, designed to assess web accessibility testing tools. Ma11y includes 25 mutation operators that intentionally violate various accessibility principles and an automated oracle to determine whether a mutant is detected by a testing tool. Evaluation on real-world websites demonstrates the practical applicability of the mutation operators and the framework’s capacity to assess tool performance. Our results demonstrate that the current tools cannot identify nearly 50% of the accessibility bugs injected by our framework, thus underscoring the need for the development of more effective accessibility testing tools. Finally, the framework’s accuracy and performance attest to its potential for seamless and automated application in practical settings.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3650212.3652113

Cites

21 of 21 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 21 references

Research works printed but not linked: 5

  1. Android. 2023. Accessibility Scanner - Apps on Google Play. https://play.google.com/store/apps/details?id=com.google.android.apps.accessibility.auditor&hl=en_US Last Accessed: December 7, 2023
  2. Kazuki Nishiura, Yuta Maezawa, Hironori Washizaki, and Shinichi Honiden. 2013. Mutation Analysis for JavaScript Web Applications Testing. Proceedings of the International Conference on Software Engineering and Knowledge Engineering, SEKE, 2013.
  3. Marian Pădure and Costin Pribeanu. 2019. Exploring the differences between five accessibility evaluation tools. In Proceedings of RoCHI 2019.
  4. Navid Salehnamadi, Forough Mehralian, and Sam Malek. 2022. GroundHog: An Automated Accessibility Crawler for Mobile Apps. In 2022 37th IEEE/ACM International Conference on Automated Software Engineering. ACM New York, NY, USA, Rochester, Michigan, USA. 13 pages.
  5. Claudia Timbi-Sisalima, Cecilia I. Martin Amor, Silvia Otón Tortosa, José R. Hilera, and Juan Aguado-Delgado. 2016. Comparative Analysis of Online Web Accessibility Evaluation Tools. In Information Systems Development: Complexity in Information Systems Development (ISD2016 Proceedings), Jerzy Gołuchowski, Malgorzata Pańkowska, Chris Barry, Michael Lang, Henry Linger, and Christoph Schneider (Eds.). University of Economics in Katowice, Katowice, Poland. isbn:978-83-7875-307-0 http://aisel.aisnet.org/isd2014/proceedings2016/CreativitySupport/3

Cited by

See all 19

Metadata

Authors
Tafreshipour, M., Deshpande, A., Mehralian, F., Ahmed, I., & Malek, S.
Affiliations
University of California, Irvine (US)
Venue
Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis
Publisher
ACM
Published
2024-09-11
Pages
100-111
OpenAlex
W4402457672
Cited by
10
References
21
Citations by year
2026: 5 · 2025: 5
FWCI
2.73
Type
conference paper
Language
en
Open access
Locations
  • ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) (publishedVersion, cc-by)
  • eScholarship (California Digital Library) (submittedVersion, cc-by)
Funders
National Science Foundation (2211790)
Topics
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
  • Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
Keywords
web accessibility testing, mutation analysis, mutation operator, accessibility testing tools, tool evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Ma11y: A Mutation Framework for Web Accessibility Testing. https://app.cochat.ai/papers/10.1145/3650212.3652113
The paper
Tafreshipour, M., Deshpande, A., Mehralian, F., Ahmed, I., & Malek, S. (2024). Ma11y: A Mutation Framework for Web Accessibility Testing. Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. https://doi.org/10.1145/3650212.3652113

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.