Accessibility Metatesting: Comparing Nine Testing Tools
Open accessAbstract
Automated web accessibility testing tools have been found complementary. The implication: To catch as many issues as possible, use multiple tools. Doing this efficiently entails integration costs. Is there a small set of tools that, together, make additional tools redundant? I approach this problem by comparing nine comprehensive accessibility testing tools that are amenable to integration: alfa, axe-core, Continuum, Equal Access, HTML CodeSniffer, Nu Html Checker, QualWeb, Tenon, and WAVE. I tested 121 web pages of interest to CVS Health with these tools. Each tool only fractionally duplicated any other tool. Each discovered numerous issue instances missed by all the others. Thus, testing with all nine tools was substantially more informative than testing with any subset.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
These tools were selected due to their widespread use in accessibility research and practice and for their established coverage of WCAG success criteria.
“These tools were selected due to their widespread use in accessibility research and practice and for their established coverage of WCAG success criteria [40, 53, 62].”
From A Multi-Agent Framework for Accessible Web Code Generation (2026) · cites this paper as [62]
The Introduction and Related Work describe accessibility tools used in practice and prior research, and §3 says the nine tools seek to measure conformance to industry standards. But §3 identifies their selection criteria as maintenance, low cost, and availability as REST APIs or NPM packages—not widespread use or established coverage of WCAG success criteria. No page numbers are available in our copy. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 2 · Cited by 14
Research works printed but not linked: 7
- Siddikjon Gaibullojonovich Abduganiev. 2017. Towards Automated Web Ac- cessibility Evaluation: A Comparative Study. Retrieved 2023-01-12 from https: //www.mecs-press.org/ijitcs/ijitcs-v9-n9/IJITCS-V9-N9-3.pdf
- Jinat Ara and Cecilia Sik-Lanyi. 2022. Investigation of COVID-19 Vaccine Infor- mation Websites across Europe and Asia Using Automated Accessibility Protocols. Retrieved 2023-03-04 from https://www.mdpi.com/1660-4601/19/5/2867
- Andreas Burkard, Gottfried Zimmermann, and Bettina Schwarzer. 2021. Monitor- ing Systems for Checking Websites on Accessibility. Retrieved 2023-03-04 from https://www.frontiersin.org/articles/10.3389/fcomp.2021.628770/full · DOI
- Andreas Burkard, Gottfried Zimmermann, and Bettina Schwarzer. 2021. Moni- toring Systems for Checking Websites on Accessibility: Supplementary Materials. Retrieved 2023-03-04 from https://www.frontiersin.org/articles/10.3389/fcomp. 2021.628770/full#supplementary-material · DOI
- Level Access. 2023. Continuum Community Edition. Retrieved 2023-01-11 from https://www.webaccessibility.com/tools/
- Carlos Alberto Silva, Arthur F. B. A. de Oliveira, Delvani Matônio Mateus, Heitor Augustus Xavier Costa, and André Pimenta Freire. 2019. Types of Problems Encountered by Automated Tool Accessibility Assessments, Expert Inspections and User Testing: A Systematic Literature Mapping. Retrieved 2023-03-13 from https: //doi.org/10.1145/3357155.3358479 · DOI
- Ulas, Yüksel, Hasan Sözer, and Murat Şensoy. 2014. Trust-based Fu- sion of Classifiers for Static Code Analysis. Retrieved 2023-01-12 from http://confcats_isif.s3.amazonaws.com/web-files/event/proceedings/html/ 2014Proceedings/papers/fusion2014_submission_118/paper118.pdf
- Authors
- Pool, J. R.
- Affiliations
- CVS Health (United States) (US)
- Corresponding author
- Pool, Jonathan Robert
- Venue
- 20th International Web for All Conference
- Publisher
- ACM
- Published
- 2023-04-30
- Pages
- 1-4
- OpenAlex
- W4366342552
- Cited by
- 14
- References
- 2
- Citations by year
- 2026: 8 · 2025: 3 · 2024: 3
- FWCI
- 5.87
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- International Web for All Conference (W4A) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- web accessibility, automated accessibility testing, tool comparison, tool integration, waves, tenon, equal access, continuum, CVS Health
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
