CoChat

Benchmarking web accessibility evaluation tools: measuring the harm of sole reliance on automated tests

Vigo, Markel, Brown, Justin, Conway, Vivienne · 2013 · Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility

The DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.

Abstract

The use of web accessibility evaluation tools is a widespread practice. Evaluation tools are heavily employed as they help in reducing the burden of identifying accessibility barriers. However, an over-reliance on automated tests often leads to setting aside further testing that entails expert evaluation and user tests. In this paper we empirically show the capabilities of current automated evaluation tools. To do so, we investigate the effectiveness of 6 state-of-the-art tools by analysing their coverage, completeness and correctness with regard to WCAG 2.0 conformance. We corroborate that relying on automated tests alone has negative effects and can have undesirable consequences. Coverage is very narrow as, at most, 50% of the success criteria are covered. Similarly, completeness ranges between 14% and 38%; however, some of the tools that exhibit higher completeness scores produce lower correctness scores (66-71%) due to the fact that catching as many violations as possible can lead to an increase in false positives. Therefore, relying on just automated tests entails that 1 of 2 success criteria will not even be analysed and among those analysed, only 4 out of 10 will be caught at the further risk of generating false positives.

Overview

Integrity signals

Registry

Crossref · 10.1145/2461121.2461124

Cites

25 of 25 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

6 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Citations

Cites

Show all 25 references

Research works printed but not linked: 2

  1. S. Abou-Zahra . Evaluation and report language (earl) 1.0 schema -- w3c working draft , 2011 . S. Abou-Zahra. Evaluation and report language (earl) 1.0 schema -- w3c working draft, 2011.
  2. W3C- WAI. Conformance evaluation of web sites for accessibility . 2005 . W3C-WAI. Conformance evaluation of web sites for accessibility. 2005.

Cited by

See all 229

Metadata

Authors
Vigo, M., Brown, J., & Conway, V.
Affiliations
  • University of Manchester (GB)
  • Edith Cowan University (AU)
Venue
Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility
Publisher
ACM
Published
2013-05-13
Pages
1-10
OpenAlex
W2059464277
Cited by
215
References
25
Citations by year
2026: 20 · 2025: 24 · 2024: 19 · 2023: 16 · 2022: 22 · 2021: 14 · 2020: 19 · 2019: 19 · 2018: 15 · 2017: 11 · 2016: 11 · 2015: 17 · 2014: 7 · 2013: 1
FWCI
19.49
Type
conference paper
Language
en
Open access
closed
Locations
  • International Cross-Disciplinary Conference on Web Accessibility (W4A) (publishedVersion)
  • Research Online (Edith Cowan University) (submittedVersion)
Topics
  • Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
Keywords
web accessibility, WCAG 2.0, conformance testing, false positives, expert evaluation, user testing, access barriers

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Benchmarking web accessibility evaluation tools: measuring the harm of sole reliance on automated tests. https://app.cochat.ai/papers/10.1145/2461121.2461124
The paper
Vigo, M., Brown, J., & Conway, V. (2013). Benchmarking web accessibility evaluation tools: measuring the harm of sole reliance on automated tests. Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility. https://doi.org/10.1145/2461121.2461124

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.