Benchmarking web accessibility evaluation tools: measuring the harm of sole reliance on automated tests
The DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
The use of web accessibility evaluation tools is a widespread practice. Evaluation tools are heavily employed as they help in reducing the burden of identifying accessibility barriers. However, an over-reliance on automated tests often leads to setting aside further testing that entails expert evaluation and user tests. In this paper we empirically show the capabilities of current automated evaluation tools. To do so, we investigate the effectiveness of 6 state-of-the-art tools by analysing their coverage, completeness and correctness with regard to WCAG 2.0 conformance. We corroborate that relying on automated tests alone has negative effects and can have undesirable consequences. Coverage is very narrow as, at most, 50% of the success criteria are covered. Similarly, completeness ranges between 14% and 38%; however, some of the tools that exhibit higher completeness scores produce lower correctness scores (66-71%) due to the fact that catching as many violations as possible can lead to an increase in false positives. Therefore, relying on just automated tests entails that 1 of 2 success criteria will not even be analysed and among those analysed, only 4 out of 10 will be caught at the further risk of generating false positives.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The study of Vigo et al. (2013) shows the problems of relying only on automated web evaluation tools.”
From Monitoring Systems for Checking Websites on Accessibility (2021) · cites this paper as Vigo et al. (2013)
In the citing paper
“Currently, there are few studies on the comparison of tools for automatic web accessibility evaluation.”
From Monitoring Systems for Checking Websites on Accessibility (2021) · cites this paper as Vigo et al. (2013)
In the citing paper
“Therefore, such tools will not replace manual checks in the foreseeable future (Vigo et al., 2013), but can only be used as a support tool.”
From Monitoring Systems for Checking Websites on Accessibility (2021) · cites this paper as Vigo et al., 2013
In the citing paper
“Nevertheless, these tools cannot replace manual accessibility checks (Vigo et al., 2013).”
From Monitoring Systems for Checking Websites on Accessibility (2021) · cites this paper as Vigo et al., 2013
In the citing paper
“While prior studies (Vigo et al., 2013;Abduganiev and Gaibullojonovich, 2017;Pȃdure and Pribeanu, 2019) have mostly evaluated free tools for checking accessibility, this work compares some commercial AMS that can monitor entire websites and evaluate their accessibility over time.”
From Monitoring Systems for Checking Websites on Accessibility (2021) · cites this paper as Vigo et al., 2013
6 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Automatic detection of text genre
Kessler, B., Numberg, G., & Schütze, H. (1997) · Proceedings of the 35th annual meeting on Association for Computational Linguistics - · doi:10.3115/976909.979622 · Cited by 358 · cited 1×
“) is feasible [13], which is useful to raise warnings in those sites that require legal commitments (see SC “3.”
BackgroundGuidelines are only half of the story: accessibility problems encountered by blind users on the web
Power, C., Freire, A., Petrie, H., & Swallow, D. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2207736 · Cited by 327 · cited 1×
“on guidelines alone leaves out a number of issues as it was found that guidelines do not cover the whole spectrum of accessibility problems, as they only cover around 50% of accessibility problems encountered by users [19].”
BackgroundImproving web accessibility: a study of webmaster perceptions
Lazar, J., Dudley-Sponaugle, A., & Greenidge, K.-D. (2004) · Computers in Human Behavior · doi:10.1016/j.chb.2003.10.018 · Cited by 326 · cited 1×
“However, a study of webmasters’ perceptions revealed that the majority of them are familiar with accessibility guidelines [15].”
BackgroundMankoff, J., Fait, H., & Tran, T. (2005) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1054972.1054979 · Cited by 207 · cited 1×
“In one of the earliest studies to explore multiple methods of evaluation and to discuss levels of expertise of evaluators, multiple approaches to website evaluation were analysed including automated tools, expert evaluations and user testing [17].”
MethodBeyond Conformance: The Role of Accessibility Evaluation Methods
Brajnik, G. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-85200-1_9 · Cited by 114 · cited 2×
“Optimally, the formative assessment process should be ongoing from the initiation of a project, and continue throughout the operational lifespan of the web page or site [6, 21].”
BackgroundAccessibility designer: visualizing usability for the blind
Takagi, H., Asakawa, C., Fukuda, K., & Maeda, J. (2003) · Proceedings of the 6th international ACM SIGACCESS conference on Computers and accessibility · doi:10.1145/1028630.1028662 · Cited by 113 · cited 1×
“The techniques used to assess the accessibility of each web page are diverse across judges: evaluation tools that diverge from the ones benchmarked (WAVE(2)), mark-up validators, browser extensions for developers (Firebug, Web Developer Toolbar, Web Accessibility Toolbar), screen readers (VoiceOver, NVDA) and evaluati…”
MethodAutomatic web accessibility metrics: Where we are and where we can go
Vigo, M., & Brajnik, G. (2011) · Interacting with Computers · doi:10.1016/j.intcom.2011.01.001 · Cited by 112 · cited 1×
“To name a few, accessibility evaluation is also employed in quality assurance of web applications, accessibility observatories, information retrieval and user-adapted interaction scenarios [25].”
BackgroundSelf-citationA comparative test of web accessibility evaluation methods
Brajnik, G. (2008) · Proceedings of the 10th international ACM SIGACCESS conference on Computers and accessibility · doi:10.1145/1414471.1414494 · Cited by 111 · cited 2×
“is not likely to distinguish reliably between important and non-important accessibility problems” [7].”
BackgroundLanguage identification in web pages
Martins, B., & Silva, M. J. (2005) · ACM Symposium on Applied Computing · doi:10.1145/1066677.1066852 · Cited by 103 · cited 1×
“Natural Language Processing) such as detecting changes on the language [18] for SC such as “3.”
BackgroundComparing accessibility evaluation tools: a method for tool effectiveness
Brajnik, G. (2004) · Universal Access in the Information Society · doi:10.1007/s10209-004-0105-y · Cited by 101 · cited 3×
“The most relevant related work upon which this paper is based on analysed the effectiveness of automated tools [5], where effectiveness encompasses completeness, correctness and specificity qualities.”
BackgroundMethodResultInfluential
Research works printed but not linked: 2
- S. Abou-Zahra . Evaluation and report language (earl) 1.0 schema -- w3c working draft , 2011 . S. Abou-Zahra. Evaluation and report language (earl) 1.0 schema -- w3c working draft, 2011.
- W3C- WAI. Conformance evaluation of web sites for accessibility . 2005 . W3C-WAI. Conformance evaluation of web sites for accessibility. 2005.
Cited by
The Current Status of Accessibility in Mobile Apps
Yan, S., & Ramachandran, P. G. (2019) · ACM Transactions on Accessible Computing · doi:10.1145/3300176 · Cited by 165
“A combination of test suites (unit test apps), test apps, and commercial apps are used to verify the correctness of the results from MAC, because test suites alone may miss some types of errors and also can produce false positives [5, 27].”
BackgroundMethodInfluentialAlsaeedi, A. (2020) · Information · doi:10.3390/info11010040 · Cited by 101
“The performance and suitability of accessibility evaluation tools can be assessed and compared in two ways [5]: by selecting a representative sample of websites or using test suites.”
BackgroundMethodInfluentialEvaluating the accessibility of public health websites: An exploratory cross-country study
Alajarmeh, N. (2022) · Universal Access in the Information Society · doi:10.1007/s10209-020-00788-7 · Cited by 89
“[31] as they highlighted some limitations and variances of automatic tools in terms of accuracy and comprehensiveness in detecting real accessibility issues when compared to manual evaluations.”
BackgroundAcosta-Vargas, P., Acosta, T., & Lujan-Mora, S. (2018) · IEEE Access · doi:10.1109/access.2018.2848978 · Cited by 92
Accessibility and Software Engineering Processes: A Systematic Literature Review
Paiva, D. M. B., Freire, A. P., & de Mattos Fortes, R. P. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110819 · Cited by 87
Metadata
- Authors
- Vigo, M., Brown, J., & Conway, V.
- Affiliations
- University of Manchester (GB)
- Edith Cowan University (AU)
- Venue
- Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility
- Publisher
- ACM
- Published
- 2013-05-13
- Pages
- 1-10
- OpenAlex
- W2059464277
- Cited by
- 215
- References
- 25
- Citations by year
- 2026: 20 · 2025: 24 · 2024: 19 · 2023: 16 · 2022: 22 · 2021: 14 · 2020: 19 · 2019: 19 · 2018: 15 · 2017: 11 · 2016: 11 · 2015: 17 · 2014: 7 · 2013: 1
- FWCI
- 19.49
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Cross-Disciplinary Conference on Web Accessibility (W4A) (publishedVersion)
- Research Online (Edith Cowan University) (submittedVersion)
- Topics
- Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- web accessibility, WCAG 2.0, conformance testing, false positives, expert evaluation, user testing, access barriers
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Benchmarking web accessibility evaluation tools: measuring the harm of sole reliance on automated tests. https://app.cochat.ai/papers/10.1145/2461121.2461124
- The paper
- Vigo, M., Brown, J., & Conway, V. (2013). Benchmarking web accessibility evaluation tools: measuring the harm of sole reliance on automated tests. Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility. https://doi.org/10.1145/2461121.2461124
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.