Benchmarking web accessibility evaluation tools: measuring the harm of sole reliance on automated tests
The DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
The use of web accessibility evaluation tools is a widespread practice. Evaluation tools are heavily employed as they help in reducing the burden of identifying accessibility barriers. However, an over-reliance on automated tests often leads to setting aside further testing that entails expert evaluation and user tests. In this paper we empirically show the capabilities of current automated evaluation tools. To do so, we investigate the effectiveness of 6 state-of-the-art tools by analysing their coverage, completeness and correctness with regard to WCAG 2.0 conformance. We corroborate that relying on automated tests alone has negative effects and can have undesirable consequences. Coverage is very narrow as, at most, 50% of the success criteria are covered. Similarly, completeness ranges between 14% and 38%; however, some of the tools that exhibit higher completeness scores produce lower correctness scores (66-71%) due to the fact that catching as many violations as possible can lead to an increase in false positives. Therefore, relying on just automated tests entails that 1 of 2 success criteria will not even be analysed and among those analysed, only 4 out of 10 will be caught at the further risk of generating false positives.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The study of Vigo et al. (2013) shows the problems of relying only on automated web evaluation tools.”
From Monitoring Systems for Checking Websites on Accessibility (2021) · cites this paper as Vigo et al. (2013)
In the citing paper
“Currently, there are few studies on the comparison of tools for automatic web accessibility evaluation.”
From Monitoring Systems for Checking Websites on Accessibility (2021) · cites this paper as Vigo et al. (2013)
In the citing paper
“Therefore, such tools will not replace manual checks in the foreseeable future (Vigo et al., 2013), but can only be used as a support tool.”
From Monitoring Systems for Checking Websites on Accessibility (2021) · cites this paper as Vigo et al., 2013
In the citing paper
“Nevertheless, these tools cannot replace manual accessibility checks (Vigo et al., 2013).”
From Monitoring Systems for Checking Websites on Accessibility (2021) · cites this paper as Vigo et al., 2013
In the citing paper
“While prior studies (Vigo et al., 2013;Abduganiev and Gaibullojonovich, 2017;Pȃdure and Pribeanu, 2019) have mostly evaluated free tools for checking accessibility, this work compares some commercial AMS that can monitor entire websites and evaluate their accessibility over time.”
From Monitoring Systems for Checking Websites on Accessibility (2021) · cites this paper as Vigo et al., 2013
6 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Automatic detection of text genre
Kessler, B., Numberg, G., & Schütze, H. (1997) · Proceedings of the 35th annual meeting on Association for Computational Linguistics - · doi:10.3115/976909.979622 · Cited by 358 · cited 1×
“) is feasible [13], which is useful to raise warnings in those sites that require legal commitments (see SC “3.”
BackgroundGuidelines are only half of the story: accessibility problems encountered by blind users on the web
Power, C., Freire, A., Petrie, H., & Swallow, D. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2207736 · Cited by 327 · cited 1×
“on guidelines alone leaves out a number of issues as it was found that guidelines do not cover the whole spectrum of accessibility problems, as they only cover around 50% of accessibility problems encountered by users [19].”
BackgroundImproving web accessibility: a study of webmaster perceptions
Lazar, J., Dudley-Sponaugle, A., & Greenidge, K.-D. (2004) · Computers in Human Behavior · doi:10.1016/j.chb.2003.10.018 · Cited by 326 · cited 1×
“However, a study of webmasters’ perceptions revealed that the majority of them are familiar with accessibility guidelines [15].”
BackgroundMankoff, J., Fait, H., & Tran, T. (2005) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1054972.1054979 · Cited by 207 · cited 1×
“In one of the earliest studies to explore multiple methods of evaluation and to discuss levels of expertise of evaluators, multiple approaches to website evaluation were analysed including automated tools, expert evaluations and user testing [17].”
MethodBeyond Conformance: The Role of Accessibility Evaluation Methods
Brajnik, G. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-85200-1_9 · Cited by 114 · cited 2×
“Optimally, the formative assessment process should be ongoing from the initiation of a project, and continue throughout the operational lifespan of the web page or site [6, 21].”
BackgroundAccessibility designer: visualizing usability for the blind
Takagi, H., Asakawa, C., Fukuda, K., & Maeda, J. (2003) · Proceedings of the 6th international ACM SIGACCESS conference on Computers and accessibility · doi:10.1145/1028630.1028662 · Cited by 113 · cited 1×
“The techniques used to assess the accessibility of each web page are diverse across judges: evaluation tools that diverge from the ones benchmarked (WAVE(2)), mark-up validators, browser extensions for developers (Firebug, Web Developer Toolbar, Web Accessibility Toolbar), screen readers (VoiceOver, NVDA) and evaluati…”
MethodAutomatic web accessibility metrics: Where we are and where we can go
Vigo, M., & Brajnik, G. (2011) · Interacting with Computers · doi:10.1016/j.intcom.2011.01.001 · Cited by 112 · cited 1×
“To name a few, accessibility evaluation is also employed in quality assurance of web applications, accessibility observatories, information retrieval and user-adapted interaction scenarios [25].”
BackgroundSelf-citationA comparative test of web accessibility evaluation methods
Brajnik, G. (2008) · Proceedings of the 10th international ACM SIGACCESS conference on Computers and accessibility · doi:10.1145/1414471.1414494 · Cited by 111 · cited 2×
“is not likely to distinguish reliably between important and non-important accessibility problems” [7].”
BackgroundLanguage identification in web pages
Martins, B., & Silva, M. J. (2005) · ACM Symposium on Applied Computing · doi:10.1145/1066677.1066852 · Cited by 103 · cited 1×
“Natural Language Processing) such as detecting changes on the language [18] for SC such as “3.”
BackgroundComparing accessibility evaluation tools: a method for tool effectiveness
Brajnik, G. (2004) · Universal Access in the Information Society · doi:10.1007/s10209-004-0105-y · Cited by 101 · cited 3×
“The most relevant related work upon which this paper is based on analysed the effectiveness of automated tools [5], where effectiveness encompasses completeness, correctness and specificity qualities.”
BackgroundMethodResultInfluentialContextual web accessibility - maximizing the benefit of accessibility guidelines
Sloan, D., Heath, A., Hamilton, F., Kelly, B., Petrie, H., & Phipps, L. (2006) · Proceedings of the 2006 international cross-disciplinary workshop on Web accessibility (W4A) Building the mobile web: rediscovering accessibility? - W4A · doi:10.1145/1133219.1133242 · Cited by 97 · cited 1×
“It also is acknowledged that context has to be taken into consideration to ensure that guidelines meet accessibility expectations [22].”
BackgroundThe Expertise Effect on Web Accessibility Evaluation Methods
Brajnik, G., Yesilada, Y., & Harper, S. (2011) · Human–Computer Interaction · doi:10.1080/07370024.2011.601670 · Cited by 83 · cited 2×
“3 experts guarantee that almost all accessibility violations can be found with a high reliability [9].”
BackgroundWeb not for all: a large scale study of web accessibility
Lopes, R., Gomes, D., & Carriço, L. (2010) · Proceedings of the 2010 International Cross Disciplinary Conference on Web Accessibility (W4A) · doi:10.1145/1805986.1806001 · Cited by 67 · cited 1×
“A number of studies indicate that the World Wide Web is not accessible [16].”
BackgroundGilbertson, T. D., & Machin, C. H. C. (2012) · Proceedings of the International Cross-Disciplinary Conference on Web Accessibility · doi:10.1145/2207016.2207024 · Cited by 34
Elementary Statistical Concepts
Hare, L. B., & Walpole, R. E. (1977) · Technometrics · doi:10.2307/1268636 · Cited by 29
Managing usability for people with disabilities in a large Web presence
King, M., Thatcher, J., Bronstad, P. M., & Easton, R. (2005) · IBM Systems Journal · doi:10.1147/sj.443.0519 · Cited by 29 · cited 1×
“The importance of sampling in large web presence is emphasised by a number of different sampling techniques for accessibility testing [8, 14].”
BackgroundEffects of sampling methods on web accessibility evaluations
Brajnik, G., Mulas, A., & Pitton, C. (2007) · Proceedings of the 9th international ACM SIGACCESS conference on Computers and accessibility · doi:10.1145/1296843.1296855 · Cited by 27 · cited 1×
“The importance of sampling in large web presence is emphasised by a number of different sampling techniques for accessibility testing [8, 14].”
BackgroundAutomatic Ergonomic Evaluation: What are the Limits?
Farenc, C., Liberati, V., & Barthet, M.-F. (1996) · CADUI · Cited by 22
Exploratory Analysis of Collaborative Web Accessibility Improvement
Sato, D., Takagi, H., Kobayashi, M., Kawanaka, S., & Asakawa, C. (2010) · ACM Transactions on Accessible Computing · doi:10.1145/1857920.1857922 · Cited by 20 · cited 1×
“Other approaches to error identification and correction come in the form of crowdsourcing [20].”
BackgroundApplication of traditional software testing methodologies to web accessibility
Shelly, C. C., & Barta, M. (2010) · International Cross-Disciplinary Conference on Web Accessibility (W4A) · doi:10.1145/1805986.1806002 · Cited by 16 · cited 1×
“Optimally, the formative assessment process should be ongoing from the initiation of a project, and continue throughout the operational lifespan of the web page or site [6, 21].”
BackgroundTransition of accessibility evaluation tools to new standards
Aizpurua, A., Arrue, M., Vigo, M., & Abascal, J. (2009) · International Web for All Conference (W4A) · doi:10.1145/1535654.1535662 · Cited by 15 · cited 1×
“This figure is consistent with previous research on the automation limits of usability evaluation [3, 10, 11], which establish that around 44-55% of tests can be automated.”
ResultSelf-citationWCAG 2.0 Test Samples Repository
Abou-Zahra, S., & Cooper, M. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-02713-0_65 · Cited by 4 · cited 1×
“contain accessibility issues violating specific SC (see examples of such test suites elsewhere [2, 23]).”
BackgroundElementary statistical concepts
Walpole, R. E. (1983) · Cited by 3
The BenToWeb Test Case Suites for the Web Content Accessibility Guidelines (WCAG) 2.0
Strobbe, C., Koch, J., Vlachogiannis, E., Ruemer, R., Velasco, C. A., & Engelen, J. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-70540-6_58 · Cited by 2 · cited 1×
“contain accessibility issues violating specific SC (see examples of such test suites elsewhere [2, 23]).”
BackgroundIntegrating Universal Design into a Global Approach for Managing Very Large Web Sites
Cooper, M., Limbourg, Q., Mariage, C., & Vanderdonckt, J. M. (1999) · HAL · Cited by 2
Research works printed but not linked: 2
- S. Abou-Zahra . Evaluation and report language (earl) 1.0 schema -- w3c working draft , 2011 . S. Abou-Zahra. Evaluation and report language (earl) 1.0 schema -- w3c working draft, 2011.
- W3C- WAI. Conformance evaluation of web sites for accessibility . 2005 . W3C-WAI. Conformance evaluation of web sites for accessibility. 2005.
Cited by
The Current Status of Accessibility in Mobile Apps
Yan, S., & Ramachandran, P. G. (2019) · ACM Transactions on Accessible Computing · doi:10.1145/3300176 · Cited by 165
“A combination of test suites (unit test apps), test apps, and commercial apps are used to verify the correctness of the results from MAC, because test suites alone may miss some types of errors and also can produce false positives [5, 27].”
BackgroundMethodInfluentialAlsaeedi, A. (2020) · Information · doi:10.3390/info11010040 · Cited by 101
“The performance and suitability of accessibility evaluation tools can be assessed and compared in two ways [5]: by selecting a representative sample of websites or using test suites.”
BackgroundMethodInfluentialEvaluating the accessibility of public health websites: An exploratory cross-country study
Alajarmeh, N. (2022) · Universal Access in the Information Society · doi:10.1007/s10209-020-00788-7 · Cited by 89
“[31] as they highlighted some limitations and variances of automatic tools in terms of accuracy and comprehensiveness in detecting real accessibility issues when compared to manual evaluations.”
BackgroundAcosta-Vargas, P., Acosta, T., & Lujan-Mora, S. (2018) · IEEE Access · doi:10.1109/access.2018.2848978 · Cited by 92
Accessibility and Software Engineering Processes: A Systematic Literature Review
Paiva, D. M. B., Freire, A. P., & de Mattos Fortes, R. P. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110819 · Cited by 87
Metadata
- Authors
- Vigo, M., Brown, J., & Conway, V.
- Affiliations
- University of Manchester (GB)
- Edith Cowan University (AU)
- Venue
- Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility
- Publisher
- ACM
- Published
- 2013-05-13
- Pages
- 1-10
- OpenAlex
- W2059464277
- Cited by
- 215
- References
- 25
- Citations by year
- 2026: 20 · 2025: 24 · 2024: 19 · 2023: 16 · 2022: 22 · 2021: 14 · 2020: 19 · 2019: 19 · 2018: 15 · 2017: 11 · 2016: 11 · 2015: 17 · 2014: 7 · 2013: 1
- FWCI
- 19.49
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Cross-Disciplinary Conference on Web Accessibility (W4A) (publishedVersion)
- Research Online (Edith Cowan University) (submittedVersion)
- Topics
- Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- web accessibility, WCAG 2.0, conformance testing, false positives, expert evaluation, user testing, access barriers
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.