Is accessibility conformance an elusive property? A study of validity and reliability of WCAG 2.0
The DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.
Abstract
The Web Content Accessibility Guidelines (WCAG) 2.0 separate testing into both “Machine” and “Human” audits; and further classify “Human Testability” into “Reliably Human Testable” and “Not Reliably Testable”; it is human testability that is the focus of this paper. We wanted to investigate the likelihood that “at least 80% of knowledgeable human evaluators would agree on the conclusion” of an accessibility audit, and therefore understand the percentage of success criteria that could be described as reliably human testable, and those that could not. In this case, we recruited twenty-five experienced evaluators to audit four pages for WCAG 2.0 conformance. These pages were chosen to differ in layout, complexity, and accessibility support, thereby creating a small but variable sample. We found that an 80% agreement between experienced evaluators almost never occurred and that the average agreement was at the 70--75% mark, while the error rate was around 29%. Further, trained—but novice—evaluators performing the same audits exhibited the same agreement to that of our more experienced ones, but a reduction on validity of 6--13% ; the validity that an untrained user would attain can only be a conjecture. Expertise appears to improve (by 19%) the ability to avoid false positives. Finally, pooling the results of two independent experienced evaluators would be the best option, capturing at most 76% of the true problems and producing only 24% of false positives. Any other independent combination of audits would achieve worse results. This means that an 80% target for agreement, when audits are conducted without communication between evaluators, is not attainable, even with experienced evaluators, when working on pages similar to the ones used in this experiment; that the error rate even for experienced evaluators is relatively high and further, that untrained accessibility auditors be they developers or quality testers from other domains, would do much worse than this.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Beyond Conformance: The Role of Accessibility Evaluation Methods
Brajnik, G. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-85200-1_9 · Cited by 114 · cited 2×
“Web accessibility evaluation methods have been widely studied and different methods have been proposed [Brajnik 2008; Abou-Zahra 2008].”
MethodSelf-citationComparing accessibility evaluation tools: a method for tool effectiveness
Brajnik, G. (2004) · Universal Access in the Information Society · doi:10.1007/s10209-004-0105-y · Cited by 101 · cited 4×
“There are many tools available [W3C/WAI 2005c], yielding different results with different levels of quality [Abou-Zahra 2008; Thatcher et al. 2006; Brajnik 2004].”
MethodResultInfluentialSelf-citationAbou-Zahra, S. (2008) · Human-Computer Interaction Series · doi:10.1007/978-1-84800-050-6_7 · Cited by 73 · cited 6×
“Conformance Review is the most widely used inspection method, where the evaluator uses a set of accessibility guidelines that focus on possible accessibility problems and has to decide if a page or Web site complies to those requirements [Abou-Zahra 2008; Thatcher et al. 2006; Henry and Grossnickle 2004; DRC 2004].”
MethodInfluentialHow much does expertise matter?: a barrier walkthrough study with experts and non-experts
Yesilada, Y., Brajnik, G., & Harper, S. (2009) · Proceedings of the 11th international ACM SIGACCESS conference on Computers and accessibility · doi:10.1145/1639642.1639678 · Cited by 53 · cited 4×
“For instance, Barrier Walkthrough (BW) is an accessibility inspection method that is inspired by heuristic evaluation [Yesilada et al. 2009].”
MethodInfluentialSelf-citation
Cited by
Exploring perceptions of web accessibility: a survey approach
Yesilada, Y., Brajnik, G., Vigo, M., & Harper, S. (2015) · Behaviour & Information Technology · doi:10.1080/0144929x.2013.848238 · Cited by 99
“Guidelines have been challenged as it has been found that WCAG 2.0 only covers the 50% of the problems encountered by users (Power et al. 2012) and that their validity and reliability are overestimated (Brajnik et al. 2012).”
BackgroundInfluentialSelf-citationVollenwyder, B., Petralito, S., Iten, G. H., Brühlmann, F., Opwis, K., & Mekler, E. D. (2023) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2022.102956 · Cited by 62
Turning manual web accessibility success criteria into automatic: an LLM-based approach
López-Gil, J.-M., & Pereira, J. (2025) · Universal Access in the Information Society · doi:10.1007/s10209-024-01108-z · Cited by 50
“There is empirical evidence that expecting high levels of agreement for reliable WCAG human testability is not attainable when involving experienced evaluators and neither with novices [24].”
Salient beliefs influencing the intention to consider Web Accessibility
Vollenwyder, B., Iten, G. H., Brühlmann, F., Opwis, K., & Mekler, E. D. (2019) · Computers in Human Behavior · doi:10.1016/j.chb.2018.11.016 · Cited by 46
Metadata
- Authors
- Brajnik, G., Yesilada, Y., & Harper, S.
- Venue
- ACM Transactions on Accessible Computing
- Published
- 2012-03
- Volume
- 4
- Issue
- 2
- Pages
- 1-28
- ISSN
- 1936-7228 (ISSN-L)
- 1936-7236
- OpenAlex
- W2062015732
- Cited by
- 52
- References
- 4
- FWCI
- 15.74
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Is accessibility conformance an elusive property? A study of validity and reliability of WCAG 2.0. https://app.cochat.ai/papers/10.1145/2141943.2141946
- The paper
- Brajnik, G., Yesilada, Y., & Harper, S. (2012). Is accessibility conformance an elusive property? A study of validity and reliability of WCAG 2.0. ACM Transactions on Accessible Computing. https://doi.org/10.1145/2141943.2141946
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.