Automatic web accessibility metrics: Where we are and where we can go
Abstract
The fact that several web accessibility metrics exist may be evidence of a lack of a comparison framework that highlights how well they work and for what purposes they are appropriate. In this paper we aim at formulating such a framework, demonstrating that it is feasible, and showing the findings we obtained when we applied it to seven existing automatic accessibility metrics. The framework encompasses validity, reliability, sensitivity, adequacy and complexity of metrics in the context of four scenarios where the metrics can be used. The experimental demonstration of the viability of the framework is based on applying seven published metrics to more than 1500 web pages and then operationalizing the notions of validity-as-conformance, adequacy and complexity. Our findings lead us to conclude that the Web Accessibility Quantitative Metric, Page Measure and Web Accessibility Barrier are the metrics that achieve the highest levels of quality (out of the seven that we examined). Finally, since we did not analyse reliability, sensitivity and validity-in-use, this paper provides guidance to address them in what are new research avenues.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The error profile of the landing page is generally similar to other pages of a given website.
“Vigo et al., [27], who found that the error profile of the landing page is generally similar to other pages of a given website.”
From Digital Accessibility in Higher Education: A Dutch case study (2024) · cites this paper as [27]
Based on the work of Vigo et al. [27], the selected method can be considered representative.
“However, based on the work of Vigo et al., [27], our selected method can be considered representative.”
From Digital Accessibility in Higher Education: A Dutch case study (2024) · cites this paper as [27]
Cites
Accessibility of Internet websites through time
Hackett, S., Parmanto, B., & Zeng, X. (2003) · Proceedings of the 6th international ACM SIGACCESS conference on Computers and accessibility · doi:10.1145/1028630.1028638 · Cited by 145 · see citing works
Beyond Conformance: The Role of Accessibility Evaluation Methods
Brajnik, G. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-85200-1_9 · Cited by 114 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Vigo, M., Brown, J., & Conway, V. (2013) · Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility · doi:10.1145/2461121.2461124 · Cited by 215 · see citing works
Self-citationThe Current Status of Accessibility in Mobile Apps
Yan, S., & Ramachandran, P. G. (2019) · ACM Transactions on Accessible Computing · doi:10.1145/3300176 · Cited by 165 · see citing works
Examining Image-Based Button Labeling for Accessibility in Android Apps through Large-Scale Analysis
Ross, A. S., Zhang, X., Fogarty, J., & Wobbrock, J. O. (2018) · Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3234695.3236364 · Cited by 90 · see citing works
Tools for web accessibility evaluation
Abascal, J., Arrue, M., & Valencia, X. (2019) · Human–Computer Interaction Series · doi:10.1007/978-1-4471-7440-0_26 · Cited by 88 · see citing works
Kim, H. K., Han, S. H., Park, J., & Park, J. (2016) · International Journal of Industrial Ergonomics · doi:10.1016/j.ergon.2016.07.002 · Cited by 80 · see citing works
Coping tactics employed by visually disabled users on the web
Vigo, M., & Harper, S. (2013) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2013.08.002 · Cited by 75 · see citing works
Self-citationThe Effects of "Not Knowing What You Don't Know" on Web Accessibility for Blind Web Users
Bigham, J. P., Lin, I., & Savage, S. (2017) · Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3132525.3132533 · Cited by 73 · see citing works
Gambino, O., Pirrone, R., & Giorgio, F. D. (2016) · Universal Access in the Information Society · doi:10.1007/s10209-014-0381-0 · Cited by 47 · see citing works
Assessment of Web Content Accessibility Levels in Spanish Official Online Education Environments
Roig-Vila, R., Ferrández, S., & Ferri-Miralles, I. (2014) · International Education Studies · doi:10.5539/ies.v7n6p31 · Cited by 19 · see citing works
Digital Accessibility in Higher Education: A Dutch case study
Smit, K., van de Hoef, A., Kuiper, N., Todorova, J., Hartman, F., Leewis, S., & Gevaert, R. (2024) · Proceedings of the 2024 8th International Conference on Software and e-Business · doi:10.1145/3715885.3715890 · Cited by 1 · see citing works
“In essence, conformance testing only landing pages for Digital Ac-cessibility is supported by Vigo et al., [27], who found that the error profile of the landing page is generally similar to other pages of a given website.”
- Authors
- Vigo, M., & Brajnik, G.
- Affiliations
- Sun Yat-sen University (CN)
- University of Udine (IT)
- Corresponding author
- Vigo, Markel
- Venue
- Interacting with Computers
- Publisher
- Oxford University Press (OUP)
- Published
- 2011-03
- Volume
- 23
- Issue
- 2
- Pages
- 137-155
- ISSN
- 0953-5438 (ISSN-L)
- OpenAlex
- W2143845053
- Cited by
- 112
- References
- 2
- Citations by year
- 2026: 9 · 2025: 4 · 2024: 4 · 2023: 7 · 2022: 10 · 2021: 4 · 2020: 5 · 2019: 8 · 2018: 5 · 2017: 8 · 2016: 10 · 2015: 8 · 2014: 7 · 2013: 16 · 2012: 6
- FWCI
- 32.73
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Interacting with Computers (publishedVersion)
- Institutional Research Information System (University of Udine) (submittedVersion)
- Topics
- Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Information Systems › Web Data Mining and Analysis
- Keywords
- web accessibility, validity, reliability statistics, sensitivity, conformance, website evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
