Comparison of Online Accessibility Evaluation Tools: An Analysis of Tool Effectiveness
Open accessAbstract
The use of online tools is a common practice for evaluating the accessibility of a website, identifying problems, and providing useful feedback on how to fix detected issues. For ease of accessibility validation, many tools have been developed and implemented successfully over the years. Yet, the results of these tools show differences in terms of coverage, correctness, and reporting-related issues. In this study, we compared online accessibility evaluation tools to understand to what extent they differ in detecting accessibility problems in websites. A total of 41 government websites of different countries were tested for violations of accessibility guidelines using six evaluation tools. We observed that each tool generated different evaluation data for the same websites. As some of the tools are complementary to each other, meaning the highest coverage and completeness can be possible with the right combination of evaluation tools. Therefore, we suggest different tools should be utilized to provide consistency and obtain reliable data from online evaluation tools, thereby improving tool effectiveness.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
These tools were selected due to their widespread use in accessibility research and practice and for their established coverage of WCAG success criteria.
“These tools were selected due to their widespread use in accessibility research and practice and for their established coverage of WCAG success criteria [40, 53, 62].”
From A Multi-Agent Framework for Accessible Web Code Generation (2026) · cites this paper as [40]
The Study Design section says the six tools were used to assess WCAG 2.0 compliance, and the Introduction describes accessibility tools used in earlier research. But the Selection of Tools section gives different reasons for choosing them—single-page reports, non-commercial availability, and on-page results. It does not establish widespread use of the selected tools or WCAG success-criteria coverage as reasons for their selection. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 25 · Cited by 33
Research works printed but not linked: 21
- R. Ismailova, and Y. Inal, "Web site accessibility and quality in use: a comparative study of government Web sites in Kyrgyzstan, Azerbaijan, Kazakhstan and Turkey," Universal Access in the Information Society, vol. 16, no. 4, pp. 987-996, 2017.
- A.P. Freire, T.J. Bittar, and R.P. Fortes, "An approach based on metrics for monitoring web accessibility in Brazilian municipalities web sites." In Proc. the 2008 ACM symposium on Applied Computing, pp. 2421-2425, 2018.
- C. Benavídez, J.L. Fuertes, E. Gutiérrez, and L. Martínez, "Semi- automatic evaluation of web accessibility with HERA 2.0," In Proc. International Conference on Computers for Handicapped Persons, Springer, Berlin, Heidelberg, pp. 199-206, 2006.
- N.E. Youngblood, and S.A. Youngblood, "User experience and accessibility: An analysis of county web portals." Journal of Usability Studies, 9(1), 2013.
- G. Brajnik, "Comparing accessibility evaluation tools: A method for tool effectiveness," Universal Access in the Information Society, vol. 3, no. 3, pp. 252-263, 2004.
- O. Gambino, R. Pirrone, and F. D. Giorgio, "Accessibility of the Italian institutional web pages: A survey on the compliance of the Italian public administration web pages to the stanca act and its 22 technical requirements for web accessibility," Universal Access in the Information Society, vol. 15, no. 2, pp. 305-312. 2016.
- R. Ismailova, and Y. Inal, "Accessibility evaluation of top university websites: a comparative study of Kyrgyzstan, Azerbaijan, Kazakhstan and Turkey," Universal Access in the Information Society, vol. 17, no. 2, pp. 437-445, 2018.
- Y. Inal, and R. Ismailova, "Effect of human development level of countries on the web accessibility and quality in use of their municipality websites." Journal of Ambient Intelligence and Humanized Computing, vol. 11, pp. 1657-1667, 2020.
- M. Pădure, and C. Pribeanu, "Exploring the differences between five accessibility evaluation tools," in Proc. RoCHI 2019 -International Conference on Human-Computer interaction, pp. 87-90, 2019.
- N.A. Karaim, and Y. Inal, "Usability and accessibility evaluation of Libyan government websites," Universal Access in the Information Society, vol. 18, no. 1, pp. 207-216, 2019.
- N. Sabev, G.N. Georgieva-Tsaneva, and G. Bogdanova, Research, analysis, and evaluation of web accessibility for a selected group of public websites in Bulgaria. Journal of Accessibility and Design for All, 10(1), 124-160, 2020.
- B. Giovanna, M. Manca, and F. Pulina, "Flexible Automatic Support for Web Accessibility Validation," in Proc. The ACM on Human- Computer Interaction, vol. 4, EICS, Article 83, 2020.
- G. Gay, and C.Q. Li. "AChecker: open, interactive, customizable, web accessibility checking," in Proc. The 2010 International Cross Disciplinary Conference on Web Accessibility (W4A), pp. 1-2, 2010.
- S. Mirri, L.A. Muratori, M. Roccetti, and P. Salomoni, "Metrics for accessibility on the Vamolà project," In Proc. the 2009 International Cross-Disciplinary Conference on Web Accessibililty (W4A), pp. 142- 145, 2009.
- M. Battistelli, S. Mirri, L.A. Muratori, P. Salomoni, and S. Spagnoli, "Making the Tree Fall Sound: Reporting Web Accessibility with the VaMoLà Monitor," In Proc. 5th International Conference on Methodologies, Technologies and Tools enabling e-Government, 2011.
- A.G. Schiavone, and F. Paternò, "An extensible environment for guideline-based accessibility evaluation of dynamic Web applications," Universal Access in the Information Society, vol. 14, no. 1, pp. 111-132, 2015.
- Y. Inal, F. Guribye, D. Rajanen, M. Rajanen, and M. Rost, "Perspectives and practices of digital accessibility: A survey of user experience professionals in Nordic countries." In Proc. 11th Nordic Conference on Human-Computer Interaction: Shaping Experiences, Shaping Society, pp. 1-11, 2020.
- Y. Inal, and R. Ismailova, "How Do Computer Engineering Students Construe Usability and Accessibility? A Comparative Study between Turkey and Kyrgyzstan". Tehnički vjesnik, vol. 25, no. 5, pp. 1339- 1347, 2018. https://doi.org/10.17559/TV-20170205083820 · DOI
- J. Lazar, A. Dudley-Sponaugle, and K.D. Greenidge, "Improving web accessibility: a study of webmaster perceptions." Computers in Human Behavior, vol. 20, no. 2, pp. 269-288, 2004.
- G. Farrelly, "Practitioner barriers to diffusion and implementation of web accessibility." Technology and Disability, vol. 23, no. 4, pp. 223- 232, 2011.
- A. Leitner, I. Ciupa, B. Meyer, and M. Howard, "Reconciling manual and automated testing: The autotest experience," In Proc. 2007 40th Annual Hawaii International Conference on System Sciences, pp. 261a-261a, 2007.
- Authors
- Ismailova, R., & Inal, Y.
- Affiliations
- Computer Engineering Department, Kyrgyz-Turkish Manas University, Bishkek, Kyrgyzstan
- Department of Design, Norwegian University of Science and Technology, Gjøvik, Norway
- Venue
- IEEE Access
- Published
- 2022
- Volume
- 10
- Pages
- 58233-58239
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W4285209525
- Cited by
- 33
- References
- 25
- Citations by year
- 2026: 7 · 2025: 11 · 2024: 4 · 2023: 7 · 2022: 4
- FWCI
- 16.55
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- IEEE Access (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- NORA - Norwegian Open Research Archives (submittedVersion, cc-by)
- Topics
- Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Information Systems › Mobile and Web Applications
- Keywords
- web accessibility, accessibility evaluation tools, tool comparison, accessibility guidelines, government websites, tool effectiveness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
