CoChat
Sign Up
Cited by 37Digital Accessibility for DisabilitiesUsability and User Interface DesignSoftware Testing and Debugging Techniques

The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.

Towards Automated Web Accessibility Evaluation: A Comparative Study

Open access
Lecturer, Khujand Polytechnic Institute of Tajik Technical University named after academician M.S. Osimi, 735700 Khujand, Tajikistan, Abduganiev, Siddikjon Gaibullojonovich·2017·International Journal of Information Technology and Computer Science
DOI 10.5815/ijitcs.2017.09.03
Open this paper in CoChat — free accountOpen PDF

Abstract

With each passing day, the Web is becoming increasingly important in our lives.Hence, the need of making it more accessible to everyone, especially for the disabled and elderly spurred a great interest in automated tools, the total registered number of which has been continuously increasing and reached from forty-five software bids in 2014 to ninety-three in 2017.The purpose of this empirical research is to assess and compare eight popular and free online automated Web accessibility evaluation tools (AWAETs) such as AChecker, Cynthia Says, EIII Checker, MAUVE, SortSite, TAW, Tenon and WAVE with regard to the WCAG 2.0 conformance.As a result, significant differences were observed in terms of tool's coverage (a maximum of 32.4%), completeness (ranges between 10% and 59%), correctness (an average of 70.7%), specificity (reaches 32%), inter-reliability (lies between 1.56% and 18.32%) and intra-reliability (the acceptable score), validity, efficiency and capacity.These eight criteria can help to determine a new role played by modern AWAETs as dependent methods in Web accessibility evaluation.Moreover, consequences of relying on AWAETs alone are quantified and concluded that applying such approaches is a great mistake since subjective and less frequent objective success criteria (SC) failed to be automated.However, using a good combination of AWAETs is highly recommended as overall results in all the mentioned quality criteria are maximized and tools could definitely validate and complete each other.Ultimately, integrating automated methods with the others is ideal and preferably at an early stage of the website development life cycle.The study also provides potential accessibility barriers that make websites inaccessible, challenges AWAETs are currently facing, nineteen pros and fourteen cons and fifteen improvement recommendations for the existing and next generation of AWAETs.Fundamentally, achieving the objectives of this study was possible due to the elaboration and implementation of a new five-phased methodology named as -5PhM-for-AWAEMs‖ for successful selection, evaluation and/or comparison of AWAEMs.In addition to providing detailed descriptions of the estimation process, this methodology represents eleven key criteria for effective selection of suitable AWAEMs and necessary numbers of web pages and expert evaluators for acceptable, normal or ideal assessment.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.5815/ijitcs.2017.09.03

Cites

38 of 38 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 38 references

Research works printed but not linked: 22

  1. [2]2. Fogg B. J., Swani P., Treinen M., Marshall J., Osipovich A., Varma C., Laraki O., Fang N., Paul J., Rangnekar A. Copyright © 2017 MECS I.J. Information Technology and Computer Science, 2017, 9, 18-44
  2. [3]3. and Shon J., Elements that affect Web credibility: Early Results From a Self-Report Study, Proceedings of CHI'00, Extended Abstracts on Human Factors in Computing Systems, 2000, pp. 287-288.
  3. [4]4. Killam B. and Holland B., Position Paper on The Suitability of Task Automated Utilities for Testing Web accessibility Compliance, Usability professionals' association conference, 2001. Available online: http://www.upassoc.org/conf2001/
  4. [7]7. Good A., An Investigation of a Method for Improving Accessibility to Web-Based information for Users with Impairments, Doctoral Thesis, University of Portsmouth, Portsmouth, 2008.
  5. [22]22. Harper S. and Yesilada Y., Web Accessibility, Springer, London, United Kingdom, 2008.
  6. [33]33. Kaur A. and Diksha D., Comparing and Evaluating the Effectiveness of Mobile Web Adequacy Evaluation Tools. Article in Universal Access in the Information Society, May 2016. DOI: 10.1007/s10209-016-0466-z.2. · DOI
  7. [34]34. Pivetta E. M., Saito D. S., Flor C. S., Ulbricht V. R., Vanzin T., Automated Accessibility Evaluation Software for Authenticated Environments -A Heuristic Usability Evaluation. HCI (7) 2014, pp. 77-88
  8. [36]36. Sukhpal K., An Automated Tool for Web Site Evaluation, International Journal of Computer Science and Information Technologies, 2012, 3 (3), pp. 4310 -4313.
  9. [40]40. Ahmad Al-A., Ibraheem Y.Y. and Ahmaro M.M., Comparison Between Web accessibility Evaluation Tools. Almadinah Islamic Studies, 2010, 1(66): Malaysian Studies. Copyright © 2017 MECS I.J. Information Technology and Computer Science, 2017, 9, 18-44
  10. [43]43. Chisholm W. Vanderheiden G. and Jacobs I. (Eds.), Web Content Accessibility Guidelines 1.0, May 5 1999.
  11. [48]48. Sevilla J., Herrera G., Martí nez and Alcantud F. Web accessibility for Individuals with Cognitive Deficits: A Comparative Study Between an Existing Commercial Web and its Cognitively Accessible Equivalent. ACM Transactions on Computer Human Interaction, 2007, 14(3), p. 12.
  12. [49]49. Power C., Freire A., Petrie H., and Swallow D., Guidelines are Only Half of the Story: Accessibility Problems Encountered by Blind Users on the Web. In Proceedings of the 2012 ACM annual conference on Human Factors in Computing Systems, CHI '12, 2012, pp. 433-442.
  13. [53]53. Mankoff J., Fait H. and Tran T., Is Your Web Page Accessible? A Comparative Study of Methods for Assessing Web Page Accessibility for the Blind. In Proc. SIGCHI Conference on Human Factors in Computing Systems, CHI'05, 2005, pp. 41-50.
  14. [57]57. Kelly B., Sloan D., Brown S., Seale J., Petrie H., Lauke P. and Ball S., Accessibility 2.0: People, Policies and Processes. In: W4A 2007: Proc. of the 2007 international cross-disciplinary conference on Web accessibility (W4A), 2007, pp. 138-147. ACM, New York.
  15. [59]59. Christopher B. and Elaine P., Development and Trial of an Educational Tool to Support the Accessibility Evaluation. Proc. In: Proceedings of the 2011 International Cross-Disciplinary Conference on Web accessibility W4A, 2011, pp. 2. Available online: http://dx.doi.org/10.1145/
  16. [61]61. Lopes R. and Carric L., Macroscopic Characterizations of Web Accessibility. New Review of Hypermedia and Multimedia, 2010, 16(3), pp. 221-243.
  17. [71]71. Gay G. and Li C.Q., AChecker: Open, Interactive, Customizable, Web accessibility Checking. In: International Cross Disciplinary Conference on Web Accessibility-W4A, 2010, pp. 1-2.
  18. [76]76. Declaring Conformance on Web Accessibility, ANEC Print version 21, May 2011, pp. 1-44. Project Reference: ANEC-R&T-2009-ICT-001final
  19. [84]84. Sears A., Heuristic walkthroughs: finding the problems without the noise. Int. Journal of Human-Computer Interaction, 1997, 9(3), pp. 213-234.
  20. [85]85. Gray W. and Salzman M. Damaged merchandise: a review of experiments that compare usability evaluation methods. Human-Computer Interaction, 1998, 13(3), pp. 203-261.
  21. [92]92. Centeno V. L., Kloos C. D., Fernandez L. S. and Fernandez N. G., Device independence for Web Wrapper Agents Workshop on Device Independent Web Engineering (DIWE'04), 26 July 2004, Munich.
  22. [98]98. Abduganiev S.G., Pils M. and Roithmayr F., Elicitation of Criteria Weights for the Web Quality Evaluation Method Universal Star: By Using Different Ranking Methods. The Strategies of Modern Science Development: Proceedings of the X International scientific-practical conference. North Charleston, USA, 12-13 April 2016. - North Charleston: CreateSpace, 2016, pp. 11-24.

Cited by

  • Monitoring Systems for Checking Websites on Accessibility

    Burkard, A., Zimmermann, G., & Schwarzer, B. (2021) · Frontiers in Computer Science · doi:10.3389/fcomp.2021.628770 · Cited by 24

    “The evaluation criterion correctness (Abduganiev and Gaibullojonovich 2017; Vigo et al., 2013) expresses how many of the TPs found by an AMS were actually TPs and not FPs. Correctness could only be ensured by manual checking.”
    BackgroundMethodResultInfluential

See all 37

Related

See all

Metadata

Authors
Lecturer, K. P. I. O. T. T. U. N. A. A. M. S. O., & Abduganiev, S. G.
Affiliations
Johannes Kepler University of Linz (AT)
Corresponding author
Abduganiev, Siddikjon Gaibullojonovich
Venue
International Journal of Information Technology and Computer Science
Publisher
MECS Publisher
Published
2017-09-08
Volume
9
Issue
9
Pages
18-44
ISSN
  • 2074-9007 (ISSN-L)
  • 2074-9015
DOI
10.5815/ijitcs.2017.09.03
OpenAlex
W2746905041
Semantic Scholar
52c9bfab6841108d1edd3d86c54a8ab0bd28f036
Cited by
37
References
38
Citations by year
2026: 7 · 2025: 7 · 2024: 3 · 2023: 3 · 2022: 5 · 2021: 5 · 2020: 4 · 2019: 3
FWCI
8.92
Type
journal article
Language
en
Open access
Locations
International Journal of Information Technology and Computer Science (publishedVersion)
Topics
  • Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
Keywords
web accessibility, tool comparison, AChecker, mauve, tenon, waves, tool selection, access barriers, disabled users

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Towards Automated Web Accessibility Evaluation: A Comparative Study. https://app.cochat.ai/papers/10.5815/ijitcs.2017.09.03
The paper
Lecturer, K. P. I. O. T. T. U. N. A. A. M. S. O., & Abduganiev, S. G. (2017). Towards Automated Web Accessibility Evaluation: A Comparative Study. International Journal of Information Technology and Computer Science. https://doi.org/10.5815/ijitcs.2017.09.03