The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.
Towards Automated Web Accessibility Evaluation: A Comparative Study
Open accessAbstract
With each passing day, the Web is becoming increasingly important in our lives.Hence, the need of making it more accessible to everyone, especially for the disabled and elderly spurred a great interest in automated tools, the total registered number of which has been continuously increasing and reached from forty-five software bids in 2014 to ninety-three in 2017.The purpose of this empirical research is to assess and compare eight popular and free online automated Web accessibility evaluation tools (AWAETs) such as AChecker, Cynthia Says, EIII Checker, MAUVE, SortSite, TAW, Tenon and WAVE with regard to the WCAG 2.0 conformance.As a result, significant differences were observed in terms of tool's coverage (a maximum of 32.4%), completeness (ranges between 10% and 59%), correctness (an average of 70.7%), specificity (reaches 32%), inter-reliability (lies between 1.56% and 18.32%) and intra-reliability (the acceptable score), validity, efficiency and capacity.These eight criteria can help to determine a new role played by modern AWAETs as dependent methods in Web accessibility evaluation.Moreover, consequences of relying on AWAETs alone are quantified and concluded that applying such approaches is a great mistake since subjective and less frequent objective success criteria (SC) failed to be automated.However, using a good combination of AWAETs is highly recommended as overall results in all the mentioned quality criteria are maximized and tools could definitely validate and complete each other.Ultimately, integrating automated methods with the others is ideal and preferably at an early stage of the website development life cycle.The study also provides potential accessibility barriers that make websites inaccessible, challenges AWAETs are currently facing, nineteen pros and fourteen cons and fifteen improvement recommendations for the existing and next generation of AWAETs.Fundamentally, achieving the objectives of this study was possible due to the elaboration and implementation of a new five-phased methodology named as -5PhM-for-AWAEMs‖ for successful selection, evaluation and/or comparison of AWAEMs.In addition to providing detailed descriptions of the estimation process, this methodology represents eleven key criteria for effective selection of suitable AWAEMs and necessary numbers of web pages and expert evaluators for acceptable, normal or ideal assessment.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Currently, there are few studies on the comparison of tools for automatic web accessibility evaluation.”
From Monitoring Systems for Checking Websites on Accessibility (2021) · cites this paper as Abduganiev and Gaibullojonovich (2017)
The cited source says what the claim attributes to it.
In the citing paper
“While prior studies (Vigo et al., 2013;Abduganiev and Gaibullojonovich, 2017;Pȃdure and Pribeanu, 2019) have mostly evaluated free tools for checking accessibility, this work compares some commercial AMS that can monitor entire websites and evaluate their accessibility over time.”
From Monitoring Systems for Checking Websites on Accessibility (2021) · cites this paper as Abduganiev and Gaibullojonovich, 2017
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Qualitative evaluation and research methods
Ellis, D. J. (1990) · International Journal of Information Management · doi:10.1016/0268-4012(90)90041-p · Cited by 17,943
The state of the art in automating usability evaluation of user interfaces
Ivory, M. Y., & Hearst, M. A. (2001) · ACM Computing Surveys · doi:10.1145/503112.503114 · Cited by 884
Web Content Accessibility Guidelines 2.0
Ribera, M., Porras, M., Boldú, M., Graells, M. T. I., Sulé, A., & París, P. (2009) · Data Technologies and Applications · doi:10.1108/00330330910998048 · Cited by 595
Damaged Merchandise? A Review of Experiments That Compare Usability Evaluation Methods
Gray, W. D., & Salzman, M. (1998) · Human-Computer Interaction · doi:10.1207/s15327051hci1303_2 · Cited by 486
The Evaluator Effect: A Chilling Fact About Usability Evaluation Methods
Hertzum, M., & Jacobsen, N. E. (2001) · International Journal of Human-Computer Interaction · doi:10.1207/s15327590ijhc1304_05 · Cited by 419
Criteria For Evaluating Usability Evaluation Methods
Hartson, H. R., Andre, T. S., & Williges, R. C. (2001) · International Journal of Human-Computer Interaction · doi:10.1207/s15327590ijhc1304_03 · Cited by 416
The relationship between accessibility and usability of websites
Petrie, H., & Kheir, O. (2007) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.1240688 · Cited by 304
Heuristic Walkthroughs: Finding the Problems Without the Noise
Sears, A. (1997) · International Journal of Human-Computer Interaction · doi:10.1207/s15327590ijhc0903_2 · Cited by 263
Constructing Accessible Web Sites
Thatcher, J., Bohman, P., Burks, M. R., Henry, S. L., Regan, B., Swierenga, S. J., Urban, M. D., & Waddell, C. D. (2002) · Apress eBooks · doi:10.1007/978-1-4302-1116-7 · Cited by 198
Criteria For Evaluating Usability Evaluation Methods
Hartson, H. R., Andre, T. S., & Williges, R. C. (2003) · International Journal of Human-Computer Interaction · doi:10.1207/s15327590ijhc1501_13 · Cited by 191
Research works printed but not linked: 22
- [2]2. Fogg B. J., Swani P., Treinen M., Marshall J., Osipovich A., Varma C., Laraki O., Fang N., Paul J., Rangnekar A. Copyright © 2017 MECS I.J. Information Technology and Computer Science, 2017, 9, 18-44
- [3]3. and Shon J., Elements that affect Web credibility: Early Results From a Self-Report Study, Proceedings of CHI'00, Extended Abstracts on Human Factors in Computing Systems, 2000, pp. 287-288.
- [4]4. Killam B. and Holland B., Position Paper on The Suitability of Task Automated Utilities for Testing Web accessibility Compliance, Usability professionals' association conference, 2001. Available online: http://www.upassoc.org/conf2001/
- [7]7. Good A., An Investigation of a Method for Improving Accessibility to Web-Based information for Users with Impairments, Doctoral Thesis, University of Portsmouth, Portsmouth, 2008.
- [22]22. Harper S. and Yesilada Y., Web Accessibility, Springer, London, United Kingdom, 2008.
- [33]33. Kaur A. and Diksha D., Comparing and Evaluating the Effectiveness of Mobile Web Adequacy Evaluation Tools. Article in Universal Access in the Information Society, May 2016. DOI: 10.1007/s10209-016-0466-z.2. · DOI
- [34]34. Pivetta E. M., Saito D. S., Flor C. S., Ulbricht V. R., Vanzin T., Automated Accessibility Evaluation Software for Authenticated Environments -A Heuristic Usability Evaluation. HCI (7) 2014, pp. 77-88
- [36]36. Sukhpal K., An Automated Tool for Web Site Evaluation, International Journal of Computer Science and Information Technologies, 2012, 3 (3), pp. 4310 -4313.
- [40]40. Ahmad Al-A., Ibraheem Y.Y. and Ahmaro M.M., Comparison Between Web accessibility Evaluation Tools. Almadinah Islamic Studies, 2010, 1(66): Malaysian Studies. Copyright © 2017 MECS I.J. Information Technology and Computer Science, 2017, 9, 18-44
- [43]43. Chisholm W. Vanderheiden G. and Jacobs I. (Eds.), Web Content Accessibility Guidelines 1.0, May 5 1999.
- [48]48. Sevilla J., Herrera G., Martí nez and Alcantud F. Web accessibility for Individuals with Cognitive Deficits: A Comparative Study Between an Existing Commercial Web and its Cognitively Accessible Equivalent. ACM Transactions on Computer Human Interaction, 2007, 14(3), p. 12.
- [49]49. Power C., Freire A., Petrie H., and Swallow D., Guidelines are Only Half of the Story: Accessibility Problems Encountered by Blind Users on the Web. In Proceedings of the 2012 ACM annual conference on Human Factors in Computing Systems, CHI '12, 2012, pp. 433-442.
- [53]53. Mankoff J., Fait H. and Tran T., Is Your Web Page Accessible? A Comparative Study of Methods for Assessing Web Page Accessibility for the Blind. In Proc. SIGCHI Conference on Human Factors in Computing Systems, CHI'05, 2005, pp. 41-50.
- [57]57. Kelly B., Sloan D., Brown S., Seale J., Petrie H., Lauke P. and Ball S., Accessibility 2.0: People, Policies and Processes. In: W4A 2007: Proc. of the 2007 international cross-disciplinary conference on Web accessibility (W4A), 2007, pp. 138-147. ACM, New York.
- [59]59. Christopher B. and Elaine P., Development and Trial of an Educational Tool to Support the Accessibility Evaluation. Proc. In: Proceedings of the 2011 International Cross-Disciplinary Conference on Web accessibility W4A, 2011, pp. 2. Available online: http://dx.doi.org/10.1145/
- [61]61. Lopes R. and Carric L., Macroscopic Characterizations of Web Accessibility. New Review of Hypermedia and Multimedia, 2010, 16(3), pp. 221-243.
- [71]71. Gay G. and Li C.Q., AChecker: Open, Interactive, Customizable, Web accessibility Checking. In: International Cross Disciplinary Conference on Web Accessibility-W4A, 2010, pp. 1-2.
- [76]76. Declaring Conformance on Web Accessibility, ANEC Print version 21, May 2011, pp. 1-44. Project Reference: ANEC-R&T-2009-ICT-001final
- [84]84. Sears A., Heuristic walkthroughs: finding the problems without the noise. Int. Journal of Human-Computer Interaction, 1997, 9(3), pp. 213-234.
- [85]85. Gray W. and Salzman M. Damaged merchandise: a review of experiments that compare usability evaluation methods. Human-Computer Interaction, 1998, 13(3), pp. 203-261.
- [92]92. Centeno V. L., Kloos C. D., Fernandez L. S. and Fernandez N. G., Device independence for Web Wrapper Agents Workshop on Device Independent Web Engineering (DIWE'04), 26 July 2004, Munich.
- [98]98. Abduganiev S.G., Pils M. and Roithmayr F., Elicitation of Criteria Weights for the Web Quality Evaluation Method Universal Star: By Using Different Ranking Methods. The Strategies of Modern Science Development: Proceedings of the X International scientific-practical conference. North Charleston, USA, 12-13 April 2016. - North Charleston: CreateSpace, 2016, pp. 11-24.
Cited by
Monitoring Systems for Checking Websites on Accessibility
Burkard, A., Zimmermann, G., & Schwarzer, B. (2021) · Frontiers in Computer Science · doi:10.3389/fcomp.2021.628770 · Cited by 24
“The evaluation criterion correctness (Abduganiev and Gaibullojonovich 2017; Vigo et al., 2013) expresses how many of the TPs found by an AMS were actually TPs and not FPs. Correctness could only be ensured by manual checking.”
BackgroundMethodResultInfluential
Metadata
- Authors
- Lecturer, K. P. I. O. T. T. U. N. A. A. M. S. O., & Abduganiev, S. G.
- Affiliations
- Johannes Kepler University of Linz (AT)
- Corresponding author
- Abduganiev, Siddikjon Gaibullojonovich
- Venue
- International Journal of Information Technology and Computer Science
- Publisher
- MECS Publisher
- Published
- 2017-09-08
- Volume
- 9
- Issue
- 9
- Pages
- 18-44
- ISSN
- 2074-9007 (ISSN-L)
- 2074-9015
- OpenAlex
- W2746905041
- Semantic Scholar
- 52c9bfab6841108d1edd3d86c54a8ab0bd28f036
- Cited by
- 37
- References
- 38
- Citations by year
- 2026: 7 · 2025: 7 · 2024: 3 · 2023: 3 · 2022: 5 · 2021: 5 · 2020: 4 · 2019: 3
- FWCI
- 8.92
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- International Journal of Information Technology and Computer Science (publishedVersion)
- Topics
- Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- web accessibility, tool comparison, AChecker, mauve, tenon, waves, tool selection, access barriers, disabled users
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
