The DOI matches its Crossref record; 37 of the 38 references it cites were found in a registry or an index.
Comparing Web Accessibility Evaluation Tools and Evaluating the Accessibility of Webpages: Proposed Frameworks
Open accessAbstract
With the growth of e-services in the past two decades, the concept of web accessibility has been given attention to ensure that every individual can benefit from these services without any barriers. Web accessibility is considered one of the main factors that should be taken into consideration while developing webpages. Web Content Accessibility Guidelines 2.0 (WCAG 2.0) have been developed to guide web developers to ensure that web contents are accessible for all users, especially disabled users. Many automatic tools have been developed to check the compliance of websites with accessibility guidelines such as WCAG 2.0 and to help web developers and content creators with designing webpages without barriers for disabled people. Despite the popularity of accessibility evaluation tools in practice, there is no systematic way to compare the performance of web accessibility evaluators. This paper first presents two novel frameworks. The first one is proposed to compare the performance of web accessibility evaluation tools in detecting web accessibility issues based on WCAG 2.0. The second framework is utilized to evaluate webpages in meeting these guidelines. Six homepages of Saudi universities were chosen as case studies to substantiate the concept of the proposed frameworks. Furthermore, two popular web accessibility evaluators, Wave and SiteImprove, are selected to compare their performance. The outcomes of studies conducted using the first proposed framework showed that SiteImprove outperformed WAVE. According to the outcomes of the studies conducted, we can conclude that web administrators would benefit from the first framework in selecting an appropriate tool based on its performance to evaluate their websites based on accessibility criteria and guidelines. Moreover, the findings of the studies conducted using the second proposed framework showed that the homepage of Taibah University is more accessible than the homepages of other Saudi universities. Based on the findings of this study, the second framework can be used by web administrators and developers to measure the accessibility of their websites. This paper also discusses the most common accessibility issues reported by WAVE and SiteImprove.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The major errors include low-contrast text and images; links without visible text; document file issues, such as inaccessible uploaded document files; missing alternative texts for images, as well as buttons, and a lack of navigation information (see [11, 12, 27, 35–39]).”
From Does the law matter? An empirical study on the accessibility of Finnish higher education institutions’ web pages (2024) · cites this paper as [35]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These tools were selected because they are widely used in accessibility research [15, 17, 19, 21] and are integrated into A11yAgent's internal evaluation pipeline, ensuring consistent measurement across all conditions.”
From A Multi-Agent Framework for Accessible Web Code Generation (2026) · cites this paper as [15]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The relationship between accessibility and usability of websites
Petrie, H., & Kheir, O. (2007) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.1240688 · Cited by 297 · cited 1×
“An example of a study in which disabled users were asked to evaluate websites’ accessibility was conducted by Petrie and Kheir [27].”
BackgroundVigo, M., Brown, J., & Conway, V. (2013) · Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility · doi:10.1145/2461121.2461124 · Cited by 215 · cited 3×
“The performance and suitability of accessibility evaluation tools can be assessed and compared in two ways [5]: by selecting a representative sample of websites or using test suites.”
BackgroundMethodInfluentiale-Government Website Accessibility: In-Depth Evaluation of Saudi Arabia and Oman
Abanumy, A. N., Al-Badi, A. H., & Mayhew, P. J. (2005) · UEA Digital Repository (University of East Anglia) · Cited by 199
The Evaluation of Accessibility, Usability, and User Experience
Petrie, H., & Bevan, N. (2009) · Human Factors and Ergonomics · doi:10.1201/9781420064995-c20 · Cited by 185 · cited 1×
“It is considered an ultimate approach for evaluating the accessibility of webpages [28].”
BackgroundBarriers to use: usability and content accessibility on the Web's most popular sites
Sullivan, T., & Matson, R. (2000) · Proceedings on the 2000 conference on Universal Usability - CUU '00 · doi:10.1145/355460.355549 · Cited by 183
The Evaluation of Accessibility, Usability, and User Experience
(2009) · The Universal Access Handbook · doi:10.1201/9781420064995-31 · Cited by 171
Remote usability evaluations with disabled people.
Petrie, H., Hamilton, F., King, N., & Pavan, P. (2006) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1124772.1124942 · Cited by 157 · cited 1×
“Despite the effectiveness of this approach, recruiting disabled people is a very difficult task [29].”
BackgroundThe Expertise Effect on Web Accessibility Evaluation Methods
Brajnik, G., Yeşilada, Y., & Harper, S. (2011) · Human-Computer Interaction · doi:10.1080/07370024.2011.601670 · Cited by 122 · cited 1×
“Brajnik et al. [22] demonstrated the ability of expert users to detect accessibility violations with more reliability.”
BackgroundMetric for Web accessibility evaluation
Parmanto, B., & Zeng, X. (2005) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20233 · Cited by 115 · cited 1×
“The total number of accessibility errors is denoted as B p and the numbers of potential accessibility errors is denoted as Parmanto [38] developed a metric based on WCAG checkpoints to automatically test the accessibility using automated accessibility evaluation tools.”
MethodWeb Accessibility Evaluation of Government Websites for People with Disabilities in Turkey
Akgül, Y., & Vatansever, K. (2015) · Journal of Advanced Management Science · doi:10.12720/joams.4.3.201-210 · Cited by 102 · cited 1×
“Yakup and Kemal [9] evaluated the web accessibility of 25 official government websites in Turkey using AChecker, eXaminator, TAW, Total Validator, WAVE, Web Accessibility Assessment Tool, Eval Access, Cynthia Says, MAGENTA, HERA, Amp and Sort Site.”
MethodComparing accessibility evaluation tools: a method for tool effectiveness
Brajnik, G. (2004) · Universal Access in the Information Society · doi:10.1007/s10209-004-0105-y · Cited by 101 · cited 7×
“Despite the novelty of Brajnik’s method [12], it is limited as it only compares a pair of tools and issues are classified manually.”
BackgroundMethodInfluentialThe accessibility of Saudi Arabia government Web sites: an exploratory study
Al-Khalifa, H. S. (2010) · Universal Access in the Information Society · doi:10.1007/s10209-010-0215-7 · Cited by 87 · cited 5×
“According to Al-Khalifa [32,33], the major violations include the following: no text alternatives, no keyboard access, and no language identification.”
BackgroundMethodInfluentialAbuaddous, H. Y., Zalisham, M., & Basir, N. (2016) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2016.071023 · Cited by 87 · cited 1×
“One of the most common failings of web accessibility evaluation tools is the difficulty in interpreting results [17].”
BackgroundEvaluating The Usability And Accessibility Of LMS “Blackboard” At King Saud University
Alturki, U., Aldraiweesh, A., & Kinshuck (2016) · Contemporary Issues in Education Research (CIER) · doi:10.19030/cier.v9i1.9548 · Cited by 85 · cited 2×
“Uthman et al. [43] evaluated the web accessibility of the LMS Blackboard at King Saud University.”
BackgroundEvaluating e-learning accessibility by automated and student-centered methods
Kumar, K., & Owston, R. (2015) · Educational Technology Research and Development · doi:10.1007/s11423-015-9413-6 · Cited by 83 · cited 1×
“An example of this approach was utilized by Kumar and Owston [30] to evaluate the accessibility of e-learning technologies using automated tools and students with a learning disability.”
MethodRevisiting the accessibility of Saudi Arabia government websites
Al-Khalifa, H. S., Baazeem, I., & Alamer, R. (2017) · Universal Access in the Information Society · doi:10.1007/s10209-016-0495-7 · Cited by 67 · cited 2×
“Al-Khalifa [34] stated that user testing is the most precise method to evaluate the accessibility of websites, but it requires time and experienced testers, causing this type of evaluation to be more challengeable.”
MethodAlahmadi, T. J., & Drew, S. (2017) · Journal of Open, Flexible and Distance Learning · doi:10.61468/jofdl.v21i1.273 · Cited by 57 · cited 2×
“[20], Alahmadi and Drew [21], and Alshamari [7], employed automatic tools to check the accessibility of websites.”
MethodAccessibility Evaluation on Malaysian E-Government Websites
Latif, M. H. A., & Masrek, M. N. (2010) · Cited by 56
Interpreting Results from Large Scale Automatic Evaluation of Web Accessibility
Bühler, C., Heck, H., Perlick, O., Nietzio, A., & Ulltveit-Moe, N. (2006) · Lecture notes in computer science · doi:10.1007/11788713_28 · Cited by 51 · cited 1×
“Buhler et al. [40] proposed a modification on the UWEM metric in order to take complexity and scalability properties into the account while evaluating web accessibility of given websites.”
BackgroundWeb accessibility highlights and trends
Brewer, J. (2004) · International Web for All Conference (W4A) · doi:10.1145/990657.990667 · Cited by 47 · cited 1×
“The evaluation of a website in terms of web accessibility is a hard task [4].”
BackgroundAccessibility Evaluation and Performance Analysis of e-Government Websites in Nigeria
Adepoju, S. A., Shehu, I. S., & Bake, P. (2016) · Journal of Advances in Information Technology · doi:10.12720/jait.7.1.49-53 · Cited by 46 · cited 1×
“Solomon and Ibrahim [10] used TAW and site analyser to report accessibility issues of Nigerian e-government websites.”
MethodAccessibility Evaluation of Arabic E-Commerce Web Sites Using Automated Tools
Alshamari, M. (2016) · Journal of Software Engineering and Applications · doi:10.4236/jsea.2016.99029 · Cited by 43 · cited 4×
“Alshamari [7] evaluated the accessibility of three well-known Arab e-commerce websites using AChecker, TAW, Eval Access, MAUVE and FAE tools.”
MethodInfluentialBanking websites in India: an accessibility evaluation
Kaur, A., & Dani, D. (2014) · CSI Transactions on ICT · doi:10.1007/s40012-014-0040-x · Cited by 42 · cited 1×
“Web accessibility checkers have been used to evaluate the web accessibility in different domains such as e-learning [6], e-Commerce [7] and banking [8].”
MethodAccessibility Evaluation of Dubai e-Government Websites: Findings and Implications
Mourad, M. B. A., & Kamoun, F. (2013) · Journal of E-Government Studies and Best Practices · doi:10.5171/2013.978647 · Cited by 41 · cited 3×
“It is vital to highlight that manual evaluation by accessibility experts or with the aid of disabled users is a time consuming and complex task [11].”
BackgroundMethodInfluentialEvaluating the accessibility and usability of top Saudi e-government services
Alfaries, A., Al-Khalifa, H. S., Al‐Razgan, M., & Al-Duwais, M. (2013) · International Conference on Theory and Practice of Electronic Governance · doi:10.1145/2591888.2591898 · Cited by 40 · cited 5×
“According to Al-Faries et al. [42], the robust principle was considered as the most common violated principle in the top e-government services in Saudi Arabia.”
BackgroundInfluentialEvaluating government website accessibility
Hong, S.-G., Katerattanakul, P., & Lee, D. (2007) · Management Research News · doi:10.1108/01409170810845930 · Cited by 24 · cited 1×
“[23] conducted a study to compare web accessibility evaluation using human experts and automated software separately.”
MethodEvaluating Web Accessibility of University Web Sites in the Kingdom of Saudi Arabia
Rana, M. M., Fakrudeen, M., & Rana, U. (2011) · The International Journal of Technology Knowledge and Society · doi:10.18848/1832-3669/cgp/v07i03/56204 · Cited by 21 · cited 5×
“This may be attributed to the web developers’ lack of knowledge of the importance of alternative texts for images [19].”
BackgroundMethodInfluentialWebsite accessibility: an Australian view
Grantham, J., Grantham, E., & Powers, D. M. W. (2012) · Australasian User Interface Conference · Cited by 20
Reliability Aware Web Accessibility Experience Metric
Song, S., Bu, J., Shen, C., Artmeier, A., Yu, Z. H., & Zhou, Q. (2018) · International Web for All Conference (W4A) · doi:10.1145/3192714.3192836 · Cited by 16 · cited 2×
“An evaluation has been performed in [41] on a collected dataset and showed that RA-WAEM performed better than the state-of-the art metrics in reflecting the user experience of disabled people while evaluating the web accessibility.”
BackgroundMethodWCAG 2.0 Semi-automatic Accessibility Evaluation System: Design and Implementation
Al-Khalifa, H. S. (2012) · Computer and Information Science · doi:10.5539/cis.v5n6p73 · Cited by 14 · cited 1×
“These tools can be categorized into two groups [16]: general and specific tools.”
BackgroundToward better web accessibility
Elkabani, I., Hamandi, L., Zantout, R., & Mansi, S. (2015) · Information and Communication Technologies and Accessibility · doi:10.1109/icta.2015.7426915 · Cited by 14 · cited 1×
“Furthermore, web accessibility evaluation using experts may include bias while finding accessibility barriers and violations [24].”
BackgroundA pilot study for evaluating Arabic websites using automated WCAG 2.0 evaluation tools
Al-Khalifa, H. S., Al-Kanhal, M., Al-Nafisah, H., Al-soukaih, N., Al-hussain, E., & Al-onzi, M. (2011) · International Conference on Innovations in Information Technology · doi:10.1109/innovations.2011.5893835 · Cited by 13 · cited 1×
“[20], Alahmadi and Drew [21], and Alshamari [7], employed automatic tools to check the accessibility of websites.”
MethodComparing Approaches to Web Accessibility Assessment
Martín, A. E., Cechich, A., & Rossi, G. H. (2008) · IGI Global eBooks · doi:10.4018/978-1-59904-847-5.ch011 · Cited by 12 · cited 1×
“It aims to help web developers check the accessibility of a given webpage to make it more accessible [13].”
BackgroundExploring the accessibility of Saudi Arabia e-government websites
Al-Khalifa, H. S. (2010) · International Conference on Theory and Practice of Electronic Governance · doi:10.1145/1930321.1930378 · Cited by 9
Miesenberger, K., Klaus, J., & Zagler, W. L. (2002) · Lecture notes in computer science · doi:10.1007/11788713 · Cited by 6
Evaluating the Blackboard System based on Web Accessibility and Usability Guidelines
Alotaibi, S. J. (2015) · International Journal for Digital Society · doi:10.20533/ijds.2040.2570.2015.0134 · Cited by 4 · cited 1×
“An interesting study was conducted by Alotaibi [36] that combined manual evaluation, automatic evaluation, and testing with the aid of disabled users to evaluate the accessibility of the Blackboard e-learning system.”
MethodAlayed, A. (2018) · ePrints Soton (University of Southampton) · Cited by 4
Research works printed but not linked: 3
- Lundqvist, S., and Ström, J. (2018, September 25). Web Accessibility in E-Learning: Identifying And Solving Accessibility Issues for Wcag 2.0 Conformance in an E-Learning Application. Student Thesis. Available online: http://urn.kb.se/resolve?urn=urn:nbn:se:liu:diva-148913.
- Khan, 2010, Effectiveness of accessibility and usability of government websites in Saudi Arabia, Can. J. Pure Appl. Sci., 4, 1127
- Cluster, W. (2008, January 9–11). Unified Web Evaluation Methodology (UWEM 10), 2006. Proceedings of the 11th International Conference (ICCHP 2008), Linz, Austria.
Cited by
Accessibility of university websites worldwide: a systematic literature review
Campoverde-Molina, M., Luján-Mora, S., & Valverde, L. (2023) · Universal Access in the Information Society · doi:10.1007/s10209-021-00825-z · Cited by 84
“[35] ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ [36] ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ [37] ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✗ ✗ ✗ [38] ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✓ ✓ ✓ ✗ [39] ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ [40] ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ [41] ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✗ [42] ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ [43] ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ [44] ✗ ✗ ✗ ✓ ✓ ✓ ✓ ✓ ✗ ✗ ✗ [45] ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ [46] ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗…”
InfluentialAccessibility engineering in web evaluation process: a systematic literature review
Ara, J., Sik‐Lányi, C., & Kelemen, A. (2023) · Universal Access in the Information Society · doi:10.1007/s10209-023-00967-2 · Cited by 96
“…in order to evaluate the well-known automatic accessibility tools in terms of webpage accessibility through their proposed measurement metrics [60]; (3) a crowdsourcing framework for web accessibility evaluation against web accessibility content guidelines checkpoints [61]; (4) an open and…”
Comparing ten WCAG tools for accessibility evaluation of websites
Kumar, S., Jeevithashree, D. V., & Biswas, P. (2021) · Technology and Disability · doi:10.3233/tad-210329 · Cited by 55
“A study by Alsaeedi [7] considered only two tools leading to incomplete evaluation of webpage.”
InfluentialComparison of Online Accessibility Evaluation Tools: An Analysis of Tool Effectiveness
Ismailova, R., & Inal, Y. (2022) · IEEE Access · doi:10.1109/access.2022.3179375 · Cited by 33
“In line with this argument, previous research pointed out that evaluating the accessibility of a website with only one tool is inadequate to decipher accessibility-related problems (e.g., [10], [11], [12], [13]).”
BackgroundResultInfluentialAccessibility in Native Mobile Applications for Users with Disabilities: A Scoping Review
Acosta-Vargas, P., Salvador-Acosta, B., Salvador-Ullauri, L., Villegas-Ch, W. E., & González, M. (2021) · Applied Sciences · doi:10.3390/app11125707 · Cited by 40
Metadata
- Authors
- Alsaeedi, A.
- Affiliations
- Taibah University (SA)
- Corresponding author
- Alsaeedi, Abdullah
- ORCID
- Alsaeedi, Abdullah
- Venue
- Information
- Publisher
- MDPI AG
- Published
- 2020-01-12
- Volume
- 11
- Issue
- 1
- Pages
- 40
- ISSN
- 2078-2489 (ISSN-L)
- OpenAlex
- W2999363798
- Cited by
- 101
- References
- 38
- Citations by year
- 2026: 16 · 2025: 24 · 2024: 11 · 2023: 22 · 2022: 15 · 2021: 11 · 2020: 2
- FWCI
- 22.48
- Type
- journal article
- Language
- en
- Open access
- gold
- dx.doi.org
- Licence
- Locations
- Information (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Information Systems › Mobile and Web Applications
- Keywords
- web accessibility, accessibility evaluation tools, WCAG 2.0, tool performance comparison, waves, disabled users, accessibility compliance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.

