The DOI matches its Crossref record; 11 of the 31 references it cites were found in a registry or an index.
Comparison of Manual Evaluation Methods for Assessing the Accessibility of Websites based on EN 301 549
Open accessAbstract
: The relevance of information technology has increased significantly over the last couple of years and therefore it is important to provide for universal access to it. Accessibility of public sector websites has become legally binding through Directive (EU) 2016/2102 for EU member states. Automatic accessibility evaluation methods can only provide a superficial impression of the accessibility status. Only manual evaluation methods can facilitate a comprehensive accessibility check. So far, there is no systematic comparison of existing manual evaluation methods available that is based on real data. In this paper, we define a generic catalog of 22 criteria for assessing the quality of accessibility evaluation methods and specify individual weights for the criteria. We then compare two representatives of manual evaluation methods: BIK BITV-Test, as a representative of conformance-based methods; and BITV-Audit, as a representative of empiric-based methods. We analyze similarities and differences between these two methods, and identify weaknesses and strengths. Our results show an advantage of BITV-Audit over BIK BITV-Test, but other weightings could yield different results.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Citations
Cites
Vigo, M., Brown, J., & Conway, V. (2013) · Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility · doi:10.1145/2461121.2461124 · Cited by 215 · cited 1×
“Automated evaluation methods can be used to assess accessibility but they cannot adequately evaluate all relevant requirements (Vigo et al., 2013).”
Mankoff, J., Fait, H., & Tran, T. (2005) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1054972.1054979 · Cited by 207 · cited 1×
“The study by Mankoff et al. (2005) compares the evaluation methods already mentioned by Lang (2004).”
MethodA comparative test of web accessibility evaluation methods
Brajnik, G. (2008) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/1414471.1414494 · Cited by 113 · cited 1×
“Brajnik (2008) further defines the following criteria for comparing evaluation methods: effectiveness, efficiency, and usefulness.”
BackgroundWeb Accessibility Testing: When the Method Is the Culprit
Brajnik, G. (2006) · Lecture Notes in Computer Science · doi:10.1007/11788713_24 · Cited by 87 · cited 1×
“In a later work, Brajnik (2006) investigates this heuristic walkthrough method, then called the barrier walkthrough method.”
MethodBeyond Web Content Accessibility Guidelines: Expert Accessibility Reviews
Calvo, R., Seyedarabi, F., & Savva, A. G. (2016) · International Conference on Software Development and Technologies for Enhancing Accessibility and Fighting Info-exclusion (DSAI) · doi:10.1145/3019943.3019955 · Cited by 65 · cited 1×
“Various studies look at the validity and reliability of WCAG 1.0 as well as 2.0 (Rømen and Svanæs, 2008; Brajnik, 2009; Calvo et al., 2016).”
BackgroundInfluentialHow much does expertise matter?: a barrier walkthrough study with experts and non-experts
Yesilada, Y., Brajnik, G., & Harper, S. (2009) · Proceedings of the 11th international ACM SIGACCESS conference on Computers and accessibility · doi:10.1145/1639642.1639678 · Cited by 53 · cited 1×
“Therefore, Yesilada et al. (2009) aim to investigate the influence of expertise on accessibility evaluation.”
BackgroundValidity and reliability of web accessibility guidelines
Brajnik, G. (2009) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/1639642.1639666 · Cited by 45 · cited 1×
“Various studies look at the validity and reliability of WCAG 1.0 as well as 2.0 (Rømen and Svanæs, 2008; Brajnik, 2009; Calvo et al., 2016).”
BackgroundInfluentialEvaluating web site accessibility
Rømen, D., & Svanæs, D. (2008) · Nordic Conference on Human-Computer Interaction · doi:10.1145/1463160.1463238 · Cited by 37 · cited 1×
“Various studies look at the validity and reliability of WCAG 1.0 as well as 2.0 (Rømen and Svanæs, 2008; Brajnik, 2009; Calvo et al., 2016).”
BackgroundInfluentialEffects of sampling methods on web accessibility evaluations
Brajnik, G., Mulas, A., & Pitton, C. (2007) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/1296843.1296855 · Cited by 34 · cited 2×
“However, Mankoff et al. indicate that each type of evaluation method has its respective weaknesses and strengths. Manual evaluation methods play an important role in the evaluation of website accessibility. Therefore, Yesilada et al. (2009) aim to investigate the influence of expertise on accessibility evaluation.”
BackgroundMonitoring Systems for Checking Websites on Accessibility
Burkard, A., Zimmermann, G., & Schwarzer, B. (2021) · Frontiers in Computer Science · doi:10.3389/fcomp.2021.628770 · Cited by 24 · cited 1×
“A study by Burkard et al. (2021) showed that with various tested automatic evaluation tools only 15 to 40% of all existing accessibility problems can be identified in the evaluated objects.”
BackgroundSelf-citation
Cited by
Pereira, L. S., & Duarte, C. (2025) · Universal Access in the Information Society · doi:10.1007/s10209-025-01210-w · Cited by 31
“Advances in accessibility evaluation advocate for a blend of conformance-based and empirical methods, focusing on the user’s experience to identify and address issues beyond standard limitations [39].”
Manual Accessibility Testing Procedures for Mobile Apps: A Comparative Evaluation
Steininger, M., Gersbacher, P., & Zimmermann, G. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-31308-9_66
Self-citation
Metadata
- Authors
- Kollotzek, G., Zimmermann, G., Ableitner, T., & Nebe, A.-M.
- Affiliations
- Stuttgart Media University (DE)
- T-Systems Multimedia Solutions GmbH, Competence Center for Digital Accessibility & Software Ergonomics, Riesaer Str. 5, 01129 Dresden, Germany, --- Select a Country ---
- Venue
- Proceedings of the 5th International Conference on Computer-Human Interaction Research and Applications
- Published
- 2021
- Pages
- 24-35
- OpenAlex
- W3212927754
- Semantic Scholar
- cfba2094c0627d238faf9a8dfbe801980e6ff4cb
- Cited by
- 2
- References
- 31
- Citations by year
- 2026: 1 · 2025: 1
- FWCI
- 0.00
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc-nd
- Locations
- International Conference on Computer-Human Interaction Research and Applications (CHIRA) (publishedVersion, cc-by-nc-nd)
- Topics
- Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Health Sciences › Health Professions › Occupational Therapy › Assistive Technology in Communication and Mobility
- Keywords
- web accessibility, accessibility evaluation, comparison of assessment methods, web accessibility standards
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A second model labelled each claim on 2026-10-12 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Comparison of Manual Evaluation Methods for Assessing the Accessibility of Websites based on EN 301 549. https://app.cochat.ai/papers/10.5220/0010647000003060
- The paper
- Kollotzek, G., Zimmermann, G., Ableitner, T., & Nebe, A.-M. (2021). Comparison of Manual Evaluation Methods for Assessing the Accessibility of Websites based on EN 301 549. Proceedings of the 5th International Conference on Computer-Human Interaction Research and Applications. https://doi.org/10.5220/0010647000003060
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.