The DOI matches its Crossref record; 6 of the 6 references it cites were found in a registry or an index; 5 of the 30 sources its claims cite were checked for support.
Analysis of web accessibility evaluation tools and guidelines for monitoring according to the Directive (EU) 2016/2102
Open accessAbstract
This paper presents a comparative analysis of automated web accessibility assessment tools and guidelines for monitoring of public websites and reporting according to the Directive (EU) 2016/2102. A review of previous research and our own comparative analysis revealed that the tools differ not only in functionality and result presentation, but also in violation discovery. Overviews of the Web Accessibility Directive Expert Group meetings` minutes and first reports by the European member states showed that unified monitoring and reporting do not exist between member states. We provided guidelines that would meet the minimum requirements of the Commission Implementing Decision (EU) 2018/1524, as well as guidelines for more detailed evaluation, which would improve results in terms of providing a clearer picture of the actual level of accessibility and enable efficient comparison between European member states.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
5 Valid · 1 Not valid · 19 Not checked · 74 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, we recommend using at least two different tools at the same time, as past research [1,3,12], as well as our comparative analysis of evaluation tools, have shown that usage of multiple tools increases the accuracy of the results.”
§ Analysis of web accessibility evaluation tools and guidelines for monitoring according to the Directive (EU) 2016/2102 > 5.2 Guidelines for monitoring according to the Directive (EU) 2016/2102
What the sources say
- Not checked yetcites [1] 1. Abduganiev, G. S. Towards Automated Web Accessibility Evaluation: A Compar- ative Study. I. J. Information Technology and Computer Science, 9, (2017). Avail- able on: https://www.mecs-press.org/ijitcs/ijitcs-v9-n9/IJITCS-V9-N9-3.pdf [1. 7. 2021].
- Partly supportedcites [3] 3. Burkard, A., Zimmermann, G., Schwarzer, B. Monitoring Systems for Checking Websites on Accessibility. In: Frontiers in Computer Science, 3, (2021). tp [1. 7. 2021]. sourceExperimental — accuracy not yet measured
The Discussion (page unavailable) says that combining two rulesets can balance their disadvantages: one finds more errors but may produce false positives, while the other finds fewer errors without false positives. That supports a narrower point about complementary checks within one system, not the claim that using two different tools increases evaluation accuracy. (LLM, conf=0.78)
- Not checked — no full text of the sourcecites [12] 12. Vigo, M., Brown, J., and Conway, V. Benchmarking web accessibility evaluation tools: measuring the harm of sole reliance on automated tests. In: Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility, 1-10, (2013). Available on: https://dl.acm.org/doi/10.1145/2461121.2461124 [1. 7. 2021]. source
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the other hand, Userway Scanner provided a link to the appropriate WCAG technique for each detected error.”
§ Analysis of web accessibility evaluation tools and guidelines for monitoring according to the Directive (EU) 2016/2102 > 5.1.2 Results of the evaluations with "not-free-of-charge" tools:
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“TAW and Tingtun Checker tools did not yet support WCAG 2.1.”
§ Analysis of web accessibility evaluation tools and guidelines for monitoring according to the Directive (EU) 2016/2102 > 5.1.1
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When researching”
§ Analysis of web accessibility evaluation tools and guidelines for monitoring according to the Directive (EU) 2016/2102 > 5.1.2 Results of the evaluations with "not-free-of-charge" tools:
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“MAUVE++ was also the only tool which detected issues with those success criteria.”
§ Analysis of web accessibility evaluation tools and guidelines for monitoring according to the Directive (EU) 2016/2102 > 5.1.1
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
27 more claims
Create a free account to see all 99 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 44
- It may state two separate claims as one. 42
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Analysis of web accessibility evaluation tools and guidelines for monitoring according to the Directive (EU) 2016/2102 > 3 RELATED WORK
“based on the requirements from the Directive (EU) 2016/2101 and Commission Implementing Decisions (EU) 2018/1524 and (EU) 2021/1339.”
§ Analysis of web accessibility evaluation tools and guidelines for monitoring according to the Directive (EU) 2016/2102 > 2 GOALS AND RESEARCH QUESTIONS
“The purpose of our work was a comprehensive research and development of guidelines that would help inspection authorities with the simplified and in-depth monitoring of public websites, and reporting of results in accordance with the Directive (EU) 2016/2102.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Analysis of web accessibility evaluation tools and guidelines for monitoring according to the Directive (EU) 2016/2102 > 3 RELATED WORK
“Major differences were observed between tools according to 8 evaluation criteria:”
§ Analysis of web accessibility evaluation tools and guidelines for monitoring according to the Directive (EU) 2016/2102 > 3 RELATED WORK
“coverage, completeness, correctness, specificity, interreliability, intra-reliability, validity, efficiency and capacity.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Analysis of web accessibility evaluation tools and guidelines for monitoring according to the Directive (EU) 2016/2102 > 4 METHODOLOGY AND PROCEDURE
“To decide which 10 free tools we wanted to include in our study”
§ Analysis of web accessibility evaluation tools and guidelines for monitoring according to the Directive (EU) 2016/2102 > 4 METHODOLOGY AND PROCEDURE
“We decided to evaluate the following free tools: WAVE, Access Monitor Plus, IBM Equal Access Accessibility Checker, Accessibility Insights for Web, BOSA, MAUVE++, Tingtun Checker, TAW and Tota11y.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Analysis of web accessibility evaluation tools and guidelines for monitoring according to the Directive (EU) 2016/2102 > 1 INTRODUCTION
“The European Parliament and the Council adopted the Directive (EU) 2016/2102 on the accessibility of the websites and mobile applications of public sector bodies on 26 October 2016 [7].”
§ Analysis of web accessibility evaluation tools and guidelines for monitoring according to the Directive (EU) 2016/2102 > References
“7. ETSI, CEN, CENELEC. EN 301 549 V2.”
Citations
Cites
Vigo, M., Brown, J., & Conway, V. (2013) · Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility · doi:10.1145/2461121.2461124 · Cited by 215
Comparing accessibility evaluation tools: a method for tool effectiveness
Brajnik, G. (2004) · Universal Access in the Information Society · doi:10.1007/s10209-004-0105-y · Cited by 101
Voss, D. F. (1991) · Science · doi:10.1126/science.253.5025.1189 · Cited by 90
Monitoring Systems for Checking Websites on Accessibility
Burkard, A., Zimmermann, G., & Schwarzer, B. (2021) · Frontiers in Computer Science · doi:10.3389/fcomp.2021.628770 · Cited by 24
Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility
Brajnik, G., Brazil, W. '. I. C.-D. C. O. W. A. 2. R. D. J., Association for Computing Machinery-Digital Library, ACM Special Interest Group on Accessible Computing, ACM Special Interest Group on Computer-Human Interaction, Hypermedia and Web ACM Special Interest Group on Hypertext, & SIGs, A. (2013) · doi:10.1145/2461121 · Cited by 21
Dostopnost digitalnih produktov za vse
Debevc, M. (2021) · doi:10.18690/978-961-286-426-2
Self-citation
Research works printed but not linked: 3
- Abduganiev, G. S. Towards Automated Web Accessibility Evaluation: A Comparative Study. I. J. Information Technology and Computer Science, 9, (2017). Available on: https://www.mecs-press.org/ijitcs/ijitcs-v9-n9/IJITCS-V9-N9-3.pdf [1. 7. 2021].
- Directive (EU) 2016/2102 of the European Parliament and of the Council of 26 October 2016 on the Accessibility of the Websites and Mobile Applications of Public Sector Bodies. OJ L 327 2 December 2016. pp. 1–15.
- Kous, K., Pavlinek, M. Longitudinalna raziskava dostopnosti spletnih mest slovenskih občin. In: Revija Uporabna informatika, 3, (2017).
Cited by
Vindigni, G. (2025) · European journal of management, economics and business. · doi:10.59324/ejmeb.2025.2(1).03 · Cited by 8
Metadata
- Authors
- Rajh, N., & Debevc, M.
- Affiliations
- University of Maribor (SI)
- Venue
- Proceedings of the 10th International Conference on Software Development and Technologies for Enhancing Accessibility and Fighting Info-exclusion
- Publisher
- ACM
- Published
- 2022-08-31
- Pages
- 195-202
- OpenAlex
- W4378211954
- Cited by
- 3
- References
- 2
- Citations by year
- 2025: 2 · 2023: 1
- FWCI
- 1.00
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- International Conference on Software Development and Technologies for Enhancing Accessibility and Fighting Info-exclusion (DSAI) (publishedVersion)
- Funders
- European Commission (KA-203, 2020-1-LV01-KA203-077455)
- Topics
- Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Social Sciences › Political Science and International Relations › E-Government and Public Services
- Keywords
- web accessibility, accessibility monitoring, public websites, violation detection, accessibility guidelines
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Analysis of web accessibility evaluation tools and guidelines for monitoring according to the Directive (EU) 2016/2102. https://app.cochat.ai/papers/10.1145/3563137.3563148
- The paper
- Rajh, N., & Debevc, M. (2022). Analysis of web accessibility evaluation tools and guidelines for monitoring according to the Directive (EU) 2016/2102. Proceedings of the 10th International Conference on Software Development and Technologies for Enhancing Accessibility and Fighting Info-exclusion. https://doi.org/10.1145/3563137.3563148
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
