How much does expertise matter?: a barrier walkthrough study with experts and non-experts
The DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Abstract
Manual accessibility evaluation plays an important role in validating the accessibility of Web pages. This role has become increasingly critical with the advent of the Web Content Accessibility Guidelines (WCAG) 2.0 and their reliance on user evaluation to validate certain conformance measures. However, the role of expertise, in such evaluations, is unknown and has not previously been studied. This paper sets out to investigate the interplay between expert and non-expert evaluation by conducting a Barrier Walkthrough (BW) study with 19 expert and 51 non-expert judges. The BW method provides an evaluation framework that can be used to manually assess the accessibility of Web pages for different user groups including motor impaired, hearing impaired, low vision, cognitive impaired, etc. We conclude that the level of expertise is an important factor in the quality of accessibility evaluation of Web pages. Expert judges spent significantly less time than non-experts; rated themselves as more productive and confident than non-experts; and ranked and rated pages differently against each type of disability. Finally, both effectiveness and reliability of the expert judges are significantly higher than non-expert judges.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For example, expertbased manual testing (Abou-Zahra, 2008), barrier walkthroughs (Yesilada et al., 2009), and automated testing (Vigo et al., 2013) use given standards as their foundation.”
From How compliance with web accessibility standards shapes the experiences of users with and without disabilities (2023) · cites this paper as Yesilada et al., 2009
Citations
Cites
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 24,532 · cited 1×
“More specifically, we used the two-way random single measure of agreement, ICC(2,1) according to conventions in [24].”
MethodThe Paradox of Choice: Why More Is Less
Schwartz, B. N. (2004) · Works - Scholarship, Research, & Creative Expression (Swarthmore College) · Cited by 2,512
Finding usability problems through heuristic evaluation
Nielsen, J. (1992) · Proceedings of the SIGCHI conference on Human factors in computing systems - CHI '92 · doi:10.1145/142750.142834 · Cited by 1,060 · cited 1×
“…Evaluation/methodology, input devices and strategies, user-centred design, interaction styles;K.4.2[ComputersandSociety]:SocialIssues Handicapped persons/special needs, assistive technologies for persons with disabilities GeneralTerms…”
BackgroundInfluentialThe state of the art in automating usability evaluation of user interfaces
Ivory, M. Y., & Hearst, M. A. (2001) · ACM Computing Surveys · doi:10.1145/503112.503114 · Cited by 875
Web Content Accessibility Guidelines 2.0: A further step towards accessible digital information
Ribera, M., Porras, M., Boldu, M., Termens, M., Sule, A., & Paris, P. (2009) · Program · doi:10.1108/00330330910998048 · Cited by 595
Damaged Merchandise? A Review of Experiments That Compare Usability Evaluation Methods
Gray, W. D., & Salzman, M. C. (1998) · Human–Computer Interaction · doi:10.1207/s15327051hci1303_2 · Cited by 488
The Evaluator Effect: A Chilling Fact About Usability Evaluation Methods
Hertzum, M., & Jacobsen, N. E. (2001) · International Journal of Human-Computer Interaction · doi:10.1207/s15327590ijhc1304_05 · Cited by 419
Criteria For Evaluating Usability Evaluation Methods
Hartson, H. R., Andre, T. S., & Williges, R. C. (2001) · International Journal of Human-Computer Interaction · doi:10.1207/s15327590ijhc1304_03 · Cited by 406
The relationship between accessibility and usability of websites
Petrie, H., & Kheir, O. (2007) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.1240688 · Cited by 297
Heuristic Walkthroughs: Finding the Problems Without the Noise
Sears, A. (1997) · International Journal of Human-Computer Interaction · doi:10.1207/s15327590ijhc0903_2 · Cited by 258
Research works printed but not linked: 4
- DRC. Formal investigation report : web accessibility. Disability Rights Commission, www.drc-gb.org , April 2004 . Visited Jan . 2006. DRC. Formal investigation report: web accessibility. Disability Rights Commission, www.drc-gb.org, April 2004. Visited Jan. 2006.
- S.L. Henry and M. Grossnickle . Just Ask: Accessibility in the User-Centered Design Process . Georgia Tech Research Corporation , Atlanta, Georgia, USA , 2004 . S.L. Henry and M. Grossnickle. Just Ask: Accessibility in the User-Centered Design Process. Georgia Tech Research Corporation, Atlanta, Georgia, USA, 2004.
- T. Lang . Comparing website accessibility evaluation methods and learnings from usability evaluation methods. http://www.peakusability.com.au , Visited May 2008 , 2003. T. Lang. Comparing website accessibility evaluation methods and learnings from usability evaluation methods. http://www.peakusability.com.au, Visited May 2008, 2003.
- A. Woolrych and G. Cockton . Assessing heuristic evaluation: mind the quality, not just the percentages . In In Proc. of HCI 2000 , pages 35 -- 36 , 2000 . A. Woolrych and G. Cockton. Assessing heuristic evaluation: mind the quality, not just the percentages. In In Proc. of HCI 2000, pages 35--36, 2000.
Cited by
Usability, user experience and accessibility: towards an integrative model
Sauer, J., Sonderegger, A., & Schmutz, S. (2020) · Ergonomics · doi:10.1080/00140139.2020.1774080 · Cited by 195
Ensuring Digital Accessibility Through Process and Policy
Lazar, J., Goldstein, D., & Taylor, A. (2015) · Elsevier eBooks · doi:10.1016/c2013-0-13367-3 · Cited by 359
The Expertise Effect on Web Accessibility Evaluation Methods
Brajnik, G., Yesilada, Y., & Harper, S. (2011) · Human–Computer Interaction · doi:10.1080/07370024.2011.601670 · Cited by 83
“Preliminary and partial results of this study were published in Yesilada, Brajnik, and Harper (2009); here we significantly extend that work.”
Self-citationAccessibility challenges and tool features: an IBM Web developer perspective
Trewin, S., Cragun, B., Swart, C., Brezin, J., & Richards, J. (2010) · Proceedings of the 2010 International Cross Disciplinary Conference on Web Accessibility (W4A) · doi:10.1145/1805986.1806029 · Cited by 70
“The reliability of human judgment of accessibility issues, however, is lower when the evaluators have less accessibility experience [5, 12 ] Brajnik [2] examined a group of relatively inexperienced web developers evaluating conformance of a Web page to accessibility guidelines, and found low reliability for both WCAG…”
BackgroundMethodInfluentialWeb not for all: a large scale study of web accessibility
Lopes, R., Gomes, D., & Carriço, L. M. (2010) · International Cross-Disciplinary Conference on Web Accessibility (W4A) · doi:10.1145/1805986.1806001 · Cited by 81
“Since most developers and designers are not accessibility experts, and since non-experts tend to incorrectly evaluate accessibility [10], we hypothesise that the optimistic rate might shed light on the real perception of accessibility by non-experts at the large.”
Background
Metadata
- Authors
- Yesilada, Y., Brajnik, G., & Harper, S.
- Affiliations
- University of Manchester, Manchester, United Kingdom
- University of Udine (IT)
- University of Manchester (GB)
- Venue
- Proceedings of the 11th international ACM SIGACCESS conference on Computers and accessibility
- Publisher
- ACM
- Published
- 2009-10-25
- Pages
- 203-210
- OpenAlex
- W2159132279
- Cited by
- 53
- References
- 23
- Citations by year
- 2025: 1 · 2023: 1 · 2022: 8 · 2021: 3 · 2020: 1 · 2019: 3 · 2018: 4 · 2017: 4 · 2016: 1 · 2015: 4 · 2014: 1 · 2013: 3 · 2012: 3
- FWCI
- 9.94
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) (publishedVersion)
- Institutional Research Information System (University of Udine) (submittedVersion)
- Funders
- Università degli Studi di Udine
- Topics
- Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Health Sciences › Health Professions › Occupational Therapy › Assistive Technology in Communication and Mobility
- Keywords
- web accessibility evaluation, expertise, Web Content Accessibility Guidelines 2.0, assessment reliability, evaluation effectiveness, disability type
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: How much does expertise matter?: a barrier walkthrough study with experts and non-experts. https://app.cochat.ai/papers/10.1145/1639642.1639678
- The paper
- Yesilada, Y., Brajnik, G., & Harper, S. (2009). How much does expertise matter?: a barrier walkthrough study with experts and non-experts. Proceedings of the 11th international ACM SIGACCESS conference on Computers and accessibility. https://doi.org/10.1145/1639642.1639678
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.