How much does expertise matter?: a barrier walkthrough study with experts and non-experts
The DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Abstract
Manual accessibility evaluation plays an important role in validating the accessibility of Web pages. This role has become increasingly critical with the advent of the Web Content Accessibility Guidelines (WCAG) 2.0 and their reliance on user evaluation to validate certain conformance measures. However, the role of expertise, in such evaluations, is unknown and has not previously been studied. This paper sets out to investigate the interplay between expert and non-expert evaluation by conducting a Barrier Walkthrough (BW) study with 19 expert and 51 non-expert judges. The BW method provides an evaluation framework that can be used to manually assess the accessibility of Web pages for different user groups including motor impaired, hearing impaired, low vision, cognitive impaired, etc. We conclude that the level of expertise is an important factor in the quality of accessibility evaluation of Web pages. Expert judges spent significantly less time than non-experts; rated themselves as more productive and confident than non-experts; and ranked and rated pages differently against each type of disability. Finally, both effectiveness and reliability of the expert judges are significantly higher than non-expert judges.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For example, expertbased manual testing (Abou-Zahra, 2008), barrier walkthroughs (Yesilada et al., 2009), and automated testing (Vigo et al., 2013) use given standards as their foundation.”
From How compliance with web accessibility standards shapes the experiences of users with and without disabilities (2023) · cites this paper as Yesilada et al., 2009
Citations
Cites
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 24,532 · cited 1×
“More specifically, we used the two-way random single measure of agreement, ICC(2,1) according to conventions in [24].”
MethodThe Paradox of Choice: Why More Is Less
Schwartz, B. N. (2004) · Works - Scholarship, Research, & Creative Expression (Swarthmore College) · Cited by 2,512
Finding usability problems through heuristic evaluation
Nielsen, J. (1992) · Proceedings of the SIGCHI conference on Human factors in computing systems - CHI '92 · doi:10.1145/142750.142834 · Cited by 1,060 · cited 1×
“…Evaluation/methodology, input devices and strategies, user-centred design, interaction styles;K.4.2[ComputersandSociety]:SocialIssues Handicapped persons/special needs, assistive technologies for persons with disabilities GeneralTerms…”
BackgroundInfluentialThe state of the art in automating usability evaluation of user interfaces
Ivory, M. Y., & Hearst, M. A. (2001) · ACM Computing Surveys · doi:10.1145/503112.503114 · Cited by 875
Web Content Accessibility Guidelines 2.0: A further step towards accessible digital information
Ribera, M., Porras, M., Boldu, M., Termens, M., Sule, A., & Paris, P. (2009) · Program · doi:10.1108/00330330910998048 · Cited by 595
Damaged Merchandise? A Review of Experiments That Compare Usability Evaluation Methods
Gray, W. D., & Salzman, M. C. (1998) · Human–Computer Interaction · doi:10.1207/s15327051hci1303_2 · Cited by 488
The Evaluator Effect: A Chilling Fact About Usability Evaluation Methods
Hertzum, M., & Jacobsen, N. E. (2001) · International Journal of Human-Computer Interaction · doi:10.1207/s15327590ijhc1304_05 · Cited by 419
Criteria For Evaluating Usability Evaluation Methods
Hartson, H. R., Andre, T. S., & Williges, R. C. (2001) · International Journal of Human-Computer Interaction · doi:10.1207/s15327590ijhc1304_03 · Cited by 406
The relationship between accessibility and usability of websites
Petrie, H., & Kheir, O. (2007) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.1240688 · Cited by 297
Heuristic Walkthroughs: Finding the Problems Without the Noise
Sears, A. (1997) · International Journal of Human-Computer Interaction · doi:10.1207/s15327590ijhc0903_2 · Cited by 258
Mankoff, J., Fait, H., & Tran, T. (2005) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1054972.1054979 · Cited by 207
Criteria For Evaluating Usability Evaluation Methods
Hartson, H. R., Andre, T. S., & Williges, R. C. (2003) · International Journal of Human-Computer Interaction · doi:10.1207/s15327590ijhc1501_13 · Cited by 186 · cited 2×
“ness (the effectiveness and usab ility of the produced results) and the method’s usability (how easily it can be understood, learned and remembered by evaluators) [23, 8, 9].”
MethodUnderstanding Inspection Methods: Lessons from an Assessment of Heuristic Evaluation
Cockton, G., & Woolrych, A. (2001) · Springer eBooks · doi:10.1007/978-1-4471-0353-0_11 · Cited by 176
The evaluator effect in usability tests
Jacobsen, N. E., Hertzum, M., & John, B. E. (1998) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/286498.286737 · Cited by 153
Web Content Accessibility Guidelines 2.0
Chisholm, W. A., & White, J. J. G. (2001) · Cited by 139
Comparative evaluation of usability tests
Molich, R., Thomsen, A. D., Karyukina, B., Schmidt, L., Ede, M., van Oel, W., & Arcuri, M. (1999) · CHI '99 extended abstracts on Human factors in computing systems - CHI '99 · doi:10.1145/632716.632768 · Cited by 95 · cited 1×
“Several studies of usability evaluation methods have shown that user testing methods may fail in yielding consistent results when performed by differ ent evaluators [22, 12] and that inspectionbased methods are not free of shortcomings either [24, 25, 6, 9].”
BackgroundWeb Accessibility Testing: When the Method Is the Culprit
Brajnik, G. (2006) · Lecture Notes in Computer Science · doi:10.1007/11788713_24 · Cited by 87 · cited 2×
“ABarrierWalkthroughStudywithExpertsandNon-Experts YelizYesiladaSchoolofComputerScienceUniversityofManchesterManchester,UKyesilady@man.ac.uk GiorgioBrajnikDept.ofComputerScienceUniversityofUdineUdine,Italygiorgio@dimi.uniud.it SimonHarperSchoolofComputerScienceUniversityofManchesterManchester,UKsharper@cs.man.ac.uk ABS…”
Self-citationUsability Evaluation and Prototype Fidelity: Users and Usability Professionals
Catani, M. B., & Biers, D. W. (1998) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/154193129804201901 · Cited by 58 · cited 2×
“Ling and Salvendy [18] show that there is a clear effect of the evaluator’s cognitive style on heuristic evaluation and Catani and Bears [4] demonstrate that the individual judgements of severity were highly personal.”
BackgroundLing, C., & Salvendy, G. (2008) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2008.11.002 · Cited by 45
Cegarra, J., & Hoc, J. (2005) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2005.06.003 · Cited by 44 · cited 1×
“Some researchers have also investigated the factors involved in the evaluator effect [13, 14, 18, 5].”
BackgroundUsability inspections by groups of specialists
Hertzum, M., Jacobsen, N. E., & Molich, R. (2002) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/506443.506534 · Cited by 41
A Study of the Evaluator Effect in Usability Testing
Hornbæk, K., & Frøkjær, E. (2008) · Human-Computer Interaction · doi:10.1080/07370020802278205 · Cited by 36
The Evaluator Effect during First-Time Use of the Cognitive Walkthrough TechniquePreprint
Hertzum, M., & Jacobsen, N. E. (1999) · Cited by 22
Research works printed but not linked: 4
- DRC. Formal investigation report : web accessibility. Disability Rights Commission, www.drc-gb.org , April 2004 . Visited Jan . 2006. DRC. Formal investigation report: web accessibility. Disability Rights Commission, www.drc-gb.org, April 2004. Visited Jan. 2006.
- S.L. Henry and M. Grossnickle . Just Ask: Accessibility in the User-Centered Design Process . Georgia Tech Research Corporation , Atlanta, Georgia, USA , 2004 . S.L. Henry and M. Grossnickle. Just Ask: Accessibility in the User-Centered Design Process. Georgia Tech Research Corporation, Atlanta, Georgia, USA, 2004.
- T. Lang . Comparing website accessibility evaluation methods and learnings from usability evaluation methods. http://www.peakusability.com.au , Visited May 2008 , 2003. T. Lang. Comparing website accessibility evaluation methods and learnings from usability evaluation methods. http://www.peakusability.com.au, Visited May 2008, 2003.
- A. Woolrych and G. Cockton . Assessing heuristic evaluation: mind the quality, not just the percentages . In In Proc. of HCI 2000 , pages 35 -- 36 , 2000 . A. Woolrych and G. Cockton. Assessing heuristic evaluation: mind the quality, not just the percentages. In In Proc. of HCI 2000, pages 35--36, 2000.
Cited by
Usability, user experience and accessibility: towards an integrative model
Sauer, J., Sonderegger, A., & Schmutz, S. (2020) · Ergonomics · doi:10.1080/00140139.2020.1774080 · Cited by 195
Ensuring Digital Accessibility Through Process and Policy
Lazar, J., Goldstein, D., & Taylor, A. (2015) · Elsevier eBooks · doi:10.1016/c2013-0-13367-3 · Cited by 359
The Expertise Effect on Web Accessibility Evaluation Methods
Brajnik, G., Yesilada, Y., & Harper, S. (2011) · Human–Computer Interaction · doi:10.1080/07370024.2011.601670 · Cited by 83
“Preliminary and partial results of this study were published in Yesilada, Brajnik, and Harper (2009); here we significantly extend that work.”
Self-citationAccessibility challenges and tool features: an IBM Web developer perspective
Trewin, S., Cragun, B., Swart, C., Brezin, J., & Richards, J. (2010) · Proceedings of the 2010 International Cross Disciplinary Conference on Web Accessibility (W4A) · doi:10.1145/1805986.1806029 · Cited by 70
“The reliability of human judgment of accessibility issues, however, is lower when the evaluators have less accessibility experience [5, 12 ] Brajnik [2] examined a group of relatively inexperienced web developers evaluating conformance of a Web page to accessibility guidelines, and found low reliability for both WCAG…”
BackgroundMethodInfluentialWeb not for all: a large scale study of web accessibility
Lopes, R., Gomes, D., & Carriço, L. (2010) · Proceedings of the 2010 International Cross Disciplinary Conference on Web Accessibility (W4A) · doi:10.1145/1805986.1806001 · Cited by 67
“Since most developers and designers are not accessibility experts, and since non-experts tend to incorrectly evaluate accessibility [10], we hypothesise that the optimistic rate might shed light on the real perception of accessibility by non-experts at the large.”
Background
Metadata
- Authors
- Yesilada, Y., Brajnik, G., & Harper, S.
- Affiliations
- University of Manchester, Manchester, United Kingdom
- University of Udine (IT)
- University of Manchester (GB)
- Venue
- Proceedings of the 11th international ACM SIGACCESS conference on Computers and accessibility
- Publisher
- ACM
- Published
- 2009-10-25
- Pages
- 203-210
- OpenAlex
- W2159132279
- Cited by
- 53
- References
- 23
- Citations by year
- 2025: 1 · 2023: 1 · 2022: 8 · 2021: 3 · 2020: 1 · 2019: 3 · 2018: 4 · 2017: 4 · 2016: 1 · 2015: 4 · 2014: 1 · 2013: 3 · 2012: 3
- FWCI
- 9.94
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) (publishedVersion)
- Institutional Research Information System (University of Udine) (submittedVersion)
- Funders
- Università degli Studi di Udine
- Topics
- Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Health Sciences › Health Professions › Occupational Therapy › Assistive Technology in Communication and Mobility
- Keywords
- web accessibility evaluation, expertise, Web Content Accessibility Guidelines 2.0, assessment reliability, evaluation effectiveness, disability type
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.