The Expertise Effect on Web Accessibility Evaluation Methods
Open accessThe DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Abstract
Web accessibility means that disabled people can effectively perceive, understand, navigate, and interact with the web. Web accessibility evaluation methods are needed to validate the accessibility of web pages. However, the role of subjectivity and of expertise in such methods is unknown and has not previously been studied. This article investigates the effect of expertise in web accessibility evaluation methods by conducting a Barrier Walkthrough (BW) study with 19 expert and 57 nonexpert judges. The BW method is an evaluation method that can be used to manually assess the accessibility of web pages for different user groups such as motor impaired, low vision, blind, and mobile users. Our results show that expertise matters, and even though the effect of expertise varies depending on the metric used to measure quality, the level of expertise is an important factor in the quality of accessibility evaluation of web pages. In brief, when pages are evaluated with nonexperts, we observe a drop in validity and reliability. We also observe a negative monotonic relationship between number of judges and reproducibility: more evaluators mean more diverse outputs. After five experts, reproducibility stabilizes, but this is not the case with nonexperts. The ability to detect all the problems increases with the number of judges: With 3 experts all problems can be found, but for such a level 14 nonexperts are needed. Even though our data show that experts rated pages differently, the difference is quite small. Finally, compared to nonexperts, experts spent much less time and the variability among them is smaller, they were significantly more confident, and they rated themselves as being more productive. The article discusses practical implications regarding how BW results should be interpreted, how to recruit evaluators, and what happens when more than one evaluator is hired. Supplemental materials are available for this article. Go to the publisher's online edition of Human-Computer Interaction for statistical details and additional measures for this article. © 2011 Taylor & Francis Group, LLC.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Usability Engineering Lifecycle
NIELSEN, J. (1993) · Usability Engineering · doi:10.1016/b978-0-08-052029-2.50007-3 · Cited by 2,657
Remote usability evaluations with disabled people.
Petrie, H., Hamilton, F., King, N., & Pavan, P. (2006) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1124772.1124942 · Cited by 157 · cited 1×
“A useful variant is remote user testing ; Petrie, Hamilton, King, and Pavan (2006) discussed pros and cons of remote usability evaluations with disabled people.”
Beyond Conformance: The Role of Accessibility Evaluation Methods
Brajnik, G. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-85200-1_9 · Cited by 114 · cited 2×
“Different methods exist to evaluate accessibility of web pages, which can be broadly categorized into five groups (Brajnik, 2008).”
Self-citationComparing accessibility evaluation tools: a method for tool effectiveness
Brajnik, G. (2004) · Universal Access in the Information Society · doi:10.1007/s10209-004-0105-y · Cited by 101 · cited 1×
“There are many tools available, 2 yielding different results with different levels of quality (Abou-Zahra, 2008; Thatcher et al., 2006; Brajnik, 2004).”
Self-citationWeb Accessibility Testing: When the Method Is the Culprit
Brajnik, G. (2006) · Lecture Notes in Computer Science · doi:10.1007/11788713_24 · Cited by 87 · cited 4×
“Barriers tested in the study can be found at Brajnik (2006b).”
InfluentialSelf-citationAbou-Zahra, S. (2008) · Human-Computer Interaction Series · doi:10.1007/978-1-84800-050-6_7 · Cited by 73 · cited 3×
“Access Board, 2001) and has to decide if a page or web site complies with those requirements ( Abou-Zahra, 2008 ; DRC, 2004; Henry & Grossnickle, 2004 ; Thatcher et al, 2006 ).”
How much does expertise matter?: a barrier walkthrough study with experts and non-experts
Yesilada, Y., Brajnik, G., & Harper, S. (2009) · Proceedings of the 11th international ACM SIGACCESS conference on Computers and accessibility · doi:10.1145/1639642.1639678 · Cited by 53 · cited 1×
“Preliminary and partial results of this study were published in Yesilada, Brajnik, and Harper (2009); here we significantly extend that work.”
Self-citation
Research works printed but not linked: 32
- Brajnik , G. March 2006b . Web accessibility testing with barriers walkthrough March , Retrieved from http://www.dimi.uniud.it/giorgio/projects/bw/bw.html
- Brajnik , G. Validity and reliability of web accessibility guidelines . Proceedings of the 11th International ACM SIGACCESS Conference on Computers and Accessibility — ASSETS 2009 . pp. 131 – 138 . New York, NY : ACM. .
- Brajnik , G. , Yesilada , Y. and Harper , S. Testability and validity of WCAG 2.0: The expertise effect . Proceedings of the 12th International ACM SIGACCESS Conference on Computers and Accessibility, ASSETS 2010 . pp. 43 – 50 . New York, NY : ACM. . doi: http://doi.acm.org/10.1145/1878803.1878813 · DOI · Paper page
- Catani , M. and Biers , D. Usability evaluation and prototype fidelity: Users and usability professionals . Proceedings of the Human Factors and Ergonomics Society 42nd Annual Meeting . pp. 1131 – 1336 . Human Factors and Ergonomics Society. .
- Coyne , K. P. and Nielsen , J. 2001 . How to conduct usability evaluations for accessibility: Methodology guidelines for testing websites and intranets with users who use assistive technology Fremont, CA : Nielsen Norman Group. . Retrieved from http://www.nngroup.com/reports/accessibility/testing
- Disability Rights Commission . April 2004 . Formal investigation report: Web accessibility April , London, , England : Author. . Retrieved from http://www.drc-gb.org/publicationsandreports/report.asp
- 10.1207/s15327051hci1303_2 · DOI · Paper page
- Harper , S. and Yesilada , Y. 2008 . “ Web accessibility and guidelines. ” . In Web accessibility: A foundation for research Edited by: Harper , S. and Yesilada , Y. 61 – 78 . London, , England : Springer. . · DOI
- 10.1207/S15327590IJHC1501_13 · DOI · Paper page
- Henry , S. L. and Grossnickle , M. 2004 . Just ask: Accessibility in the user-centered design process Atlanta : Georgia Tech Research Corporation. .
- Hertzum , M. and Jacobsen , N. E. 1999 . “ The evaluator effect during first-time use of the cognitive walkthrough technique. ” . In Human-Computer Interaction: Ergonomics and User Interfaces Edited by: Bullinger , H. J. and Ziegler , J. Vol. 1 , 1063 – 1067 . Lawrence Erlbaum Associates. .
- Hertzum , M. and Jacobsen , N. E. 2001 . The evaluator effect: A chilling fact about usability evaluation methods . International Journal of Human-Computer Interaction , 1 : 421 – 443 . · DOI · Paper page
- Hertzum , M. , Jacobsen , N. E. and Molich , R. 2002 . “ Usability inspections by groups of specialists: Perceived agreement in spite of disparate observations. ” . In CHI 2002 Extended Abstracts 662 – 663 . New York, NY : ACM. .
- 10.1080/07370020802278205 · DOI
- 10.1145/1735223.1735255 · DOI · Paper page
- Jacobsen , N. E. , Hertzum , M. and John , B. The evaluator effect in usability tests . CHI 1998, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems . pp. 255 – 256 . New York, NY : ACM. .
- Jay , C. , Lunn , D. and Michailidou , E. 2008 . “ End user evaluations. ” . In Web accessibility: A foundation for research Edited by: Harper , S. and Yesilada , Y. 107 – 126 . London, , England : Springer. . · doi:10.1007/978-1-84800-050-6_8
- Lang , T. 2003 . Comparing website accessibility evaluation methods and learnings from usability evaluation methods Retrieved from http://www.peakusability.com.au/about-us/pdf/website_accessibility.pdf
- 10.1016/j.ijhcs.2008.11.002 · DOI
- Mankoff , J. , Fait , H. and Tran , T. Is your web page accessible? A comparative study of methods for assessing web page accessibility for the blind . CHI 2005, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems . pp. 41 – 50 . New York, NY : ACM. .
- Molich , R. , Bevan , N. , Curson , I. , Butler , S. , Kindlund , E. , Miller , D. and Kirakowski , J. Comparative evaluation of usability tests . Proceedings of the Usability Professionals Association Conference . pp. 189 – 200 . Washington, DC : UPA. .
- Nielsen , J. Finding usability problems through heuristic evaluation . CHI 1992, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems . pp. 373 – 380 . New York, NY : ACM. .
- Nielsen , J. Reliability of severity estimates for usability problems found by heuristic evaluation . CHI 1992: Posters and short talks of the 1992 SIGCHI Conference on Human Factors in Computing Systems . pp. 129 – 130 . New York, NY : ACM. .
- Nielsen , J. and Molich , R. Heuristic evaluation of user interfaces . CHI 1990, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems . pp. 249 – 256 . New York, NY : ACM. .
- Petrie , H. and Kheir , O. 2007 . “ The relationship between accessibility and usability of websites. ” . In Proceedings of CHI 2007 397 – 406 . New York, NY : ACM. .
- Rubin , J. and Chisnell , D. 2008 . Handbook of usability testing, , 2nd ed. Hoboken, NJ : Wiley. .
- 10.1207/s15327590ijhc0903_2 · DOI · Paper page
- Swets , J. A. 1996 . Signal detection theory and ROC analysis in psychology and diagnostics: Collected papers Lawrence Erlbaum Associates. .
- Takagi , H. , Kawanaka , S. , Kobayashi , M. , Sato , D. and Asakawa , C. Collaborative web accessibility improvement: Challenges and possibilities . Proceedings of the 11th International ACM SIGACCESS conference on Computers and Accessibility . pp. 195 – 202 . New York, NY : ACM. . doi: http://doi.acm.org/10.1145/1639642.1639677 · DOI
- Thatcher , J. , Burks , M. , Heilmann , C. , Henry , S. , Kirkpatrick , A. , Lauke , P. , Lawson , B. , Regan , B. , Rutter , R. , Urban , M. and Waddel , C. 2006 . Web accessibility: Web standards and regulatory compliance New York, NY : FriendsofED. .
- Trochim , W. M. 2006 . The research methods knowledge base, , 2nd ed. Retrieved from http://www.socialresearchmethods.net/kb/
- Woolrych , A. , Cockton , G. and Hindmarch , M. 2004 . “ Falsification testing for usability inspection method assessment. ” . In Proceedings of HCI 2004 137 – 140 . London, , England : Springer-Verlag. .
Cited by
Vigo, M., Brown, J., & Conway, V. (2013) · Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility · doi:10.1145/2461121.2461124 · Cited by 215
“3 experts guarantee that almost all accessibility violations can be found with a high reliability [9].”
BackgroundAlsaeedi, A. (2020) · Information · doi:10.3390/info11010040 · Cited by 101
“Brajnik et al. [22] demonstrated the ability of expert users to detect accessibility violations with more reliability.”
BackgroundAbuaddous, H. Y., Zalisham, M., & Basir, N. (2016) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2016.071023 · Cited by 87
Accessibility of university websites worldwide: a systematic literature review
Campoverde-Molina, M., Luján-Mora, S., & Valverde, L. (2023) · Universal Access in the Information Society · doi:10.1007/s10209-021-00825-z · Cited by 84
“” On the other hand, evaluations with real users [22] are informal or formal experiments that are carried out with people who navigate a website in a normal way and their behavior is observed by the evaluators.”
BackgroundVollenwyder, B., Petralito, S., Iten, G. H., Brühlmann, F., Opwis, K., & Mekler, E. D. (2023) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2022.102956 · Cited by 62
Metadata
- Authors
- Brajnik, G., Yesilada, Y., & Harper, S.
- Venue
- Human–Computer Interaction
- Publisher
- Informa UK Limited
- Published
- 2011-08-30
- Volume
- 26
- Issue
- 3
- Pages
- 246-283
- ISSN
- 0737-0024 (ISSN-L)
- 1532-7051
- OpenAlex
- W1535223020
- Cited by
- 83
- References
- 7
- FWCI
- 11.89
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The Expertise Effect on Web Accessibility Evaluation Methods. https://app.cochat.ai/papers/10.1080/07370024.2011.601670
- The paper
- Brajnik, G., Yesilada, Y., & Harper, S. (2011). The Expertise Effect on Web Accessibility Evaluation Methods. Human–Computer Interaction. https://doi.org/10.1080/07370024.2011.601670
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.