CoChat

The Expertise Effect on Web Accessibility Evaluation Methods

Open access
Brajnik, Giorgio, Yesilada, Yeliz, Harper, Simon · 2011 · Human–Computer Interaction
Cited by 83

The DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.

Abstract

Web accessibility means that disabled people can effectively perceive, understand, navigate, and interact with the web. Web accessibility evaluation methods are needed to validate the accessibility of web pages. However, the role of subjectivity and of expertise in such methods is unknown and has not previously been studied. This article investigates the effect of expertise in web accessibility evaluation methods by conducting a Barrier Walkthrough (BW) study with 19 expert and 57 nonexpert judges. The BW method is an evaluation method that can be used to manually assess the accessibility of web pages for different user groups such as motor impaired, low vision, blind, and mobile users. Our results show that expertise matters, and even though the effect of expertise varies depending on the metric used to measure quality, the level of expertise is an important factor in the quality of accessibility evaluation of web pages. In brief, when pages are evaluated with nonexperts, we observe a drop in validity and reliability. We also observe a negative monotonic relationship between number of judges and reproducibility: more evaluators mean more diverse outputs. After five experts, reproducibility stabilizes, but this is not the case with nonexperts. The ability to detect all the problems increases with the number of judges: With 3 experts all problems can be found, but for such a level 14 nonexperts are needed. Even though our data show that experts rated pages differently, the difference is quite small. Finally, compared to nonexperts, experts spent much less time and the variability among them is smaller, they were significantly more confident, and they rated themselves as being more productive. The article discusses practical implications regarding how BW results should be interpreted, how to recruit evaluators, and what happens when more than one evaluator is hired. Supplemental materials are available for this article. Go to the publisher's online edition of Human-Computer Interaction for statistical details and additional measures for this article. © 2011 Taylor & Francis Group, LLC.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1080/07370024.2011.601670

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 7 references

Research works printed but not linked: 32

  1. Brajnik , G. March 2006b . Web accessibility testing with barriers walkthrough March , Retrieved from http://www.dimi.uniud.it/giorgio/projects/bw/bw.html
  2. Brajnik , G. Validity and reliability of web accessibility guidelines . Proceedings of the 11th International ACM SIGACCESS Conference on Computers and Accessibility — ASSETS 2009 . pp. 131 – 138 . New York, NY : ACM. .
  3. Brajnik , G. , Yesilada , Y. and Harper , S. Testability and validity of WCAG 2.0: The expertise effect . Proceedings of the 12th International ACM SIGACCESS Conference on Computers and Accessibility, ASSETS 2010 . pp. 43 – 50 . New York, NY : ACM. . doi: http://doi.acm.org/10.1145/1878803.1878813 · DOI · Paper page
  4. Catani , M. and Biers , D. Usability evaluation and prototype fidelity: Users and usability professionals . Proceedings of the Human Factors and Ergonomics Society 42nd Annual Meeting . pp. 1131 – 1336 . Human Factors and Ergonomics Society. .
  5. Coyne , K. P. and Nielsen , J. 2001 . How to conduct usability evaluations for accessibility: Methodology guidelines for testing websites and intranets with users who use assistive technology Fremont, CA : Nielsen Norman Group. . Retrieved from http://www.nngroup.com/reports/accessibility/testing
  6. Disability Rights Commission . April 2004 . Formal investigation report: Web accessibility April , London, , England : Author. . Retrieved from http://www.drc-gb.org/publicationsandreports/report.asp
  7. 10.1207/s15327051hci1303_2 · DOI · Paper page
  8. Harper , S. and Yesilada , Y. 2008 . “ Web accessibility and guidelines. ” . In Web accessibility: A foundation for research Edited by: Harper , S. and Yesilada , Y. 61 – 78 . London, , England : Springer. . · DOI
  9. 10.1207/S15327590IJHC1501_13 · DOI · Paper page
  10. Henry , S. L. and Grossnickle , M. 2004 . Just ask: Accessibility in the user-centered design process Atlanta : Georgia Tech Research Corporation. .
  11. Hertzum , M. and Jacobsen , N. E. 1999 . “ The evaluator effect during first-time use of the cognitive walkthrough technique. ” . In Human-Computer Interaction: Ergonomics and User Interfaces Edited by: Bullinger , H. J. and Ziegler , J. Vol. 1 , 1063 – 1067 . Lawrence Erlbaum Associates. .
  12. Hertzum , M. and Jacobsen , N. E. 2001 . The evaluator effect: A chilling fact about usability evaluation methods . International Journal of Human-Computer Interaction , 1 : 421 – 443 . · DOI · Paper page
  13. Hertzum , M. , Jacobsen , N. E. and Molich , R. 2002 . “ Usability inspections by groups of specialists: Perceived agreement in spite of disparate observations. ” . In CHI 2002 Extended Abstracts 662 – 663 . New York, NY : ACM. .
  14. 10.1080/07370020802278205 · DOI
  15. 10.1145/1735223.1735255 · DOI · Paper page
  16. Jacobsen , N. E. , Hertzum , M. and John , B. The evaluator effect in usability tests . CHI 1998, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems . pp. 255 – 256 . New York, NY : ACM. .
  17. Jay , C. , Lunn , D. and Michailidou , E. 2008 . “ End user evaluations. ” . In Web accessibility: A foundation for research Edited by: Harper , S. and Yesilada , Y. 107 – 126 . London, , England : Springer. . · doi:10.1007/978-1-84800-050-6_8
  18. Lang , T. 2003 . Comparing website accessibility evaluation methods and learnings from usability evaluation methods Retrieved from http://www.peakusability.com.au/about-us/pdf/website_accessibility.pdf
  19. 10.1016/j.ijhcs.2008.11.002 · DOI
  20. Mankoff , J. , Fait , H. and Tran , T. Is your web page accessible? A comparative study of methods for assessing web page accessibility for the blind . CHI 2005, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems . pp. 41 – 50 . New York, NY : ACM. .
  21. Molich , R. , Bevan , N. , Curson , I. , Butler , S. , Kindlund , E. , Miller , D. and Kirakowski , J. Comparative evaluation of usability tests . Proceedings of the Usability Professionals Association Conference . pp. 189 – 200 . Washington, DC : UPA. .
  22. Nielsen , J. Finding usability problems through heuristic evaluation . CHI 1992, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems . pp. 373 – 380 . New York, NY : ACM. .
  23. Nielsen , J. Reliability of severity estimates for usability problems found by heuristic evaluation . CHI 1992: Posters and short talks of the 1992 SIGCHI Conference on Human Factors in Computing Systems . pp. 129 – 130 . New York, NY : ACM. .
  24. Nielsen , J. and Molich , R. Heuristic evaluation of user interfaces . CHI 1990, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems . pp. 249 – 256 . New York, NY : ACM. .
  25. Petrie , H. and Kheir , O. 2007 . “ The relationship between accessibility and usability of websites. ” . In Proceedings of CHI 2007 397 – 406 . New York, NY : ACM. .
  26. Rubin , J. and Chisnell , D. 2008 . Handbook of usability testing, , 2nd ed. Hoboken, NJ : Wiley. .
  27. 10.1207/s15327590ijhc0903_2 · DOI · Paper page
  28. Swets , J. A. 1996 . Signal detection theory and ROC analysis in psychology and diagnostics: Collected papers Lawrence Erlbaum Associates. .
  29. Takagi , H. , Kawanaka , S. , Kobayashi , M. , Sato , D. and Asakawa , C. Collaborative web accessibility improvement: Challenges and possibilities . Proceedings of the 11th International ACM SIGACCESS conference on Computers and Accessibility . pp. 195 – 202 . New York, NY : ACM. . doi: http://doi.acm.org/10.1145/1639642.1639677 · DOI
  30. Thatcher , J. , Burks , M. , Heilmann , C. , Henry , S. , Kirkpatrick , A. , Lauke , P. , Lawson , B. , Regan , B. , Rutter , R. , Urban , M. and Waddel , C. 2006 . Web accessibility: Web standards and regulatory compliance New York, NY : FriendsofED. .
  31. Trochim , W. M. 2006 . The research methods knowledge base, , 2nd ed. Retrieved from http://www.socialresearchmethods.net/kb/
  32. Woolrych , A. , Cockton , G. and Hindmarch , M. 2004 . “ Falsification testing for usability inspection method assessment. ” . In Proceedings of HCI 2004 137 – 140 . London, , England : Springer-Verlag. .

Cited by

See all 83

Metadata

Authors
Brajnik, G., Yesilada, Y., & Harper, S.
Venue
Human–Computer Interaction
Published
2011-08-30
Volume
26
Issue
3
Pages
246-283
ISSN
  • 0737-0024 (ISSN-L)
  • 1532-7051
OpenAlex
W1535223020
Cited by
83
References
7
FWCI
11.89
Type
journal article
Language
en
Open access
hybrid

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: The Expertise Effect on Web Accessibility Evaluation Methods. https://app.cochat.ai/papers/10.1080/07370024.2011.601670
The paper
Brajnik, G., Yesilada, Y., & Harper, S. (2011). The Expertise Effect on Web Accessibility Evaluation Methods. Human–Computer Interaction. https://doi.org/10.1080/07370024.2011.601670

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.