The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.
The Evaluator Effect: A Chilling Fact About Usability Evaluation Methods
Abstract
Computer professionals have a need for robust, easy-to-use usability evaluation methods (UEMs) to help them systematically improve the usability of computer artifacts. However, cognitive walkthrough (CW), heuristic evaluation (HE), and thinking- aloud study (TA)-3 of the most widely used UEMs-suffer from a substantial evaluator effect in that multiple evaluators evaluating the same interface with the same UEM detect markedly different sets of problems. A review of 11 studies of these 3 UEMs reveals that the evaluator effect exists for both novice and experienced evaluators, for both cosmetic and severe problems, for both problem detection and severity assessment, and for evaluations of both simple and complex systems. The average agreement between any 2 evaluators who have evaluated the same system using the same UEM ranges from 5% to 65%, and no 1 of the 3 UEMs is consistently better than the others. Although evaluator effects of this magnitude may not be surprising for a UEM as informal as HE, it is certainly notable that a substantial evaluator effect persists for evaluators who apply the strict procedure of CW or observe users thinking out loud. Hence, it is highly questionable to use a TA with 1 evaluator as an authoritative statement about what problems an interface contains. Generally, the application of the UEMs is characterized by (a) vague goal analyses leading to variability in the task scenarios, (b) vague evaluation procedures leading to anchoring, or (c) vague problem criteria leading to anything being accepted as a usability problem, or all of these. The simplest way of coping with the evaluator effect, which cannot be completely eliminated, is to involve multiple evaluators in usability evaluations.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(1993) · Cited by 9,879
Heuristic evaluation of user interfaces
Nielsen, J., & Molich, R. (1990) · Proceedings of the SIGCHI conference on Human factors in computing systems Empowering people - CHI '90 · doi:10.1145/97243.97281 · Cited by 3,533
The Usability Engineering Lifecycle
NIELSEN, J. (1993) · Usability Engineering · doi:10.1016/b978-0-08-052029-2.50007-3 · Cited by 2,657
Nielsen, J. (1995) · Conference companion on Human factors in computing systems - CHI '95 · doi:10.1145/223355.223730 · Cited by 2,620
Handbook of Human-Computer Interaction
(1988) · Elsevier eBooks · doi:10.1016/c2009-0-12113-x · Cited by 2,058
A mathematical model of the finding of usability problems
Nielsen, J., & Landauer, T. K. (1993) · Proceedings of the SIGCHI conference on Human factors in computing systems - CHI '93 · doi:10.1145/169059.169166 · Cited by 1,961
Nielsen, J. (1994) · Cited by 1,897
A Practical Guide to Usability Testing
Dumas, J. F., & Redish, J. C. (1993) · Cited by 1,335
Refining the Test Phase of Usability Evaluation: How Many Subjects Is Enough?
Virzi, R. A. (1992) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/001872089203400407 · Cited by 1,190
Finding usability problems through heuristic evaluation
Nielsen, J. (1992) · Proceedings of the SIGCHI conference on Human factors in computing systems - CHI '92 · doi:10.1145/142750.142834 · Cited by 1,060
Cited by
Beyond Conformance: The Role of Accessibility Evaluation Methods
Brajnik, G. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-85200-1_9 · Cited by 114
Web Accessibility Testing: When the Method Is the Culprit
Brajnik, G. (2006) · Lecture Notes in Computer Science · doi:10.1007/11788713_24 · Cited by 87
Human-AI Collaboration for UX Evaluation: Effects of Explanation and Synchronization
Fan, M., Yang, X., Yu, T., Liao, Q. V., & Zhao, J. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3512943 · Cited by 81
How much does expertise matter?: a barrier walkthrough study with experts and non-experts
Yesilada, Y., Brajnik, G., & Harper, S. (2009) · Proceedings of the 11th international ACM SIGACCESS conference on Computers and accessibility · doi:10.1145/1639642.1639678 · Cited by 53
Estimación del tamaño de la muestra para pruebas de usabilidad
Cazañas, A., De San Miguel, A., & Parra, E. (2017) · Enfoque UTE · doi:10.29019/enfoqueute.v8n1.126 · Cited by 52
Metadata
- Authors
- Hertzum, M., & Jacobsen, N. E.
- Venue
- International Journal of Human-Computer Interaction
- Publisher
- Informa UK Limited
- Published
- 2001-12
- Volume
- 13
- Issue
- 4
- Pages
- 421-443
- ISSN
- 1044-7318 (ISSN-L)
- 1532-7590
- OpenAlex
- W2101451656
- Cited by
- 419
- References
- 34
- FWCI
- 10.30
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The Evaluator Effect: A Chilling Fact About Usability Evaluation Methods. https://app.cochat.ai/papers/10.1207/s15327590ijhc1304_05
- The paper
- Hertzum, M., & Jacobsen, N. E. (2001). The Evaluator Effect: A Chilling Fact About Usability Evaluation Methods. International Journal of Human-Computer Interaction. https://doi.org/10.1207/s15327590ijhc1304_05
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.