The Evaluator Effect: A Chilling Fact About Usability Evaluation Methods
The DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.
Abstract
Computer professionals have a need for robust, easy-to-use usability evaluation methods (UEMs) to help them systematically improve the usability of computer artifacts. However, cognitive walkthrough (CW), heuristic evaluation (HE), and thinking- aloud study (TA)-3 of the most widely used UEMs-suffer from a substantial evaluator effect in that multiple evaluators evaluating the same interface with the same UEM detect markedly different sets of problems. A review of 11 studies of these 3 UEMs reveals that the evaluator effect exists for both novice and experienced evaluators, for both cosmetic and severe problems, for both problem detection and severity assessment, and for evaluations of both simple and complex systems. The average agreement between any 2 evaluators who have evaluated the same system using the same UEM ranges from 5% to 65%, and no 1 of the 3 UEMs is consistently better than the others. Although evaluator effects of this magnitude may not be surprising for a UEM as informal as HE, it is certainly notable that a substantial evaluator effect persists for evaluators who apply the strict procedure of CW or observe users thinking out loud. Hence, it is highly questionable to use a TA with 1 evaluator as an authoritative statement about what problems an interface contains. Generally, the application of the UEMs is characterized by (a) vague goal analyses leading to variability in the task scenarios, (b) vague evaluation procedures leading to anchoring, or (c) vague problem criteria leading to anything being accepted as a usability problem, or all of these. The simplest way of coping with the evaluator effect, which cannot be completely eliminated, is to involve multiple evaluators in usability evaluations.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Usability Engineering Lifecycle
NIELSEN, J. (1993) · Usability Engineering · doi:10.1016/b978-0-08-052029-2.50007-3 · Cited by 2,657
A mathematical model of the finding of usability problems
Nielsen, J., & Landauer, T. K. (1993) · Proceedings of the SIGCHI conference on Human factors in computing systems - CHI '93 · doi:10.1145/169059.169166 · Cited by 1,961
Sample sizes for usability studies: Additional considerations
Lewis, J. R. (1994) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/001872089403600215 · Cited by 341
Interobserver variability in dermatopathology
Cramer, S. F. (1997) · Archives of Dermatology · doi:10.1001/archderm.1997.03890440117019 · Cited by 19
Cited by
The mHealth App Usability Questionnaire (MAUQ): Development and Validation Study
Zhou, L., Bao, J., Setiawan, I. M. A., Saptono, A., & Parmanto, B. (2019) · JMIR mHealth and uHealth · doi:10.2196/11500 · Cited by 675
Towards automated web accessibility evaluation: a comparative study
Lecturer, K. P. I. O. T. T. U. N. A. A. M. S. O., & Abduganiev, S. G. (2017) · International Journal of Information Technology and Computer Science · doi:10.5815/ijitcs.2017.09.03 · Cited by 37
(2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-41264-1 · Cited by 22
Glaser, S. L. C., Fraterman, I., van Brummelen, N., Tibollo, V., Del Campo, L. M., Mallo, H., Wilgenhof, S., Wilk, S., Gisko, V., Khadakou, V., et al. (2025) · JMIR Formative Research · doi:10.2196/57659 · Cited by 5
Criticmate: Stagewise Human–AI Co-Critique in Single-Screen UI Evaluation
Ko, J., Choi, J., Morales, C., Kim, D., & Ko, M. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790929 · Cited by 1
“Because evaluators bring different tastes and priorities, reviews of the same interface often diverge, a phenomenon known as the “evaluator effect” [36, 39, 45], which makes conflicting feedback difficult to reconcile.”
Metadata
- Authors
- Hertzum, M., & Jacobsen, N. E.
- Venue
- International Journal of Human-Computer Interaction
- Publisher
- Informa UK Limited
- Published
- 2003-02
- Volume
- 15
- Issue
- 1
- Pages
- 183-204
- ISSN
- 1044-7318 (ISSN-L)
- 1532-7590
- OpenAlex
- W4233642407
- Cited by
- 118
- References
- 4
- FWCI
- 3.70
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The Evaluator Effect: A Chilling Fact About Usability Evaluation Methods. https://app.cochat.ai/papers/10.1207/s15327590ijhc1501_14
- The paper
- Hertzum, M., & Jacobsen, N. E. (2003). The Evaluator Effect: A Chilling Fact About Usability Evaluation Methods. International Journal of Human-Computer Interaction. https://doi.org/10.1207/s15327590ijhc1501_14
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.