The DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
Criteria For Evaluating Usability Evaluation Methods
Abstract
The current variety of alternative approaches to usability evaluation methods (UEMs) designed to assess and improve usability in software systems is offset by a general lack of understanding of the capabilities and limitations of each. Practitioners need to know which methods are more effective and in what ways and for what purposes. However, UEMs cannot be evaluated and compared reliably because of the lack of standard criteria for comparison. In this article, we present a practical discussion of factors, comparison criteria, and UEM performance measures useful in studies comparing UEMs. In demonstrating the importance of developing appropriate UEM evaluation criteria, we offer operational definitions and possible measures of UEM performance. We highlight specific challenges that researchers and practitioners face in comparing UEMs and provide a point of departure for further discussion and refinement of the principles and techniques used to approach UEM evaluation and comparison.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Statistical principles in experimental design.
Winer, B. J. (1962) · McGraw-Hill Book Company · doi:10.1037/11774-000 · Cited by 26,755
Foundations of statistical natural language processing
Manning, C. D., & Schütze, H. (1999) · Cited by 9,816
Introduction to Modern Information Retrieval
Salton, G., & McGill, M. J. (1983) · Virtual Defense Library (Ministerio de Defensa) · Cited by 9,680
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733
The critical incident technique.
Flanagan, J. C. (1954) · Psychological Bulletin · doi:10.1037/h0061470 · Cited by 8,191
Design and Analysis: A Researcher's Handbook
Beaton, A. E., & Keppel, G. (1975) · American Educational Research Journal · doi:10.2307/1162588 · Cited by 5,822
(2007) · Encyclopedia of Industrial and Organizational Psychology · doi:10.4135/9781412952651.n55 · Cited by 5,287
Protocol Analysis: Verbal Reports as Data
Venkatesan, M., Ericsson, K. A., & Simon, H. A. (1986) · Journal of Marketing Research · doi:10.2307/3151491 · Cited by 4,347
Statistical Principles in Experimental Design
Green, P. E., Winer, B. J., Brown, D. R., & Michels, K. M. (1992) · Journal of Marketing Research · doi:10.2307/3172747 · Cited by 4,333
Cited by
Usability, user experience and accessibility: towards an integrative model
Sauer, J., Sonderegger, A., & Schmutz, S. (2020) · Ergonomics · doi:10.1080/00140139.2020.1774080 · Cited by 195
Beyond Conformance: The Role of Accessibility Evaluation Methods
Brajnik, G. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-85200-1_9 · Cited by 114
Characterising user capabilities to support inclusive design evaluation
Persad, U., Langdon, P., & Clarkson, J. (2007) · Universal Access in the Information Society · doi:10.1007/s10209-007-0083-y · Cited by 113
Silva, A., Schrum, M., Hedlund-Botti, E., Gopalan, N., & Gombolay, M. (2023) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2022.2101698 · Cited by 104
Adolescent Online Safety: Moving Beyond Formative Evaluations to Designing Solutions for the Future
Pinter, A. T., Wisniewski, P. J., Xu, H., Rosson, M. B., & Caroll, J. M. (2017) · Proceedings of the 2017 Conference on Interaction Design and Children · doi:10.1145/3078072.3079722 · Cited by 79
Metadata
- Authors
- Hartson, H. R., Andre, T. S., & Williges, R. C.
- Venue
- International Journal of Human-Computer Interaction
- Publisher
- Informa UK Limited
- Published
- 2001-12
- Volume
- 13
- Issue
- 4
- Pages
- 373-410
- ISSN
- 1044-7318 (ISSN-L)
- 1532-7590
- OpenAlex
- W2151854647
- Cited by
- 406
- References
- 60
- FWCI
- 9.32
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Criteria For Evaluating Usability Evaluation Methods. https://app.cochat.ai/papers/10.1207/s15327590ijhc1304_03
- The paper
- Hartson, H. R., Andre, T. S., & Williges, R. C. (2001). Criteria For Evaluating Usability Evaluation Methods. International Journal of Human-Computer Interaction. https://doi.org/10.1207/s15327590ijhc1304_03
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.