The DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Sample Sizes for Usability Studies: Additional Considerations
Abstract
Recently, Virzi (1992) presented data that support three claims regarding sample sizes for usability studies: (1) observing four or five participants will allow a usability practitioner to discover 80% of a product's usability problems, (2) observing additional participants will reveal fewer and fewer new usability problems, and (3) more severe usability problems are easier to detect with the first few participants. Results from an independent usability study clearly support the second claim, partially support the first, but fail to support the third. Problem discovery shows diminishing returns as a function of sample size. Observing four to five participants will uncover about 80% of a product's usability problems as long as the average likelihood of problem detection ranges between 0.32 and 0.42, as in Virzi. If the average likelihood of problem detection is lower, then a practitioner will need to observe more than five participants to discover 80% of the problems. Using behavioral categories for problem severity (or impact), these data showed no correlation between problem severity (impact) and rate of discovery. The data provided evidence that the binomial probability formula may provide a good model for predicting problem discovery curves, given an estimate of the average likelihood of problem detection. Finally, data from economic simulations that estimated return on investment (ROI) under a variety of settings showed that only the average likelihood of problem detection strongly influenced the range of sample sizes for maximum ROI.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“While this number of participants was sufficient to conduct a usability test, from a scientific perspective [26]”
Citations
Cites
Nonparametric Statistical Methods
(1999) · Technometrics · doi:10.1080/00401706.1999.10485954 · Cited by 6,090
Software Engineering Economics
Boehm, B. W. (2001) · Pioneers and Their Contributions to Software Engineering · doi:10.1007/978-3-642-48354-7_5 · Cited by 5,981
Refining the Test Phase of Usability Evaluation: How Many Subjects Is Enough?
Virzi, R. A. (1992) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/001872089203400407 · Cited by 1,190
Nonparametric Statistical Methods
Gibbons, J. D. (1974) · Technometrics · doi:10.1080/00401706.1974.10489221 · Cited by 1,115
How Many Subjects?: Statistical Power Analysis in Research
Kraemer, H. C., & Blasey, C. (2016) · SAGE Publications Ltd eBooks · doi:10.4135/9781483398761 · Cited by 910
Design rules based on analyses of human error
Norman, D. A. (1983) · Communications of the ACM · doi:10.1145/2163.358092 · Cited by 460
Software engineering economics
Al Hady, M. M. (2007) · Cybrarians Journal · doi:10.70000/cj.2007.14.426 · Cited by 356
Gould, J. D. (1987) · Human–Computer Interaction–INTERACT '87 · doi:10.1016/b978-0-444-70304-0.50010-8 · Cited by 330
Comparison of empirical testing and walkthrough methods in user interface evaluation
Karat, C.-M., Campbell, R., & Fiegel, T. (1992) · Proceedings of the SIGCHI conference on Human factors in computing systems - CHI '92 · doi:10.1145/142750.142873 · Cited by 300
How Many Subjects? Statistical Power Analysis in Research.
Anderson, K. L., Kraemer, H. C., & Thiemann, S. (1988) · Contemporary Sociology · doi:10.2307/2073648 · Cited by 203
Research works printed but not linked: 1
- Nunnally, J. C., 1978, Psychometric theory.
Cited by
Damaged Merchandise? A Review of Experiments That Compare Usability Evaluation Methods
Gray, W. D., & Salzman, M. C. (1998) · Human–Computer Interaction · doi:10.1207/s15327051hci1303_2 · Cited by 488
van den Haak, M., De Jong, M., & Jan Schellens, P. (2003) · Behaviour & Information Technology · doi:10.1080/0044929031000 · Cited by 486
The Evaluator Effect: A Chilling Fact About Usability Evaluation Methods
Hertzum, M., & Jacobsen, N. E. (2001) · International Journal of Human-Computer Interaction · doi:10.1207/s15327590ijhc1304_05 · Cited by 419
Number of people required for usability evaluation: the 10±2 rule
Hwang, W., & Salvendy, G. (2010) · Communications of the ACM · doi:10.1145/1735223.1735255 · Cited by 410
Criteria For Evaluating Usability Evaluation Methods
Hartson, H. R., Andre, T. S., & Williges, R. C. (2001) · International Journal of Human-Computer Interaction · doi:10.1207/s15327590ijhc1304_03 · Cited by 406
Metadata
- Authors
- Lewis, J. R.
- Affiliations
- IBM (United States) (US)
- Corresponding author
- Lewis, James R.
- ORCID
- Lewis, James R.
- Venue
- Human Factors: The Journal of the Human Factors and Ergonomics Society
- Publisher
- SAGE Publications
- Published
- 1994-06
- Volume
- 36
- Issue
- 2
- Pages
- 368-378
- ISSN
- 0018-7208 (ISSN-L)
- 1547-8181
- OpenAlex
- W1635961553
- Semantic Scholar
- 42456ab83078584e750f12cde130f97eb1d2bdb4
- Cited by
- 341
- References
- 23
- Citations by year
- 2026: 14 · 2025: 16 · 2024: 23 · 2023: 16 · 2022: 13 · 2021: 15 · 2020: 12 · 2019: 13 · 2018: 17 · 2017: 13 · 2016: 18 · 2015: 12 · 2014: 10 · 2013: 16 · 2012: 13
- FWCI
- 5.59
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Human Factors The Journal of the Human Factors and Ergonomics Society (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Yale University
- Uniformed Services University of the Health Sciences
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Physical Sciences › Computer Science › Human-Computer Interaction › Persona Design and Applications
- Keywords
- sample size, usability testing, problem finding, usability problems, problem severity, return on investment, economic simulation, usability evaluation
- MeSH terms
- Equipment Design, Evaluation Studies as Topic, Humans, Ergonomics, Monte Carlo Method, Office Automation, Probability, Research Design, Sampling Studies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Sample Sizes for Usability Studies: Additional Considerations. https://app.cochat.ai/papers/10.1177/001872089403600215
- The paper
- Lewis, J. R. (1994). Sample Sizes for Usability Studies: Additional Considerations. Human Factors: The Journal of the Human Factors and Ergonomics Society. https://doi.org/10.1177/001872089403600215
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.