Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.

Sample Sizes for Usability Studies: Additional Considerations

Lewis, James R. · 1994 · Human Factors: The Journal of the Human Factors and Ergonomics Society

Abstract

Recently, Virzi (1992) presented data that support three claims regarding sample sizes for usability studies: (1) observing four or five participants will allow a usability practitioner to discover 80% of a product's usability problems, (2) observing additional participants will reveal fewer and fewer new usability problems, and (3) more severe usability problems are easier to detect with the first few participants. Results from an independent usability study clearly support the second claim, partially support the first, but fail to support the third. Problem discovery shows diminishing returns as a function of sample size. Observing four to five participants will uncover about 80% of a product's usability problems as long as the average likelihood of problem detection ranges between 0.32 and 0.42, as in Virzi. If the average likelihood of problem detection is lower, then a practitioner will need to observe more than five participants to discover 80% of the problems. Using behavioral categories for problem severity (or impact), these data showed no correlation between problem severity (impact) and rate of discovery. The data provided evidence that the binomial probability formula may provide a good model for predicting problem discovery curves, given an estimate of the average likelihood of problem detection. Finally, data from economic simulations that estimated return on investment (ROI) under a variety of settings showed that only the average likelihood of problem detection strongly influenced the range of sample sizes for maximum ROI.

Overview

Integrity signals

Registry

Crossref · 10.1177/001872089403600215

Cites

23 of 23 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 23 references

Research works printed but not linked: 1

  1. Nunnally, J. C., 1978, Psychometric theory.

Cited by

See all 368

Metadata

Authors
Lewis, J. R.
Affiliations
IBM (United States) (US)
Corresponding author
Lewis, James R.
Venue
Human Factors: The Journal of the Human Factors and Ergonomics Society
Published
1994-06
Volume
36
Issue
2
Pages
368-378
ISSN
  • 0018-7208 (ISSN-L)
  • 1547-8181
OpenAlex
W1635961553
Cited by
341
References
23
Citations by year
2026: 14 · 2025: 16 · 2024: 23 · 2023: 16 · 2022: 13 · 2021: 15 · 2020: 12 · 2019: 13 · 2018: 17 · 2017: 13 · 2016: 18 · 2015: 12 · 2014: 10 · 2013: 16 · 2012: 13
FWCI
5.59
Type
journal article
Language
en
Open access
closed
Licence
journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
Locations
  • Human Factors The Journal of the Human Factors and Ergonomics Society (publishedVersion)
  • PubMed (publishedVersion)
Funders
  • Yale University
  • Uniformed Services University of the Health Sciences
Topics
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
  • Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
  • Physical Sciences › Computer Science › Human-Computer Interaction › Persona Design and Applications
Keywords
sample size, usability testing, problem finding, usability problems, problem severity, return on investment, economic simulation, usability evaluation
MeSH terms
Equipment Design, Evaluation Studies as Topic, Humans, Ergonomics, Monte Carlo Method, Office Automation, Probability, Research Design, Sampling Studies

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Sample Sizes for Usability Studies: Additional Considerations. https://app.cochat.ai/papers/10.1177/001872089403600215
The paper
Lewis, J. R. (1994). Sample Sizes for Usability Studies: Additional Considerations. Human Factors: The Journal of the Human Factors and Ergonomics Society. https://doi.org/10.1177/001872089403600215

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.