Reviewing and Extending the Five-User Assumption: A Grounded Procedure for Interaction Evaluation
Open accessThe DOI matches its Crossref record; 6 of the 6 references it cites were found in a registry or an index.
Abstract
The debate concerning how many participants represents a sufficient number for interaction testing is well-established and long-running, with prominent contributions arguing that five users provide a good benchmark when seeking to discover interaction problems. We argue that adoption of five users in this context is often done with little understanding of the basis for, or implications of, the decision. We present an analysis of relevant research to clarify the meaning of the five-user assumption and to examine the way in which the original research that suggested it has been applied. This includes its blind adoption and application in some studies, and complaints about its inadequacies in others. We argue that the five-user assumption is often misunderstood, not only in the field of Human-Computer Interaction, but also in fields such as medical device design, or in business and information applications. The analysis that we present allows us to define a systematic approach for monitoring the sample discovery likelihood, in formative and summative evaluations, and for gathering information in order to make critical decisions during the interaction testing, while respecting the aim of the evaluation and allotted budget. This approach -- which we call the Grounded Procedure-- is introduced and its value argued.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Nielsen, J. (1994) · Conference companion on Human factors in computing systems - CHI '94 · doi:10.1145/259963.260531 · Cited by 2,759
A mathematical model of the finding of usability problems
Nielsen, J., & Landauer, T. K. (1993) · Proceedings of the SIGCHI conference on Human factors in computing systems - CHI '93 · doi:10.1145/169059.169166 · Cited by 1,961
Beyond the five-user assumption: benefits of increased sample sizes in usability testing
Faulkner, L. (2003) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03195514 · Cited by 1,053 · cited 1×
“Faulkner concluded that “the risk of relying on any one set of five users was that nearly half of the identified problems could have been missed; however, each addition of users markedly increased the odds of finding the problems” [Faulkner 2003].”
BackgroundSample sizes for usability studies: Additional considerations
Lewis, J. R. (1994) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/001872089403600215 · Cited by 341
Sample size in usability studies
Schmettow, M. (2012) · Communications of the ACM · doi:10.1145/2133806.2133824 · Cited by 123
Sauro, J., & Lewis, J. R. (2012) · Quantifying the User Experience · doi:10.1016/b978-0-12-384968-7.00002-3 · Cited by 117 · cited 2×
“While the reduction of assessment costs remains a key outcome associated with p -value estimation, the debate has become increasingly focused on the reliability of the data gathered when using a small sample users [Borsci et al. 2012; Nielsen 2012; Sauro and Lewis 2012; Schmettow 2012].”
Method
Cited by
Vasileiou, K., Barnett, J., Thorpe, S., & Young, T. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0594-7 · Cited by 3,115
Self-citationLocal Standards for Sample Size at CHI
Caine, K. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858498 · Cited by 657
Usability: Lessons Learned … and Yet to Be Learned
Lewis, J. R. (2014) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2014.930311 · Cited by 311
Atwal, A., Money, A., & Harvey, M. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.3723 · Cited by 52
“This number of participants is in excess of the recommended threshold of five participants typically required to carry out effective think-aloud interaction and usability testing [40,41].”
BackgroundResult
Metadata
- Authors
- Borsci, S., Macredie, R. D., Barnett, J., Martin, J., Kuljis, J., & Young, T.
- Venue
- ACM Transactions on Computer-Human Interaction
- Published
- 2013-11
- Volume
- 20
- Issue
- 5
- Pages
- 1-23
- ISSN
- 1073-0516 (ISSN-L)
- 1557-7325
- OpenAlex
- W2018069482
- Cited by
- 51
- References
- 6
- FWCI
- 10.77
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.