The DOI matches its Crossref record; 8 of the 9 references it cites were found in a registry or an index.
Beyond the five-user assumption: Benefits of increased sample sizes in usability testing
Open accessAbstract
It is widely assumed that 5 participants suffice for usability testing. In this study, 60 users were tested and random sets of 5 or more were sampled from the whole, to demonstrate the risks of using only 5 participants and the benefits of using more. Some of the randomly selected sets of 5 participants found 99% of the problems; other sets found only 55%. With 10 users, the lowest percentage of problems revealed by any one set was increased to 80%, and with 20 users, to 95%.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Purposive sampling was used to select 10 individuals 43 with at least 50% of the following characteristics: 65þ years old, individuals of diverse race/ethnicity, male, education less than college, and taking at least 3 medications.”
From A chatbot for hypertension self-management support: user-centered design, development, and usability testing (2023) · cites this paper as [43]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Prior research suggests that our sample size is adequate for usability testing, 43”
From A chatbot for hypertension self-management support: user-centered design, development, and usability testing (2023) · cites this paper as [43]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Ten participants were chosen at each stage of the recruitment process—the mean percentage of problems identified in usability studies rises from 85% to 95% when the number of participants increases from 5 to 10 [36].”
From A user-centred design framework for mHealth (2020) · cites this paper as [36]
The cited source says what the claim attributes to it.
In the citing paper
“Our sample size was chosen in line with prior work around formative testing for usability [19,43]: our goal is to explore the frst,”
From Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts (2023) · cites this paper as [19]
In the citing paper
“This may be reasonable, given that simulations suggest that 10 users will identify a minimum of 80 % of the problems with the technology during usability testing, and 20 users will identify 95 % of the problems [42].”
From Maximising the impact of qualitative research in feasibility studies for randomised controlled trials: guidance for researchers (2015) · cites this paper as [42]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(1993) · Cited by 9,879
(1993) · Elsevier eBooks · doi:10.1016/c2009-0-21512-1 · Cited by 5,552
Managing the Risks of Organizational Accidents
REASON, J. T. (2016) · Routledge eBooks · doi:10.4324/9781315543543 · Cited by 4,523
A mathematical model of the finding of usability problems
Nielsen, J., & Landauer, T. K. (1993) · Proceedings of the SIGCHI conference on Human factors in computing systems - CHI '93 · doi:10.1145/169059.169166 · Cited by 1,961
Refining the Test Phase of Usability Evaluation: How Many Subjects Is Enough?
Virzi, R. A. (1992) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1177/001872089203400407 · Cited by 1,269
Spool, J. M., & Schroeder, W. (2001) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/634067.634236 · Cited by 162
Why and when five test users aren’t enough
Woolrych, A., & Cockton, G. (2001) · Cited by 99
How many users does it take to change a Web site?
Hudson, W. (2001) · ACM SIGCHI Bulletin · doi:10.1145/967222.967230 · Cited by 9
Flagged first, then by citations
1 of the 9 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- Grosvenor, L. (1999).Software usability: Challenging the myths and assumptions in an emerging field. Unpublished master’s thesis, University of Texas, Austin.
- Wentworth, R. J. (1996, March).Group chairman’s factual report (American Airlines Flight 965). Washington, DC: National Transportation Safety Board, Office of Aviation Safety.
Cited by
Sorted by citations
Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts
Zamfirescu-Pereira, J. D., Wong, R. Y., Hartmann, B., & Yang, Q. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581388 · Cited by 1,274
“Our sample size was chosen in line with prior work around formative testing for usability [19, 43]: our goal is to explore the frst, “easy-to-fnd” problems users encounter when engaged in our task, and our experience with a pilot user suggested that problem discoverability was likely to be high.”
MethodA user-centered model for designing consumer mobile health (mHealth) applications (apps)
Schnall, R., Rojas, M., Bakken, S., Brown, W., Carballo-Dieguez, A., Carry, M., Gelaude, D., Mosley, J. P., & Travers, J. (2016) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2016.02.002 · Cited by 583
Biddle, R., Chiasson, S., & van Oorschot, P. C. (2012) · ACM Computing Surveys · doi:10.1145/2333112.2333114 · Cited by 577
O’Cathain, A., Hoddinott, P., Lewin, S., Thomas, K. J., Young, B., Adamson, J., Jansen, Y. J., Mills, N., Moore, G., & Donovan, J. L. (2015) · Pilot and Feasibility Studies · doi:10.1186/s40814-015-0026-y · Cited by 458
Guo, C., Ashrafian, H., Ghafur, S., Fontana, G., Gardner, C., & Prime, M. S. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00314-2 · Cited by 345
Metadata
- Authors
- Faulkner, L.
- Affiliations
- Applied Research Laboratories, The University of Texas at Austin (US)
- Corresponding author
- Faulkner, Laura
- Venue
- Behavior Research Methods, Instruments, & Computers
- Published
- 2003-08
- Volume
- 35
- Issue
- 3
- Pages
- 379-383
- ISSN
- 0743-3808 (ISSN-L)
- 1532-5970
- OpenAlex
- W2088028234
- Semantic Scholar
- 7369703e67a6812b4476fc95f8085c4d20ebe2ff
- Cited by
- 1,053
- References
- 9
- Citations by year
- 2026: 95 · 2025: 83 · 2024: 98 · 2023: 83 · 2022: 83 · 2021: 92 · 2020: 81 · 2019: 59 · 2018: 61 · 2017: 51 · 2016: 46 · 2015: 39 · 2014: 33 · 2013: 33 · 2012: 25
- FWCI
- 8.34
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- www.springer.com/tdm (tdm)
- Locations
- Behavior Research Methods, Instruments, & Computers (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Human-Computer Interaction › Persona Design and Applications
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Keywords
- usability testing, sample size, number of participants, random sampling, problem identification
- MeSH terms
- Consumer Behavior, Data Collection, Humans, Product Surveillance, Postmarketing, Software, Task Performance and Analysis, User-Computer Interface, Sample Size, Internet
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
