Sample size in usability studies
Open accessAbstract
Magic numbers are strictly hocus-pocus, so usability studies must test many more subjects than is usually assumed.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A total of 25 neonates (n = 798 data entries) were recruited to assess the first generation of the device.
“Using Schmettow’s methodology for sample size estimation22, a total of 25 neonates (n = 798 data entries) were recruited to assess the first generation of the device.”
From Soft, all-in-one, nanomembrane wearable system for advancing neonatal health monitoring in Ethiopia (2025) · cites this paper as [22]
Sample size was estimated using Schmettow's methodology.
“Using Schmettow’s methodology for sample size estimation22, a total of 25 neonates (n = 798 data entries) were recruited to assess the first generation of the device.”
From Soft, all-in-one, nanomembrane wearable system for advancing neonatal health monitoring in Ethiopia (2025) · cites this paper as [22]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Multimodel Inference: Understanding AIC and BIC in Model Selection
Burnham, K. P., & Anderson, D. R. (2004) · Sociological Methods & Research · doi:10.1177/0049124104268644 · Cited by 11,648 · see citing works
A mathematical model of the finding of usability problems
Nielsen, J., & Landauer, T. K. (1993) · Proceedings of the SIGCHI conference on Human factors in computing systems - CHI '93 · doi:10.1145/169059.169166 · Cited by 1,961 · see citing works
Refining the Test Phase of Usability Evaluation: How Many Subjects Is Enough?
Virzi, R. A. (1992) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1177/001872089203400407 · Cited by 1,237 · see citing works
International encyclopedia of ergonomics and human factors
(2002) · Choice Reviews Online · doi:10.5860/choice.39-4939 · Cited by 790 · see citing works
Number of people required for usability evaluation: the 10±2 rule
Hwang, W., & Salvendy, G. (2010) · Communications of the ACM · doi:10.1145/1735223.1735255 · Cited by 410 · see citing works
What do usability evaluators do in practice?
Nørgaard, M., & Hornbæk, K. (2006) · Designing Interactive Systems Conference · doi:10.1145/1142405.1142439 · Cited by 230 · see citing works
Dorazio, R. M., & Royle, J. A. (2003) · Biometrics · doi:10.1111/1541-0420.00042 · Cited by 226 · see citing works
Determining Usability Test Sample Size
Nielsen, J., Lewis, J. D., & Turner, C. (2006) · CRC Press eBooks · doi:10.1201/9780849375477.ch597 · Cited by 194 · see citing works
A comprehensive evaluation of capture-recapture models for estimating software defect content
Briand, L., Emam, K. E., Freimut, B., et al. (2000) · IEEE Transactions on Software Engineering · doi:10.1109/32.852741 · Cited by 168 · see citing works
Tracking the effectiveness of usability evaluation methods
John, B. E., & Marks, S. J. (1997) · Behaviour and Information Technology · doi:10.1080/014492997119789 · Cited by 165 · see citing works
Spool, J. M., & Schroeder, W. (2001) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/634067.634236 · Cited by 162 · see citing works
Comparative usability evaluation (CUE-4)
Molich, R., & Dumas, J. S. (2006) · Behaviour and Information Technology · doi:10.1080/01449290600959062 · Cited by 112 · see citing works
Why and when five test users aren’t enough
Woolrych, A., & Cockton, G. (2001) · Cited by 99 · see citing works
Relaxing the homogeneity assumption in usability testing
Caulton, D. A. (2001) · Behaviour and Information Technology · doi:10.1080/01449290010020648 · Cited by 90 · see citing works
Evaluation of Procedures for Adjusting Problem-Discovery Rates Estimated From Small Samples
Lewis, J. R. (2001) · International Journal of Human-Computer Interaction · doi:10.1207/s15327590ijhc1304_06 · Cited by 72 · see citing works
Analysis of combinatorial user effect in international usability tests
Law, E. L., & Hvannberg, E. Þ. (2004) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/985692.985694 · Cited by 63 · see citing works
Comparison of techniques for matching of usability problem descriptions
Hornbæk, K., & Frøkjær, E. (2008) · Interacting with Computers · doi:10.1016/j.intcom.2008.08.005 · Cited by 45 · see citing works
Heterogeneity in the Usability Evaluation Process
Schmettow, M. (2008) · Electronic workshops in computing · doi:10.14236/ewic/hci2008.9 · Cited by 22 · see citing works
Self-citationTesting Small System Customer Set-Up
Lewis, J. R. (1982) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/154193128202600810 · Cited by 21 · see citing works
Evaluation of capture-recapture models for estimating the abundance of naturally-occurring defects
Walia, G., & Carver, J. C. (2008) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/1414004.1414031 · Cited by 17 · see citing works
Controlling the usability evaluation process under varying defect visibility
Schmettow, M. (2009) · University of Twente Research Information · doi:10.5555/1671011.1671034 · Cited by 11 · see citing works
Self-citationTesting small system customer set-up
Lewis, J. R. (1982) · PsycEXTRA Dataset · doi:10.1037/e573992012-010 · Cited by 7 · see citing works
Linden, G., Bernstein, M. S., & Poole, E. S. (2010) · Communications of the ACM · doi:10.1145/1787234.1787238 · Cited by 2 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- molich, r. how many participants needed to test usability? Commun. ACM 53, 8 (aug. 2010), 7.
- Karwowski, ed. crc Press, boca raton, Fl, 2006, 3084-3088.
Cited by
Sorted by citations
A usability and safety analysis of electronic health records: a multi-center study
Ratwani, R. M., Savage, E., Will, A., Arnold, R., Khairat, S., Miller, K., Fairbanks, R. J., Hodgkins, M., & Hettinger, A. Z. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy088 · Cited by 180 · see citing works
Soft, all-in-one, nanomembrane wearable system for advancing neonatal health monitoring in Ethiopia
Zhou, L., Joseph, M., Lee, Y. J., Yadav, D., Nayak, L., Woodall, J., Matthews, J., Soltis, I., Mekuria, F. M., Yeshanew, K. H., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01974-8 · Cited by 3 · see citing works
- Authors
- Schmettow, M.
- Affiliations
- University of Twente (NL)
- Corresponding author
- Schmettow, Martin
- ORCID
- Schmettow, Martin
- Venue
- Communications of the ACM
- Published
- 2012-04
- Volume
- 55
- Issue
- 4
- Pages
- 64-70
- ISSN
- 0001-0782 (ISSN-L)
- 1557-7317
- OpenAlex
- W1993640684
- Cited by
- 123
- References
- 23
- Citations by year
- 2026: 7 · 2025: 10 · 2024: 10 · 2023: 11 · 2022: 4 · 2021: 7 · 2020: 13 · 2019: 6 · 2018: 9 · 2017: 12 · 2016: 9 · 2015: 6 · 2014: 11 · 2013: 5 · 2012: 3
- FWCI
- 14.77
- Type
- journal article
- Language
- en
- Open access
- green
- research.utwente.nl
- Locations
- Communications of the ACM (publishedVersion)
- University of Twente Research Information (submittedVersion)
- University of Twente Research Information (submittedVersion, other-oa)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Physical Sciences › Computer Science › Human-Computer Interaction › Persona Design and Applications
- Keywords
- sample size, usability testing, magic numbers, statistical significance, user testing, sample size determination
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
