Number of people required for usability evaluation: the 10±2 rule
The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
Introduction Usability evaluation is essential to make sure that software products newly released are easy to use, efficient, and effective to reach goals, and satisfactory to users. For example, when a software company wants to develop and sell a new product, the company needs to evaluate usability of the new product before launching it at a market to avoid the possibility that the new product may contain usability problems, which span from cosmetic problems to severe functional problems. Three widely used methods for usability evaluation are Think Aloud (TA), Heuristic Evaluation (HE) and Cognitive Walkthrough (CW). TA method is commonly employed with a lab-based user testing, while there are variants of TA methods, including thinking out aloud at user's workplace instead of at labs. What we discuss here is the TA method that is combined with a lab-based user testing, in which test users use products while simultaneously and continuously thinking out aloud, and experimenters record users' behaviors and verbal protocols in the laboratory. HE is a usability inspection method, in which a small number of evaluators find usability problems in a user interface design by examining an interface and judging its compliance with well-known usability principles, called heuristics. CW is a theory-based method, in which evaluators evaluate every step necessary to perform a scenario-based task, and look for usability problems that would interfere with learning by exploration. These three methods have their own advantages and disadvantages. For instance, TA method provides good qualitative data from a small number of test users, but laboratory environment may influence test user's behaviors. HE is a cheap, fast and easy-to-use method, while it often finds too specific and low-priority usability problems, including even not real problems. CW helps find mismatches between users' and designers' conceptualization of a task, but it needs extensive knowledge of cognitive psychology and technical details to apply. However, even though these advantages and disadvantages show overall characteristics of three major usability evaluation methods, we cannot compare them quantitatively and see their efficiency clearly. Because one of reasons why so-called discounted methods, such as HE and CW, were developed is to save costs of usability evaluation, cost-related criteria for comparing usability evaluation are meaningful to usability practitioners as well as usability researchers. One of the most disputable issues related to cost of usability evaluation is sample size. That is, how many users or evaluators are needed to achieve a targeted usability evaluation performance, for example, 80% of overall discovery rate? The sample size of usability evaluation is known to depend on an estimate of problem discovery rate across participants. The overall discovery rate is a common quantitative measure that is used to show the effectiveness of a specific usability evaluation method in most of usability evaluation studies. It is also called overall detection rate or thoroughness measure, which is the ratio of 'the sum of unique usability problems detected by all experiment participants' against 'the number of usability problems that exist in the evaluated systems', ranging between 0 and 1. The overall discovery rates were reported more than any other criterion measure in the usability evaluation experiments and also a key component for projecting required sample size for usability evaluation study. Thus, how many test users or evaluators participate in the usability evaluation is a critical issue, considering its cost-effectiveness.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“However, a sample size of 10±2 is sufficient for discovering 80 per cent of usability issues [55].”
From A user-centred design framework for mHealth (2020) · cites this paper as [55]
Citations
Cites
Heuristic evaluation of user interfaces
Nielsen, J., & Molich, R. (1990) · Proceedings of the SIGCHI conference on Human factors in computing systems Empowering people - CHI '90 · doi:10.1145/97243.97281 · Cited by 3,533
Refining the Test Phase of Usability Evaluation: How Many Subjects Is Enough?
Virzi, R. A. (1992) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/001872089203400407 · Cited by 1,190
Estimating the number of subjects needed for a thinking aloud test
Nielsen, J. (1994) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1994.1065 · Cited by 476
Sample sizes for usability studies: Additional considerations
Lewis, J. R. (1994) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/001872089403600215 · Cited by 341
What is gained and lost when using evaluation methods other than empirical testing
Desurvire, H., Kondziela, J. M., & Atwood, M. E. (1993) · Cited by 107
Relaxing the homogeneity assumption in usability testing
Caulton, D. A. (2001) · Behaviour & Information Technology · doi:10.1080/01449290010020648 · Cited by 90
Law, L.-C., & Hvannberg, E. Þ. (2002) · Nordic Conference on Human-Computer Interaction · doi:10.1145/572020.572030 · Cited by 87
Evaluation of Procedures for Adjusting Problem-Discovery Rates Estimated From Small Samples
Lewis, J. R. (2001) · International Journal of Human-Computer Interaction · doi:10.1207/s15327590ijhc1304_06 · Cited by 72
Analysis of combinatorial user effect in international usability tests
Law, E. L., & Hvannberg, E. Þ. (2004) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/985692.985694 · Cited by 63
Novice heuristic evaluations of a complex interface
Slavković, A., & Cross, K. (1999) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/632716.632902 · Cited by 31
Cited by
Items 501–563 of 563
Hammad, F. (2017) · Nottingham ePrints (University of Nottingham)
Digitaalinen työkalu tunnetilojen seurantaan
Hallasmaa, T. (2021) · Aaltodoc (Aalto University)
Análise, design e inovação de modelos de negócios para servitização
Costa, R. M. (2017) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/t.12.2017.tde-17082017-094629
Billestrup, J. (2017) · VBN Forskningsportal (Aalborg Universitet) · doi:10.5278/vbn.phd.tech.00033
Hyötyläinen, S. (2023) · Aaltodoc (Aalto University)
UX Calculator: An Online Tool to Support User Testing
Wang, R. W. Y., Chen, S. L., Labbé, C., Frédette, M., Abdessemed, A., Courtemanche, F., Coursaris, C. K., Sénécal, S., & Léger, P. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-35702-2_7
A usability evaluation model for accessible mobile e-book applications for the visually impaired
Matraf, M. S. B. (2023) · UUM Electronic Theses and Dissertation [eTheses] (Northern University of Malaysia)
An Experiment: The Optimal Number of Participants for the Usability Testing of Mobile Apps
Zhao, X., & Lal, R. (2020) · AIS Electronic Library (AISeL)
Rogers, N. (2019) · ePrints Soton (University of Southampton)
Student-Centered Design of a Mental Health Self-Management Tool
Ramos, E. T. (2023) · OakTrust (Texas A&M University Libraries)
Bernini, S., Panzarasa, S., Quaglini, S., Costa, A., Picascia, M., Cappa, S. F., Cerami, C., Tassorelli, C., Vecchi, T., & Bottiroli, S. (2023) · doi:10.3389/fmed.2023.1129914.s001
Exploring the use of patient feedback in pharmacy consultations
Al‐Jabr, H. (2019) · UEA Digital Repository (University of East Anglia)
Frisina, C. S. (2019) · VTechWorks (Virginia Tech)
Sustainable Technology and Elderly Life
Iglesias, A. M., Morato, J., Sánchez-Cuadrado, S., & Fernández-Panadero, C. (2022) · MDPI eBooks · doi:10.3390/books978-3-0365-4270-6
Tehtävän monimutkaisuusanalyysi: Mobiilisovellus tapaustutkimus
Nieminen, S. (2022) · Aaltodoc (Aalto University)
Alvarez, A. N. (2014) · Institutional Repositories DataBase (IRDB)
To Bento or Not to Bento? A usability study of prototypes of Oria and Bento Box
Mogensen, F. C. (2020) · NVA (Nasjonalt vitenarkiv)
Schaaf, J. (2021) · Qucosa (Saxon State and University Library Dresden)
Fu, Y.-L., Dai, R., Xie, X., & Song, W. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e13624
A data transformation model for relational and non-relational data
Hasan, H. F. F. (2023) · UUM Electronic Theses and Dissertation [eTheses] (Northern University of Malaysia)
Accessibility Concepts to Include Elderly in the Digital World
Christersson, O. (2024) · Uppsala University Publications (Uppsala University)
Vigo, M., Matentzoglu, N., Jay, C., & Stevens, R. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3254910
Bui, D. D. (2022) · Institutional Repositories DataBase (IRDB)
Examining the Effect of Crossmodal Perception and Implicit Memory on Gestural Input Behaviour
Feng, F. (2019) · Queen Mary Research Online (Queen Mary University of London)
Lazarou, I., Nikolopoulos, S., Georgiadis, K., Oikonomou, V. P., Mariakaki, A., & Kompatsiaris, I. (2022) · doi:10.3389/fdgth.2022.846963.s001
Improved Presentation of Search Results on a Map
Johansson, M. (2010) · Uppsala University Publications (Uppsala University)
USABILITY EVALUATION OF AN E-LEARNING TUTORIAL USING TWO EVALUATION METHODS
Nyang'or, J. O. (2012) · Unisa Institutional Repository (University of South Africa)
Incorporating usability methods in software development
Blent, A. (2015) · DiVA (Linkoping University)
QUTLINE: Segmentación de Líneas a Partir de Distancias o Partes Equivalentes
Nieto, A. V. B., Poveda, G. E. S., & Cortés, C. M. (2015)
En studie kring utvecklingen av en webbapplikation - Presentshoppen
Bouveng, R., Gullbrandson, M., Ho, P., Lagerby, O., Lindahl, C., Lodenius, J., Ottosson, H., Ramstrand, M., & Wallgren, J. (2016) · DiVA (Linkoping University)
Mobile Communication within Healthcare Environment : A Case Study at Danderyds University Hospital
Söderström, P. (2017) · KTH Publication Database DiVA (KTH Royal Institute of Technology)
Query disambiguation approach using triple-filter
Agaie, A. I. (2017) · Universiti Putra Malaysia Institutional Repository (Universiti Putra Malaysia)
Evaluating the Navigation in Software Visualization in Augmented Reality with Speech Processing
Stock, A. (2019) · elib (German Aerospace Center)
Huang, H., Liu, L., Fu, Z., Zhang, Y., Zhang, J., Tsai, Y.-L., Guo, L., & Rau, P. P. (2021) · Mobile Information Systems · doi:10.1155/2021/9987332
Att utveckla en webbapplikation för att sälja presentböcker
Norberg, O. (2014) · DiVA (Linkoping University)
Kühl, F. L., & Papí, J. F. (2023) · Transportation research procedia · doi:10.1016/j.trpro.2023.11.761
A Preliminary Usability Evaluation for Online Annotation and Student Clustering System
Chang, M.-H., Kuo, R., Essalmi, F., Chang, M., Kumar, V., & Hsu-Yang, K. U. N. G. (2019) · International Conference on Computers in Education · doi:10.58459/icce.2019.768
Casanova, G. J. B., Pino, A. V., Erazo, O., & Ordóñez, J. A. P. (2025) · REVISTA ODIGOS · doi:10.35290/ro.v6n3.2025.1677
Användarvänligt e-handelssystem
Törnemyr, N. (2011) · DiVA (Linnaeus University)
Kühl, F. L., & Papí, J. F. (2022) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.7339415
Tegström, E. (2014) · DiVA (Södertörn University)
Nytsemisprófanir á tutor-web netnámskerfinu
Torfason, K. E. (2018) · Skemman
Dahlström, F., Eirik, F., Gudmundsson, T., Lindberg, S., Nilsson, F., Olsson, M., Svensk, H., & Sörensen, J. (2018) · DiVA (Linkoping University)
Bauer, L., & Kühnel, S. (2025) · AIS Electronic Library (AISeL)
Digitalizing Football Referee Note-Taking
Tjärnhage, A. (2023) · DiVA at Umeå University (Umeå University)
Methods of usability evaluations of mobile devices
Freeman, M., & Alshehri, F. (2012) · AIS Electronic Library (AISeL)
Designing an electricity order creation workflow for novice and expert users
Wikar, M. (2023) · DiVA at Umeå University (Umeå University)
Zamora de la Barra, N. M. (2016) · Universidad de Chile
Salas Arbaiza, C. E., Salas Arbaiza, C. E., & Salas Arbaiza, C. E. (2015) · Americanae (AECID Library)
Hermann, J. (2024) · Qucosa (Saxon State and University Library Dresden)
Käytettävyystutkimus: Verkkokaupan hallintapaneeli
Lampi, H. (2015) · Aaltodoc (Aalto University)
Apostolakis, D., Angelidis, G., Argyriou, V., Sarigiannidis, P., & Papadopoulos, G. T. (2026) · International Conference in Electronic Engineering & Information Technology (EEITE) · doi:10.1109/eeite69609.2026.11679767
Become the Beast: Exploring Human-Quadruped Locomotion for ExergamesPreprint
Ahmed, S. A., Bhatnagar, P., & Hämäläinen, P. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.15428
Bhachu, A. S. (2012) · Discovery Research Portal (University of Dundee)
Nake, L., Betke, H., Damarowsky, J., Böcker, T., Kühnel, S., & Sackmann, S. (2026) · AIS Electronic Library (AISeL)
Chavez, J. M., Cruz, A. D., Unisa, J., Cruz, R. A. D., Ual, J. R., Manzanero, J., & Torres, J. D. (2026) · Isabela State University Linker Journal of Engineering Computing and Technology · doi:10.65141/ject.v3i1.n6
Zakaria, M. H., Wei, T. Z., Sulaiman, H. A., Abdullah, M. H. L., & Maksom, Z. (2026) · International Journal of Research and Innovation in Social Science · doi:10.47772/ijriss.2026.100601429
Psoroulas, G., Randazzo, M., Salatino, C., Cagliani, L., Spinelli, F., Consentino, C., Ramella, M., Converti, R. M., & Gower, V. (2026) · JMIR Rehabilitation and Assistive Technologies · doi:10.2196/106550
Oh, Y.-J., Ha, T.-W., Yoon, S.-W., & Park, S.-H. (2026) · Physical therapy rehabilitation science · doi:10.14474/ptrs.2026.15.3.368
Metadata
- Authors
- Hwang, W., & Salvendy, G.
- Affiliations
- Soongsil University (KR)
- Purdue University West Lafayette (US)
- ORCID
- Hwang, Wonil
- Venue
- Communications of the ACM
- Published
- 2010-05
- Volume
- 53
- Issue
- 5
- Pages
- 130-133
- ISSN
- 0001-0782 (ISSN-L)
- 1557-7317
- OpenAlex
- W2016022053
- Cited by
- 410
- References
- 11
- Citations by year
- 2026: 40 · 2025: 36 · 2024: 26 · 2023: 43 · 2022: 27 · 2021: 21 · 2020: 29 · 2019: 30 · 2018: 22 · 2017: 24 · 2016: 23 · 2015: 15 · 2014: 24 · 2013: 17 · 2012: 17
- FWCI
- 31.16
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Communications of the ACM (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Keywords
- usability evaluation, sample size, think-aloud, heuristic evaluation, cognitive walkthrough, detection rate, evaluation costs, user interface design, verbal protocols, user testing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.