Number of people required for usability evaluation: the 10±2 rule
The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
Introduction Usability evaluation is essential to make sure that software products newly released are easy to use, efficient, and effective to reach goals, and satisfactory to users. For example, when a software company wants to develop and sell a new product, the company needs to evaluate usability of the new product before launching it at a market to avoid the possibility that the new product may contain usability problems, which span from cosmetic problems to severe functional problems. Three widely used methods for usability evaluation are Think Aloud (TA), Heuristic Evaluation (HE) and Cognitive Walkthrough (CW). TA method is commonly employed with a lab-based user testing, while there are variants of TA methods, including thinking out aloud at user's workplace instead of at labs. What we discuss here is the TA method that is combined with a lab-based user testing, in which test users use products while simultaneously and continuously thinking out aloud, and experimenters record users' behaviors and verbal protocols in the laboratory. HE is a usability inspection method, in which a small number of evaluators find usability problems in a user interface design by examining an interface and judging its compliance with well-known usability principles, called heuristics. CW is a theory-based method, in which evaluators evaluate every step necessary to perform a scenario-based task, and look for usability problems that would interfere with learning by exploration. These three methods have their own advantages and disadvantages. For instance, TA method provides good qualitative data from a small number of test users, but laboratory environment may influence test user's behaviors. HE is a cheap, fast and easy-to-use method, while it often finds too specific and low-priority usability problems, including even not real problems. CW helps find mismatches between users' and designers' conceptualization of a task, but it needs extensive knowledge of cognitive psychology and technical details to apply. However, even though these advantages and disadvantages show overall characteristics of three major usability evaluation methods, we cannot compare them quantitatively and see their efficiency clearly. Because one of reasons why so-called discounted methods, such as HE and CW, were developed is to save costs of usability evaluation, cost-related criteria for comparing usability evaluation are meaningful to usability practitioners as well as usability researchers. One of the most disputable issues related to cost of usability evaluation is sample size. That is, how many users or evaluators are needed to achieve a targeted usability evaluation performance, for example, 80% of overall discovery rate? The sample size of usability evaluation is known to depend on an estimate of problem discovery rate across participants. The overall discovery rate is a common quantitative measure that is used to show the effectiveness of a specific usability evaluation method in most of usability evaluation studies. It is also called overall detection rate or thoroughness measure, which is the ratio of 'the sum of unique usability problems detected by all experiment participants' against 'the number of usability problems that exist in the evaluated systems', ranging between 0 and 1. The overall discovery rates were reported more than any other criterion measure in the usability evaluation experiments and also a key component for projecting required sample size for usability evaluation study. Thus, how many test users or evaluators participate in the usability evaluation is a critical issue, considering its cost-effectiveness.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“However, a sample size of 10±2 is sufficient for discovering 80 per cent of usability issues [55].”
From A user-centred design framework for mHealth (2020) · cites this paper as [55]
Citations
Cites
Heuristic evaluation of user interfaces
Nielsen, J., & Molich, R. (1990) · Proceedings of the SIGCHI conference on Human factors in computing systems Empowering people - CHI '90 · doi:10.1145/97243.97281 · Cited by 3,533
Refining the Test Phase of Usability Evaluation: How Many Subjects Is Enough?
Virzi, R. A. (1992) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/001872089203400407 · Cited by 1,190
Estimating the number of subjects needed for a thinking aloud test
Nielsen, J. (1994) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1994.1065 · Cited by 476
Sample sizes for usability studies: Additional considerations
Lewis, J. R. (1994) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/001872089403600215 · Cited by 341
What is gained and lost when using evaluation methods other than empirical testing
Desurvire, H., Kondziela, J. M., & Atwood, M. E. (1993) · Cited by 107
Relaxing the homogeneity assumption in usability testing
Caulton, D. A. (2001) · Behaviour & Information Technology · doi:10.1080/01449290010020648 · Cited by 90
Law, L.-C., & Hvannberg, E. Þ. (2002) · Nordic Conference on Human-Computer Interaction · doi:10.1145/572020.572030 · Cited by 87
Evaluation of Procedures for Adjusting Problem-Discovery Rates Estimated From Small Samples
Lewis, J. R. (2001) · International Journal of Human-Computer Interaction · doi:10.1207/s15327590ijhc1304_06 · Cited by 72
Analysis of combinatorial user effect in international usability tests
Law, E. L., & Hvannberg, E. Þ. (2004) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/985692.985694 · Cited by 63
Novice heuristic evaluations of a complex interface
Slavković, A., & Cross, K. (1999) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/632716.632902 · Cited by 31
Cited by
Items 401–500 of 563
Simon-Liedtke, T. H. A. J. T. (2021) · IADIS International Conference on Interfaces and Human Computer Interaction and International Conference on Game and Entertainment Technologies (IHCI/GET) · doi:10.33965/ihci_get2021_202105l002 · Cited by 1
User-interface design and evaluation in a mobile application for detecting latent tuberculosis
Farao, J. (2019) · Open University of Cape Town (University of Cape Town) · Cited by 1
Identificación empírica de beneficios de usabilidad
Aveledo, M. (2014) · UPM Digital Archive (Technical University of Madrid) · doi:10.20868/upm.thesis.33132 · Cited by 1
Aljahdali, S. (2024) · ORCA Online Research @Cardiff (Cardiff University) · Cited by 1
Improving customer support efficiency through decision support powered by machine learning
Boman, S. (2023) · DiVA (Linkoping University) · Cited by 1
How technology can help visually impaired young adults participate in physical activity
De Almeida Fontinele Zadra, G. (2025) · ePrints Soton (University of Southampton) · Cited by 1
Usability Inspection of Web Portals: Results and Insights from Empirical Study
Granić, A., Marangunić, N., & Mitrović, I. (2013) · Cited by 1
Anke, J., & Schoormann, T. (2025) · Electronic Markets · doi:10.1007/s12525-025-00839-y · Cited by 1
Tackling Big Data Variety using Metadata
Vranopoulos, G. (2022) · PEARL (University of Plymouth) · doi:10.24382/474 · Cited by 1
Giudice, G. (n.d.) · AMS Degree Thesis (University of Bologna) · Cited by 1
Evaluating the Usability of Mobile Instant Messaging Apps on iOS Devices
Caro‐Álvaro, S., García‐Cabot, A., García‐López, E., de‐Marcos, L., & Martínez-Herráiz, J.-J. (2015) · International Conference on Software Engineering Advances · Cited by 1
Evaluating the Effectiveness of Enhancing Webcasts With Digital Tagging
MANICLANG, M. A. F., YONG, R. Y. H., MARZIN, M., Amellal, K., Rajendran, K., Lee, E., & Sng, J. (2018) · Asian Journal of the Scholarship of Teaching and Learning · doi:10.24112/ajsotl.83089 · Cited by 1
A Systems Engineering Analysis of Opportunities for Pharmacists on Diabetes Care Teams
Jahn, M. A. (2019) · Figshare · doi:10.25394/pgs.7866161 · Cited by 1
Online Cultural Heritage : facilitating complex query making through Tangible User Interfaces
Pereda, J. (2016) · ePrints Soton (University of Southampton) · Cited by 1
Symmetry Measures on Complex Networks
Garrido, Á. (2017) · MDPI eBooks · doi:10.3390/books978-3-03842-499-4 · Cited by 1
A formative usability study of workflow management systems in label-free digital pathology
Herrmann, T. A. (2022) · F1000Research · doi:10.12688/f1000research.108924.1 · Cited by 1
A new framework and learning tool to enhance the usability of software
Almansour, F. (2017) · PEARL (University of Plymouth) · doi:10.24382/796 · Cited by 1
From Sand to Screen: Virtual Beach Tourism for People with Mobility Impairments
Santoso, H. B., Senapartha, I. K. D., Nendya, M. B., Tamtama, G. I. W., Hapsari, M. K. S. N., Wibisono, M., Dewi, C. A. K., Figueroa, R. B., Jr., & Wuttisittikulkij, L. (2025) · International Conference on Awareness Science and Technology (iCAST) · doi:10.1109/icast68191.2025.11300185 · Cited by 1
Designing a Digital Twin to Improve Adherence and Care
Karlström, M. (2023) · DiVA at Umeå University (Umeå University) · Cited by 1
Bir Akademik Dergi Yönetim Sistemi Olarak DergiPark'ın Kullanılabilirlik Değerlendirmesi
Ermişoğlu, D. (2019) · Hacettepe University Institutional Repository (hacettepe.edu.tr) · Cited by 1
Hayat, H. (2020) · Loughborough University Research Repository (Loughborough University) · doi:10.26174/thesis.lboro.12609179 · Cited by 1
Howard, S. B. (2018) · ThinkTech (Texas Tech University) · Cited by 1
The Usability State of Nine Public Self-Service Applications in Denmark
Billestrup, J., Stage, J., & Bruun, A. (2017) · Advances in Computer-Human Interaction · Cited by 1
El enriquecimiento injustificado como mecanismo de tutela frente de los derechos de la personalidad
Campos, R. G. (2023) · Advocatus · doi:10.26439/advocatus2023.n043.6428 · Cited by 1
Reconceptualisation of Digital Cultures of DMOs through an Enhanced Digital Platform
ZAINAL ABIDIN, H. B. (2021) · University of Surrey Open Research repository · doi:10.15126/thesis.900136 · Cited by 1
Conservation of Limited Resources: Design Principles for Security and Usability on Mobile Devices
Horcher, A.-M. (2018) · NSUWorks (Nova Southeastern University) · Cited by 1
Poppe, L. (2019) · Ghent University Academic Bibliography (Ghent University) · Cited by 1
Usability Testing of an Online Farming Investment Website
Portus, A. J., & Flores, P. J. C. (2016) · Advances in intelligent systems and computing · doi:10.1007/978-3-319-41685-4_17 · Cited by 1
Adiguna, C., Afwani, R., & Wijaya, I. G. P. S. (2023) · Jurnal Teknologi Informasi Komputer dan Aplikasinya (JTIKA ) · doi:10.29303/jtika.v5i1.213 · Cited by 1
Wibowo, A. W. A., Ismianti, I., Husaini, R., Mastrisiswadi, H., Kasih, P. H., Rahmawati, K., & Atsani, S. I. (2025) · OPSI · doi:10.31315/opsi.v18i2.15537 · Cited by 1
Useable Authentication Mechanisms for Secure Online Banking in Saudi Arabia
Alzahrani, A. (2018) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.21693135 · Cited by 1
Hannan, R., Laform, S., Boyle, K., Danziger, S., Mahyoub, M. A., & Smith, T. G. (2020) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/1071181320641344 · Cited by 1
Specific Heuristics for Smartwatch Usability Evaluation: Development, Validation, and Comparison
Li, Y., Richart, M., & Hermawati, S. (2025) · Repository@Nottingham (University of Nottingham) · Cited by 1
Sistema de avaliação de serviços de cloud computing
Reixa, M. C. G. (2012) · Repositório do ISCTE-IUL · Cited by 1
García‐López, E. (2013) · e_Buah · Cited by 1
Gamification of a Physics Simulation Tool
Baldwin, A., & Dahlberg, S. (2015) · DiVA (Digitala Vetenskapliga Arkivet) · Cited by 1
Mobile application analytics : specification for an online analytics tool
Juhlin, D. (2010) · DiVA (Luleå University of Technology) · Cited by 1
Wolfram, L. J. (2011) · Digital Archive @ GSU · doi:10.57709/1955145 · Cited by 1
Multimedia Learning Technology for Rural Malaysia
ABDUL LATEF, A. S. (2020) · University of Surrey Open Research repository · doi:10.15126/thesis.900063 · Cited by 1
Matloobtalab, M., & Iversen, P. (2023) · DiVA (Linnaeus University) · Cited by 1
Eco-Wearables: Merging Art and Technology for Environmental Crisis Awareness
Dehghani, M. (2024) · OCAD University Open Research Repository (OCAD University) · Cited by 1
Automated Usability Testing Of Websites Using Link Structure
Korvald, C. (2014) · UND Scholarly Commons (University of North Dakota) · Cited by 1
Development and evaluation of a mobile web-based core nursing skills management system
조은미 (2014) · Seoul National University Open Repository (Seoul National University) · Cited by 1
Enhancing the User Interface of Open-Source Erp for Educational Use in Higher Education
Muhammad, H. A. F., & Putri, N. K. S. (2025) · IEEE International Conference on Cryptography, Informatics, and Cybersecurity (ICoCICs) · doi:10.1109/icocics68032.2025.11384048
Cenkner, D. P., June, J., Schueller, S. M., Zalta, A. K., Grammer, J., Ekman, E., Baldini, K., LaCroix, J. M., & Ghahramanlou‐Holloway, M. (2026) · JMIR Formative Research · doi:10.2196/89994
Developing Innovative Tool to Enhance the Effectiveness of Decision Support System
Almansour, F., & Stuart, L. (2016) · Lecture notes in business information processing · doi:10.1007/978-3-319-32877-5_14
CatalaBlocks: A Block-Based Visual Tool for Programming the Law
Russo, N. M., Schmitt, M., Iyer, A., & Hong, J. (2026) · Designing Interactive Systems Conference · doi:10.1145/3800645.3813091
Aldhoayan, M. D. (2018) · D-Scholarship@Pitt (University of Pittsburgh)
de Wit, M., Coetzee, M., van Greunen, D., & Plastow, N. A. (2026) · BMC Digital Health · doi:10.1186/s44247-026-00272-x
Method of neuro-fuzzy estimation of graphical user interface usability
Королёва, О. ".-К. К. ". И. С. П., & Корвяков, В. П. (2016) · Herald of the Bauman Moscow State Technical University Series Instrument Engineering · doi:10.18698/0236-3933-2016-5-61-74
Augello, A., Caggianese, G., Antaranian, S., Zissis, G., Pietro, G. D., & Gallo, L. (2026) · Human Behavior and Emerging Technologies · doi:10.1155/hbe2/6622167
Explaining Scientific Hypotheses in Drug Development with Knowledge Graphs
Nunes, S., & Pesquita, C. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-25156-5_6
Joshi, A., Agrawal, S., Grover, A., Malhotra, B., Gauniyal, M., Rajkhowa, S., & Rishi, B. (2026) · BMC Digital Health · doi:10.1186/s44247-026-00248-x
Kelly, M. M., Dempsey, A. E., Ameral, V., Petrakis, B. A., Reilly, E. D., Quigley, K. S., Bricker, J. B., & Heffner, J. L. (2026) · JMIR Formative Research · doi:10.2196/75394
Chen, X., Ning, H., Peng, L., Zhang, C., Zhang, J., Jiang, D., Li, X., Wan, X., Huang, J., Xie, Q., et al. (2026) · JMIR Serious Games · doi:10.2196/93081
Stankić, A., Vujošević, D., Radosavljevic, N., & Jeremić, A. (2026) · Health Informatics Journal · doi:10.1177/14604582261485150
Computational curation of traditional martial arts: from archives to systems
Hou, Y. (2026) · Digital Scholarship in the Humanities · doi:10.1093/llc/fqag038
Scholtz, B. M., Calitz, A. P., & Snyman, I. (2015) · South African Computer Journal · doi:10.18489/sacj.v0i57.269
Pfister, P. B. (2021) · Research Publications (Maastricht University) · doi:10.26481/dis.20210617pb
Can we configure COOK, a cognitive Orthosis for meal preparation, with efficiency and effectiveness?
Santos, M. A. D., Rondeau, M., Gagnon‐Roy, M., Lignon, C., Ngankam, H. K., Bier, N., Giroux, S., & Bottari, C. L. (2025) · Disability and Rehabilitation Assistive Technology · doi:10.1080/17483107.2025.2557443
Orellano-Colón, E. M., Bonilla-Díaz, W., Revilla-Orellano, R., Mejías-Castro, J., & Roche-Lima, A. (2025) · JMIR Formative Research · doi:10.2196/69645
Jamaludin, S. A., Abdul-Razak, S., Nasir, N. M. N. M., Kassim, S. S., Miptah, H. N., Kanoo, L. L., & Zainuddin, M. Z. (2026) · Digital Health · doi:10.1177/20552076261443058
Leveraging Immersive Virtual Reality to Prevent Child Injuries in the Home: Proof-of-Concept Study
Osborne, M. C., Zhang, L., Garefino, A. C., & Freeman, R. (2026) · JMIR Pediatrics and Parenting · doi:10.2196/90006
Jang, A., Jeong, H., & Kim, Y.-H. (2025) · Applied Sciences · doi:10.3390/app152111650
Improving Informed Consent for English and Spanish Speakers in Clinical Trials
Peinado, S., Thompson, J. E., Plouffe, L., Raynor, D. K., & Mccormack, L. A. (2025) · Therapeutic Innovation & Regulatory Science · doi:10.1007/s43441-025-00885-w
โนรี, ส., & รัตนวิชา, พ. (2017) · Chula Digiverse · doi:10.58837/chula.the.2017.697
Towards a Taxonomy for Neighborhood Volunteering Management Platforms
Betke, H., Bosse, M., Tzavala-Reusch, F., Kühnel, S., & Sackmann, S. (2025) · Lecture notes in information systems and organisation · doi:10.1007/978-3-031-80119-8_19
Effectiveness of the Adaptive Heuristic Evaluation Method for Augmented Reality Interfaces
Lim, D., Suh, G., Shin, S., & Hwang, W. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2561767
Self-citationDesigning Adaptive AI Assistance for Block-Based Modelling: a Wizard of Oz Study with Domain Experts
Mannari, C., Turchi, T., Bacco, M., Ferrari, A., Conati, C., & Malizia, A. (2026) · International Conference on Advanced Visual Interfaces (AVI) · doi:10.1145/3811427.3811448
Markiewicz, A., Vuillermoz, E., Davis, C., & Riley, M. (2026) · doi:10.2196/preprints.103489
Designing an AI Avatar for an Interactive Companion in the MR Environment
Arman, M. H., Ismail, A. W., Fadzli, F. E., & Ahmad, M. A. (2025) · IEEE International Conference on System Engineering and Technology (ICSET) · doi:10.1109/icset65917.2025.11284180
EDUCATIONAL REWARD INFORMATION COMMUNICATION API (ERIC API): A PRELIMINARY STUDY RESULT
Kuo, R., Chen, C., Lu, Z.-X., Chang, M., & Chang, H. (2019) · Revista Produção e Desenvolvimento · doi:10.32358/rpd.2019.v5.395
Hsu, H.-H., Cheng, K.-C., & Chiu, Y.-L. (2026) · JMIR Rehabilitation and Assistive Technologies · doi:10.2196/107314
The role of user session on the heuristic evaluation
Alqurni, J. S. (2019)
Yusuf, A. O., Leal, A. N., Akanmu, A. A., Murzi, H., Shojaei, A., & Agee, P. R. (2024) · ASCE Construction Research Congress (CRC) · doi:10.1061/9780784485293.030
Brizzi, G., Signorelli, A., & Riva, G. (2026) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-032-26235-6_1
Remote medical monitoring decision support system and user interface usability
Tosine, A. (2011) · Carleton University's Institutional Repository (MacOdrum Library, Carleton University) · doi:10.22215/etd/2011-09190
Weermeijer, J. D. M. (2023) · Lirias
Chiam, H. (2017) · TSpace (University of Toronto)
이예림 (2017) · Seoul National University Open Repository (Seoul National University)
Towards Adaptive Switches through implementation of visual feedback in assistive devices
Cascone, F. D., Martorelli, M., Gloria, A., Nathan-Roberts, D., & Lanzotti, A. (2019) · IEEE International Workshop on Metrology for Industry 4.0 & IoT (MetroInd4.0&IoT) · doi:10.1109/metroi4.2019.8792893
Young, S., Awuor, D. R., Owuor, P., Thavarajah, W., Kiprotich, K., Aggarwal, R., & Lucks, J. (2022) · Research Square · doi:10.21203/rs.3.rs-1839702/v1
Roncete, G. P., Gentile, G., Vicentini, F. C., Vicentini, A. T. C., Antolli, E., Morandini, M., Danilovic, A., Batagello, C., Marchini, G. S., Torricelli, F. C. M., et al. (2026) · Applied Clinical Informatics · doi:10.1055/a-2880-6345
BlocksDeFix: Guided Debugging Hints for Young Learners in Block-based Programming Environments
van Uijthoven, E., & Seraj, M. (2026) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3803437.3805787
Kamsteeg, M., Algera, F., van der Rijt, C., Mulder, M., & Torkamaan, H. (2026) · Proceedings - IEEE Symposium on Computer-Based Medical Systems · doi:10.1109/cbms69103.2026.00276
Zou, Y., Pianca, E., & D'Cunha, N. M. (2026) · Dementia · doi:10.1177/14713012261492102
(2015) · Elsevier eBooks · doi:10.1016/b978-0-12-800232-2.09983-1
Ribeiro, F. G. C., Rodrigues, L., Xavier, C., da Costa, N. T., Cavalcanti, A. P., Lobo, J., Gašević, D., Drachsler, H., & Mello, R. F. (2026) · Journal of Computer Assisted Learning · doi:10.1002/jcal.70362
Ansari, R., Herman, R., Cartwright, M., Escobedo, L., & Wehbe, R. R. (2026) · International Journal of Child-Computer Interaction · doi:10.1016/j.ijcci.2026.100860
Developer Driven and User Driven Usability Evaluations
Bruun, A. (2013) · VBN Forskningsportal (Aalborg Universitet)
Rojas, L. E. P., Araque, O., Figueroa-Olaya, M. X., & Riveros, Ó. C. V. (2026) · Green Technologies and Sustainability · doi:10.1016/j.grets.2026.100447
Shift-Integrated mHealth for Safety-Critical Work: Formative Evaluation of a Police Wellbeing App
Ayorinde, O. T., Doğan, H., Adedoyin, F. F., Jiang, N., Bitters, F., & Dempsey, S. (2026) · Electronic workshops in computing · doi:10.14236/ewic/britchi26.27
Gagnon‐Roy, M. (2023) · Papyrus : Institutional Repository (Université de Montréal) · doi:10.71781/26575
Kokemuksia käytettävyystestauksesta: Ääneenajattelun ja ohjaajan läsnäolon vaikutuksia
Riihiaho, S. (2015) · Aaltodoc (Aalto University)
(2013) · Elsevier eBooks · doi:10.1016/b978-0-12-410391-7.00014-2
An Expert-In-The-Loop Design Utilising Knowledge Graphs to Prompt LLMs in Professional Writing
Yu, X., He, C., Zhou, X., Zhang, L., & Miao, C. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-30849-8_31
Development of a smartphone application for rheumatoid arthritis self-management
Azevedo, A. R. P. (2014) · Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT) · doi:10.34626/z4j1-8f15
Metadata
- Authors
- Hwang, W., & Salvendy, G.
- Affiliations
- Soongsil University (KR)
- Purdue University West Lafayette (US)
- ORCID
- Hwang, Wonil
- Venue
- Communications of the ACM
- Published
- 2010-05
- Volume
- 53
- Issue
- 5
- Pages
- 130-133
- ISSN
- 0001-0782 (ISSN-L)
- 1557-7317
- OpenAlex
- W2016022053
- Cited by
- 410
- References
- 11
- Citations by year
- 2026: 40 · 2025: 36 · 2024: 26 · 2023: 43 · 2022: 27 · 2021: 21 · 2020: 29 · 2019: 30 · 2018: 22 · 2017: 24 · 2016: 23 · 2015: 15 · 2014: 24 · 2013: 17 · 2012: 17
- FWCI
- 31.16
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Communications of the ACM (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Keywords
- usability evaluation, sample size, think-aloud, heuristic evaluation, cognitive walkthrough, detection rate, evaluation costs, user interface design, verbal protocols, user testing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.