Number of people required for usability evaluation: the 10±2 rule
The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
Introduction Usability evaluation is essential to make sure that software products newly released are easy to use, efficient, and effective to reach goals, and satisfactory to users. For example, when a software company wants to develop and sell a new product, the company needs to evaluate usability of the new product before launching it at a market to avoid the possibility that the new product may contain usability problems, which span from cosmetic problems to severe functional problems. Three widely used methods for usability evaluation are Think Aloud (TA), Heuristic Evaluation (HE) and Cognitive Walkthrough (CW). TA method is commonly employed with a lab-based user testing, while there are variants of TA methods, including thinking out aloud at user's workplace instead of at labs. What we discuss here is the TA method that is combined with a lab-based user testing, in which test users use products while simultaneously and continuously thinking out aloud, and experimenters record users' behaviors and verbal protocols in the laboratory. HE is a usability inspection method, in which a small number of evaluators find usability problems in a user interface design by examining an interface and judging its compliance with well-known usability principles, called heuristics. CW is a theory-based method, in which evaluators evaluate every step necessary to perform a scenario-based task, and look for usability problems that would interfere with learning by exploration. These three methods have their own advantages and disadvantages. For instance, TA method provides good qualitative data from a small number of test users, but laboratory environment may influence test user's behaviors. HE is a cheap, fast and easy-to-use method, while it often finds too specific and low-priority usability problems, including even not real problems. CW helps find mismatches between users' and designers' conceptualization of a task, but it needs extensive knowledge of cognitive psychology and technical details to apply. However, even though these advantages and disadvantages show overall characteristics of three major usability evaluation methods, we cannot compare them quantitatively and see their efficiency clearly. Because one of reasons why so-called discounted methods, such as HE and CW, were developed is to save costs of usability evaluation, cost-related criteria for comparing usability evaluation are meaningful to usability practitioners as well as usability researchers. One of the most disputable issues related to cost of usability evaluation is sample size. That is, how many users or evaluators are needed to achieve a targeted usability evaluation performance, for example, 80% of overall discovery rate? The sample size of usability evaluation is known to depend on an estimate of problem discovery rate across participants. The overall discovery rate is a common quantitative measure that is used to show the effectiveness of a specific usability evaluation method in most of usability evaluation studies. It is also called overall detection rate or thoroughness measure, which is the ratio of 'the sum of unique usability problems detected by all experiment participants' against 'the number of usability problems that exist in the evaluated systems', ranging between 0 and 1. The overall discovery rates were reported more than any other criterion measure in the usability evaluation experiments and also a key component for projecting required sample size for usability evaluation study. Thus, how many test users or evaluators participate in the usability evaluation is a critical issue, considering its cost-effectiveness.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“However, a sample size of 10±2 is sufficient for discovering 80 per cent of usability issues [55].”
From A user-centred design framework for mHealth (2020) · cites this paper as [55]
Citations
Cites
Heuristic evaluation of user interfaces
Nielsen, J., & Molich, R. (1990) · Proceedings of the SIGCHI conference on Human factors in computing systems Empowering people - CHI '90 · doi:10.1145/97243.97281 · Cited by 3,533
Refining the Test Phase of Usability Evaluation: How Many Subjects Is Enough?
Virzi, R. A. (1992) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/001872089203400407 · Cited by 1,190
Estimating the number of subjects needed for a thinking aloud test
Nielsen, J. (1994) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1994.1065 · Cited by 476
Sample sizes for usability studies: Additional considerations
Lewis, J. R. (1994) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/001872089403600215 · Cited by 341
What is gained and lost when using evaluation methods other than empirical testing
Desurvire, H., Kondziela, J. M., & Atwood, M. E. (1993) · Cited by 107
Relaxing the homogeneity assumption in usability testing
Caulton, D. A. (2001) · Behaviour & Information Technology · doi:10.1080/01449290010020648 · Cited by 90
Law, L.-C., & Hvannberg, E. Þ. (2002) · Nordic Conference on Human-Computer Interaction · doi:10.1145/572020.572030 · Cited by 87
Evaluation of Procedures for Adjusting Problem-Discovery Rates Estimated From Small Samples
Lewis, J. R. (2001) · International Journal of Human-Computer Interaction · doi:10.1207/s15327590ijhc1304_06 · Cited by 72
Analysis of combinatorial user effect in international usability tests
Law, E. L., & Hvannberg, E. Þ. (2004) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/985692.985694 · Cited by 63
Novice heuristic evaluations of a complex interface
Slavković, A., & Cross, K. (1999) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/632716.632902 · Cited by 31
Cited by
Items 101–200 of 563
Pantazos, K., & Vatrapu, R. (2016) · Hawaii International Conference on System Sciences (HICSS) · doi:10.1109/hicss.2016.14 · Cited by 24
Responsible Use of AI in Military Systems
Schraagen, J. M. (2024) · Chapman & Hall/CRC eBooks · doi:10.1201/9781003410379 · Cited by 24
Design and Usability of Interactive User Profiles for Online Health Communities
Hartzler, A. L., Weis, B., Cahill, C., Pratt, W., Park, A., Backonja, U., & McDonald, D. W. (2016) · ACM Transactions on Computer-Human Interaction · doi:10.1145/2903718 · Cited by 24
Ponce, P., Balderas, D., Peffer, T. E., & Molina, A. (2017) · Energy and Buildings · doi:10.1016/j.enbuild.2017.12.043 · Cited by 24
Jiang, M., Liu, S., Feng, Q., Gao, J., & Zhang, Q. (2018) · Medical Science Monitor · doi:10.12659/msm.909933 · Cited by 24
Kopka, M., Slagman, A. C., Schorr, C., Krampe, H., Altendorf, M. B., Balzer, F., Bolanaki, M., Kuschick, D., Möckel, M., Napierala, H., et al. (2024) · Smart Health · doi:10.1016/j.smhl.2024.100517 · Cited by 23
GroupvsIndividual Web Accessibility Evaluations: Effects with Novice Evaluators
Brajnik, G., Vigo, M., Yeşilada, Y., & Harper, S. (2016) · Interacting with Computers · doi:10.1093/iwc/iww006 · Cited by 23
Exploring user experience and usability of mobile and open banking for digital natives in Turkiye
Soyupak, O., & İpek, H. (2025) · Qualitative Research in Financial Markets · doi:10.1108/qrfm-02-2024-0051 · Cited by 23
'Surprised, Shocked, Worried'}: User Reactions to Facebook Data Collection from Third Parties
Arias-Cabarcos, P., Khalili, S., & Strufe, T. (2023) · Proceedings on Privacy Enhancing Technologies · doi:10.56553/popets-2023-0023 · Cited by 23
Evaluation of Flexible Graphical User Interface for Intuitive Human Robot Interactions
Dániel, B., Köröndi, P., Sziebig, G., & Thomessen, T. (2014) · Acta Polytechnica Hungarica · doi:10.12700/aph.11.01.2014.01.9 · Cited by 23
Reddit Insights: Improving Online Discussion Culture by Contextualizing User Profiles
Waltenberger, F., Höferlin, S., & Froehlich, M. (2023) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3544549.3585671 · Cited by 23
Cross-Platform Usability Model Evaluation
Majrashi, K., Hamilton, M., Uitdenbogerd, A. L., & Al-Megren, S. (2020) · Multimodal Technologies and Interaction · doi:10.3390/mti4040080 · Cited by 23
Determining fidelity of mixed prototypes: Effect of media and physical interaction
Zhou, X., & Rau, P. P. (2019) · Applied Ergonomics · doi:10.1016/j.apergo.2019.05.007 · Cited by 23
Afonso, L., Rodrigues, R., Reis, E., Miller, K. M., Castro, J., Parente, N., Teixeira, C., Fraga, A., & Torres, S. (2020) · IEEE Transactions on Games · doi:10.1109/tg.2020.3015804 · Cited by 21
Lazar, J. K., Feng, J. H., & Hochheiser, H. (2017) · Elsevier eBooks · doi:10.1016/b978-0-12-805390-4.00001-7 · Cited by 21
“I Feel My Abs”: Exploring Non-standing VR Locomotion
Kontio, R., Laattala, M., Welsch, R., & Hämäläinen, P. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3611069 · Cited by 21
Heuristic evaluation on Augmented Reality courseware for the deaf
Zainuddin, N. M. M., Zaman, H. B., & Ahmad, A. (2011) · International Conference on User Science and Engineering · doi:10.1109/iuser.2011.6150562 · Cited by 21
Hernando, R., & Macías, J. A. (2022) · Behaviour and Information Technology · doi:10.1080/0144929x.2021.2022209 · Cited by 21
Usability evaluation guidelines for business intelligence applications
Jooste, C., van Biljon, J. A., & Mentz, J. C. (2013) · South African Institute of Computer Scientists and Information Technologists · doi:10.1145/2513456.2513478 · Cited by 21
Testing the Usability of OpenStreetMap's iD Tool
Behrens, J., Elzakker, C. P. J. M. V., & Schmidt, M. (2015) · The Cartographic Journal · doi:10.1080/00087041.2015.1119457 · Cited by 20
Amouzgar, K., & Willebrand, J. (2025) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2025.05.019 · Cited by 20
Santarnecchi, E., Balloni, E., Paolanti, M., Frontoni, E., Stacchio, L., Zingaretti, P., & Pierdicca, R. (2025) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2025.3549160 · Cited by 19
User-centred design for Chinese-oriented spoken english learning system
Yu, P., Pan, Y., Li, C., Zhang, Z., Shi, Q., Chu, W., Liu, M., & Zhu, Z. (2015) · Computer Assisted Language Learning · doi:10.1080/09588221.2015.1121877 · Cited by 19
Reichold, M., Heß, M., Kolominsky‐Rabas, P. L., Graessel, E., & Prokosch, H. (2021) · JMIR Formative Research · doi:10.2196/31649 · Cited by 19
Vigo, M., Matentzoglu, N., Jay, C., & Stevens, R. (2018) · Journal of Web Semantics · doi:10.1016/j.websem.2018.09.004 · Cited by 19
Bernini, S., Panzarasa, S., Quaglini, S., Costa, A. C., Picascia, M., Cappa, S. F., Cerami, C., Tassorelli, C., Vecchi, T., & Bottiroli, S. (2023) · Frontiers in Medicine · doi:10.3389/fmed.2023.1129914 · Cited by 18
MyPDDL: Tools for Efficiently Creating PDDL Domains and Problems
Strobel, V., & Kirsch, A. (2020) · Springer eBooks · doi:10.1007/978-3-030-38561-3_4 · Cited by 18
A Decision Support System for Global Team Management: Expert Evaluation
Beecham, S., Carroll, N., & Noll, J. (2012) · IEEE International Conference on Global Software Engineering Workshops (ICGSEW) · doi:10.1109/icgsew.2012.14 · Cited by 18
The effect of task assignments and instruction types on remote asynchronous usability testing
Bruun, A., & Stage, J. (2012) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2208364 · Cited by 17
Analysis and Design of Service Business Models
Zolnowski, A. (2015) · Cited by 17
Digital Health and Care Study on Elderly Monitoring
Gaiduk, M., Seepold, R., Madrid, N. M., & Ortega, J. A. (2021) · Sustainability · doi:10.3390/su132313376 · Cited by 17
Argus: Interactive a priori Power Analysis
Wang, X., Eiselmayer, A., Mackay, W. E., Hornbæk, K., & Wacharamanotham, C. (2020) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2020.3028894 · Cited by 17
Interactive generative system supporting participatory house design
Kwieciński, K., & Słyk, J. (2022) · Automation in Construction · doi:10.1016/j.autcon.2022.104665 · Cited by 17
An experiment with content distribution methods in touchscreen mobile devices
García‐López, E., García‐Cabot, A., & de‐Marcos, L. (2015) · Applied Ergonomics · doi:10.1016/j.apergo.2015.02.009 · Cited by 17
A Systematic Evaluation of Mobile Applications for Instant Messaging on iOS Devices
Caro‐Álvaro, S., García‐López, E., García‐Cabot, A., de‐Marcos, L., & Martínez, J. M. G. (2017) · Mobile Information Systems · doi:10.1155/2017/1294193 · Cited by 17
Usability Inspection in Model-Driven Web Development: Empirical Validation in WebML
Fernandez, A., Abrahão, S., Insfrán, E., & Matera, M. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-41533-3_45 · Cited by 17
Empirical Formal Methods: Guidelines for Performing Empirical Studies on Formal Methods
ter Beek, M. H., & Ferrari, A. (2022) · Software · doi:10.3390/software1040017 · Cited by 16
Martín‐Gutiérrez, J., & Guerra, M. S. D. R. (2021) · Sensors · doi:10.3390/s21041328 · Cited by 16
Heuristic Evaluation in Visualization: An Empirical Study : Position paper
Santos, B. S., Silva, S., & Dias, P. (2018) · IEEE Evaluation and Beyond - Methodological Approaches for Visualization (BELIV) · doi:10.1109/beliv.2018.8634108 · Cited by 16
An Empirical Usability Analysis of the Google Authentication API
Wijayarathna, C., & Arachchilage, N. A. G. (2019) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3319008.3319350 · Cited by 16
Zhang, H., Guo, G., Comstock, E., Chen, B., Chen, X., Song, C., Ajay, J., Langan, J., Bhattacharjya, S., Cavuoto, L. A., et al. (2020) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3386901.3389028 · Cited by 16
Simulator-based Mixed Reality eVTOL Pilot Training: The Instructor Operator Station
Kimura, S., Zintl, M., Hammann, C., & Holzapfel, F. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642060 · Cited by 16
Haliburton, L., Lammel, M., Karolus, J., & Schmidt, A. (2022) · International Conference on Mobile and Ubiquitous Multimedia (MUM) · doi:10.1145/3568444.3568452 · Cited by 16
A Critical Analysis of the Evaluation Practice in Medical Visualization
Preim, B., Ropinski, T., & Isenberg, P. (2018) · HAL · doi:10.2312/vcbm.20181228 · Cited by 16
Planning in the Wild: Modeling Tools for PDDL
Strobel, V., & Kirsch, A. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-11206-0_27 · Cited by 16
Towards to the validation of a usability evaluation method for model-driven web development
Fernandez, A., Abrahão, S., & Insfrán, E. (2010) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/1852786.1852855 · Cited by 16
Aldughayfiq, B., & Sampalli, S. (2021) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2021.104509 · Cited by 15
Gerard, F. J., Ammerer, K., Mentzelopoulos, M., & Economou, D. (2024) · Journal of Criminal Justice Education · doi:10.1080/10511253.2024.2324179 · Cited by 15
Lehong, S., van Biljon, J. A., & Sanders, I. D. (2022) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2022.2121205 · Cited by 15
Dual-method usability evaluation of e-commerce websites : in quest of better user experience
Moczarny, I. M. (2011) · Unisa Institutional Repository (University of South Africa) · Cited by 15
Evaluating multimodal affective fusion using physiological signals
Gilroy, S. W., Cavazza, M., & Vervondel, V. (2011) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/1943403.1943413 · Cited by 15
RITA: a useR Interface evaluaTion frAmework
Kolski, C., Ezzedine, H., & Charfi, S. (2015) · HAL · doi:10.3217/jucs-021-04-0526 · Cited by 15
A Framework to Conduct and Report on Empirical User Studies in Semantic Web Contexts
Pesquita, C., Ivanova, V., Lohmann, S., & Lambrix, P. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-03667-6_36 · Cited by 15
Usability research for interpreter-centred technology
Frittella, F. M. (2023) · doi:10.17169/refubium-37470 · Cited by 15
Ahmad, W. F. W., Sulaiman, S. B., & Johari, F. S. (2010) · International Conference on User Science and Engineering · doi:10.1109/iuser.2010.5716734 · Cited by 15
Yalçın, Ö. N., Lallé, S., & Conati, C. (2022) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3490099.3511124 · Cited by 14
Xiong, M., Chan, N. N., Wong, B. E., Xie, X., & Na, M. (2025) · Frontiers in Psychology · doi:10.3389/fpsyg.2025.1691422 · Cited by 14
Formative Evaluation of Data-Driven Business Models – The Data Insight Generator
Kühne, B., & Böhmann, T. (2020) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2020.053 · Cited by 14
Zahib, M. A., Effendy, V., & Darwiyanto, E. (2022) · Journal of Information System Research (JOSH) · doi:10.47065/josh.v4i1.2246 · Cited by 14
Hokkanen, L. (2017) · Tampere University Institutional Repository (Tampere University) · Cited by 14
MDWA: a model-driven Web augmentation approach—coping with client- and server-side support
Urbieta, M., Firmenich, S., Bosetti, G., Maglione, P., Rossi, G. H., & Olivero, M. Á. (2020) · Software & Systems Modeling · doi:10.1007/s10270-020-00779-5 · Cited by 14
Frustration control during driving using auditory false heart rate feedback
Ota, K., Fujiwara, K., & Hiraoka, T. (2024) · Transportation Research Part F Traffic Psychology and Behaviour · doi:10.1016/j.trf.2024.01.014 · Cited by 13
Fontecha, J., Gónzalez, I., & Bravo, J. (2019) · Journal of Ambient Intelligence and Humanized Computing · doi:10.1007/s12652-019-01369-0 · Cited by 13
GamiDOC: The Importance of Designing Gamification in a Proper Way
Bassanelli, S., Bucchiarone, A., & Gini, F. (2024) · IEEE Transactions on Games · doi:10.1109/tg.2024.3364061 · Cited by 13
LLMs Enable Context-Aware Augmented Reality in Surgical Navigation
Javaheri, H., Ghamarnejad, O., Lukowicz, P., Stavrou, G. A., & Karolus, J. (2025) · Designing Interactive Systems Conference · doi:10.1145/3715336.3735708 · Cited by 13
Usability and Acceptance of an Interactive Tablet-Based Exercise Application: A Mixed Methods Study
Pfister, P. B., Tobler-Ammann, B., Knols, R. H., de Bruin, E. D., & de Bie, R. A. (2020) · Frontiers in Digital Health · doi:10.3389/fdgth.2020.578281 · Cited by 13
Auditory Feedback of False Heart Rate for Video Game Experience Improvement
Ogawa, S., Fujiwara, K., & Kano, M. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.3039874 · Cited by 13
Zambo, D., Moyo, D., Tanthuma, G., Chapman, J., Re, V. L., Graziani, A., Lowenthal, E. D., Hanrahan, N. P., Littman–Quinn, R., Kovarik, C., et al. (2017) · Methods of Information in Medicine · doi:10.3414/me17-01-0045 · Cited by 13
Hodson, N., Woods, P., Solano, J. L., Talbot, C., & Giacco, D. (2024) · JMIR Pediatrics and Parenting · doi:10.2196/53907 · Cited by 12
Ogunseiju, O. R., Gonsalves, N. J., Akanmu, A. A., Abraham, Y. S., & Nnaji, C. A. (2023) · Smart and Sustainable Built Environment · doi:10.1108/sasbe-07-2022-0129 · Cited by 12
A Veteran-Centric Web-Based Decision Aid for Lung Cancer Screening: Usability Analysis
Schapira, M. M., Chhatre, S., Prigge, J. M., Meline, J., Kaminstein, D. S., Rodriguez, K. L., Fraenkel, L., Kravetz, J. D., Whittle, J., Bastian, L. A., et al. (2022) · JMIR Formative Research · doi:10.2196/29039 · Cited by 12
Zainuddin, N. M. M., Maarop, N. B., & Hassan, W. A. W. (2021) · Asian Journal of University Education · doi:10.24191/ajue.v17i4.16214 · Cited by 12
Less users more confidence: How AOIs don’t affect scanpath trend analysis
Eraslan, S., Yeşilada, Y., & Harper, S. (2017) · Journal of Eye Movement Research · doi:10.16910/jemr.10.4.6 · Cited by 12
Facilitating and automating usability testing of educational technologies
Villamañe, M., & Álvarez, A. (2024) · Computer Applications in Engineering Education · doi:10.1002/cae.22725 · Cited by 12
Better state pictures facilitating state machine characteristic conjecture
Bui, D. D., & Ogata, K. (2021) · Multimedia Tools and Applications · doi:10.1007/s11042-021-10992-z · Cited by 12
Giri, I. G. P. Y., Dewi, L. J. E., & Sunarya, I. M. G. (2023) · Journal Of Computer Networks Architecture and High Performance Computing · doi:10.47709/cnahpc.v5i2.2511 · Cited by 12
Evaluating the Usability of Mashups Applications
Insfrán, E., Cedillo, P., Fernandez, A., Abrahão, S., & Matera, M. (2012) · International Conference on the Quality of Information and Communications Technology (QUATIC) · doi:10.1109/quatic.2012.28 · Cited by 12
Brizee, A. (2014) · Computers & composition/Computers and composition · doi:10.1016/j.compcom.2014.04.007 · Cited by 12
Measuring Malaysian M-commerce user behaviour
Tzuaan, S. S., Sivaji, A., Yong, L. T., Zanegenh, M. H. T., & Shan, L. (2014) · International Conference on Computer and Information Sciences · doi:10.1109/iccoins.2014.6868405 · Cited by 12
FiberStars: Visual Comparison of Diffusion Tractography Data between Multiple Subjects
Franke, L., Weidele, D. K. I., Zhang, F., Karayumak, S. C., Pieper, S., O’Donnell, L. J., Rathi, Y., & Haehn, D. (2021) · IEEE Pacific Visualization Symposium · doi:10.1109/pacificvis52677.2021.00023 · Cited by 11
Jones, C. E., & Maquil, V. (2016) · Communications in computer and information science · doi:10.1007/978-3-319-29589-3_7 · Cited by 11
Bailey, S. K. T., Lewis, J. E., Ciccone, B. A., Friedland, R. L., & Reiner, C. C. (2022) · Simulation in Healthcare The Journal of the Society for Simulation in Healthcare · doi:10.1097/sih.0000000000000637 · Cited by 11
What sample sizes do we need? Part 1: summative studies
Sauro, J., & Lewis, J. R. (2016) · Elsevier eBooks · doi:10.1016/b978-0-12-802308-2.00006-0 · Cited by 11
Zainuddin, N. M. M., Maarop, N. B., & Hassan, W. A. W. (2022) · Asian Journal of University Education · doi:10.24191/ajue.v18i2.17990 · Cited by 11
Usability Evaluation for User Interface Redesign of Financial Technology Application
Nugraha, A. P., Rolando, R., Puspasari, M. A., & Syaifullah, D. H. (2019) · IOP Conference Series Materials Science and Engineering · doi:10.1088/1757-899x/505/1/012101 · Cited by 11
O'Connor, S. R., Wilson, V., Brown, D., Cleland, I., & McCance, T. (2024) · Digital Health · doi:10.1177/20552076241271788 · Cited by 10
Willis, M., Hein, L. B., Hu, Z., Saran, R., Argentina, M., Bragg‐Gresham, J. L., Krein, S. L., Gillespie, B. W., Zheng, K., & Veinot, T. C. (2021) · JMIR Human Factors · doi:10.2196/26012 · Cited by 10
Hartzler, A. L., Bartlett, L., Hobler, M. R., Reid, N., Pryor, J. B., Kapnadak, S. G., Berry, D. L., Lober, W. B., Goss, C. H., Ramos, K. J., et al. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac176 · Cited by 10
Saly, L., Provvidenza, C. F., Al-Hakeem, H., Hickling, A., Stevens, S. A., Kakonge, L., Hunt, A. W., Bennett, S. M., Martinussen, R., & Scratch, S. E. (2023) · JMIR Human Factors · doi:10.2196/43129 · Cited by 10
Othman, M. K., Ong, L. W., & Aman, S. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-022-11991-4 · Cited by 10
Mixed-method usability investigation of ARROWS: augmented reality for roadway work zone safety
Sabeti, S., Morris, N. L., & Shoghli, O. (2024) · International Journal of Occupational Safety and Ergonomics · doi:10.1080/10803548.2023.2295660 · Cited by 10
Understanding How People Approach Constraint Modelling and Solving
Hoffmann, R., Zhu, X., Akgün, Ö., & Nacenta, M. A. (2022) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.cp.2022.28 · Cited by 10
The Influence of Icon Background Colors and Icon Symbols on Menu Item Selection for Smartphone
Punchoojit, L., & Hongwarittorrn, N. (2018) · NAFOSTED Conference on Information and Computer Science (NICS) · doi:10.1109/nics.2018.8606854 · Cited by 9
Morrow, E. L., Nelson, L. A., Duff, M. C., & Mayberry, L. S. (2024) · JMIR Rehabilitation and Assistive Technologies · doi:10.2196/59630 · Cited by 9
Nurrachman, A. P., Priyandari, Y., & Yuniaristanto, Y. (2022) · Techno Com · doi:10.33633/tc.v21i3.6307 · Cited by 9
Bonn, M. M., Graham, L. J., Marrocco, S. L., Jeske, S., Moran, B., & Wolfe, D. L. (2023) · Digital Health · doi:10.1177/20552076231183555 · Cited by 9
Santos‐Costa, P., Alves, M., Sousa, C., Sousa, L. B., Paiva-Santos, F., Bernardes, R. A., Ventura, F. I. Q. S., Salgueiro‐Oliveira, A., Parreira, P., Vieira, M. M. D. S., et al. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph191711130 · Cited by 9
Analysis and measurement of internal usability metrics through code annotations
Schramme, M., & Macías, J. A. (2019) · Software Quality Journal · doi:10.1007/s11219-019-09455-4 · Cited by 9
Martínez, J. G., Quiñones, I. T., & Robles, Á. S. (2011) · Edutec Revista Electrónica de Tecnología Educativa · doi:10.21556/edutec.2011.37.393 · Cited by 9
Gray, K. M., Clarke, K., Alzougool, B., Hines, C., Tidhar, G. A., & Frukhtman, F. (2014) · JMIR Research Protocols · doi:10.2196/resprot.3201 · Cited by 9
Metadata
- Authors
- Hwang, W., & Salvendy, G.
- Affiliations
- Soongsil University (KR)
- Purdue University West Lafayette (US)
- ORCID
- Hwang, Wonil
- Venue
- Communications of the ACM
- Published
- 2010-05
- Volume
- 53
- Issue
- 5
- Pages
- 130-133
- ISSN
- 0001-0782 (ISSN-L)
- 1557-7317
- OpenAlex
- W2016022053
- Cited by
- 410
- References
- 11
- Citations by year
- 2026: 40 · 2025: 36 · 2024: 26 · 2023: 43 · 2022: 27 · 2021: 21 · 2020: 29 · 2019: 30 · 2018: 22 · 2017: 24 · 2016: 23 · 2015: 15 · 2014: 24 · 2013: 17 · 2012: 17
- FWCI
- 31.16
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Communications of the ACM (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Keywords
- usability evaluation, sample size, think-aloud, heuristic evaluation, cognitive walkthrough, detection rate, evaluation costs, user interface design, verbal protocols, user testing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.