Number of people required for usability evaluation: the 10±2 rule
The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
Introduction Usability evaluation is essential to make sure that software products newly released are easy to use, efficient, and effective to reach goals, and satisfactory to users. For example, when a software company wants to develop and sell a new product, the company needs to evaluate usability of the new product before launching it at a market to avoid the possibility that the new product may contain usability problems, which span from cosmetic problems to severe functional problems. Three widely used methods for usability evaluation are Think Aloud (TA), Heuristic Evaluation (HE) and Cognitive Walkthrough (CW). TA method is commonly employed with a lab-based user testing, while there are variants of TA methods, including thinking out aloud at user's workplace instead of at labs. What we discuss here is the TA method that is combined with a lab-based user testing, in which test users use products while simultaneously and continuously thinking out aloud, and experimenters record users' behaviors and verbal protocols in the laboratory. HE is a usability inspection method, in which a small number of evaluators find usability problems in a user interface design by examining an interface and judging its compliance with well-known usability principles, called heuristics. CW is a theory-based method, in which evaluators evaluate every step necessary to perform a scenario-based task, and look for usability problems that would interfere with learning by exploration. These three methods have their own advantages and disadvantages. For instance, TA method provides good qualitative data from a small number of test users, but laboratory environment may influence test user's behaviors. HE is a cheap, fast and easy-to-use method, while it often finds too specific and low-priority usability problems, including even not real problems. CW helps find mismatches between users' and designers' conceptualization of a task, but it needs extensive knowledge of cognitive psychology and technical details to apply. However, even though these advantages and disadvantages show overall characteristics of three major usability evaluation methods, we cannot compare them quantitatively and see their efficiency clearly. Because one of reasons why so-called discounted methods, such as HE and CW, were developed is to save costs of usability evaluation, cost-related criteria for comparing usability evaluation are meaningful to usability practitioners as well as usability researchers. One of the most disputable issues related to cost of usability evaluation is sample size. That is, how many users or evaluators are needed to achieve a targeted usability evaluation performance, for example, 80% of overall discovery rate? The sample size of usability evaluation is known to depend on an estimate of problem discovery rate across participants. The overall discovery rate is a common quantitative measure that is used to show the effectiveness of a specific usability evaluation method in most of usability evaluation studies. It is also called overall detection rate or thoroughness measure, which is the ratio of 'the sum of unique usability problems detected by all experiment participants' against 'the number of usability problems that exist in the evaluated systems', ranging between 0 and 1. The overall discovery rates were reported more than any other criterion measure in the usability evaluation experiments and also a key component for projecting required sample size for usability evaluation study. Thus, how many test users or evaluators participate in the usability evaluation is a critical issue, considering its cost-effectiveness.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“However, a sample size of 10±2 is sufficient for discovering 80 per cent of usability issues [55].”
From A user-centred design framework for mHealth (2020) · cites this paper as [55]
Citations
Cites
Heuristic evaluation of user interfaces
Nielsen, J., & Molich, R. (1990) · Proceedings of the SIGCHI conference on Human factors in computing systems Empowering people - CHI '90 · doi:10.1145/97243.97281 · Cited by 3,533
Refining the Test Phase of Usability Evaluation: How Many Subjects Is Enough?
Virzi, R. A. (1992) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/001872089203400407 · Cited by 1,190
Estimating the number of subjects needed for a thinking aloud test
Nielsen, J. (1994) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1994.1065 · Cited by 476
Sample sizes for usability studies: Additional considerations
Lewis, J. R. (1994) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/001872089403600215 · Cited by 341
What is gained and lost when using evaluation methods other than empirical testing
Desurvire, H., Kondziela, J. M., & Atwood, M. E. (1993) · Cited by 107
Relaxing the homogeneity assumption in usability testing
Caulton, D. A. (2001) · Behaviour & Information Technology · doi:10.1080/01449290010020648 · Cited by 90
Law, L.-C., & Hvannberg, E. Þ. (2002) · Nordic Conference on Human-Computer Interaction · doi:10.1145/572020.572030 · Cited by 87
Evaluation of Procedures for Adjusting Problem-Discovery Rates Estimated From Small Samples
Lewis, J. R. (2001) · International Journal of Human-Computer Interaction · doi:10.1207/s15327590ijhc1304_06 · Cited by 72
Analysis of combinatorial user effect in international usability tests
Law, E. L., & Hvannberg, E. Þ. (2004) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/985692.985694 · Cited by 63
Novice heuristic evaluations of a complex interface
Slavković, A., & Cross, K. (1999) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/632716.632902 · Cited by 31
Cited by
Local Standards for Sample Size at CHI
Caine, K. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858498 · Cited by 657
Understanding Your Users: A Practical Guide to User Research Methods
Baxter, K. K., Courage, C., & Caine, K. E. (2015) · Cited by 356
Usability: Lessons Learned … and Yet to Be Learned
Lewis, J. R. (2014) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2014.930311 · Cited by 311
Improving service quality through customer journey analysis
Halvorsrud, R., Kvale, K., & Følstad, A. (2016) · Journal of Service Theory and Practice · doi:10.1108/jstp-05-2015-0111 · Cited by 267
USABILITY AND USER EXPERIENCE: DESIGN AND EVALUATION
Lewis, J. R., & Sauro, J. (2021) · Wiley eBooks · doi:10.1002/9781119636113.ch38 · Cited by 243
A user-centred design framework for mHealth
Farao, J., Malila, B., Conrad, N., Mutsvangwa, T., Rangaka, M. X., & Douglas, T. S. (2020) · PLoS ONE · doi:10.1371/journal.pone.0237910 · Cited by 203
“However, a sample size of 10±2 is sufficient for discovering 80 per cent of usability issues [55].”
BackgroundHermawati, S., & Lawson, G. (2016) · Applied Ergonomics · doi:10.1016/j.apergo.2015.11.016 · Cited by 196
Lyon, A. R., Munson, S. A., Renn, B. N., Atkins, D. C., Pullmann, M. D., Friedman, E., & Areán, P. A. (2019) · JMIR Research Protocols · doi:10.2196/14990 · Cited by 187
How many participants are really enough for usability studies?
Alroobaea, R., & Mayhew, P. J. (2014) · 2014 Science and Information Conference · doi:10.1109/sai.2014.6918171 · Cited by 178
Smartphone Technology and Apps: Rapidly Changing Health Promotion.
Kratzke, C., & Cox, C. C. (2012) · The international electronic journal of health education · Cited by 165
Exploring user experience of learning management system
Maslov, I., Nikou, S., & Hansen, P. (2021) · International Journal of Information and Learning Technology · doi:10.1108/ijilt-03-2021-0046 · Cited by 157
Some Whys and Hows of Experiments in Human—Computer Interaction
Hornbæk, K. (2013) · Foundations and Trends® in Human–Computer Interaction · doi:10.1561/1100000043 · Cited by 140
Sample size in usability studies
Schmettow, M. (2012) · Communications of the ACM · doi:10.1145/2133806.2133824 · Cited by 123
Dombek, M. (2014) · Dublin City University Open Access Institutional Repository (Dublin City University) · Cited by 114
Simpl: A toolkit for domain-specific language development in enterprise information systems
Freudenthal, M. (2013) · DSpace repository (University of Tartu) · Cited by 107
Academic methods for usability evaluation of serious games: a systematic review
Yáñez-Gómez, R., Cascado-Caballero, D., & Sevillano, J. L. (2016) · Multimedia Tools and Applications · doi:10.1007/s11042-016-3845-9 · Cited by 101
User Interface Inspection Methods
Wilson, C. (2014) · Elsevier eBooks · doi:10.1016/c2012-0-06519-2 · Cited by 98
Design guidelines for Classroom Multiplayer Presential Games (CMPG)
Páucar, M. A. V., Gajardo, I., Nussbaum, M., Andreu, J. J., Echeverría, A., & Plass, J. L. (2011) · Computers & Education · doi:10.1016/j.compedu.2011.05.003 · Cited by 96
A Survey on Adaptive Authentication
Arias-Cabarcos, P., Krupitzer, C., & Becker, C. (2019) · ACM Computing Surveys · doi:10.1145/3336117 · Cited by 89
Evaluating the perceived and estimated quality in use of Web 2.0 applications
Orehovački, T., Granić, A., & Kermek, D. (2013) · Journal of Systems and Software · doi:10.1016/j.jss.2013.05.071 · Cited by 87
The Expertise Effect on Web Accessibility Evaluation Methods
Brajnik, G., Yesilada, Y., & Harper, S. (2011) · Human–Computer Interaction · doi:10.1080/07370024.2011.601670 · Cited by 83
Guidance for publishing qualitative research in informatics
Ancker, J. S., Benda, N. C., Reddy, M., Unertl, K. M., & Veinot, T. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab195 · Cited by 82
Riihiaho, S. (2017) · Wiley eBooks · doi:10.1002/9781118976005.ch14 · Cited by 82
Salomoni, P., Prandi, C., Roccetti, M., Casanova, L., Marchetti, L., & Marfia, G. (2017) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-016-0236-5 · Cited by 80
Empirical validation of a usability inspection method for model-driven Web development
Fernandez, A., Abrahão, S., & Insfrán, E. (2012) · Journal of Systems and Software · doi:10.1016/j.jss.2012.07.043 · Cited by 77
Morotti, E., Stacchio, L., Donatiello, L., Roccetti, M., Tarabelli, J., & Marfia, G. (2021) · Virtual Reality · doi:10.1007/s10055-021-00602-6 · Cited by 76
Bridging the accessibility gap in Open Educational Resources
Navarrete, R., & Luján‐Mora, S. (2017) · Universal Access in the Information Society · doi:10.1007/s10209-017-0529-9 · Cited by 74
A Checklist to Evaluate Augmented Reality Applications
Guimarães, M. D. P., & Martins, V. F. (2014) · Symposium on Virtual and Augmented Reality (SVR) · doi:10.1109/svr.2014.17 · Cited by 74
Lyon, A. R., Koerner, K., & Chung, J. (2020) · Implementation Research and Practice · doi:10.1177/2633489520932924 · Cited by 73
Aesthetic perception analysis of destination pictures using #beautifuldestinations on Instagram
Hauser, D., Leopold, A., Egger, R., Ganewita, H., & Herrgessell, L. (2022) · Journal of Destination Marketing & Management · doi:10.1016/j.jdmm.2022.100702 · Cited by 71
Chen, J., Druhl, E., Ramesh, B. P., Houston, T. K., Brandt, C., Zulman, D. M., Vimalananda, V. G., Malkani, S., & Yu, H. Q. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.8669 · Cited by 70
How can usability contribute to user experience? a study in the domain of e-commerce
Moczarny, I. M., de Villiers, R., & van Biljon, J. A. (2012) · South African Institute of Computer Scientists and Information Technologists · doi:10.1145/2389836.2389862 · Cited by 68
Ferrari, A., Mazzanti, F., Basile, D., & ter Beek, M. H. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3124677 · Cited by 62
Systematic guidance on usability methods in user-centered software development
Pérez, L. C., & Macías, J. A. (2018) · Information and Software Technology · doi:10.1016/j.infsof.2018.01.010 · Cited by 61
Rationalization for explainable NLP: a survey
Gurrapu, S., Kulkarni, A., Huang, L., Lourentzou, I., & Batarseh, F. A. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1225093 · Cited by 58
Olde Keizer, R. A. C. M., van Velsen, L., Moncharmont, M., Riche, B., Ammour, N., Del Signore, S., Zia, G., Hermens, H., & N’Dja, A. (2019) · Health and Technology · doi:10.1007/s12553-019-00320-9 · Cited by 57
Doubleday, E. G., O’Loughlin, V. D., & Doubleday, A. F. (2011) · Anatomical Sciences Education · doi:10.1002/ase.252 · Cited by 56
Estimación del tamaño de la muestra para pruebas de usabilidad
Cazañas, A., De San Miguel, A., & Parra, E. (2017) · Enfoque UTE · doi:10.29019/enfoqueute.v8n1.126 · Cited by 52
“Several studies (Virzi, 1992), (Nielsen & Landauer, 1993), (Lewis, 1994), (Caulton, 2001), (Turner, Lewis, & Nielsen, 2006), (Hwang & Salvendy , 2010) suggest that when conducting user testing it is sufficient to use rules of thumb such as 4±1and 10±2 to estimate the number of users required.”
BackgroundIs accessibility conformance an elusive property? A study of validity and reliability of WCAG 2.0
Brajnik, G., Yesilada, Y., & Harper, S. (2012) · ACM Transactions on Accessible Computing · doi:10.1145/2141943.2141946 · Cited by 52
Seiler-Hwang, S., Arias-Cabarcos, P., Marín, A., Almenárez, F., Díaz-Sánchez, D., & Becker, C. (2019) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3319535.3354192 · Cited by 49
Supporting user-perceived usability benchmarking through a developed quantitative metric
Veral, R., & Macías, J. A. (2018) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2018.09.012 · Cited by 49
Usability Evaluation for Business Intelligence Applications: A User Support Perspective
Jooste, C., van Biljon, J. A., & Mentz, J. C. (2014) · South African Computer Journal · doi:10.18489/sacj.v53i0.198 · Cited by 49
Sun, C. J., Anderson, K. M., Kuhn, T., Mayer, L., & Klein, C. H. (2020) · JMIR mhealth and uhealth · doi:10.2196/15888 · Cited by 48
Design and Evaluation of Interactive Proofreading Tools for Connectomics
Haehn, D., Knowles-Barley, S., Roberts, M., Beyer, J., Kasthuri, N., Lichtman, J. W., & Pfister, H. (2014) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2014.2346371 · Cited by 47
Eye tracking scanpath analysis on web pages: how many users?
Eraslan, S., Yeşilada, Y., & Harper, S. (2016) · ACM Symposium on Eye Tracking Research and Applications (ETRA) · doi:10.1145/2857491.2857519 · Cited by 47
RuleKeeper: GDPR-Aware Personal Data Compliance for Web Frameworks
Ferreira, M., Brito, T., Santos, J. F., & Santos, N. (2023) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp46215.2023.10179395 · Cited by 47
Designing a Guardian Angel: Giving an Automated Vehicle the Possibility to Override its Driver
Maurer, S., Erbach, R., Kraiem, I., Kuhnert, S., Grimm, P., & Rukzio, E. (2018) · International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI) · doi:10.1145/3239060.3239078 · Cited by 46
AR Graphic Representation of Musical Notes for Self-Learning on Guitar
Guerra, M. S. D. R., Martín‐Gutiérrez, J., López-Chao, V., Parra, R. F., & Sosa, M. A. R. (2019) · Applied Sciences · doi:10.3390/app9214527 · Cited by 46
Sivaji, A., Abdollah, N., Tzuaan, S. S., Chuan, N. K., Nor, Z. M., Rasidi, S. H., & Yoong, S. W. (2014) · International Conference on User Science and Engineering · doi:10.1109/iuser.2014.7002677 · Cited by 45
Schuman-Olivier, Z., Borodovsky, J. T., Steinkamp, J., Munir, Q., Butler, K., Greene, M. A., Goldblatt, J., Xie, H. Y., & Marsch, L. A. (2018) · Addiction Science & Clinical Practice · doi:10.1186/s13722-018-0122-4 · Cited by 44
Fleming, J. B., Hill, Y. N., & Burns, M. N. (2017) · JMIR Human Factors · doi:10.2196/humanfactors.7392 · Cited by 44
Kim, H., Lee, S.-H., Cho, N.-B., You, H., Choi, T., & Kim, J. (2020) · JMIR mHealth and uHealth · doi:10.2196/19585 · Cited by 42
Wilansky, P., Eklund, J. M., Milner, T., Kreindler, D., Cheung, A., Kovacs, T., Shooshtari, S., Astell, A., Ohinmaa, A., Henderson, J., et al. (2016) · JMIR Research Protocols · doi:10.2196/resprot.5841 · Cited by 42
Jonathan, G., Dopke, C. A., Michaels, T., Bank, A., Martin, C. R., Adhikari, K., Krakauer, R. L., Ryan, C., McBride, A., Babington, P., et al. (2021) · JMIR Mental Health · doi:10.2196/20424 · Cited by 41
Combining human and machine intelligence for clinical trial eligibility querying
Fang, Y., Idnay, B., Sun, Y., Liu, H., Chen, Z., Marder, K. S., Xu, H., Schnall, R., & Weng, C. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac051 · Cited by 41
Seinsche, J., de Bruin, E. D., Saibene, E., Rizzo, F., Carpinella, I., Ferrarin, M., Moza, S., Ritter, T., & Giannouli, E. (2023) · JMIR Human Factors · doi:10.2196/48845 · Cited by 41
Scorolli, C., Grasso, E. N., Stacchio, L., Armandi, V., Matteucci, G., & Marfia, G. (2023) · Computers in Human Behavior · doi:10.1016/j.chb.2023.107910 · Cited by 41
Alessa, T., Hawley, M., Alsulamy, N. A., & de Witte, L. (2021) · JMIR mhealth and uhealth · doi:10.2196/24177 · Cited by 40
AQUArIUM - A suite of software measures for HCI quality evaluation of ubiquitous mobile applications
Carvalho, R. M., Andrade, R. M. C., & de Oliveira, K. M. (2017) · Journal of Systems and Software · doi:10.1016/j.jss.2017.11.022 · Cited by 40
Rebsamen, S., Knols, R. H., Pfister, P. B., & de Bruin, E. D. (2019) · Frontiers in Physiology · doi:10.3389/fphys.2019.01019 · Cited by 39
“Participants were encouraged to “think aloud,” a procedure often used in video game development studies (Hwang and Salvendy, 2010; Pirker and Gütl, 2015; May, 2019), while operating the software and playing the exergames (Wuest et al., 2014a; Eccles and Arsal, 2017).”
MethodPoppe, L., der Mispel, C. V., Bourdeaudhuij, I. D., Verloigne, M., Shadid, S., & Crombez, G. (2017) · PLoS ONE · doi:10.1371/journal.pone.0190020 · Cited by 38
Three patterns for user involvement in startups
Hokkanen, L., & Leppänen, M. (2015) · European Conference on Pattern Languages of Programs · doi:10.1145/2855321.2855373 · Cited by 38
Eicher, C., Haesner, M., Spranger, M., Kuzmicheva, O., Gräser, A., & Steinhagen-Thiessen, E. (2019) · Assistive Technology · doi:10.1080/10400435.2017.1352051 · Cited by 37
Schmettow, M., Vos, W. M., & Schraagen, J. M. (2013) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2013.04.007 · Cited by 37
Developing a Low-Cost IoT-Based Remote Cardiovascular Patient Monitoring System in Cameroon
Armand, T. P. T., Mozumder, M. A. I., Ali, S., Amaechi, A. O., & Kim, H. (2023) · Healthcare · doi:10.3390/healthcare11020199 · Cited by 37
EMGuitar: Assisting Guitar Playing with Electromyography
Karolus, J., Schuff, H., Kosch, T., Woźniak, P. W., & Schmidt, A. (2018) · Designing Interactive Systems Conference · doi:10.1145/3196709.3196803 · Cited by 36
Kim, J. C., Saguna, S., & Åhlund, C. (2023) · JMIR Human Factors · doi:10.2196/42145 · Cited by 36
Assessing the quality of mobile graphical user interfaces using multi-objective optimization
Soui, M., Chouchane, M., Mkaouer, M. W., Kessentini, M., & Ghédira, K. (2019) · Soft Computing · doi:10.1007/s00500-019-04391-8 · Cited by 35
Barbara, A. M., Dobbins, M., Haynes, R. B., Iorio, A., Lavis, J. N., Raina, P. S., & Levinson, A. J. (2016) · JMIR Human Factors · doi:10.2196/humanfactors.4800 · Cited by 35
Stay Vigilant: The Threat of a Replication Crisis in VR Locomotion Research
Zielasko, D., & Weißker, T. (2023) · ACM Symposium on Virtual Reality Software and Technology (VRST) · doi:10.1145/3611659.3615697 · Cited by 34
Bhattacharjya, S., Stafford, M., Cavuoto, L. A., Yang, Z., Song, C., Subryan, H., Xu, W., & Langan, J. (2019) · Journal of NeuroEngineering and Rehabilitation · doi:10.1186/s12984-019-0592-y · Cited by 34
Student Searching with EBSCO Discovery: A Usability Study
Owens, E. E., Jones, G., McMain, L. M., Shen, L., & Vieira, S. (2014) · Journal of Electronic Resources Librarianship · doi:10.1080/1941126x.2014.877331 · Cited by 34
A Biofeedback App for Migraine: Development and Usability Study
Ingvaldsen, S. H., Tronvik, E., Brenner, E., Winnberg, I. G., Olsen, A., Gravdahl, G. B., & Stubberud, A. (2021) · JMIR Formative Research · doi:10.2196/23229 · Cited by 33
FORMATIVE EVALUATION OF BUSINESS MODEL REPRESENTATIONS - THE SERVICE BUSINESS MODEL CANVAS
Zolnowski, A., & Böhmann, T. (2014) · AIS Electronic Library (AISeL) · Cited by 33
Hit the Thumb Jack! Using Electromyography to Augment the Piano Keyboard
Karolus, J., Kilian, A., Kosch, T., Schmidt, A., & Woźniak, P. W. (2020) · Designing Interactive Systems Conference · doi:10.1145/3357236.3395500 · Cited by 32
Kim, W., Lee, S., Kim, S., Jo, S., Yoo, C., Hwang, I., Kang, S., & Song, J. (2020) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3411815 · Cited by 32
Characteristics of Users and Nonusers of Symptom Checkers in Germany: Cross-Sectional Survey Study
Kopka, M., Scatturin, L., Napierala, H., Fürstenau, D., Feufel, M. A., Balzer, F., & Schmieding, M. L. (2023) · Journal of Medical Internet Research · doi:10.2196/46231 · Cited by 32
Usability and user evaluation of an integrated multimedia e-learning management system
Rupere, T., Jakovljević, M., & |, |. (2021) · Knowledge Management & E-Learning An International Journal · doi:10.34105/j.kmel.2021.13.018 · Cited by 32
Guided Proofreading of Automatic Segmentations for Connectomics
Haehn, D., Kaynig, V., Tompkin, J., Lichtman, J. W., & Pfister, H. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00971 · Cited by 31
Lazar, J. K., Feng, J. H., & Hochheiser, H. (2017) · Elsevier eBooks · doi:10.1016/b978-0-12-805390-4.00010-8 · Cited by 31
Integrating Blink Click interaction into a head tracking system: implementation and usability issues
Krapic, L., Lenac, K., & Ljubić, S. (2013) · Universal Access in the Information Society · doi:10.1007/s10209-013-0343-y · Cited by 31
eTACTS: A method for dynamically filtering clinical trial search results
Miotto, R., Jiang, S. Y., & Weng, C. (2013) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2013.07.014 · Cited by 31
Web‐based 3D medical image visualization framework for biomedical engineering education
Settapat, S., Achalakul, T., & Ohkura, M. (2011) · Computer Applications in Engineering Education · doi:10.1002/cae.20548 · Cited by 31
Heuristic usability evaluation on games: a modular approach
Yáñez-Gómez, R., Font, J. L., Cascado-Caballero, D., & Sevillano, J. L. (2018) · Multimedia Tools and Applications · doi:10.1007/s11042-018-6593-1 · Cited by 30
Responsible use of AI in military systems: prospects and challenges
Schraagen, J. M. (2023) · Ergonomics · doi:10.1080/00140139.2023.2278394 · Cited by 30
Wijayarathna, C., & Arachchilage, N. A. G. (2018) · Computers & Security · doi:10.1016/j.cose.2018.09.007 · Cited by 30
Usability Study of a Multicomponent Exergame Training for Older Adults with Mobility Limitations
Thalmann, M., Ringli, L., Adcock, M., Swinnen, N., de Jong, J., Dumoulin, C. L., Guimar�ães, V., & de Bruin, E. D. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph182413422 · Cited by 30
A Brief Online Implicit Bias Intervention for School Mental Health Clinicians
Liu, F. F., Coifman, J., McRee, E., Stone, J., Law, A., Gaias, L. M., Reyes, R., Lai, C. K., Blair, I. V., Yu, C.-L., et al. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19020679 · Cited by 29
Weermeijer, J. D. M., Wampers, M., de Thurah, L., Bonnier, R., Piot, M., Kuppens, P., Myin‐Germeys, I., & Kiekens, G. (2023) · JMIR Formative Research · doi:10.2196/48821 · Cited by 29
Jones, C. E., & Weber, P. (2012) · Transactions in GIS · doi:10.1111/j.1467-9671.2012.01319.x · Cited by 29
Feasibility and usability of an ontology-based mobile intervention for patients with hypertension
Wheeler, T. S., Vallis, M., Giacomantonio, N., & Abidi, S. (2018) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2018.08.002 · Cited by 28
Choosing a User Experience Research Activity
Baxter, K. K., Courage, C., & Caine, K. E. (2015) · Elsevier eBooks · doi:10.1016/b978-0-12-800232-2.00005-5 · Cited by 27
Usability Evaluation of a Private Social Network on Mental Health for Relatives
Toribio-Guzmán, J. M., García‐Holgado, A., Soto‐Pérez, F., García‐Peñalvo, F. J., & Franco, M. (2017) · Journal of Medical Systems · doi:10.1007/s10916-017-0780-x · Cited by 27
Biofeedback Treatment App for Pediatric Migraine: Development and Usability Study
Stubberud, A., Tronvik, E., Olsen, A., Gravdahl, G. B., & Linde, M. (2020) · Headache The Journal of Head and Face Pain · doi:10.1111/head.13772 · Cited by 27
Using simulation to aid decision making in managing the usability evaluation process
Rodríguez, N. H., Ruiz, M., Orta, E., & Valderrama, J. T. (2014) · Information and Software Technology · doi:10.1016/j.infsof.2014.06.001 · Cited by 27
Cloud services evaluation framework
Reixa, M. C. G., Costa, C., & Aparício, M. (2012) · Workshop on Open Source and Design of Communication (OSDOC) · doi:10.1145/2316936.2316948 · Cited by 27
Methods of usability evaluations of mobile devices
Alshehri, F., & Freeman, M. B. (2012) · Research Online (University of Wollongong) · Cited by 25
The accuracy and usability of point-of-use fluoride biosensors in rural Kenya
Thavarajah, W., Owuor, P. M., Awuor, D. R., Kiprotich, K., Aggarwal, R., Lucks, J. B., & Young, S. L. (2023) · npj Clean Water · doi:10.1038/s41545-023-00221-5 · Cited by 25
Schoormann, T., Behrens, D., Michael Fellmann, M. A., & Knackstedt, R. (2021) · Lecture notes in information systems and organisation · doi:10.1007/978-3-030-86790-4_38 · Cited by 24
Shegog, R., Armistead, L., Markham, C. M., Dube, S., Song, H.-Y., Chaudhary, P., Spencer, A., Peskin, M. F., Maria, D. S., Wilkerson, J. M., et al. (2021) · JMIR Serious Games · doi:10.2196/23088 · Cited by 24
Metadata
- Authors
- Hwang, W., & Salvendy, G.
- Affiliations
- Soongsil University (KR)
- Purdue University West Lafayette (US)
- ORCID
- Hwang, Wonil
- Venue
- Communications of the ACM
- Published
- 2010-05
- Volume
- 53
- Issue
- 5
- Pages
- 130-133
- ISSN
- 0001-0782 (ISSN-L)
- 1557-7317
- OpenAlex
- W2016022053
- Cited by
- 410
- References
- 11
- Citations by year
- 2026: 40 · 2025: 36 · 2024: 26 · 2023: 43 · 2022: 27 · 2021: 21 · 2020: 29 · 2019: 30 · 2018: 22 · 2017: 24 · 2016: 23 · 2015: 15 · 2014: 24 · 2013: 17 · 2012: 17
- FWCI
- 31.16
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Communications of the ACM (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Keywords
- usability evaluation, sample size, think-aloud, heuristic evaluation, cognitive walkthrough, detection rate, evaluation costs, user interface design, verbal protocols, user testing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.