The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
A mathematical model of the finding of usability problems
Open accessAbstract
For 11 studies, we find that the detection of usability problems as a function of number of users tested or heuristic evaluators employed is well modeled as a Poisson process. The model can be used to plan the amount of evaluation required to achieve desired levels of thoroughness or benefits. Results of early tests can provide estimates of the number of problems left to be found and the number of additional evaluations needed to find a given fraction. With quantitative evaluation costs and detection values, the model can estimate the numbers of evaluations at which optimal cost/benefit ratios are obtained and at which marginal utility vanishes. For a “medium” example, we estimate that 16 evaluations would be worth their cost, with maximum benefit/cost ratio at four.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“The recommended number of participants for usability testing is at least 10 people [14].”
From The PAediatric Risk Assessment (PARA) Mobile App to Reduce Postdischarge Child Mortality: Design, Usability, and Feasibility for Health Care Workers in Uganda (2016) · cites this paper as [14]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Nevertheless, there is some consensus that usability tests may include a minimum of 5 participants per iteration and that approximately 80% of usability issues are discovered with as few as 4-6 participants [80].”
From Development and Evaluation of the Usefulness, Usability, and Feasibility of iNNOV Breast Cancer: Mixed Methods Study (2022) · cites this paper as [80]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“which meets the optimal requirement for detecting all usability problems [78]”
From Development and usability testing of a web-based and therapist-assisted coping skills program for managing psychosocial problems in individuals with hand and upper limb injuries: Mixed methods study (2020) · cites this paper as [78]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(1993) · Cited by 9,879
Self-citation(1993) · Elsevier eBooks · doi:10.1016/c2009-0-21512-1 · Cited by 5,552
Heuristic evaluation of user interfaces
Nielsen, J., & Molich, R. (1990) · Proceedings of the SIGCHI conference on Human factors in computing systems Empowering people - CHI '90 · doi:10.1145/97243.97281 · Cited by 3,533
Self-citationThe Usability Engineering Lifecycle
NIELSEN, J. (1993) · Usability Engineering · doi:10.1016/b978-0-08-052029-2.50007-3 · Cited by 2,657
Self-citationNielsen, J. (1995) · Conference companion on Human factors in computing systems - CHI '95 · doi:10.1145/223355.223730 · Cited by 2,620
Self-citationIntroduction to Stochastic Processes
Toral, R., & Colet, P. (2014) · Wiley eBooks · doi:10.1002/9783527683147.ch6 · Cited by 2,406
Nielsen, J. (1994) · Cited by 1,778
Self-citationRefining the Test Phase of Usability Evaluation: How Many Subjects Is Enough?
Virzi, R. A. (1992) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1177/001872089203400407 · Cited by 1,269
Finding usability problems through heuristic evaluation
Nielsen, J. (1992) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/142750.142834 · Cited by 1,139
Self-citationImproving a human-computer dialogue
Molich, R., & Nielsen, J. (1990) · Communications of the ACM · doi:10.1145/77481.77486 · Cited by 821
Self-citationUser interface evaluation in the real world
Jeffries, R., Miller, J. R., Wharton, C., & Uyeda, K. M. (1991) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/108844.108862 · Cited by 619
The usability engineering life cycle
Nielsen, J. (1992) · Computer · doi:10.1109/2.121503 · Cited by 581
Self-citationEstimating the number of subjects needed for a thinking aloud test
Nielsen, J. (1994) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1994.1065 · Cited by 476
Self-citationIterative user-interface design
Nielsen, J. (1993) · Computer · doi:10.1109/2.241424 · Cited by 444
Self-citationComparison of empirical testing and walkthrough methods in user interface evaluation
Karat, C.-M., Campbell, R., & Fiegel, T. (1992) · ACM SIGCHI Conference on Human Factors in Computing Systems (CHI) · doi:10.1145/142750.142873 · Cited by 301
Introduction to Stochastic Processes
Pavliotis, G. A. (2014) · Texts in applied mathematics · doi:10.1007/978-1-4939-1323-7_1 · Cited by 285
Usability engineering at a discount
Nielsen, J. (1989) · International Conference on Human-Computer Interaction · Cited by 240
Self-citationWhen Should One Stop Testing Software?
Dalal, S. R., & Mallows, C. L. (1988) · Journal of the American Statistical Association · doi:10.1080/01621459.1988.10478676 · Cited by 196
Cost/benefit analysis for incorporating human factors in the software lifecycle
Mantei, M. M., & Teorey, T. J. (1988) · Communications of the ACM · doi:10.1145/42404.42408 · Cited by 181
The precis of Project Ernestine or an overview of a validation of GOMS
Gray, W. D., John, B. E., & Atwood, M. E. (1992) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/142750.142821 · Cited by 152
Introduction to Stochastic Processes
Ray, W. D. (1976) · Journal of the Operational Research Society · doi:10.1057/jors.1976.47 · Cited by 128
The usability engineering lifecycle
Mayhew, D. J. (1999) · CHI '01 extended abstracts on Human factors in computer systems - CHI '01 · doi:10.1145/632780.632805 · Cited by 125
Streamlining the Design Process: Running Fewer Subjects
Virzi, R. A. (1990) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/154193129003400411 · Cited by 119
A cost-effective evaluation method for use by designers
Wright, P. C., & Monk, A. F. (1991) · International Journal of Man-Machine Studies · doi:10.1016/s0020-7373(05)80167-1 · Cited by 111
Designing and using human-computer interfaces and knowledge based systems
Salvendy, G., & Smith, M. J. (1989) · Elsevier eBooks · Cited by 95
Some graphical aids for deciding when to stop testing software
Dalal, S. R., & Mallows, C. L. (1990) · IEEE Journal on Selected Areas in Communications · doi:10.1109/49.46868 · Cited by 85
Evaluating the thinking-aloud technique for use by computer scientists
Nielsen, J. (1993) · Ablex Publishing Corp. eBooks · Cited by 81
Self-citationIntegrated office software benchmarks: A case study
Lewis, J. R., Henry, S. C., & Mack, R. L. (1990) · International Conference on Human-Computer Interaction · Cited by 31
Using the thinking-aloud method in system development
Jørgensen, A. H. (1989) · International Conference on Human-Computer Interaction · Cited by 30
When Should One Stop Testing Software?
Dalal, S. R., & Mallows, C. L. (1988) · Journal of the American Statistical Association · doi:10.2307/2289319 · Cited by 28
Flagged first, then by citations
Research works printed but not linked: 2
- and Knowledge Based Systems. Amsterdam: Elsevier Science Publishers, 743-750.
- Nielsen, J. (1993). Iterative design of user interfaces. IEEE Computer 26 (to appear, probably in the July issue).
Cited by
Sorted by citations
How Many Interviews Are Enough?: An Experiment with Data Saturation and Variability
Guest, G., Bunce, A., & Johnson, L. (2006) · Field Methods · doi:10.1177/1525822x05279903 · Cited by 20,808
(1993) · Cited by 9,879
Self-citationNielsen, J. (1994) · Conference companion on Human factors in computing systems - CHI '94 · doi:10.1145/259963.260531 · Cited by 2,759
Self-citationNielsen, J. (1995) · Conference companion on Human factors in computing systems - CHI '95 · doi:10.1145/223355.223730 · Cited by 2,620
Self-citationMethods to support human-centred design
MAGUIRE, M. (2001) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.2001.0503 · Cited by 1,057
Metadata
- Authors
- Nielsen, J., & Landauer, T. K.
- ORCID
- Nielsen, Jakob
- Venue
- Proceedings of the SIGCHI conference on Human factors in computing systems - CHI '93
- Publisher
- ACM Press
- Published
- 1993
- Pages
- 206-213
- OpenAlex
- W2099473180
- Cited by
- 1,961
- References
- 29
- Citations by year
- 2026: 110 · 2025: 125 · 2024: 123 · 2023: 121 · 2022: 128 · 2021: 129 · 2020: 108 · 2019: 92 · 2018: 93 · 2017: 96 · 2016: 82 · 2015: 92 · 2014: 76 · 2013: 76 · 2012: 65
- FWCI
- 6.58
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Keywords
- usability evaluation, heuristic evaluation, Poisson process model, evaluation planning, user testing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
