CoChat
Sign Up
Cited by 1,961Usability and User Interface DesignSoftware Engineering Techniques and PracticesHuman-Automation Interaction and Safety

A mathematical model of the finding of usability problems

Open access
Nielsen, Jakob, Landauer, Thomas K.·1993·Proceedings of the SIGCHI conference on Human factors in computing systems - CHI '93
DOI 10.1145/169059.169166
Open this paper in CoChat — free accountOpen PDF

Abstract

For 11 studies, we find that the detection of usability problems as a function of number of users tested or heuristic evaluators employed is well modeled as a Poisson process. The model can be used to plan the amount of evaluation required to achieve desired levels of thoroughness or benefits. Results of early tests can provide estimates of the number of problems left to be found and the number of additional evaluations needed to find a given fraction. With quantitative evaluation costs and detection values, the model can estimate the numbers of evaluations at which optimal cost/benefit ratios are obtained and at which marginal utility vanishes. For a “medium” example, we estimate that 16 evaluations would be worth their cost, with maximum benefit/cost ratio at four.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/169059.169166

Cites

30 of 30 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The recommended number of participants for usability testing is at least 10 people.

    “The recommended number of participants for usability testing is at least 10 people [14].”

    From The PAediatric Risk Assessment (PARA) Mobile App to Reduce Postdischarge Child Mortality: Design, Usability, and Feasibility for Health Care Workers in Uganda (2016) · cites this paper as [14]

    In “A Priori Estimates of the Optimum Number of Evaluators and Test Users,” Table 3 estimates seven test users for a small project, below the claimed minimum of 10. The same section gives 3.2 test users as a rough estimate for each iteration of a design. The paper therefore does not recommend at least 10 participants for usability testing in general. (LLM, conf=0.88)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • There is some consensus that usability tests may include a minimum of 5 participants per iteration and that approximately 80% of usability issues are discovered with as few as 4-6 participants.

    “Nevertheless, there is some consensus that usability tests may include a minimum of 5 participants per iteration and that approximately 80% of usability issues are discovered with as few as 4-6 participants [80].”

    From Development and Evaluation of the Usefulness, Usability, and Feasibility of iNNOV Breast Cancer: Mixed Methods Study (2022) · cites this paper as [80]

    In “A Priori Estimates of the Optimum Number of Evaluators and Test Users” (page not specified), the paper mentions an earlier recommendation of about five test users and models the effort per design iteration. Its finding model and mean single-user detection probability of about 0.31 are consistent with roughly 80% of problems being found by 4–6 users. It does not establish a consensus or prescribe five participants as a minimum; its modeled number per iteration varies with costs. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Four ICT experts meets the optimal requirement for detecting all usability problems.

    “which meets the optimal requirement for detecting all usability problems [78]”

    From Development and usability testing of a web-based and therapist-assisted coping skills program for managing psychosocial problems in individuals with hand and upper limb injuries: Mixed methods study (2020) · cites this paper as [78]

    In “A Priori Estimates of the Optimum Number of Evaluators and Test Users” (page not provided), the paper estimates that 4.4 heuristic evaluators give the highest benefit-to-cost ratio under its assumptions, as a rough guide for each design iteration. It does not say that four ICT experts will detect all usability problems; it also gives larger optimal evaluator numbers for other project scenarios. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 2

  1. and Knowledge Based Systems. Amsterdam: Elsevier Science Publishers, 743-750.
  2. Nielsen, J. (1993). Iterative design of user interfaces. IEEE Computer 26 (to appear, probably in the July issue).

Cited by

Sorted by citations

Related

Authors
Nielsen, J., & Landauer, T. K.
ORCID
Nielsen, Jakob
Venue
Proceedings of the SIGCHI conference on Human factors in computing systems - CHI '93
Publisher
ACM Press
Published
1993
Pages
206-213
DOI
10.1145/169059.169166
OpenAlex
W2099473180
Cited by
1,961
References
29
Citations by year
2026: 110 · 2025: 125 · 2024: 123 · 2023: 121 · 2022: 128 · 2021: 129 · 2020: 108 · 2019: 92 · 2018: 93 · 2017: 96 · 2016: 82 · 2015: 92 · 2014: 76 · 2013: 76 · 2012: 65
FWCI
6.58
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
CHI Conference on Human Factors in Computing Systems (publishedVersion)
Topics
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
Keywords
usability evaluation, heuristic evaluation, Poisson process model, evaluation planning, user testing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review