Utility of human-computer interactions: toward a science of preference measurement
The DOI matches its Crossref record; 25 of the 26 references it cites were found in a registry or an index.
Abstract
The success of a computer system depends upon a user choosing it, but the field of Human-Computer Interaction has little ability to predict this user choice. We present a new method that measures user choice, and quantifies it as a measure of utility. Our method has two core features. First, it introduces an economic definition of utility, one that we can operationalize through economic experiments. Second, we employ a novel method of crowdsourcing that enables the collection of thousands of economic judgments from real users.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
(1993) · Cited by 9,879
An Inquiry into the Nature and Causes of the Wealth of Nations
Smith, A., Cannan, E., & Stigler, G. J. (1977) · University of Chicago Press eBooks · doi:10.7208/chicago/9780226763750.001.0001 · Cited by 6,806
Journal of the American Statistical Association
Koehler, A. B. (1995) · International Journal of Forecasting · doi:10.1016/0169-2070(95)90067-5 · Cited by 5,337
Probable Inference, the Law of Succession, and Statistical Inference
Wilson, E. B. (1927) · Journal of the American Statistical Association · doi:10.1080/01621459.1927.10502953 · Cited by 4,476 · cited 1×
“We compute 95% confidence intervals using Wilson’s estimate [24], since survival data is binomial.”
MethodMiller, R. G. (1981) · Cited by 3,336
Science and Engineering Indicators
Washington, E., & Warren, M. E. (2004) · Cited by 2,822
Predictably irrational: the hidden forces that shape our decisions
(2008) · Choice Reviews Online · doi:10.5860/choice.46-0969 · Cited by 2,514 · cited 1×
“Second, the researcher must be careful to avoid situations where the use of a money incentive adversely effects one’s decision process, as has been recorded in Behavioral Economics [1].”
BackgroundLabeling images with a computer game
von Ahn, L., & Dabbish, L. (2004) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/985692.985733 · Cited by 2,204 · cited 1×
“And some software is designed specifically to give new goals to users: video gamers pay money to find compelling tasks to spend their time on; consumers purchase persuasive products, such as Nike+ iPod, hoping to influence their choices in life; Human Computation and Crowdsourcing applications find ways to wrap tasks…”
BackgroundRising Above The Gathering Storm: Energizing and Employing America for a Brighter Economic Future
Augustine, N. R. (2006) · Cited by 2,064
Crowdsourcing user studies with Mechanical Turk
Kittur, A., Chi, E. H., & Suh, B. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357127 · Cited by 1,923 · cited 1×
“[13], Heer and Bostock [7], and a growing number of others.”
Background
Research works printed but not linked: 1
- Simon , H.A. "Designing Organizations for an Information-Rich World. in Martin Greenberger," Computers, Communication, and the Public Interest , Baltimore, MD : The Johns Hopkins Press ( 1971 ). Simon, H.A. "Designing Organizations for an Information-Rich World. in Martin Greenberger," Computers, Communication, and the Public Interest, Baltimore, MD: The Johns Hopkins Press (1971).
Cited by
Kelley, P. G., Komanduri, S., Mazurek, M. L., Shay, R., Vidas, T., Bauer, L., Christin, N., Cranor, L. F., & Lopez, J. (2012) · 2012 IEEE Symposium on Security and Privacy · doi:10.1109/sp.2012.38 · Cited by 397
Explanations Can Reduce Overreliance on AI Systems During Decision-Making
Vasconcelos, H., Jörke, M., Grunde-McLaughlin, M., Gerstenberg, T., Bernstein, M. S., & Krishna, R. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579605 · Cited by 402
Crowds in two seconds: enabling realtime crowd-powered interfaces
Bernstein, M. S., Brandt, J., Miller, R. C., & Karger, D. R. (2011) · Proceedings of the 24th annual ACM symposium on User interface software and technology · doi:10.1145/2047196.2047201 · Cited by 302
“Toomim [25] used crowds to evaluate alternatives more thoroughly, though with much longer latency.”
MethodHow does your password measure up? the effect of strength meters on password creation
Ur, B., Kelley, P. G., Komanduri, S., Lee, J., Maass, M., Mazurek, M. L., Passaro, T., Shay, R., Vidas, T. M., Bauer, L., et al. (2012) · USENIX Security Symposium · Cited by 366
Real-time crowd control of existing interfaces
Lasecki, W. S., Murray, K. I., White, S., Miller, R. C., & Bigham, J. P. (2011) · Proceedings of the 24th annual ACM symposium on User interface software and technology · doi:10.1145/2047196.2047200 · Cited by 164
“These can be measured empirically for a specific crowd source, but are expected to be task-dependent [21].”
Background
Metadata
- Authors
- Toomim, M., Kriplean, T., Pörtner, C., & Landay, J.
- Affiliations
- Microsoft (United States) (US)
- University of Washington (US)
- Venue
- Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2011-05-07
- Pages
- 2275-2284
- OpenAlex
- W2137244916
- Cited by
- 64
- References
- 26
- Citations by year
- 2025: 4 · 2024: 2 · 2023: 1 · 2022: 3 · 2021: 2 · 2020: 1 · 2019: 3 · 2018: 7 · 2017: 3 · 2016: 6 · 2015: 6 · 2014: 7 · 2013: 10 · 2012: 5
- FWCI
- 6.20
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Keywords
- user choice, utility measurement, crowdsourcing, economic experiments, user preferences, human-computer interaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Utility of human-computer interactions: toward a science of preference measurement. https://app.cochat.ai/papers/10.1145/1978942.1979277
- The paper
- Toomim, M., Kriplean, T., Pörtner, C., & Landay, J. (2011). Utility of human-computer interactions: toward a science of preference measurement. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/1978942.1979277
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.