Utility of human-computer interactions: toward a science of preference measurement
The DOI matches its Crossref record; 25 of the 26 references it cites were found in a registry or an index.
Abstract
The success of a computer system depends upon a user choosing it, but the field of Human-Computer Interaction has little ability to predict this user choice. We present a new method that measures user choice, and quantifies it as a measure of utility. Our method has two core features. First, it introduces an economic definition of utility, one that we can operationalize through economic experiments. Second, we employ a novel method of crowdsourcing that enables the collection of thousands of economic judgments from real users.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
(1993) · Cited by 9,879
An Inquiry into the Nature and Causes of the Wealth of Nations
Smith, A., Cannan, E., & Stigler, G. J. (1977) · University of Chicago Press eBooks · doi:10.7208/chicago/9780226763750.001.0001 · Cited by 6,806
Journal of the American Statistical Association
Koehler, A. B. (1995) · International Journal of Forecasting · doi:10.1016/0169-2070(95)90067-5 · Cited by 5,337
Probable Inference, the Law of Succession, and Statistical Inference
Wilson, E. B. (1927) · Journal of the American Statistical Association · doi:10.1080/01621459.1927.10502953 · Cited by 4,476 · cited 1×
“We compute 95% confidence intervals using Wilson’s estimate [24], since survival data is binomial.”
MethodMiller, R. G. (1981) · Cited by 3,336
Science and Engineering Indicators
Washington, E., & Warren, M. E. (2004) · Cited by 2,822
Predictably irrational: the hidden forces that shape our decisions
(2008) · Choice Reviews Online · doi:10.5860/choice.46-0969 · Cited by 2,514 · cited 1×
“Second, the researcher must be careful to avoid situations where the use of a money incentive adversely effects one’s decision process, as has been recorded in Behavioral Economics [1].”
BackgroundLabeling images with a computer game
von Ahn, L., & Dabbish, L. (2004) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/985692.985733 · Cited by 2,204 · cited 1×
“And some software is designed specifically to give new goals to users: video gamers pay money to find compelling tasks to spend their time on; consumers purchase persuasive products, such as Nike+ iPod, hoping to influence their choices in life; Human Computation and Crowdsourcing applications find ways to wrap tasks…”
BackgroundRising Above The Gathering Storm: Energizing and Employing America for a Brighter Economic Future
Augustine, N. R. (2006) · Cited by 2,064
Crowdsourcing user studies with Mechanical Turk
Kittur, A., Chi, E. H., & Suh, B. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357127 · Cited by 1,923 · cited 1×
“[13], Heer and Bostock [7], and a growing number of others.”
BackgroundSurvival Analysis: Techniques for Censored and Truncated Data
Klein, J. P., & Moeschberger, M. L. (2003) · Statistics for Biology and Health · doi:10.1007/b97377 · Cited by 1,816
Fitts' Law as a Research and Design Tool in Human-Computer Interaction
MacKenzie, I. S. (1992) · Human–Computer Interaction · doi:10.1207/s15327051hci0701_3 · Cited by 1,637 · cited 1×
“Consider the classic Fitts’ Law experiment [16].”
Predicting protein structures with a multiplayer online game
Cooper, S., Khatib, F., Treuille, A., Barbero, J., Lee, J., Beenen, M., Leaver-Fay, A., Baker, D., Popović, Z., & players, F. (2010) · Nature · doi:10.1038/nature09304 · Cited by 1,368
Who are the crowdworkers?: shifting demographics in mechanical turk
Ross, J., Irani, L., Silberman, M. S., Zaldivar, A., & Tomlinson, B. (2010) · CHI '10 Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/1753846.1753873 · Cited by 1,004
Quality management on Amazon Mechanical Turk
Ipeirotis, P. G., Provost, F., & Wang, J. (2010) · Proceedings of the ACM SIGKDD Workshop on Human Computation · doi:10.1145/1837885.1837906 · Cited by 973 · cited 1×
“This is the basis of a growing body of quality-assurance techniques used on Mechanical Turk [11].”
MethodAnalyzing the Amazon Mechanical Turk marketplace
Ipeirotis, P. G. (2010) · XRDS: Crossroads, The ACM Magazine for Students · doi:10.1145/1869086.1869094 · Cited by 879 · cited 1×
“A substantial portion of workers have bachelors, masters, and PhD degrees, for instance [9,19].”
BackgroundCrowdsourcing graphical perception: using mechanical turk to assess visualization design
Heer, J., & Bostock, M. (2010) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753357 · Cited by 816 · cited 1×
“[13], Heer and Bostock [7], and a growing number of others.”
BackgroundCapet, M.-F., & Lloyd, C. (1968) · Revue économique · doi:10.2307/3498855 · Cited by 664
The labor economics of paid crowdsourcing
Horton, J. J., & Chilton, L. B. (2010) · Proceedings of the 11th ACM conference on Electronic commerce · doi:10.1145/1807342.1807376 · Cited by 601 · cited 3×
“Horton and Chilton [8] measured a worker’s reservation wage — the wage below which she will not do a task—by starting with a high wage and incrementally reducing it until the worker quit.”
BackgroundInfluentialHeuristics for designing enjoyable user interfaces: Lessons from computer games
Malone, T. W. (1982) · Proceedings of the 1982 conference on Human factors in computing systems - CHI '82 · doi:10.1145/800049.801756 · Cited by 454
Practical guide to controlled experiments on the web: listen to your customers not to the hippo
Kohavi, R., Henne, R. M., & Sommerfield, D. (2007) · Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1281192.1281295 · Cited by 377 · cited 2×
“" [15] By paying users, we have more discretion to ask them survey questions to elicit “why,” and we also gain the ability to test abstract interfaces and tasks—such as our Fitts’ law interface—to isolate factors and remove confounds of real-world websites.”
BackgroundUtility and usability: research issues and development contexts
Grudin, J. (1992) · Interacting with Computers · doi:10.1016/0953-5438(92)90005-z · Cited by 220 · cited 2×
“The CHI research community grew from the need to understand discretionary use of computer interfaces [5].”
BackgroundEconomic and subjective measures of the perceived value of aesthetics and usability
Ben-Bassat, T., Meyer, J., & Tractinsky, N. (2006) · ACM Transactions on Computer-Human Interaction · doi:10.1145/1165734.1165737 · Cited by 172 · cited 2×
“[2] used a traditional second-price auction to determine user preference for interfaces.”
MethodGreene, W. H., Econometric analysis
Webel, K. (2010) · Statistical Papers · doi:10.1007/s00362-010-0315-8 · Cited by 39
Grudin, J. T., & MacLean, A. (1985) · Cited by 15
Research works printed but not linked: 1
- Simon , H.A. "Designing Organizations for an Information-Rich World. in Martin Greenberger," Computers, Communication, and the Public Interest , Baltimore, MD : The Johns Hopkins Press ( 1971 ). Simon, H.A. "Designing Organizations for an Information-Rich World. in Martin Greenberger," Computers, Communication, and the Public Interest, Baltimore, MD: The Johns Hopkins Press (1971).
Cited by
Kelley, P. G., Komanduri, S., Mazurek, M. L., Shay, R., Vidas, T., Bauer, L., Christin, N., Cranor, L. F., & Lopez, J. (2012) · 2012 IEEE Symposium on Security and Privacy · doi:10.1109/sp.2012.38 · Cited by 397
Explanations Can Reduce Overreliance on AI Systems During Decision-Making
Vasconcelos, H., Jörke, M., Grunde-McLaughlin, M., Gerstenberg, T., Bernstein, M. S., & Krishna, R. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579605 · Cited by 402
Crowds in two seconds: enabling realtime crowd-powered interfaces
Bernstein, M. S., Brandt, J., Miller, R. C., & Karger, D. R. (2011) · Proceedings of the 24th annual ACM symposium on User interface software and technology · doi:10.1145/2047196.2047201 · Cited by 302
“Toomim [25] used crowds to evaluate alternatives more thoroughly, though with much longer latency.”
MethodHow does your password measure up? the effect of strength meters on password creation
Ur, B., Kelley, P. G., Komanduri, S., Lee, J., Maass, M., Mazurek, M. L., Passaro, T., Shay, R., Vidas, T. M., Bauer, L., et al. (2012) · USENIX Security Symposium · Cited by 366
Real-time crowd control of existing interfaces
Lasecki, W. S., Murray, K. I., White, S., Miller, R. C., & Bigham, J. P. (2011) · Proceedings of the 24th annual ACM symposium on User interface software and technology · doi:10.1145/2047196.2047200 · Cited by 164
“These can be measured empirically for a specific crowd source, but are expected to be task-dependent [21].”
Background
Metadata
- Authors
- Toomim, M., Kriplean, T., Pörtner, C., & Landay, J.
- Affiliations
- Microsoft (United States) (US)
- University of Washington (US)
- Venue
- Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2011-05-07
- Pages
- 2275-2284
- OpenAlex
- W2137244916
- Cited by
- 64
- References
- 26
- Citations by year
- 2025: 4 · 2024: 2 · 2023: 1 · 2022: 3 · 2021: 2 · 2020: 1 · 2019: 3 · 2018: 7 · 2017: 3 · 2016: 6 · 2015: 6 · 2014: 7 · 2013: 10 · 2012: 5
- FWCI
- 6.20
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Keywords
- user choice, utility measurement, crowdsourcing, economic experiments, user preferences, human-computer interaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.