The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
Crowdsourcing user studies with Mechanical Turk
Abstract
User studies are important for many aspects of the design process and involve techniques ranging from informal surveys to rigorous laboratory studies. However, the costs involved in engaging users often requires practitioners to trade off between sample size, time requirements, and monetary costs. Micro-task markets, such as Amazon's Mechanical Turk, offer a potential paradigm for engaging a large number of users for low time and monetary costs. Here we investigate the utility of a micro-task market for collecting user measurements, and discuss design considerations for developing remote micro user evaluation tasks. Although micro-task markets have great potential for rapidly collecting user measurements at low costs, we found that special care is needed in formulating tasks in order to harness the capabilities of the approach.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Coase's penguin, or, linux and “the nature of the firm”
Benkler, Y. (2002) · The Yale Law Journal · doi:10.2307/1562247 · Cited by 1,572
Viégas, F. B., Wattenberg, M., & McKeon, M. M. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-73257-0_49 · Cited by 195
Cited by
VADER: a parsimonious rule-based model for sentiment analysis of social media text
Hutto, C., & Gilbert, E. (2014) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v8i1.14550 · Cited by 6,087
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Conducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172
Inside the Turk: Understanding Mechanical Turk as a Participant Pool
Paolacci, G., & Chandler, J. (2014) · Current Directions in Psychological Science · doi:10.1177/0963721414531598 · Cited by 2,136
Cheap and fast---but is it good?: evaluating non-expert annotations for natural language tasks
Snow, R., O'Connor, B., Jurafsky, D., & Ng, A. Y. (2008) · Proceedings of the Conference on Empirical Methods in Natural Language Processing - EMNLP '08 · doi:10.3115/1613715.1613751 · Cited by 1,887
Hauser, D. J., & Schwarz, N. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0578-z · Cited by 1,739
Everyone's an influencer: quantifying influence on twitter
Bakshy, E., Hofman, J. M., Mason, W. A., & Watts, D. J. (2011) · Proceedings of the fourth ACM international conference on Web search and data mining · doi:10.1145/1935826.1935845 · Cited by 1,689
Using Mechanical Turk to Study Clinical Populations
Shapiro, D. N., Chandler, J., & Mueller, P. A. (2013) · Clinical Psychological Science · doi:10.1177/2167702612469015 · Cited by 1,139
Conducting Clinical Research Using Crowdsourced Convenience Samples
Chandler, J., & Shapiro, D. (2016) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-021815-093623 · Cited by 1,072
An MTurk Crisis? Shifts in Data Quality and the Impact on Study Results
Chmielewski, M., & Kucker, S. C. (2020) · Social Psychological and Personality Science · doi:10.1177/1948550619875149 · Cited by 1,027
Who are the crowdworkers?: shifting demographics in mechanical turk
Ross, J., Irani, L., Silberman, M. S., Zaldivar, A., & Tomlinson, B. (2010) · CHI '10 Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/1753846.1753873 · Cited by 1,004
Kittur, A., Nickerson, J. V., Bernstein, M., Gerber, E., Shaw, A., Zimmerman, J., Lease, M., & Horton, J. (2013) · Proceedings of the 2013 conference on Computer supported cooperative work · doi:10.1145/2441776.2441923 · Cited by 1,034
Self-citationChandler, J., Mueller, P., & Paolacci, G. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0365-7 · Cited by 950
Training deep networks for facial expression recognition with crowd-sourced label distribution
Barsoum, E., Zhang, C., Ferrer, C. C., & Zhang, Z. (2016) · Proceedings of the 18th ACM International Conference on Multimodal Interaction · doi:10.1145/2993148.2993165 · Cited by 926
A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems
Mohseni, S., Zarei, N., & Ragan, E. D. (2021) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3387166 · Cited by 785
A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems
Mohseni, S., Zarei, N., & Ragan, E. D. (2018) · arXiv · doi:10.48550/arxiv.1811.11839 · Cited by 783
Soylent: a word processor with a crowd inside
Bernstein, M. S., Little, G., Miller, R. C., Hartmann, B., Ackerman, M. S., Karger, D. R., Crowell, D., & Panovich, K. (2010) · Proceedings of the 23nd annual ACM symposium on User interface software and technology · doi:10.1145/1866029.1866078 · Cited by 686
VizWiz: nearly real-time answers to visual questions
Bigham, J. P., Jayant, C., Ji, H., Little, G., Miller, A., Miller, R. C., Miller, R., Tatarowicz, A., White, B., White, S., et al. (2010) · Proceedings of the 23nd annual ACM symposium on User interface software and technology · doi:10.1145/1866029.1866080 · Cited by 601
"I regretted the minute I pressed share": a qualitative study of regrets on Facebook
Wang, Y., Norcie, G., Komanduri, S., Acquisti, A., Leon, P. G., & Cranor, L. F. (2011) · Proceedings of the Seventh Symposium on Usable Privacy and Security · doi:10.1145/2078827.2078841 · Cited by 479
On the naturalness of software
Hindle, A., Barr, E. T., Su, Z., Gabel, M., & Devanbu, P. (2012) · 2012 34th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2012.6227135 · Cited by 425
Fair and Balanced? Quantifying Media Bias through Crowdsourced Content Analysis
Budak, C., Goel, S., & Rao, J. M. (2016) · Public Opinion Quarterly · doi:10.1093/poq/nfw007 · Cited by 336
Shepherding the crowd yields better work
Dow, S., Kulkarni, A., Klemmer, S., & Hartmann, B. (2012) · Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work · doi:10.1145/2145204.2145355 · Cited by 334
Keith, M. G., Tay, L., & Harms, P. D. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.01359 · Cited by 319
Need fulfillment and experiences on social media: A case on Facebook and WhatsApp
Karapanos, E., Teixeira, P., & Gouveia, R. (2016) · Computers in Human Behavior · doi:10.1016/j.chb.2015.10.015 · Cited by 316
Bartneck, C., Duenser, A., Moltchanova, E., & Zawieska, K. (2015) · PLOS ONE · doi:10.1371/journal.pone.0121595 · Cited by 317
Reputation-based incentive protocols in crowdsourcing applications
Zhang, Y., & van der Schaar, M. (2012) · 2012 Proceedings IEEE INFOCOM · doi:10.1109/infcom.2012.6195597 · Cited by 315
On the naturalness of software
Hindle, A., Barr, E. T., Gabel, M., Su, Z., & Devanbu, P. (2016) · Communications of the ACM · doi:10.1145/2902362 · Cited by 307
Lowry, P. B., D’Arcy, J., Hammer, B., & Moody, G. D. (2016) · The Journal of Strategic Information Systems · doi:10.1016/j.jsis.2016.06.002 · Cited by 301
(2015) · doi:10.1007/s10664-013-9279-3 · Cited by 273
Global mapping of infectious disease
Hay, S. I., Battle, K. E., Pigott, D. M., Smith, D. L., Moyes, C. L., Bhatt, S., Brownstein, J. S., Collier, N., Myers, M. F., George, D. B., et al. (2013) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2012.0250 · Cited by 243
Sharing Knowledge and Expertise: The CSCW View of Knowledge Management
Ackerman, M. S., Dachtera, J., Pipek, V., & Wulf, V. (2013) · Computer Supported Cooperative Work (CSCW) · doi:10.1007/s10606-013-9192-8 · Cited by 229
The Persuasive Power of Data Visualization
Pandey, A. V., Manivannan, A., Nov, O., Satterthwaite, M., & Bertini, E. (2014) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2014.2346419 · Cited by 217
The YouTube Lens: Crowdsourced Personality Impressions and Audiovisual Analysis of Vlogs
Biel, J.-I., & Gatica-Perez, D. (2013) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2012.2225032 · Cited by 209
Serf and turf: crowdturfing for fun and profit
Wang, G., Wilson, C., Zhao, X., Zhu, Y., Mohanlal, M., Zheng, H., & Zhao, B. Y. (2012) · Proceedings of the 21st international conference on World Wide Web · doi:10.1145/2187836.2187928 · Cited by 197
PeerStudio: Rapid Peer Feedback Emphasizes Revision and Improves Performance
Kulkarni, C. E., Bernstein, M. S., & Klemmer, S. R. (2015) · Proceedings of the Second (2015) ACM Conference on Learning @ Scale · doi:10.1145/2724660.2724670 · Cited by 192
O’Leary, M. B., Wilson, J. M., & Metiu, A. (2014) · MIS Quarterly · doi:10.25300/misq/2014/38.4.13 · Cited by 191
(2010) · doi:10.5555/1734454.1734544 · Cited by 163
Field trial of Tiramisu: crowd-sourcing bus arrival times to spur co-design
Zimmerman, J., Tomasic, A., Garrod, C., Yoo, D., Hiruncharoenvate, C., Aziz, R., Thiruvengadam, N. R., Huang, Y., & Steinfeld, A. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979187 · Cited by 150
Examining Crowd Work and Gig Work Through The Historical Lens of Piecework
Alkhatib, A., Bernstein, M. S., & Levi, M. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025974 · Cited by 148
What the App is That? Deception and Countermeasures in the Android User Interface
Bianchi, A., Corbetta, J., Invernizzi, L., Fratantonio, Y., Kruegel, C., & Vigna, G. (2015) · 2015 IEEE Symposium on Security and Privacy · doi:10.1109/sp.2015.62 · Cited by 147
Incentivizing High Quality Crowdwork
Ho, C.-J., Slivkins, A., Suri, S., & Vaughan, J. W. (2015) · Proceedings of the 24th International Conference on World Wide Web · doi:10.1145/2736277.2741102 · Cited by 146
Cognitive Biases in Crowdsourcing
Eickhoff, C. (2018) · Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining · doi:10.1145/3159652.3159654 · Cited by 144
Personifying programming tool feedback improves novice programmers' learning
Lee, M. J., & Ko, A. J. (2011) · Proceedings of the seventh international workshop on Computing education research · doi:10.1145/2016911.2016934 · Cited by 125
Can you ever trust a wiki?: impacting perceived trustworthiness in wikipedia
Kittur, A., Suh, B., & Chi, E. H. (2008) · Proceedings of the 2008 ACM conference on Computer supported cooperative work · doi:10.1145/1460563.1460639 · Cited by 122
Self-citationThe Who in XAI: How AI Background Shapes Perceptions of AI Explanations
Ehsan, U., Passi, S., Liao, Q. V., Chan, L., Lee, I.-H., Muller, M., & Riedl, M. O. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642474 · Cited by 123
"Why would anybody do this?": Understanding Older Adults' Motivations and Challenges in Crowd Work
Brewer, R., Morris, M. R., & Piper, A. M. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858198 · Cited by 111
Investigating intelligibility for uncertain context-aware applications
Lim, B. Y., & Dey, A. K. (2011) · Proceedings of the 13th international conference on Ubiquitous computing · doi:10.1145/2030112.2030168 · Cited by 101
Mitra, T., Hutto, C. J., & Gilbert, E. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702553 · Cited by 102
Working Together Apart: Collaboration over the Internet
Olson, J. S., & Olson, G. M. (2014) · Synthesis Lectures on Human-Centered Informatics · doi:10.2200/s00542ed1v01y201310hci020 · Cited by 95
Improving Data Quality Using Amazon Mechanical Turk Through Platform Setup
Lu, L., Neale, N., Line, N. D., & Bonn, M. (2022) · Cornell Hospitality Quarterly · doi:10.1177/19389655211025475 · Cited by 94
Information quality assessment of community generated content: a user study of Wikipedia
Yaari, E., Baruchson-Arbib, S., & Bar-Ilan, J. (2011) · Journal of Information Science · doi:10.1177/0165551511416065 · Cited by 83
Effect of Information Presentation on Fairness Perceptions of Machine Learning Predictors
van Berkel, N., Goncalves, J., Russo, D., Hosio, S., & Skov, M. B. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445365 · Cited by 84
Auditory interfaces in automated driving: an international survey
Bazilinskyy, P., & de Winter, J. (2015) · PeerJ Computer Science · doi:10.7717/peerj-cs.13 · Cited by 83
LemonAid: selection-based crowdsourced contextual help for web applications
Chilana, P. K., Ko, A. J., & Wobbrock, J. O. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2208620 · Cited by 79
Ways of Knowing When Research Subjects Care
Howard, D., & Irani, L. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300327 · Cited by 70
Utility of human-computer interactions: toward a science of preference measurement
Toomim, M., Kriplean, T., Pörtner, C., & Landay, J. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979277 · Cited by 64
Complementing Text Entry Evaluations with a Composition Task
Vertanen, K., & Kristensson, P. O. (2014) · ACM Transactions on Computer-Human Interaction · doi:10.1145/2555691 · Cited by 62
Gaze and Attention Management for Embodied Conversational Agents
Pejsa, T., Andrist, S., Gleicher, M., & Mutlu, B. (2015) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/2724731 · Cited by 61
Distributed analogical idea generation: inventing with crowds
Yu, L., Kittur, A., & Kraut, R. E. (2014) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2556288.2557371 · Cited by 60
Self-citationPardos, Z. A., Tang, M., Anastasopoulos, I., Sheel, S. K., & Zhang, E. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581574 · Cited by 55
Aroyo, L., & Welty, C. (2014) · Human Computation · doi:10.15346/hc.v1i1.3 · Cited by 43
Preparing Future Designers for Human-AI Collaboration in Persona Creation
Goel, T., Shaer, O., Delcourt, C., Gu, Q., & Cooper, A. (2023) · Proceedings of the 2nd Annual Meeting of the Symposium on Human-Computer Interaction for Work · doi:10.1145/3596671.3598574 · Cited by 37
Crowdsourcing the Unknown: The Satellite Search for Genghis Khan
Lin, A. Y.-M., Huynh, A., Lanckriet, G., & Barrington, L. (2014) · PLoS ONE · doi:10.1371/journal.pone.0114046 · Cited by 35
Understanding Trust, Privacy and Financial Fears in Online Payment
El Haddad, G., Aïmeur, E., & Hage, H. (2018) · 2018 17th IEEE International Conference On Trust, Security And Privacy In Computing And Communications/ 12th IEEE International Conference On Big Data Science And Engineering (TrustCom/BigDataSE) · doi:10.1109/trustcom/bigdatase.2018.00015 · Cited by 35
Rabbitt, S. M., Kazdin, A. E., & Hong, J. E. (2014) · Anthrozoös · doi:10.2752/175303714x13903827487881 · Cited by 34
Can colored sidewalk nudge city tourists to walk? An experimental study of the effect of nudges
Chen, J., Lehto, X., Lehto, M., & Day, J. (2023) · Tourism Management · doi:10.1016/j.tourman.2022.104683 · Cited by 31
Simons, R. N., Gurari, D., & Fleischmann, K. R. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415176 · Cited by 27
Interactive IR User Study Design, Evaluation, and Reporting
Liu, J., & Shah, C. (2019) · Synthesis Lectures on Information Concepts, Retrieval, and Services · doi:10.1007/978-3-031-02319-4 · Cited by 26
Simulation Experiments on (the Absence of) Ratings Bias in Reputation Systems
Thebault-Spieker, J., Kluver, D., Klein, M. A., Halfaker, A., Hecht, B., Terveen, L., & Konstan, J. A. (2017) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3134736 · Cited by 26
Shirahama, K., & Grzegorzek, M. (2016) · Multimedia Tools and Applications · doi:10.1007/s11042-014-2292-8 · Cited by 24
Designing information savvy societies: an introduction to assessability
Forte, A., Andalibi, N., Park, T., & Willever-Farr, H. (2014) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2556288.2557072 · Cited by 23
O'Leary, M. B., Wilson, J. M., & Metiu, A. (2013) · doi:10.2139/ssrn.2322710 · Cited by 16
Kumar, H., Yoo, S., Bernuy, A., Shi, J., Luo, H., Williams, J. J., Kuzminykh, A., Anderson, A., & Kornfield, R. (2024) · doi:10.2139/ssrn.4892998 · Cited by 9
Barriers to collaborative information seeking in organizations
Karunakaran, A., & Reddy, M. (2012) · Proceedings of the American Society for Information Science and Technology · doi:10.1002/meet.14504901169 · Cited by 8
LLMs as Research Tools: A Large Scale Survey of Researchers' Usage and Perceptions
Liao, Z., Antoniak, M., Cheong, I., Cheng, E. Y.-Y., Lee, A.-H., Lo, K., Chang, J. C., & Zhang, A. X. (2024) · arXiv · doi:10.48550/arxiv.2411.05025 · Cited by 7
On the State of Reporting in Crowdsourcing Experiments and a Checklist to Aid Current Practices
Ramírez, J., Sayin, B., Baez, M., Casati, F., Cernuzzi, L., Benatallah, B., & Demartini, G. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479531 · Cited by 4
Metadata
- Authors
- Kittur, A., Chi, E. H., & Suh, B.
- Venue
- Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2008-04-06
- Pages
- 453-456
- OpenAlex
- W2151401338
- Cited by
- 1,923
- References
- 2
- FWCI
- 88.96
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
