Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Open accessAbstract
ABSTRACT Mechanical Turk (MTurk), an online labor system run by Amazon.com, provides quick, easy, and inexpensive access to online research participants. As use of MTurk has grown, so have questions from behavioral researchers about its participants, reliability, and low compensation. In this article, we review recent research about MTurk and compare MTurk participants with community and student samples on a set of personality dimensions and classic decision‐making biases. Across two studies, we find many similarities between MTurk participants and traditional samples, but we also find important differences. For instance, MTurk participants are less likely to pay attention to experimental materials, reducing statistical power. They are more likely to use the Internet to find answers, even with no incentive for correct responses. MTurk participants have attitudes about money that are different from a community sample's attitudes but similar to students' attitudes. Finally, MTurk participants are less extraverted and have lower self‐esteem than other participants, presenting challenges for some research domains. Despite these differences, MTurk participants produce reliable results consistent with standard decision‐making biases: they are present biased, risk‐averse for gains, risk‐seeking for losses, show delay/expedite asymmetries, and show the certainty effect—with almost no significant differences in effect sizes from other samples. We conclude that MTurk offers a highly valuable opportunity for data collection and recommend that researchers using MTurk (1) include screening questions that gauge attention and language comprehension; (2) avoid questions with factual answers; and (3) consider how individual differences in financial and social domains may influence results. Copyright © 2012 John Wiley & Sons, Ltd.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
MTurk is an online crowdsourcing platform widely used for research purposes.
“MTurk is an online crowdsourcing platform widely used for research purposes (Goodman et al., 2013).”
From Virtual reality adoption during the COVID-19 pandemic: A uses and gratifications perspective (2021) · cites this paper as Goodman et al., 2013
Cites
Prospect Theory: An Analysis of Decision under Risk
Kahneman, D., & Tversky, A. (1979) · Econometrica · doi:10.2307/1914185 · Cited by 48,524 · see citing works
Prospect theory: An analysis of decision under risk
Kahneman, D. J., & Tversky, A. (1988) · Cambridge University Press eBooks · doi:10.1017/cbo9780511609220.014 · Cited by 32,700 · see citing works
Tversky, A., & Kahneman, D. (1974) · Science · doi:10.1126/science.185.4157.1124 · Cited by 28,810 · see citing works
Judgment under Uncertainty: Heuristics and Biases
Tversky, A., & Kahneman, D. (1975) · Utility, Probability, and Human Decision Making · doi:10.1007/978-94-010-1834-0_8 · Cited by 22,650 · see citing works
The Framing of Decisions and the Psychology of Choice
Tversky, A., & Kahneman, D. (1981) · Science · doi:10.1126/science.7455683 · Cited by 17,604 · cited 1× · see citing works
“Except for one study showing thatMTurk workers were more risk-averse than non-MTurk participants (Paolacci et al., 2010) in the Asian Disease problem (Tversky & Kahneman, 1981), research has not identified significant differences between MTurk participants and traditional samples.”
ResultAdvances in prospect theory: Cumulative representation of uncertainty
Tversky, A., & Kahneman, D. (1992) · Journal of Risk and Uncertainty · doi:10.1007/bf00122574 · Cited by 15,467 · see citing works
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996 · see citing works
A very brief measure of the Big-Five personality domains
Gosling, S. D., Rentfrow, P. J., & Swann, W. B. (2003) · Journal of Research in Personality · doi:10.1016/s0092-6566(03)00046-1 · Cited by 8,307 · see citing works
(1999) · Cited by 7,995 · see citing works
Handbook of Personality: Theory and Research
Krueger, R. F., & Johnson, W. (2008) · Cited by 7,884 · see citing works
Cognitive Reflection and Decision Making
Frederick, S. (2005) · Journal of Economic Perspectives · doi:10.1257/089533005775196732 · Cited by 5,256 · see citing works
The World Is Flat: A Brief History of the Twenty-First Century
Ikenberry, G. J., & Friedman, T. L. (2005) · Foreign Affairs · doi:10.2307/20031715 · Cited by 4,227 · see citing works
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767 · see citing works
Instructional manipulation checks: Detecting satisficing to increase statistical power
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2009.03.009 · Cited by 3,493 · see citing works
Conducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172 · see citing works
Robins, R. W., Hendin, H. M., & Trzesniewski, K. H. (2001) · Personality and Social Psychology Bulletin · doi:10.1177/0146167201272002 · Cited by 3,041 · see citing works
The Dishonesty of Honest People: A Theory of Self-Concept Maintenance
Mazar, N., Amir, O., & Ariely, D. (2008) · Journal of Marketing Research · doi:10.1509/jmkr.45.6.633 · Cited by 2,875 · see citing works
Some empirical evidence on dynamic inconsistency
Thaler, R. (1981) · Economics Letters · doi:10.1016/0165-1765(81)90067-7 · Cited by 2,394 · see citing works
Judgment under Uncertainty: Heuristics and Biases
Tversky, A., & Kahneman, D. J. (1978) · Elsevier eBooks · doi:10.1016/b978-0-12-214850-7.50008-5 · Cited by 2,060 · see citing works
Morvan, C., & Jenkins, W. J. (2017) · Macat Library eBooks · doi:10.4324/9781912282562 · Cited by 1,891 · see citing works
Decisions from Experience and the Effect of Rare Events in Risky Choice
Hertwig, R., Barron, G., Weber, E. U., & Erev, I. (2004) · Psychological Science · doi:10.1111/j.0956-7976.2004.00715.x · Cited by 1,657 · see citing works
The world is flat: a brief history of the twenty-first century
(2006) · Choice Reviews Online · doi:10.5860/choice.43-4141 · Cited by 1,634 · see citing works
The Online Laboratory: Conducting Experiments in a Real Labor Market∗Preprint
Zeckhauser, R. J. (2016) · Cited by 1,454 · see citing works
The Material Values Scale: Measurement Properties and Development of a Short Form
Richins, M. L. (2004) · Journal of Consumer Research · doi:10.1086/383436 · Cited by 1,427 · see citing works
Peterson, R. A. (2001) · Journal of Consumer Research · doi:10.1086/323732 · Cited by 1,258 · see citing works
Contrasts and Effect Sizes in Behavioral Research
Rosenthal, R., Rosnow, R. L., & Rubin, D. B. (1999) · Cambridge University Press eBooks · doi:10.1017/cbo9780511804403 · Cited by 1,144 · see citing works
Contrasts and Effect Sizes in Behavioral Research: A Correlational Approach
Rosenthal, R., Rosnow, R. L., & Rubin, D. B. (1999) · Cited by 854 · see citing works
Stability of self-esteem across the life span.
Trzesniewski, K. H., Donnellan, M. B., & Robins, R. W. (2003) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.84.1.205 · Cited by 807 · see citing works
Small feedback‐based decisions and their limited correspondence to description‐based decisions
Barron, G., & Erev, I. (2003) · Journal of Behavioral Decision Making · doi:10.1002/bdm.443 · Cited by 739 · see citing works
Rand, D. G. (2011) · Journal of Theoretical Biology · doi:10.1016/j.jtbi.2011.03.004 · Cited by 686 · see citing works
Shu, L. L., Francesca Gino, B., & Bazerman, M. H. (2011) · Personality and Social Psychology Bulletin · doi:10.1177/0146167211398138 · Cited by 671 · see citing works
Frames of mind in intertemporal choice
Loewenstein, G. (1988) · Management Science · doi:10.1287/mnsc.34.2.200 · Cited by 667 · see citing works
Demographics of Mechanical Turk
Ipeirotis, P. G. (2010) · Faculty Digital Archive (New York University Florence) · Cited by 494 · see citing works
On the External Validity of Experiments in Consumer Research
Lynch, J. G. (1982) · Journal of Consumer Research · doi:10.1086/208919 · Cited by 471 · see citing works
Decision‐making competence: External validation through an individual‐differences approach
Parker, A. M., & Fischhoff, B. (2005) · Journal of Behavioral Decision Making · doi:10.1002/bdm.481 · Cited by 444 · see citing works
Rick, S. I., Cryder, C. E., & Loewenstein, G. F. (2008) · Journal of Consumer Research · doi:10.1086/523285 · Cited by 436 · see citing works
Cooperation and Contagion in Web-Based, Networked Public Goods Experiments
Suri, S., & Watts, D. J. (2011) · PLoS ONE · doi:10.1371/journal.pone.0016836 · Cited by 355 · see citing works
Experimentation in the 21st Century: The Importance of External Validity
Winer, R. S. (1999) · Journal of the Academy of Marketing Science · doi:10.1177/0092070399273005 · Cited by 346 · see citing works
Cooperation and contagion in web-based, networked public goods experiments
Suri, S., & Watts, D. J. (2011) · ACM SIGecom Exchanges · doi:10.1145/1998549.1998550 · Cited by 318 · see citing works
A short form of the Maximization Scale: Factor structure, reliability and validity studies
Nenkov, G. Y., Morrin, M., Ward, A. C., Schwartz, B. N., & Hulland, J. S. (2008) · Judgment and Decision Making · doi:10.1017/s1930297500000395 · Cited by 284 · see citing works
Deep thoughts and shallow frames: on the susceptibility to framing effects
LeBoeuf, R. A., & Shafir, E. B. (2003) · Journal of Behavioral Decision Making · doi:10.1002/bdm.433 · Cited by 239 · see citing works
Emotional reactions to losing explain gender differences in entering a risky lottery
Eriksson, K., & Simpson, B. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500001030 · Cited by 171 · see citing works
Malkoc, S. A., & Zauberman, G. S. (2006) · Journal of Marketing Research · doi:10.1509/jmkr.43.4.618 · Cited by 147 · see citing works
Anchors aweigh: A demonstration of cross-modality anchoring and magnitude priming
Oppenheimer, D. M., LeBoeuf, R. A., & Brewer, N. T. (2007) · Cognition · doi:10.1016/j.cognition.2006.12.008 · Cited by 114 · see citing works
Deciphering the Role of RND Efflux Transporters in Burkholderia cenocepacia
Bazzini, S., Udine, C., Sass, A. M., Pasca, M. R., Longo, F., Emiliani, G., Fondi, M., Perrin, E., Decorosi, F., Viti, C., et al. (2011) · PLoS ONE · doi:10.1371/journal.pone.0018902 · Cited by 99 · see citing works
Flagged first, then by citations
1 of the 47 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Beyond the Turk: Alternative platforms for crowdsourcing behavioral research
Peer, E., Brandimarte, L., Samat, S., & Acquisti, A. (2017) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2017.01.006 · Cited by 2,957 · see citing works
Inside the Turk: Understanding Mechanical Turk as a Participant Pool
Paolacci, G., & Chandler, J. (2014) · Current Directions in Psychological Science · doi:10.1177/0963721414531598 · Cited by 2,136 · see citing works
jsPsych: A JavaScript library for creating behavioral experiments in a Web browser
de Leeuw, J. R. (2014) · Behavior Research Methods · doi:10.3758/s13428-014-0458-y · Cited by 1,783 · see citing works
Hauser, D. J., & Schwarz, N. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0578-z · Cited by 1,739 · see citing works
Data quality of platforms and panels for online behavioral research
Peer, E., Rothschild, D., Gordon, A., Evernden, Z., & Damer, E. (2021) · Behavior Research Methods · doi:10.3758/s13428-021-01694-3 · Cited by 1,718 · see citing works
Reputation as a sufficient condition for data quality on Amazon Mechanical Turk
Peer, E., Vosgerau, J., & Acquisti, A. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0434-y · Cited by 1,618 · see citing works
The Generalizability of Survey Experiments
Mullinix, K. J., Leeper, T. J., Druckman, J. N., & Jeremy Freese, A. (2015) · Journal of Experimental Political Science · doi:10.1017/xps.2015.19 · Cited by 1,235 · see citing works
Go, E., & Sundar, S. S. (2019) · Computers in Human Behavior · doi:10.1016/j.chb.2019.01.020 · Cited by 1,089 · see citing works
Hulland, J., Baumgartner, H., & Smith, K. M. (2018) · Journal of the Academy of Marketing Science · doi:10.1007/s11747-017-0532-y · Cited by 1,067 · see citing works
An MTurk Crisis? Shifts in Data Quality and the Impact on Study Results
Chmielewski, M., & Kucker, S. C. (2020) · Social Psychological and Personality Science · doi:10.1177/1948550619875149 · Cited by 1,027 · see citing works
MTurk Research: Review and Recommendations
Aguinis, H., Villamor, I., & Ramani, R. S. (2020) · Journal of Management · doi:10.1177/0149206320969787 · Cited by 942 · see citing works
Chandler, J., Mueller, P., & Paolacci, G. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0365-7 · Cited by 950 · see citing works
Conducting Clinical Research Using Crowdsourced Convenience Samples
Chandler, J., & Shapiro, D. (2016) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-021815-093623 · Cited by 1,072 · see citing works
Kees, J., Berry, C., Burton, S., & Sheehan, K. (2017) · Journal of Advertising · doi:10.1080/00913367.2016.1269304 · Cited by 902 · see citing works
Gatt, A., & Krahmer, E. (2018) · Journal of Artificial Intelligence Research · doi:10.1613/jair.5477 · Cited by 776 · see citing works
An Evaluation of Amazon’s Mechanical Turk, Its Rapid Rise, and Its Effective Use
Buhrmester, M. D., Talaifar, S., & Gosling, S. D. (2018) · Perspectives on Psychological Science · doi:10.1177/1745691617706516 · Cited by 768 · see citing works
Are samples drawn from Mechanical Turk valid for research on political ideology?
Clifford, S., Jewell, R., & Waggoner, P. (2015) · Research & Politics · doi:10.1177/2053168015622072 · Cited by 680 · see citing works
Online panels in social science research: Expanding sampling methods beyond Mechanical Turk
Chandler, J., Rosenzweig, C., Moss, A. J., Robinson, J., & Litman, L. (2019) · Behavior Research Methods · doi:10.3758/s13428-019-01273-7 · Cited by 668 · see citing works
Validity and Mechanical Turk: An assessment of exclusion methods and interactive experiments
Thomas, K. A., & Clifford, S. (2017) · Computers in Human Behavior · doi:10.1016/j.chb.2017.08.038 · Cited by 625 · see citing works
People are averse to machines making moral decisions.
Bigman, Y. E., & Gray, K. (2018) · Cognition · doi:10.1016/j.cognition.2018.08.003 · Cited by 738 · see citing works
Crowdsourcing Consumer Research
Goodman, J. K., & Paolacci, G. (2017) · Journal of Consumer Research · doi:10.1093/jcr/ucx047 · Cited by 594 · see citing works
Weinberg, J. D., Jeremy Freese, A., & McElhattan, D. (2014) · Sociological Science · doi:10.15195/v1.a19 · Cited by 582 · see citing works
The experienced psychological benefits of place attachment
Scannell, L., & Gifford, R. (2017) · Journal of Environmental Psychology · doi:10.1016/j.jenvp.2017.04.001 · Cited by 548 · see citing works
“Participant” Perceptions of Twitter Research Ethics
Fiesler, C. L., & Proferes, N. (2018) · Social Media + Society · doi:10.1177/2056305118763366 · Cited by 545 · see citing works
Steffens, N. K., Haslam, S. A., Reicher, S. D., Platow, M. J., Fransen, K., Yang, J., Ryan, M. K., Jetten, J., Peters, K., & Boen, F. (2014) · The Leadership Quarterly · doi:10.1016/j.leaqua.2014.05.002 · Cited by 487 · see citing works
Demand Effects in Survey Experiments: An Empirical Assessment
MUMMOLO, J., & PETERSON, E. (2019) · American Political Science Review · doi:10.1017/s0003055418000837 · Cited by 548 · see citing works
Zhou, H., & Ayelet Fishbach, B. (2016) · Journal of Personality and Social Psychology · doi:10.1037/pspa0000056 · Cited by 462 · see citing works
Crowdsourcing research: Data collection with Amazon’s Mechanical Turk
Sheehan, K. B. (2018) · Communication Monographs · doi:10.1080/03637751.2017.1342043 · Cited by 408 · see citing works
Lowry, P. B., Zhang, J., Wang, C., & Siponen, M. T. (2016) · Information Systems Research · doi:10.1287/isre.2016.0671 · Cited by 411 · see citing works
Reversed item bias: An integrative model.
Weijters, B., Baumgartner, H., & Schillewaert, N. (2013) · Psychological Methods · doi:10.1037/a0032121 · Cited by 409 · see citing works
Are Attention Check Questions a Threat to Scale Validity?
Kung, F. Y. H., Kwok, N., & Brown, D. J. (2017) · Applied Psychology · doi:10.1111/apps.12108 · Cited by 392 · see citing works
Happiness from Ordinary and Extraordinary Experiences
Bhattacharjee, A., & Mogilner, C. (2014) · Journal of Consumer Research · doi:10.1086/674724 · Cited by 380 · see citing works
Weingarten, E., Chen, Q.-J., McAdams, M., Yi, J., Hepler, J., & Albarracín, D. (2015) · Psychological bulletin · doi:10.1037/bul0000030 · Cited by 275 · see citing works
Development of a new scale to measure subjective career success: A mixed-methods study
Shockley, K. M., Ureksoy, H., Rodopman, O. B., Poteat, L. F., & Dullaghan, T. R. (2015) · Journal of Organizational Behavior · doi:10.1002/job.2046 · Cited by 375 · see citing works
Min, S., So, K. K. F., & Jeong, M. (2019) · Journal of Travel & Tourism Marketing · doi:10.1080/10548408.2018.1507866 · Cited by 489 · see citing works
Conducting interactive experiments online
Arechar, A. A., Gächter, S., & Molleman, L. (2017) · Experimental Economics · doi:10.1007/s10683-017-9527-2 · Cited by 362 · see citing works
Nelson, S. K., Layous, K., Cole, S. W., & Lyubomirsky, S. (2016) · Emotion · doi:10.1037/emo0000178 · Cited by 356 · see citing works
Understanding online safety behaviors: A protection motivation theory perspective
Tsai, H.-Y. S., Jiang, M., Alhabash, S., LaRose, R., Rifon, N. J., & Cotten, S. R. (2016) · Computers & Security · doi:10.1016/j.cose.2016.02.009 · Cited by 363 · see citing works
The Use of Online Panel Data in Management Research: A Review and Recommendations
Porter, C. O. L. H., Outlaw, R., Gale, J., & Cho, T. S. (2018) · Journal of Management · doi:10.1177/0149206318811569 · Cited by 346 · see citing works
Fair and Balanced? Quantifying Media Bias through Crowdsourced Content Analysis
Budak, C., Goel, S., & Rao, J. M. (2016) · Public Opinion Quarterly · doi:10.1093/poq/nfw007 · Cited by 336 · see citing works
Pizzetti, M., Gatti, L., & Seele, P. (2019) · Journal of Business Ethics · doi:10.1007/s10551-019-04406-2 · Cited by 332 · see citing works
MTurk Character Misrepresentation: Assessment and Solutions
Sharpe Wessling, K., Huber, J., & Netzer, O. (2017) · Journal of Consumer Research · doi:10.1093/jcr/ucx053 · Cited by 330 · see citing works
Koo, B., Curtis, C., & Ryan, B. (2020) · International Journal of Hospitality Management · doi:10.1016/j.ijhm.2020.102763 · Cited by 351 · see citing works
Keith, M. G., Tay, L., & Harms, P. D. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.01359 · Cited by 319 · see citing works
Klain, S. C., Olmsted, P., Chan, K. M. A., & Satterfield, T. A. (2017) · PLoS ONE · doi:10.1371/journal.pone.0183962 · Cited by 327 · see citing works
Differentiating environmental concern in the context of psychological adaption to climate change
Helm, S. V., Pollitt, A., Barnett, M. A., Curran, M. A., & Craig, Z. R. (2018) · Global Environmental Change · doi:10.1016/j.gloenvcha.2017.11.012 · Cited by 307 · see citing works
Subjective Knowledge in Consumer Financial Decisions
Hadar, L., Sood, S., & Fox, C. R. (2013) · Journal of Marketing Research · doi:10.1509/jmr.10.0518 · Cited by 318 · see citing works
Public Service Motivation Concepts and Theory: A Critique
Bozeman, B. L., & Su, X. (2014) · Public Administration Review · doi:10.1111/puar.12248 · Cited by 317 · see citing works
Bartneck, C., Duenser, A., Moltchanova, E., & Zawieska, K. (2015) · PLOS ONE · doi:10.1371/journal.pone.0121595 · Cited by 317 · see citing works
Huang, J. L. W., Bowling, N. A., Liu, M., & Li, Y. (2014) · Journal of Business and Psychology · doi:10.1007/s10869-014-9357-6 · Cited by 315 · see citing works
The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers
Stewart, N., Ungemach, C., Harris, A. J. L., Bartels, D. M., Newell, B. R., Paolacci, G., & Chandler, J. (2015) · Judgment and Decision Making · doi:10.1017/s1930297500005611 · Cited by 304 · see citing works
“MTurk workers are used in psychology, economics, and political science, with classic findings replicated in all three domains (Berinsky et al., 2012; Goodman, Cryder, & Cheema, 2013; Horton, Rand, We thank Jon Baron, Leif Nelson, Stian Reimers, and two anonymous reviewers for suggestions.”
The importance of assessing clinical phenomena in Mechanical Turk research.
Hall, K. A. A., Çek, D., Shaw, A. M., & Timpano, K. R. (2015) · Psychological Assessment · doi:10.1037/pas0000217 · Cited by 311 · see citing works
Gächter, S., Starmer, C., & Tufano, F. (2015) · PLoS ONE · doi:10.1371/journal.pone.0129478 · Cited by 311 · see citing works
The myth of harmless wrongs in moral cognition: Automatic dyadic completion from sin to suffering.
Gray, K. J., Schein, C., & Ward, A. F. (2014) · Journal of Experimental Psychology General · doi:10.1037/a0036149 · Cited by 306 · see citing works
Conducting perception research over the internet: A tutorial review
Woods, A. T., Velasco, C., Levitan, C. A., Wan, X., & Spence, C. (2015) · PeerJ · doi:10.7717/peerj.1058 · Cited by 301 · see citing works
Deng, X. N., Joshi, K. D., & Galliers, R. D. (2016) · MIS Quarterly · doi:10.25300/misq/2016/40.2.01 · Cited by 316 · see citing works
Lowry, P. B., D’Arcy, J., Hammer, B., & Moody, G. D. (2016) · The Journal of Strategic Information Systems · doi:10.1016/j.jsis.2016.06.002 · Cited by 301 · see citing works
Investigating logistics service quality in omni-channel retailing
Murfield, M. L. U., Boone, C. A., Rutner, P., & Thomas, R. (2017) · International Journal of Physical Distribution & Logistics Management · doi:10.1108/ijpdlm-06-2016-0161 · Cited by 316 · see citing works
Crowd-sourced Text Analysis: Reproducible and Agile Production of Political Data
Benoit, K. R., Conway, D., Lauderdale, B. E., Laver, M., & Mikhaylov, S. (2016) · American Political Science Review · doi:10.1017/s0003055416000058 · Cited by 296 · see citing works
The use of sampling methods in advertising research: a gap between theory and practice
Sarstedt, M., Bengart, P., Shaltoni, A. M., & Lehmann, S. (2017) · International Journal of Advertising · doi:10.1080/02650487.2017.1348329 · Cited by 294 · see citing works
VLAT: Development of a Visualization Literacy Assessment Test
Lee, S., Kim, S.-H., & Kwon, B. C. (2016) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2016.2598920 · Cited by 292 · see citing works
Scoundrels or Stars? Theory and Evidence on the Quality of Workers in Online Labor Markets
Farrell, A. M., Grenier, J. H., & Leiby, J. (2016) · The Accounting Review · doi:10.2308/accr-51447 · Cited by 274 · see citing works
A study of antecedents and outcomes of social media WOM towards luxury brand purchase intention
Park, J., Hyun, H., & Thavisay, T. (2020) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2020.102272 · Cited by 315 · see citing works
Chandler, J., & Paolacci, G. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550617698203 · Cited by 280 · see citing works
Evaluating CloudResearch’s Approved Group as a solution for problematic data quality on MTurk
Hauser, D., Moss, A. J., Rosenzweig, C., Jaffe, S. N., Robinson, J., & Litman, L. (2022) · Behavior Research Methods · doi:10.3758/s13428-022-01999-x · Cited by 276 · see citing works
Testing the Waters: Behavior across Participant Pools
Snowberg, E., & Yariv, L. (2021) · American Economic Review · doi:10.1257/aer.20181065 · Cited by 274 · see citing works
Haslam, S. A., McMahon, C., Cruwys, T., Haslam, C., Jetten, J., & Steffens, N. K. (2017) · Social Science & Medicine · doi:10.1016/j.socscimed.2017.12.020 · Cited by 265 · see citing works
Umucu, E., & Lee, B. (2020) · Rehabilitation Psychology · doi:10.1037/rep0000328 · Cited by 260 · see citing works
Associations between loot box use, problematic gaming and gambling, and gambling-related cognitions
Brooks, G. A., & Clark, L. (2019) · Addictive Behaviors · doi:10.1016/j.addbeh.2019.04.009 · Cited by 266 · see citing works
Individual and work factors related to perceived work ability and labor force outcomes.
McGonagle, A. K., Fisher, G. G., Barnes-Farrell, J. L., & Grosch, J. W. (2014) · Journal of Applied Psychology · doi:10.1037/a0037974 · Cited by 254 · see citing works
Experiential Avoidance: An Examination of the Construct Validity of the AAQ-II and MEAQ
Rochefort, C., Baldwin, A. S., & Chmielewski, M. (2018) · Behavior Therapy · doi:10.1016/j.beth.2017.08.008 · Cited by 247 · see citing works
Ståhl, T., & van Prooijen, J.-W. (2018) · Personality and Individual Differences · doi:10.1016/j.paid.2017.10.026 · Cited by 245 · see citing works
The effect of hedonic and utilitarian values on satisfaction and loyalty of Airbnb users
Lee, S., & Kim, D.-Y. (2018) · International Journal of Contemporary Hospitality Management · doi:10.1108/ijchm-09-2016-0504 · Cited by 242 · see citing works
Who Are the Turkers? A Characterization of MTurk Workers Using the Personality Assessment Inventory
McCredie, M. N., & Morey, L. C. (2018) · Assessment · doi:10.1177/1073191118760709 · Cited by 235 · see citing works
Adapa, S., Fazal‐e‐Hasan, S. M., Balaji, M. S., Azeem, M. M., & Mortimer, G. (2019) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2019.101901 · Cited by 227 · see citing works
Jang, Y. J., Kim, W. G., & Lee, H. (2014) · International Journal of Hospitality Management · doi:10.1016/j.ijhm.2014.10.001 · Cited by 226 · see citing works
Crowdsourcing Samples in Cognitive Science.
Stewart, N., Chandler, J. J., & Paolacci, G. (2017) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2017.06.007 · Cited by 210 · see citing works
Mena, P. (2019) · Policy & Internet · doi:10.1002/poi3.214 · Cited by 224 · see citing works
Swami, V., Furnham, A. F., Smyth, N., Weis, L., Lay, A., & Clow, A. (2016) · Personality and Individual Differences · doi:10.1016/j.paid.2016.04.084 · Cited by 219 · see citing works
Petrescu, D. C., Hollands, G. J., Couturier, D., Ng, Y., & Marteau, T. M. (2016) · PLoS ONE · doi:10.1371/journal.pone.0155995 · Cited by 217 · see citing works
Candidate Choice Without Party Labels:
Kirkland, P. A., & Coppock, A. (2017) · Political Behavior · doi:10.1007/s11109-017-9414-8 · Cited by 217 · see citing works
Moore, S. G. (2015) · Journal of Consumer Research · doi:10.1093/jcr/ucv003 · Cited by 216 · see citing works
Fan, A., Wu, L., Miao, L., & Mattila, A. S. (2019) · Journal of Hospitality Marketing & Management · doi:10.1080/19368623.2019.1639095 · Cited by 214 · see citing works
Pillai, S. G., Kim, W. G., Haldorai, K., & Kim, H. (2022) · International Journal of Hospitality Management · doi:10.1016/j.ijhm.2022.103275 · Cited by 248 · see citing works
Human-Robot Interaction: An Introduction
Bartneck, C., Belpaeme, T., Eyssel, F., Kanda, T., Keijsers, M., & Šabanović, S. (2020) · doi:10.1017/9781108676649 · Cited by 211 · see citing works
Hair, J. F., & Sarstedt, M. (2021) · The Journal of Marketing Theory and Practice · doi:10.1080/10696679.2020.1860683 · Cited by 224 · see citing works
Clifford, S., & Jerit, J. (2014) · Journal of Experimental Political Science · doi:10.1017/xps.2014.5 · Cited by 206 · see citing works
Robinson, J., Rosenzweig, C., Moss, A. J., & Litman, L. (2019) · PLoS ONE · doi:10.1371/journal.pone.0226394 · Cited by 203 · see citing works
Psychological research in the internet age: The quality of web-based data
Ramsey, S. R., Thompson, K. L., McKenzie, M. D., & Rosenbaum, A. (2016) · Computers in Human Behavior · doi:10.1016/j.chb.2015.12.049 · Cited by 199 · see citing works
Reh, S., Tröster, C., & Quaquebeke, N. V. (2017) · Journal of Applied Psychology · doi:10.1037/apl0000281 · Cited by 184 · see citing works
Cryder, C. E., Loewenstein, G. F., & Scheines, R. (2012) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2012.08.002 · Cited by 199 · see citing works
It’s about time: New perspectives and insights on time management
Aeon, B., & Aguinis, H. (2017) · Academy of Management Perspectives · doi:10.5465/amp.2016.0166 · Cited by 197 · see citing works
The accuracy of several pose estimation methods for 3D joint centre localisation
Needham, L., Evans, M., Cosker, D. P., Wade, L., McGuigan, P. M., Bilzon, J., & Colyer, S. (2021) · Scientific Reports · doi:10.1038/s41598-021-00212-x · Cited by 205 · see citing works
Artificial intelligence in public services: When and why citizens accept its usage
Gesk, T. S., & Leyer, M. (2022) · Government Information Quarterly · doi:10.1016/j.giq.2022.101704 · Cited by 195 · see citing works
Vohs, K. D. (2015) · Journal of Experimental Psychology General · doi:10.1037/xge0000091 · Cited by 193 · see citing works
Positive and negative eWOM motivations and hotel customers’ eWOM behavior: Does personality matter?
Hu, Y., & Kim, H. J. (2018) · International Journal of Hospitality Management · doi:10.1016/j.ijhm.2018.03.004 · Cited by 193 · see citing works
Redmiles, E. M., Kross, S., & Mazurek, M. L. (2019) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp.2019.00014 · Cited by 193 · see citing works
Comparing customer perceptions of hotel and peer-to-peer accommodation advantages and disadvantages
Birinci, H., Berezina, K., & Çobanoğlu, C. (2018) · International Journal of Contemporary Hospitality Management · doi:10.1108/ijchm-09-2016-0506 · Cited by 190 · see citing works
Work experiences on MTurk: Job satisfaction, turnover, and information sharing
Brawley, A. M., & Pury, C. L. S. (2016) · Computers in Human Behavior · doi:10.1016/j.chb.2015.08.031 · Cited by 188 · see citing works
Cyberbullying on Social Networking Sites: The Crime Opportunity and Affordance Perspectives
Chan, T. K. H., Cheung, C. M. K., & Wong, R. Y. M. (2019) · Journal of Management Information Systems · doi:10.1080/07421222.2019.1599500 · Cited by 190 · see citing works
- Authors
- Goodman, J. K., Cryder, C. E., & Cheema, A.
- Affiliations
- Washington University in St. Louis (US)
- University of Virginia (US)
- Corresponding author
- Cryder, Cynthia E.
- Venue
- Journal of Behavioral Decision Making
- Publisher
- Wiley
- Published
- 2012-04-02
- Volume
- 26
- Issue
- 3
- Pages
- 213-224
- ISSN
- 0894-3257 (ISSN-L)
- 1099-0771
- OpenAlex
- W2126302787
- Cited by
- 2,242
- References
- 47
- Citations by year
- 2026: 47 · 2025: 99 · 2024: 121 · 2023: 123 · 2022: 172 · 2021: 268 · 2020: 244 · 2019: 212 · 2018: 208 · 2017: 245 · 2016: 201 · 2015: 171 · 2014: 81 · 2013: 41 · 2012: 7
- FWCI
- 155.03
- Type
- journal article
- Language
- en
- Open access
- green
- autopapers.ssrn.com
- Locations
- Journal of Behavioral Decision Making (publishedVersion)
- SSRN Electronic Journal (submittedVersion)
- ACR North American Advances
- Funders
- Carnegie Mellon University
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Social Sciences › Safety Research › Experimental Behavioral Economics Studies
- Keywords
- Mechanical Turk, online labour platforms, sample representativeness, participant selection, decision bias, present bias, risk aversion, delay discounting, certainty effect, personality differences, self-esteem, extraversion, attitudes toward money, student samples, community sample, screening questions
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
