The promise of Mechanical Turk: How online labor markets can help theorists run behavioral experiments
The DOI matches its Crossref record; 82 of the 82 references it cites were found in a registry or an index.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Axelrod, R., & Hamilton, W. D. (1981) · Science · doi:10.1126/science.7466396 · Cited by 20,065
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996
Five Rules for the Evolution of Cooperation
Nowak, M. A. (2006) · Science · doi:10.1126/science.1133755 · Cited by 5,894 · cited 3×
“Evolutionary game theory uses mathematics to formalize the process of evolution (Hofbauer and Sigmund, 1998; Nowak, 2006b).”
BackgroundEvolutionary Games and Population Dynamics
Hofbauer, J., & Sigmund, K. (1998) · doi:10.1017/cbo9781139173179 · Cited by 5,502 · cited 1×
“Evolutionary game theory uses mathematics to formalize the process of evolution (Hofbauer and Sigmund, 1998; Nowak, 2006b).”
BackgroundWhy Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
Cooperation and Punishment in Public Goods Experiments
Fehr, E., & Gächter, S. (2000) · American Economic Review · doi:10.1257/aer.90.4.980 · Cited by 4,547 · cited 1×
“In their replication experiment, they had N¼96 subjects play the same linear repeated public goods game of Fehr and Gächter (2000).”
BackgroundConnolly, T., & Axelrod, R. (1984) · Administrative Science Quarterly · doi:10.2307/2392957 · Cited by 3,890
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767 · cited 1×
“A third direct replication used various social psychology tests and compared N ¼318 AMT subjects to both N ¼141 students from a Midwestern University and N ¼137 visitors to an online discussion board (Paolacci et al., 2010).”
MethodBehavioral Game Theory: Experiments in Strategic Interaction
Camerer, C. F. (2003) · Cited by 3,722
Cited by
Prolific.ac—A subject pool for online experiments
Palan, S., & Schitter, C. (2018) · Journal of Behavioral and Experimental Finance · doi:10.1016/j.jbef.2017.12.004 · Cited by 3,450
Conducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172
Beyond the Turk: Alternative platforms for crowdsourcing behavioral research
Peer, E., Brandimarte, L., Samat, S., & Acquisti, A. (2017) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2017.01.006 · Cited by 2,957
“…purposes, in and outside the behavioral sciences (e.g., Buhrmester, Kwang, & Gosling, 2011; Crump, McDonnell, & Gureckis, 2013; Fort, Adda, & Cohen, 2011; Goodman, Cryder, & Cheema, 2013; Mason & Suri, 2012; Paolacci, Chandler, & Ipeirotis, 2010; Rand, 2012; Simcox & Fiez, 2014; Sprouse, 2011).”
BackgroundData collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242
Inside the Turk: Understanding Mechanical Turk as a Participant Pool
Paolacci, G., & Chandler, J. (2014) · Current Directions in Psychological Science · doi:10.1177/0963721414531598 · Cited by 2,136
Statistical physics of human cooperation
Perc, M., Jordan, J. J., Rand, D. G., Wang, Z., Boccaletti, S., & Szolnoki, A. (2017) · Physics Reports · doi:10.1016/j.physrep.2017.05.004 · Cited by 1,590
Self-citationSpontaneous giving and calculated greed
Rand, D. G., Greene, J. D., & Nowak, M. A. (2012) · Nature · doi:10.1038/nature11467 · Cited by 1,334
“Rand, D. G., Dreber, A., Ellingsen, T., Fudenberg, D. & Nowak, M. A. Positive interactions promote public cooperation. Science 325, 1272–1275 (2009). 8.”
Self-citationRand, D. G., & Nowak, M. A. (2013) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2013.06.003 · Cited by 1,308
Self-citationUsing Mechanical Turk to Study Clinical Populations
Shapiro, D. N., Chandler, J., & Mueller, P. A. (2013) · Clinical Psychological Science · doi:10.1177/2167702612469015 · Cited by 1,139
“As a result, the MTurk community is governed by strong norms of honesty and accuracy (Rand, 2012; Suri, Goldstein, & Mason, 2011).”
Conducting Clinical Research Using Crowdsourced Convenience Samples
Chandler, J., & Shapiro, D. (2016) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-021815-093623 · Cited by 1,072
How Elastic Are Preferences for Redistribution? Evidence from Randomized Survey Experiments
Kuziemko, I., Norton, M. I., Saez, E., & Stantcheva, S. (2015) · American Economic Review · doi:10.1257/aer.20130360 · Cited by 1,067
Chandler, J., Mueller, P., & Paolacci, G. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0365-7 · Cited by 950
An Evaluation of Amazon’s Mechanical Turk, Its Rapid Rise, and Its Effective Use
Buhrmester, M. D., Talaifar, S., & Gosling, S. D. (2018) · Perspectives on Psychological Science · doi:10.1177/1745691617706516 · Cited by 767
Social heuristics shape intuitive cooperation
Rand, D. G., Peysakhovich, A., Kraft-Todd, G. T., Newman, G. E., Wurzbacher, O., Nowak, M. A., & Greene, J. D. (2014) · Nature Communications · doi:10.1038/ncomms4677 · Cited by 720
Self-citationIntroducing the Open Affective Standardized Image Set (OASIS)
Kurdi, B., Lozano, S., & Banaji, M. R. (2017) · Behavior Research Methods · doi:10.3758/s13428-016-0715-3 · Cited by 664
Dynamic social networks promote cooperation in experiments with humans
Rand, D. G., Arbesman, S., & Christakis, N. A. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1108243108 · Cited by 653
Self-citationValidity and Mechanical Turk: An assessment of exclusion methods and interactive experiments
Thomas, K. A., & Clifford, S. (2017) · Computers in Human Behavior · doi:10.1016/j.chb.2017.08.038 · Cited by 625
Crowdsourcing Consumer Research
Goodman, J. K., & Paolacci, G. (2017) · Journal of Consumer Research · doi:10.1093/jcr/ucx047 · Cited by 588
Weinberg, J., Freese, J., & McElhattan, D. (2014) · Sociological Science · doi:10.15195/v1.a19 · Cited by 582
The experienced psychological benefits of place attachment
Scannell, L., & Gifford, R. (2017) · Journal of Environmental Psychology · doi:10.1016/j.jenvp.2017.04.001 · Cited by 525
Moshagen, M., Hilbig, B. E., & Zettler, I. (2018) · Psychological Review · doi:10.1037/rev0000111 · Cited by 519
The Evolution of Cooperation in Infinitely Repeated Games: Experimental Evidence
Dal Bó, P., & Fréchette, G. R. (2011) · American Economic Review · doi:10.1257/aer.101.1.411 · Cited by 495
Humans display a ‘cooperative phenotype’ that is domain general and temporally stable
Peysakhovich, A., Nowak, M. A., & Rand, D. G. (2014) · Nature Communications · doi:10.1038/ncomms5939 · Cited by 426
Self-citationThe moral standing of animals: Towards a psychology of speciesism.
Caviola, L., Everett, J. A. C., & Faber, N. S. (2019) · Journal of Personality and Social Psychology · doi:10.1037/pspp0000182 · Cited by 409
Hauser, O. P., Rand, D. G., Peysakhovich, A., & Nowak, M. A. (2014) · Nature · doi:10.1038/nature13530 · Cited by 378
Self-citationDigital Footprints: Opportunities and Challenges for Online Social Research
Golder, S. A., & Macy, M. W. (2014) · Annual Review of Sociology · doi:10.1146/annurev-soc-071913-043145 · Cited by 378
Analysis of a consumer survey on plug-in hybrid electric vehicles
Krupa, J. S., Rizzo, D. M., Eppstein, M. J., Brad Lanute, D., Gaalema, D. E., Lakkaraju, K., & Warrender, C. E. (2014) · Transportation Research Part A: Policy and Practice · doi:10.1016/j.tra.2014.02.019 · Cited by 371
Economic games on the internet: the effect of $1 stakes
Amir, O., Rand, D. G., & Gal, Y. K. (2012) · PLoS ONE · doi:10.1371/journal.pone.0031461 · Cited by 364
“Recent work has replicated classical findings such as framing and priming on MTurk [8– 10], found a high level of test-retest reliability on Mturk [10–12], and shown quantitative agreement in behavior between MTurk and the physical laboratory [6,8].”
ResultSelf-citationThe evolution of antisocial punishment in optional public goods games
Rand, D. G., & Nowak, M. A. (2011) · Nature Communications · doi:10.1038/ncomms1442 · Cited by 350
Self-citationConducting interactive experiments online
Arechar, A. A., Gächter, S., & Molleman, L. (2018) · Experimental Economics · doi:10.1007/s10683-017-9527-2 · Cited by 340
Searching for explanations: How the Internet inflates estimates of internal knowledge.
Fisher, M., Goddu, M. K., & Keil, F. C. (2015) · Journal of Experimental Psychology: General · doi:10.1037/xge0000070 · Cited by 330
“01) from the United States completed the study through Amazon’s Mechanical Turk (Rand, 2012).”
MethodKeith, M. G., Tay, L., & Harms, P. D. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.01359 · Cited by 319
Bartneck, C., Duenser, A., Moltchanova, E., & Zawieska, K. (2015) · PLOS ONE · doi:10.1371/journal.pone.0121595 · Cited by 317
“Other limitations to AMT studies include lack of possibility to exert control over participants’ environments compared to lab studies [27] and potentially lower English proficiency [31].”
BackgroundResultHögberg, J., Hamari, J., & Wästlund, E. (2019) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-019-09223-w · Cited by 299
Zhu, S., Song, J., Hazen, B. T., Lee, K., & Cegielski, C. (2018) · International Journal of Physical Distribution & Logistics Management · doi:10.1108/ijpdlm-11-2017-0341 · Cited by 294
What's in a name? Consumer perceptions of in vitro meat under different names
Bryant, C. J., & Barnett, J. C. (2019) · Appetite · doi:10.1016/j.appet.2019.02.021 · Cited by 292
Static network structure can stabilize human cooperation
Rand, D. G., Nowak, M. A., Fowler, J. H., & Christakis, N. A. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1400406111 · Cited by 290
Self-citationCoffman, K. B., Coffman, L. C., & Ericson, K. M. M. (2017) · Management Science · doi:10.1287/mnsc.2016.2503 · Cited by 287
How Black Are Lakisha and Jamal? Racial Perceptions from Names Used in Correspondence Audit Studies
Gaddis, S. (2017) · Sociological Science · doi:10.15195/v4.a19 · Cited by 285
Phase transitions in models of human cooperation
Perc, M. (2016) · Physics Letters A · doi:10.1016/j.physleta.2016.06.017 · Cited by 277
Umucu, E., & Lee, B. (2020) · Rehabilitation Psychology · doi:10.1037/rep0000328 · Cited by 260
Experimental subjects are not different
Exadaktylos, F., Espín, A. M., & Brañas-Garza, P. (2013) · Scientific Reports · doi:10.1038/srep01213 · Cited by 257
Planning and Conducting Experimental Advertising Research and Questionnaire Design
Geuens, M., & De Pelsmacker, P. (2017) · Journal of Advertising · doi:10.1080/00913367.2016.1225233 · Cited by 234
“…hard-to-reach target groups, a good response rate for follow-up studies, truly random allocation to experimental conditions, and importantly, reliable, consistent, and highquality data (Buhrmester, Kwang, and Gosling 2011; Berinsky, Huber, and Lenz 2012; Mason and Suri 2012; Rand 2012; Shank 2016).”
BackgroundMetacognitive Failure as a Feature of Those Holding Radical Beliefs
Rollwage, M., Dolan, R. J., & Fleming, S. M. (2018) · Current Biology · doi:10.1016/j.cub.2018.10.053 · Cited by 193
“Mechanical Turk has been shown to be more representative of the population than typical college student samples [41, 42], and produces high quality data [40] with good internal and external validity [43, 44], even when using complex behavioral tasks [45].”
BackgroundProsocial Conformity: Prosocial Norms Generalize Across Behavior and Empathy
Nook, E. C., Ong, D. C., Morelli, S. A., Mitchell, J. P., & Zaki, J. (2016) · Personality and Social Psychology Bulletin · doi:10.1177/0146167216649932 · Cited by 186
“We recruited 102 participants from Amazon Mechanical Turk (mTurk), an online workplace that provides reliable behavioral data (Buhrmester, Kwang, & Gosling, 2011; Crump, McDonnell, & Gureckis, 2013; Horton, Rand, & Zeckhauser, 2011; Rand, 2012).”
MethodThe Dark Triad and dispositional aggression
Jones, D. N., & Neria, A. L. (2015) · Personality and Individual Differences · doi:10.1016/j.paid.2015.06.021 · Cited by 178
“In fact, recent surveys have found that the reliability, validity, and interest in taking surveys are all greater in MTurk samples when compared to students (Rand, 2012).”
BackgroundReputation Transferability in Online Labor Markets
Kokkodis, M., & Ipeirotis, P. G. (2016) · Management Science · doi:10.1287/mnsc.2015.2217 · Cited by 166
Deep learning-aided decision support for diagnosis of skin disease across skin tones
Groh, M., Badri, O., Daneshjou, R., Koochek, A., Harris, C., Soenksen, L. R., Doraiswamy, P. M., & Picard, R. (2024) · Nature Medicine · doi:10.1038/s41591-023-02728-3 · Cited by 160
Fairness violations elicit greater punishment on behalf of another than for oneself
FeldmanHall, O., Sokol-Hessner, P., Van Bavel, J. J., & Phelps, E. A. (2014) · Nature Communications · doi:10.1038/ncomms6306 · Cited by 106
What’s mine is mine and what’s yours is mine: The Dark Triad and gambling with your neighbor’s money
Jones, D. N. (2013) · Journal of Research in Personality · doi:10.1016/j.jrp.2013.04.005 · Cited by 103
“’’ MTurk is widely used by psychologists as a powerful source of data that surpasses student samples in most respects (Buhrmester, Kwang, & Gosling, 2011; Paolacci, Chandler, & Ipeirotis, 2010; Rand, 2012).”
BackgroundThird-party punishers are rewarded, but third-party helpers even more so
Raihani, N. J., & Bshary, R. (2015) · Evolution · doi:10.1111/evo.12637 · Cited by 101
“Subjects made their decision in isolation, where third parties were matched to the decision of dictators ex-post, and bystanders matched with the decisions of third parties ex-post (as in Raihani and McAuliffe 2012; Rand 2012).”
Moral polarization and out-party hostility in the US political context
Tappin, B. M., & McKay, R. T. (2019) · Journal of Social and Political Psychology · doi:10.5964/jspp.v7i1.1090 · Cited by 79
“Subjects were supporters of the US Republican or Democratic Party, recruited via Amazon’s Mechanical Turk (MTurk), an online labour market commonly used for psychological research (Arechar et al., 2018; Chandler & Shapiro, 2016; Rand, 2012).”
MethodReflecting on explanatory ability: A mechanism for detecting gaps in causal knowledge.
Johnson, D. R., Murphy, M. P., & Messer, R. M. (2016) · Journal of Experimental Psychology: General · doi:10.1037/xge0000161 · Cited by 30
Munzel, A. (2015) · Recherche et Applications en Marketing (English Edition) · doi:10.1177/2051570715604155 · Cited by 24
“In recent empirical investigations, AMT received significant support as a source for data collection in academic experimental studies (Paolacci et al., 2010; Rand, 2012).”
BackgroundMethodInfluentialDunbar, A., & Kubrin, C. E. (2018) · Journal of Experimental Criminology · doi:10.1007/s11292-018-9342-6 · Cited by 24
The impact of relationship-contingent self-esteem on mate retention and reactions to threat
Holden, C. J., Zeigler‐Hill, V., Shackelford, T. K., & Welling, L. L. M. (2018) · Personal Relationships · doi:10.1111/pere.12255 · Cited by 10
On the State of Reporting in Crowdsourcing Experiments and a Checklist to Aid Current Practices
Ramírez, J., Sayin, B., Baez, M., Casati, F., Cernuzzi, L., Benatallah, B., & Demartini, G. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479531 · Cited by 4
Metadata
- Authors
- Rand, D. G.
- Venue
- Journal of Theoretical Biology
- Publisher
- Elsevier BV
- Published
- 2012-04
- Volume
- 299
- Pages
- 172-179
- ISSN
- 0022-5193 (ISSN-L)
- OpenAlex
- W2169800142
- Cited by
- 683
- References
- 82
- FWCI
- 116.77
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.