Evaluating amazon’s mechanical turk as a tool for experimental behavioral research
Open accessAbstract
Amazon Mechanical Turk (AMT) is an online crowdsourcing service where anonymous online workers complete web-based tasks for small sums of money. The service has attracted attention from experimental psychologists interested in gathering human subject data more efficiently. However, relative to traditional laboratory studies, many aspects of the testing environment are not under the experimenter's control. In this paper, we attempt to empirically evaluate the fidelity of the AMT system for use in cognitive behavioral experiments. These types of experiment differ from simple surveys in that they require multiple trials, sustained attention from participants, comprehension of complex instructions, and millisecond accuracy for response recording and stimulus presentation. We replicate a diverse body of tasks from experimental psychology including the Stroop, Switching, Flanker, Simon, Posner Cuing, attentional blink, subliminal priming, and category learning tasks using participants recruited using AMT. While most of replications were qualitatively successful and validated the approach of collecting data anonymously online using a web-browser, others revealed disparity between laboratory results and online results. A number of important lessons were encountered in the process of conducting these replications that should be of value to other researchers.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Online data collection using a web-browser seems mostly in line with laboratory results, so long as the experiment methods are solid.
“Regarding online experiments in general, Crump et al. [32] showed that the data collected online using a web-browser seemed mostly in line with laboratory results, so long as the experiment methods were solid.”
From Adaptive trust calibration for human-AI collaboration (2020) · cites this paper as [32]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Studies of interference in serial verbal reactions
Stroop, J. R. (1935) · Journal of Experimental Psychology · doi:10.1037/h0054651 · Cited by 18,622 · see citing works
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996 · see citing works
Effects of noise letters upon the identification of a target letter in a nonsearch task
Eriksen, B. A., & Eriksen, C. W. (1974) · Perception & Psychophysics · doi:10.3758/bf03203267 · Cited by 7,230 · see citing works
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567 · see citing works
Half a century of research on the Stroop effect: An integrative review.
MacLeod, C. M. (1991) · Psychological Bulletin · doi:10.1037/0033-2909.109.2.163 · Cited by 5,500 · see citing works
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767 · see citing works
Instructional manipulation checks: Detecting satisficing to increase statistical power
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2009.03.009 · Cited by 3,493 · see citing works
Conducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172 · see citing works
Monsell, S. (2003) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(03)00028-7 · Cited by 2,984 · see citing works
Components of visual orienting
Posner, M. I. (1984) · Attention and performance · Cited by 2,980 · see citing works
Gosling, S. D., Vazire, S., Srivastava, S., & John, O. P. (2004) · American Psychologist · doi:10.1037/0003-066x.59.2.93 · Cited by 2,932 · see citing works
Temporary suppression of visual processing in an RSVP task: An attentional blink?
Raymond, J. E., Shapiro, K. L., & Arnell, K. M. (1992) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037//0096-1523.18.3.849 · Cited by 2,276 · see citing works
Klein, R. M. (2000) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(00)01452-2 · Cited by 2,017 · see citing works
Temporary suppression of visual processing in an RSVP task: an attentional blink?
Raymond, J. E., Shapiro, K. L., & Arnell, K. M. (1992) · Journal of Experimental Psychology: Human Perception and Performance · doi:10.1037/0096-1523.18.3.849 · Cited by 1,649 · see citing works
ALCOVE: An exemplar-based connectionist model of category learning.
Kruschke, J. K. (1992) · Psychological Review · doi:10.1037/0033-295x.99.1.22 · Cited by 1,582 · see citing works
Control and interference in task switching—A review
Kiesel, A., Steinhauser, M., Wendt, M., Falkenstein, M., Jost, K., Philipp, A. M., & Koch, I. (2010) · Psychological Bulletin · doi:10.1037/a0019842 · Cited by 1,485 · see citing works
Who are the crowdworkers?: shifting demographics in mechanical turk
Ross, J., Irani, L., Silberman, M. S., Zaldivar, A., & Tomlinson, B. (2010) · CHI '10 Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/1753846.1753873 · Cited by 1,004 · see citing works
Lu, C.-H., & Proctor, R. W. (1995) · Psychonomic Bulletin & Review · doi:10.3758/bf03210959 · Cited by 962 · see citing works
A Solution to the Effect of Sample Size on Outlier Elimination
Selst, M. V., & Jolicœur, P. (1994) · The Quarterly Journal of Experimental Psychology Section A · doi:10.1080/14640749408401131 · Cited by 844 · see citing works
Learning and memorization of classifications.
Shepard, R. N., Hovland, C. I., & Jenkins, H. M. (1961) · The Psychological Monographs · doi:10.1037/h0093825 · Cited by 824 · see citing works
SUSTAIN: A Network Model of Category Learning.
Love, B. C., Medin, D. L., & Gureckis, T. M. (2004) · Psychological Review · doi:10.1037/0033-295x.111.2.309 · Cited by 778 · see citing works
Standards for Internet-Based Experimenting
Reips, U. (2002) · Experimental Psychology (formerly Zeitschrift für Experimentelle Psychologie) · doi:10.1026/1618-3169.49.4.243 · Cited by 685 · see citing works
Germine, L., Nakayama, K., Duchaine, B. C., Chabris, C. F., Chatterjee, G., & Wilmer, J. B. (2012) · Psychonomic Bulletin & Review · doi:10.3758/s13423-012-0296-9 · Cited by 669 · see citing works
Shapiro, K. L., Raymond, J. E., & Arnell, K. M. (1997) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(97)01094-2 · Cited by 654 · see citing works
Jersild, A. T. (2019) · Cited by 634 · see citing works
Standards for Internet-Based Experimenting
Reips, U. (2002) · Experimental Psychology (formerly Zeitschrift für Experimentelle Psychologie) · doi:10.1026//1618-3169.49.4.243 · Cited by 557 · see citing works
Financial incentives and the "performance of crowds"
Mason, W., & Watts, D. J. (2009) · doi:10.1145/1600150.1600175 · Cited by 544 · see citing works
Effects of masked stimuli on motor activation: Behavioral and electrophysiological evidence.
Eimer, M., & Schlaghecken, F. (1998) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037//0096-1523.24.6.1737 · Cited by 467 · see citing works
Eriksen, C. W. (1995) · Visual Cognition · doi:10.1080/13506289508401726 · Cited by 412 · see citing works
Economic games on the internet: the effect of $1 stakes
Amir, O., Rand, D. G., & Gal, Y. K. (2012) · PLoS ONE · doi:10.1371/journal.pone.0031461 · Cited by 364 · see citing works
Effects of masked stimuli on motor activation: Behavioral and electrophysiological evidence.
Eimer, M., & Schlaghecken, F. (1998) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037/0096-1523.24.6.1737 · Cited by 356 · see citing works
Cooperation and Contagion in Web-Based, Networked Public Goods Experiments
Suri, S., & Watts, D. J. (2011) · PLoS ONE · doi:10.1371/journal.pone.0016836 · Cited by 355 · see citing works
Does IOR occur in discrimination tasks? Yes, it does, but later
Lupiáñez, J., Milán, E. G., Tornay, F. J., Madrid, E., & Tudela, P. (1997) · Perception & Psychophysics · doi:10.3758/bf03214211 · Cited by 339 · see citing works
Craft, J. L., & Simon, J. (1970) · Journal of Experimental Psychology · doi:10.1037/h0028843 · Cited by 333 · see citing works
Cooperation and contagion in web-based, networked public goods experiments
Suri, S., & Watts, D. J. (2011) · ACM SIGecom Exchanges · doi:10.1145/1998549.1998550 · Cited by 318 · see citing works
Eyetracking and selective attention in category learning
Rehder, B., & Hoffman, A. B. (2005) · Cognitive Psychology · doi:10.1016/j.cogpsych.2004.11.001 · Cited by 315 · see citing works
Nosofsky, R. M., Gluck, M. A., Palmeri, T. J., McKinley, S. C., & Glauthier, P. (1994) · Memory & Cognition · doi:10.3758/bf03200862 · Cited by 292 · see citing works
Task Switching Across the Life Span: Effects of Age on General and Specific Switch Costs.
Reimers, S., & Maylor, E. A. (2005) · Developmental Psychology · doi:10.1037/0012-1649.41.4.661 · Cited by 273 · see citing works
Comparing supervised and unsupervised category learning
Love, B. C. (2002) · Psychonomic Bulletin & Review · doi:10.3758/bf03196342 · Cited by 242 · see citing works
Links between conscious awareness and response inhibition: Evidence from masked priming
Martin Eimer, A., & Schlaghecken, F. (2002) · Psychonomic Bulletin & Review · doi:10.3758/bf03196307 · Cited by 196 · see citing works
Task-set reconfiguration processes do not imply a control homunuculus: Reply to Altmann
Monsell, S. (2003) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(03)00170-0 · Cited by 182 · see citing works
Processing irrelevant location information: Practice and transfer effects in choice-reaction tasks
Proctor, R. W., & Lu, C.-H. (1999) · Memory & Cognition · doi:10.3758/bf03201214 · Cited by 150 · see citing works
Stroop-type interference: Congruity effects in color naming with typewritten responses.
Logan, G. D., & Zbrodoff, N. J. (1998) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037/0096-1523.24.3.978 · Cited by 112 · see citing works
Using Amazon Mechanical Turk for linguistic research
Schnoebelen, T., & Kuperman, V. (2010) · Psihologija · doi:10.2298/psi1004441s · Cited by 110 · see citing works
Working memory capacity and categorization: Individual differences and modeling.
Lewandowsky, S. (2011) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/a0022639 · Cited by 101 · see citing works
Adobe Flash as a medium for online experimentation: A test of reaction time measurement capabilities
Reimers, S., & Stewart, N. (2007) · Behavior Research Methods · doi:10.3758/bf03193004 · Cited by 93 · see citing works
Time resolution of clocks: Effects on reaction time measurement—Good news for bad clocks
Ulrich, R., & Giray, M. (1989) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/j.2044-8317.1989.tb01111.x · Cited by 65 · see citing works
Response time accuracy in Apple Macintosh computers
Neath, I., Earle, A., Hallett, D., & Surprenant, A. M. (2011) · Behavior Research Methods · doi:10.3758/s13428-011-0069-9 · Cited by 57 · see citing works
Conflict adaptation in time: Foreperiods as contextual cues for attentional adjustment
Wendt, M., & Kiesel, A. (2011) · Psychonomic Bulletin & Review · doi:10.3758/s13423-011-0119-4 · Cited by 56 · see citing works
The QWERTY Effect: How typing shapes the meanings of words.
Jasmin, K. M., & Casasanto, D. J. (2012) · Psychonomic Bulletin & Review · doi:10.3758/s13423-012-0229-7 · Cited by 53 · see citing works
Kurtz, K. J., Levering, K. R., Stanton, R. D., Romero, J., & Morris, S. N. (2012) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/a0029178 · Cited by 50 · see citing works
Flagged first, then by citations
1 of the 52 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- [13]Ipeirotis PG (2010) Demographics of Mechanical Turk (Tech. Rep. No. CeDER-10-01). New York: New York University. Available: http://hdl.handle.net/2451/29585. Accessed 2012 Jun 1.
Cited by
Sorted by citations
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00068 · Cited by 14,623 · see citing works
Prolific.ac—A subject pool for online experiments
Palan, S., & Schitter, C. (2018) · Journal of Behavioral and Experimental Finance · doi:10.1016/j.jbef.2017.12.004 · Cited by 3,450 · see citing works
Beyond the Turk: Alternative platforms for crowdsourcing behavioral research
Peer, E., Brandimarte, L., Samat, S., & Acquisti, A. (2017) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2017.01.006 · Cited by 2,957 · see citing works
Human-level concept learning through probabilistic program induction
Lake, B. M., Salakhutdinov, R., & Tenenbaum, J. B. (2015) · Science · doi:10.1126/science.aab3050 · Cited by 2,817 · see citing works
Research Methods in Applied Linguistics
(2021) · Research methods in applied linguistics · doi:10.1075/rmal · Cited by 2,782 · see citing works
Inside the Turk: Understanding Mechanical Turk as a Participant Pool
Paolacci, G., & Chandler, J. (2014) · Current Directions in Psychological Science · doi:10.1177/0963721414531598 · Cited by 2,136 · see citing works
Gorilla in our midst: An online behavioral experiment builder
Anwyl-Irvine, A. L., Massonnié, J., Flitton, A., Kirkham, N. Z., & Evershed, J. K. (2019) · Behavior Research Methods · doi:10.3758/s13428-019-01237-x · Cited by 1,878 · see citing works
jsPsych: A JavaScript library for creating behavioral experiments in a Web browser
de Leeuw, J. R. (2014) · Behavior Research Methods · doi:10.3758/s13428-014-0458-y · Cited by 1,783 · see citing works
Hauser, D. J., & Schwarz, N. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0578-z · Cited by 1,739 · see citing works
Conducting Clinical Research Using Crowdsourced Convenience Samples
Chandler, J., & Shapiro, D. (2016) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-021815-093623 · Cited by 1,072 · see citing works
MTurk Research: Review and Recommendations
Aguinis, H., Villamor, I., & Ramani, R. S. (2020) · Journal of Management · doi:10.1177/0149206320969787 · Cited by 942 · see citing works
Kees, J., Berry, C., Burton, S., & Sheehan, K. (2017) · Journal of Advertising · doi:10.1080/00913367.2016.1269304 · Cited by 902 · see citing works
Age and gender differences in self-esteem—A cross-cultural window.
(2015) · doi:10.1037/pspp0000078 · Cited by 684 · see citing works
Are samples drawn from Mechanical Turk valid for research on political ideology?
Clifford, S., Jewell, R., & Waggoner, P. (2015) · Research & Politics · doi:10.1177/2053168015622072 · Cited by 680 · see citing works
Characterizing a psychiatric symptom dimension related to deficits in goal-directed control
Gillan, C. M., Kosinski, M., Whelan, R., Phelps, E. A., & Daw, N. D. (2016) · eLife · doi:10.7554/elife.11305 · Cited by 586 · see citing works
The Value of Direct Replication
Simons, D. J. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691613514755 · Cited by 630 · see citing works
Validity and Mechanical Turk: An assessment of exclusion methods and interactive experiments
Thomas, K. A., & Clifford, S. (2017) · Computers in Human Behavior · doi:10.1016/j.chb.2017.08.038 · Cited by 625 · see citing works
Crowdsourcing Consumer Research
Goodman, J. K., & Paolacci, G. (2017) · Journal of Consumer Research · doi:10.1093/jcr/ucx047 · Cited by 597 · see citing works
Large-scale analysis of test–retest reliabilities of self-regulation measures
Enkavi, A. Z., Eisenberg, I. W., Bissett, P. G., Mazza, G. L., MacKinnon, D. P., Marsch, L. A., & Poldrack, R. A. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1818430116 · Cited by 489 · see citing works
Abbey, J. D., & Meloy, M. G. (2017) · Journal of Operations Management · doi:10.1016/j.jom.2017.06.001 · Cited by 522 · see citing works
The development and initial validation of the Decent Work Scale.
Duffy, R. D., Allan, B. A., England, J. W., Blustein, D. L., Autin, K. L., Douglass, R. P., Ferreira, J. A., & Santos, E. J. R. (2017) · Journal of Counseling Psychology · doi:10.1037/cou0000191 · Cited by 528 · see citing works
Lange, K., Kühn, S., & Filevich, E. (2015) · PLoS ONE · doi:10.1371/journal.pone.0130834 · Cited by 481 · see citing works
Internet Research in Psychology
Gosling, S. D., & Mason, W. (2015) · Annual Review of Psychology · doi:10.1146/annurev-psych-010814-015321 · Cited by 458 · see citing works
Amazon mechanical Turk in organizational psychology: an evaluation and practical recommendations
Cheung, J. H., Burns, D. K., Sinclair, R. R., & Sliter, M. (2017) · Journal of Business and Psychology · doi:10.1007/s10869-016-9458-5 · Cited by 474 · see citing works
Johnson, V., Kiser, A. I. T., Washington, R., & Torres, R. (2017) · Computers in Human Behavior · doi:10.1016/j.chb.2017.10.035 · Cited by 464 · see citing works
A review of experiments in tourism and hospitality
(2020) · doi:10.1016/j.annals.2020.102858 · Cited by 455 · see citing works
Headphone screening to facilitate web-based auditory experiments
Woods, K. J. P., Siegel, M., Traer, J. A., & McDermott, J. H. (2017) · Attention Perception & Psychophysics · doi:10.3758/s13414-017-1361-2 · Cited by 434 · see citing works
Inflation Expectations, Learning, and Supermarket Prices: Evidence from Survey Experiments
Cavallo, A. F., Cruces, G., & Truglia, R. N. P. (2017) · American Economic Journal Macroeconomics · doi:10.1257/mac.20150147 · Cited by 424 · see citing works
From Patches to Pictures (PaQ-2-PiQ): Mapping the Perceptual Space of Picture Quality
Ying, Z., Niu, H., Gupta, P., Mahajan, D., Ghadiyaram, D., & Bovik, A. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00363 · Cited by 406 · see citing works
Meta-analysis of emotion recognition deficits in major depressive disorder
Dalili, M. N., Penton-Voak, I. S., Harmer, C. J., & Munafò, M. R. (2015) · Psychological Medicine · doi:10.1017/s0033291714002591 · Cited by 349 · see citing works
The Use of Online Panel Data in Management Research: A Review and Recommendations
Porter, C. O. L. H., Outlaw, R., Gale, J., & Cho, T. S. (2018) · Journal of Management · doi:10.1177/0149206318811569 · Cited by 346 · see citing works
Keith, M. G., Tay, L., & Harms, P. D. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.01359 · Cited by 319 · see citing works
Kim, J., Giroux, M., & Lee, J. C. (2021) · Psychology & Marketing · doi:10.1002/mar.21498 · Cited by 353 · see citing works
Crowdsourcing—Harnessing the Masses to Advance Health and Medicine, a Systematic Review
Ranard, B. L., Ha, Y. P., Meisel, Z. F., Asch, D. A., Hill, S. S., Becker, L. B., Seymour, A. K., & Merchant, R. M. (2013) · Journal of General Internal Medicine · doi:10.1007/s11606-013-2536-8 · Cited by 324 · see citing works
Conducting perception research over the internet: A tutorial review
Woods, A. T., Velasco, C., Levitan, C. A., Wan, X., & Spence, C. (2015) · PeerJ · doi:10.7717/peerj.1058 · Cited by 301 · see citing works
lab.js: A free, open, online study builder
Henninger, F., Shevchenko, Y., Mertens, U. K., Kieslich, P. J., & Hilbig, B. E. (2021) · Behavior Research Methods · doi:10.3758/s13428-019-01283-5 · Cited by 307 · see citing works
How supply chain analytics enables operational supply chain transparency
Zhu, S., Song, J., Hazen, B. T., Lee, K., & Cegielski, C. G. (2018) · International Journal of Physical Distribution & Logistics Management · doi:10.1108/ijpdlm-11-2017-0341 · Cited by 307 · see citing works
Model-based learning protects against forming habits
Gillan, C. M., Otto, A. R., Phelps, E. A., & Daw, N. D. (2015) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/s13415-015-0347-6 · Cited by 283 · see citing works
Scoundrels or Stars? Theory and Evidence on the Quality of Workers in Online Labor Markets
Farrell, A. M., Grenier, J. H., & Leiby, J. (2016) · The Accounting Review · doi:10.2308/accr-51447 · Cited by 274 · see citing works
No Harm in Checking: Using Factual Manipulation Checks to Assess Attentiveness in Experiments
Kane, J., & Barabas, J. (2018) · American Journal of Political Science · doi:10.1111/ajps.12396 · Cited by 276 · see citing works
Effect of tourist photographs on attitudes towards destination: Manifest and latent content
Kim, H., & Stepchenkova, S. O. (2015) · Tourism Management · doi:10.1016/j.tourman.2015.02.004 · Cited by 272 · see citing works
Laupichler, M. C., Aster, A., Haverkamp, N., & Raupach, T. (2023) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2023.100338 · Cited by 269 · see citing works
psiTurk: An open-source framework for conducting replicable behavioral experiments online
Gureckis, T. M., Martin, J. B., McDonnell, J. V., Rich, A. S., Markant, D. B., Coenen, A., Halpern, D., Hamrick, J. B., & Chan, P. (2015) · Behavior Research Methods · doi:10.3758/s13428-015-0642-8 · Cited by 256 · see citing works
de Leeuw, J. R., & Motz, B. A. (2015) · Behavior Research Methods · doi:10.3758/s13428-015-0567-2 · Cited by 242 · see citing works
The use of crowdsourcing in addiction science research: Amazon Mechanical Turk.
Strickland, J. C., & Stoops, W. W. (2018) · Experimental and Clinical Psychopharmacology · doi:10.1037/pha0000235 · Cited by 249 · see citing works
Sacrifice One For the Good of Many?: People Apply Different Moral Norms to Human and Robot Agents
Malle, B. F., Scheutz, M., Arnold, T., Voiklis, J., & Cusimano, C. (2015) · Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction · doi:10.1145/2696454.2696458 · Cited by 246 · see citing works
Adaptive trust calibration for human-AI collaboration
Okamura, K., & Yamada, S. (2020) · PLOS ONE · doi:10.1371/journal.pone.0229132 · Cited by 239 · see citing works
Reaction time effects in lab- versus Web-based research: Experimental evidence
Hilbig, B. E. (2015) · Behavior Research Methods · doi:10.3758/s13428-015-0678-9 · Cited by 224 · see citing works
Crowdsourcing Samples in Cognitive Science
Stewart, N., Chandler, J., & Paolacci, G. (2017) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2017.06.007 · Cited by 216 · see citing works
Franklin, J. C., Fox, K. R., Franklin, C. R., Kleiman, E. M., Ribeiro, J. D., Jaroszewski, A. C., Hooley, J. M., & Nock, M. K. (2016) · Journal of Consulting and Clinical Psychology · doi:10.1037/ccp0000093 · Cited by 217 · see citing works
Self-Affirmation Reduces Delay Discounting of the Financially Deprived
Moeini-Jazani, M., Albalooshi, S., & Seljeseth, I. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.01729 · Cited by 77 · see citing works
Robinson, J., Rosenzweig, C., Moss, A. J., & Litman, L. (2019) · PLoS ONE · doi:10.1371/journal.pone.0226394 · Cited by 203 · see citing works
Boss phubbing, trust, job satisfaction and employee performance
Roberts, J. A., & David, M. E. (2019) · Personality and Individual Differences · doi:10.1016/j.paid.2019.109702 · Cited by 225 · see citing works
Perceptions and Price: Evidence from CEO Presentations at IPO Roadshows
Blankespoor, E., Hendricks, B. E., & Miller, G. S. (2017) · Journal of Accounting Research · doi:10.1111/1475-679x.12164 · Cited by 192 · see citing works
Artificial Intelligence and Declined Guilt: Retailing Morality Comparison Between Human and AI
Giroux, M., Kim, J., Lee, J. C., & Park, J. (2022) · Journal of Business Ethics · doi:10.1007/s10551-022-05056-7 · Cited by 194 · see citing works
Presentation and response timing accuracy in Adobe Flash and HTML5/JavaScript Web experiments
Reimers, S., & Stewart, N. (2014) · Behavior Research Methods · doi:10.3758/s13428-014-0471-1 · Cited by 171 · see citing works
Metacognitive Failure as a Feature of Those Holding Radical Beliefs
Rollwage, M., Dolan, R. J., & Fleming, S. M. (2018) · Current Biology · doi:10.1016/j.cub.2018.10.053 · Cited by 193 · see citing works
Spontaneous synchronization to speech reveals neural mechanisms facilitating language learning
Assaneo, M. F., Ripollés, P., Orpella, J., Lin, W. M., de Diego‐Balaguer, R., & Poeppel, D. (2019) · Nature Neuroscience · doi:10.1038/s41593-019-0353-z · Cited by 191 · see citing works
Prosocial Conformity: Prosocial Norms Generalize Across Behavior and Empathy
Nook, E. C., Ong, D. C., Morelli, S. A., Mitchell, J. P., & Zaki, J. (2016) · Personality and Social Psychology Bulletin · doi:10.1177/0146167216649932 · Cited by 186 · see citing works
Cognitive Control Predicts Use of Model-based Reinforcement Learning
Otto, A. R., Skatova, A., Madlon‐Kay, S., & Daw, N. D. (2014) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_00709 · Cited by 186 · see citing works
Li, Q., Joo, S. J., Yeatman, J. D., & Reinecke, K. (2020) · Scientific Reports · doi:10.1038/s41598-019-57204-1 · Cited by 185 · see citing works
Inferring word meanings by assuming that speakers are informative
Frank, M. C., & Goodman, N. D. (2014) · Cognitive Psychology · doi:10.1016/j.cogpsych.2014.08.002 · Cited by 179 · see citing works
Stritch, J. M., Pedersen, M. J., & Taggart, G. (2017) · International Public Management Journal · doi:10.1080/10967494.2016.1276493 · Cited by 181 · see citing works
Miller, J. D., Crowe, M., Weiss, B., Maples-Keller, J. L., & Lynam, D. R. (2017) · Personality Disorders Theory Research and Treatment · doi:10.1037/per0000191 · Cited by 179 · see citing works
Noyes, E., Davis, J. P., Petrov, N., Gray, K. L. H., & Ritchie, K. L. (2021) · Royal Society Open Science · doi:10.1098/rsos.201169 · Cited by 179 · see citing works
QRTEngine: An easy solution for running online reaction time experiments using Qualtrics
Barnhoorn, J. S., Haasnoot, E., Bocanegra, B. R., & van Steenbergen, H. (2015) · Behavior Research Methods · doi:10.3758/s13428-014-0530-7 · Cited by 166 · see citing works
Using Crowdsourcing Websites for Sociological Research: The Case of Amazon Mechanical Turk
Shank, D. B. (2015) · The American Sociologist · doi:10.1007/s12108-015-9266-9 · Cited by 176 · see citing works
Mueller, M. L., Dunlosky, J. T., Tauber, S. K., & Rhodes, M. G. (2013) · Journal of Memory and Language · doi:10.1016/j.jml.2013.09.007 · Cited by 173 · see citing works
Visual working memory directly alters perception
Teng, C., & Kravitz, D. J. (2019) · Nature Human Behaviour · doi:10.1038/s41562-019-0640-4 · Cited by 169 · see citing works
Strategic offloading of delayed intentions into the external environment
Gilbert, S. J. (2015) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470218.2014.972963 · Cited by 164 · see citing works
Lukasik, K. M., Waris, O., Soveri, A., Lehtonen, M., & Laine, M. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.00004 · Cited by 177 · see citing works
Gilbert, S. J. (2015) · Consciousness and Cognition · doi:10.1016/j.concog.2015.01.006 · Cited by 170 · see citing works
A Note on the Use of Survey Research Firms to Enable Empirical Data Collection
Schoenherr, T., Ellram, L. M., & Tate, W. L. (2015) · Journal of Business Logistics · doi:10.1111/jbl.12092 · Cited by 169 · see citing works
Studying determinants of crowd-shipping use
Punel, A., Ermagun, A., & Stathopoulos, A. (2018) · Travel Behaviour and Society · doi:10.1016/j.tbs.2018.03.005 · Cited by 166 · see citing works
Building, Hosting and Recruiting: A Brief Introduction to Running Behavioral Experiments Online
Sauter, M., Draschkow, D., & Mack, W. (2020) · Brain Sciences · doi:10.3390/brainsci10040251 · Cited by 164 · see citing works
Online psychophysics: reaction time effects in cognitive experiments
Semmelmann, K., & Weigelt, S. (2016) · Behavior Research Methods · doi:10.3758/s13428-016-0783-4 · Cited by 164 · see citing works
Mechanical Turk data collection in addiction research: utility, concerns and best practices
Mellis, A. M., & Bickel, W. K. (2020) · Addiction · doi:10.1111/add.15032 · Cited by 164 · see citing works
A High Bar or a Double Standard? Gender, Competence, and Information in Political Campaigns
Ditonto, T. (2016) · Political Behavior · doi:10.1007/s11109-016-9357-5 · Cited by 162 · see citing works
Why women don’t run: Experimental evidence on gender differences in political competition aversion
Preece, J., & Stoddard, O. B. (2015) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2015.04.019 · Cited by 161 · see citing works
Necka, E. A., Cacioppo, S., Norman, G. J., & Cacioppo, J. T. (2016) · PLoS ONE · doi:10.1371/journal.pone.0157732 · Cited by 161 · see citing works
Serving You Depletes Me? A Leader-Centric Examination of Servant Leadership Behaviors
Liao, C., Lee, H. W., Johnson, R. E., & Lin, S. (2020) · Journal of Management · doi:10.1177/0149206320906883 · Cited by 165 · see citing works
Batching smartphone notifications can improve well-being
Fitz, N., Kushlev, K., Jagannathan, R., Lewis, T., Paliwal, D., & Ariely, D. (2019) · Computers in Human Behavior · doi:10.1016/j.chb.2019.07.016 · Cited by 142 · see citing works
The role of a “common is moral” heuristic in the stability and change of moral norms.
Lindström, B., Jangard, S., Selbing, I., & Olsson, A. (2017) · Journal of Experimental Psychology General · doi:10.1037/xge0000365 · Cited by 150 · see citing works
Improving the Efficacy of Games for Change Using Personalization Models
Orji, R., Mandryk, R. L., & Vassileva, J. (2017) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3119929 · Cited by 157 · see citing works
Comparing attentional disengagement between prolific and MTurk samples
Albert, D. A., & Smilek, D. (2023) · Scientific Reports · doi:10.1038/s41598-023-46048-5 · Cited by 166 · see citing works
Common Concerns with MTurk as a Participant Pool: Evidence and SolutionsPreprint
Hauser, D., Paolacci, G., & Chandler, J. (2018) · doi:10.31234/osf.io/uq45c · Cited by 156 · see citing works
Atoms of recognition in human and computer vision
Ullman, S., Assif, L., Fetaya, E., & Harari, D. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1513198113 · Cited by 151 · see citing works
Jack, A. I., Friedman, J. P., Boyatzis, R. E., & Taylor, S. N. (2016) · PLoS ONE · doi:10.1371/journal.pone.0149989 · Cited by 155 · see citing works
The racially diverse affective expression (RADIATE) face stimulus set
Conley, M. I., Dellarco, D. V., Rubien-Thomas, E., Cohen, A. O., Cervera, A., Tottenham, N., & Casey, B. J. (2018) · Psychiatry Research · doi:10.1016/j.psychres.2018.04.066 · Cited by 151 · see citing works
The opportunity cost of time modulates cognitive effort
Otto, A. R., & Daw, N. D. (2019) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2018.05.006 · Cited by 138 · see citing works
Mapping of Crowdsourcing in Health: Systematic Review
Créquit, P., Mansouri, G., Benchoufi, M., Vivot, A., & Ravaud, P. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.9330 · Cited by 135 · see citing works
Determinants of congruency sequence effects without learning and memory confounds.
Weissman, D. H., Jiang, J., & Egner, T. (2014) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037/a0037454 · Cited by 144 · see citing works
Episodic memories predict adaptive value-based decision-making.
Murty, V. P., FeldmanHall, O., Hunter, L. E., Phelps, E. A., & Davachi, L. (2016) · Journal of Experimental Psychology General · doi:10.1037/xge0000158 · Cited by 140 · see citing works
Influencing climate change attitudes in the United States: A systematic review and meta-analysis
Rode, J. B., Dent, A. L., Benedict, C. N., Brosnahan, D., Martinez, R. L., & Ditto, P. H. (2021) · Journal of Environmental Psychology · doi:10.1016/j.jenvp.2021.101623 · Cited by 144 · see citing works
Advantages and Disadvantages of Using Internet-Based Survey Methods in Aviation-Related Research
Rice, S. C., Winter, S. R., Doherty, S. M., & Milner, M. N. (2017) · Journal of Aviation Technology and Engineering · doi:10.7771/2159-6670.1160 · Cited by 142 · see citing works
Saravi, B., Hassel, F., Ülkümen, S., Zink, A., Shavlokhova, V., Couillard‐Després, S., Boeker, M., Obid, P., & Lang, G. M. (2022) · Journal of Personalized Medicine · doi:10.3390/jpm12040509 · Cited by 146 · see citing works
Jones, D. N. (2014) · Personality and Individual Differences · doi:10.1016/j.paid.2014.01.030 · Cited by 137 · see citing works
The smarter, the better?! Customer well-being, engagement, and perceptions in smart service systems
Henkens, B., Verleye, K., & Larivière, B. (2020) · International Journal of Research in Marketing · doi:10.1016/j.ijresmar.2020.09.006 · Cited by 141 · see citing works
Formalizing Neurath’s ship: approximate algorithms for online causal learning.
Bramley, N. R., Dayan, P., Griffiths, T. L., & Lagnado, D. A. (2017) · Psychological Review · doi:10.1037/rev0000061 · Cited by 136 · see citing works
Using MTurk to Distribute a Survey or Experiment: Methodological Considerations
Hunt, N. C., & Scheetz, A. M. (2018) · Journal of Information Systems · doi:10.2308/isys-52021 · Cited by 136 · see citing works
- Authors
- Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M.
- Affiliations
- City University of New York (US)
- New York University (US)
- Corresponding author
- Crump, Matthew J. C.
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2013-03-13
- Volume
- 8
- Issue
- 3
- Pages
- e57410
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC3596391
- OpenAlex
- W2080211902
- Cited by
- 1,617
- References
- 52
- Citations by year
- 2026: 25 · 2025: 63 · 2024: 83 · 2023: 108 · 2022: 162 · 2021: 171 · 2020: 164 · 2019: 138 · 2018: 164 · 2017: 155 · 2016: 146 · 2015: 129 · 2014: 86 · 2013: 20
- FWCI
- 192.44
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS ONE (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- CUNY Academic Works (City University of New York) (submittedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Social Sciences › Safety Research › Experimental Behavioral Economics Studies
- Keywords
- Amazon Mechanical Turk, online crowdsourcing, replication, Stroop task, attentional blink, subliminal priming, category learning, Simon task, flanker task, stimulus presentation
- MeSH terms
- Attention, Cues, Humans, Learning, Reaction Time, Internet, Behavioral Research, Learning Curve
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
