Systems perspective of Amazon mechanical Turk for organizational research: review and recommendations
Open accessAbstract
Amazon Mechanical Turk (MTurk) is becoming a prevalent source of quick and cost effective data for organizational research, but there are questions about the appropriateness of the platform for organizational research. To answer these questions, we conducted an integrative review based on 75 papers evaluating the MTurk platform and 250 MTurk samples used in organizational research. This integrative review provides four contributions: (1) we analyze the trends associated with the use of MTurk samples in organizational research; (2) we develop a systems perspective (recruitment system, selection system, and work management system) to synthesize and organize the key factors influencing data collected on MTurk that may affect generalizability and data quality; (3) within each factor, we also use available MTurk samples from the organizational literature to analyze key issues (e.g., sample characteristics, use of attention checks, payment); and (4) based on our review, we provide specific recommendations and a checklist for data reporting in order to improve data transparency and enable further research on this issue.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The attrition observed in the present studies is consistent with patterns reported in the broader crowdsourcing literature.
“In this context, the attrition observed in the present studies is consistent with patterns reported in the broader crowdsourcing literature.”
From Testing the testing effect on prolific: when retrieval practice fails to boost learning (2026) · cites this paper as Keith et al., 2017
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996 · see citing works
Schneider, B. (1987) · Personnel Psychology · doi:10.1111/j.1744-6570.1987.tb00609.x · Cited by 4,634 · see citing works
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008 · see citing works
Identifying careless responses in survey data.
Meade, A. W., & Craig, S. B. (2012) · Psychological Methods · doi:10.1037/a0028085 · Cited by 3,865 · see citing works
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767 · see citing works
Conducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172 · see citing works
Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242 · see citing works
Inside the Turk: Understanding Mechanical Turk as a Participant Pool
Paolacci, G., & Chandler, J. (2014) · Current Directions in Psychological Science · doi:10.1177/0963721414531598 · Cited by 2,136 · see citing works
TurkPrime.com: A versatile crowdsourcing data acquisition platform for the behavioral sciences
Litman, L., Robinson, J., & Abberbock, T. (2016) · Behavior Research Methods · doi:10.3758/s13428-016-0727-z · Cited by 1,978 · see citing works
Crowdsourcing user studies with Mechanical Turk
Kittur, A., Ed, H., & Suh, B. (2008) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357127 · Cited by 1,941 · see citing works
Casler, K., Bickel, L., & Hackett, E. (2013) · Computers in Human Behavior · doi:10.1016/j.chb.2013.05.009 · Cited by 1,662 · cited 12× · see citing works
“3 Paolacci et al., 2010; Behrend et al., 2011; Berinsky et al., 2012; Goodman et al., 2012; Johnson and Borden, 2012; Simons and Chabris, 2012; Casler et al., 2013; Steelman et al., 2014; Feitosa et al., 2015; Roulin, 2015.”
BackgroundResultInfluentialReputation as a sufficient condition for data quality on Amazon Mechanical Turk
Peer, E., Vosgerau, J., & Acquisti, A. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0434-y · Cited by 1,618 · see citing works
Evaluating amazon’s mechanical turk as a tool for experimental behavioral research
Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0057410 · Cited by 1,617 · see citing works
The Online Laboratory: Conducting Experiments in a Real Labor Market∗Preprint
Zeckhauser, R. J. (2016) · Cited by 1,454 · see citing works
Investigating Variation in Replicability: A “Many Labs” Replication Project
Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Bahník, Š., Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Brumbaugh, C. C., et al. (2014) · Social Psychology · doi:10.1027/1864-9335/a000178 · Cited by 1,208 · see citing works
Using Mechanical Turk to Study Clinical Populations
Shapiro, D. N., Chandler, J., & Mueller, P. A. (2013) · Clinical Psychological Science · doi:10.1177/2167702612469015 · Cited by 1,139 · see citing works
The viability of crowdsourcing for survey research
Behrend, T. S., Sharek, D. J., Meade, A. W., & Wiebe, E. (2011) · Behavior Research Methods · doi:10.3758/s13428-011-0081-0 · Cited by 1,049 · see citing works
Is psychology suffering from a replication crisis? What does “failure to replicate” really mean?
Maxwell, S. E., Lau, M. Y., & Howard, G. S. (2015) · American Psychologist · doi:10.1037/a0039400 · Cited by 1,018 · see citing works
Chandler, J., Mueller, P., & Paolacci, G. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0365-7 · Cited by 950 · see citing works
Eby, L. T. D. T., Hurst, C. S., & Butts, M. M. (2008) · Cited by 926 · see citing works
The influence of management journals in the 1980s and 1990s
Podsakoff, P. M., MacKenzie, S. B., Bachrach, D. G., & Podsakoff, N. P. (2005) · Strategic Management Journal · doi:10.1002/smj.454 · Cited by 912 · cited 1× · see citing works
“These journals represent top journals in the field of organization science (Zickar and Highhouse, 2001; Podsakoff et al., 2005), and give a good indication of how MTurk is currently being utilized in the organizational sciences.”
BackgroundAmazon Mechanical Turk: Gold Mine or Coal Mine?
Fort, K., Adda, G., & Cohen, K. B. (2011) · Computational Linguistics · doi:10.1162/coli_a_00057 · Cited by 772 · see citing works
Assessing the non-random sampling effects of subject attrition in longitudinal research
Goodman, J. S., & Blum, T. C. (1996) · Journal of Management · doi:10.1177/014920639602200405 · Cited by 766 · see citing works
Landers, R. N., & Behrend, T. S. (2015) · Industrial and Organizational Psychology · doi:10.1017/iop.2015.13 · Cited by 714 · see citing works
Is the Replicability Crisis Overblown? Three Arguments Examined
Pashler, H., & Harris, C. R. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463401 · Cited by 697 · see citing works
Best practice recommendations for data screening
DeSimone, J. A., Harms, P. D., & DeSimone, A. J. (2014) · Journal of Organizational Behavior · doi:10.1002/job.1962 · Cited by 690 · see citing works
Self-citationRand, D. G. (2011) · Journal of Theoretical Biology · doi:10.1016/j.jtbi.2011.03.004 · Cited by 686 · see citing works
Gustavson, K., von Soest, T., Karevold, E., & Røysamb, E. (2012) · BMC Public Health · doi:10.1186/1471-2458-12-918 · Cited by 679 · see citing works
Financial incentives and the "performance of crowds"
Mason, W., & Watts, D. J. (2009) · doi:10.1145/1600150.1600175 · Cited by 544 · see citing works
Demographics of Mechanical Turk
Ipeirotis, P. G. (2010) · Faculty Digital Archive (New York University Florence) · Cited by 494 · see citing works
Milkman, K. L., Akinola, M., & Chugh, D. (2015) · Journal of Applied Psychology · doi:10.1037/apl0000022 · Cited by 488 · see citing works
Amazon mechanical Turk in organizational psychology: an evaluation and practical recommendations
Cheung, J. H., Burns, D. K., Sinclair, R. R., & Sliter, M. (2017) · Journal of Business and Psychology · doi:10.1007/s10869-016-9458-5 · Cited by 474 · see citing works
More than fun and money. Worker motivation in crowdsourcing - a study on mechanical turk
Kaufmann, N., Schulze, T., & Veit, D. J. (2011) · MADOC (University of Mannheim) · Cited by 469 · see citing works
Financial incentives and the "performance of crowds"
Mason, W., & Watts, D. J. (2010) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1809400.1809422 · Cited by 463 · see citing works
Zhou, H., & Ayelet Fishbach, B. (2016) · Journal of Personality and Social Psychology · doi:10.1037/pspa0000056 · Cited by 462 · see citing works
Are your participants gaming the system?
Downs, J. S., Holbrook, M. B., Sheng, S., & Cranor, L. F. (2010) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753688 · Cited by 422 · see citing works
Insufficient effort responding: Examining an insidious confound in survey data.
Huang, J. L. W., Liu, M., & Bowling, N. A. (2014) · Journal of Applied Psychology · doi:10.1037/a0038510 · Cited by 421 · see citing works
A reliability analysis of Mechanical Turk data
Rouse, S. V. (2014) · Computers in Human Behavior · doi:10.1016/j.chb.2014.11.004 · Cited by 414 · see citing works
Data Collection in the Digital Age: Innovative Alternatives to Student Samples1
Steelman, Z. R., Hammer, B. I., & Limayem, M. (2014) · MIS Quarterly · doi:10.25300/misq/2014/38.2.02 · Cited by 406 · see citing works
Employee recruitment: Current knowledge and important areas for future research
Breaugh, J. A. (2008) · Human Resource Management Review · doi:10.1016/j.hrmr.2008.07.003 · Cited by 403 · see citing works
The slippery slope: How small ethical transgressions pave the way for larger future transgressions.
Welsh, D. T., Ordóñez, L. D., Snyder, D. G., & Christian, M. S. (2014) · Journal of Applied Psychology · doi:10.1037/a0036950 · Cited by 392 · see citing works
Economic games on the internet: the effect of $1 stakes
Amir, O., Rand, D. G., & Gal, Y. K. (2012) · PLoS ONE · doi:10.1371/journal.pone.0031461 · Cited by 364 · see citing works
Authenticity and Consumer Value Ratings: Empirical Tests from the Restaurant Domain
Kovács, B., Carroll, G. R., & Lehman, D. W. (2013) · Organization Science · doi:10.1287/orsc.2013.0843 · Cited by 318 · see citing works
Bartneck, C., Duenser, A., Moltchanova, E., & Zawieska, K. (2015) · PLOS ONE · doi:10.1371/journal.pone.0121595 · Cited by 317 · see citing works
Implications of direct and indirect range restriction for meta-analysis methods and findings.
Hunter, J. E., Schmidt, F. L., & Le, H. (2006) · Journal of Applied Psychology · doi:10.1037/0021-9010.91.3.594 · Cited by 315 · see citing works
Huang, J. L. W., Bowling, N. A., Liu, M., & Li, Y. (2014) · Journal of Business and Psychology · doi:10.1007/s10869-014-9357-6 · Cited by 315 · see citing works
Brandon, D. M., Long, J. H., Loraas, T. M., Mueller, J. M., & Vansant, B. (2013) · Behavioral Research in Accounting · doi:10.2308/bria-50651 · Cited by 315 · see citing works
The importance of assessing clinical phenomena in Mechanical Turk research.
Hall, K. A. A., Çek, D., Shaw, A. M., & Timpano, K. R. (2015) · Psychological Assessment · doi:10.1037/pas0000217 · Cited by 311 · see citing works
The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers
Stewart, N., Ungemach, C., Harris, A. J. L., Bartels, D. M., Newell, B. R., Paolacci, G., & Chandler, J. (2015) · Judgment and Decision Making · doi:10.1017/s1930297500005611 · Cited by 304 · see citing works
The Contaminating Effects of Building Instrumental Ties
Casciaro, T., Francesca Gino, B., & Kouchaki, M. (2014) · Administrative Science Quarterly · doi:10.1177/0001839214554990 · Cited by 278 · see citing works
Litman, L., Robinson, J., & Rosenzweig, C. (2014) · Behavior Research Methods · doi:10.3758/s13428-014-0483-x · Cited by 275 · see citing works
An Assessment of Intrinsic and Extrinsic Motivation on Task Performance in Crowdsourcing Markets.
Rogstadius, J., Kostakos, V., Kittur, A., et al. (2011) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.13140/rg.2.2.19170.94401 · Cited by 268 · see citing works
A mazon Mechanical Turk and the commodification of labour
Bergvall‐Kåreborn, B., & Howcroft, D. (2014) · New Technology Work and Employment · doi:10.1111/ntwe.12038 · Cited by 258 · see citing works
Individual and work factors related to perceived work ability and labor force outcomes.
McGonagle, A. K., Fisher, G. G., Barnes-Farrell, J. L., & Grosch, J. W. (2014) · Journal of Applied Psychology · doi:10.1037/a0037974 · Cited by 254 · see citing works
Mindfulness buffers retaliatory responses to injustice: A regulatory approach.
Long, E. C., & Christian, M. S. (2015) · Journal of Applied Psychology · doi:10.1037/apl0000019 · Cited by 226 · see citing works
Stanley, D. J., & Spence, J. R. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614528518 · Cited by 211 · see citing works
Bowling, N. A., Huang, J. L. W., Bragg, C. B., Khazon, S., Liu, M., & Blackmore, C. E. (2016) · Journal of Personality and Social Psychology · doi:10.1037/pspp0000085 · Cited by 206 · see citing works
Science and Ethics in Conducting, Analyzing, and Reporting Psychological Research
Rosenthal, R. (1994) · Psychological Science · doi:10.1111/j.1467-9280.1994.tb00646.x · Cited by 201 · see citing works
Response Rates in National Panel Surveys
Schoeni, R. F., Stafford, F. P., McGonagle, K. A., & Andreski, P. M. (2012) · The Annals of the American Academy of Political and Social Science · doi:10.1177/0002716212456363 · Cited by 200 · see citing works
Psychological research in the internet age: The quality of web-based data
Ramsey, S. R., Thompson, K. L., McKenzie, M. D., & Rosenbaum, A. (2016) · Computers in Human Behavior · doi:10.1016/j.chb.2015.12.049 · Cited by 199 · see citing works
A convenient solution: Using MTurk to sample from hard-to-reach populations
Smith, N. A., Sabat, I. E., Martinez, L. R., Weaver, K., & Xu, S. (2015) · Industrial and Organizational Psychology · doi:10.1017/iop.2015.29 · Cited by 190 · see citing works
Work experiences on MTurk: Job satisfaction, turnover, and information sharing
Brawley, A. M., & Pury, C. L. S. (2016) · Computers in Human Behavior · doi:10.1016/j.chb.2015.08.031 · Cited by 188 · see citing works
Using Nonnaive Participants Can Reduce Effect Sizes
Chandler, J., Paolacci, G., Peer, E., Mueller, P., & Ratliff, K. A. (2015) · Psychological Science · doi:10.1177/0956797615585115 · Cited by 185 · see citing works
Credé, M., & Harms, P. D. (2015) · Journal of Organizational Behavior · doi:10.1002/job.2008 · Cited by 181 · see citing works
Self-citationSimons, D. J., & Chabris, C. F. (2012) · PLoS ONE · doi:10.1371/journal.pone.0051876 · Cited by 180 · see citing works
How Careless Responding and Acquiescence Response Bias Can Influence Construct Dimensionality
Kam, C. C. S., & Meyer, J. P. (2015) · Organizational Research Methods · doi:10.1177/1094428115571894 · Cited by 180 · see citing works
It’s a Trap! Instructional Manipulation Checks Prompt Systematic Thinking on “Tricky” Tasks
Hauser, D., & Schwarz, N. (2015) · SAGE Open · doi:10.1177/2158244015584617 · Cited by 173 · see citing works
Task search in a human computation market
Chilton, L. B., Horton, J. J., Miller, R. C., & Azenkot, S. (2010) · doi:10.1145/1837885.1837889 · Cited by 172 · see citing works
Who gets credit for input? Demographic and structural status cues in voice recognition.
Howell, T. M., Harrison, D. A., Burris, E. R., & Detert, J. R. (2015) · Journal of Applied Psychology · doi:10.1037/apl0000025 · Cited by 171 · see citing works
Assessing the reliability of the M5-120 on Amazon’s mechanical Turk
Holden, C. J., Dennie, T., & Hicks, A. D. (2013) · Computers in Human Behavior · doi:10.1016/j.chb.2013.02.020 · Cited by 169 · see citing works
Using crowdsourcing for TREC relevance assessment
Alonso, O., & Mizzaro, S. (2012) · Information Processing & Management · doi:10.1016/j.ipm.2012.01.004 · Cited by 161 · see citing works
Samples in applied psychology: Over a decade of research in review.
Shen, W., Kiger, T. B., Davies, S. E., Rasch, R. L., Simon, K. M., & Ones, D. S. (2011) · Journal of Applied Psychology · doi:10.1037/a0023322 · Cited by 160 · see citing works
Newman, D. A., & Lyon, J. S. (2009) · Journal of Applied Psychology · doi:10.1037/a0013472 · Cited by 158 · see citing works
Inattentive responding in MTurk and other online samples
Fleischer, A., Mead, A. D., & Huang, J. (2015) · Industrial and Organizational Psychology · doi:10.1017/iop.2015.25 · Cited by 151 · see citing works
Chua, R. Y. J. (2012) · Academy of Management Journal · doi:10.5465/amj.2011.0971 · Cited by 149 · see citing works
Bergman, M., & Jean, V. A. (2015) · Industrial and Organizational Psychology · doi:10.1017/iop.2015.70 · Cited by 138 · see citing works
Participant Self-Selection Biases as a Function of Individual Differences in Time Perspective
Harber, K. D., Zimbardo, P. G., & Boyd, J. N. (2003) · Basic and Applied Social Psychology · doi:10.1207/s15324834basp2503_08 · Cited by 136 · see citing works
Who are the Turkers? Worker Demographics in Amazon Mechanical Turk
Ross, J., Zaldivar, A., Irani, L. C., & Tomlinson, B. (2009) · Sound Ideas (University of Puget Sound) · Cited by 136 · see citing works
Do Samples Really Matter That Much?
Highhouse, S., & Gillespie, J. Z. (2010) · Routledge eBooks · doi:10.4324/9780203867266-19 · Cited by 136 · see citing works
Society for Industrial and Organizational Psychology
Tetrick, L. E. (2010) · The Corsini Encyclopedia of Psychology · doi:10.1002/9780470479216.corpsy0906 · Cited by 135 · see citing works
Karim, M. N., Kaminsky, S. E., & Behrend, T. S. (2014) · Journal of Business and Psychology · doi:10.1007/s10869-014-9343-z · Cited by 126 · see citing works
What's the right price? pricing tasks for finishing on time
Faridani, S., Hartmann, B., & Ipeirotis, P. G. (2011) · Cited by 126 · see citing works
Social desirability bias and self-reports of motivation
Antin, J., & Shaw, A. (2012) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2208699 · Cited by 125 · see citing works
Swapping bricks for clicks: Crowdsourcing longitudinal data on Amazon Turk
Daly, T. M., & Nataraajan, R. (2015) · Journal of Business Research · doi:10.1016/j.jbusres.2015.05.001 · Cited by 119 · see citing works
Participants at Your Fingertips
Johnson, D. R., & Borden, L. A. (2012) · Teaching of Psychology · doi:10.1177/0098628312456615 · Cited by 119 · see citing works
Feitosa, J., Joseph, D. L., & Newman, D. A. (2014) · Personality and Individual Differences · doi:10.1016/j.paid.2014.11.017 · Cited by 118 · see citing works
Using Nonnaive Participants Can Reduce Effect SizesPreprint
Chandler, J., Paolacci, G., Péer, E., Mueller, P. A., & Ratliff, K. A. (2015) · RePEc: Research Papers in Economics · Cited by 118 · see citing works
Mechanical Turk upends social sciences
Bohannon, J. (2016) · Science · doi:10.1126/science.352.6291.1263 · Cited by 116 · see citing works
Roulin, N. (2015) · Industrial and Organizational Psychology · doi:10.1017/iop.2015.24 · Cited by 115 · see citing works
Rosette, A. S., Mueller, J., & Lebel, R. D. (2015) · The Leadership Quarterly · doi:10.1016/j.leaqua.2015.02.001 · Cited by 112 · see citing works
Using Mechanical Turk to Study Family Processes and Youth Mental Health: A Test of Feasibility
Schleider, J. L., & Weisz, J. R. (2015) · Journal of Child and Family Studies · doi:10.1007/s10826-015-0126-6 · Cited by 112 · see citing works
Caution! MTurk Workers Ahead—Fines Doubled
Harms, P. D., & DeSimone, J. A. (2015) · Industrial and Organizational Psychology · doi:10.1017/iop.2015.23 · Cited by 109 · see citing works
Self-citationPower, Moral Clarity, and Punishment in the Workplace
Wiltermuth, S. S., & Flynn, F. J. (2012) · Academy of Management Journal · doi:10.5465/amj.2010.0960 · Cited by 107 · see citing works
Understanding internal, external, and relational attributions for abusive supervision
Burton, J. P., Taylor, S. G., & Barber, L. K. (2014) · Journal of Organizational Behavior · doi:10.1002/job.1939 · Cited by 106 · see citing works
Web workers unite! addressing challenges of online laborers
Bederson, B. B., & Quinn, A. J. (2011) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/1979742.1979606 · Cited by 106 · see citing works
Marchiondo, L. A., Myers, C. G., & Kopelman, S. (2015) · The Leadership Quarterly · doi:10.1016/j.leaqua.2015.06.006 · Cited by 105 · see citing works
Cheapened altruism: Discounting personally affected prosocial actors
Lin, F., & Small, D. A. (2011) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2011.11.006 · Cited by 101 · see citing works
Spisak, B. R., Grabo, A. E., Arvey, R. D., & van Vugt, M. (2014) · The Leadership Quarterly · doi:10.1016/j.leaqua.2014.06.001 · Cited by 101 · see citing works
Can we turn shirkers into workers?
Berinsky, A. J., Margolis, M. F., & Sances, M. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.09.010 · Cited by 96 · see citing works
Time pressure reverses risk preferences
Saqib, N. U., & Chan, E. Y. (2015) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2015.06.004 · Cited by 77 · see citing works
Woo, S. E., Keith, M. G., & Thornton, M. A. (2015) · Industrial and Organizational Psychology · doi:10.1017/iop.2015.21 · Cited by 75 · see citing works
Self-citationCho, E., & Allen, T. D. (2011) · Journal of Vocational Behavior · doi:10.1016/j.jvb.2011.12.002 · Cited by 73 · see citing works
Random and Systematic Error Effects of Insomnia on Survey Behavior
Barber, L. K., Barnes, C. M., & Carlson, K. D. (2013) · Organizational Research Methods · doi:10.1177/1094428113493120 · Cited by 73 · see citing works
Risks and Rewards of Crowdsourcing Marketplaces
Chandler, J., Paolacci, G., & Mueller, P. A. (2013) · Springer eBooks · doi:10.1007/978-1-4614-8806-4_30 · Cited by 70 · see citing works
Hardy, B., & Ford, L. R. (2014) · Organizational Research Methods · doi:10.1177/1094428113520185 · Cited by 69 · see citing works
Inesi, M. E., & Cable, D. M. (2014) · Personnel Psychology · doi:10.1111/peps.12083 · Cited by 69 · see citing works
Selectively Recruiting Participants from Amazon Mechanical Turk Using QualtricsPreprint
Péer, E., Paolacci, G., Chandler, J., & Mueller, P. A. (2012) · SSRN Electronic Journal · doi:10.2139/ssrn.2100631 · Cited by 67 · see citing works
Nostalgia buffers the negative impact of low procedural justice on cooperation
van Dijke, M., Wildschut, T., Leunissen, J. M., & Sedikides, C. (2015) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2014.11.005 · Cited by 65 · see citing works
An online daily diary study of alcohol use using A mazon's M echanical T urk
Boynton, M. H., & Richman, L. S. (2014) · Drug and Alcohol Review · doi:10.1111/dar.12163 · Cited by 63 · see citing works
Hypocrisy by association: When organizational membership increases condemnation for wrongdoing
Effron, D. A., Lucas, B. J., & O’Connor, K. (2015) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2015.05.001 · Cited by 60 · see citing works
Reconnection Choices: Selecting the Most Valuable (vs. Most Preferred) Dormant Ties
Walter, J., Levin, D. Z., & Murnighan, J. K. (2015) · Organization Science · doi:10.1287/orsc.2015.0996 · Cited by 60 · see citing works
What do people desire in their leaders? The role of leadership level on trait desirability
Nichols, A., & Cottrell, C. A. (2014) · The Leadership Quarterly · doi:10.1016/j.leaqua.2014.04.001 · Cited by 60 · see citing works
Parker, S. L., Jimmieson, N. L., Walsh, A. J., & Loakes, J. (2014) · Journal of Business and Psychology · doi:10.1007/s10869-014-9383-4 · Cited by 55 · see citing works
Lorenzo‐Seva, U., & Ferrando, P. J. (2010) · Behavior Research Methods · doi:10.3758/s13428-010-0043-y · Cited by 51 · see citing works
Pseudo-dyadic “interaction” on Amazon’s Mechanical Turk
Summerville, A. L., & Chartier, C. R. (2012) · Behavior Research Methods · doi:10.3758/s13428-012-0250-9 · Cited by 50 · see citing works
Conducting Psychology Student Research Via the Mechanical Turk Crowdsourcing Service
Bates, J. A., & Lanza, B. A. (2013) · North American journal of psychology · Cited by 47 · see citing works
Siegel, J. T., Navarro, M. A., & Thomson, A. L. (2015) · Social Science & Medicine · doi:10.1016/j.socscimed.2015.08.020 · Cited by 46 · see citing works
Fifty Days an MTurk Worker: The Social and Motivational Context for Amazon Mechanical Turk Workers
Schmidt, G. B. (2015) · Industrial and Organizational Psychology · doi:10.1017/iop.2015.20 · Cited by 42 · see citing works
Wide of the mark: Evidence on the underlying causes of overprecision in judgment
Moore, D. A., Carter, A. B., & Yang, H. H. J. (2015) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2015.09.003 · Cited by 41 · see citing works
Collecting Narrative Data on Amazon's Mechanical Turk
Grysman, A. (2015) · Applied Cognitive Psychology · doi:10.1002/acp.3140 · Cited by 40 · see citing works
Proactive Responding to Anticipated Discrimination Based on Chronic Illness: Double-Edged Sword?
McGonagle, A. K., & Hamblin, L. E. (2013) · Journal of Business and Psychology · doi:10.1007/s10869-013-9324-7 · Cited by 39 · see citing works
Wiltermuth, S. S., Bennett, V. M., & Pierce, L. (2013) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2013.10.001 · Cited by 39 · see citing works
Measuring Prestige of Journals in Industrial-Organizational Psychology
Zickar, M. J., & Highhouse, S. (2001) · PsycEXTRA Dataset · doi:10.1037/e576972011-002 · Cited by 38 · see citing works
Crowdsourcing content analysis for managerial research
Conley, C. A., & Tosti‐Kharas, J. (2014) · Management Decision · doi:10.1108/md-03-2012-0156 · Cited by 37 · see citing works
Savani, K., & King, D. (2015) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2015.05.002 · Cited by 27 · see citing works
Two-stage decisions increase preference for hedonic options
Bhargave, R. P., Chakravarti, A., & Guha, A. (2015) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2015.06.003 · Cited by 27 · see citing works
Is Mechanical Turk the Answer to Our Sampling Woes?
Keith, M. G., & Harms, P. D. (2016) · Industrial and Organizational Psychology · doi:10.1017/iop.2015.130 · Cited by 27 · see citing works
Self-citationDifference in Response Effort Across Sample Types: Perception or Reality?
Ran, S., Liu, M., Marchiondo, L. A., & Huang, J. L. W. (2015) · Industrial and Organizational Psychology · doi:10.1017/iop.2015.26 · Cited by 26 · see citing works
The New Fruit Fly for Applied Psychological Research
Highhouse, S., & Zhang, D. C. (2015) · Industrial and Organizational Psychology · doi:10.1017/iop.2015.22 · Cited by 24 · see citing works
Proleader and antitrailer information distortion and their effects on choice and postchoice memory
DeKay, M. L., Miller, S. A., Schley, D. R., & Erford, B. M. (2014) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2014.07.003 · Cited by 20 · see citing works
Karim, M. N., & Behrend, T. S. (2013) · Journal of Business and Psychology · doi:10.1007/s10869-013-9309-6 · Cited by 20 · see citing works
Participant Motivation: A Critical Consideration
McGonagle, A. K. (2015) · Industrial and Organizational Psychology · doi:10.1017/iop.2015.27 · Cited by 19 · see citing works
Age-Related Slowing in Online Samples
Bui, D. C., Myerson, J., & Hale, S. (2015) · The Psychological Record · doi:10.1007/s40732-015-0135-2 · Cited by 18 · see citing works
Fine, S., & Pirak, M. (2015) · Journal of Business and Psychology · doi:10.1007/s10869-015-9398-5 · Cited by 17 · see citing works
Proudfoot, D., Kay, A. C., & Mann, H. (2015) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2015.06.008 · Cited by 15 · see citing works
Is there a place for sympathy in negotiation? Finding strength in weakness
Shirako, A., Kilduff, G. J., & Kray, L. J. (2015) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2015.09.004 · Cited by 14 · see citing works
The online laboratory: conducting experiments in a real labor market
Horton, J. J., Rand, D. G., & Zeckhauser, R. (2011) · Experimental Economics · doi:10.1007/s10683-011-9273-9 · Cited by 9 · see citing works
Costa‐jussà, M. R., Grivolla, J., Mellebeek, B., et al. (2014) · Information Sciences · doi:10.1016/j.ins.2014.01.043 · Cited by 6 · see citing works
A preliminary development and validation of a measure of safety performance
Yuan, Z. (2014) · Purdue e-Pubs (Purdue University System) · doi:10.7912/c2/1075 · Cited by 1 · see citing works
The Impact of Social Class in the Hiring Process
Young, N. C. J., & Reilly, G. (2016) · Academy of Management Proceedings · doi:10.5465/ambpp.2016.234 · Cited by 1 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Data quality of platforms and panels for online behavioral research
Peer, E., Rothschild, D., Gordon, A., Evernden, Z., & Damer, E. (2021) · Behavior Research Methods · doi:10.3758/s13428-021-01694-3 · Cited by 1,718 · see citing works
Cooksey Stowers, K., Jiang, Q., Atoloye, A., Lucan, S., & Gans, K. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17197143 · Cited by 117 · see citing works
Mindfulness buffers the impact of COVID-19 outbreak information on sleep duration.
Zheng, M. X., Yao, J., & Narayanan, J. (2020) · doi:10.31234/osf.io/wuh94 · Cited by 77 · see citing works
Wörtler, B., Van Yperen, N. W., & Barelds, D. P. H. (2020) · Motivation and Emotion · doi:10.1007/s11031-019-09775-9 · Cited by 53 · see citing works
Rhythm perception, speaking rate entrainment, and conversational quality: a mediated model
Wynn, C. J., Barrett, T. S., & Borrie, S. A. (2022) · Journal of Speech, Language, and Hearing Research · doi:10.1044/2022_jslhr-21-00293 · Cited by 31 · see citing works
Testing the testing effect on prolific: when retrieval practice fails to boost learning
Sigayret, K., Parmentier, J.-F., & Silvestre, F. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1727423
- Authors
- Keith, M. G., Tay, L., & Harms, P. D.
- Affiliations
- Purdue University West Lafayette (US)
- University of Alabama (US)
- Corresponding author
- Keith, Melissa G.
- Venue
- Frontiers in Psychology
- Publisher
- Frontiers Media SA
- Published
- 2017-08-08
- Volume
- 8
- Pages
- 1359-1359
- ISSN
- 1664-1078 (ISSN-L)
- PMC
- PMC5550837
- OpenAlex
- W2742643783
- Cited by
- 319
- References
- 140
- Citations by year
- 2026: 20 · 2025: 27 · 2024: 39 · 2023: 44 · 2022: 51 · 2021: 60 · 2020: 41 · 2019: 25 · 2018: 11 · 2017: 1
- FWCI
- 4.86
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Frontiers in Psychology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- Amazon Mechanical Turk, crowdsourcing, organizational analysis, recruitment system, selection system, work management system, generalizability, data quality, attention checks, payments, data transparency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
