Amazon mechanical Turk in organizational psychology: an evaluation and practical recommendations

Common method biases in behavioral research: A critical review of the literature and recommended remedies

Podsakoff, P. M., MacKenzie, S. B., Lee, J.-Y., & Podsakoff, N. P. (2003) · Journal of Applied Psychology · doi:10.1037/0021-9010.88.5.879 · Cited by 82,194 · see citing works

Sources of method bias in social science research and recommendations on how to control it

Podsakoff, P. M., MacKenzie, S. B., & Podsakoff, N. P. (2012) · Annual Review of Psychology · doi:10.1146/annurev-psych-120710-100452 · Cited by 15,744 · see citing works

Untitled work

(2002) · Cited by 13,211 · see citing works

The weirdest people in the world?

Henrich, J., Heine, S. J., & Norenzayan, A. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0999152x · Cited by 12,641 · see citing works

Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?

Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996 · see citing works

WEIRD languages have misled us, too

Majid, A., & Levinson, S. C. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x1000018x · Cited by 4,169 · see citing works

Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk

Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008 · see citing works

Identifying careless responses in survey data.

Meade, A. W., & Craig, S. B. (2012) · Psychological Methods · doi:10.1037/a0028085 · Cited by 3,865 · see citing works

Running experiments on Amazon Mechanical Turk

Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767 · see citing works

Instructional manipulation checks: Detecting satisficing to increase statistical power

Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2009.03.009 · Cited by 3,493 · see citing works

Method Variance in Organizational Research: Truth or Urban Legend?

Spector, P. E. (2006) · Organizational Research Methods · doi:10.1177/1094428105284955 · Cited by 3,445 · see citing works

Conducting behavioral research on Amazon’s Mechanical Turk

Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172 · see citing works

Inside the Turk: Understanding Mechanical Turk as a Participant Pool

Paolacci, G., & Chandler, J. (2014) · Current Directions in Psychological Science · doi:10.1177/0963721414531598 · Cited by 2,136 · see citing works

Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants

Hauser, D. J., & Schwarz, N. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0578-z · Cited by 1,739 · see citing works

Reputation as a sufficient condition for data quality on Amazon Mechanical Turk

Peer, E., Vosgerau, J., & Acquisti, A. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0434-y · Cited by 1,618 · see citing works

Evaluating amazon’s mechanical turk as a tool for experimental behavioral research

Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0057410 · Cited by 1,617 · see citing works

The Online Laboratory: Conducting Experiments in a Real Labor Market∗Preprint

Zeckhauser, R. J. (2016) · Cited by 1,454 · see citing works

Detecting and Deterring Insufficient Effort Responding to Surveys

Huang, J. L. W., Curran, P. G., Keeney, J., Poposki, E. M., & DeShon, R. P. (2011) · Journal of Business and Psychology · doi:10.1007/s10869-011-9231-8 · Cited by 1,315 · see citing works

Using Mechanical Turk to Study Clinical Populations

Shapiro, D. N., Chandler, J., & Mueller, P. A. (2013) · Clinical Psychological Science · doi:10.1177/2167702612469015 · Cited by 1,139 · see citing works

The viability of crowdsourcing for survey research

Behrend, T. S., Sharek, D. J., Meade, A. W., & Wiebe, E. (2011) · Behavior Research Methods · doi:10.3758/s13428-011-0081-0 · Cited by 1,049 · see citing works

Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers

Chandler, J., Mueller, P., & Paolacci, G. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0365-7 · Cited by 950 · see citing works

An inconvenient truth: Arbitrary distinctions between organizational, mechanical Turk, and other convenience samples

Landers, R. N., & Behrend, T. S. (2015) · Industrial and Organizational Psychology · doi:10.1017/iop.2015.13 · Cited by 714 · see citing works

Best practice recommendations for data screening

DeSimone, J. A., Harms, P. D., & DeSimone, A. J. (2014) · Journal of Organizational Behavior · doi:10.1002/job.1962 · Cited by 690 · see citing works

Factors Defined by Negatively Keyed Items: The Result of Careless Respondents?

Schmitt, N., & Stuits, D. M. (1985) · Applied Psychological Measurement · doi:10.1177/014662168500900405 · Cited by 536 · see citing works

Seriousness checks are useful to improve data validity in online research

Aust, F., Diedenhofen, B., Ullrich, S., & Musch, J. (2012) · Behavior Research Methods · doi:10.3758/s13428-012-0265-2 · Cited by 528 · see citing works

Demographics of Mechanical Turk

Ipeirotis, P. G. (2010) · Faculty Digital Archive (New York University Florence) · Cited by 494 · see citing works

A validation of Amazon Mechanical Turk for the collection of acceptability judgments in linguistic theory

Sprouse, J. (2010) · Behavior Research Methods · doi:10.3758/s13428-010-0039-7 · Cited by 492 · see citing works

Insufficient effort responding: Examining an insidious confound in survey data.

Huang, J. L. W., Liu, M., & Bowling, N. A. (2014) · Journal of Applied Psychology · doi:10.1037/a0038510 · Cited by 421 · see citing works

A reliability analysis of Mechanical Turk data

Rouse, S. V. (2014) · Computers in Human Behavior · doi:10.1016/j.chb.2014.11.004 · Cited by 414 · see citing works

Careless Responding to Reverse-Worded Items: Implications for Confirmatory Factor Analysis

Woods, C. M. (2006) · Journal of Psychopathology and Behavioral Assessment · doi:10.1007/s10862-005-9004-7 · Cited by 402 · see citing works

Implications of direct and indirect range restriction for meta-analysis methods and findings.

Hunter, J. E., Schmidt, F. L., & Le, H. (2006) · Journal of Applied Psychology · doi:10.1037/0021-9010.91.3.594 · Cited by 315 · see citing works

Detecting Insufficient Effort Responding with an Infrequency Scale: Evaluating Validity and Participant Reactions

Huang, J. L. W., Bowling, N. A., Liu, M., & Li, Y. (2014) · Journal of Business and Psychology · doi:10.1007/s10869-014-9357-6 · Cited by 315 · see citing works

The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers

Stewart, N., Ungemach, C., Harris, A. J. L., Bartels, D. M., Newell, B. R., Paolacci, G., & Chandler, J. (2015) · Judgment and Decision Making · doi:10.1017/s1930297500005611 · Cited by 304 · see citing works

Creating Speech and Language Data With Amazon’s Mechanical Turk

Callison-Burch, C., & Dredze, M. H. (2010) · Cited by 298 · see citing works

Evidence for response bias as a source of error variance in applied assessment.

McGrath, R. E., Mitchell, M., Kim, B. H., & Hough, L. (2010) · Psychological Bulletin · doi:10.1037/a0019216 · Cited by 282 · see citing works

The Industrial-Organizational Psychologist

(2009) · PsycEXTRA Dataset · doi:10.1037/e578882011-001 · Cited by 277 · see citing works

A mazon Mechanical Turk and the commodification of labour

Bergvall‐Kåreborn, B., & Howcroft, D. (2014) · New Technology Work and Employment · doi:10.1111/ntwe.12038 · Cited by 258 · see citing works

A convenient solution: Using MTurk to sample from hard-to-reach populations

Smith, N. A., Sabat, I. E., Martinez, L. R., Weaver, K., & Xu, S. (2015) · Industrial and Organizational Psychology · doi:10.1017/iop.2015.29 · Cited by 190 · see citing works

Using Nonnaive Participants Can Reduce Effect Sizes

Chandler, J., Paolacci, G., Peer, E., Mueller, P., & Ratliff, K. A. (2015) · Psychological Science · doi:10.1177/0956797615585115 · Cited by 185 · see citing works

How Careless Responding and Acquiescence Response Bias Can Influence Construct Dimensionality

Kam, C. C. S., & Meyer, J. P. (2015) · Organizational Research Methods · doi:10.1177/1094428115571894 · Cited by 180 · see citing works

eLancing: A review and research agenda for bridging the science–practice gap

Aguinis, H., & Lawal, S. O. (2012) · Human Resource Management Review · doi:10.1016/j.hrmr.2012.06.003 · Cited by 161 · see citing works

Inattentive responding in MTurk and other online samples

Fleischer, A., Mead, A. D., & Huang, J. (2015) · Industrial and Organizational Psychology · doi:10.1017/iop.2015.25 · Cited by 151 · see citing works

Where Have All the “Workers” Gone? A Critical Analysis of the Unrepresentativeness of Our Samples Relative to the Labor Market in the Industrial–Organizational Psychology Literature

Bergman, M., & Jean, V. A. (2015) · Industrial and Organizational Psychology · doi:10.1017/iop.2015.70 · Cited by 138 · see citing works

Social desirability bias and self-reports of motivation

Antin, J., & Shaw, A. (2012) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2208699 · Cited by 125 · see citing works

Using Nonnaive Participants Can Reduce Effect SizesPreprint

Chandler, J., Paolacci, G., Péer, E., Mueller, P. A., & Ratliff, K. A. (2015) · RePEc: Research Papers in Economics · Cited by 118 · see citing works

Don't Throw the Baby Out With the Bathwater: Comparing Data Quality of Crowdsourcing, Online Panels, and Student Samples

Roulin, N. (2015) · Industrial and Organizational Psychology · doi:10.1017/iop.2015.24 · Cited by 115 · see citing works

Caution! MTurk Workers Ahead—Fines Doubled

Harms, P. D., & DeSimone, J. A. (2015) · Industrial and Organizational Psychology · doi:10.1017/iop.2015.23 · Cited by 109 · see citing works

Conducting field experiments using eLancing's natural environment

Aguinis, H., & Lawal, S. O. (2012) · Journal of Business Venturing · doi:10.1016/j.jbusvent.2012.01.002 · Cited by 106 · see citing works

Amazon Mechanical Turk for Industrial and Organizational Psychology: Advantages, Challenges, and Practical Recommendations

Woo, S. E., Keith, M. G., & Thornton, M. A. (2015) · Industrial and Organizational Psychology · doi:10.1017/iop.2015.21 · Cited by 75 · see citing works

Internet Panels, Professional Respondents, and Data Quality

Matthijsse, S. M., de Leeuw, E. D., & Hox, J. J. (2015) · Methodology · doi:10.1027/1614-2241/a000094 · Cited by 62 · see citing works

Insufficient Effort Survey Responding: An Under‐Appreciated Problem in Work and Organisational Health Psychology Research

McGonagle, A. K., Huang, J. L. W., & Walsh, B. M. (2015) · Applied Psychology · doi:10.1111/apps.12058 · Cited by 61 · see citing works

Research strategies in industrial and organizational psychology: Nonexperimental, quasi-experimental, and randomized experimental research in special purpose and nonspecial purpose settings.

Stone‐Romero, E. F. (2010) · American Psychological Association eBooks · doi:10.1037/12169-002 · Cited by 61 · see citing works

Fifty Days an MTurk Worker: The Social and Motivational Context for Amazon Mechanical Turk Workers

Schmidt, G. B. (2015) · Industrial and Organizational Psychology · doi:10.1017/iop.2015.20 · Cited by 42 · see citing works

Building and developing the organization

Zedeck, S. (2011) · American Psychological Association eBooks · Cited by 28 · see citing works

Difference in Response Effort Across Sample Types: Perception or Reality?

Ran, S., Liu, M., Marchiondo, L. A., & Huang, J. L. W. (2015) · Industrial and Organizational Psychology · doi:10.1017/iop.2015.26 · Cited by 26 · see citing works

The New Fruit Fly for Applied Psychological Research

Highhouse, S., & Zhang, D. C. (2015) · Industrial and Organizational Psychology · doi:10.1017/iop.2015.22 · Cited by 24 · see citing works

Stop Apologizing for Your Samples, Start Embracing Them

Zhu, X., Barnes-Farrell, J. L., & Dalal, D. K. (2015) · Industrial and Organizational Psychology · doi:10.1017/iop.2015.30 · Cited by 20 · see citing works

Participant Motivation: A Critical Consideration

McGonagle, A. K. (2015) · Industrial and Organizational Psychology · doi:10.1017/iop.2015.27 · Cited by 19 · see citing works

Data quality of platforms and panels for online behavioral research

Peer, E., Rothschild, D., Gordon, A., Evernden, Z., & Damer, E. (2021) · Behavior Research Methods · doi:10.3758/s13428-021-01694-3 · Cited by 1,718 · see citing works

Systems perspective of Amazon mechanical Turk for organizational research: review and recommendations

Keith, M. G., Tay, L., & Harms, P. D. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.01359 · Cited by 319 · see citing works

Examining the inverted U-shaped relationship between workload and innovative work behavior: the role of work engagement and mindfulness.

Montani, F., Vandenberghe, C., Khedhaouria, A., & Courcy, F. (2020) · Human Relations · doi:10.1177/0018726718819055 · Cited by 252 · see citing works

The quality of data collected online: An investigation of careless responding in a crowdsourced sample

Brühlmann, F., Petralito, S., Aeschbach, L. F., & Opwis, K. (2020) · Methods in Psychology · doi:10.1016/j.metip.2020.100022 · Cited by 168 · see citing works

Coping during the COVID-19 pandemic: Relations with mental health and quality of life.

Shamblaw, A. L., Rumas, R. L., & Best, M. W. (2021) · Canadian Psychology / Psychologie canadienne · doi:10.1037/cap0000263 · Cited by 156 · see citing works

Rise in Use of Digital Mental Health Tools and Technologies in the United States During the COVID-19 Pandemic: Survey Study

Sorkin, D. H., Janio, E. A., Eikey, E. V., Schneider, M., Davis, K., Schueller, S. M., Stadnick, N. A., Zheng, K., Neary, M., Safani, D., et al. (2021) · Journal of Medical Internet Research · doi:10.2196/26994 · Cited by 108 · see citing works

Assessing Perceived Barriers to Meditation: the Determinants of Meditation Practice Inventory-Revised (DMPI-R)

Hunt, C. A., Hoffman, M. A., Mohr, J. J., & Williams, A.-L. (2020) · Mindfulness · doi:10.1007/s12671-020-01308-7 · Cited by 52 · see citing works

A Systems-Based Approach to Fostering Robust Science in Industrial-Organizational Psychology

Grand, J. A., Rogelberg, S. G., Allen, T. D., Landis, R. S., Reynolds, D. H., Scott, J. C., Tonidandel, S., & Truxillo, D. M. (2018) · Industrial and Organizational Psychology · doi:10.1017/iop.2017.55 · Cited by 52 · see citing works

Do individual differences in need strength moderate the relations between basic psychological need satisfaction and organizational citizenship behavior?

Wörtler, B., Van Yperen, N. W., & Barelds, D. P. H. (2020) · Motivation and Emotion · doi:10.1007/s11031-019-09775-9 · Cited by 53 · see citing works

Cognitive rigidity, habitual tendencies, and obsessive-compulsive symptoms: Individual differences and compensatory interactions

Ramakrishnan, S., Robbins, T. W., & Zmigrod, L. (2022) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2022.865896 · Cited by 29 · see citing works

Testing the testing effect on prolific: when retrieval practice fails to boost learning

Sigayret, K., Parmentier, J.-F., & Silvestre, F. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1727423

Can there be a trade off between internal and external validity

Vaeth, M. (2019)

Designing Experimental Research

Miksza, P., Shaw, J. T., Richerme, L. K., et al. (2023) · Music Education Research · doi:10.1093/oso/9780197639757.003.0014

Research Validity

Rosenstein, L. D. (2019) · doi:10.1002/9781119563600.ch2

A New Way to Think About Internal and External Validity

Trafimow, D. (2022) · Perspectives on Psychological Science · doi:10.1177/17456916221136117 · Cited by 30 · see citing works

Assessing the Validity of Clinical Trials

Akobeng, A. K. (2008) · Journal of Pediatric Gastroenterology and Nutrition · doi:10.1097/mpg.0b013e31816c749f · Cited by 81 · see citing works