The DOI matches its Crossref record; 87 of the 87 references it cites were found in a registry or an index.
MTurk Research: Review and Recommendations
Open accessAbstract
The use of Amazon’s Mechanical Turk (MTurk) in management research has increased over 2,117% in recent years, from 6 papers in 2012 to 133 in 2019. Among scholars, though, there is a mixture of excitement about the practical and logistical benefits of using MTurk and skepticism about the validity of the data. Given that the practice is rapidly increasing but scholarly opinions diverge, the Journal of Management commissioned this review and consideration of best practices. We hope the recommendations provided here will serve as a catalyst for more robust, reproducible, and trustworthy MTurk-based research in management and related fields.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767
Conducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172
Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242
Synthesizing information systems knowledge: A typology of literature reviews
Paré, G., Trudel, M.-C., Jaana, M., & Kitsiou, S. (2015) · Information & Management · doi:10.1016/j.im.2014.08.008 · Cited by 1,987
A Critical Review and Best‐Practice Recommendations for Control Variable Usage
Bernerth, J. B., & Aguinis, H. (2016) · Personnel Psychology · doi:10.1111/peps.12103 · Cited by 1,731 · cited 1×
“Therefore, when Big Five traits are expected to influence substantive results, they can be used as statistical controls so that results and inferences are attributable to the hypothesized predictors and not to variability in personality traits between samples (Bernerth & Aguinis, 2016).”
BackgroundSelf-citationCasler, K., Bickel, L., & Hackett, E. (2013) · Computers in Human Behavior · doi:10.1016/j.chb.2013.05.009 · Cited by 1,662
Reputation as a sufficient condition for data quality on Amazon Mechanical Turk
Peer, E., Vosgerau, J., & Acquisti, A. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0434-y · Cited by 1,618 · cited 1×
“…familiarity with common manipulations, attention check techniques, and experimental tasks and questions; Lovett, Bajaba, Lovett, & Simmering, 2018; Peer, Vosgerau, & Acquisti, 2014) or, alternatively, employ screening questions to gauge MTurker familiarity with research subject, stimuli, and if…”
BackgroundEvaluating amazon’s mechanical turk as a tool for experimental behavioral research
Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0057410 · Cited by 1,617
CAPTCHA: Using Hard AI Problems for Security
von Ahn, L., Blum, M., Hopper, N. J., & Langford, J. (2003) · Lecture Notes in Computer Science · doi:10.1007/3-540-39200-9_18 · Cited by 1,447
The viability of crowdsourcing for survey research
Behrend, T. S., Sharek, D. J., Meade, A. W., & Wiebe, E. N. (2011) · Behavior Research Methods · doi:10.3758/s13428-011-0081-0 · Cited by 1,018
Berinsky, A. J., Margolis, M. F., & Sances, M. W. (2014) · American Journal of Political Science · doi:10.1111/ajps.12081 · Cited by 951 · cited 3×
“MTurkers who score higher on self-reports of response effort or fail to comply with directed questions are more likely to have engaged in careless responding or IER (Berinsky, Margolis, & Sances, 2014; Maniaci & Rogge, 2014).”
BackgroundChandler, J., Mueller, P., & Paolacci, G. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0365-7 · Cited by 950
Kees, J., Berry, C., Burton, S., & Sheehan, K. (2017) · Journal of Advertising · doi:10.1080/00913367.2016.1269304 · Cited by 902
Caring about carelessness: Participant inattention and its effects on research
Maniaci, M. R., & Rogge, R. D. (2014) · Journal of Research in Personality · doi:10.1016/j.jrp.2013.09.008 · Cited by 852
Crowdsourcing graphical perception: using mechanical turk to assess visualization design
Heer, J., & Bostock, M. (2010) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753357 · Cited by 816
Transparency and replicability in qualitative research: the case of interviews with elite informants
Aguinis, H., & Solarino, A. M. (2019) · Strategic Management Journal · doi:10.1002/smj.3015 · Cited by 791
Self-citationAn Evaluation of Amazon’s Mechanical Turk, Its Rapid Rise, and Its Effective Use
Buhrmester, M. D., Talaifar, S., & Gosling, S. D. (2018) · Perspectives on Psychological Science · doi:10.1177/1745691617706516 · Cited by 767
Landers, R. N., & Behrend, T. S. (2015) · Industrial and Organizational Psychology · doi:10.1017/iop.2015.13 · Cited by 714
The demographic and political composition of Mechanical Turk samples
Levay, K. E., Freese, J., & Druckman, J. N. (2016) · Sage Open · doi:10.1177/2158244016636433 · Cited by 676
Online panels in social science research: Expanding sampling methods beyond Mechanical Turk
Chandler, J., Rosenzweig, C., Moss, A. J., Robinson, J., & Litman, L. (2019) · Behavior Research Methods · doi:10.3758/s13428-019-01273-7 · Cited by 668
Validity and Mechanical Turk: An assessment of exclusion methods and interactive experiments
Thomas, K. A., & Clifford, S. (2017) · Computers in Human Behavior · doi:10.1016/j.chb.2017.08.038 · Cited by 625
Ascertaining the validity of individual protocols from Web-based personality inventories
Johnson, J. A. (2005) · Journal of Research in Personality · doi:10.1016/j.jrp.2004.09.009 · Cited by 617
Reynolds, C. R. (2008) · Encyclopedia of Special Education · doi:10.1002/9780470373699.speced0485 · Cited by 617
The labor economics of paid crowdsourcing
Horton, J. J., & Chilton, L. B. (2010) · Proceedings of the 11th ACM conference on Electronic commerce · doi:10.1145/1807342.1807376 · Cited by 601
Weinberg, J., Freese, J., & McElhattan, D. (2014) · Sociological Science · doi:10.15195/v1.a19 · Cited by 582
Measuring and Bounding Experimenter Demand
de Quidt, J., Haushofer, J., & Roth, C. (2018) · American Economic Review · doi:10.1257/aer.20171330 · Cited by 568
Demand Effects in Survey Experiments: An Empirical Assessment
MUMMOLO, J., & PETERSON, E. (2019) · American Political Science Review · doi:10.1017/s0003055418000837 · Cited by 548
A Tale of Two Sample Sources: Do Results from Online Panel Data and Conventional Data Converge?
Walter, S. L., Seibert, S. E., Goering, D., & O’Boyle, E. H. (2019) · Journal of Business and Psychology · doi:10.1007/s10869-018-9552-y · Cited by 544
The shape of and solutions to the MTurk quality crisis
Kennedy, R., Clifford, S., Burleigh, T., Waggoner, P. D., Jewell, R., & Winter, N. J. G. (2020) · Political Science Research and Methods · doi:10.1017/psrm.2020.6 · Cited by 506
Sprouse, J. (2011) · Behavior Research Methods · doi:10.3758/s13428-010-0039-7 · Cited by 487
Amazon mechanical Turk in organizational psychology: an evaluation and practical recommendations
Cheung, J. H., Burns, D. K., Sinclair, R. R., & Sliter, M. (2017) · Journal of Business and Psychology · doi:10.1007/s10869-016-9458-5 · Cited by 474
What You See Is What You Get? Enhancing Methodological Transparency in Management Research
Aguinis, H., Ramani, R. S., & Alabduljader, N. (2018) · Academy of Management Annals · doi:10.5465/annals.2016.0011 · Cited by 463
Self-citationZhou, H., & Fishbach, A. (2016) · Journal of Personality and Social Psychology · doi:10.1037/pspa0000056 · Cited by 459
A reliability analysis of Mechanical Turk data
Rouse, S. V. (2015) · Computers in Human Behavior · doi:10.1016/j.chb.2014.11.004 · Cited by 414
External validity: From do-calculus to transportability across populations
Pearl, J., & Bareinboim, E. (2013) · Cited by 398
External Validity: From Do-Calculus to Transportability Across Populations
Pearl, J., & Bareinboim, E. (2014) · Statistical Science · doi:10.1214/14-sts486 · Cited by 345
Conducting interactive experiments online
Arechar, A. A., Gächter, S., & Molleman, L. (2018) · Experimental Economics · doi:10.1007/s10683-017-9527-2 · Cited by 340
The Use of Online Panel Data in Management Research: A Review and Recommendations
Porter, C. O. L. H., Outlaw, R., Gale, J. P., & Cho, T. S. (2019) · Journal of Management · doi:10.1177/0149206318811569 · Cited by 338
MTurk Character Misrepresentation: Assessment and Solutions
Sharpe Wessling, K., Huber, J., & Netzer, O. (2017) · Journal of Consumer Research · doi:10.1093/jcr/ucx053 · Cited by 330 · cited 2×
“…pay everyone who participates, eliminate those who do not match desired criteria, and invite those who meet the qualifications/pass the screener to participate in the focal study (Chandler, Mueller, & Paolacci, 2014; Hydock, 2018; Siegel, Navarro, & Thomson, 2015; Wessling, Huber, & Netzer, 2017).”
BackgroundKeith, M. G., Tay, L., & Harms, P. D. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.01359 · Cited by 319
Huang, J. L., Bowling, N. A., Liu, M., & Li, Y. (2015) · Journal of Business and Psychology · doi:10.1007/s10869-014-9357-6 · Cited by 314
(2020) · Conducting Health Research: Principles, Process, and Methods · doi:10.4135/9781071909621.n12 · Cited by 310
The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers
Stewart, N., Ungemach, C., Harris, A. J. L., Bartels, D. M., Newell, B. R., Paolacci, G., & Chandler, J. (2015) · Judgment and Decision Making · doi:10.1017/s1930297500005611 · Cited by 304 · cited 1×
“This helps identify MTurkers who attempt self-misrepre-sentation to qualify for a particular study (Stewart et al., 2015).”
BackgroundDeng, X. (., Joshi, K. D., & Galliers, R. D. (2016) · MIS Quarterly · doi:10.25300/misq/2016/40.2.01 · Cited by 303
Creating Speech and Language Data With Amazon’s Mechanical Turk
Callison-Burch, C., & Dredze, M. H. (2010) · Cited by 294
Chandler, J. J., & Paolacci, G. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550617698203 · Cited by 277
Litman, L., Robinson, J., & Rosenzweig, C. (2015) · Behavior Research Methods · doi:10.3758/s13428-014-0483-x · Cited by 275
Amazon Mechanical Turk and the commodification of labour
Bergvall‐Kåreborn, B., & Howcroft, D. (2014) · New Technology, Work and Employment · doi:10.1111/ntwe.12038 · Cited by 256
Methods to detect low quality data and its implication for psychological research
Buchanan, E. M., & Scofield, J. E. (2018) · Behavior Research Methods · doi:10.3758/s13428-018-1035-6 · Cited by 254
Wood, D., Harms, P. D., Lowman, G. H., & DeSimone, J. A. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550617703168 · Cited by 243
Aguinis, H., Ramani, R. S., & Alabduljader, N. (2023) · Organizational Research Methods · doi:10.1177/1094428120943281 · Cited by 222
Self-citationClifford, S., & Jerit, J. (2014) · Journal of Experimental Political Science · doi:10.1017/xps.2014.5 · Cited by 206
Psychological research in the internet age: The quality of web-based data
Ramsey, S. R., Thompson, K. L., McKenzie, M., & Rosenbaum, A. (2016) · Computers in Human Behavior · doi:10.1016/j.chb.2015.12.049 · Cited by 203
Unfairness by Design? The Perceived Fairness of Digital Labor on Crowdworking Platforms
Fieseler, C., Bucher, E., & Hoffmann, C. P. (2019) · Journal of Business Ethics · doi:10.1007/s10551-017-3607-2 · Cited by 199
Work experiences on MTurk: Job satisfaction, turnover, and information sharing
Brawley, A. M., & Pury, C. L. S. (2016) · Computers in Human Behavior · doi:10.1016/j.chb.2015.08.031 · Cited by 188 · cited 3×
“…(e.g., Turker Nation, MTurk Crowd) to gauge MTurkers reactions to the study (if any), check if pertinent information is being shared, and respond promptly to any questions or concerns raised by participants (Barchard & Williams, 2008; Brawley & Pury, 2016; Deng, Joshi, & Galliers, 2016).”
BackgroundUsing Nonnaive Participants Can Reduce Effect Sizes
Chandler, J., Paolacci, G., Peer, E., Mueller, P., & Ratliff, K. A. (2015) · Psychological Science · doi:10.1177/0956797615585115 · Cited by 185
Stritch, J. M., Pedersen, M. J., & Taggart, G. (2017) · International Public Management Journal · doi:10.1080/10967494.2016.1276493 · Cited by 181
Necka, E. A., Cacioppo, S., Norman, G. J., & Cacioppo, J. T. (2016) · PLOS ONE · doi:10.1371/journal.pone.0157732 · Cited by 161
Using crowdsourcing for TREC relevance assessment
Alonso, O., & Mizzaro, S. (2012) · Information Processing & Management · doi:10.1016/j.ipm.2012.01.004 · Cited by 160
Feitosa, J., Joseph, D. L., & Newman, D. A. (2014) · Personality and Individual Differences · doi:10.1016/j.paid.2014.11.017 · Cited by 145
Barchard, K. A., & Williams, J. E. (2008) · Behavior Research Methods · doi:10.3758/brm.40.4.1111 · Cited by 145
Callison-Burch, C., & Association for Computing Machinery-Digital Library (2010) · Cited by 144
Antin, J., & Shaw, A. (2012) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2208699 · Cited by 134
Casey, L. S., Chandler, J., Levine, A. S., Proctor, A., & Strolovitch, D. Z. (2017) · SAGE Open · doi:10.1177/2158244017712774 · Cited by 132
Lovett, M., Bajaba, S. M., Lovett, M., & Simmering, M. J. (2017) · Applied Psychology · doi:10.1111/apps.12124 · Cited by 125
Barends, A. J., & de Vries, R. E. (2019) · Personality and Individual Differences · doi:10.1016/j.paid.2019.02.015 · Cited by 124 · cited 3×
“Sources used to derive recommendations: 1 Antin and Shaw (2012); 2 Arechar, Gächter, and Molleman (2018); 3 Barends and de Vries (2019); 4 Bederson and Quinn (2011); 5 Behrend, Sharek, Meade, and Wiebe (2011); 6 Bergvall-Kåreborn and Howcroft (2014); 7 Brawley and Pury (2016); 8 Buchanan and…”
BackgroundMethodUsing Nonnaive Participants Can Reduce Effect SizesPreprint
Chandler, J., Paolacci, G., Péer, E., Mueller, P. A., & Ratliff, K. A. (2015) · RePEc: Research Papers in Economics · Cited by 118
Gleibs, I. H. (2016) · Behavior Research Methods · doi:10.3758/s13428-016-0789-y · Cited by 114
Hamby, T., & Taylor, W. (2016) · Educational and Psychological Measurement · doi:10.1177/0013164415627349 · Cited by 111
Methods of Detecting Insufficient Effort Responding: Comparisons and Practical Recommendations
Hong, M. R., Steedle, J. T., & Cheng, Y. (2019) · Educational and Psychological Measurement · doi:10.1177/0013164419865316 · Cited by 109
Web workers unite! addressing challenges of online laborers
Bederson, B. B., & Quinn, A. J. (2011) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/1979742.1979606 · Cited by 106
The Shape of and Solutions to the MTurk Quality CrisisPreprint
Kennedy, R., Clifford, S., Burleigh, T. J., Jewell, R., & Waggoner, P. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3272468 · Cited by 99
Can Nonprobability Samples be Used for Social Science Research? A cautionary tale
Zack, E. S., KENNEDY, J. M., & Long, J. S. (2017) · Survey research methods · doi:10.18148/srm/2019.v13i2.7262 · Cited by 99
Actionable recommendations for narrowing the science-practice gap in open science
Aguinis, H., Banks, G. C., Rogelberg, S. G., & Cascio, W. F. (2020) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2020.02.007 · Cited by 90
Self-citationKan, I. P., & Drummey, A. B. (2018) · Computers in Human Behavior · doi:10.1016/j.chb.2018.02.005 · Cited by 68
Assessing and overcoming participant dishonesty in online data collection
Hydock, C. (2017) · Behavior Research Methods · doi:10.3758/s13428-017-0984-5 · Cited by 59
Pseudo-dyadic “interaction” on Amazon’s Mechanical Turk
Summerville, A. L., & Chartier, C. R. (2012) · Behavior Research Methods · doi:10.3758/s13428-012-0250-9 · Cited by 50
The Use of Online Panel Data in Management Research: A Review and Recommendations
Porter, C. O. L. H., Outlaw, R., Gale, J. P., & Cho, T. S. (2018) · Sage Journals Data · doi:10.25384/sage.c.4310936 · Cited by 49
EXPLORING TASK PROPERTIES IN CROWDSOURCING – AN EMPIRICAL STUDY ON MECHANICAL TURK
Schulze, T., Seedorf, S., Geiger, D., Kaufmann, N., & Schader, M. (2011) · MADOC (University of Mannheim) · Cited by 46
Siegel, J. T., Navarro, M. A., & Thomson, A. L. (2015) · Social Science & Medicine · doi:10.1016/j.socscimed.2015.08.020 · Cited by 45
Deterring Unethical Behavior in Online Labor Markets
Brink, W. D., Eaton, T. V., Grenier, J. H., & Reffett, A. B. (2017) · Journal of Business Ethics · doi:10.1007/s10551-017-3570-y · Cited by 40
Colman, D. E., Vineyard, J., & Letzring, T. D. (2017) · Personality and Individual Differences · doi:10.1016/j.paid.2017.06.023 · Cited by 39
On the Transportability of Laboratory Results
Bader, F., Baumeister, B., Berger, R. D., & Keuschnigg, M. (2019) · Sociological Methods & Research · doi:10.1177/0049124119826151 · Cited by 28
Comparing Amazon Mechanical Turk with unpaid internet resources in online clinical trials
Bunge, E. L., Cook, H. M., Bond, M. J., Williamson, R. E., Cano, M. T., Barrera, A. Z., Leykin, Y., & Muñoz, R. F. (2018) · Internet Interventions · doi:10.1016/j.invent.2018.04.001 · Cited by 24
Microworkers as research participants: Does underpaying Turkers lead to cognitive dissonance?
Liu, B., & Sundar, S. S. (2018) · Computers in Human Behavior · doi:10.1016/j.chb.2018.06.017 · Cited by 19
Tosti‐Kharas, J., & Conley, C. A. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.00741 · Cited by 17
Research works printed but not linked: 2
- Bederson B. B., 2011, ACM CHI Extended Abstracts on Human Factors in Computing Systems, 2011, 97
- Dennis S. A., Goodson B. M., Pearson C. 2019. Virtual private servers and the limitations of IP-based screening procedures: Lessons from the MTurk quality crisis of 2018 (Publication No. 3233954). SSRN.
Cited by
Douglas, B. D., Ewell, P. J., & Brauer, M. (2023) · PLOS ONE · doi:10.1371/journal.pone.0279720 · Cited by 1,532
Survey response rates: Trends and a validity assessment framework
Holtom, B., Baruch, Y., Aguinis, H., & A Ballinger, G. (2022) · Human Relations · doi:10.1177/00187267211070769 · Cited by 380
Self-citationDo Investors Care about Impact?
Heeb, F., Kölbel, J. F., Paetzold, F., & Zeisberger, S. (2023) · The Review of Financial Studies · doi:10.1093/rfs/hhac066 · Cited by 332
Attitude Toward the Color Blue: An Ideal Marker Variable
Miller, B. K., & Simmering, M. J. (2023) · Organizational Research Methods · doi:10.1177/10944281221075361 · Cited by 298
Consumer Responses to AI-Generated Charitable Giving Ads
Arango, L., Singaraju, S. P., & Niininen, O. (2023) · Journal of Advertising · doi:10.1080/00913367.2023.2183285 · Cited by 243
Metadata
- Authors
- Aguinis, H., Villamor, I., & Ramani, R. S.
- Venue
- Journal of Management
- Publisher
- SAGE Publications
- Published
- 2020-11-19
- Volume
- 47
- Issue
- 4
- Pages
- 823-837
- ISSN
- 0149-2063 (ISSN-L)
- 1557-1211
- OpenAlex
- W3108819963
- Cited by
- 984
- References
- 87
- FWCI
- 103.08
- Type
- journal article
- Language
- en
- Open access
- bronze
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.