The shape of and solutions to the MTurk quality crisis
Open accessAbstract
Amazon's Mechanical Turk is widely used for data collection; however, data quality may be declining due to the use of virtual private servers to fraudulently gain access to studies. Unfortunately, we know little about the scale and consequence of this fraud, and tools for social scientists to detect and prevent this fraud are underdeveloped. We first analyze 38 studies and show that this fraud is not new, but has increased recently. We then show that these fraudulent respondents provide particularly low-quality data and can weaken treatment effects. Finally, we provide two solutions: an easy-to-use application for identifying fraud in the existing datasets and a method for blocking fraudulent respondents in Qualtrics surveys.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Research has shown that data from MTurk workers is comparable to data from other pools (e.g., commercial panels, social media, colleges), but recently there has been a decrease in data quality.
“Research has shown data from MTurk workers is comparable to data from other pools (e.g., commercial panels, social media, colleges) [12, 20, 23, 32, 69, 80], but recently there has been a decrease in data quality [28, 49, 66, 68, 104, 108].”
From "I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust (2024) · cites this paper as [49]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008 · see citing works
Instructional manipulation checks: Detecting satisficing to increase statistical power
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2009.03.009 · Cited by 3,493 · see citing works
Inside the Turk: Understanding Mechanical Turk as a Participant Pool
Paolacci, G., & Chandler, J. (2014) · Current Directions in Psychological Science · doi:10.1177/0963721414531598 · Cited by 2,136 · see citing works
Hauser, D. J., & Schwarz, N. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0578-z · Cited by 1,739 · see citing works
Reputation as a sufficient condition for data quality on Amazon Mechanical Turk
Peer, E., Vosgerau, J., & Acquisti, A. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0434-y · Cited by 1,618 · see citing works
The Online Laboratory: Conducting Experiments in a Real Labor Market∗Preprint
Zeckhauser, R. J. (2016) · Cited by 1,454 · see citing works
The Generalizability of Survey Experiments
Mullinix, K. J., Leeper, T. J., Druckman, J. N., & Jeremy Freese, A. (2015) · Journal of Experimental Political Science · doi:10.1017/xps.2015.19 · Cited by 1,235 · see citing works
reCAPTCHA: Human-Based Character Recognition via Web Security Measures
von Ahn, L., Maurer, B., McMillen, C. P., Abraham, D., & Blum, M. (2008) · Science · doi:10.1126/science.1160379 · Cited by 1,116 · see citing works
Berinsky, A. J., Margolis, M. F., & Sances, M. W. (2014) · American Journal of Political Science · doi:10.1111/ajps.12081 · Cited by 951 · cited 1× · see citing works
“Although researchers often use instructional manipulation checks (Oppenheimer et al ., 2009; Berinsky et al ., 2013), we avoid this style of attention check because the format is easily recognizable, making it less diagnostic of attention among professional survey respondents (Thomas and Clifford,…”
How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It
Montgomery, J. M., Nyhan, B., & Torres, M. (2018) · American Journal of Political Science · doi:10.1111/ajps.12357 · Cited by 800 · see citing works
Are samples drawn from Mechanical Turk valid for research on political ideology?
Clifford, S., Jewell, R., & Waggoner, P. (2015) · Research & Politics · doi:10.1177/2053168015622072 · Cited by 680 · see citing works
Self-citationValidity and Mechanical Turk: An assessment of exclusion methods and interactive experiments
Thomas, K. A., & Clifford, S. (2017) · Computers in Human Behavior · doi:10.1016/j.chb.2017.08.038 · Cited by 625 · see citing works
Self-citationWeinberg, J. D., Jeremy Freese, A., & McElhattan, D. (2014) · Sociological Science · doi:10.15195/v1.a19 · Cited by 582 · see citing works
Generalizing from Survey Experiments Conducted on Mechanical Turk: A Replication Approach
Coppock, A. (2019) · Political Science Research and Methods · doi:10.1017/psrm.2018.10 · Cited by 515 · see citing works
Zhou, H., & Ayelet Fishbach, B. (2016) · Journal of Personality and Social Psychology · doi:10.1037/pspa0000056 · Cited by 462 · see citing works
Chandler, J., & Paolacci, G. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550617698203 · Cited by 280 · see citing works
Cross-Sample Comparisons and External Validity
Krupnikov, Y., & Levine, A. S. (2014) · Journal of Experimental Political Science · doi:10.1017/xps.2014.7 · Cited by 251 · see citing works
Using Nonnaive Participants Can Reduce Effect Sizes
Chandler, J., Paolacci, G., Peer, E., Mueller, P., & Ratliff, K. A. (2015) · Psychological Science · doi:10.1177/0956797615585115 · Cited by 185 · see citing works
Stritch, J. M., Pedersen, M. J., & Taggart, G. (2017) · International Public Management Journal · doi:10.1080/10967494.2016.1276493 · Cited by 181 · see citing works
Using Crowdsourcing Websites for Sociological Research: The Case of Amazon Mechanical Turk
Shank, D. B. (2015) · The American Sociologist · doi:10.1007/s12108-015-9266-9 · Cited by 176 · see citing works
Random Responding as a Threat to the Validity of Effect Size Estimates in Correlational Research
Credé, M. (2010) · Educational and Psychological Measurement · doi:10.1177/0013164410366686 · Cited by 161 · see citing works
Using Nonnaive Participants Can Reduce Effect SizesPreprint
Chandler, J., Paolacci, G., Péer, E., Mueller, P. A., & Ratliff, K. A. (2015) · RePEc: Research Papers in Economics · Cited by 118 · see citing works
Why So Serious?: Survey Trolls and MisinformationPreprint
Lopez, J., & Hillygus, D. S. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3131087 · Cited by 105 · see citing works
Cheating on Political Knowledge Questions in Online Surveys
Clifford, S., & Jerit, J. (2016) · Public Opinion Quarterly · doi:10.1093/poq/nfw030 · Cited by 89 · see citing works
Self-citationDennis, S. A., Goodson, B. M., & Pearson, C. A. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3233954 · Cited by 66 · see citing works
Compassionate Democrats and Tough Republicans: How Ideology Shapes Partisan Stereotypes
Clifford, S. (2019) · Political Behavior · doi:10.1007/s11109-019-09542-z · Cited by 55 · see citing works
Self-citationA Simplified Protocol to Screen Out VPS and International Respondents Using QualtricsPreprint
Winter, N., Burleigh, T. J., Kennedy, R., et al. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3327274 · Cited by 55 · see citing works
Self-citationDetecting Fraud in Online Surveys by Tracing, Scoring, and Visualizing IP Addresses
Waggoner, P., Kennedy, R., & Clifford, S. (2019) · The Journal of Open Source Software · doi:10.21105/joss.01285 · Cited by 52 · see citing works
Self-citationTaking the time? Explaining effortful participation among low-cost online survey participants
Anson, I. G. (2018) · Research & Politics · doi:10.1177/2053168018785483 · Cited by 50 · see citing works
Psychologists grow increasingly dependent on online research subjects
(2017) · AAAS Articles DO Group · doi:10.1126/science.aag0592 · Cited by 8 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Attitudes towards AI: measurement and associations with personality
Stein, J.-P., Messingschlager, T., Gnambs, T., Hutmacher, F., & Appel, M. (2024) · Scientific Reports · doi:10.1038/s41598-024-53335-2 · Cited by 309 · see citing works
“With our data, we observed a very good Cronbach’s α of 0.86 for the CMQ.”
MethodInfluentialKim, S. S. Y., Liao, Q. V., Vorvoreanu, M., Ballard, S., & Vaughan, J. W. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3658941 · Cited by 147 · see citing works
“Research has shown data from MTurk workers is comparable to data from other pools (e.g., commercial panels, social media, colleges) [12, 20, 23, 32, 63, 74], but recently there has been a decrease in data quality [28, 44, 60, 62, 94, 98].”
ResultImproving Data Quality Using Amazon Mechanical Turk Through Platform Setup
Lu, L., Neale, N., Line, N. D., & Bonn, M. (2022) · Cornell Hospitality Quarterly · doi:10.1177/19389655211025475 · Cited by 94 · see citing works
The detection of political deepfakes
Appel, M., & Prietzel, F. (2022) · Journal of Computer-Mediated Communication · doi:10.1093/jcmc/zmac008 · Cited by 95 · see citing works
Veselovsky, V., Ribeiro, M. H., & West, R. (2023) · arXiv · doi:10.48550/arxiv.2306.07899 · Cited by 73 · see citing works
“…tasks (Draws et al., 2021; Bragg et al., 2013), conducted au-dits on the overall quality of crowdsourced annotations (Chmielewski and Kucker, 2020; Kennedy et al., 2020; Smith et al., 2016), and shed light on the demographics and socioeconomic conditions of workers using crowdsourcing platforms…”
The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation
Karpinska, M., Akoury, N., & Iyyer, M. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.97 · Cited by 56 · see citing works
Trust within human-machine collectives depends on the perceived consensus about cooperative norms
Makovi, K., Sargsyan, A., Li, W., Bonnefon, J.-F., & Rahwan, T. (2023) · Nature Communications · doi:10.1038/s41467-023-38592-5 · Cited by 36 · see citing works
Who Broke Amazon Mechanical Turk?: An Analysis of Crowdsourcing Data Quality over Time
Marshall, C. C., Goguladinne, P. S. R., Maheshwari, M., Sathe, A., & Shipman, F. M. (2023) · Proceedings of the 15th ACM Web Science Conference 2023 · doi:10.1145/3578503.3583622 · Cited by 20 · see citing works
Understanding the costs and benefits of politics among adolescents within a sociocultural context
Oosterhoff, B., Poppler, A., Hill, R. M., Fitzgerald, H., & Shook, N. J. (2022) · Infant and Child Development · doi:10.1002/icd.2280 · Cited by 21 · see citing works
Vázquez, A. L., Culianos, D., Flores, C. M. N., Alvarez, M. D. L. C., Barrett, T. S., & Domenech Rodríguez, M. M. (2022) · Child & Youth Care Forum · doi:10.1007/s10566-021-09656-8 · Cited by 15 · see citing works
- Authors
- Kennedy, R., Clifford, S., Burleigh, T., Waggoner, P. D., Jewell, R., & Winter, N. J. G.
- Affiliations
- University of Houston (US)
- University of Chicago (US)
- University of Virginia (US)
- Corresponding author
- Clifford, Scott
- Venue
- Political Science Research and Methods
- Publisher
- Cambridge University Press (CUP)
- Published
- 2020-04-24
- Volume
- 8
- Issue
- 4
- Pages
- 614-629
- ISSN
- 2049-8470 (ISSN-L)
- 2049-8489
- OpenAlex
- W3123404044
- Cited by
- 506
- References
- 30
- Citations by year
- 2026: 45 · 2025: 56 · 2024: 87 · 2023: 96 · 2022: 111 · 2021: 85 · 2020: 21 · 2019: 4 · 2018: 1
- FWCI
- 71.82
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Political Science Research and Methods (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Physical Sciences › Computer Science › Information Systems › Spam and Phishing Detection
- Keywords
- Amazon Mechanical Turk, virtual private server, data quality, treatment effect estimation, fraud detection, Qualtrics, crowdsourced data collection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
