Who Broke Amazon Mechanical Turk?: An Analysis of Crowdsourcing Data Quality over Time
Abstract
We present the results of a survey fielded in June of 2022 as a lens to examine recent data reliability issues on Amazon Mechanical Turk. We contrast bad data from this survey with bad data from the same survey fielded among US workers in October 2013, April 2018, and February 2019. Application of an established data cleaning scheme reveals that unusable data has risen from a little over 2% in 2013 to almost 90% in 2022. Through symptomatic diagnosis, we attribute the data reliability drop not to an increase in bad faith work, but rather to a continuum of English proficiency levels. A qualitative analysis of workers’ responses to open-ended questions allows us to distinguish between low fluency workers, ultra-low fluency workers, satisficers, and bad faith workers. We go on to show the effects of the new low fluency work on Likert scale data and on the study's qualitative results. Attention checks are shown to be much less effective than they once were at identifying survey responses that should be discarded.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Research has shown that data from MTurk workers is comparable to data from other pools (e.g., commercial panels, social media, colleges), but recently there has been a decrease in data quality.
“Research has shown data from MTurk workers is comparable to data from other pools (e.g., commercial panels, social media, colleges) [12, 20, 23, 32, 69, 80], but recently there has been a decrease in data quality [28, 49, 66, 68, 104, 108].”
From "I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust (2024) · cites this paper as [68]
Cites
Identifying careless responses in survey data.
Meade, A. W., & Craig, S. B. (2012) · Psychological Methods · doi:10.1037/a0028085 · Cited by 3,865 · see citing works
Hauser, D. J., & Schwarz, N. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0578-z · Cited by 1,739 · see citing works
Data quality of platforms and panels for online behavioral research
Peer, E., Rothschild, D., Gordon, A., Evernden, Z., & Damer, E. (2021) · Behavior Research Methods · doi:10.3758/s13428-021-01694-3 · Cited by 1,718 · see citing works
Reputation as a sufficient condition for data quality on Amazon Mechanical Turk
Peer, E., Vosgerau, J., & Acquisti, A. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0434-y · Cited by 1,618 · see citing works
An MTurk Crisis? Shifts in Data Quality and the Impact on Study Results
Chmielewski, M., & Kucker, S. C. (2020) · Social Psychological and Personality Science · doi:10.1177/1948550619875149 · Cited by 1,027 · see citing works
Who are the crowdworkers?: shifting demographics in mechanical turk
Ross, J., Irani, L., Silberman, M. S., Zaldivar, A., & Tomlinson, B. (2010) · CHI '10 Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/1753846.1753873 · Cited by 1,004 · see citing works
van Selm, M., & Jankowski, N. W. (2006) · Quality & Quantity · doi:10.1007/s11135-005-8081-8 · Cited by 922 · see citing works
Demographics and Dynamics of Mechanical Turk Workers
Difallah, D. E., Filatova, E., & Ipeirotis, P. G. (2018) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3159652.3159661 · Cited by 512 · see citing works
The shape of and solutions to the MTurk quality crisis
Kennedy, R., Clifford, S., Burleigh, T., Waggoner, P. D., Jewell, R., & Winter, N. J. G. (2020) · Political Science Research and Methods · doi:10.1017/psrm.2020.6 · Cited by 506 · see citing works
Got Bots? Practical Recommendations to Protect Online Survey Data from Bot Attacks
Storozuk, A., Ashley, M., Delage, V., & Maloney, E. A. (2020) · The Quantitative Methods for Psychology · doi:10.20982/tqmp.16.5.p472 · Cited by 329 · see citing works
Methods to detect low quality data and its implication for psychological research
Buchanan, E. M., & Scofield, J. E. (2018) · Behavior Research Methods · doi:10.3758/s13428-018-1035-6 · Cited by 253 · see citing works
Understanding Malicious Behavior in Crowdsourcing Platforms
Gadiraju, U., Kawase, R., Dietze, S., & Demartini, G. (2015) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702443 · Cited by 241 · see citing works
The Crowd is a Collaborative Network
Gray, M. L., Suri, S., Ali, S. S., & Kulkarni, D. (2016) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2818048.2819942 · Cited by 239 · see citing works
Bentley, F. R., Daskalova, N., & White, B. (2017) · Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/3027063.3053335 · Cited by 91 · cited 1× · see citing works
“This type of work is perhaps the most frustrating to qualitative researchers who appreciated the type of diverse participation Mechanical Turk offered [5] [31] [33].”
BackgroundClarity is a Worthwhile Quality
Gadiraju, U., Yang, J., & Bozzon, A. (2017) · ACM Conference on Hypertext and Social Media (HT) · doi:10.1145/3078714.3078715 · Cited by 87 · see citing works
Survey Satisficing Inflates Reliability and Validity Measures
Hamby, T., & Taylor, W. (2016) · Educational and Psychological Measurement · doi:10.1177/0013164415627349 · Cited by 81 · see citing works
Collecting Online Survey Data: A Comparison of Data Quality among a Commercial Panel & MTurk
Zhang, B., & Gearhart, S. (2020) · Survey Practice · doi:10.29115/sp-2020-0015 · Cited by 73 · see citing works
Dennis, S. A., Goodson, B. M., & Pearson, C. A. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3233954 · Cited by 66 · see citing works
Hargittai, E., & Shaw, A. (2020) · Socius Sociological Research for a Dynamic World · doi:10.1177/2378023119889834 · Cited by 64 · see citing works
Striving to Earn More: A Survey of Work Strategies and Tool Use Among Crowd Workers
Kaplan, T., Saito, S., Hara, K., & Bigham, J. P. (2018) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v6i1.13327 · Cited by 63 · see citing works
Privacy, Power, and Invisible Labor on Amazon Mechanical Turk
Sannon, S., & Cosley, D. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300512 · Cited by 63 · see citing works
The Perpetual Work Life of Crowdworkers
Williams, A. C., Mark, G. J., Milland, K., Lank, E., & Law, E. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359126 · Cited by 62 · see citing works
A hunt for the Snark: Annotator Diversity in Data Practices
Kapania, S., Taylor, A. S., & Wang, D. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580645 · Cited by 50 · see citing works
Modus Operandi of Crowd Workers
Gadiraju, U., Checco, A., Gupta, N., & Demartini, G. (2017) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3130914 · Cited by 41 · see citing works
The Hidden Cost of Using Amazon Mechanical Turk for Research
Saravanos, A., Zervoudakis, S., Zheng, D., Stott, N., Hawryluk, B., & Delfino, D. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-90238-4_12 · Cited by 29 · see citing works
All That Glitters Is Gold — An Attack Scheme on Gold Questions in Crowdsourcing
Checco, A., Bates, J., & Demartini, G. (2018) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v6i1.13332 · Cited by 21 · see citing works
Debugging a Crowdsourced Task with Low Inter-Rater Agreement
Alonso, O., Marshall, C. C., & Najork, M. (2015) · ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL) · doi:10.1145/2756406.2757741 · Cited by 19 · see citing works
Self-citationCrowdsourcing Quality Concerns: An Examination of Amazon’s Mechanical Turk
Dupuis, M., Renaud, K., & Searle, R. (2022) · Conference on Information Technology Education · doi:10.1145/3537674.3555783 · Cited by 17 · see citing works
Marshall, C. C., & Shipman, F. M., III (2017) · Communications of the ACM · doi:10.1145/2996181 · Cited by 11 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Kim, S. S. Y., Liao, Q. V., Vorvoreanu, M., Ballard, S., & Vaughan, J. W. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3658941 · Cited by 147 · see citing works
Humans vs. ChatGPT: Evaluating Annotation Methods for Financial Corpora
Kaikaus, J., Li, H., & Brunner, R. J. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386425 · Cited by 8 · see citing works
Towards Fair and Equitable Incentives to Motivate Paid and Unpaid Crowd Contributions
Wallace, S., Massachi, T., Su, J., Miller, D., & Huang, J. J. S. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714195 · Cited by 8 · see citing works
Barends, A. J., & Andrews, W. (2023) · Social Influence · doi:10.1080/15534510.2023.2256492 · Cited by 6 · see citing works
The Self-Control Ability Scale: Measuring a Key Construct of Situational Action Theory
Hasselhorn, F. A., Sattler, S., Kroneberg, C., & Seddig, D. (2024) · Justice Quarterly · doi:10.1080/07418825.2024.2413584 · Cited by 6 · see citing works
Kazari, K., Chen, Y., & Shakeri, Z. (2025) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc58623.2025.11253134 · Cited by 5 · see citing works
Hart, C. G., & McLaughlin, H. (2024) · Socius Sociological Research for a Dynamic World · doi:10.1177/23780231241290545 · Cited by 4 · see citing works
Case Law Grounding: Using Precedents to Align Decision-Making for Humans and AI
Chen, Q. Z., & Zhang, A. X. (2025) · ACM Collective Intelligence Conference (CI) · doi:10.1145/3715928.3737487 · Cited by 4 · see citing works
Cova, F., & Martinez, T. (2024) · Erkenntnis · doi:10.1007/s10670-024-00844-1 · Cited by 4 · see citing works
Is Crowdsourcing a Puppet Show? Detecting a New Type of Fraud in Online Platforms
Wang, S., Jui, I. J., & Thorpe, J. (2024) · New Security Paradigms Workshop · doi:10.1145/3703465.3703472 · Cited by 3 · see citing works
Suviranta, R., & Hiippala, T. (2025) · Digital Scholarship in the Humanities · doi:10.1093/llc/fqaf110 · Cited by 1 · see citing works
Tragedy of the Commons in Crowd Work-Based Research
Xia, H. (2023) · ACM Journal on Responsible Computing · doi:10.1145/3626493 · Cited by 1 · see citing works
High, G., Nussbaum, P., & Green, P. (2023) · Color Research & Application · doi:10.1002/col.22913 · Cited by 1 · see citing works
Dynamic payment on microtasking platforms using bee colony optimization
Moayedikia, A. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124862 · Cited by 1 · see citing works
Paulino, D., Ferreira, J. R., Netto, A. T. C., Correia, A., Ribeiro, J., Guimarães, D., Barroso, J., & Paredes, H. (2024) · SeGAH ... IEEE ... International Conference on Serious Games and Applications for Health/IEEE International Conference on Serious Games and Applications for Health · doi:10.1109/segah61285.2024.10639597 · Cited by 1 · see citing works
The scientific enterprise is a human enterprise
Richardson, R. (2024) · doi:10.59350/9hra6-hdt61
Xia, H., & Liu, J. (2024) · Journal of Information Communication and Ethics in Society · doi:10.1108/jices-01-2024-0011
Are some parties immune from scandal? Party family, scandal exposure, and party evaluation
Leiter, D. L., & Vonnahme, G. (2025) · Journal of Elections Public Opinion and Parties · doi:10.1080/17457289.2025.2474423
The Magnitude of Truth: On Using Magnitude Estimation for Truthfulness Assessment
Soprano, M., Tapu, D. E., Barbera, D. L., Roitero, K., & Mizzaro, S. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730091
Attract & Engage Visitors for Tabular Data Maintenance: a Longitudinal Naturalistic Field Study
Wallace, S., Lee, N. K., Peng, Z., Massachi, T., Do, L., Rathore, S., Brown, S. M., & Huang, J. J. S. (2026) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3816901
- Authors
- Marshall, C. C., Goguladinne, P. S. R., Maheshwari, M., Sathe, A., & Shipman, F. M.
- Affiliations
- Texas A&M University (US)
- Department of Computer Science and Engineering, Texas A&M University, USA
- Venue
- Proceedings of the 15th ACM Web Science Conference 2023
- Publisher
- ACM
- Published
- 2023-04-30
- Pages
- 335-345
- OpenAlex
- W4367046989
- Cited by
- 20
- References
- 29
- Citations by year
- 2026: 1 · 2025: 6 · 2024: 9 · 2023: 4
- FWCI
- 6.05
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Web Science Conference (WebSci) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- Amazon Mechanical Turk, data reliability, online labour platforms, English proficiency, attention checks, data cleaning, survey response quality, satisficers, Likert scale data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
