The DOI matches its Crossref record; 45 of the 46 references it cites were found in a registry or an index, and 1 are not checked yet.
An MTurk Crisis? Shifts in Data Quality and the Impact on Study Results
Abstract
Amazon’s Mechanical Turk (MTurk) is arguably one of the most important research tools of the past decade. The ability to rapidly collect large amounts of high-quality human subjects data has advanced multiple fields, including personality and social psychology. Beginning in summer 2018, concerns arose regarding MTurk data quality leading to questions about the utility of MTurk for psychological research. We present empirical evidence of a substantial decrease in data quality using a four-wave naturalistic experimental design: pre-, during, and post-summer 2018. During and to some extent post-summer 2018, we find significant increases in participants failing response validity indicators, decreases in reliability and validity of a widely used personality measure, and failures to replicate well-established findings. However, these detrimental effects can be mitigated by using response validity indicators and screening the data. We discuss implications and offer suggestions to ensure data quality.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Also, a large proportion of sample 3 data (35%) was collected from MTURK. While we used attention checks and an open-ended response item to reduce the incidence of “bots,” automated, or inattentive responding, there is growing concern that the quality of MTURK data has become increasingly poor since 2018 (Chmielewski & Kucker, 2020), which limits the conclusions we can draw.”
From Exploratory and Confirmatory Bayesian Networks Identify the Central Role of Non-judging in Symptoms of Depression (2021) · cites this paper as Chmielewski & Kucker, 2020
In the citing paper
“Research has shown data from MTurk workers is comparable to data from other pools (e.g., commercial panels, social media, colleges) [12, 20, 23, 32, 69, 80], but recently there has been a decrease in data quality [28, 49, 66, 68, 104, 108].”
From "I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust (2024) · cites this paper as [28]
In the citing paper
“Although some recent concerns about MTurk quality have emerged, data support that diverse validity and attention checks, such as those included in the present studies, adequately protect data quality [90–92].”
From Not so terrifying after all? A set of failed replications of the mortality salience effects of Terror Management Theory (2023) · cites this paper as [92]
In the citing paper
“The literature on the reliability and validity of data obtained from crowdsourcing panels (such as Clickworker.de)emphasize the necessity of thorough data screening for low diligence participants (e.g., Chmielewski & Kucker, 2020;Kennedy et al., 2020).”
From The detection of political deepfakes (2022) · cites this paper as Chmielewski & Kucker, 2020
In the citing paper
“For the online studies (Studies 2 and 3), we additionally implemented the following exclusion criteria: Participants were excluded if they failed instructed response questions, which served as an attention check (Kennedy et al., 2020), or they provided inconsistent information between their birth year at the start of the study and their age later in the study (Chmielewski & Kucker, 2020).”
From Mind wandering during first- and foreign-language reading (2026) · cites this paper as Chmielewski & Kucker, 2020
Citations
Cites
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996
Tripartite model of anxiety and depression: psychometric evidence and taxonomic implications
Clark, L. A., & Watson, D. (1991) · Journal of Abnormal Psychology · doi:10.1037/0021-843x.100.3.316 · Cited by 4,601
Instructional manipulation checks: Detecting satisficing to increase statistical power
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2009.03.009 · Cited by 3,493
Conducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172 · cited 2×
“There are long-standing recommendations to include response validity indicators when using MTurk to remove inva-lid or low-quality data (Barger, Behrend, Sharek, & Sinar, 2011; Kittur, Chi, & Suh, 2008; Mason & Suri, 2011; Zhu & Carterette, 2010).”
BackgroundKotov, R., Gamez, W., Schmidt, F., & Watson, D. (2010) · Psychological Bulletin · doi:10.1037/a0020327 · Cited by 2,677 · cited 2×
“Specifically, decades of research and multiple meta-analyses (Kotov et al., 2010; Mal-ouff et al., 2005) demonstrate that neuroticism has substantial links to the mood and anxiety disorders, whereas extraversion, conscientiousness, and agreeableness tend to demonstrate weaker (negative)…”
BackgroundData collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242 · cited 2×
“…a variety of study designs and numerous types of data (Behrend et al., 2011; Buhrmester et al., 2011; Casler et al., 2013; Eriksson & Simpson, 2010; Goodman, Cryder, & Cheema, 2013; Horton, Rand, & Zeckhauser, 2011; Kees et al., 2017; Mason & Watts, 2009; Paolacci & Chandler, 2014; Shapiro,…”
BackgroundHandbook of Personality Theory and Research.
Schwartz, M., Borgatta, E. F., & Lambert, W. W. (1970) · American Sociological Review · doi:10.2307/2093310 · Cited by 2,163
Inside the Turk: Understanding Mechanical Turk as a Participant Pool
Paolacci, G., & Chandler, J. (2014) · Current Directions in Psychological Science · doi:10.1177/0963721414531598 · Cited by 2,136
Crowdsourcing user studies with Mechanical Turk
Kittur, A., Chi, E. H., & Suh, B. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357127 · Cited by 1,923
Casler, K., Bickel, L., & Hackett, E. (2013) · Computers in Human Behavior · doi:10.1016/j.chb.2013.05.009 · Cited by 1,662 · cited 3×
“…or superior in quality to data collected in the lab, from professional online panels, and using marketing research companies (Behrend, Sharek, Meade, & Wiebe, 2011; Buhrme-ster et al., 2011; Casler, Bickel, & Hackett, 2013; Kees, Berry, Burton, & Sheehan, 2017; Paolacci & Chandler, 2014).”
BackgroundMethod
Research works printed but not linked: 5
- Barger P., 2011, The Industrial-Organizational Psychologist, 49, 11
- John O. P., 1991, The Big Five Inventory—versions 4a and 54
- Sylaska K., 2019, It’s 2019: Do we need super attention check items to conduct web-based survey research? The evolution of MTurk survey respondents
- Vannette D. L., 2016, Paper presented at the 71st Annual Conference of the American Association for Public Opinion Research
- Zhu D., Carterette B. (2010). An analysis of assessor behavior in crowdsourced preference judgments. Presented at the SIGIR 2010 Workshop on Crowdsourcing for Search Evaluations, Geneva, Switzerland, 17–20.
Cited by
Douglas, B. D., Ewell, P. J., & Brauer, M. (2023) · PLOS ONE · doi:10.1371/journal.pone.0279720 · Cited by 1,532
Sharma, M., Tong, M., Korbak, T., Duperrouzel, D., Was, H., Korbak, J., Cheng, M., Askell, A., Bai, Y., Brain, C., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.13548 · Cited by 1,313
“Human evaluators are imperfect (Saunders et al., 2022; Gudibande et al., 2023), make mistakes e.g., due to limited time (Chmielewski & Kucker, 2020) or cognitive biases (Pandey et al., 2022), and sometimes have diverse, contradictory preferences (Bakker et al., 2022).”
ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks
Gilardi, F., Alizadeh, M., & Kubli, M. (2023) · arXiv · doi:10.48550/arxiv.2303.15056 · Cited by 1,207
ChatGPT outperforms crowd workers for text-annotation tasks
Gilardi, F., Alizadeh, M., & Kubli, M. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2305016120 · Cited by 1,197
“Moreover, there have been concerns that MTurk data quality has decreased (Chmielewski and Kucker, 2020), while alternative platforms such as CrowdFlower and FigureEight are no longer practicable options for academic research since they were acquired by Appen, a company that is focused on a business…”
BackgroundDesigning Information Provision Experiments
Haaland, I., Roth, C., & Wohlfart, J. (2023) · Journal of Economic Literature · doi:10.1257/jel.20211658 · Cited by 541
Metadata
- Authors
- Chmielewski, M., & Kucker, S. C.
- Affiliations
- Southern Methodist University (US)
- Oklahoma State University (US)
- Corresponding author
- Chmielewski, Michael
- Venue
- Social Psychological and Personality Science
- Publisher
- SAGE Publications
- Published
- 2019-10-10
- Volume
- 11
- Issue
- 4
- Pages
- 464-473
- ISSN
- 1948-5506 (ISSN-L)
- 1948-5514
- OpenAlex
- W2979864122
- Cited by
- 1,027
- References
- 46
- Citations by year
- 2026: 89 · 2025: 129 · 2024: 176 · 2023: 206 · 2022: 184 · 2021: 181 · 2020: 57 · 2019: 3
- FWCI
- 115.20
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Social Psychological and Personality Science (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- Amazon Mechanical Turk, data quality, response validity, personality measures, reliability statistics, replication, online labour platforms, participant screening, naturalistic experiment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: An MTurk Crisis? Shifts in Data Quality and the Impact on Study Results. https://app.cochat.ai/papers/10.1177/1948550619875149
- The paper
- Chmielewski, M., & Kucker, S. C. (2020). An MTurk Crisis? Shifts in Data Quality and the Impact on Study Results. Social Psychological and Personality Science. https://doi.org/10.1177/1948550619875149
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.