An MTurk Crisis? Shifts in Data Quality and the Impact on Study Results
The DOI matches its Crossref record; 45 of the 46 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Amazon’s Mechanical Turk (MTurk) is arguably one of the most important research tools of the past decade. The ability to rapidly collect large amounts of high-quality human subjects data has advanced multiple fields, including personality and social psychology. Beginning in summer 2018, concerns arose regarding MTurk data quality leading to questions about the utility of MTurk for psychological research. We present empirical evidence of a substantial decrease in data quality using a four-wave naturalistic experimental design: pre-, during, and post-summer 2018. During and to some extent post-summer 2018, we find significant increases in participants failing response validity indicators, decreases in reliability and validity of a widely used personality measure, and failures to replicate well-established findings. However, these detrimental effects can be mitigated by using response validity indicators and screening the data. We discuss implications and offer suggestions to ensure data quality.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Also, a large proportion of sample 3 data (35%) was collected from MTURK. While we used attention checks and an open-ended response item to reduce the incidence of “bots,” automated, or inattentive responding, there is growing concern that the quality of MTURK data has become increasingly poor since 2018 (Chmielewski & Kucker, 2020), which limits the conclusions we can draw.”
From Exploratory and Confirmatory Bayesian Networks Identify the Central Role of Non-judging in Symptoms of Depression (2021) · cites this paper as Chmielewski & Kucker, 2020
In the citing paper
“Research has shown data from MTurk workers is comparable to data from other pools (e.g., commercial panels, social media, colleges) [12, 20, 23, 32, 69, 80], but recently there has been a decrease in data quality [28, 49, 66, 68, 104, 108].”
From "I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust (2024) · cites this paper as [28]
In the citing paper
“Although some recent concerns about MTurk quality have emerged, data support that diverse validity and attention checks, such as those included in the present studies, adequately protect data quality [90–92].”
From Not so terrifying after all? A set of failed replications of the mortality salience effects of Terror Management Theory (2023) · cites this paper as [92]
In the citing paper
“The literature on the reliability and validity of data obtained from crowdsourcing panels (such as Clickworker.de)emphasize the necessity of thorough data screening for low diligence participants (e.g., Chmielewski & Kucker, 2020;Kennedy et al., 2020).”
From The detection of political deepfakes (2022) · cites this paper as Chmielewski & Kucker, 2020
In the citing paper
“For the online studies (Studies 2 and 3), we additionally implemented the following exclusion criteria: Participants were excluded if they failed instructed response questions, which served as an attention check (Kennedy et al., 2020), or they provided inconsistent information between their birth year at the start of the study and their age later in the study (Chmielewski & Kucker, 2020).”
From Mind wandering during first- and foreign-language reading (2026) · cites this paper as Chmielewski & Kucker, 2020
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996 · cited 4×
“In fact, these approaches are how researchers originally established MTurk’s high-quality data (Behrend et al., 2011; Buhrmester et al., 2011; Casler et al., 2013; Good-man et al., 2013; Kees et al., 2017; Mason & Watts, 2009; Sha-piro et al., 2013).”
InfluentialTripartite model of anxiety and depression: psychometric evidence and taxonomic implications
Clark, L. A., & Watson, D. (1991) · Journal of Abnormal Psychology · doi:10.1037/0021-843x.100.3.316 · Cited by 4,601 · cited 1×
“…Inventory (BFI; John, Donahue, & Kentle, 1991) and test whether well-established associations with psychopathology replicate (Bagby et al., 1996; Clark & Watson, 1991; Kotov, Gamez, Schmidt, & Watson, 2010; Krueger & Tackett, 2006; Malouff, Thorsteinsson, & Schutte, 2005; Widiger & Trull, 2007).”
Instructional manipulation checks: Detecting satisficing to increase statistical power
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2009.03.009 · Cited by 3,493 · cited 1×
“Similarly, instructional manipulation checks (IMCs) that were previously recommended as a way to ensure high-quality data (see Oppenheimer, Meyvis, & Davidenko, 2009) have been found to alter participants’ responses in potentially problematic ways (see Hauser & Schwarz, 2015).”
BackgroundConducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172 · cited 2×
“There are long-standing recommendations to include response validity indicators when using MTurk to remove inva-lid or low-quality data (Barger, Behrend, Sharek, & Sinar, 2011; Kittur, Chi, & Suh, 2008; Mason & Suri, 2011; Zhu & Carterette, 2010).”
BackgroundKotov, R., Gamez, W., Schmidt, F., & Watson, D. (2010) · Psychological Bulletin · doi:10.1037/a0020327 · Cited by 2,677 · cited 2×
“Specifically, decades of research and multiple meta-analyses (Kotov et al., 2010; Mal-ouff et al., 2005) demonstrate that neuroticism has substantial links to the mood and anxiety disorders, whereas extraversion, conscientiousness, and agreeableness tend to demonstrate weaker (negative)…”
BackgroundData collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242 · cited 2×
“…a variety of study designs and numerous types of data (Behrend et al., 2011; Buhrmester et al., 2011; Casler et al., 2013; Eriksson & Simpson, 2010; Goodman, Cryder, & Cheema, 2013; Horton, Rand, & Zeckhauser, 2011; Kees et al., 2017; Mason & Watts, 2009; Paolacci & Chandler, 2014; Shapiro,…”
BackgroundHandbook of Personality Theory and Research.
Schwartz, M., Borgatta, E. F., & Lambert, W. W. (1970) · American Sociological Review · doi:10.2307/2093310 · Cited by 2,163
Inside the Turk: Understanding Mechanical Turk as a Participant Pool
Paolacci, G., & Chandler, J. (2014) · Current Directions in Psychological Science · doi:10.1177/0963721414531598 · Cited by 2,136 · cited 2×
“…or superior in quality to data collected in the lab, from professional online panels, and using marketing research companies (Behrend, Sharek, Meade, & Wiebe, 2011; Buhrme-ster et al., 2011; Casler, Bickel, & Hackett, 2013; Kees, Berry, Burton, & Sheehan, 2017; Paolacci & Chandler, 2014).”
BackgroundCrowdsourcing user studies with Mechanical Turk
Kittur, A., Chi, E. H., & Suh, B. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357127 · Cited by 1,923 · cited 2×
“…validity indicators (Aust, Die-denhofen, Ullrich, & Musch, 2013; Barger et al., 2011; Kennedy, Clifford, Burleigh, Jewell, & Waggoner, 2018; Kittur et al., 2008; Mason & Suri, 2011; Wood et al., 2017; Zhu & Carterette, 2010) and recommend screening all responses before approving HITs so…”
BackgroundCasler, K., Bickel, L., & Hackett, E. (2013) · Computers in Human Behavior · doi:10.1016/j.chb.2013.05.009 · Cited by 1,662 · cited 3×
“…or superior in quality to data collected in the lab, from professional online panels, and using marketing research companies (Behrend, Sharek, Meade, & Wiebe, 2011; Buhrme-ster et al., 2011; Casler, Bickel, & Hackett, 2013; Kees, Berry, Burton, & Sheehan, 2017; Paolacci & Chandler, 2014).”
BackgroundMethodReputation as a sufficient condition for data quality on Amazon Mechanical Turk
Peer, E., Vosgerau, J., & Acquisti, A. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0434-y · Cited by 1,618 · cited 2×
“Nevertheless, replicating the current pattern of results in studies using higher HIT requirements (Peer et al., 2014) is important.”
BackgroundResultThe Online Laboratory: Conducting Experiments in a Real Labor Market∗Preprint
Zeckhauser, R. J. (2016) · Cited by 1,530
Using Mechanical Turk to Study Clinical Populations
Shapiro, D. N., Chandler, J., & Mueller, P. A. (2013) · Clinical Psychological Science · doi:10.1177/2167702612469015 · Cited by 1,139 · cited 1×
“…types of data (Behrend et al., 2011; Buhrmester et al., 2011; Casler et al., 2013; Eriksson & Simpson, 2010; Goodman, Cryder, & Cheema, 2013; Horton, Rand, & Zeckhauser, 2011; Kees et al., 2017; Mason & Watts, 2009; Paolacci & Chandler, 2014; Shapiro, Chandler, & Mueller, 2013; Suri & Watts, 2011).”
The viability of crowdsourcing for survey research
Behrend, T. S., Sharek, D. J., Meade, A. W., & Wiebe, E. N. (2011) · Behavior Research Methods · doi:10.3758/s13428-011-0081-0 · Cited by 1,018 · cited 3×
“In fact, these approaches are how researchers originally established MTurk’s high-quality data (Behrend et al., 2011; Buhrmester et al., 2011; Casler et al., 2013; Good-man et al., 2013; Kees et al., 2017; Mason & Watts, 2009; Sha-piro et al., 2013).”
BackgroundPlate tectonics in the classification of personality disorder: Shifting to a dimensional model.
Widiger, T. A., & Trull, T. J. (2007) · American Psychologist · doi:10.1037/0003-066x.62.2.71 · Cited by 957 · cited 1×
“…Inventory (BFI; John, Donahue, & Kentle, 1991) and test whether well-established associations with psychopathology replicate (Bagby et al., 1996; Clark & Watson, 1991; Kotov, Gamez, Schmidt, & Watson, 2010; Krueger & Tackett, 2006; Malouff, Thorsteinsson, & Schutte, 2005; Widiger & Trull, 2007).”
Berinsky, A. J., Margolis, M. F., & Sances, M. W. (2014) · American Journal of Political Science · doi:10.1111/ajps.12081 · Cited by 951 · cited 1×
“…(e.g., those lower in conscientiousness, individuals with lower cognitive ability, particular age groups, specific demographic groups; see Berinsky, Margolis, & Sances, 2014; Vannette, 2016) thereby biasing samples; this has led Qualtrics to recommend against their use (Vannette, 2017).”
BackgroundKees, J., Berry, C., Burton, S., & Sheehan, K. (2017) · Journal of Advertising · doi:10.1080/00913367.2016.1269304 · Cited by 902 · cited 3×
“…or superior in quality to data collected in the lab, from professional online panels, and using marketing research companies (Behrend, Sharek, Meade, & Wiebe, 2011; Buhrme-ster et al., 2011; Casler, Bickel, & Hackett, 2013; Kees, Berry, Burton, & Sheehan, 2017; Paolacci & Chandler, 2014).”
BackgroundAn Evaluation of Amazon’s Mechanical Turk, Its Rapid Rise, and Its Effective Use
Buhrmester, M. D., Talaifar, S., & Gosling, S. D. (2018) · Perspectives on Psychological Science · doi:10.1177/1745691617706516 · Cited by 767 · cited 2×
“Recent years have seen an exponential increase in psychological studies being conducted on MTurk (Buhrmester, Talaifar, & Gosling, 2018), likely because MTurk is an efficient method for collecting large amounts of data (Buhrmester, Kwang, & Gosling, 2011; Buhrmester et al., 2018; Stewart et al.,…”
MethodConstruct Validation in Social and Personality Research: Current Practice and Recommendations
Flake, J. K., Pek, J., & Hehman, E. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550617693063 · Cited by 697 · cited 1×
“This is critical as optimal measurement is essential for the continued advancement of science, and poor measurement has been suggested as one cause of the replication crisis (Chmielewski et al., 2016; Flake, Pek, & Hehman, 2017).”
BackgroundMalouff, J. M., Thorsteinsson, E. B., & Schutte, N. S. (2005) · Journal of Psychopathology and Behavioral Assessment · doi:10.1007/s10862-005-5384-y · Cited by 675 · cited 1×
“…Inventory (BFI; John, Donahue, & Kentle, 1991) and test whether well-established associations with psychopathology replicate (Bagby et al., 1996; Clark & Watson, 1991; Kotov, Gamez, Schmidt, & Watson, 2010; Krueger & Tackett, 2006; Malouff, Thorsteinsson, & Schutte, 2005; Widiger & Trull, 2007).”
The online laboratory: conducting experiments in a real labor market
Horton, J. J., Rand, D. G., & Zeckhauser, R. (2011) · Experimental Economics · doi:10.1007/s10683-011-9273-9 · Cited by 567 · cited 1×
“…types of data (Behrend et al., 2011; Buhrmester et al., 2011; Casler et al., 2013; Eriksson & Simpson, 2010; Goodman, Cryder, & Cheema, 2013; Horton, Rand, & Zeckhauser, 2011; Kees et al., 2017; Mason & Watts, 2009; Paolacci & Chandler, 2014; Shapiro, Chandler, & Mueller, 2013; Suri &…”
Seriousness checks are useful to improve data validity in online research
Aust, F., Diedenhofen, B., Ullrich, S., & Musch, J. (2013) · Behavior Research Methods · doi:10.3758/s13428-012-0265-2 · Cited by 524
Financial incentives and the "performance of crowds"
Mason, W., & Watts, D. J. (2009) · Proceedings of the ACM SIGKDD Workshop on Human Computation · doi:10.1145/1600150.1600175 · Cited by 520
The shape of and solutions to the MTurk quality crisis
Kennedy, R., Clifford, S., Burleigh, T., Waggoner, P. D., Jewell, R., & Winter, N. J. G. (2020) · Political Science Research and Methods · doi:10.1017/psrm.2020.6 · Cited by 506 · cited 1×
“Even if the current crisis passes, similar issues may arise again (Kennedy et al., 2018).”
BackgroundSoto, C. J., John, O. P., Gosling, S. D., & Potter, J. (2008) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.94.4.718 · Cited by 498
Crowdsourcing research: Data collection with Amazon’s Mechanical Turk
Sheehan, K. B. (2018) · Communication Monographs · doi:10.1080/03637751.2017.1342043 · Cited by 408 · cited 2×
“…or superior in quality to data collected in the lab, from professional online panels, and using marketing research companies (Behrend, Sharek, Meade, & Wiebe, 2011; Buhrme-ster et al., 2011; Casler, Bickel, & Hackett, 2013; Kees, Berry, Burton, & Sheehan, 2017; Paolacci & Chandler, 2014).”
BackgroundCooperation and Contagion in Web-Based, Networked Public Goods Experiments
Suri, S., & Watts, D. J. (2011) · PLoS ONE · doi:10.1371/journal.pone.0016836 · Cited by 354
The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers
Stewart, N., Ungemach, C., Harris, A. J. L., Bartels, D. M., Newell, B. R., Paolacci, G., & Chandler, J. (2015) · Judgment and Decision Making · doi:10.1017/s1930297500005611 · Cited by 304 · cited 1×
“…years have seen an exponential increase in psychological studies being conducted on MTurk (Buhrmester, Talaifar, & Gosling, 2018), likely because MTurk is an efficient method for collecting large amounts of data (Buhrmester, Kwang, & Gosling, 2011; Buhrmester et al., 2018; Stewart et al., 2015).”
MethodLitman, L., Robinson, J., & Rosenzweig, C. (2015) · Behavior Research Methods · doi:10.3758/s13428-014-0483-x · Cited by 275 · cited 1×
“The internal consistency reliability of the BFI in MTurk samples is generally in the mid .80s (e.g., mean a ¼ .83 to .87; Chmielewski, Sala, Tang, & Baldwin, 2016; Litman, Robinson, & Rosenzweig, 2015).”
BackgroundThe Industrial-Organizational Psychologist: (578882011-001)
(2009) · PsycEXTRA Dataset · doi:10.1037/e578882011-001 · Cited by 274
Wood, D., Harms, P. D., Lowman, G. H., & DeSimone, J. A. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550617703168 · Cited by 243 · cited 5×
“In addition to the validity indicators included, recent research indicates that response time in seconds per item (SPI) and profile correlations are important validity indicators (Wood et al., 2017); SPI in particular is easily implemented and places no burden on participants.”
BackgroundMethodInfluentialUsing the Amazon Mechanical Turk for transcription of spoken language
Marge, M., Banerjee, S., & Rudnicky, A. I. (2010) · 2010 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2010.5494979 · Cited by 224 · cited 1×
“Furthermore, compensation level appears to have no impact on data quality (Buhrmester et al., 2011; Marge, Banerjee, & Rudnicky, 2010; Mason & Watts, 2009).”
BackgroundEmotional reactions to losing explain gender differences in entering a risky lottery
Eriksson, K., & Simpson, B. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500001030 · Cited by 172 · cited 1×
“…been demonstrated across a variety of study designs and numerous types of data (Behrend et al., 2011; Buhrmester et al., 2011; Casler et al., 2013; Eriksson & Simpson, 2010; Goodman, Cryder, & Cheema, 2013; Horton, Rand, & Zeckhauser, 2011; Kees et al., 2017; Mason & Watts, 2009; Paolacci &…”
BackgroundIt’s a Trap! Instructional Manipulation Checks Prompt Systematic Thinking on “Tricky” Tasks
Hauser, D. J., & Schwarz, N. (2015) · Sage Open · doi:10.1177/2158244015584617 · Cited by 168 · cited 1×
“Similarly, instructional manipulation checks (IMCs) that were previously recommended as a way to ensure high-quality data (see Oppenheimer, Meyvis, & Davidenko, 2009) have been found to alter participants’ responses in potentially problematic ways (see Hauser & Schwarz, 2015).”
BackgroundMethod matters: Understanding diagnostic reliability in DSM-IV and DSM-5.
Chmielewski, M., Clark, L. A., Bagby, R. M., & Watson, D. (2015) · Journal of Abnormal Psychology · doi:10.1037/abn0000069 · Cited by 161 · cited 1×
“…should also report the psychometric properties of the measures in the studied sample and compare them to previous research, when available, as this provides valuable information about the performance of the measure and the quality of data itself (Chmielewski, Clark, Bagby, & Watson, 2015).”
Self-citationThe impact of acquiescence on the evaluation of personality structure.
Rammstedt, B., & Farmer, R. F. (2013) · Psychological Assessment · doi:10.1037/a0033323 · Cited by 124
Bipolar disorder, unipolar depression and the Five-Factor Model of personality
Bagby, R. M., Young, L. T., Schuller, D. R., Bindseil, K. D., Cooke, R. G., Dickens, S. E., LEVITT, A. J., & Joffe, R. T. (1996) · Journal of Affective Disorders · doi:10.1016/0165-0327(96)00060-2 · Cited by 116 · cited 2×
“…of the Big Five Inventory (BFI; John, Donahue, & Kentle, 1991) and test whether well-established associations with psychopathology replicate (Bagby et al., 1996; Clark & Watson, 1991; Kotov, Gamez, Schmidt, & Watson, 2010; Krueger & Tackett, 2006; Malouff, Thorsteinsson, & Schutte, 2005;…”
Personality and Psychopathology
(1999) · American Psychiatric Association Publishing eBooks · doi:10.1176/appi.books.9781615376322 · Cited by 102
The Shape of and Solutions to the MTurk Quality CrisisPreprint
Kennedy, R., Clifford, S., Burleigh, T. J., Jewell, R., & Waggoner, P. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3272468 · Cited by 99
How serious is the ‘carelessness’ problem on Mechanical Turk?
Aruguete, M. S., Huynh, H., Browne, B. L., Jurs, B., Flint, E., & McCutcheon, L. E. (2019) · International Journal of Social Research Methodology · doi:10.1080/13645579.2018.1563966 · Cited by 89 · cited 2×
“However, the percentage of participants failing validity indicators in Waves 3 and 4 is similar to informal reports and recent publications (Aruguete et al., 2019; Courrégé, Skeel, Feder, & Boress, 2019; Dreyfuss, 2018).”
ResultDennis, S. A., Goodson, B. M., & Pearson, C. A. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3233954 · Cited by 78
TaskMaster: A Tool for Determining When Subjects Are on Task
Permut, S., Fisher, M., & Oppenheimer, D. M. (2019) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919838479 · Cited by 55 · cited 1×
“…tracking geolocations/IP addresses, incentivizing participants providing high-quality data (Barger et al., 2011), programs tracking whether participants are “on task” (Permut, Fisher, & Oppenheimer, 2019), and other online sample sources (i.e., Prolific) may offer additional tools for researchers.”
BackgroundExamining the construct validity of affective judgments of physical activity measures.
Chmielewski, M. S., Sala, M., Tang, R., & Baldwin, A. S. (2016) · Psychological Assessment · doi:10.1037/pas0000322 · Cited by 40 · cited 2×
“This is critical as optimal measurement is essential for the continued advancement of science, and poor measurement has been suggested as one cause of the replication crisis (Chmielewski et al., 2016; Flake, Pek, & Hehman, 2017).”
BackgroundSelf-citationCrowdsourcing a News Query Classification Dataset
McCreadie, R., Macdonald, C., & Ounis, I. (2010) · Cited by 32
Taking parent personality and child temperament into account in child language development
Kucker, S. C., Zimmerman, C., & Chmielewski, M. S. (2021) · British Journal of Developmental Psychology · doi:10.1111/bjdp.12379 · Cited by 32
Self-citationThe ADHD Symptom Infrequency Scale (ASIS): A novel measure designed to detect adult ADHD simulators.
Courrege, S. C., Skeel, R. L., Feder, A. H., & Boress, K. S. J. (2019) · Psychological Assessment · doi:10.1037/pas0000706 · Cited by 17 · cited 1×
“However, the percentage of participants failing validity indicators in Waves 3 and 4 is similar to informal reports and recent publications (Aruguete et al., 2019; Courre´ge´, Skeel, Feder, & Boress, 2019; Dreyfuss, 2018).”
Result
Research works printed but not linked: 5
- Barger P., 2011, The Industrial-Organizational Psychologist, 49, 11
- John O. P., 1991, The Big Five Inventory—versions 4a and 54
- Sylaska K., 2019, It’s 2019: Do we need super attention check items to conduct web-based survey research? The evolution of MTurk survey respondents
- Vannette D. L., 2016, Paper presented at the 71st Annual Conference of the American Association for Public Opinion Research
- Zhu D., Carterette B. (2010). An analysis of assessor behavior in crowdsourced preference judgments. Presented at the SIGIR 2010 Workshop on Crowdsourcing for Search Evaluations, Geneva, Switzerland, 17–20.
Cited by
Douglas, B. D., Ewell, P. J., & Brauer, M. (2023) · PLOS ONE · doi:10.1371/journal.pone.0279720 · Cited by 1,532
“, Cronbach’s alphas and internal validity between scales) [2].”
BackgroundSharma, M., Tong, M., Korbak, T., Duperrouzel, D., Was, H., Korbak, J., Cheng, M., Askell, A., Bai, Y., Brain, C., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.13548 · Cited by 1,313
“Human evaluators are imperfect (Saunders et al., 2022; Gudibande et al., 2023), make mistakes e.g., due to limited time (Chmielewski & Kucker, 2020) or cognitive biases (Pandey et al., 2022), and sometimes have diverse, contradictory preferences (Bakker et al., 2022).”
ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks
Gilardi, F., Alizadeh, M., & Kubli, M. (2023) · arXiv · doi:10.48550/arxiv.2303.15056 · Cited by 1,207
“Moreover, there have been concerns that MTurk data quality has decreased (Chmielewski and Kucker, 2020), while alternative platforms such as CrowdFlower and FigureEight are no longer practicable options for academic research since they were acquired by Appen, a company that is focused on a business…”
BackgroundChatGPT outperforms crowd workers for text-annotation tasks
Gilardi, F., Alizadeh, M., & Kubli, M. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2305016120 · Cited by 1,197
“Moreover, there have been concerns that MTurk data quality has decreased (Chmielewski and Kucker, 2020), while alternative platforms such as CrowdFlower and FigureEight are no longer practicable options for academic research since they were acquired by Appen, a company that is focused on a business…”
BackgroundDesigning Information Provision Experiments
Haaland, I., Roth, C., & Wohlfart, J. (2023) · Journal of Economic Literature · doi:10.1257/jel.20211658 · Cited by 541
Metadata
- Authors
- Chmielewski, M., & Kucker, S. C.
- Affiliations
- Southern Methodist University (US)
- Oklahoma State University (US)
- Corresponding author
- Chmielewski, Michael
- Venue
- Social Psychological and Personality Science
- Publisher
- SAGE Publications
- Published
- 2019-10-10
- Volume
- 11
- Issue
- 4
- Pages
- 464-473
- ISSN
- 1948-5506 (ISSN-L)
- 1948-5514
- OpenAlex
- W2979864122
- Cited by
- 1,027
- References
- 46
- Citations by year
- 2026: 89 · 2025: 129 · 2024: 176 · 2023: 206 · 2022: 184 · 2021: 181 · 2020: 57 · 2019: 3
- FWCI
- 115.20
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Social Psychological and Personality Science (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- Amazon Mechanical Turk, data quality, response validity, personality measures, reliability statistics, replication, online labour platforms, participant screening, naturalistic experiment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.