Do imposters threaten data quality? An examination of worker misrepresentation and downstream consequences in Amazon's Mechanical Turk workforce
The DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
Amazon's Mechanical Turk (MTurk) is fast becoming the most popular online research platform, and as such, it is crucial for researchers to recognize its advantages and shortcomings. Here, we focused on the issue of worker deception and examined the downstream consequences of demographic misrepresentation in MTurk. In two studies, we asked: Are we testing who we think we are testing? and Does demographic deception ultimately have an impact on data quality? We found that in the presence of explicit eligibility requirements, an alarmingly high proportion of our samples misrepresented themselves in order to qualify for the studies (55.8% in Study 1 and 21.8% in Study 2). We also found that the nature of the downstream consequences of demographic deception varied across studies. Specifically, thescope of the impact may rest with the relationship between the demographic variable of interest and the outcome measure. In sum, the impact of demographic deception on data quality is multi-faceted, and a fruitful avenue of future research is to identify additional motivating factors that may underlie such deception. Worker misrepresentation is prevalent when eligibility requirement is present.The impact of MTurk worker demographic deception is evaluated in two studies.Deception status does not impact worker attentiveness.Demographic deception threatens data quality and could result in faulty conclusions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767
Henrich, J., Heine, S. J., & Norenzayan, A. (2010) · Nature · doi:10.1038/466029a · Cited by 3,441
Beyond the Turk: Alternative platforms for crowdsourcing behavioral research
Peer, E., Brandimarte, L., Samat, S., & Acquisti, A. (2017) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2017.01.006 · Cited by 2,957
The Influence of a Sense of Time on Human Development
Carstensen, L. L. (2006) · Science · doi:10.1126/science.1127488 · Cited by 2,554
A checklist for testing measurement invariance
van de Schoot, R., Lugtig, P., & Hox, J. (2012) · European Journal of Developmental Psychology · doi:10.1080/17405629.2012.686740 · Cited by 2,064
Socioemotional selectivity theory and the regulation of emotion in the second half of life
Carstensen, L. L., Fung, H. H., & Charles, S. T. (2003) · Motivation and Emotion · doi:10.1023/a:1024569803230 · Cited by 1,978
Hauser, D. J., & Schwarz, N. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0578-z · Cited by 1,739
Casler, K., Bickel, L., & Hackett, E. (2013) · Computers in Human Behavior · doi:10.1016/j.chb.2013.05.009 · Cited by 1,662
Evaluating amazon’s mechanical turk as a tool for experimental behavioral research
Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0057410 · Cited by 1,617
Conducting Clinical Research Using Crowdsourced Convenience Samples
Chandler, J., & Shapiro, D. (2016) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-021815-093623 · Cited by 1,072
Cited by
Douglas, B. D., Ewell, P. J., & Brauer, M. (2023) · PLOS ONE · doi:10.1371/journal.pone.0279720 · Cited by 1,532
MTurk Research: Review and Recommendations
Aguinis, H., Villamor, I., & Ramani, R. S. (2021) · Journal of Management · doi:10.1177/0149206320969787 · Cited by 984
Data Collection via Online Platforms: Challenges and Recommendations for Future Research
Newman, A., Bavik, Y. L., Mount, M., & Shao, B. (2021) · Applied Psychology · doi:10.1111/apps.12302 · Cited by 306
Brühlmann, F., Petralito, S., Aeschbach, L. F., & Opwis, K. (2020) · Methods in Psychology · doi:10.1016/j.metip.2020.100022 · Cited by 168
“As a primary advantage, crowdsourcing platforms offer a more diverse population compared to typically homogenous samples from psychological studies (Kan and Drummey, 2018): In the case of MTurk, these workers are composed of a demographic containing more than 5000000 individuals from 190 countries (Paolacci and Chandl…”
BackgroundMethodInfluentialSorensen, K., & Johnson Jorgensen, J. (2019) · Social Sciences · doi:10.3390/socsci8090244 · Cited by 52
A four-dimensional model of Asian American stereotypes
Bu, W., & Borgida, E. (2021) · Group Processes & Intergroup Relations · doi:10.1177/1368430220936360 · Cited by 9
“We are aware of other critiques of MTurk as a source of research participants (e.g., Kan & Drummey, 2018; Matherly, 2019).”
Background
Metadata
- Authors
- Kan, I. P., & Drummey, A. B.
- Venue
- Computers in Human Behavior
- Publisher
- Elsevier BV
- Published
- 2018-06
- Volume
- 83
- Pages
- 243-253
- ISSN
- 0747-5632 (ISSN-L)
- OpenAlex
- W2793743208
- Cited by
- 58
- References
- 14
- FWCI
- 9.84
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.