The DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Mechanical Turk data collection in addiction research: utility, concerns and best practices
Open accessAbstract
Aims: Amazon Mechanical Turk (MTurk) provides a crowdsourcing platform for the engagement of potential research participants with data collection instruments. This review (1) provides an introduction to the mechanics and validity of MTurk research; (2) gives examples of MTurk research; and (3) discusses current limitations and best practices in MTurk research. Methods: We review four use cases of MTurk for research relevant to addictions: (1) the development of novel measures, (2) testing interventions, (3) the collection of longitudinal use data to determine the feasibility of longer‐term studies of substance use and (4) the completion of large batteries of assessments to characterize the relationships between measured constructs. We review concerns with the platform, ways of mitigating these and important information to include when presenting findings. Results: MTurk has proved to be a useful source of data for behavioral science more broadly, with specific applications to addiction science. However, it is still not appropriate for all use cases, such as population‐level inference. To live up to the potential of highly transparent, reproducible science from MTurk, researchers should clearly report inclusion/exclusion criteria, data quality checks and reasons for excluding collected data, how and when data were collected and both targeted and actual participant compensation. Conclusions: Although on‐line survey research is not a substitute for random sampling or clinical recruitment, the Mechanical Turk community of both participants and researchers has developed multiple tools to promote data quality, fairness and rigor. Overall, Mechanical Turk has provided a useful source of convenience samples despite its limitations and has demonstrated utility in the engagement of relevant groups for addiction science.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Prolific.ac—A subject pool for online experiments
Palan, S., & Schitter, C. (2018) · Journal of Behavioral and Experimental Finance · doi:10.1016/j.jbef.2017.12.004 · Cited by 3,450
Conducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172
Beyond the Turk: Alternative platforms for crowdsourcing behavioral research
Peer, E., Brandimarte, L., Samat, S., & Acquisti, A. (2017) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2017.01.006 · Cited by 2,957
Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242 · cited 2×
“This could theoretically allow for fully open source research, in which the complete code for both experimental procedures and analysis are made available for replication (as in (25)).”
BackgroundHauser, D. J., & Schwarz, N. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0578-z · Cited by 1,739
Casler, K., Bickel, L., & Hackett, E. (2013) · Computers in Human Behavior · doi:10.1016/j.chb.2013.05.009 · Cited by 1,662 · cited 1×
“Multiple studies have sought to replicate a battery of psychological assessments in both labbased and online samples, including samples of unpaid volunteers (7) and in comparisons between computer-based tasks completed on MTurk and purely behavioral tasks (8).”
BackgroundReputation as a sufficient condition for data quality on Amazon Mechanical Turk
Peer, E., Vosgerau, J., & Acquisti, A. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0434-y · Cited by 1,618 · cited 16×
“These include both the use of checks to ensure that participants understand task instructions clearly (9), and use of an inclusion criterion based on Worker’s “reputation,” or total percentage of approved HITs (39), in addition to other attention check questions (40) and task-specific data quality assessments (e.”
BackgroundInfluentialEvaluating amazon’s mechanical turk as a tool for experimental behavioral research
Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0057410 · Cited by 1,617
An MTurk Crisis? Shifts in Data Quality and the Impact on Study Results
Chmielewski, M., & Kucker, S. C. (2020) · Social Psychological and Personality Science · doi:10.1177/1948550619875149 · Cited by 1,027
Delayed reward discounting and addictive behavior: a meta-analysis
MacKillop, J., Amlung, M. T., Few, L. R., Ray, L. A., Sweet, L. H., & Munafò, M. R. (2011) · Psychopharmacology · doi:10.1007/s00213-011-2229-0 · Cited by 1,021 · cited 1×
“Specific findings related to discounting rates and addiction, including observation of steeper delay discounting among individuals with substance use disorders compared to those without, have also been replicated and demonstrated in meta-analyses (12,13).”
Result
Cited by
Why the COVID-19 pandemic is a traumatic stressor
Bridgland, V. M. E., Moeck, E. K., Green, D. M., Swain, T. L., Nayda, D. M., Matson, L. A., Hutchison, N. P., & Takarangi, M. K. T. (2021) · PLOS ONE · doi:10.1371/journal.pone.0240146 · Cited by 501
Rundle, S. M., Cunningham, J. A., & Hendershot, C. S. (2021) · Drug and Alcohol Review · doi:10.1111/dar.13244 · Cited by 61
Metadata
- Authors
- Mellis, A. M., & Bickel, W. K.
- Venue
- Addiction
- Publisher
- Wiley
- Published
- 2020-03-24
- Volume
- 115
- Issue
- 10
- Pages
- 1960-1968
- ISSN
- 0965-2140 (ISSN-L)
- 1360-0443
- OpenAlex
- W3010009477
- Cited by
- 164
- References
- 23
- FWCI
- 25.71
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Mechanical Turk data collection in addiction research: utility, concerns and best practices. https://app.cochat.ai/papers/10.1111/add.15032
- The paper
- Mellis, A. M., & Bickel, W. K. (2020). Mechanical Turk data collection in addiction research: utility, concerns and best practices. Addiction. https://doi.org/10.1111/add.15032
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.