Reliability and validity of data obtained from alcohol, cannabis, and gambling populations on Amazon’s Mechanical Turk.
The DOI matches its Crossref record.
Abstract
Researchers recently have begun using Mechanical Turk (MTurk), an online crowdsourcing platform, to recruit addiction populations. However, whether the data obtained from substance users and gamblers on MTurk are reliable and valid is unknown. Herein, we assessed the internal and retest reliability of and concurrent and convergent validity of data obtained from addiction populations on MTurk. Current drinkers (N = 208), cannabis users (N = 200), and gamblers (N = 200) residing in the United States completed measures of alcohol, cannabis, and gambling severity, psychological constructs (e.g., impulsivity) related to addictions, overt and subtle measures of valid responding, and motivations for completing MTurk studies. Of the original sample, 88-92% of participants who provided informed consent for recontact completed a reassessment 1 week later. The internal consistency of the addiction severity measures ranged from α = .75 to .93. The stability over 1 week ranged from κ = .57 to .70 for categorical classification, and intraclass correlation coefficient (ICC) = .71 to .86 for continuous measures. The addiction measures were significantly correlated with each other and with other constructs related to addictive behaviors. Overall, 80-85% of participants provided valid responses. They reported attending and answering questions honestly, with financial motives being the most frequently endorsed motivation. After invalid responses were excluded, results remained the same for alcohol and gambling, but significant differences emerged for the cannabis sample. The results suggest that the self-report data obtained from alcohol and gambling populations are of high quality, however, caution is warranted with cannabis populations. MTurk shows promise as a recruitment tool for some addictive behaviors. (PsycINFO Database Record
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Mturk is an online survey management system that has been found to yield valid and representative data (Kim & Hodgins, 2017).”
From Evaluating Experiential Avoidance in terms of COVID-19 Fear and Pandemic Emotional Distress Symptoms Among Latinx Adults (2022) · cites this paper as Kim & Hodgins, 2017
Citations
Cites
Still being checked.
Cited by
Mortensen, K., & Hughes, T. L. (2018) · Journal of General Internal Medicine · doi:10.1007/s11606-017-4246-0 · Cited by 441
Wardell, J. D., Kempe, T., Rapinda, K. K., Single, A., Bilevicius, E., Frohlich, J. R., Hendershot, C. S., & Keough, M. T. (2020) · Alcoholism: Clinical and Experimental Research · doi:10.1111/acer.14425 · Cited by 288
The use of crowdsourcing in addiction science research: Amazon Mechanical Turk.
Strickland, J. C., & Stoops, W. W. (2019) · Experimental and Clinical Psychopharmacology · doi:10.1037/pha0000235 · Cited by 250
The relationship of loot box purchases to problem video gaming and problem gambling
Li, W., Mills, D., & Nower, L. (2019) · Addictive Behaviors · doi:10.1016/j.addbeh.2019.05.016 · Cited by 205
McPhee, M. D., Keough, M. T., Rundle, S. M., Heath, L. M., Wardell, J. D., & Hendershot, C. S. (2020) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2020.574676 · Cited by 154
Metadata
- Authors
- Kim, H. S., & Hodgins, D. C.
- Affiliations
- University of Calgary (CA)
- Venue
- Psychology of Addictive Behaviors
- Published
- 2017-02
- Volume
- 31
- Issue
- 1
- Pages
- 85-94
- ISSN
- 0893-164X (ISSN-L)
- 1939-1501
- OpenAlex
- W2558243715
- Cited by
- 206
- References
- 0
- Citations by year
- 2026: 2 · 2025: 10 · 2024: 15 · 2023: 25 · 2022: 25 · 2021: 29 · 2020: 30 · 2019: 31 · 2018: 23 · 2017: 13
- FWCI
- 30.54
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychology of Addictive Behaviors (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Alberta Gambling Research Institute
- Social Sciences and Humanities Research Council of Canada
- Alberta Innovates Health Solutions
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Health Sciences › Medicine › Epidemiology › Substance Abuse Treatment and Outcomes
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- Amazon Mechanical Turk, addiction research, alcohol consumption, cannabis use, gambling, reliability statistics, validity, test-retest reliability, concurrent validity, convergent validity, self-report validity, response validity, economic motivation, impulsivity, addiction severity
- MeSH terms
- Adult, Marijuana Abuse, Data Collection, Female, Gambling, Humans, Male, United States, Reproducibility of Results, Behavior, Addictive, Alcohol-Related Disorders, Internet, Young Adult
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.