CoChat

Reliability and validity of data obtained from alcohol, cannabis, and gambling populations on Amazon’s Mechanical Turk.

Kim, Hyoun S., Hodgins, David C. · 2017 · Psychology of Addictive Behaviors

The DOI matches its Crossref record.

Abstract

Researchers recently have begun using Mechanical Turk (MTurk), an online crowdsourcing platform, to recruit addiction populations. However, whether the data obtained from substance users and gamblers on MTurk are reliable and valid is unknown. Herein, we assessed the internal and retest reliability of and concurrent and convergent validity of data obtained from addiction populations on MTurk. Current drinkers (N = 208), cannabis users (N = 200), and gamblers (N = 200) residing in the United States completed measures of alcohol, cannabis, and gambling severity, psychological constructs (e.g., impulsivity) related to addictions, overt and subtle measures of valid responding, and motivations for completing MTurk studies. Of the original sample, 88-92% of participants who provided informed consent for recontact completed a reassessment 1 week later. The internal consistency of the addiction severity measures ranged from α = .75 to .93. The stability over 1 week ranged from κ = .57 to .70 for categorical classification, and intraclass correlation coefficient (ICC) = .71 to .86 for continuous measures. The addiction measures were significantly correlated with each other and with other constructs related to addictive behaviors. Overall, 80-85% of participants provided valid responses. They reported attending and answering questions honestly, with financial motives being the most frequently endorsed motivation. After invalid responses were excluded, results remained the same for alcohol and gambling, but significant differences emerged for the cannabis sample. The results suggest that the self-report data obtained from alcohol and gambling populations are of high quality, however, caution is warranted with cannabis populations. MTurk shows promise as a recruitment tool for some addictive behaviors. (PsycINFO Database Record

Overview

Integrity signals

Registry

Crossref · 10.1037/adb0000219

Cites

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Still being checked.

Cited by

See all 207

Metadata

Authors
Kim, H. S., & Hodgins, D. C.
Affiliations
University of Calgary (CA)
Venue
Psychology of Addictive Behaviors
Published
2017-02
Volume
31
Issue
1
Pages
85-94
ISSN
  • 0893-164X (ISSN-L)
  • 1939-1501
OpenAlex
W2558243715
Cited by
206
References
0
Citations by year
2026: 2 · 2025: 10 · 2024: 15 · 2023: 25 · 2022: 25 · 2021: 29 · 2020: 30 · 2019: 31 · 2018: 23 · 2017: 13
FWCI
30.54
Type
journal article
Language
en
Open access
closed
Locations
  • Psychology of Addictive Behaviors (publishedVersion)
  • PubMed (publishedVersion)
Funders
  • Alberta Gambling Research Institute
  • Social Sciences and Humanities Research Council of Canada
  • Alberta Innovates Health Solutions
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
  • Health Sciences › Medicine › Epidemiology › Substance Abuse Treatment and Outcomes
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
Keywords
Amazon Mechanical Turk, addiction research, alcohol consumption, cannabis use, gambling, reliability statistics, validity, test-retest reliability, concurrent validity, convergent validity, self-report validity, response validity, economic motivation, impulsivity, addiction severity
MeSH terms
Adult, Marijuana Abuse, Data Collection, Female, Gambling, Humans, Male, United States, Reproducibility of Results, Behavior, Addictive, Alcohol-Related Disorders, Internet, Young Adult

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Reliability and validity of data obtained from alcohol, cannabis, and gambling populations on Amazon’s Mechanical Turk. https://app.cochat.ai/papers/10.1037/adb0000219
The paper
Kim, H. S., & Hodgins, D. C. (2017). Reliability and validity of data obtained from alcohol, cannabis, and gambling populations on Amazon’s Mechanical Turk. Psychology of Addictive Behaviors. https://doi.org/10.1037/adb0000219

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.