Mechanical Turk data collection in addiction research: utility, concerns and best practices
Open accessThe DOI matches its Crossref record; 47 of the 47 references it cites were found in a registry or an index.
Abstract
Aims: Amazon Mechanical Turk (MTurk) provides a crowdsourcing platform for the engagement of potential research participants with data collection instruments. This review (1) provides an introduction to the mechanics and validity of MTurk research; (2) gives examples of MTurk research; and (3) discusses current limitations and best practices in MTurk research. Methods: We review four use cases of MTurk for research relevant to addictions: (1) the development of novel measures, (2) testing interventions, (3) the collection of longitudinal use data to determine the feasibility of longer‐term studies of substance use and (4) the completion of large batteries of assessments to characterize the relationships between measured constructs. We review concerns with the platform, ways of mitigating these and important information to include when presenting findings. Results: MTurk has proved to be a useful source of data for behavioral science more broadly, with specific applications to addiction science. However, it is still not appropriate for all use cases, such as population‐level inference. To live up to the potential of highly transparent, reproducible science from MTurk, researchers should clearly report inclusion/exclusion criteria, data quality checks and reasons for excluding collected data, how and when data were collected and both targeted and actual participant compensation. Conclusions: Although on‐line survey research is not a substitute for random sampling or clinical recruitment, the Mechanical Turk community of both participants and researchers has developed multiple tools to promote data quality, fairness and rigor. Overall, Mechanical Turk has provided a useful source of convenience samples despite its limitations and has demonstrated utility in the engagement of relevant groups for addiction science.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996
Prolific.ac—A subject pool for online experiments
Palan, S., & Schitter, C. (2018) · Journal of Behavioral and Experimental Finance · doi:10.1016/j.jbef.2017.12.004 · Cited by 3,450 · cited 2×
“As new methods and norms in crowdsourcing platforms emerge (e.g., the rating of requesters based on pay, rejection rates, and responsiveness, as on Turker Nation) and competitors innovate with additional tools to support research (e.g., by screening for naivete to experimental manipulations across tasks, as Prolific A…”
BackgroundConducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172
Beyond the Turk: Alternative platforms for crowdsourcing behavioral research
Peer, E., Brandimarte, L., Samat, S., & Acquisti, A. (2017) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2017.01.006 · Cited by 2,957 · cited 1×
“Indeed, despite the existence of multiple alternatives to MTurk (including comparable crowdsourcing websites such as CrowdFlower, and research-focused participant pools such as Prolific Academic (5); for a more complete review of online crowdsourcing alternatives to MTurk, see (6)), MTurk remains the most highly used…”
BackgroundData collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242 · cited 2×
“This could theoretically allow for fully open source research, in which the complete code for both experimental procedures and analysis are made available for replication (as in (25)).”
BackgroundHauser, D. J., & Schwarz, N. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0578-z · Cited by 1,739 · cited 1×
“Some additional research has specifically explored differences in survey completion strategy between MTurk and other samples, revealing possibly higher rates of survey satisficing among online, forpay survey completers (16), and sensitivity of MTurk participants to slight changes in instructions (17).”
BackgroundCasler, K., Bickel, L., & Hackett, E. (2013) · Computers in Human Behavior · doi:10.1016/j.chb.2013.05.009 · Cited by 1,662 · cited 1×
“Multiple studies have sought to replicate a battery of psychological assessments in both labbased and online samples, including samples of unpaid volunteers (7) and in comparisons between computer-based tasks completed on MTurk and purely behavioral tasks (8).”
BackgroundReputation as a sufficient condition for data quality on Amazon Mechanical Turk
Peer, E., Vosgerau, J., & Acquisti, A. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0434-y · Cited by 1,618 · cited 16×
“These include both the use of checks to ensure that participants understand task instructions clearly (9), and use of an inclusion criterion based on Worker’s “reputation,” or total percentage of approved HITs (39), in addition to other attention check questions (40) and task-specific data quality assessments (e.”
BackgroundInfluentialEvaluating amazon’s mechanical turk as a tool for experimental behavioral research
Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0057410 · Cited by 1,617 · cited 2×
“Overall, task performance was comparable in the online sample as in previously published results, especially after the introduction of instruction comprehension checks (9).”
BackgroundResultAn MTurk Crisis? Shifts in Data Quality and the Impact on Study Results
Chmielewski, M., & Kucker, S. C. (2020) · Social Psychological and Personality Science · doi:10.1177/1948550619875149 · Cited by 1,027 · cited 1×
“Specifically, recent concerns have been articulated about rates of low-quality data and nonsensical, distracted, or fraudulent responding, which may be increasing since 2018 (19).”
BackgroundDelayed reward discounting and addictive behavior: a meta-analysis
MacKillop, J., Amlung, M. T., Few, L. R., Ray, L. A., Sweet, L. H., & Munafò, M. R. (2011) · Psychopharmacology · doi:10.1007/s00213-011-2229-0 · Cited by 1,021 · cited 1×
“Specific findings related to discounting rates and addiction, including observation of steeper delay discounting among individuals with substance use disorders compared to those without, have also been replicated and demonstrated in meta-analyses (12,13).”
ResultCoppock, A., & McClellan, O. A. (2019) · Research & Politics · doi:10.1177/2053168018822174 · Cited by 987 · cited 1×
“However, by and large, a replication-based approach has indicated that results observed on MTurk can generalize to other laboratory populations (18).”
ResultBerinsky, A. J., Margolis, M. F., & Sances, M. W. (2014) · American Journal of Political Science · doi:10.1111/ajps.12081 · Cited by 951 · cited 1×
“These include both the use of checks to ensure that participants understand task instructions clearly (9), and use of an inclusion criterion based on Worker’s “reputation,” or total percentage of approved HITs (39), in addition to other attention check questions (40) and task-specific data quality assessments (e.”
BackgroundChandler, J., Mueller, P., & Paolacci, G. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0365-7 · Cited by 950
Huff, C., & Tingley, D. (2015) · Research & Politics · doi:10.1177/2053168015604648 · Cited by 903 · cited 1×
“Within the US, MTurk workers are also more likely to be politically liberal (26).”
BackgroundTurkopticon: interrupting worker invisibility in amazon mechanical turk
Irani, L. C., & Silberman, M. S. (2013) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2470742 · Cited by 798
Rand, D. G. (2012) · Journal of Theoretical Biology · doi:10.1016/j.jtbi.2011.03.004 · Cited by 683 · cited 1×
“This review (1) provides an introduction to the mechanics and validity of MTurk research; (2) gives examples of MTurk research; and (3) discusses current limitations and best practices in MTurk research.”
BackgroundGermine, L., Nakayama, K., Duchaine, B. C., Chabris, C. F., Chatterjee, G., & Wilmer, J. B. (2012) · Psychonomic Bulletin & Review · doi:10.3758/s13423-012-0296-9 · Cited by 669 · cited 1×
“Multiple studies have sought to replicate a battery of psychological assessments in both labbased and online samples, including samples of unpaid volunteers (7) and in comparisons between computer-based tasks completed on MTurk and purely behavioral tasks (8).”
BackgroundOnline panels in social science research: Expanding sampling methods beyond Mechanical Turk
Chandler, J., Rosenzweig, C., Moss, A. J., Robinson, J., & Litman, L. (2019) · Behavior Research Methods · doi:10.3758/s13428-019-01273-7 · Cited by 668
Amlung, M., Vedelago, L., Acker, J., Balodis, I., & MacKillop, J. (2017) · Addiction · doi:10.1111/add.13535 · Cited by 626
An algorithm for identifying nonsystematic delay-discounting data.
Johnson, M. W., & Bickel, W. K. (2008) · Experimental and Clinical Psychopharmacology · doi:10.1037/1064-1297.16.3.264 · Cited by 555
Self-citationThe shape of and solutions to the MTurk quality crisis
Kennedy, R., Clifford, S., Burleigh, T., Waggoner, P. D., Jewell, R., & Winter, N. J. G. (2020) · Political Science Research and Methods · doi:10.1017/psrm.2020.6 · Cited by 506
Demographics and Dynamics of Mechanical Turk Workers
Difallah, D., Filatova, E., & Ipeirotis, P. (2018) · Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining · doi:10.1145/3159652.3159661 · Cited by 501 · cited 1×
“However, at any given time the estimated population available is under 2,500 workers (22).”
BackgroundA Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk
Hara, K., Adams, A., Milland, K., Savage, S., Callison-Burch, C., & Bigham, J. P. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174023 · Cited by 497
Large-scale analysis of test–retest reliabilities of self-regulation measures
Enkavi, A. Z., Eisenberg, I. W., Bissett, P. G., Mazza, G. L., MacKinnon, D. P., Marsch, L. A., & Poldrack, R. A. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1818430116 · Cited by 489 · cited 2×
“Analysis of open-source data has also been critical to identifying common patterns used among participants employing VPSs to circumvent location restrictions, and helped to address scientific questions of both theoretical and methodological import (35) Taken together, the entirely-online MTurk environment means that b…”
BackgroundMortensen, K., & Hughes, T. L. (2018) · Journal of General Internal Medicine · doi:10.1007/s11606-017-4246-0 · Cited by 441 · cited 1×
“Overall, a review of 35 manuscripts comparing performance between MTurk and other samples identified that responses are comparable across populations (15).”
BackgroundThe average laboratory samples a population of 7,300 Amazon Mechanical Turk workers
Stewart, N., Ungemach, C., Harris, A. J. L., Bartels, D. M., Newell, B. R., Paolacci, G., & Chandler, J. (2015) · Judgment and Decision Making · doi:10.1017/s1930297500005611 · Cited by 304 · cited 4×
“Posting research instruments as single, large batches of HITs may increase the effective population size available, reducing the probability of “repeat participants” in research where naivete is an important consideration (23).”
BackgroundInfluentialIdentification and management of nonsystematic purchase task data: Toward best practice.
Stein, J. S., Koffarnus, M. N., Snider, S. E., Quisenberry, A. J., & Bickel, W. K. (2015) · Experimental and Clinical Psychopharmacology · doi:10.1037/pha0000020 · Cited by 281
Self-citationUnstuck in time: episodic future thinking reduces delay discounting and cigarette smoking
Stein, J. S., Wilson, A. G., Koffarnus, M. N., Daniel, T. O., Epstein, L. H., & Bickel, W. K. (2016) · Psychopharmacology · doi:10.1007/s00213-016-4410-y · Cited by 267 · cited 1×
“Additionally, compared to a control condition, episodic future thinking reduced delay discounting and self-administration of cigarettes (32); see Figure 2, top panel.”
BackgroundSelf-citationThe use of crowdsourcing in addiction science research: Amazon Mechanical Turk.
Strickland, J. C., & Stoops, W. W. (2019) · Experimental and Clinical Psychopharmacology · doi:10.1037/pha0000235 · Cited by 250 · cited 1×
“Much ink has been spilled reviewing the use of MTurk in academic research (1–3), including in-depth review with specific applications to addiction science (4).”
BackgroundEpisodic future thinking: expansion of the temporal window in individuals with alcohol dependence
Snider, S. E., LaConte, S. M., & Bickel, W. K. (2016) · Alcoholism: Clinical and Experimental Research · doi:10.1111/acer.13112 · Cited by 241 · cited 1×
“Episodic future thinking has been shown to reduce discounting and valuation of alcohol (31).”
BackgroundSelf-citationRobinson, J., Rosenzweig, C., Moss, A. J., & Litman, L. (2019) · PLOS ONE · doi:10.1371/journal.pone.0226394 · Cited by 205
Stein, J. S., Tegge, A. N., Turner, J. K., & Bickel, W. K. (2018) · Journal of Behavioral Medicine · doi:10.1007/s10865-017-9908-1 · Cited by 159 · cited 1×
“In an extension of this work, Stein and colleagues (24) examined the effects of episodic future thinking in an MTurk sample of 117 cigarette smokers, all of whom passed internal consistency measures serving as data quality checks.”
BackgroundSelf-citationDupuis, M., Meier, E., & Cuneo, F. (2018) · Behavior Research Methods · doi:10.3758/s13428-018-1103-y · Cited by 146
McInnis, B., Cosley, D., Nam, C., & Leshed, G. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858539 · Cited by 119
Opportunity costs of reward delays and the discounting of hypothetical money and cigarettes
Johnson, P. S., Herrmann, E. S., & Johnson, M. W. (2015) · Journal of the Experimental Analysis of Behavior · doi:10.1002/jeab.110 · Cited by 113 · cited 1×
“Johnson and colleagues (14) attempted to replicate six phenomena previously observed in delay discounting research.”
BackgroundThe Shape of and Solutions to the MTurk Quality CrisisPreprint
Kennedy, R., Clifford, S., Burleigh, T. J., Jewell, R., & Waggoner, P. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3272468 · Cited by 99
Gleibs, I. H. (2017) · Behavior Research Methods · doi:10.3758/s13428-016-0789-y · Cited by 93
Hamby, T., & Taylor, W. (2016) · Educational and Psychological Measurement · doi:10.1177/0013164415627349 · Cited by 81
Toward Narrative Theory: Interventions for Reinforcer Pathology in Health Behavior
Bickel, W. K., Stein, J. S., Moody, L. N., Snider, S. E., Mellis, A. M., & Quisenberry, A. J. (2017) · Nebraska Symposium on Motivation/The Nebraska symposium on motivation · doi:10.1007/978-3-319-51721-6_8 · Cited by 76 · cited 1×
“Narrative interventions for addiction (30) may be particularly well-suited to development via crowdsourcing.”
BackgroundSelf-citationApplying novel technologies and methods to inform the ontology of self-regulation
Eisenberg, I. W., Bissett, P. G., Canning, J. R., Dallery, J., Enkavi, A. Z., Whitfield-Gabrieli, S., Gonzalez, O., Green, A. I., Greene, M. A., Kiernan, M., et al. (2018) · Behaviour Research and Therapy · doi:10.1016/j.brat.2017.09.014 · Cited by 61 · cited 1×
“In an ambitious research project examining underlying features defining the construct of self-control, participants have been recruited from MTurk to engage in lengthy batteries of self-control measures (34).”
BackgroundDetecting Fraud in Online Surveys by Tracing, Scoring, and Visualizing IP Addresses
Waggoner, P., Kennedy, R., & Clifford, S. (2019) · The Journal of Open Source Software · doi:10.21105/joss.01285 · Cited by 59
Strickland, J. C., & Stoops, W. W. (2018) · Journal of the Experimental Analysis of Behavior · doi:10.1002/jeab.445 · Cited by 55 · cited 2×
“Participants must be fairly compensated, and recent manuscripts have demonstrated model behaviors in reporting both planned and actual hourly compensation rates (33).”
BackgroundA Simplified Protocol to Screen Out VPS and International Respondents Using QualtricsPreprint
Winter, N., Burleigh, T. J., Kennedy, R., & Clifford, S. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3327274 · Cited by 54
Studying Cannabis Use Behaviors With Facebook and Web Surveys: Methods and Insights
Borodovsky, J. T., Marsch, L. A., & Budney, A. J. (2018) · JMIR Public Health and Surveillance · doi:10.2196/publichealth.9408 · Cited by 54 · cited 1×
“However, MTurk has potential (as does other online, social-media-based recruitment (27)) to offer a nonprobability-based method for oversampling participants with particular conditions relevant to addiction, such as heavy drinkers or individuals reporting chronic pain--similar to flyer-based community recruitment of i…”
BackgroundCunningham, J. A., Godinho, A., & Kushnir, V. (2017) · Internet Interventions · doi:10.1016/j.invent.2017.08.005 · Cited by 47
Tompkins, D. A., Huhn, A. S., Johnson, P. S., Smith, M. T., Strain, E. C., Edwards, R. R., & Johnson, M. W. (2017) · Addiction · doi:10.1111/add.13922 · Cited by 13 · cited 1×
“The external validity of this measure was demonstrated, identifying that individuals at the highest risk for opioid misuse using a standard clinical measure were also the most likely to take a novel analgesic with high potential for addiction (29).”
Background
Cited by
Why the COVID-19 pandemic is a traumatic stressor
Bridgland, V. M. E., Moeck, E. K., Green, D. M., Swain, T. L., Nayda, D. M., Matson, L. A., Hutchison, N. P., & Takarangi, M. K. T. (2021) · PLOS ONE · doi:10.1371/journal.pone.0240146 · Cited by 501
Rundle, S. M., Cunningham, J. A., & Hendershot, C. S. (2021) · Drug and Alcohol Review · doi:10.1111/dar.13244 · Cited by 61
Metadata
- Authors
- Mellis, A. M., & Bickel, W. K.
- Venue
- Addiction
- Publisher
- Wiley
- Published
- 2020-03-24
- Volume
- 115
- Issue
- 10
- Pages
- 1960-1968
- ISSN
- 0965-2140 (ISSN-L)
- 1360-0443
- OpenAlex
- W3010009477
- Cited by
- 164
- References
- 47
- FWCI
- 25.71
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.