The DOI matches its Crossref record; 174 of the 177 references it cites were found in a registry or an index.
Conducting Clinical Research Using Crowdsourced Convenience Samples
Abstract
Crowdsourcing has had a dramatic impact on the speed and scale at which scientific research can be conducted. Clinical scientists have particularly benefited from readily available research study participants and streamlined recruiting and payment systems afforded by Amazon Mechanical Turk (MTurk), a popular labor market for crowdsourcing workers. MTurk has been used in this capacity for more than five years. The popularity and novelty of the platform have spurred numerous methodological investigations, making it the most studied nonprobability sample available to researchers. This article summarizes what is known about MTurk sample composition and data quality with an emphasis on findings relevant to clinical psychological research. It then addresses methodological issues with using MTurk--many of which are common to other nonprobability samples but unfamiliar to clinical science researchers--and suggests concrete steps to avoid these issues or minimize their impact.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Chandler and Shapiro (2016), including empirical work by Shapiro and colleagues (2013), demonstrate that MTurk-recruited research study participants are similarly honest, attentive, and reliable compared to non-MTurk samples, and that MTurk is particularly well suited to reaching populations with social anxiety symptoms.”
From A Randomized Trial of Brief Online Interventions to Facilitate Treatment Seeking for Social Anxiety. (2022) · cites this paper as Chandler and Shapiro, 2016
In the citing paper
“However, as noted, research reviewed by Chandler and Shapiro (2016), including empirical work by Shapiro and colleagues (2013), illustrates that MTurk workers are comparably honest, attentive, and reliable compared to non-MTurk samples, and that MTurk is well suited to reaching populations with clinically relevant symptoms.”
From A Randomized Trial of Brief Online Interventions to Facilitate Treatment Seeking for Social Anxiety. (2022) · cites this paper as Chandler and Shapiro (2016)
Citations
Cites
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996
Cognitive Reflection and Decision Making
Frederick, S. (2005) · Journal of Economic Perspectives · doi:10.1257/089533005775196732 · Cited by 5,256
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767
Self-citationInstructional manipulation checks: Detecting satisficing to increase statistical power
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2009.03.009 · Cited by 3,493
Conducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708
Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242
Inside the Turk: Understanding Mechanical Turk as a Participant Pool
Paolacci, G., & Chandler, J. (2014) · Current Directions in Psychological Science · doi:10.1177/0963721414531598 · Cited by 2,136
Self-citationCrowdsourcing user studies with Mechanical Turk
Kittur, A., Ed, H., & Suh, B. (2008) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357127 · Cited by 1,941
Research works printed but not linked: 12
- Barnhoorn JS, 2014, Behav. Res. Methods, 530, 1
- Beramendi P, Duch RM, Matsuo A. 2014. When lab subjects meet real people: comparing different modes of experiments. Presented at Asian Polit. Methodol. Conf., Taipai
- Casey L, Chandler J, Levine AS, Proctor A, Strolovitch D. 2015. Demographic characteristics of a large sample of US workers. Mathematica Policy Res. Rep. In press
- Downes-Le Guin T, 2006, ESOMAR World Res. Conf. Panel Res.
- Higgins C, 2010, Proc. NAACL HLT 2010 Workshop on Creat. Speech and Language Data with Amazon's Mechanical Turk, Los Angeles, 89
- 10.1007/s10683-011-9273-9 · DOI
- 10.1145/2676431.2676437 · DOI
- Lang ASID, 2011, Code4Lib J., 15, 1
- Lebreton P, 2015, Proc. SPIE Conf. Hum. Vis. Electron. Imaging, 20th, San Francisco
- 10.1026//1618-3169.49.4.243 · DOI
- Shao W, 2015, Digit. Cult. Educ., 7
- Suri S, 2011, AAAI Workshops, 25th, San Francisco, 61
Cited by
Sorted by citations
Understanding and managing the threat of common method bias: Detection, prevention and control
Kock, F., Berbekova, A., & Assaf, A. G. (2021) · Tourism Management · doi:10.1016/j.tourman.2021.104330 · Cited by 1,692
An Evaluation of Amazon’s Mechanical Turk, Its Rapid Rise, and Its Effective Use
Buhrmester, M. D., Talaifar, S., & Gosling, S. D. (2018) · Perspectives on Psychological Science · doi:10.1177/1745691617706516 · Cited by 815
Online panels in social science research: Expanding sampling methods beyond Mechanical Turk
Chandler, J., Rosenzweig, C., Moss, A. J., Robinson, J., & Litman, L. (2019) · Behavior Research Methods · doi:10.3758/s13428-019-01273-7 · Cited by 668
Self-citationCustomer service chatbots: Anthropomorphism and adoption
Sheehan, B., Jin, H. S., & Gottlieb, U. (2020) · Journal of Business Research · doi:10.1016/j.jbusres.2020.04.030 · Cited by 667
Marroquín, B. M., Vine, V., & Morgan, R. M. (2020) · Psychiatry Research · doi:10.1016/j.psychres.2020.113419 · Cited by 667
Metadata
- Authors
- Chandler, J., & Shapiro, D.
- Affiliations
- Mathematica Policy Research (US)
- University of Michigan (US)
- ORCID
- Chandler, Jesse
- Venue
- Annual Review of Clinical Psychology
- Publisher
- Annual Reviews
- Published
- 2016-03-28
- Volume
- 12
- Issue
- 1
- Pages
- 53-81
- ISSN
- 1548-5943 (ISSN-L)
- 1548-5951
- OpenAlex
- W2126493218
- Cited by
- 1,072
- References
- 177
- Citations by year
- 2026: 27 · 2025: 70 · 2024: 105 · 2023: 103 · 2022: 150 · 2021: 168 · 2020: 122 · 2019: 115 · 2018: 111 · 2017: 77 · 2016: 24
- FWCI
- 17.58
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Annual Review of Clinical Psychology (publishedVersion)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Ethics in Clinical Research
- Keywords
- crowdsourcing, Amazon Mechanical Turk, convenience samples, non-probability sampling, sample composition, data quality, participant recruitment, digital labour markets
- MeSH terms
- Humans, Biomedical Research, Crowdsourcing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
