Cited by 1,072Mobile Crowdsensing and CrowdsourcingSurvey Methodology and NonresponseEthics in Clinical Research

Conducting Clinical Research Using Crowdsourced Convenience Samples

Chandler, Jesse, Shapiro, Danielle·2016·Annual Review of Clinical Psychology

Abstract

Crowdsourcing has had a dramatic impact on the speed and scale at which scientific research can be conducted. Clinical scientists have particularly benefited from readily available research study participants and streamlined recruiting and payment systems afforded by Amazon Mechanical Turk (MTurk), a popular labor market for crowdsourcing workers. MTurk has been used in this capacity for more than five years. The popularity and novelty of the platform have spurred numerous methodological investigations, making it the most studied nonprobability sample available to researchers. This article summarizes what is known about MTurk sample composition and data quality with an emphasis on findings relevant to clinical psychological research. It then addresses methodological issues with using MTurk--many of which are common to other nonprobability samples but unfamiliar to clinical science researchers--and suggests concrete steps to avoid these issues or minimize their impact.

Integrity signals

Registry

Crossref · 10.1146/annurev-clinpsy-021815-093623

Cites

175 of 175 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Chandler and Shapiro (2016) demonstrate that MTurk-recruited research study participants are similarly honest, attentive, and reliable compared to non-MTurk samples, and that MTurk is particularly well suited to reaching populations with social anxiety symptoms.

    “Chandler and Shapiro (2016), including empirical work by Shapiro and colleagues (2013), demonstrate that MTurk-recruited research study participants are similarly honest, attentive, and reliable compared to non-MTurk samples, and that MTurk is particularly well suited to reaching populations with social anxiety symptoms.”
  • MTurk workers are comparably honest, attentive, and reliable compared to non-MTurk samples, and MTurk is well suited to reaching populations with clinically relevant symptoms.

    “However, as noted, research reviewed by Chandler and Shapiro (2016), including empirical work by Shapiro and colleagues (2013), illustrates that MTurk workers are comparably honest, attentive, and reliable compared to non-MTurk samples, and that MTurk is well suited to reaching populations with clinically relevant symptoms.”

Cites

Flagged first, then by citations

2 of the 177 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Authors
Chandler, J., & Shapiro, D.
Affiliations
  • Mathematica Policy Research (US)
  • University of Michigan (US)
Venue
Annual Review of Clinical Psychology
Published
2016-03-28
Volume
12
Issue
1
Pages
53-81
ISSN
  • 1548-5943 (ISSN-L)
  • 1548-5951
OpenAlex
W2126493218
Cited by
1,072
References
177
Citations by year
2026: 27 · 2025: 70 · 2024: 105 · 2023: 103 · 2022: 150 · 2021: 168 · 2020: 122 · 2019: 115 · 2018: 111 · 2017: 77 · 2016: 24
FWCI
17.58
Type
journal article
Language
en
Open access
closed
Locations
  • Annual Review of Clinical Psychology (publishedVersion)
  • PubMed (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
  • Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Ethics in Clinical Research
Keywords
crowdsourcing, Amazon Mechanical Turk, convenience samples, non-probability sampling, sample composition, data quality, participant recruitment, digital labour markets
MeSH terms
Humans, Biomedical Research, Crowdsourcing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review