CoChat
Sign Up
Cited by 668Mobile Crowdsensing and CrowdsourcingSurvey Methodology and NonresponseFocus Groups and Qualitative Methods

Online panels in social science research: Expanding sampling methods beyond Mechanical Turk

Open access
Chandler, Jesse, Rosenzweig, Cheskie, Moss, Aaron J., Robinson, Jonathan, Litman, Leib·2019·Behavior Research Methods
DOI 10.3758/s13428-019-01273-7PMC PMC6797699
Open this paper in CoChat — free accountOpen PDF

Abstract

Amazon Mechanical Turk (MTurk) is widely used by behavioral scientists to recruit research participants. MTurk offers advantages over traditional student subject pools, but it also has important limitations. In particular, the MTurk population is small and potentially overused, and some groups of interest to behavioral scientists are underrepresented and difficult to recruit. Here we examined whether online research panels can avoid these limitations. Specifically, we compared sample composition, data quality (measured by effect sizes, internal reliability, and attention checks), and the non-naivete of participants recruited from MTurk and Prime Panels—an aggregate of online research panels. Prime Panels participants were more diverse in age, family composition, religiosity, education, and political attitudes. Prime Panels participants also reported less exposure to classic protocols and produced larger effect sizes, but only after screening out several participants who failed a screening task. We conclude that online research panels offer a unique opportunity for research, yet one with some important trade-offs.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3758/s13428-019-01273-7

Cites

70 of 70 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Participants in Study 1 were recruited via MTurk, a method known to introduce potential demographic biases that may affect generalizability.

    “participants were recruited via MTurk, a method known to introduce potential demographic biases, such as higher digital literacy and specific cultural or political orientations, which may affect generalizability of our findings”

    From The associations between social comparison on social media and young adults’ mental health (2025) · cites this paper as Chandler et al., 2019

    The paper’s discussion of MTurk sample composition says its workers differ from the US population in age, education, religiosity, and political orientation. Its Introduction also notes that demographic differences can limit whether findings replicate in nationally representative samples. The copy does not establish that MTurk workers have higher digital literacy; the citing authors’ Study 1 recruitment is their own description, not a finding attributed to this source. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • MTurk is not representative, particularly when it comes to self-reported political identity and ideology.

    “and is, in fact, not representative, particularly when it comes to self-reported political identity and ideology51–53”

    From Accuracy prompts are a replicable and generalizable approach for reducing the spread of misinformation (2022) · cites this paper as [51]

  • MTurk offers convenience samples and makes no attempt to be nationally representative.

    “MTurk offers convenience samples and makes no attempt to be nationally representative”

    From Accuracy prompts are a replicable and generalizable approach for reducing the spread of misinformation (2022) · cites this paper as [51]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 2

  1. Pew Research Center. (2016b). A wider ideological gap between more and less educated adults. Retrieved from http://www.people-press.org/2016/04/26/a-wider-ideological-gap-between-more-and-less-educated-adults/
  2. Waggoner, B. J. (2018). Atheism and the effects of mortality salience and limited analytic processing ability on religious faith and teleological thinking (Unpublished master’s thesis). Florida International University, Miami, FL. Retrieved from https://etd.ohiolink.edu/!etd.send_file?accession=csu1530473668176033&disposition=inline

Cited by

Sorted by citations

Related

Authors
Chandler, J., Rosenzweig, C., Moss, A. J., Robinson, J., & Litman, L.
Affiliations
  • University of Michigan (US)
  • Columbia University (US)
  • Lander University (US)
Corresponding author
Litman, Leib
ORCID
Venue
Behavior Research Methods
Publisher
Springer Science and Business Media LLC
Published
2019-09-11
Volume
51
Issue
5
Pages
2022-2038
ISSN
  • 1554-351X (ISSN-L)
  • 1554-3528
DOI
10.3758/s13428-019-01273-7
PMC
PMC6797699
OpenAlex
W2972421655
Cited by
668
References
70
Citations by year
2026: 67 · 2025: 123 · 2024: 110 · 2023: 112 · 2022: 111 · 2021: 93 · 2020: 47 · 2019: 4 · 2016: 1
FWCI
69.12
Type
journal article
Language
en
Open access
Licence
Locations
  • Behavior Research Methods (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
  • Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
  • Social Sciences › Sociology and Political Science › Focus Groups and Qualitative Methods
Keywords
Amazon Mechanical Turk, participant recruitment, sample composition, data quality, effect size, internal consistency, attention checks, participant screening, participant diversity
MeSH terms
Data Accuracy, Attention, Humans, Mass Screening, Social Sciences, Students, Reproducibility of Results, Internet, Behavioral Research, Crowdsourcing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review