Student or educator? Sign up with your school email and get 6 months of Pro free.
Cited by 1,532Mobile Crowdsensing and CrowdsourcingSurvey Methodology and NonresponseResearch Data Management Practices

The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.

Data quality in online human-subjects research: Comparisons between MTurk, Prolific, CloudResearch, Qualtrics, and SONA

Open access
Douglas, Benjamin D., Ewell, Patrick J., Brauer, Markus·2023·PLOS ONE

Abstract

With the proliferation of online data collection in human-subjects research, concerns have been raised over the presence of inattentive survey participants and non-human respondents (bots). We compared the quality of the data collected through five commonly used platforms. Data quality was indicated by the percentage of participants who meaningfully respond to the researcher’s question (high quality) versus those who only contribute noise (low quality). We found that compared to MTurk, Qualtrics, or an undergraduate student sample (i.e., SONA), participants on Prolific and CloudResearch were more likely to pass various attention checks, provide meaningful answers, follow instructions, remember previously presented information, have a unique IP address and geolocation, and work slowly enough to be able to read all the items. We divided the samples into high- and low-quality respondents and computed the cost we paid per high-quality respondent. Prolific ($1.90) and CloudResearch ($2.00) were cheaper than MTurk ($4.36) and Qualtrics ($8.17). SONA cost $0.00, yet took the longest to collect the data.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1371/journal.pone.0279720

Cites

45 of 45 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

1 more claim about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 44 references

Research works printed but not linked: 2

  1. C Kennedy, 2020, Assessing the Risks to Online Polls from Bogus Respondents, Internet]. Pew Research Center
  2. J Ternovski, 2022, A Note on Increases in Inattentive Online Survey-Takers Since 2020. Journal of Quantitative Description, Digital Media, 2, 1

Cited by

See all 1,532

Metadata

Authors
Douglas, B. D., Ewell, P. J., & Brauer, M.
Affiliations
  • University of Wisconsin–Madison (US)
  • Kenyon College (US)
Venue
PLOS ONE
Published
2023-03-14
Volume
18
Issue
3
Pages
e0279720
ISSN
1932-6203 (ISSN-L)
OpenAlex
W4324129606
Cited by
1,532
References
44
Citations by year
2026: 558 · 2025: 571 · 2024: 329 · 2023: 69
FWCI
347.46
Type
journal article
Language
en
Open access
Locations
  • PLoS ONE (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
BRITE Lab Grant: Behavioral Research Insights Through Experiments Lab, <https://brite.wisc.edu/>
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
  • Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
  • Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
Keywords
online data collection, data quality, crowdsourcing platforms, Amazon Mechanical Turk, Prolific, Qualtrics, Sona, attention checks, bot detection, online surveys
MeSH terms
Data Accuracy, Humans, Surveys and Questionnaires, Research Subjects

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Data quality in online human-subjects research: Comparisons between MTurk, Prolific, CloudResearch, Qualtrics, and SONA. https://app.cochat.ai/papers/10.1371/journal.pone.0279720
The paper
Douglas, B. D., Ewell, P. J., & Brauer, M. (2023). Data quality in online human-subjects research: Comparisons between MTurk, Prolific, CloudResearch, Qualtrics, and SONA. PLOS ONE. https://doi.org/10.1371/journal.pone.0279720

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.