Cited by 94Mobile Crowdsensing and CrowdsourcingSurvey Methodology and NonresponseComputational and Text Analysis Methods

Improving Data Quality Using Amazon Mechanical Turk Through Platform Setup

Lu, Lu, Neale, Nathan, Line, Nathaniel D., Bonn, Mark·2022·Cornell Hospitality Quarterly

Abstract

As the use of Amazon’s Mechanical Turk (MTurk) has increased among social science researchers, so, too, has research into the merits and drawbacks of the platform. However, while many endeavors have sought to address issues such as generalizability, the attentiveness of workers, and the quality of the associated data, there has been relatively less effort concentrated on integrating the various strategies that can be used to generate high-quality data using MTurk samples. Accordingly, the purpose of this research is twofold. First, existing studies are integrated into a set of strategies/best practices that can be used to maximize MTurk data quality. Second, focusing on task setup, selected platform-level strategies that have received relatively less attention in previous research are empirically tested to further enhance the contribution of the proposed best practices for MTurk usage.

Integrity signals

Registry

Crossref · 10.1177/19389655211025475

Cites

47 of 47 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Research has shown that data from MTurk workers is comparable to data from other pools (e.g., commercial panels, social media, colleges), but recently there has been a decrease in data quality.

    “Research has shown data from MTurk workers is comparable to data from other pools (e.g., commercial panels, social media, colleges) [12, 20, 23, 32, 69, 80], but recently there has been a decrease in data quality [28, 49, 66, 68, 104, 108].”

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Authors
Lu, L., Neale, N., Line, N. D., & Bonn, M.
Affiliations
  • Temple University (US)
  • University of Houston - Downtown (US)
  • Florida State University (US)
Corresponding author
Lu, Lu
Venue
Cornell Hospitality Quarterly
Published
2021-07-01
Volume
63
Issue
2
Pages
231-246
ISSN
  • 1938-9655 (ISSN-L)
  • 1938-9663
OpenAlex
W3173414067
Cited by
94
References
47
Citations by year
2026: 7 · 2025: 18 · 2024: 31 · 2023: 23 · 2022: 13 · 2021: 2
FWCI
14.60
Type
journal article
Language
en
Open access
closed
Locations
Cornell Hospitality Quarterly (publishedVersion)
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
  • Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
Keywords
Amazon Mechanical Turk, data quality, crowdsourcing, task design, best practices, online labour platforms

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review