CoChat

A panel for lemons? Positivity bias, reputation systems and data quality on MTurk

Matherly, Ted · 2019 · European Journal of Marketing
Cited by 75

The DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index.

Abstract

Purpose: The purpose of this paper is to investigate how the effectiveness of systems for ensuring cooperation in online transactions is impacted by a positivity bias in the evaluation of the work that is produced. The presence of this bias can reduce the informativeness of the reputation system and negatively impact its ability to ensure quality. Design/methodology/approach: This research combines survey and experimental methods, collecting data from 1,875 Mechanical Turk (MTurk) workers in five studies designed to investigate the informativeness of the MTurk reputation system. Findings: The findings demonstrate the presence of a positivity bias in evaluations of workers on MTurk, which leaves them undifferentiated, except at the extremity of the reputation system and by status markers. Research limitations/implications: Because MTurk workers self-select tasks, the findings are limited in that they may only be generalizable to those who are interested in research-related work. Further, the tasks used in this research are largely subjective in nature, which may decrease their sensitivity to differences in quality. Practical implications: For researchers, the results suggest that requiring 99 per cent approval rates (rather than the previously advised 95 per cent) should be used to identify high-quality workers on MTurk. Originality/value: The research provides insights into the design and use of reputation systems and demonstrates how design decisions can exacerbate the effect of naturally occurring biases in evaluations to reduce the utility of these systems.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1108/ejm-07-2017-0491

Cites

24 of 24 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 24 references

Cited by

See all 75

Metadata

Authors
Matherly, T.
Venue
European Journal of Marketing
Publisher
Emerald
Published
2019-02-11
Volume
53
Issue
2
Pages
195-223
ISSN
0309-0566 (ISSN-L)
OpenAlex
W2908862202
Cited by
75
References
24
FWCI
11.16
Type
journal article
Language
en
Open access
closed

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A panel for lemons? Positivity bias, reputation systems and data quality on MTurk. https://app.cochat.ai/papers/10.1108/ejm-07-2017-0491
The paper
Matherly, T. (2019). A panel for lemons? Positivity bias, reputation systems and data quality on MTurk. European Journal of Marketing. https://doi.org/10.1108/ejm-07-2017-0491

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.