CoChat
Sign Up
Cited by 2,957Mobile Crowdsensing and CrowdsourcingSurvey Methodology and NonresponseExperimental Behavioral Economics Studies

Beyond the Turk: Alternative platforms for crowdsourcing behavioral research

Open access
Peer, Eyal, Brandimarte, Laura, Samat, Sonam, Acquisti, Alessandro·2017·Journal of Experimental Social Psychology
DOI 10.1016/j.jesp.2017.01.006
Open this paper in CoChat — free accountOpen PDF

AI snapshot

Written from the paper’s full text.

TL;DR

Peer et al. compare Amazon Mechanical Turk (MTurk) with alternative crowdsourcing platforms, especially CrowdFlower (CF) and Prolific Academic (ProA), on data quality, naivety, diversity, and honesty. They conclude ProA is the most viable alternative to MTurk, while CF offers fast recruitment but weaker attention and reproducibility.

Key findings

  • In Study 1, CF collected 200 responses fastest (44 minutes), followed by MTurk (1:48 hours) and ProA (4:37 hours); CF also had the highest attention-check failure rate, with only 27.1% of participants retained under a strict exclusion policy versus about half for MTurk, ProA, and CBDR. Average failed ACQs were 0.67 (MTurk), 0.81 (ProA), 1.04 (CBDR), and 1.76 (CF).
  • MTurk and ProA replicated all tested effects in Study 1, but CF did not show the Sunk Cost or Gambler’s Fallacy effects; CF also had lower reliability on the RSES and NFC scales, improving after ACQ exclusions.
  • Participants were least naïve on MTurk (60.3% of tasks reported as seen for the first time), followed by ProA (68.3%), CBDR (72.2%), and CF (80.8%). Die-roll cheating was highest on MTurk (mean claimed bonus 46.87 cents; Cohen’s d = 1.88), followed by ProA (42.29; d = 0.92) and CF (40.68; d = 0.70), against a 35-cent honest benchmark.
  • In Study 2 (1,374 recruited; 1,205 completed), MTurk was faster than ProA (151 minutes vs. almost 10 hours to reach 600 responses; 3.99 vs. 1.01 responses per minute). MTurk had more participants passing all three ACQs (60.6% vs. 48.4%), but reliability remained high on both platforms and the simulation heuristic replicated on both.

Methods

  • Study 1 was an online survey run on CF, ProA, CBDR, and MTurk, targeting about 200 participants per platform and yielding 831 total participants in January 2016. It measured attention with four ACQs, reliability with the NFC and RSES scales, naivety, reproducibility of four judgment-and-decision tasks, and dishonest behavior via a die-roll task.
  • Study 2 focused on MTurk and ProA, recruiting 1,374 participants (691 MTurk, 683 ProA) and analyzing 1,205 completers (604 MTurk, 601 ProA). It used the Consideration for Future Consequences scale, three ACQs, the simulation heuristic, and demographic/usage questions.
  • The paper also reports a preliminary study that attempted six alternative platforms and successfully sampled CrowdFlower, MicroWorkers, RapidWorkers, MTurk, and CBDR, with 890 total participants.

Limitations

  • Stated: the roots and causes of platform differences remain unclear because the authors could control sampling but not allocation of participants across platforms; it is also unclear how constant or transient the findings are, since some differences (e.g., demographics) seemed stable while others (e.g., response rates, naivety) could be temporary.
  • Stated: after Study 1, ProA changed its pricing scheme, which may have affected response rates in Study 2; the authors note this as a possible reason for the larger MTurk–ProA response-rate gap.
  • Evident from the text: Study 2 excluded some “obvious” hypothesis questions under editorial advice because their interpretation was ambiguous; the platform comparisons are also limited to the specific platforms, tasks, payment schemes, and time periods studied.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1016/j.jesp.2017.01.006

Cites

31 of 31 references verified0 citation errors · 0 not checkable

Statistics

23 tests recomputed · 0 with the reported and recomputed p on opposite sides of .05

Claims the paper makes

0 of 17 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    Among CBDR participants, excluding based on ACQs generally increased effect sizes except for Quote Attribution.

    “Among CBDR participants, excluding based on ACQs generally increased effect sizes except for the case of the Quote Attribution task.”
    Quote in the paperNot a substantial claim
  • Findingown data

    The most familiar tasks were RSES and NFC scales, followed by Asian Disease problem.

    “As Figure 4 shows, the most familiar tasks were the RSES and NFC scales, followed by the Asian Disease problem.”
    Quote in the paperNot a substantial claim
  • Findingown data

    MTurk participants had a higher income than the other samples.

    “MTurk participants had a higher income than the other samples.”
    Quote in the paperNot a substantial claim
  • Findingown data

    There were statistically significant differences between the samples in ethnicity, education, and income.

    “We found statistically significant differences between the samples in ethnicity, χ 2 (15) = 92.64,p < .01,education, χ 2 (6) = 17.85, p < .01,and income, χ 2 (18) = 61.5, p < .01”
    Quote in the paperSubstantial claim
  • Findingown data

    All post-hoc differences in failed ACQs, except between ProA and CBDR, were statistically significant after Bonferroni correction.

    “All post-hoc differences, except between ProA and CBDR, were statistically significant after applying Bonferroni's correction (p < .05).”
    Quote in the paperSubstantial claim
  • May combine two claims. 75
  • Classified as probably not a substantial claim. 21
  • The quote only partly states this claim. 1

  • May combine two claims.

    Findingown data

    Excluding participants based on ACQs on MTurk had little to no impact on observed effect sizes, and it somewhat increased effect sizes on ProA.

    “As can be seen in Table 3, excluding participants based on ACQs on MTurk had little to no impact on the observed effect sizes, and it somewhat increased effect sizes on ProA.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The vast majority of MTurk and CBDR participants reported residing in North America, while CF and ProA showed a much more diverse distribution across the globe.

    “Regarding location, while the vast majority of MTurk (and CBDR) participants reportedfoot_2 that they currently resided in North America (U.S. and Canada), CF and ProA showed a much more diverse distribution across the globe.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Differences in reliability among CF groups were statistically significant for both RSES and NFC.

    “Using Hakstian and Whalen's (1976) method to compare between independent reliability coefficients, we found the differences in reliability among CF, between the groups stated above, were statistically significant for both the RSES and NFC (χ 2 (2) = 8.21, 17.95; p = .02,p < .01).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Both RSES and NFC scales showed expected high reliability scores overall (Cronbach's alpha = 0.898, 0.901 respectively).

    “Overall, both scales showed the expected high reliability scores (Cronbach's alpha = 0.898, 0.901 respectively).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    CF showed the fastest response rates, with 200 responses collected within 44 minutes, followed by MTurk (1:48 hours), ProA (4:37 hours), and CBDR (stopped after a week with 195 responses).

    “As can be seen, CF showed the fastest response rates, with 200 responses collected within 44 minutes, followed by MTurk, where it took 1:48 hours to collect 200 responses.On ProA, it took 4:37 hours to collect 200 responses, and collection was stopped after a week on CBDR (which had provided 195 responses at that time).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    All four experimental effects were statistically significant in MTurk and ProA samples; CF participants did not show either the Sunk Cost or Gambler's Fallacy effects; CBDR participants did not exhibit the Gambler's Fallacy effect.

    “We first looked at overall replicability and, as Table 3 shows, found all effects to be statistically significant in MTurk and ProA samples.However, CF participants did not show either the Sunk Cost or Gambler's Fallacy effects.CBDR participants did not exhibit the Gambler's Fallacy effect either.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Dropout rates were around 10% for all platforms, with no significant differences between the platforms.

    “As detailed in Table 2, dropout rates were around 10% for all platforms, with no significant differences between the platforms, χ 2 (3) = 3.43, p = .33.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Under a lenient exclusion policy, CF's sample is reduced the most to about only 45% of its original size; this difference is significant.

    “Using the lenient policy of allowing participants to fail one ACQ reduces the sample size less for all platforms, but still CF's sample is reduced the most to about only 45% of its original size (χ 2 (3) = 80.83, p < .01).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The average response rate was best on CF and MTurk (3.85 and 5.62 minutes required for 10 responses), followed by ProA (12.94 minutes per 10 responses) and CBDR (about 9 hours per 10 responses).

    “The average response rate was best on CF and MTurk (3.85 and 5.62 minutes required for 10 responses), followed by ProA (12.94 minutes per 10 responses) and CBDR (about 9 hours per 10 responses).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Among CF participants, strict exclusion had a mixed effect: increased effect size of Asian Disease and Gambler's Fallacy, reduced effect sizes on other tasks.

    “Among CF participants, the strict exclusion policy had a mixed effect as it increased the effect size of the Asian Disease and Gambler's Fallacy tasks, while it reduced effect sizes on the other tasks.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    MTurk participants were typically more familiar with the tasks, while CF participants were more naïve.

    “Between the platforms, MTurk participants were typically more familiar with the tasks, while CF participants were more naïve to the tasks.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For RSES, reliability was adequately high (around or above 0.90) on all platforms except CF; for CF, reliability improved significantly from 0.837 to 0.901 under lenient exclusion and was 0.891 under strict.

    “As shown in Figure 3a, reliability measures for the RSES were adequately high (around or above 0.90) on all platforms except CF, and that did not change considerably under the lenient or strict exclusion policies.For CF, reliability improved significantly (from 0.837 to 0.901) when applying the lenient exclusion policy, and it was similarly high (0.891) under the strict policy.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Caucasians were more prevalent on MTurk and ProA than on CF, which included a higher proportion of Asian and Latin/Hispanic participants.

    “In general, Caucasians were more prevalent on MTurk and ProA than on CF, which included a higher proportion of Asian and Latin/Hispanic participants”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Median completion time was lowest on CBDR (10 minutes), followed by MTurk (11 minutes), ProA (14 minutes), and highest on CF (16 minutes); differences were statistically significant.

    “Because the time distribution was highly skewed, we compared medians across groups and found that it was lowest on CBDR (10 minutes), followed by MTurk (11 minutes), ProA (14 minutes), and highest on CF (16 minutes).A Kruskal-Wallis test showed that these differences were statistically significant (p < .01).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For NFC, reliability was high for all platforms except CF; for CF, reliability was lower in overall sample (0.689) and improved significantly to 0.836 under both exclusion policies.

    “This pattern appeared similarly for the NFC: For all platforms, except CF, reliability was high for the overall sample and also after excluding based on ACQs.For CF, reliability was lower in the overall sample (0.689) and improved significantly (to 0.836) under both exclusion policies.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    60.6% of MTurk participants passed all three ACQs, while only 48.4% of ProA's participants passed all ACQs.

    “While 60.6% of participants on MTurk passed all three ACQs, only 48.4% of ProA's participants passed all ACQs.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    More than 70% of MTurk participants reported earning more than $50 a week, while about 85% of ProA participants reported earning less than $10 a week.

    “whereas more than 70% of participants on MTurk reported earning more than $50 a week, about 85% of ProA participants reported earning less than $10 a week.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    ProA participants included slightly more Asians and Hispanics, and slightly fewer African-American and Caucasians than MTurk.

    “ProA participants included slightly more Asians and Hispanics, and slightly fewer African-American and Caucasians than MTurk”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    On ShortTask, the authors received a credit card error despite using a working credit card, and repeated efforts to contact the site were left unrequited.

    “On ShortTask, we received a credit card error despite using a working credit card, and our repeated efforts to contact them were left unrequited.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In Study 1, the response rate difference between MTurk and ProA was about 2.5 times in favor of MTurk; in Study 2, that ratio increased to 4 times in favor of MTurk.

    “Whereas in Study 1 the difference between the response rate on MTurk to ProA was about 2.5 times in favor of MTurk, that ratio increased to 4 times in favor of MTurk in Study 2.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    ProA participants were somewhat more educated and had lower income compared to MTurk.

    “they were somewhat more educated and had lower income compared to MTurk.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The authors attempted to set up a survey on all six identified platforms but were successful on only three of them.

    “We attempted to set up a survey on all six of the aforementioned platforms, but were successful in doing so only on three of them.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    North Americans comprised only 25.9% of ProA's participants, which also included 30.8% from the U.K., another 27.1% from Europe, 8.1% from South America, and 6% from India.

    “North Americans comprised only 25.9% of ProA's participants, which also included 30.8% from the U.K., another 27.1% from Europe, 8.1% from South America, and 6% from India”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    Unlike MTurk, some of the identified platforms require that a task posted by a requester be reviewed before it is made available to workers.

    “Unlike MTurk, some of these platforms require that a”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    MiniJobz rejected the survey after reviewing it without providing any reason.

    “MiniJobz rejected our survey after reviewing it, without providing any reason for rejection.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Reliability remained high on both platforms, and remained consistently high when excluding participants who failed ACQs.

    “Reliability remained high on both platforms, and it remained consistently high when excluding participants who failed ACQs, on both sites.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The simulation heuristic was replicated successfully on both sites, with no significant differences between the sites.

    “On both sites, the simulation heuristic was replicated successfully, with no significant differences between the sites.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The percent of participants failing one, two or all ACQs on MTurk were 26%, 9.6% and 3.8% while on ProA these were 19%, 20% and 12%.

    “The percent of participants failing one, two or all ACQs on MTurk were 26%, 9.6% and 3.8% while on ProA these were 19%, 20% and 12%.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    ProA had more diverse and naïve participants than MTurk.

    “with more diverse and naïve participants”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The effect was slightly stronger under the exclusion policies, but these differences were not statistically significant.

    “The effect was slightly stronger under the exclusion policies, but these differences were not statistically significant, as is evident by the overlap of the 95% confidence interval bars in Figure 12.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    The reviewed sites offered researchers the ability to sample participants based on characteristics such as country and language.

    “These sites also offer researchers the ability to sample participants based on other characteristics, such as country and language.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingreasoning

    The authors identified six potential crowdsourcing platforms that seemed similar to Mechanical Turk in general design and purpose: CrowdFlower, MicroWorkers, RapidWorkers, MiniJobz, ClickWorker, and ShortTask.

    “By searching for crowdsourcing websites, we identified six potential platforms that seem similar to Mechanical Turk in general design and purpose: CrowdFlower, MicroWorkers, RapidWorkers, MiniJobz, ClickWorker and ShortTask.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    During the one-week sampling period, the authors were able to sample about 200 participants from all platforms except RapidWorkers, ending with a total sample of 890 participants.

    “During that week, we were able to sample about 200 participants from all platforms except for RapidWorkers, and ended up with a total sample of 890 participants.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    At the time of data collection, only MicroWorkers provided the ability to pay bonus payments to participants after they submitted their work.

    “At the time of our data collection, only MicroWorkers provided the ability to pay bonus payments to participants after they submit their work (since then, CrowdFlower has also added this functionality).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The degree of overlap between platforms seems quite small, with the highest overlap among the 22% of ProA users who also used MTurk.

    “Generally, the degree of overlap between platforms seems to be quite small, with the highest overlap among the 22% of ProA users who also used MTurk.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    MTurk participants were least naïve (60.3% tasks seen first time), followed by ProA (68.3%) and CBDR (72.2%); CF participants were most naïve (80.8%).

    “MTurk participants were the least naïve, with an average of 60.3% of tasks reported as seen for the first time, followed by ProA and CBDR (68.3%, 72.2%) participants; CF participants seemed the most naïve, as they reported a mean of 80.8% tasks seen for the first time.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    All of the sites included surveys, and many included tasks requiring users to sign up for other sites, click links, and download applications.

    “All of the sites also include surveys, and many of them include tasks that require users to sign-up for various other sites, to click on links, and download applications.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Effect sizes of cheating degree were significantly highest on MTurk, followed by ProA, and lowest among CF participants (Cohen's d = 1.88, 0.92, 0.70).

    “However, the effect sizes of cheating degree were significantly highest on MTurk, followed by ProA, and lowest among CF participants, Cohen's d = 1.88, 0.92, 0.70, respectively, F (2, 633) = 9.49, p < .01.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    90.5% of MTurk participants were from North America, and 6.8% from India.

    “While 90.5% of MTurk participants were from North America, and 6.8% from India”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingcitation

    Most of the reviewed sites had supervision and reputation mechanisms designed to measure the quality of submitted work and to allow researchers to invite workers with high reputation.

    “Most of the sites we reviewed have supervision and reputation mechanisms designed to measure the quality of submitted work, and to allow researchers to invite workers with high reputation (similar to MTurk's”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Peer et al., 2014 source

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    MTurk's cheating rate was significantly higher than both ProA's and CF's, but the difference between ProA and CF was not significant.

    “Post-hoc comparisons, using Bonferroni's correction, showed that MTurk's cheating rate was significantly higher than both ProA's and CF's (p < .01),but that the difference between the latter two samples was not (p = 0.79).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    ProA participants spent considerably less time on the platform than MTurk and CF participants, with 69.1% reporting spending between 1 and 8 hours per week.

    “ProA users spent considerably less time, with 69.1% reporting spending between 1 to 8 hours per week.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Participants' self-reported platform usage, earnings, number of tasks completed, and approval scores are accurate.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingreasoning

    Only one participant completed the survey during that night and the others joined the following day.

    “Apparently, only one participant completed it during that night and the others joined the following day.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The majority of users from all sites except CrowdFlower came from the U.S.; about half of CrowdFlower users were from Europe, and the rest were from India (16.3%) or other mostly South American and Asian countries (28.1%).

    “whereas the majority of users from all sites except CrowdFlower came from the U.S. (see Figure 5), about half of the users on CrowdFlower were from Europe, and the rest were from India (16.3%) or other (mostly South American and Asian) countries (28.1%).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Median completion time was lowest on RapidWorkers (11.58 mins), followed by MTurk (13.06 mins), MicroWorkers and CBDR (16.77 and 16.88 mins), and highest on CrowdFlower (17.67 mins).

    “Because the time distribution was highly skewed, we compared medians across groups to find it was lowest on RapidWorkers (11.58 mins), followed by MTurk (13.06 mins), MicroWorkers and CBDR (16.77 and 16.88) and the highest on CrowdFlower (17.67 mins).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    MTurk and CrowdFlower participants seemed almost solely focused on using their respective platforms, and CBDR participants rarely used any of these platforms.

    “MTurk and CrowdFlower participants seem to be almost solely focused on using their respective platforms, and CBDR participants rarely used any of these platforms.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    MTurk, MicroWorkers, and CBDR participants were predominantly Caucasian (more than 60%), whereas CrowdFlower had a relatively high rate of Asian participants, as did RapidWorkers and CBDR.

    “with MTurk, MicroWorkers and CBDR participants being predominantly Caucasian (more than 60%), whereas CrowdFlower having a relatively high rate of Asian participants, as did RapidWorkers and CBDR.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    MTurk participants reported the highest average pay (M=$3.69, SD=$0.5), followed by CF (M=$2.54, SD=$0.9), ProA (M=$1.81, SD=$0.81) and CBDR (M=$1.3, SD=$0.57).

    “as MTurk participants reported the highest average pay (M=$3.69,SD=$0.5),followed by CF (M=$2.54,SD=$0.9),ProA (M=$1.81,SD=$0.81) and CBDR (M=$1.3,SD=$0.57).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Participants' self-reported platform usage, earnings, number of tasks completed, and approval scores are accurate.

  • May combine two claims.

    Findingown data

    Only MicroWorkers and CBDR participants showed a statistically significant effect in the sunk-cost task, although the effect was still in the expected direction on MTurk and CrowdFlower.

    “However, only MicroWorkers and CBDR participants showed a statistically significant effect in the sunk-cost task (although the effect was still in the expected direction on MTurk and CrowdFlower).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    RapidWorkers participants sometimes used MicroWorkers and CrowdFlower but not MTurk, and MicroWorkers participants sometimes used MTurk.

    “In contrast, RapidWorkers participants sometimes used MicroWorkers and CrowdFlower (but not MTurk), and MicroWorkers' participants sometimes used MTurk.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For the gain vs. loss framing effect, all but RapidWorkers' participants showed the effect in the expected direction, and these effects were statistically significant.

    “For the gain vs. loss framing effect, all but RapidWorkers' participants showed the effect in the expected direction (over-choice of sure option in the positive vs. the negative framing), and these effects were statistically significant.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The median number of tasks participants reported completing on the platform was highest among MTurk (7,100), lower on CF (1,000), and much lower on ProA (30) and CBDR (6).

    “Consistently, the median number of tasks participants reported completing on the platform was highest among MTurk (7,100), lower on CF (1,000) and much lower on ProA (30) and CBDR (6).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Participants' self-reported platform usage, earnings, number of tasks completed, and approval scores are accurate.

  • May combine two claims.

    Findingown data

    After Bonferroni correction, there were significant differences on the total score when comparing MTurk and CBDR to the other samples (p<.01), but no differences between MTurk and CBDR (p=1).

    “We examined pair-wise differences on the total score and found, after Bonferroni correction, significant differences when comparing MTurk and CBDR to the other samples (p < .01),but no differences between MTurk and CBDR (p = 1).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Dropout rates ranged between 10% for CBDR and about 17% for RapidWorkers, except for MicroWorkers, where 44% of respondents who started the survey did not complete it.

    “dropout rates ranged between 10% (for CBDR) to about 17% (for RapidWorkers), except for MicroWorkers, where 44% of respondents that started the survey did not complete it.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    MicroWorkers, RapidWorkers, MiniJobz, ClickWorker and ShortTask did not prove as effective as CF and ProA in data quality, response rate, or cost of recruitment.

    “These websites did not prove as effective as the ones we have chosen to report on -either in their data quality or response rate or the cost of recruitment -and so we do not discuss them in this paper.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Many MicroWorkers and CBDR participants indicated they used the platform for passing time.

    “Many MicroWorkers and CBDR participants indicated they use the platform for passing time.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    MTurk, CBDR, and CrowdFlower participants mostly held an undergraduate degree, whereas MicroWorkers participants mostly had a highschool education level.

    “mostly held an undergraduate degree, whereas MicroWorkers participants mostly had a highschool education level.”
    Quote in the paperNot a substantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    There were differences in education level, income level, and U.S. citizenship between samples, χ2(4)=128.37, 125.84, 568.74, p<.01.

    “S. citizenship between samples (χ 2 (4= 128.37, 125.84, 568.74, p < .01)”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Comparing the distribution of die outcomes to a uniform random distribution showed no statistical differences among any samples that received this part (MTurk, CBDR, and MicroWorkers; p=.83, .13, .51), showing no evidence for dishonest reporting on the die-roll task.

    “Comparing the distribution of die outcomes to a uniform random distribution showed no statistical differences among any of the samples that received this part of the study (MTurk, CBDR and MicroWorkers; p = .83,.13,.51,respectively), showing no evidence for dishonest reporting on the die roll task.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For the anchoring tasks, a MANOVA found statistically significant effects for anchoring condition, sample, and their interaction, Wilk's λ=.78, .90, .89, F(4/16, 326/996)=22.77, 2.24, 2.43, p<.01.

    “We found a statistically significant effect for both the anchoring condition and the sample, as well as their interaction, Wilk's λ = .78,.90,.89,F (4/16, 326/996) = 22.77, 2.24, 2.43, p < .01,respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The effects of anchoring were not different between samples, except for an unusually high effect for the Mississippi question in the RapidWorkers sample.

    “As can be seen in Table 5, the effects of the anchoring was not different between samples, except for an unusually high effect for the Mississippi question in the RapidWorkers sample.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Reliability was high on both sites, with MTurk showing somewhat higher reliability.

    “Reliability was high on both sites, with MTurk showing somewhat higher reliability.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    MTurk and CBDR participants obtained the highest CRT scores overall, followed by CrowdFlower.

    “As seen in Figure 2, MTurk and CBDR participants obtained the highest scores overall, followed by CrowdFlower,”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Reliability measures for participants who passed both attention-check questions, compared to those who did not, were typically not different, and sometimes even in the opposite expected direction.

    “As can be seen in Table 4, reliability measures for participants who passed both attention-check questions, compared to those who did not, were typically not different, and sometimes even in the opposite expected direction.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    RapidWorkers participants showed high reliability on the IUIPC scale, but very low reliability on the NFC and mediocre reliability on the RSES scales.

    “RapidWorkers participants showed high reliability on the IUIPC scale, but very low reliability on the NFC and mediocre reliability on the RSES scales.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    Reliability was assessed using the Consideration for Future Consequences scale (Strathman et al., 1994).

    “To assess reliability, we used the Consideration for Future Consequences scale (Strathman, Gleicher, Boninger, & Edwards, 1994).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Strathman, Gleicher, Boninger, & Edwards, 1994 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    MicroWorkers and RapidWorkers published the survey about 24 hours after submission, CBDR published it in the evening of the same day, and the survey started immediately on all other platforms.

    “Due to their process, MicroWorkers and RapidWorkers published our survey about 24 hours after we submitted it, and CBDR published our survey in the evening of the same day, whereas our survey was started immediately on all other platforms.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Scholars have urged the scientific community to expand beyond WEIRD participants, and specifically beyond U.S.-based participants.

    “scholars have urged the scientific community to expand beyond western, industrialized, educated, rich and democratic participants (or WEIRD; see Henrich, Heine, & Norenzayan, 2010), and specifically beyond U.S.-based participants”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Henrich, Heine, & Norenzayan, 2010 source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The sampling time was limited to one week to set a common timeframe for the study.

    “We limited the sampling time to one week, in order to set a common timeframe for the study.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    On both MTurk and ProA, most participants pay attention to instructions and consistently complete questionnaires carefully.

    “The results suggest that on both MTurk and ProA, most of the participants pay attention to instructions and consistently complete questionnaires carefully.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data

    Reliability above 0.8 indicates adequate reliability.

    “indicating adequate reliability.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Both CrowdFlower and MicroWorkers seemed to be possible alternatives for MTurk: questionnaire reliability on those samples was adequate in most cases, the samples typically reproduced the known effects of the Asian-disease problem, sunk-cost, and anchoring, and MicroWorkers participants did not try to cheat by over-reporting their performance.

    “In contrast, both CrowdFlower and MicroWorkers seem to be possible alternatives for MTurk: The reliability of questionnaire on those samples was adequate on most cases; the samples typically reproduced the known effects of the Asian-disease problem, sunk-cost and anchoring; and MicroWorkers participants did not try to cheat the researcher by over-reporting their performance.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The order of the scales was randomized between participants.

    “The order of the scales was randomized between participants.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    CrowdFlower showed superiority over MicroWorkers in terms of response rates, achieving a response rate identical to MTurk's.

    “over MicroWorkers in terms of response rates (achieving a response rate identical to that of MTurk's).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The order of tasks and questions within each task was randomized between participants, and allocation to conditions was randomized within each task.

    “The order of these tasks, as well as the questions within each task, were randomized between participants, and allocation to conditions was randomized within each of these tasks.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    From the perspective of replicability of known effects, both CrowdFlower and MicroWorkers could be considered a good alternative for MTurk, whereas RapidWorkers's results were less than adequate.

    “To summarize, from the perspective of replicability of known effects, both CrowdFlower and MicroWorkers could be considered a good alternative for MTurk, whereas RapidWorker's results were less than adequate.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    CrowdFlower taps into completely different populations than MTurk, namely European participants that are absent on MTurk and the other sites.

    “These differences clearly show that CrowdFlower taps into completely different populations than MTurk, namely European participants that are absent on MTurk and the other sites.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The high attention-check failure did not seem to impact CrowdFlower participants' ability to provide reliable responses to questionnaires or reproduce effects in known tasks.

    “This did not, however, seem to impact their ability to provide reliable responses to the questionnaires or reproduce the effects in the known tasks.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    CrowdFlower was distinctly different from the other samples in ethnicity and origin, but not in income and education, in which MicroWorkers participants were a little less educated but of higher income than the other sites.

    “To summarize, it appears that CrowdFlower is distinctly different from the other samples in aspects of ethnicity, origin, but not in aspects of income and education (in which MicroWorkers participants were a little bit less educated, but of higher income, than the other sites).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    MicroWorkers had a satisfactory response rate, although somewhat slower than the university pool.

    “MicroWorkers also had a satisfactory response rate, although somewhat slower than the university pool”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Both CrowdFlower and MicroWorkers exhibited a high level of data quality, not inferior to MTurk's on most respects, whereas RapidWorkers's results were unsatisfactory on most aspects examined.

    “To summarize thus far, it appears that both CrowdFlower and MicroWorkers exhibit a high level of data quality, not inferior to that of MTurk's on most respects, whereas RapidWorker's results were unsatisfactory on most of the aspects examined in this study.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    All analyses included only participants who completed the entire study.

    “All our analyses include only participants who completed the entire study.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    The scales used (IUIPC, NFC, RSES) are reliable and validated.

    “they are reliable and validated scales, and b) they have been previously used with success to measure data quality on MTurk (Peer et al., 2014).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Peer et al., 2014 source

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    CrowdFlower provided a comparable alternative to MTurk in terms of response rates and had a higher response rate than the CBDR university pool.

    “To summarize, CrowdFlower provided a comparable alternative to MTurk in terms of response rates, and had a higher response rate than the CBDR university pool.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The first stage of the survey consisted of psychometric questionnaires and experimental tasks adopted from prominent behavioral and psychological studies, administered in random order.

    “The first stage consisted of several psychometric questionnaires and experimental tasks adopted from prominent behavioral and psychological studies, which were administered in random order and designed to test attention, reliability, reproducibility, non-naivety, and individual differences.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    A high failure rate on attention-check questions could be considered desirable because it implies these sites' participants are still somewhat naïve to experimental materials.

    “and a high failure rate could actually be considered desirable because it implies these sites' participants are still somewhat naïve to experimental materials.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The Asian-disease framing effect is a known effect.

    “The first task was based on the Asian-disease framing effect (Tversky & Kahneman, 1981),”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Tversky & Kahneman, 1981 source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    The Cognitive Reflection Task can be used to explore non-naivety among MTurk participants.

    “we included the Cognitive Reflection Task, as Chandler et al. (2014) used to explore non-naivety among MTurk participants.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Chandler et al., 2014 source

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    From the perspective of reliability on established scales, MicroWorkers, followed by CrowdFlower, provided the best alternative to MTurk, whereas RapidWorkers results were less than satisfactory.

    “To summarize, from the perspective of reliability on established scales, MicroWorkers, followed by CrowdFlower, provided the best alternative to MTurk, whereas RapidWorkers results were less than satisfactory.”
    Quote in the paperSubstantial claim
89 of 89

  • The paper relies on this without support.

    MTurk's terms of service prohibit tasks that ask participants to download or install software or applications, or to disclose identifiable personal information.

    “For example, MTurk's terms of service prohibit tasks that ask participants to download or install software or applications, or to disclose identifiable personal information (including email addresses).”

  • The paper relies on this without support.

    ProA changed their pricing scheme to significantly raise the commission paid by researchers after Study 1.

    “after Study 1 was conducted, ProA changed their pricing scheme to significantly raise the commission paid by researchers.”

  • The paper relies on this without support.

    MTurk has been available for several years before ProA was launched.

    “MTurk has been available for several years before ProA was launched.”

  • The paper relies on this without support.

    MTurk provides a suitable benchmark for evaluating the data quality of alternative crowdsourcing platforms.

    “comparable alternative to MTurk in terms of response rates”

  • The paper relies on this without support.

    Failing attention-check questions indicates that participants did not fully read the survey instructions.

    “failed both the attention-check questions, suggesting these participants may not fully read the instructions in the survey questions.”

  • The paper relies on this without support.

    Participants' self-reported frequency of platform use accurately reflects their actual platform use.

    “We asked participants from each platform to indicate the frequency with which they have used that platform, as well as the other crowdsourcing platforms”

  • The paper relies on this without support.

    Participants' self-reported platform usage, earnings, number of tasks completed, and approval scores are accurate.

    “The median approval score (percentage of approved submissions) participants reported having was close to 100% for all platforms except for CF (89%).”

  • The paper relies on this without support.

    Participants' reported locations and IP addresses accurately indicate their geographic location.

    “We compared participants' reported locations to the location of their IP addresses, and confirmed that about 97% of location reports were compatible with the coordinates of their IP address.”

  • The paper relies on this without support.

    The four mundane hypotheses used to assess reproducibility (men are taller than women; women own more shoes than men; liberals value social equality more than conservatives; smokers think smoking kills less than non-smokers) are true known effects.

    “we adopted the approach of Simmons, Nelson, & Simonsohn (2013), and included in the study several questions that were aimed to test four relatively mundane hypotheses: a) that men are taller than women; b) that women own more shoes than men; c) that liberals think social equality is more important than conservatives; d) that smokers think that smoking kills less than non-smokers do.”

  • The paper relies on this without support.

    The samples from different platforms are comparable enough that observed differences reflect platform differences rather than sample composition.

    “The differences between average pay/week between the samples were statistically significant, F (3, 769) = 371.46,p < .01,”
10 of 10

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Statistics that don’t add up

Each reported p-value recomputed from its test statistic and degrees of freedom.

  • Inconsistent p-value: χ 2 (4) = 6.63, p = .17

    Minor

    The paper reports p = .17 here; recomputing from the test statistic and its degrees of freedom gives p = .157. The check cannot tell whether the statistic or the p-value is the misprint.

    Reported p = .17 · recomputed p = .157

1 of 1

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • Doesn’t add upχ²(4) = 6.63reported p = .17 · recomputed p = .157

    “χ 2 (4) = 6.63, p = .17”

  • Consistentχ²(15) = 92.64reported p < .01 · recomputed p < .001

    “χ 2 (15) = 92.64,p < .01”

  • Consistentχ²(6) = 17.85reported p < .01 · recomputed p = .007

    “χ 2 (6) = 17.85, p < .01”

  • Consistentχ²(18) = 61.5reported p < .01 · recomputed p < .001

    “χ 2 (18) = 61.5, p < .01”

  • Consistentχ²(3) = 3.43reported p = .33 · recomputed p = .330

    “χ 2 (3) = 3.43, p = .33”

  • ConsistentF(3, 827) = 37.41reported p < .01 · recomputed p < .001

    “F (3, 827) = 37.41, p < .01”

  • Consistentt(219) = 2.93reported p < .01 · recomputed p = .004

    “t (219) = 2.93, p < .01”

  • ConsistentF(3, 827) = 25.34reported p < .01 · recomputed p < .001

    “F (3, 827) = 25.34,p < .01”

  • ConsistentF(2, 633) = 9.49reported p < .01 · recomputed p < .001

    “F (2, 633) = 9.49, p < .01”

  • Consistentχ²(1) = 1.04reported p = .31 · recomputed p = .308

    “χ 2 (1) = 1.04, p = .31”

  • Consistentχ²(5) = 25.51reported p < .01 · recomputed p < .001

    “χ 2 (5) = 25.51,p < .01”

  • Consistentχ²(9) = 60.04reported p < .01 · recomputed p < .001

    “χ 2 (9) = 60.04,p < .01”

  • Consistentχ²(7) = 147.02reported p < .01 · recomputed p < .001

    “χ 2 (7) = 147.02,p < .01”

  • Consistentχ²(6) = 575.2reported p < .01 · recomputed p < .001

    “χ 2 (6) = 575.2,p < .01”

  • Consistentχ²(3) = 64.03reported p < .01 · recomputed p < .001

    “χ 2 (3) = 64.03,p < .01”

  • Consistentχ²(1) = 61.18reported p < .01 · recomputed p < .001

    “χ 2 (1) = 61.18,p < .01”

  • Consistentχ²(4) = 385.46reported p < .01 · recomputed p < .001

    “χ 2 (4) = 385.46,p < .01”

  • Consistentχ²(4) = 29.99reported p < .01 · recomputed p < .001

    “χ 2 (4) = 29.99,p < .01”

  • Consistentt(888) = 7.44reported p < .01 · recomputed p < .001

    “t (888) = 7.44, p < .01”

  • ConsistentF(4, 329) = 4.61reported p < .01 · recomputed p = .001

    “F (4, 329) = 4.61, p < .01”

  • ConsistentF(4, 329) = 4.58reported p < .01 · recomputed p = .001

    “F (4, 329) = 4.58, p < .01”

  • ConsistentF(3, 769) = 371.46reported p < .01 · recomputed p < .001

    “F (3, 769) = 371.46,p < .01”

  • Consistentt(592) = 0.69reported p = .49 · recomputed p = .490

    “t (592) = 0.69, p = .49”

23 of 23

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Participants reported as failing both checks outnumber those failing the second check

    Minor

    The preliminary-study table labels 14.00% of MTurk participants as failing both questions, although only 3.50% failed the second. Its "Failed both" column cannot represent participants who failed both checks.

  • ProA is labeled mostly U.S. despite its reported locations

    Minor

    Study 1 reports only 30% of ProA participants in North America and 56% in the U.K. and Europe combined, contradicting the summary table's "Mostly U.S." designation.

2 of 2

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Peer, E., Brandimarte, L., Samat, S., & Acquisti, A.
Affiliations
  • University of Arizona (US)
  • Carnegie Mellon University (US)
ORCID
Venue
Journal of Experimental Social Psychology
Publisher
Elsevier BV
Published
2017-05
Volume
70
Pages
153-163
ISSN
0022-1031 (ISSN-L)
DOI
10.1016/j.jesp.2017.01.006
OpenAlex
W2585292421
Semantic Scholar
ca9f3a04a1b6524105f51676abdd198b118a7d18
Cited by
2,957
References
31
Citations by year
2026: 317 · 2025: 371 · 2024: 429 · 2023: 400 · 2022: 446 · 2021: 405 · 2020: 256 · 2019: 193 · 2018: 86 · 2017: 50 · 2016: 2 · 2014: 1
FWCI
192.46
Type
journal article
Language
en
Open access
Licence
www.elsevier.com/tdm/userlicense/1.0/ (tdm)
Locations
  • Journal of Experimental Social Psychology (publishedVersion)
  • UA Campus Repository (The University of Arizona) (acceptedVersion)
  • Figshare (submittedVersion, cc-by-nc-nd)
  • KiltHub Repository
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
  • Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
  • Social Sciences › Safety Research › Experimental Behavioral Economics Studies
Keywords
crowdsourcing, Amazon Mechanical Turk, CrowdFlower, dishonesty, response rate, replication, data quality, participant diversity, online research platform

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review