CoChat
Sign Up
Cited by 4,008Mobile Crowdsensing and CrowdsourcingSurvey Methodology and NonresponseNames, Identity, and Discrimination Research

Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk

Open access
Berinsky, Adam J., Huber, Gregory A., Lenz, Gabriel S.·2012·Political Analysis
DOI 10.1093/pan/mpr057
Open this paper in CoChat — free accountOpen PDF

Abstract

We examine the trade-offs associated with using Amazon.com's Mechanical Turk (MTurk) interface for subject recruitment. We first describe MTurk and its promise as a vehicle for performing low-cost and easy-to-field experiments. We then assess the internal and external validity of experiments performed using MTurk, employing a framework that can be used to evaluate other subject pools. We first investigate the characteristics of samples drawn from the MTurk population. We show that respondents recruited in this manner are often more representative of the U.S. population than in-person convenience samples—the modal sample in published experimental political science—but less representative than subjects in Internet-based panels or national probability samples. Finally, we replicate important published experimental work using MTurk samples.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper evaluates Amazon Mechanical Turk as a subject pool for political science experiments, comparing MTurk samples with convenience, Internet-panel, and national probability samples and replicating three published experiments. It concludes MTurk is inexpensive and often more representative than in-person convenience samples, though younger and more liberal than the U.S. public, with treatment effects generally similar to original studies.

Key findings

  • The MTurk sample (N = 551, February/March 2010) was 60.1% female, mean age 32.3 years, mean education 14.9 years, 83.5% white, and 40.8% Democratic; compared with ANESP/CPS/ANES benchmarks it was younger, lower income, less likely to be married, own a home, or report a religious affiliation, and more ideologically liberal.
  • Replications produced similar effects: the welfare vs. “assistance to the poor” wording gap was 38 percentage points (p < .001) vs. 44% in the 2010 GSS; the Asian disease problem yielded 74% choosing the certain option in the lives-saved frame vs. 38% in the lives-lost frame (p < .001), compared with 72% vs. 22% in the original student sample; the Kam and Simas risk-frame coefficient was 1.18 in MTurk vs. 1.07 in the original article.
  • Habitual participation was modest: respondents reported a mean of 1.7 political surveys in the last month, 39% took no other survey, and 78% took two or fewer; across seven experiments there were 1,574 unique subjects, 70% participated in only one experiment, 18% in two, and 2% in five or more.
  • Internal-validity checks found 14 of 551 responses (2.4%) came from seven IP addresses with two responses each; on an attention check, 60% of MTurk respondents answered correctly vs. 49% on Polimetrix/YouGov and 46% on SSI. Recruitment cost was low: more than 200 subjects per day at $.25 per survey and over 300 per day at $.50–$.75, with a per-respondent cost of $.55 for a 5-min survey at $.50 including Amazon’s 10% surcharge.

Methods

  • Surveyed 551 MTurk workers in February and March 2010, advertised as about 10 min and paid 50 cents; restricted to U.S. residents aged 18+ with prior approval rates of at least 95%, and excluded 32 respondents (5.8%) who did not take the survey from U.S. IP addresses. Additional MTurk surveys included N = 699 in May 2011 for need-for-cognition/evaluation measures, N = 450 in January 2010 for the Asian disease replication, and N = 329 for the welfare wording experiment.
  • Benchmarked MTurk against convenience samples from APSR, AJPS, and JOP from January 2005 to June 2010 (961 articles, 51 experimental, 44 using U.S. subjects exclusively, more than half using convenience samples), the ANESP Internet panel, CPS 2008, and ANES 2008, using identical question wording where possible.
  • Replicated three experiments (Rasinski welfare wording, Tversky and Kahneman Asian disease, Kam and Simas framing/risk) and used descriptive comparisons, significance tests, ordered probits for heterogeneous treatment effects, and coefficient comparisons.

Limitations

  • Stated: MTurk subjects are notably younger and more ideologically liberal than the public, which may limit suitability for some research topics; they appear to pay more attention to tasks; and as use increases, habitual responding may pose more of an external validity problem. Demand characteristics may also be greater, and future work is needed to assess how serious this is for MTurk.
  • Stated: the MTurk sample does not perfectly match the U.S. population; if treatment effects differ by factors for which MTurk is less representative (e.g., older or more conservative citizens), estimates may be inaccurate for the larger population, and large samples may be needed to estimate subgroup differences.
  • Evident: the sample is restricted to U.S. adults with at least 95% approval rates, so findings may not generalize to all MTurk workers; comparisons rely on unweighted ANESP results and self-reported measures; IP-address checks cannot definitively identify repeat takers, and the paper notes that IP locator accuracy is unbenchmarked.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1093/pan/mpr057

Cites

37 of 37 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 19 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    At higher pay rates between $.50 and $.75 per completion, over 300 subjects per day can be recruited.

    “At higher pay rates-between $.50 and $.75 per completion-we have been able to recruit over 300 subjects per day.”
    Quote in the paperSubstantial claim
  • Findingown data

    When payments were lower (e.g., $.15 per HIT), recruitment was somewhat slower.

    “When payments were lower (e.g., $.15 per HIT), recruitment was somewhat slower.”
    Quote in the paperNot a substantial claim
  • Findingown data

    For short surveys advertised as taking between 2 and 4 min, more than 200 subjects per day can be obtained when paying as little as $.25 per survey.

    “for short surveys advertised as taking between 2 and 4 min, we have been able to obtain more than 200 subjects per day when paying as little as $.25 per survey.”
    Quote in the paperSubstantial claim
  • Findingown data

    Estimates of average treatment effects from MTurk samples are similar to those from the original convenience and nationally representative samples in replicated published experiments.

    “Second, we replicate experimental studies previously conducted using convenience and nationally representative samples, finding that the estimates of average treatment effects are similar in the MTurk and original samples.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The original published experiments used as replication benchmarks provide valid estimates of treatment effects for their samples.
  • Findingreasoning

    The MTurk interface provides a mechanism for performing panel studies.

    “The MTurk interface provides a mechanism for performing these sorts of panel studies.”
    Quote in the paperNot a substantial claim
  • May combine two claims. 49
  • The quote only partly states this claim. 1
  • Classified as probably not a substantial claim. 15

  • May combine two claims.

    Findingown data

    Relative to other experimental pools, MTurk is extremely inexpensive both in terms of the cost of subjects and the time required to implement studies.

    “Relative to other experimental pools, MTurk is extremely inexpensive both in terms of the cost of subjects and the time required to implement studies.”
    • Also stated in: § 2.1 Cost per Subject and Ease of Recruitment · plain
      “MTurk is, in short, extremely inexpensive relative to nearly every alternative other than uncompensated students.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    The highest pay rate used on MTurk of $.50 for a 5-min survey (effective hourly rate $6.00) is associated with a per-respondent cost of $.55 including Amazon's 10% surcharge, or $.11 per survey minute.

    “even the highest pay rate we have used on MTurk of $.50 for a 5-min survey (an effective hourly rate of $6.00) is still associated with a per-respondent cost of $.55 (including Amazon.com's10% surcharge) or $.11 per survey minute.”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data

    The demographic characteristics of domestic MTurk users are more representative and diverse than the corresponding student and convenience samples typically used in experimental political science studies.

    “We show that respondents recruited in this manner are often more representative of the U.S. population than in-person convenience samples-the modal sample in published experimental political science-but less representative than subjects in Internet-based panels or national probability samples.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.
    • Rests on an assumption: MTurk workers' self-reported demographic and political characteristics are accurate.

  • May combine two claims.

    Findingown data

    Concerns about heterogeneous treatment effects, subject attentiveness, and the prevalence of habitual survey takers are not large problems in practice for MTurk.

    “Third, we find that potential limitations to using MTurk to recruit subjects and conduct research-in particular, concerns about heterogeneous treatment effects, subject attentiveness, and the prevalence of habitual survey takers-are not large problems in practice.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.

  • May combine two claims.

    Findingown data

    In racial composition, the MTurk sample is much less white than the student sample by Kam et al. but similar to the adult sample by Kam et al. and the Ann Arbor sample by Berinsky and Kinder.

    “Finally, in terms of racial composition, the MTurk sample is much less white than the student sample by Kam et al. (which is again very different from national samples; see Table 3) but similar to the adult sample by Kam et al. and the Ann Arbor sample by Berinsky and Kinder.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.
    • Rests on an assumption: MTurk workers' self-reported demographic and political characteristics are accurate.

  • May combine two claims.

    Findingown data

    The MTurk sample has a similar high education level as compared to the two samples by Berinsky and Kinder, whereas the adult sample reported by Kam et al. is much less educated.

    “The MTurk sample has a similar (high) education level as compared to the two samples by Berinsky and Kinder”
    • Also stated in: § 14 · plain
      “reported by Kam et al. is much less educated (even compared to national probability samples; see Table 3).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.
    • Rests on an assumption: MTurk workers' self-reported demographic and political characteristics are accurate.

  • May combine two claims.

    Findingnone given

    The MTurk and student samples are similar in gender distribution, but the adult sample reported by Kam et al. and the Ann Arbor samples from Berinsky and Kinder are substantially more female.

    “The MTurk and student samples are similar in terms of gender distribution, but the adult sample reported by Kam et al. and the Ann Arbor samples from Berinsky and Kinder are substantially more female.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.
    • Rests on an assumption: MTurk workers' self-reported demographic and political characteristics are accurate.

  • May combine two claims.

    Findingnone given

    Relative to an average student sample, the MTurk population is substantially older, but it is younger than any of the three nonstudent adult samples.

    “relative to an average student sample, the MTurk population is substantially older, but it is younger than any of the three nonstudent adult samples.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.
    • Rests on an assumption: MTurk workers' self-reported demographic and political characteristics are accurate.

  • May combine two claims.

    Findingown data

    MTurk respondents are somewhat more interested in politics than ANESP respondents, and both samples are considerably more interested than ANES respondents.

    “MTurk respondents are also somewhat more interested in politics than ANESP respondents are, and both samples are considerably more interested than are ANES respondents.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.
    • Rests on an assumption: The American National Election 2008-09 Panel Study (ANESP) is a high-quality Internet sample and a valid benchmark for comparing MTurk.
    • Rests on an assumption: The Current Population Survey and the 2008 American National Election Study are the gold standard for survey sampling.

  • May combine two claims.

    Findingown data

    About a quarter of MTurk subjects correctly answered the vice presidential question, exactly what one would expect by chance.

    “About a quarter did correctly answer the vice presidential question (Thomas Marshall), exactly what one would expect by chance.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.

  • May combine two claims.

    Findingown data

    On both tax increase items, MTurk subjects are only a few percentage points more liberal than ANESP, and these differences are not statistically significant.

    “On both tax increase items, MTurk subjects are only a few percentage points more liberal in their views (and these differences are not statistically significant).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.
    • Rests on an assumption: The American National Election 2008-09 Panel Study (ANESP) is a high-quality Internet sample and a valid benchmark for comparing MTurk.

  • May combine two claims.

    Findingnone given

    The MTurk sample is broadly similar to the other samples in region of residence, with perhaps a slightly larger prevalence of those living in the Northeast.

    “the MTurk sample is broadly similar to the other samples in region of residence, with perhaps a slightly larger prevalence of those living in the Northeast.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.
    • Rests on an assumption: The American National Election 2008-09 Panel Study (ANESP) is a high-quality Internet sample and a valid benchmark for comparing MTurk.
    • Rests on an assumption: The Current Population Survey and the 2008 American National Election Study are the gold standard for survey sampling.

  • May combine two claims.

    Findingown data

    MTurk responses match the ANES well on universal health care-about 50% of both samples support it-whereas ANESP is less supportive at 42%.

    “The MTurk responses match the ANES well on universal health care-about 50% of both samples support it-whereas those in the ANESP are somewhat less supportive at 42%.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.
    • Rests on an assumption: The American National Election 2008-09 Panel Study (ANESP) is a high-quality Internet sample and a valid benchmark for comparing MTurk.
    • Rests on an assumption: The Current Population Survey and the 2008 American National Election Study are the gold standard for survey sampling.

  • May combine two claims.

    Findingnone given

    On race, MTurk is similar to the CPS on percent white, but underrepresents blacks and Hispanics.

    “On race, MTurk's characteristics are mixed: It is similar to the CPS on percent white, but underrepresents blacks and Hispanics.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.
    • Rests on an assumption: The Current Population Survey and the 2008 American National Election Study are the gold standard for survey sampling.

  • May combine two claims.

    Findingown data

    For both need for cognition and need to evaluate, the MTurk sample scores higher than the ANESP, which is itself higher than the ANES 2008.

    “Finally, for both need for cognition and need to evaluate, the MTurk sample scores higher than the ANESP, which is itself higher than the ANES 2008.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.
    • Rests on an assumption: The American National Election 2008-09 Panel Study (ANESP) is a high-quality Internet sample and a valid benchmark for comparing MTurk.
    • Rests on an assumption: The Current Population Survey and the 2008 American National Election Study are the gold standard for survey sampling.

  • May combine two claims.

    Findingreasoning

    Consistent with the age gap, both mean and median income in the MTurk sample are lower than in the other samples.

    “Consistent with this age gap, both mean and median income are lower than in the other samples.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.
    • Rests on an assumption: The American National Election 2008-09 Panel Study (ANESP) is a high-quality Internet sample and a valid benchmark for comparing MTurk.

  • May combine two claims.

    Findingown data

    MTurk respondents are slightly more Democratic in partisan identification than ANESP respondents and substantially more liberal in ideology.

    “MTurk respondents are slightly more Democratic in their partisan identification than are ANESP respondents and are substantially more liberal in their ideology”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.
    • Rests on an assumption: The American National Election 2008-09 Panel Study (ANESP) is a high-quality Internet sample and a valid benchmark for comparing MTurk.

  • May combine two claims.

    Findingown data

    MTurk subjects appear more knowledgeable than ANESP respondents, but the gap is not large (statistically significant in four of six cases).

    “Based on their responses, MTurk subjects appear more knowledgeable than ANESP respondents, but the gap is not large (though it is statistically significant in four of six cases).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.
    • Rests on an assumption: The American National Election 2008-09 Panel Study (ANESP) is a high-quality Internet sample and a valid benchmark for comparing MTurk.

  • May combine two claims.

    Findingown data

    Compared to the ANESP, MTurk subjects express somewhat more liberal views on all three additional policy items, with only 16% supporting a constitutional amendment banning gay marriage, compared to 31% in the ANESP.

    “Compared to the ANESP, MTurk subjects express somewhat more liberal views on all three items, with only 16% supporting a constitutional amendment banning gay marriage, compared to 31% in the ANESP”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.
    • Rests on an assumption: The American National Election 2008-09 Panel Study (ANESP) is a high-quality Internet sample and a valid benchmark for comparing MTurk.

  • May combine two claims.

    Findingnone given

    MTurk subjects are more likely to have never married (51%), rent rather than own their home (53%), and report no religious affiliation (42%).

    “MTurk subjects are more likely to: have never married (51%), rent rather than own their home (53%), and report no religious affiliation (42%).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.
    • Rests on an assumption: MTurk workers' self-reported demographic and political characteristics are accurate.

  • May combine two claims.

    Findingown data

    In the MTurk Experiment 1, 17% said too little was being spent on welfare, whereas 55% said too little was being spent on assistance to the poor.

    “Only 17% of MTurkers said too little was being spent on welfare, whereas 55% said too little was being spent on assistance to the poor.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    In the 2010 GSS, 24% think too little is being spent on welfare, whereas 68% think too little is spent on assistance to the poor.

    “Specifically, in 2010, the GSS data show that 24% think that too little is being spent on welfare, whereas 68% think that too little is spent on assistance to the poor.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In both the 2010 GSS and the MTurk sample, the experimental effect does not differ reliably across gender, education, or racial lines.

    “In both the 2010 GSS and the MTurk sample, the experimental effect does not differ reliably across gender, education, or racial lines.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The MTurk and GSS samples are comparable enough to pool in ordered probits testing heterogeneous treatment effects.

  • May combine two claims.

    Findingown data

    Higher levels of risk acceptance are associated with greater support for the risky policy option in both experiments, and the coefficients are similar across all three specifications.

    “Additionally, higher levels of risk acceptance are associated with greater support for the risky policy option in both experiments and the coefficients are also similar across all three specifications.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In the MTurk survey, the mean number of political surveys taken in the last month on MTurk was 1.7; 39% took no other survey, and 78% took two or fewer surveys.

    “The mean was 1.7; 39% of the respondents took no other survey, whereas 78% took two or fewer surveys.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: MTurk workers' self-reported number of political surveys taken in the last month is accurate.

  • May combine two claims.

    Findingown data

    In the MTurk sample, 74% pick the certain choice in the lives saved condition and 38% select the certain choice in the lives lost condition, a statistically significant difference (p < .001).

    “We find a similar pattern among our MTurk sample: 74% pick the certain choice in the lives saved condition and 38% select the certain choice in the lives lost condition ( p value of test of statistical significance of difference < .001).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Across seven MTurk experiments conducted from January 2010 to April 2010, there were 1,574 unique subjects; 70% participated in only one experiment, 18% in two, and 2% in five or more.

    “Across the seven experiments, there were a total of 1,574 unique subjects.Of these subjects, 70% participated in only one experiment; another 18% participated in two experiments.Only 2% of the subjects participated in five or more experiments.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The seven MTurk studies examined are representative enough of MTurk experiments to support broader trends about habitual participation.
    • Rests on an assumption: Unique MTurk ID numbers identify unique individuals.

  • May combine two claims.

    Findingown data

    In a study by one of the authors, 60% of MTurk respondents answered a political office question correctly, compared with 49% on Polimetrix/YouGov and 46% on SSI.

    “On the MTurk study, 60% of the respondents answered the question correctly.An identical question concerning the same article was also included on experiments run through Polimetrix/YouGov (another high-quality Internet panel) and with a sample collected by Survey Sampling International (SSI).The correct answer rates on these platforms were markedly lower than in the MTurk sample-49% on Polimetrix/”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The multiple-choice political office question is a valid measure of attention to experimental instruments.

  • May combine two claims.

    Findingown data

    In the experiments presented above, the effects did not differ statistically or substantively when habitual and nonhabitual respondents were examined separately.

    “In the experiments presented above, we found that the effects did not differ-in either a statistical or a substantive sense-when we examined the habitual respondents and the nonhabitual respondents separately.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Other convenience samples, such as student or local intercept samples, may also have significant numbers of habitual experimental participants.

    “It should be noted that other convenience samples, such as student or local intercept samples, may also have significant numbers of habitual experimental participants.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Per subject costs for typical undergraduate samples are about $5-10, for nonstudent campus samples about $30, and for temporary agency subjects between $15 and $20.

    “per subject costs for typical undergraduate samples are about $5-10, for nonstudent campus samples about $30 (Kam, Wilking, and Zechmeister 2007), and for temporary agency subjects between $15 and $20.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Kam, Wilking, and Zechmeister 2007

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    In designing the MTurk survey, the authors followed the ANESP as closely as possible, using identical question wordings and branching formats.

    “Additionally, in designing our MTurk survey, we followed the ANESP as closely as possible, using identical question wordings and branching formats.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    In the MTurk Experiment 1, respondents were randomly assigned to either the welfare or the assistance to the poor version of the question.

    “Respondents were randomly assigned to either the welfare or assistance to the poor version of the question.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The MTurk sample does not perfectly match the demographic and attitudinal characteristics of the U.S. population but does not present a wildly distorted view of the U.S. population either.

    “All told, these comparisons reinforce the conclusion that the MTurk sample does not perfectly match the demographic and attitudinal characteristics of the U.S. population but does not present a wildly distorted view of the U.S. population either.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.
    • Rests on an assumption: The American National Election 2008-09 Panel Study (ANESP) is a high-quality Internet sample and a valid benchmark for comparing MTurk.
    • Rests on an assumption: The Current Population Survey and the 2008 American National Election Study are the gold standard for survey sampling.
    • Rests on an assumption: MTurk workers' self-reported demographic and political characteristics are accurate.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Even though welfare and assistance to the poor are thought by policy experts to refer to the same policy, the study found important differences in levels of support between the two question forms.

    “Even though welfare and assistance to the poor are thought by policy experts to refer to the same policy, the study found important differences in levels of support between the two question forms.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Rasinski 1989

      Not checked in the free run.

    • Rests on an assumption: The original published experiments used as replication benchmarks provide valid estimates of treatment effects for their samples.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Although 20%-25% of respondents in each year said too little was being spent on welfare, 63%-65% said too little was being spent on assistance to the poor.

    “Although 20%-25% of the respondents in each year said that too little was being spent on welfare, 63%-65% said that too little was being spent on assistance to the poor (Rasinski 1989, 391).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Rasinski 1989

      Not checked in the free run.

    • Rests on an assumption: The original published experiments used as replication benchmarks provide valid estimates of treatment effects for their samples.

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Given the relative dearth of older or more conservative individuals in the MTurk pool, large sample sizes may be necessary to obtain sufficient diversity on these dimensions to estimate differences in treatment effects for these groups.

    “Furthermore, given the relative dearth of these sorts of individuals in the MTurk pool, large sample sizes may be necessary to obtain sufficient diversity on these dimensions to estimate differences in treatment effects for these groups”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.

  • Classified as probably not a substantial claim.

    Premisecitation

    Rasinski (1989) reports results from a question wording experiment that asked representative samples from the General Social Surveys (GSS) from 1984 to 1986 whether too much or too little was being spent on welfare or assistance to the poor.

    “Rasinski (1989) reports results from a question wording experiment that asked representative samples from the General Social Surveys (GSS) from 1984 to 1986 whether too much or too little was being spent on either ''welfare'' or ''assistance to the poor.''”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Rasinski (1989)

      Not checked in the free run.

    • Rests on an assumption: The original published experiments used as replication benchmarks provide valid estimates of treatment effects for their samples.

  • May combine two claims.

    Interpretationreasoning

    MTurk samples will often be more diverse than convenience samples and will always be more diverse than student samples.

    “MTurk samples will often be more diverse than convenience samples and will always be more diverse than student samples.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The GSS has continued to ask the spending experiment and the gap in support for increasing spending between the question forms remains similar over time, ranging from 28% to 50%, with an average difference of 37%.

    “The GSS has continued to ask the spending experiment and the gap in support for increasing spending between the question forms remains similar over time, ranging from 28% to 50%, with an average difference of 37% (Green and Kern 2010).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Green and Kern 2010

      Not checked in the free run.

    • Rests on an assumption: The original published experiments used as replication benchmarks provide valid estimates of treatment effects for their samples.

  • May combine two claims.

    Interpretationown data

    Statistically significant differences exist between the MTurk sample and the benchmark surveys, but these differences are substantively small.

    “Statistically significant differences exist between the MTurk sample and the benchmark surveys, but these differences are substantively small.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.
    • Rests on an assumption: The American National Election 2008-09 Panel Study (ANESP) is a high-quality Internet sample and a valid benchmark for comparing MTurk.
    • Rests on an assumption: The Current Population Survey and the 2008 American National Election Study are the gold standard for survey sampling.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given

    The GSS is a nationally representative face-to-face interview sample with characteristics similar to the ANES face-to-face surveys.

    “The GSS is a nationally representative face-to-face interview sample with characteristics similar to the ANES face-to-face surveys.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The original published experiments used as replication benchmarks provide valid estimates of treatment effects for their samples.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Prior work finds no evidence that gender or race are associated with differences in effect sizes in the GSS in years earlier than 2010.

    “Prior work similarly finds no evidence that gender or race are associated with differences in effect sizes in the GSS in years earlier than 2010 (Green and Kern 2010).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Green and Kern 2010

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Framing outcomes in positive terms produced a reversal of participants' preferences for the two programs compared to negative terms.

    “Framing the outcomes in positive terms therefore produced a reversal of participants' preferences for the two programs compared to when it was presented in negative terms.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Tversky and Kahneman (1981)

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Tversky and Kahneman report that when the problem was framed in terms of lives saved, respondents were more likely to pick the certain choice, while when framed in terms of lives lost, respondents were more likely to pick the risky choice.

    “Tversky and Kahneman report that, when the problem was framed in terms of ''lives saved,'' respondents were more likely to pick the certain choice, while when it was framed in terms of ''lives lost,'' respondents were more likely to pick the risky choice.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Tversky and Kahneman (1981)

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The Asian Disease Problem experiment was first conducted using a student sample but has been replicated in many other settings.

    “This experiment was first conducted using a student sample but has been replicated in many other settings (e.g., Takemura 1994; Kuhberger 1995; Jou, Shanteau, and Harris 1996; Bless, Betsch, and Franzen 1998; Druckman 2001).”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Takemura 1994

      Not checked in the free run.

    • cites Kuhberger 1995

      Not checked in the free run.

    • cites Jou, Shanteau, and Harris 1996

      Not checked in the free run.

  • May combine two claims.

    Interpretationnone given

    The first two studies replicated were both straightforward experiments with large effect sizes.

    “The first two studies we replicated were both straightforward experiments with large effect sizes.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    A final source of concern about external validity for any self-selected sample is the potential for habitual customers; if the same subjects take several surveys, there is potential for cross-experiment stimuli contamination.

    “A final source of concern about external validity for any self-selected sample is the potential for ''habitual customers.''If the same subjects take several surveys, there is the potential for cross-experiment stimuli contamination (Transue, Lee, and Aldrich 2009).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Transue, Lee, and Aldrich 2009

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Kam and Simas used a modified version of the Tversky and Kahneman framing experiment and an original scale measuring individual differences in risk orientation to explore how frame and risk proclivities affected choice under uncertainty.

    “Kam and Simas used a modified version of the Tversky and Kahneman framing experiment and an original scale measuring individual differences in risk orientation to explore how both the frame of a problem and variation in risk proclivities affected choice under uncertainty.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Kam and Simas (2010)

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    In the original Tversky and Kahneman experiment with student samples, 72% picked the certain choice in the lives saved condition, compared to 22% in the lives lost condition.

    “In the original Tversky and Kahneman experiment, which was run with student samples, 72% of respondents picked the certain choice in the ''lives saved'' condition, as compared to 22% who picked the certain choice in the ''lives lost'' condition.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Tversky and Kahneman (1981)

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    The MTurk sample shows the large preference reversal reported by Tversky and Kahneman, although the gap is smaller than in the student sample.

    “Thus, although the gap is smaller than in the student sample, we still observe the large preference reversal reported in Tversky and Kahneman (and replicated in subsequent experiments with different samples).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Tversky and Kahneman

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    Overall, the basic pattern of effects is the same between the original Kam and Simas study and the MTurk replication, although the MTurk estimate of the predictive power of risk assessment is modestly larger.

    “Overall, although our estimate of the predictive power of risk assessment is modestly larger than in the original article, the basic pattern of effects is the same.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Even if the duplicate IP responses are cases of repeat survey taking, only a handful of responses would be contaminated, suggesting repeat survey taking is not a large problem in the MTurk subject pool.

    “But even if these are cases of repeat survey taking, only a handful of responses would be contaminated, suggesting that repeat survey taking is not a large problem in the MTurk subject pool.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisereasoning

    MTurk Requesters often specify at least a 95% prior approval rate, giving respondents an incentive to read instructions carefully and consider their responses.

    “Since Requesters often specify at least a 95% prior ''approval rate''-that is, previous Requesters accepted 95% or more of the HITs submitted by an individual-respondents have an incentive to read instructions carefully and consider their responses.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Relative to other convenience samples often used in experimental research in political science, MTurk subjects are often more representative of the general population and substantially less expensive to recruit.

    “We demonstrate that relative to other convenience samples often used in experimental research in political science, MTurk subjects are often more representative of the general population and substantially less expensive to recruit.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The MTurk pool is apparently not currently excessively overused, and habitual responding appears to be a minor concern.

    “They are apparently also not currently an excessively overused pool, and habitual responding appears to be a minor concern.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationnone given

    MTurk subjects are notably younger and more ideologically liberal than the public, which may limit their suitability for some research topics.

    “In particular, MTurk subjects are notably younger and more ideologically liberal than the public, which may limit their suitability for some research topics.”
    Quote in the paperNot a substantial claim
57 of 57

  • The paper relies on this without support.

    The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.

    “We surveyed 551 MTurk workers in February and March of 2010.”

  • The paper relies on this without support.

    The American National Election 2008-09 Panel Study (ANESP) is a high-quality Internet sample and a valid benchmark for comparing MTurk.

    “we compare MTurk to a high-quality Internet survey, the American National Election 2008-09 Panel Study (ANESP).”

  • The paper relies on this without support.

    The Current Population Survey and the 2008 American National Election Study are the gold standard for survey sampling.

    “These latter two studies use face-to-face probability samples and are widely considered the ''gold standard'' for survey sampling.”

  • The paper relies on this without support.

    The original published experiments used as replication benchmarks provide valid estimates of treatment effects for their samples.

    “we also replicated the results reported in three experiments.”

  • The paper relies on this without support.

    IP address locators accurately identify U.S. residency.

    “although IP address locators seem reliable, we are unaware of research benchmarking their accuracy.”

  • The paper relies on this without support.

    MTurk workers' self-reported demographic and political characteristics are accurate.

    “we compare measured characteristics of MTurk survey participants to characteristics of participants in three distinct types of research samples”

  • The paper relies on this without support.

    The approval rate threshold of 95% selects a sample comparable to the broader MTurk population.

    “We also excluded individuals with approval rates below 95% on previous MTurk tasks.”

  • The paper relies on this without support.

    MTurk workers' self-reported number of political surveys taken in the last month is accurate.

    “To assess the severity of this problem, we asked our respondents how many political surveys they had taken in the last month on MTurk.”

  • The paper relies on this without support.

    The seven MTurk studies examined are representative enough of MTurk experiments to support broader trends about habitual participation.

    “Although this set of experiments represents only a small proportion of those conducted on MTurk, our findings may illuminate broader trends.”

  • The paper relies on this without support.

    Unique MTurk ID numbers identify unique individuals.

    “We gathered the unique MTurk ID number for all workers who participated in each of seven studies we conducted from January 2010 to April 2010.”

  • The paper relies on this without support.

    The multiple-choice political office question is a valid measure of attention to experimental instruments.

    “In a study conducted by one of the authors, subjects were asked to identify the political office held by a person mentioned in a story they had just read.The format of this question was a multiple-choice item with five possible responses.”

  • The paper relies on this without support.

    The GSS welfare/assistance question wording experiment provides a valid benchmark for comparing MTurk treatment effects.

    “We found a statistically significant 38 percentage point gap ( p value < .001) between the two conditions that is similar in magnitude to that found using the GSS (the comparable gap was 44% in the 2010 GSS).”

  • The paper relies on this without support.

    The MTurk and GSS samples are comparable enough to pool in ordered probits testing heterogeneous treatment effects.

    “To conduct these tests, we pooled the MTurk and GSS samples.”
13 of 13

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The Internet has opened the way for improving standard surveys, experiments, and health reporting.

    “The Internet has opened the way for improving standard surveys, experiments, and health reporting ( 35).”

    From The Parable of Google Flu: Traps in Big Data Analysis (2014) · cites this paper as [35]

    Section 2 describes using an Internet platform to run surveys with embedded experiments, and the Conclusion says MTurk can make experimental research easier and less expensive. The paper does not address improvements to health reporting; the disease scenario in section 4.3.2 is a framing experiment, not health reporting. Page numbers are unavailable in our copy. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Research has shown that data from MTurk workers is comparable to data from other pools (e.g., commercial panels, social media, colleges), but recently there has been a decrease in data quality.

    “Research has shown data from MTurk workers is comparable to data from other pools (e.g., commercial panels, social media, colleges) [12, 20, 23, 32, 69, 80], but recently there has been a decrease in data quality [28, 49, 66, 68, 104, 108].”

    From "I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust (2024) · cites this paper as [12]

  • Online platforms that distribute tasks across workers, such as Amazon's Mechanical Turk, can make acquiring data efficient.

    “online platforms that distribute tasks across workers, such as Amazon's Mechanical Turk, can make acquiring data efficient (Berinsky, Huber, and Lenz 2012).”

    From Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts (2013) · cites this paper as Berinsky, Huber, and Lenz 2012

  • Amazon Mechanical Turk workers appear more representative of the general population than the typical first year psychology students comprising many reviewed studies.

    “This study will re-examine these assertions based on two samples of Amazon Mechanical Turk workers, who appear more representative of the general population than are the typical first year psychology students comprising many of the studies reviewed in the preceding (see e.g., Berinsky et al., 2012).”

    From A psychometric analysis of the reading the mind in the eyes test: toward a brief form for research and applied settings (2015) · cites this paper as Berinsky et al., 2012

  • MTurk is not representative, particularly when it comes to self-reported political identity and ideology.

    “and is, in fact, not representative, particularly when it comes to self-reported political identity and ideology51–53”

    From Accuracy prompts are a replicable and generalizable approach for reducing the spread of misinformation (2022) · cites this paper as [53]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Replication survey has conflicting field dates

    Minor

    The text identifies the May 2011 survey as containing the Kam and Simas replication, while Table 6 dates that replication survey to May 2010.

  • Reported percentage of responses from shared IP addresses is too low

    Minor

    Fourteen of 551 responses is 2.54%, which rounds to 2.5%, not 2.4%.

    Reported 2.4 · recomputed 2.540835

2 of 2

Cites

Flagged first, then by citations

1 of the 38 references couldn’t be matched to a known paper.

Research works printed but not linked: 4

  1. The MTurk platform is of course limited to conducting research that does not require physical interactions between the subject and either the researcher or other subjects (e.g., to gather DNA samples, administer physical interventions, or observe face-to-face interactions among subjects). © The Author 2012. Published by Oxford University Press on behalf of the Society for Political Methodology. All rights reserved. For Permissions, please email: journals.permissions@oup.com Adam J. Berinsky et al.
  2. Ross, 2010, CHI EA 2010
  3. Snow, 2008, Evaluating non-expert annotations for natural language tasks
  4. Sheng, 2008, Get another label?

Cited by

Sorted by citations

Related

Authors
Berinsky, A. J., Huber, G. A., & Lenz, G. S.
Affiliations
  • Massachusetts Institute of Technology (US)
  • Yale University (US)
  • University of California, Berkeley (US)
Corresponding author
Berinsky, Adam J.
ORCID
Venue
Political Analysis
Publisher
Cambridge University Press (CUP)
Published
2012
Volume
20
Issue
3
Pages
351-368
ISSN
  • 1047-1987 (ISSN-L)
  • 1476-4989
DOI
10.1093/pan/mpr057
OpenAlex
W2119298903
Semantic Scholar
ff6a7eac8dc9e2a529f5299a9117c741b30c2aee
Cited by
4,008
References
38
Citations by year
2026: 107 · 2025: 134 · 2024: 166 · 2023: 232 · 2022: 361 · 2021: 393 · 2020: 421 · 2019: 443 · 2018: 462 · 2017: 400 · 2016: 312 · 2015: 278 · 2014: 173 · 2013: 93 · 2012: 20
FWCI
290.34
Type
journal article
Language
en
Open access
Licence
www.cambridge.org/core/terms (unspecified, 1830 d)
Locations
  • Political Analysis (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
  • Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
  • Social Sciences › Sociology and Political Science › Names, Identity, and Discrimination Research
Keywords
Amazon Mechanical Turk, digital labour markets, participant recruitment, internal validity, generalizability, sample representativeness, convenience samples, replication, online panels, probability samples

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review