The DOI matches its Crossref record; 37 of the 38 references it cites were found in a registry or an index; none of the 19 sources its claims cite is checked for support yet.
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Open accessAbstract
We examine the trade-offs associated with using Amazon.com's Mechanical Turk (MTurk) interface for subject recruitment. We first describe MTurk and its promise as a vehicle for performing low-cost and easy-to-field experiments. We then assess the internal and external validity of experiments performed using MTurk, employing a framework that can be used to evaluate other subject pools. We first investigate the characteristics of samples drawn from the MTurk population. We show that respondents recruited in this manner are often more representative of the U.S. population than in-person convenience samples—the modal sample in published experimental political science—but less representative than subjects in Internet-based panels or national probability samples. Finally, we replicate important published experimental work using MTurk samples.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper evaluates Amazon Mechanical Turk as a subject pool for political science experiments, comparing MTurk samples with convenience, Internet-panel, and national probability samples and replicating three published experiments. It concludes MTurk is inexpensive and often more representative than in-person convenience samples, though younger and more liberal than the U.S. public, with treatment effects generally similar to original studies.
Key findings
- The MTurk sample (N = 551, February/March 2010) was 60.1% female, mean age 32.3 years, mean education 14.9 years, 83.5% white, and 40.8% Democratic; compared with ANESP/CPS/ANES benchmarks it was younger, lower income, less likely to be married, own a home, or report a religious affiliation, and more ideologically liberal.
- Replications produced similar effects: the welfare vs. “assistance to the poor” wording gap was 38 percentage points (p < .001) vs. 44% in the 2010 GSS; the Asian disease problem yielded 74% choosing the certain option in the lives-saved frame vs. 38% in the lives-lost frame (p < .001), compared with 72% vs. 22% in the original student sample; the Kam and Simas risk-frame coefficient was 1.18 in MTurk vs. 1.07 in the original article.
- Habitual participation was modest: respondents reported a mean of 1.7 political surveys in the last month, 39% took no other survey, and 78% took two or fewer; across seven experiments there were 1,574 unique subjects, 70% participated in only one experiment, 18% in two, and 2% in five or more.
- Internal-validity checks found 14 of 551 responses (2.4%) came from seven IP addresses with two responses each; on an attention check, 60% of MTurk respondents answered correctly vs. 49% on Polimetrix/YouGov and 46% on SSI. Recruitment cost was low: more than 200 subjects per day at $.25 per survey and over 300 per day at $.50–$.75, with a per-respondent cost of $.55 for a 5-min survey at $.50 including Amazon’s 10% surcharge.
Methods
- Surveyed 551 MTurk workers in February and March 2010, advertised as about 10 min and paid 50 cents; restricted to U.S. residents aged 18+ with prior approval rates of at least 95%, and excluded 32 respondents (5.8%) who did not take the survey from U.S. IP addresses. Additional MTurk surveys included N = 699 in May 2011 for need-for-cognition/evaluation measures, N = 450 in January 2010 for the Asian disease replication, and N = 329 for the welfare wording experiment.
- Benchmarked MTurk against convenience samples from APSR, AJPS, and JOP from January 2005 to June 2010 (961 articles, 51 experimental, 44 using U.S. subjects exclusively, more than half using convenience samples), the ANESP Internet panel, CPS 2008, and ANES 2008, using identical question wording where possible.
- Replicated three experiments (Rasinski welfare wording, Tversky and Kahneman Asian disease, Kam and Simas framing/risk) and used descriptive comparisons, significance tests, ordered probits for heterogeneous treatment effects, and coefficient comparisons.
Limitations
- Stated: MTurk subjects are notably younger and more ideologically liberal than the public, which may limit suitability for some research topics; they appear to pay more attention to tasks; and as use increases, habitual responding may pose more of an external validity problem. Demand characteristics may also be greater, and future work is needed to assess how serious this is for MTurk.
- Stated: the MTurk sample does not perfectly match the U.S. population; if treatment effects differ by factors for which MTurk is less representative (e.g., older or more conservative citizens), estimates may be inaccurate for the larger population, and large samples may be needed to estimate subgroup differences.
- Evident: the sample is restricted to U.S. adults with at least 95% approval rates, so findings may not generalize to all MTurk workers; comparisons rely on unweighted ANESP results and self-reported measures; IP-address checks cannot definitively identify repeat takers, and the paper notes that IP locator accuracy is unbenchmarked.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 of 19 cited sources checked · 0 supported · 0 flagged
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“At higher pay rates-between $.50 and $.75 per completion-we have been able to recruit over 300 subjects per day.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 2.1 Cost per Subject and Ease of Recruitment
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When payments were lower (e.g., $.15 per HIT), recruitment was somewhat slower.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 2.1 Cost per Subject and Ease of Recruitment
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“for short surveys advertised as taking between 2 and 4 min, we have been able to obtain more than 200 subjects per day when paying as little as $.25 per survey.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 2.1 Cost per Subject and Ease of Recruitment
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Second, we replicate experimental studies previously conducted using convenience and nationally representative samples, finding that the estimates of average treatment effects are similar in the MTurk and original samples.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 1 Introduction
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Rests on an assumption: The original published experiments used as replication benchmarks provide valid estimates of treatment effects for their samples.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The MTurk interface provides a mechanism for performing these sorts of panel studies.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 2.1 Cost per Subject and Ease of Recruitment
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesThe paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“private survey firms we have worked with charge at least $10 per subject for a 5-min survey when respondents are drawn from an Internet panel.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 2.1 Cost per Subject and Ease of Recruitment
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In a second study, respondents were offered 50 cents for a 3-min follow-up survey conducted 1-3 months after a first-wave interview.Within 8 days, almost 60% of the original respondents took the follow-up.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 2.1 Cost per Subject and Ease of Recruitment
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We examined all issues of the American Political Science Review, the American Journal of Political Science, and the Journal of Politics from January 2005 to June 2010.Of the 961 articles in these issues, 51 used experimental data.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 4.1 Comparison of Respondent Characteristics: Local Convenience Samples
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Local convenience samples are the modal means of subject recruitment among recent published survey and laboratory experimental research in political science.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 4.1 Comparison of Respondent Characteristics: Local Convenience Samples
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Analysis of our two studies suggests that the demographic profile does not change significantly in the follow-up survey.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 2.1 Cost per Subject and Ease of Recruitment
Load-bearingStrength:hedged· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- It may state two separate claims as one. 49
- The quote only partly states this claim. 1
- Probably not a substantial claim. 15
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Relative to other experimental pools, MTurk is extremely inexpensive both in terms of the cost of subjects and the time required to implement studies.”
- Also stated in: § 2.1 Cost per Subject and Ease of Recruitment · plain
“MTurk is, in short, extremely inexpensive relative to nearly every alternative other than uncompensated students.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 1 Introduction
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Also stated in: § 2.1 Cost per Subject and Ease of Recruitment · plain
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“even the highest pay rate we have used on MTurk of $.50 for a 5-min survey (an effective hourly rate of $6.00) is still associated with a per-respondent cost of $.55 (including Amazon.com's10% surcharge) or $.11 per survey minute.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 2.1 Cost per Subject and Ease of Recruitment
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We show that respondents recruited in this manner are often more representative of the U.S. population than in-person convenience samples-the modal sample in published experimental political science-but less representative than subjects in Internet-based panels or national probability samples.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > Abstract
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesThe quote only partly states this claim.
- Rests on an assumption: The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.
- Rests on an assumption: MTurk workers' self-reported demographic and political characteristics are accurate.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Third, we find that potential limitations to using MTurk to recruit subjects and conduct research-in particular, concerns about heterogeneous treatment effects, subject attentiveness, and the prevalence of habitual survey takers-are not large problems in practice.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 1 Introduction
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally, in terms of racial composition, the MTurk sample is much less white than the student sample by Kam et al. (which is again very different from national samples; see Table 3) but similar to the adult sample by Kam et al. and the Ann Arbor sample by Berinsky and Kinder.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 14
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.
- Rests on an assumption: MTurk workers' self-reported demographic and political characteristics are accurate.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The MTurk sample has a similar (high) education level as compared to the two samples by Berinsky and Kinder”
- Also stated in: § 14 · plain
“reported by Kam et al. is much less educated (even compared to national probability samples; see Table 3).”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 4.1 Comparison of Respondent Characteristics: Local Convenience Samples
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.
- Rests on an assumption: MTurk workers' self-reported demographic and political characteristics are accurate.
- Also stated in: § 14 · plain
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The MTurk and student samples are similar in terms of gender distribution, but the adult sample reported by Kam et al. and the Ann Arbor samples from Berinsky and Kinder are substantially more female.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 4.1 Comparison of Respondent Characteristics: Local Convenience Samples
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.
- Rests on an assumption: MTurk workers' self-reported demographic and political characteristics are accurate.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“relative to an average student sample, the MTurk population is substantially older, but it is younger than any of the three nonstudent adult samples.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 4.1 Comparison of Respondent Characteristics: Local Convenience Samples
Strength:plain· not yet calibratedn = 3Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.
- Rests on an assumption: MTurk workers' self-reported demographic and political characteristics are accurate.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“MTurk respondents are also somewhat more interested in politics than ANESP respondents are, and both samples are considerably more interested than are ANES respondents.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 4.2 Comparison of Respondent Characteristics: Internet Samples and High-Quality, Face-To-Face, Probability Samples
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.
- Rests on an assumption: The American National Election 2008-09 Panel Study (ANESP) is a high-quality Internet sample and a valid benchmark for comparing MTurk.
- Rests on an assumption: The Current Population Survey and the 2008 American National Election Study are the gold standard for survey sampling.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“About a quarter did correctly answer the vice presidential question (Thomas Marshall), exactly what one would expect by chance.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 22
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On both tax increase items, MTurk subjects are only a few percentage points more liberal in their views (and these differences are not statistically significant).”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 22
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.
- Rests on an assumption: The American National Election 2008-09 Panel Study (ANESP) is a high-quality Internet sample and a valid benchmark for comparing MTurk.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“the MTurk sample is broadly similar to the other samples in region of residence, with perhaps a slightly larger prevalence of those living in the Northeast.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 4.2 Comparison of Respondent Characteristics: Internet Samples and High-Quality, Face-To-Face, Probability Samples
Strength:hedged· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.
- Rests on an assumption: The American National Election 2008-09 Panel Study (ANESP) is a high-quality Internet sample and a valid benchmark for comparing MTurk.
- Rests on an assumption: The Current Population Survey and the 2008 American National Election Study are the gold standard for survey sampling.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The MTurk responses match the ANES well on universal health care-about 50% of both samples support it-whereas those in the ANESP are somewhat less supportive at 42%.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 22
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.
- Rests on an assumption: The American National Election 2008-09 Panel Study (ANESP) is a high-quality Internet sample and a valid benchmark for comparing MTurk.
- Rests on an assumption: The Current Population Survey and the 2008 American National Election Study are the gold standard for survey sampling.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“On race, MTurk's characteristics are mixed: It is similar to the CPS on percent white, but underrepresents blacks and Hispanics.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 4.2 Comparison of Respondent Characteristics: Internet Samples and High-Quality, Face-To-Face, Probability Samples
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.
- Rests on an assumption: The Current Population Survey and the 2008 American National Election Study are the gold standard for survey sampling.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally, for both need for cognition and need to evaluate, the MTurk sample scores higher than the ANESP, which is itself higher than the ANES 2008.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 4.2 Comparison of Respondent Characteristics: Internet Samples and High-Quality, Face-To-Face, Probability Samples
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.
- Rests on an assumption: The American National Election 2008-09 Panel Study (ANESP) is a high-quality Internet sample and a valid benchmark for comparing MTurk.
- Rests on an assumption: The Current Population Survey and the 2008 American National Election Study are the gold standard for survey sampling.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Consistent with this age gap, both mean and median income are lower than in the other samples.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 4.2 Comparison of Respondent Characteristics: Internet Samples and High-Quality, Face-To-Face, Probability Samples
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.
- Rests on an assumption: The American National Election 2008-09 Panel Study (ANESP) is a high-quality Internet sample and a valid benchmark for comparing MTurk.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“MTurk respondents are slightly more Democratic in their partisan identification than are ANESP respondents and are substantially more liberal in their ideology”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 4.2 Comparison of Respondent Characteristics: Internet Samples and High-Quality, Face-To-Face, Probability Samples
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.
- Rests on an assumption: The American National Election 2008-09 Panel Study (ANESP) is a high-quality Internet sample and a valid benchmark for comparing MTurk.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Based on their responses, MTurk subjects appear more knowledgeable than ANESP respondents, but the gap is not large (though it is statistically significant in four of six cases).”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 4.2 Comparison of Respondent Characteristics: Internet Samples and High-Quality, Face-To-Face, Probability Samples
Strength:hedged· not yet calibratedn = 4Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.
- Rests on an assumption: The American National Election 2008-09 Panel Study (ANESP) is a high-quality Internet sample and a valid benchmark for comparing MTurk.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Compared to the ANESP, MTurk subjects express somewhat more liberal views on all three items, with only 16% supporting a constitutional amendment banning gay marriage, compared to 31% in the ANESP”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 22
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.
- Rests on an assumption: The American National Election 2008-09 Panel Study (ANESP) is a high-quality Internet sample and a valid benchmark for comparing MTurk.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“MTurk subjects are more likely to: have never married (51%), rent rather than own their home (53%), and report no religious affiliation (42%).”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 4.2 Comparison of Respondent Characteristics: Internet Samples and High-Quality, Face-To-Face, Probability Samples
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.
- Rests on an assumption: MTurk workers' self-reported demographic and political characteristics are accurate.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Only 17% of MTurkers said too little was being spent on welfare, whereas 55% said too little was being spent on assistance to the poor.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 4.3.1
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Specifically, in 2010, the GSS data show that 24% think that too little is being spent on welfare, whereas 68% think that too little is spent on assistance to the poor.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 26
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In both the 2010 GSS and the MTurk sample, the experimental effect does not differ reliably across gender, education, or racial lines.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 4.3.1
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: The MTurk and GSS samples are comparable enough to pool in ordered probits testing heterogeneous treatment effects.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Additionally, higher levels of risk acceptance are associated with greater support for the risky policy option in both experiments and the coefficients are also similar across all three specifications.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 4.3.3 Experiment 3: Framing and risk
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The mean was 1.7; 39% of the respondents took no other survey, whereas 78% took two or fewer surveys.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 4.4 Habitual Participants Analysis
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: MTurk workers' self-reported number of political surveys taken in the last month is accurate.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find a similar pattern among our MTurk sample: 74% pick the certain choice in the lives saved condition and 38% select the certain choice in the lives lost condition ( p value of test of statistical significance of difference < .001).”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 27
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Across the seven experiments, there were a total of 1,574 unique subjects.Of these subjects, 70% participated in only one experiment; another 18% participated in two experiments.Only 2% of the subjects participated in five or more experiments.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 4.4 Habitual Participants Analysis
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: The seven MTurk studies examined are representative enough of MTurk experiments to support broader trends about habitual participation.
- Rests on an assumption: Unique MTurk ID numbers identify unique individuals.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the MTurk study, 60% of the respondents answered the question correctly.An identical question concerning the same article was also included on experiments run through Polimetrix/YouGov (another high-quality Internet panel) and with a sample collected by Survey Sampling International (SSI).The correct answer rates on these platforms were markedly lower than in the MTurk sample-49% on Polimetrix/”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 5.2 Attention, Demand, and Subject Motivation
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: The multiple-choice political office question is a valid measure of attention to experimental instruments.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In the experiments presented above, we found that the effects did not differ-in either a statistical or a substantive sense-when we examined the habitual respondents and the nonhabitual respondents separately.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 4.4 Habitual Participants Analysis
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It should be noted that other convenience samples, such as student or local intercept samples, may also have significant numbers of habitual experimental participants.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 2.1 Cost per Subject and Ease of Recruitment
Strength:hedged· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“per subject costs for typical undergraduate samples are about $5-10, for nonstudent campus samples about $30 (Kam, Wilking, and Zechmeister 2007), and for temporary agency subjects between $15 and $20.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 2.1 Cost per Subject and Ease of Recruitment
What the sources say
- Not checked yetcites Kam, Wilking, and Zechmeister 2007
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Additionally, in designing our MTurk survey, we followed the ANESP as closely as possible, using identical question wordings and branching formats.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 4.2 Comparison of Respondent Characteristics: Internet Samples and High-Quality, Face-To-Face, Probability Samples
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Respondents were randomly assigned to either the welfare or assistance to the poor version of the question.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 4.3.1
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All told, these comparisons reinforce the conclusion that the MTurk sample does not perfectly match the demographic and attitudinal characteristics of the U.S. population but does not present a wildly distorted view of the U.S. population either.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 22
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.
- Rests on an assumption: The American National Election 2008-09 Panel Study (ANESP) is a high-quality Internet sample and a valid benchmark for comparing MTurk.
- Rests on an assumption: The Current Population Survey and the 2008 American National Election Study are the gold standard for survey sampling.
- Rests on an assumption: MTurk workers' self-reported demographic and political characteristics are accurate.
In the paper
“Even though welfare and assistance to the poor are thought by policy experts to refer to the same policy, the study found important differences in levels of support between the two question forms.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 4.3.1
What the sources say
- Not checked yetcites Rasinski 1989
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Rests on an assumption: The original published experiments used as replication benchmarks provide valid estimates of treatment effects for their samples.
In the paper
“Although 20%-25% of the respondents in each year said that too little was being spent on welfare, 63%-65% said that too little was being spent on assistance to the poor (Rasinski 1989, 391).”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 4.3.1
What the sources say
- Not checked yetcites Rasinski 1989
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may state two separate claims as one.
- Rests on an assumption: The original published experiments used as replication benchmarks provide valid estimates of treatment effects for their samples.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Furthermore, given the relative dearth of these sorts of individuals in the MTurk pool, large sample sizes may be necessary to obtain sufficient diversity on these dimensions to estimate differences in treatment effects for these groups”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 22
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
- Rests on an assumption: The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.
In the paper
“Rasinski (1989) reports results from a question wording experiment that asked representative samples from the General Social Surveys (GSS) from 1984 to 1986 whether too much or too little was being spent on either ''welfare'' or ''assistance to the poor.''”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 4.3.1
What the sources say
- Not checked yetcites Rasinski (1989)
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
- Rests on an assumption: The original published experiments used as replication benchmarks provide valid estimates of treatment effects for their samples.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“MTurk samples will often be more diverse than convenience samples and will always be more diverse than student samples.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 22
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes? Based: unclearIt may state two separate claims as one.
- Rests on an assumption: The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.
In the paper
“The GSS has continued to ask the spending experiment and the gap in support for increasing spending between the question forms remains similar over time, ranging from 28% to 50%, with an average difference of 37% (Green and Kern 2010).”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 4.3.1
What the sources say
- Not checked yetcites Green and Kern 2010
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Rests on an assumption: The original published experiments used as replication benchmarks provide valid estimates of treatment effects for their samples.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Statistically significant differences exist between the MTurk sample and the benchmark surveys, but these differences are substantively small.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 22
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.
- Rests on an assumption: The American National Election 2008-09 Panel Study (ANESP) is a high-quality Internet sample and a valid benchmark for comparing MTurk.
- Rests on an assumption: The Current Population Survey and the 2008 American National Election Study are the gold standard for survey sampling.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The GSS is a nationally representative face-to-face interview sample with characteristics similar to the ANES face-to-face surveys.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 4.3.1
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
- Rests on an assumption: The original published experiments used as replication benchmarks provide valid estimates of treatment effects for their samples.
In the paper
“Prior work similarly finds no evidence that gender or race are associated with differences in effect sizes in the GSS in years earlier than 2010 (Green and Kern 2010).”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 27
What the sources say
- Not checked yetcites Green and Kern 2010
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Framing the outcomes in positive terms therefore produced a reversal of participants' preferences for the two programs compared to when it was presented in negative terms.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 27
What the sources say
- Not checked yetcites Tversky and Kahneman (1981)
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Tversky and Kahneman report that, when the problem was framed in terms of ''lives saved,'' respondents were more likely to pick the certain choice, while when it was framed in terms of ''lives lost,'' respondents were more likely to pick the risky choice.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 27
What the sources say
- Not checked yetcites Tversky and Kahneman (1981)
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“This experiment was first conducted using a student sample but has been replicated in many other settings (e.g., Takemura 1994; Kuhberger 1995; Jou, Shanteau, and Harris 1996; Bless, Betsch, and Franzen 1998; Druckman 2001).”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 4.3.2 Experiment 2: Asian disease problem
What the sources say
- Not checked yetcites Takemura 1994
- Not checked yetcites Kuhberger 1995
- Not checked yetcites Jou, Shanteau, and Harris 1996
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The first two studies we replicated were both straightforward experiments with large effect sizes.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 4.3.3 Experiment 3: Framing and risk
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
In the paper
“A final source of concern about external validity for any self-selected sample is the potential for ''habitual customers.''If the same subjects take several surveys, there is the potential for cross-experiment stimuli contamination (Transue, Lee, and Aldrich 2009).”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 4.4 Habitual Participants Analysis
What the sources say
- Not checked yetcites Transue, Lee, and Aldrich 2009
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
In the paper
“Kam and Simas used a modified version of the Tversky and Kahneman framing experiment and an original scale measuring individual differences in risk orientation to explore how both the frame of a problem and variation in risk proclivities affected choice under uncertainty.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 4.3.3 Experiment 3: Framing and risk
What the sources say
- Not checked yetcites Kam and Simas (2010)
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
In the paper
“In the original Tversky and Kahneman experiment, which was run with student samples, 72% of respondents picked the certain choice in the ''lives saved'' condition, as compared to 22% who picked the certain choice in the ''lives lost'' condition.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 27
What the sources say
- Not checked yetcites Tversky and Kahneman (1981)
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
In the paper
“Thus, although the gap is smaller than in the student sample, we still observe the large preference reversal reported in Tversky and Kahneman (and replicated in subsequent experiments with different samples).”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 27
What the sources say
- Not checked yetcites Tversky and Kahneman
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, although our estimate of the predictive power of risk assessment is modestly larger than in the original article, the basic pattern of effects is the same.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 4.3.3 Experiment 3: Framing and risk
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“But even if these are cases of repeat survey taking, only a handful of responses would be contaminated, suggesting that repeat survey taking is not a large problem in the MTurk subject pool.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 5.1 Do MTurk Workers Violate Assignment by Participating in Experiments Multiple Times?
Load-bearingStrength:hedged· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Since Requesters often specify at least a 95% prior ''approval rate''-that is, previous Requesters accepted 95% or more of the HITs submitted by an individual-respondents have an incentive to read instructions carefully and consider their responses.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 5.2 Attention, Demand, and Subject Motivation
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We demonstrate that relative to other convenience samples often used in experimental research in political science, MTurk subjects are often more representative of the general population and substantially less expensive to recruit.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 6 Conclusion
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“They are apparently also not currently an excessively overused pool, and habitual responding appears to be a minor concern.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 6 Conclusion
Load-bearingStrength:hedged· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In particular, MTurk subjects are notably younger and more ideologically liberal than the public, which may limit their suitability for some research topics.”
§ Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk > 6 Conclusion
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes? Based: unclearIt may state two separate claims as one.
The paper relies on this without support.
The 551 MTurk workers surveyed in February and March 2010 are representative of the broader domestic MTurk population.
“We surveyed 551 MTurk workers in February and March of 2010.”
The paper relies on this without support.
The American National Election 2008-09 Panel Study (ANESP) is a high-quality Internet sample and a valid benchmark for comparing MTurk.
“we compare MTurk to a high-quality Internet survey, the American National Election 2008-09 Panel Study (ANESP).”
The paper relies on this without support.
The Current Population Survey and the 2008 American National Election Study are the gold standard for survey sampling.
“These latter two studies use face-to-face probability samples and are widely considered the ''gold standard'' for survey sampling.”
The paper relies on this without support.
The original published experiments used as replication benchmarks provide valid estimates of treatment effects for their samples.
“we also replicated the results reported in three experiments.”
The paper relies on this without support.
IP address locators accurately identify U.S. residency.
“although IP address locators seem reliable, we are unaware of research benchmarking their accuracy.”
The paper relies on this without support.
MTurk workers' self-reported demographic and political characteristics are accurate.
“we compare measured characteristics of MTurk survey participants to characteristics of participants in three distinct types of research samples”
The paper relies on this without support.
The approval rate threshold of 95% selects a sample comparable to the broader MTurk population.
“We also excluded individuals with approval rates below 95% on previous MTurk tasks.”
The paper relies on this without support.
MTurk workers' self-reported number of political surveys taken in the last month is accurate.
“To assess the severity of this problem, we asked our respondents how many political surveys they had taken in the last month on MTurk.”
The paper relies on this without support.
The seven MTurk studies examined are representative enough of MTurk experiments to support broader trends about habitual participation.
“Although this set of experiments represents only a small proportion of those conducted on MTurk, our findings may illuminate broader trends.”
The paper relies on this without support.
Unique MTurk ID numbers identify unique individuals.
“We gathered the unique MTurk ID number for all workers who participated in each of seven studies we conducted from January 2010 to April 2010.”
The paper relies on this without support.
The multiple-choice political office question is a valid measure of attention to experimental instruments.
“In a study conducted by one of the authors, subjects were asked to identify the political office held by a person mentioned in a story they had just read.The format of this question was a multiple-choice item with five possible responses.”
The paper relies on this without support.
The GSS welfare/assistance question wording experiment provides a valid benchmark for comparing MTurk treatment effects.
“We found a statistically significant 38 percentage point gap ( p value < .001) between the two conditions that is similar in magnitude to that found using the GSS (the comparable gap was 44% in the 2010 GSS).”
The paper relies on this without support.
The MTurk and GSS samples are comparable enough to pool in ordered probits testing heterogeneous treatment effects.
“To conduct these tests, we pooled the MTurk and GSS samples.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The Internet has opened the way for improving standard surveys, experiments, and health reporting ( 35).”
From The Parable of Google Flu: Traps in Big Data Analysis (2014) · cites this paper as [35]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Research has shown data from MTurk workers is comparable to data from other pools (e.g., commercial panels, social media, colleges) [12, 20, 23, 32, 69, 80], but recently there has been a decrease in data quality [28, 49, 66, 68, 104, 108].”
From "I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust (2024) · cites this paper as [12]
The cited source says what the claim attributes to it.
In the citing paper
“online platforms that distribute tasks across workers, such as Amazon's Mechanical Turk, can make acquiring data efficient (Berinsky, Huber, and Lenz 2012).”
From Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts (2013) · cites this paper as Berinsky, Huber, and Lenz 2012
The cited source says what the claim attributes to it.
In the citing paper
“This study will re-examine these assertions based on two samples of Amazon Mechanical Turk workers, who appear more representative of the general population than are the typical first year psychology students comprising many of the studies reviewed in the preceding (see e.g., Berinsky et al., 2012).”
From A psychometric analysis of the reading the mind in the eyes test: toward a brief form for research and applied settings (2015) · cites this paper as Berinsky et al., 2012
In the citing paper
“and is, in fact, not representative, particularly when it comes to self-reported political identity and ideology51–53”
From Accuracy prompts are a replicable and generalizable approach for reducing the spread of misinformation (2022) · cites this paper as [53]
In the citing paper
“MTurk offers convenience samples and makes no attempt to be nationally representative”
From Accuracy prompts are a replicable and generalizable approach for reducing the spread of misinformation (2022) · cites this paper as [53]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Replication survey has conflicting field dates
MinorThe text identifies the May 2011 survey as containing the Kam and Simas replication, while Table 6 dates that replication survey to May 2010.
“These items were placed on a separate survey of 699 MTurk respondents conducted in May 2011.This study also contained the Kam and Simas (2010) replication discussed below.”
“Table: 6 Table 6 Replication of Table2byKam and Simas (2010)-risk acceptance and preference for the probabilistic outcome MTurk survey is from May 2010.”
Reported percentage of responses from shared IP addresses is too low
MinorFourteen of 551 responses is 2.54%, which rounds to 2.5%, not 2.4%.
Reported 2.4 · recomputed 2.540835
“We found that a total of seven IP addresses produced two responses each to our demographic survey (i.e., 14 of 551 responses or 2.4% of the total responses).”
Citations
Cites
The Framing of Decisions and the Psychology of Choice
Tversky, A., & Kahneman, D. (1981) · Science · doi:10.1126/science.7455683 · Cited by 18,497
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996
Orne, M. T. (1962) · American Psychologist · doi:10.1037/h0043424 · Cited by 4,041
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767
Gosling, S. D., Vazire, S., Srivastava, S., & John, O. P. (2004) · American Psychologist · doi:10.1037/0003-066x.59.2.93 · Cited by 2,932 · cited 1×
“Based on a comprehensive analysis, for example, Gosling et al. (2004) conclude that Internet samples tend to be diverse, are not adversely affected by nonserious or habitual responders, and produce findings consistent with traditional methods.”
ResultSears, D. O. (1986) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.51.3.515 · Cited by 2,111
Crowdsourcing user studies with Mechanical Turk
Kittur, A., Ed, H., & Suh, B. (2008) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357127 · Cited by 1,941
Cheap and fast---but is it good?: evaluating non-expert annotations for natural language tasks
Snow, R., O'Connor, B., Jurafsky, D., & Ng, A. Y. (2008) · Proceedings of the Conference on Empirical Methods in Natural Language Processing - EMNLP '08 · doi:10.3115/1613715.1613751 · Cited by 1,887
The Online Laboratory: Conducting Experiments in a Real Labor Market∗Preprint
Zeckhauser, R. J. (2016) · Cited by 1,454
Get another label? improving data quality and data mining using multiple, noisy labelers
Sheng, V. S., Provost, F. J., & Ipeirotis, P. G. (2008) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/1401890.1401965 · Cited by 1,180
Flagged first, then by citations
1 of the 38 references couldn’t be matched to a known paper.
Research works printed but not linked: 4
- The MTurk platform is of course limited to conducting research that does not require physical interactions between the subject and either the researcher or other subjects (e.g., to gather DNA samples, administer physical interventions, or observe face-to-face interactions among subjects). © The Author 2012. Published by Oxford University Press on behalf of the Society for Political Methodology. All rights reserved. For Permissions, please email: journals.permissions@oup.com Adam J. Berinsky et al.
- Ross, 2010, CHI EA 2010
- Snow, 2008, Evaluating non-expert annotations for natural language tasks
- Sheng, 2008, Get another label?
Cited by
Sorted by citations
Items 801–900 of 1,024
Science demands explanation, religion tolerates mystery
Liquin, E. G., Metz, S. E., & Lombrozo, T. (2020) · Cognition · doi:10.1016/j.cognition.2020.104398 · Cited by 50
Bastiaens, I., & Rudra, N. (2018) · Cambridge University Press eBooks · doi:10.1017/9781108556668 · Cited by 51
Employee accountability: development of a multidimensional scale
Han, Y., & Perry, J. L. (2019) · International Public Management Journal · doi:10.1080/10967494.2019.1690606 · Cited by 57
Weigold, A., & Weigold, I. K. (2021) · Social Science Computer Review · doi:10.1177/08944393211006847 · Cited by 49
Overgaard, C. S. B. (2021) · Journalism · doi:10.1177/14648849211062738 · Cited by 60
Lab vs online experiments: No differences
Prissé, B., & Jorrat, D. (2022) · Journal of Behavioral and Experimental Economics · doi:10.1016/j.socec.2022.101910 · Cited by 49
Kim, S. J., & Hancock, J. T. (2016) · Communication Research · doi:10.1177/0093650216644017 · Cited by 49
When and why does education matter? Motivation and resource effects in political efficacy
Rasmussen, S. H. R., & Nørgaard, A. S. (2017) · European Journal of Political Research · doi:10.1111/1475-6765.12213 · Cited by 57
Disambiguating authenticity: Interpretations of value and appeal
O'Connor, K. S., Carroll, G. R., & Kovács, B. (2017) · PLoS ONE · doi:10.1371/journal.pone.0179187 · Cited by 53
Chung, M. (2018) · Computers in Human Behavior · doi:10.1016/j.chb.2018.10.011 · Cited by 49
Giner‐Sorolla, R. (2018) · European Review of Social Psychology · doi:10.1080/10463283.2018.1542902 · Cited by 52
Cross-Domain Association in Metacognitive Efficiency Depends on First-Order Task Types
Lee, A., Ruby, E., Giles, N., & Lau, H. C. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.02464 · Cited by 49
An Improved Question Format for Measuring Conspiracy Beliefs
Clifford, S., Kim, Y., & Sullivan, B. W. (2019) · Public Opinion Quarterly · doi:10.1093/poq/nfz049 · Cited by 45
Cherry, T. L., James, A. G., & Murphy, J. J. (2021) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2021.04.006 · Cited by 49
Jensen‐Doss, A., Patel, Z. S., Casline, E. P., Ringle, V. A. M., & Timpano, K. R. (2021) · Journal of Clinical Child & Adolescent Psychology · doi:10.1080/15374416.2020.1815205 · Cited by 50
Rubel, A. N., & Burleigh, T. J. (2018) · Sexualities · doi:10.1177/1363460718779781 · Cited by 49
Amazeen, M. A. (2020) · Journalism & Mass Communication Quarterly · doi:10.1177/1077699020952131 · Cited by 30
Li, S., Jain, K., & Tzini, K. (2021) · Journal of Business Ethics · doi:10.1007/s10551-021-04797-1 · Cited by 49
The Origins and Consequences of Racialized Schemas about U.S. Parties
Zhirkov, K., & Valentino, N. A. (2022) · The Journal of Race Ethnicity and Politics · doi:10.1017/rep.2022.4 · Cited by 49
D’Amore, C., van Zomeren, M., & Koudenburg, N. (2024) · Journal of Personality and Social Psychology · doi:10.1037/pspi0000454 · Cited by 50
Rains, S. A., & Brunner, S. R. (2018) · Communication Research · doi:10.1177/0093650215598836 · Cited by 53
Briones, E. M., & Benham, G. (2017) · Behavior Research Methods · doi:10.3758/s13428-016-0710-8 · Cited by 48
A Preference for War: How Fairness and Rhetoric Influence Leadership Incentives in Crises
Gottfried, M. S., & Trager, R. (2016) · International Studies Quarterly · doi:10.1093/isq/sqv025 · Cited by 50
Keusch, F., & Yan, T. (2016) · Social Science Computer Review · doi:10.1177/0894439316675566 · Cited by 48
Wallsten, K., Nteta, T., McCarthy, L. A., & Tarsi, M. R. (2017) · Political Research Quarterly · doi:10.1177/1065912916685186 · Cited by 42
Attitudes toward attending the 2016 Olympic Games and visiting Brazil after the games
Rocha, C., & Fink, J. S. (2017) · Tourism Management Perspectives · doi:10.1016/j.tmp.2017.01.001 · Cited by 48
Hayes, J. L., Shan, Y., & King, K. W. (2017) · International Journal of Advertising · doi:10.1080/02650487.2017.1360576 · Cited by 51
Barrantes, R. J., & Eaton, A. A. (2018) · Sex Roles · doi:10.1007/s11199-018-0894-8 · Cited by 48
Strickland, J. C., & Stoops, W. W. (2018) · Journal of the Experimental Analysis of Behavior · doi:10.1002/jeab.445 · Cited by 47
Klas, A., Zinkiewicz, L., Zhou, J., & Clarke, E. J. R. (2018) · Environmental Communication · doi:10.1080/17524032.2018.1488755 · Cited by 48
Crowdsourcing visual perception experiments: a case of contrast threshold
Sasaki, K., & Yamada, Y. (2019) · PeerJ · doi:10.7717/peerj.8339 · Cited by 54
Justifying Social Inequalities: The Role of Social Darwinism
Rudman, L. A., & Saud, L. (2020) · Personality and Social Psychology Bulletin · doi:10.1177/0146167219896924 · Cited by 48
Conscription, Inequality, and Partisan Support for War
Kriner, D. L., & Shen, F. X. (2015) · Journal of Conflict Resolution · doi:10.1177/0022002715590877 · Cited by 47
An Audit of Political Behavior Research
Robison, J., Stevenson, R. T., Druckman, J. N., Jackman, S. D., Katz, J. N., & Vavreck, L. (2018) · SAGE Open · doi:10.1177/2158244018794769 · Cited by 47
The Democracy of Dating: How Political Affiliations Shape Relationship Formation
Easton, M. J., & Holbein, J. B. (2020) · Journal of Experimental Political Science · doi:10.1017/xps.2020.21 · Cited by 50
Overgaard, C. S. B. (2021) · Journalism Studies · doi:10.1080/1461670x.2021.1971107 · Cited by 56
Which Information Matters? Measuring Landlord Assessment of Tenant Screening Reports
So, W. (2022) · Housing Policy Debate · doi:10.1080/10511482.2022.2113815 · Cited by 50
Hiring Behavior Models for Online Labor Markets
Kokkodis, M., Papadimitriou, P., & Ipeirotis, P. G. (2015) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/2684822.2685299 · Cited by 47
Tang, T., & Cooper, R. (2017) · Communication & Sport · doi:10.1177/2167479516688438 · Cited by 47
Choma, B. L., & Prusaczyk, E. (2018) · Psychology of Women Quarterly · doi:10.1177/0361684317747845 · Cited by 49
When do customers engage in brand pages? Effects of social presence
Song, J., Moon, H. G., & Kim, M. (2019) · International Journal of Contemporary Hospitality Management · doi:10.1108/ijchm-10-2018-0816 · Cited by 49
Management Control System Design and Employees' Autonomous Motivation
Chen, C. X., Lill, J. B., & Vance, T. W. (2019) · Journal of Management Accounting Research · doi:10.2308/jmar-17-068 · Cited by 57
Shahid, F., Beshai, S., & Rosario, N. D. (2020) · Journal of Religion and Health · doi:10.1007/s10943-020-01024-5 · Cited by 48
Hall, M. G., Lazard, A. J., Grummon, A. H., Higgins, I. C. A., Bercholz, M., Richter, A. P. C., & Taillie, L. S. (2021) · Preventive Medicine · doi:10.1016/j.ypmed.2021.106562 · Cited by 47
Responding to Negative Electronic Word of Mouth to Improve Purchase Intention
Zinko, R., Patrick, A., Furner, C. P., Gaines, S., Kim, M. D., Negri, M., Orellana, E., Torres, S., & Villarreal, C. (2021) · Journal of theoretical and applied electronic commerce research · doi:10.3390/jtaer16060109 · Cited by 47
Analysis of reactions towards failures and recovery strategies for autonomous robots
Brooks, D. J., Begum, M., & Yanco, H. A. (2016) · 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN) · doi:10.1109/roman.2016.7745162 · Cited by 46
Classified or Coverup? The Effect of Redactions on Conspiracy Theory Beliefs
Nyhan, B. J., Dickinson, F., Dudding, S., Dylgjeri, E., Neiley, E., Pullerits, C., Seog, M., Simpson, A., Szilagyi, H., & Walmsley, C. (2016) · Journal of Experimental Political Science · doi:10.1017/xps.2015.21 · Cited by 46
Schuh, S. C., Cai, Y., Kaluza, A. J., Steffens, N. K., David, E., & Haslam, S. A. (2021) · Human Resource Management · doi:10.1002/hrm.22060 · Cited by 49
Culatta, E., & Clay‐Warner, J. (2021) · Society and Mental Health · doi:10.1177/2156869321991892 · Cited by 51
Opportunities for Expanding Access to Veterinary Care: Lessons From COVID-19
Smith, S. M., George, Z., Duncan, C., & Frey, D. M. (2022) · Frontiers in Veterinary Science · doi:10.3389/fvets.2022.804794 · Cited by 52
Scaling Up? Unpacking the Effect of Deliberative Mini-Publics on Legitimacy Perceptions
Germann, M., Mariën, S., & Muradova, L. (2022) · Political Studies · doi:10.1177/00323217221137444 · Cited by 65
Stoinski, L. M., Perkuhn, J., & Hebart, M. N. (2023) · Behavior Research Methods · doi:10.3758/s13428-023-02110-8 · Cited by 59
Kimel, S. Y., Huesmann, R., Kunst, J. R., & Halperin, E. (2016) · Personality and Social Psychology Bulletin · doi:10.1177/0146167216642196 · Cited by 46
Lansford, K. L., Borrie, S. A., & Bystricky, L. (2016) · American Journal of Speech-Language Pathology · doi:10.1044/2015_ajslp-15-0059 · Cited by 46
Tella, R. D., & Rotemberg, J. J. (2018) · Journal of Comparative Economics · doi:10.1016/j.jce.2018.03.001 · Cited by 50
Nonprobability Sampling and Twitter
Rafail, P. (2017) · Social Science Computer Review · doi:10.1177/0894439317709431 · Cited by 46
Religious anti-gay prejudice as a predictor of mental health, abuse, and substance use.
Sowe, B. J., Taylor, A., & Brown, J. (2017) · American Journal of Orthopsychiatry · doi:10.1037/ort0000297 · Cited by 48
Initial evidence that non-clinical autistic traits are associated with lower income
Skylark, W. J., & Baron‐Cohen, S. (2017) · Molecular Autism · doi:10.1186/s13229-017-0179-z · Cited by 45
Han, J., & Federico, C. M. (2018) · Journal of Communication · doi:10.1093/joc/jqy025 · Cited by 55
Cotropia, C. A. (2019) · Women s Reproductive Health · doi:10.1080/23293691.2019.1653575 · Cited by 46
Peifer, J. L., Chugani, S. K., & Roos, J. M. (2019) · Psychology and Marketing · doi:10.1002/mar.21277 · Cited by 53
Factors influencing the organizational adoption of cloud computing: a survey among cloud workers
Stieninger, M., Nedbal, D., Wetzlinger, W., Wagner, G., & Erskine, M. A. (2022) · International journal of information systems and project management · doi:10.12821/ijispm060101 · Cited by 46
Examining broad intellectual abilities obtained within an mTurk internet sample
Merz, Z. C., Lace, J. W., & Eisenstein, A. M. (2020) · Current Psychology · doi:10.1007/s12144-020-00741-0 · Cited by 52
The effects of police performance on agency trustworthiness and citizen participation
Porumbescu, G. A., Neshkova, M. I., & Huntoon, M. (2019) · Public Management Review · doi:10.1080/14719037.2018.1473473 · Cited by 45
Kalkan, K. O., Layman, G. C., & Green, J. C. (2018) · Politics and Religion · doi:10.1017/s1755048318000342 · Cited by 45
What drives the perceived legitimacy of collaborative governance? An experimental study
Lee, S., & Esteve, M. (2023) · Public Management Review · doi:10.1080/14719037.2022.2026692 · Cited by 45
Why tourists choose Airbnb: A motivation-based segmentation study underpinned by innovation concepts
Guttentag, D. A. (2016) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 44
Tingley, D. H. (2017) · International Organization · doi:10.1017/s0020818316000448 · Cited by 46
A Pairwise Comparison Framework for Fast, Flexible, and Reliable Human Coding of Political Texts
CARLSON, D., & Montgomery, J. (2017) · American Political Science Review · doi:10.1017/s0003055417000302 · Cited by 45
Non‐Scientific Criteria for Belief Sustain Counter‐Scientific Beliefs
Metz, S. E., Weisberg, D. S., & Weisberg, M. (2018) · Cognitive Science · doi:10.1111/cogs.12584 · Cited by 45
To Whom Do We Confide Our Secrets?
Slepian, M. L., & Kirby, J. N. (2018) · Personality and Social Psychology Bulletin · doi:10.1177/0146167218756032 · Cited by 46
Mindfulness, Money Attitudes, and Credit
Pereira, M. C., & Coelho, F. (2019) · Journal of Consumer Affairs · doi:10.1111/joca.12197 · Cited by 45
Comparing public concern and support for drone regulation to the current legal framework
Zwickle, A., Farber, H. B., & Hamm, J. A. (2018) · Behavioral Sciences & the Law · doi:10.1002/bsl.2357 · Cited by 45
Amazon Mechanical Turk workers can provide consistent and economically meaningful data
Johnson, D. B., & Ryan, J. B. (2020) · Southern Economic Journal · doi:10.1002/soej.12451 · Cited by 45
Body odor disgust sensitivity is associated with prejudice towards a fictive group of immigrants
Zakrzewska, M., Olofsson, J., Lindholm, T., Blomkvist, A., & Liuzza, M. T. (2019) · Physiology & Behavior · doi:10.1016/j.physbeh.2019.01.006 · Cited by 44
Emotion regulation choice: a broad examination of external factors
Young, G. R., & Suri, G. (2019) · Cognition & Emotion · doi:10.1080/02699931.2019.1611544 · Cited by 46
Bradt, J. T. (2019) · Behavioural Public Policy · doi:10.1017/bpp.2019.31 · Cited by 45
Quinn, D. M. (2020) · Educational Researcher · doi:10.3102/0013189x20932469 · Cited by 45
Experimental studies in family business research
Lude, M. J., & Prügl, R. (2020) · Journal of Family Business Strategy · doi:10.1016/j.jfbs.2020.100361 · Cited by 45
Measuring Perceptions of Economic Inequality and Justice: An Empirical Assessment
Heiserman, N., & Simpson, B. (2021) · Social Justice Research · doi:10.1007/s11211-021-00368-x · Cited by 47
Attitudes Toward Economic Inequality: The Illusory Agreement
Pedersen, R. T., & Mutz, D. C. (2019) · Political Science Research and Methods · doi:10.1017/psrm.2018.18 · Cited by 54
Who cares about human rights? Public opinion about human rights foreign policy
Allendoerfer, M. G. (2015) · Journal of Human Rights · doi:10.1080/14754835.2015.1129603 · Cited by 44
Eliciting Time versus Money: Time Scarcity Underlies Asymmetric Wage Rates
Monga, A., May, F. B., & Bagchi, R. (2017) · Journal of Consumer Research · doi:10.1093/jcr/ucx066 · Cited by 44
Black, J., Belicki, K., Emberley-Ralph, J., & McCann, A. (2020) · Death Studies · doi:10.1080/07481187.2020.1737274 · Cited by 53
Occupational Prestige: The Status Component of Socioeconomic Status
Hughes, B. T., Srivastava, S., Leszko, M., & Condon, D. M. (2024) · Collabra: Psychology · doi:10.1525/collabra.92882 · Cited by 44
Bauer, L., Cranor, L. F., Komanduri, S., Mazurek, M. L., Reiter, M. K., Sleeper, M., & Ur, B. (2013) · ACM Workshop on Privacy in the Electronic Society (WPES) · doi:10.1145/2517840.2517859 · Cited by 43
Shilton, K., & Sayles, S. (2016) · Hawaii International Conference on System Sciences (HICSS) · doi:10.1109/hicss.2016.242 · Cited by 44
Third-person perception of online comments: Civil ones persuade you more than me
Chen, G. M., & Ng, Y. M. M. (2015) · Computers in Human Behavior · doi:10.1016/j.chb.2015.10.014 · Cited by 44
Scopelliti, I., Min, H. L., McCormick, E. N., Kassam, K., & Morewedge, C. K. (2017) · Management Science · doi:10.1287/mnsc.2016.2668 · Cited by 44
Kim, Y., & Park, H. (2017) · Corporate Reputation Review · doi:10.1057/s41299-017-0016-5 · Cited by 44
Backing up, not backing down: Mitigating audience costs through policy substitution
Lin-Greenberg, E. (2019) · Journal of Peace Research · doi:10.1177/0022343319832641 · Cited by 48
Can Social Media Incivility Induce Enthusiasm?
Kosmidis, S., & Theocharis, Y. (2020) · Public Opinion Quarterly · doi:10.1093/poq/nfaa014 · Cited by 44
How partisanship and sexism influence voters’ reactions to political #MeToo scandals
Costa, M., Briggs, T., Chahal, A., Fried, J., Garg, R., Kriz, S., Lei, L. H.-I., Milne, A., & Slayton, J. (2020) · Research & Politics · doi:10.1177/2053168020941727 · Cited by 44
Harell, A., & Lieberman, E. S. (2021) · Social Science & Medicine · doi:10.1016/j.socscimed.2021.113884 · Cited by 47
The Influence of Weight-of-Evidence Messages on (Vaccine) Attitudes: A Sequential Mediation Model
Clarke, C. E., McKeever, B. W., Holton, A. E., & Dixon, G. (2015) · Journal of Health Communication · doi:10.1080/10810730.2015.1023959 · Cited by 43
Epistemic Factors in Selective Exposure and Political Misperceptions on the Right and Left
Meirick, P. C., & Bessarabova, E. (2015) · Analyses of Social Issues and Public Policy · doi:10.1111/asap.12101 · Cited by 45
Friedman, J. A., Lerner, J. S., & Zeckhauser, R. (2017) · International Organization · doi:10.1017/s0020818317000352 · Cited by 43
Caillier, J. G. (2018) · Public Performance & Management Review · doi:10.1080/15309576.2018.1431138 · Cited by 43
Perceived versus actual autism knowledge in the general population
McMahon, C. M., Stoll, B., & Linthicum, M. (2019) · Research in Autism · doi:10.1016/j.rasd.2019.101499 · Cited by 43
Krishna, A., & Amazeen, M. A. (2022) · Public Relations Review · doi:10.1016/j.pubrev.2022.102251 · Cited by 46
Metadata
- Authors
- Berinsky, A. J., Huber, G. A., & Lenz, G. S.
- Affiliations
- Massachusetts Institute of Technology (US)
- Yale University (US)
- University of California, Berkeley (US)
- Corresponding author
- Berinsky, Adam J.
- Venue
- Political Analysis
- Publisher
- Cambridge University Press (CUP)
- Published
- 2012
- Volume
- 20
- Issue
- 3
- Pages
- 351-368
- ISSN
- 1047-1987 (ISSN-L)
- 1476-4989
- OpenAlex
- W2119298903
- Semantic Scholar
- ff6a7eac8dc9e2a529f5299a9117c741b30c2aee
- Cited by
- 4,008
- References
- 38
- Citations by year
- 2026: 107 · 2025: 134 · 2024: 166 · 2023: 232 · 2022: 361 · 2021: 393 · 2020: 421 · 2019: 443 · 2018: 462 · 2017: 400 · 2016: 312 · 2015: 278 · 2014: 173 · 2013: 93 · 2012: 20
- FWCI
- 290.34
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Political Analysis (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Social Sciences › Sociology and Political Science › Names, Identity, and Discrimination Research
- Keywords
- Amazon Mechanical Turk, digital labour markets, participant recruitment, internal validity, generalizability, sample representativeness, convenience samples, replication, online panels, probability samples
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
