The DOI matches its Crossref record; 85 of the 85 references it cites were found in a registry or an index.
Validity and Mechanical Turk: An assessment of exclusion methods and interactive experiments
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Identifying careless responses in survey data.
Meade, A. W., & Craig, S. B. (2012) · Psychological Methods · doi:10.1037/a0028085 · Cited by 3,865
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767
Instructional manipulation checks: Detecting satisficing to increase statistical power
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2009.03.009 · Cited by 3,493
Conducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172
Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242
Introducing the Short Dark Triad (SD3): A Brief Measure of Dark Personality Traits
Jones, D. N., & Paulhus, D. L. (2014) · Assessment · doi:10.1177/1073191113514105 · Cited by 2,162
Hauser, D. J., & Schwarz, N. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0578-z · Cited by 1,739
Casler, K., Bickel, L., & Hackett, E. (2013) · Computers in Human Behavior · doi:10.1016/j.chb.2013.05.009 · Cited by 1,662 · cited 2×
“…Chartier, 2013; Suri & Watts, 2011), social psychology (Horton et al., 2011; Rand et al., 2012; Summerville & Chartier, 2013), cognitive psychology (Casler et al., 2013; Crump, McDonnell, & Gureckis, 2013), clinical psychology (Shapiro et al., 2013), and political science (Berinsky et al., 2012,…”
BackgroundReputation as a sufficient condition for data quality on Amazon Mechanical Turk
Peer, E., Vosgerau, J., & Acquisti, A. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0434-y · Cited by 1,618
Evaluating amazon’s mechanical turk as a tool for experimental behavioral research
Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0057410 · Cited by 1,617
The Online Laboratory: Conducting Experiments in a Real Labor Market∗Preprint
Zeckhauser, R. J. (2016) · Cited by 1,529
Advantages and Disadvantages of Cognitive Heuristics in Political Decision Making
Lau, R. R., & Redlawsk, D. P. (2001) · American Journal of Political Science · doi:10.2307/2669334 · Cited by 1,421
Spontaneous giving and calculated greed
Rand, D. G., Greene, J. D., & Nowak, M. A. (2012) · Nature · doi:10.1038/nature11467 · Cited by 1,334
The Generalizability of Survey Experiments
Mullinix, K. J., Leeper, T. J., Druckman, J. N., & Freese, J. (2015) · Journal of Experimental Political Science · doi:10.1017/xps.2015.19 · Cited by 1,228
Investigating Variation in Replicability: A “Many Labs” Replication Project
Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Bahník, Š., Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Brumbaugh, C. C., et al. (2014) · Social Psychology · doi:10.1027/1864-9335/a000178 · Cited by 1,208
Using Mechanical Turk to Study Clinical Populations
Shapiro, D. N., Chandler, J., & Mueller, P. A. (2013) · Clinical Psychological Science · doi:10.1177/2167702612469015 · Cited by 1,139
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
(2010) · doi:10.1145/1753326 · Cited by 989
Berinsky, A. J., Margolis, M. F., & Sances, M. W. (2014) · American Journal of Political Science · doi:10.1111/ajps.12081 · Cited by 951
Chandler, J., Mueller, P., & Paolacci, G. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0365-7 · Cited by 950
Huff, C., & Tingley, D. (2015) · Research & Politics · doi:10.1177/2053168015604648 · Cited by 903
Caring about carelessness: Participant inattention and its effects on research
Maniaci, M. R., & Rogge, R. D. (2014) · Journal of Research in Personality · doi:10.1016/j.jrp.2013.09.008 · Cited by 852
The Consquences of Adjustment for a Concomitant Variable That Has Been Affected by the Treatment
Rosenbaum, P. R. (1984) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2981697 · Cited by 849
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
(2014) · doi:10.1145/2556288 · Cited by 818
Are samples drawn from Mechanical Turk valid for research on political ideology?
Clifford, S., Jewell, R. M., & Waggoner, P. D. (2015) · Research & Politics · doi:10.1177/2053168015622072 · Cited by 686
Self-citationRand, D. G. (2012) · Journal of Theoretical Biology · doi:10.1016/j.jtbi.2011.03.004 · Cited by 683
An Open, Large-Scale, Collaborative Effort to Estimate the Reproducibility of Psychological Science
Open Science Collaboration (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612462588 · Cited by 614
Generalizing from Survey Experiments Conducted on Mechanical Turk: A Replication Approach
Coppock, A. (2019) · Political Science Research and Methods · doi:10.1017/psrm.2018.10 · Cited by 515
Demographics of Mechanical Turk
Ipeirotis, P. G. (2010) · Faculty Digital Archive (New York University Florence) · Cited by 494
Who Toes the Party Line? Cues, Values, and Individual Differences
Kam, C. D. (2005) · Political Behavior · doi:10.1007/s11109-005-1764-y · Cited by 470
Zhou, H., & Fishbach, A. (2016) · Journal of Personality and Social Psychology · doi:10.1037/pspa0000056 · Cited by 459
Are your participants gaming the system?: screening mechanical turk workers
Downs, J. S., Holbrook, M. B., Sheng, S., & Cranor, L. F. (2010) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753688 · Cited by 418
A reliability analysis of Mechanical Turk data
Rouse, S. V. (2015) · Computers in Human Behavior · doi:10.1016/j.chb.2014.11.004 · Cited by 414
Reversed item bias: An integrative model.
Weijters, B., Baumgartner, H., & Schillewaert, N. (2013) · Psychological Methods · doi:10.1037/a0032121 · Cited by 386
Economic games on the internet: the effect of $1 stakes
Amir, O., Rand, D. G., & Gal, Y. K. (2012) · PLoS ONE · doi:10.1371/journal.pone.0031461 · Cited by 364
O’Reilly, J., Eichhorst, W., Gábos, A., Hadjivassiliou, K., Lain, D., Leschke, J., McGuinness, S., Kureková, L. M., Nazio, T., Ortlieb, R., et al. (2015) · Sage Open · doi:10.1177/2158244015574962 · Cited by 360
Cooperation and Contagion in Web-Based, Networked Public Goods Experiments
Suri, S., & Watts, D. J. (2011) · PLoS ONE · doi:10.1371/journal.pone.0016836 · Cited by 354
The FAD–Plus: Measuring Lay Beliefs Regarding Free Will and Related Constructs
Paulhus, D. L., & Carey, J. M. (2011) · Journal of Personality Assessment · doi:10.1080/00223891.2010.528483 · Cited by 336
Self-talk as a regulatory mechanism: How you do it matters.
Kross, E., Bruehlman-Senecal, E., Park, J., Burson, A., Dougherty, A., Shablack, H., Bremner, R., Moser, J., & Ayduk, O. (2014) · Journal of Personality and Social Psychology · doi:10.1037/a0035173 · Cited by 332
Cooperation and contagion in web-based, networked public goods experiments
Suri, S., & Watts, D. J. (2011) · ACM SIGecom Exchanges · doi:10.1145/1998549.1998550 · Cited by 318
More than a body: Mind perception and the nature of objectification.
Gray, K., Knobe, J., Sheskin, M., Bloom, P., & Barrett, L. F. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0025883 · Cited by 277
Personality and Persuasive Technology: An Exploratory Study on Health-Promoting Mobile Applications
Halko, S., & Kientz, J. A. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-13226-1_16 · Cited by 252
Cross-Sample Comparisons and External Validity
Krupnikov, Y., & Levine, A. S. (2014) · Journal of Experimental Political Science · doi:10.1017/xps.2014.7 · Cited by 251
Can Partisan Cues Diminish Democratic Accountability?
Arceneaux, K. (2008) · Political Behavior · doi:10.1007/s11109-007-9044-7 · Cited by 233
Learning and the diffusion of regime contention in the Arab Spring
Bamert, J., Gilardi, F., & Wasserfallen, F. (2015) · Research & Politics · doi:10.1177/2053168015593306 · Cited by 213
Intuitive confidence: Choosing between intuitive and nonintuitive alternatives.
Simmons, J. P., & Nelson, L. D. (2006) · Journal of Experimental Psychology: General · doi:10.1037/0096-3445.135.3.409 · Cited by 212
Clifford, S., & Jerit, J. (2014) · Journal of Experimental Political Science · doi:10.1017/xps.2014.5 · Cited by 206
Self-citationPsychological research in the internet age: The quality of web-based data
Ramsey, S. R., Thompson, K. L., McKenzie, M., & Rosenbaum, A. (2016) · Computers in Human Behavior · doi:10.1016/j.chb.2015.12.049 · Cited by 203
Cryder, C. E., Loewenstein, G., & Scheines, R. (2013) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2012.08.002 · Cited by 190
Goal gradient in helping behavior
Cryder, C. E., Loewenstein, G. F., & Seltman, H. J. (2013) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2013.07.003 · Cited by 177
The psychology of coordination and common knowledge.
Thomas, K. A., DeScioli, P., Haque, O. S., & Pinker, S. (2014) · Journal of Personality and Social Psychology · doi:10.1037/a0037037 · Cited by 170
Self-citationIt’s a Trap! Instructional Manipulation Checks Prompt Systematic Thinking on “Tricky” Tasks
Hauser, D. J., & Schwarz, N. (2015) · Sage Open · doi:10.1177/2158244015584617 · Cited by 168
Crowdsourcing performance evaluations of user interfaces
Komarov, S., Reinecke, K., & Gajos, K. Z. (2013) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2470684 · Cited by 165
Missing the trees for the forest: A construal level account of the illusion of explanatory depth.
Alter, A. L., Oppenheimer, D. M., & Zemla, J. C. (2010) · Journal of Personality and Social Psychology · doi:10.1037/a0020218 · Cited by 158
How to Measure Discount Rates? An Experimental Comparison of Three Methods
Hardisty, D. J., Thompson, K. F., Krantz, D. H., & Weber, E. U. (2011) · Judgment and Decision Making · doi:10.1017/s1930297500005957 · Cited by 158
How naming and shaming affects human rights perceptions in the shamed country
Ausderan, J. (2014) · Journal of Peace Research · doi:10.1177/0022343313510014 · Cited by 158
The Exception Is the Rule: Underestimating and Overspending on Exceptional Expenses
Sussman, A. B., & Alter, A. L. (2012) · Journal of Consumer Research · doi:10.1086/665833 · Cited by 154
Who are the Turkers? Worker Demographics in Amazon Mechanical Turk
Ross, J., Zaldivar, A., Irani, L. C., & Tomlinson, B. (2009) · Sound Ideas (University of Puget Sound) · Cited by 136
Hawkins, C. B., & Nosek, B. A. (2012) · Personality and Social Psychology Bulletin · doi:10.1177/0146167212452313 · Cited by 123
Group-Based Dominance and Opposition to Equality Correspond to Different Psychological Motives
Kugler, M. B., Cooper, J., & Nosek, B. A. (2010) · Social Justice Research · doi:10.1007/s11211-010-0112-5 · Cited by 120
Kurtz, J. E., & Parrish, C. L. (2001) · Journal of Personality Assessment · doi:10.1207/s15327752jpa7602_12 · Cited by 106
Parents Reap What They Sow: Child-Centrism and Parental Well-Being
Ashton-James, C. E., Kushlev, K., & Dunn, E. W. (2013) · Social Psychological and Personality Science · doi:10.1177/1948550613479804 · Cited by 97
Do Attempts to Improve Respondent Attention Increase Social Desirability Bias?
Clifford, S., & Jerit, J. (2015) · Public Opinion Quarterly · doi:10.1093/poq/nfv027 · Cited by 91
Self-citationMeyvis, T., Goldsmith, K., & Dhar, R. (2011) · Journal of Marketing Research · doi:10.1509/jmr.08.0060 · Cited by 86
Goodman, J. K., & Irmak, C. (2012) · Journal of Marketing Research · doi:10.1509/jmr.10.0396 · Cited by 85
Does affluence impoverish the experience of parenting?
Kushlev, K., Dunn, E. W., & Ashton-James, C. E. (2012) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2012.06.001 · Cited by 67
When deviant leaders are punished more than non-leaders: The role of deviance severity
Karelaia, N., & Keck, S. (2013) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2013.04.003 · Cited by 66
Framing Influences Willingness to Pay but Not Willingness to Accept
Yang, Y., Vosgerau, J., & Loewenstein, G. F. (2013) · Journal of Marketing Research · doi:10.1509/jmr.12.0430 · Cited by 63
Zervanou, K., & Lendvai, P. (2011) · Cited by 62
“Not One of Us”: Predictors and Consequences of Denying Ingroup Characteristics to Ambiguous Targets
Kteily, N., Cotterill, S., Sidanius, J., Sheehy-Skeffington, J. A., & Bergh, R. (2014) · Personality and Social Psychology Bulletin · doi:10.1177/0146167214539708 · Cited by 60
Pseudo-dyadic “interaction” on Amazon’s Mechanical Turk
Summerville, A. L., & Chartier, C. R. (2012) · Behavior Research Methods · doi:10.3758/s13428-012-0250-9 · Cited by 50
Sirota, M., Kostovičová, L., & Juanchich, M. (2013) · Psychonomic Bulletin & Review · doi:10.3758/s13423-013-0555-4 · Cited by 49
Schuldt, J. P., & Roh, S. (2014) · Social Cognition · doi:10.1521/soco.2014.32.3.217 · Cited by 45
Attitudes of Lay People to Withdrawal of Treatment in Brain Damaged Patients
Gipson, J., Kahane, G., & Savulescu, J. (2013) · Neuroethics · doi:10.1007/s12152-012-9174-4 · Cited by 44
Do I amuse you? Asymmetric predictors for humor appreciation and humor production
Moran, J. M., Rain, M., Page‐Gould, E., & Mar, R. A. (2013) · Journal of Research in Personality · doi:10.1016/j.jrp.2013.12.002 · Cited by 41
Ad-hoc scalar implicature in adults and children
Stiller, A. J., Goodman, N. D., & Frank, M. C. (2011) · eScholarship (California Digital Library) · Cited by 38
Gromet, D. M., & Darley, J. M. (2011) · Journal of Empirical Legal Studies · doi:10.1111/j.1740-1461.2011.01242.x · Cited by 37
Functions in biological kind classification
Lombrozo, T., & Rehder, B. (2012) · Cognitive Psychology · doi:10.1016/j.cogpsych.2012.06.002 · Cited by 33
Spatial Metaphor and Real Estate: North–South Location Biases Housing Preference
Meier, B. P., Moller, A. C., Chen, J. J., & Riemer-Peltz, M. (2011) · Social Psychological and Personality Science · doi:10.1177/1948550611401042 · Cited by 33
Investigator Characteristics and Respondent Behavior in Online Surveys
White, A. R., Strezhnev, A., Lucas, C., Kruszewska, D., & Huff, C. (2018) · Journal of Experimental Political Science · doi:10.1017/xps.2017.25 · Cited by 27
Barone, M. J., Lyle, K. B., & Winterich, K. P. (2014) · Marketing Letters · doi:10.1007/s11002-013-9276-8 · Cited by 16
Reducing stigma toward individuals with mental illnesses: A brief, online manipulationPreprint
Menatti, A. R., Smyth, F. L., Teachman, B. A., & Nosek, B. A. (2016) · doi:10.31234/osf.io/ghfa3 · Cited by 11
Løhre, E., & Teigen, K. H. (2014) · Acta Psychologica · doi:10.1016/j.actpsy.2014.01.005 · Cited by 9
Crowdsourced Data Preprocessing with R and Amazon Mechanical Turk
Thomas, J. L. (2016) · The R Journal · doi:10.32614/rj-2016-020 · Cited by 9
Cited by
Data quality of platforms and panels for online behavioral research
Peer, E., Rothschild, D., Gordon, A., Evernden, Z., & Damer, E. (2021) · Behavior Research Methods · doi:10.3758/s13428-021-01694-3 · Cited by 1,718
MTurk Research: Review and Recommendations
Aguinis, H., Villamor, I., & Ramani, R. S. (2021) · Journal of Management · doi:10.1177/0149206320969787 · Cited by 984
Online panels in social science research: Expanding sampling methods beyond Mechanical Turk
Chandler, J., Rosenzweig, C., Moss, A. J., Robinson, J., & Litman, L. (2019) · Behavior Research Methods · doi:10.3758/s13428-019-01273-7 · Cited by 668
The shape of and solutions to the MTurk quality crisis
Kennedy, R., Clifford, S., Burleigh, T., Waggoner, P. D., Jewell, R., & Winter, N. J. G. (2020) · Political Science Research and Methods · doi:10.1017/psrm.2020.6 · Cited by 506
Self-citationAre Manipulation Checks Necessary?
Hauser, D. J., Ellsworth, P. C., & Gonzalez, R. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.00998 · Cited by 480
Metadata
- Authors
- Thomas, K. A., & Clifford, S.
- Venue
- Computers in Human Behavior
- Publisher
- Elsevier BV
- Published
- 2017-12
- Volume
- 77
- Pages
- 184-197
- ISSN
- 0747-5632 (ISSN-L)
- OpenAlex
- W2747118274
- Cited by
- 625
- References
- 85
- FWCI
- 53.26
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.