Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers
Open accessAbstract
Crowdsourcing services--particularly Amazon Mechanical Turk--have made it easy for behavioral scientists to recruit research participants. However, researchers have overlooked crucial differences between crowdsourcing and traditional recruitment methods that provide unique opportunities and challenges. We show that crowdsourced workers are likely to participate across multiple related experiments and that researchers are overzealous in the exclusion of research participants. We describe how both of these problems can be avoided using advanced interface features that also allow prescreening and longitudinal data collection. Using these techniques can minimize the effects of previously ignored drawbacks and expand the scope of crowdsourcing as a tool for psychological research.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Cites
Statistical power analyses using G∗Power 3.1: tests for correlation and regression analyses
Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (2009) · Behavior Research Methods · doi:10.3758/brm.41.4.1149 · Cited by 37,615 · see citing works
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996 · see citing works
Stereotype threat and the intellectual test performance of African Americans.
Steele, C. M., & Aronson, J. (1995) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.69.5.797 · Cited by 8,181 · see citing works
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567 · see citing works
Cognitive Reflection and Decision Making
Frederick, S. (2005) · Journal of Economic Perspectives · doi:10.1257/089533005775196732 · Cited by 5,256 · see citing works
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008 · see citing works
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767 · see citing works
Self-citationInstructional manipulation checks: Detecting satisficing to increase statistical power
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2009.03.009 · Cited by 3,493 · see citing works
Conducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172 · see citing works
Gosling, S. D., Vazire, S., Srivastava, S., & John, O. P. (2004) · American Psychologist · doi:10.1037/0003-066x.59.2.93 · Cited by 2,932 · see citing works
The Efficient Assessment of Need for Cognition
Cacioppo, J. T., Petty, R. E., & Feng Kao, C. (1984) · Journal of Personality Assessment · doi:10.1207/s15327752jpa4803_13 · Cited by 2,913 · see citing works
Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242 · see citing works
Crowdsourcing user studies with Mechanical Turk
Kittur, A., Ed, H., & Suh, B. (2008) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357127 · Cited by 1,941 · see citing works
Likableness ratings of 555 personality-trait words.
Anderson, N. H. (1968) · Journal of Personality and Social Psychology · doi:10.1037/h0025907 · Cited by 1,672 · see citing works
Lintott, C. J., Schawinski, K., Slosar, A., Land, K., Bamford, S., Thomas, D., Raddick, M. J., Nichol, R. C., Szalay, A., Andreescu, D., et al. (2008) · Monthly Notices of the Royal Astronomical Society · doi:10.1111/j.1365-2966.2008.13689.x · Cited by 1,488 · see citing works
The Online Laboratory: Conducting Experiments in a Real Labor Market∗Preprint
Zeckhauser, R. J. (2016) · Cited by 1,454 · see citing works
Rosch, E. (1975) · Cognitive Psychology · doi:10.1016/0010-0285(75)90021-3 · Cited by 1,423 · see citing works
Predicting protein structures with a multiplayer online game
Cooper, S., Khatib, F., Treuille, A., Barbero, J., Lee, J., Beenen, M., Leaver-Fay, A., Baker, D., Popović, Z., & players, F. (2010) · Nature · doi:10.1038/nature09304 · Cited by 1,368 · see citing works
Using Mechanical Turk to Study Clinical Populations
Shapiro, D. N., Chandler, J., & Mueller, P. A. (2013) · Clinical Psychological Science · doi:10.1177/2167702612469015 · Cited by 1,139 · see citing works
Self-citationreCAPTCHA: Human-Based Character Recognition via Web Security Measures
von Ahn, L., Maurer, B., McMillen, C. P., Abraham, D., & Blum, M. (2008) · Science · doi:10.1126/science.1160379 · Cited by 1,116 · see citing works
The viability of crowdsourcing for survey research
Behrend, T. S., Sharek, D. J., Meade, A. W., & Wiebe, E. (2011) · Behavior Research Methods · doi:10.3758/s13428-011-0081-0 · Cited by 1,049 · see citing works
Rand, D. G. (2011) · Journal of Theoretical Biology · doi:10.1016/j.jtbi.2011.03.004 · Cited by 686 · see citing works
Ascertaining the validity of individual protocols from Web-based personality inventories
Johnson, J. A. (2004) · Journal of Research in Personality · doi:10.1016/j.jrp.2004.09.009 · Cited by 618 · see citing works
Stereotypes as Judgmental Heuristics: Evidence of Circadian Variations in Discrimination
Bodenhausen, G. V. (1990) · Psychological Science · doi:10.1111/j.1467-9280.1990.tb00226.x · Cited by 614 · see citing works
Divine intuition: Cognitive style influences belief in God.
Shenhav, A., Rand, D. G., & Greene, J. D. (2012) · Journal of Experimental Psychology: General · doi:10.1037/a0025391 · Cited by 557 · see citing works
Demographics of Mechanical Turk
Ipeirotis, P. G. (2010) · Faculty Digital Archive (New York University Florence) · Cited by 494 · see citing works
Sprouse, J. (2010) · Behavior Research Methods · doi:10.3758/s13428-010-0039-7 · Cited by 492 · see citing works
Reips, U. (2000) · Elsevier eBooks · doi:10.1016/b978-012099980-4/50005-8 · Cited by 462 · see citing works
Validity of Web-Based Psychological Research
Krantz, J. H., & Dalal, R. S. (2000) · Elsevier eBooks · doi:10.1016/b978-012099980-4/50003-4 · Cited by 425 · see citing works
Are your participants gaming the system?
Downs, J. S., Holbrook, M. B., Sheng, S., & Cranor, L. F. (2010) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753688 · Cited by 422 · see citing works
Reflection and Reasoning in Moral Judgment
Paxton, J. M., Ungar, L. D. J., & Greene, J. D. (2011) · Cognitive Science · doi:10.1111/j.1551-6709.2011.01210.x · Cited by 418 · see citing works
Cognitive sophistication does not attenuate the bias blind spot.
West, R. F., Meserve, R. J., & Stanovich, K. E. (2012) · Journal of Personality and Social Psychology · doi:10.1037/a0028857 · Cited by 361 · see citing works
Cooperation and Contagion in Web-Based, Networked Public Goods Experiments
Suri, S., & Watts, D. J. (2011) · PLoS ONE · doi:10.1371/journal.pone.0016836 · Cited by 355 · see citing works
Cooperation and contagion in web-based, networked public goods experiments
Suri, S., & Watts, D. J. (2011) · ACM SIGecom Exchanges · doi:10.1145/1998549.1998550 · Cited by 318 · see citing works
The Sage Handbook of Social Psychology
Langelaan, S., & JONGE, J. D. (2004) · Gedrag & Organisatie · doi:10.5117/2004.017.004.008 · Cited by 273 · see citing works
Ribisl, K. M., Walton, M. A., Mowbray, C. T., Luke, D. A., Davidson, W. S., & BootsMiller, B. J. (1996) · Evaluation and Program Planning · doi:10.1016/0149-7189(95)00037-2 · Cited by 249 · see citing works
Practice Effects on Commonly Used Measures of Executive Function Across Twelve Months
Basso, M. R., Bornstein, R. A., & Lang, J. M. (1999) · The Clinical Neuropsychologist · doi:10.1076/clin.13.3.283.1743 · Cited by 245 · see citing works
Demand Artifacts in Laboratory Experiments in Consumer Research
Sawyer, A. G. (1975) · Journal of Consumer Research · doi:10.1086/208604 · Cited by 239 · see citing works
Presenting diverse political opinions
Munson, S. A., & Resnick, P. (2010) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753543 · Cited by 236 · see citing works
Developing a tool for measuring the decision-making competence of older adults.
Finucane, M. L., & Gullion, C. M. (2010) · Psychology and Aging · doi:10.1037/a0019106 · Cited by 202 · see citing works
Stereotype Threat in Applied Settings Re‐Examined
Danaher, K., & Crandall, C. S. (2008) · Journal of Applied Social Psychology · doi:10.1111/j.1559-1816.2008.00362.x · Cited by 200 · see citing works
Pope, D. G., & Simonsohn, U. (2010) · Psychological Science · doi:10.1177/0956797610391098 · Cited by 173 · see citing works
Task search in a human computation market
Chilton, L. B., Horton, J. J., Miller, R. C., & Azenkot, S. (2010) · doi:10.1145/1837885.1837889 · Cited by 172 · see citing works
Potential of the Internet for Personality Research
Buchanan, T. (2000) · Elsevier eBooks · doi:10.1016/b978-012099980-4/50006-x · Cited by 171 · see citing works
Attrition in Substance Abuse Prevention Research
Hansen, W. B., Tobler, N. S., & Graham, J. A. (1990) · Evaluation Review · doi:10.1177/0193841x9001400608 · Cited by 156 · see citing works
“Debriefing” and susceptibility to subsequent experimental manipulations
Brock, T. C., & Becker, L. A. (1966) · Journal of Experimental Social Psychology · doi:10.1016/0022-1031(66)90087-4 · Cited by 108 · see citing works
Risks and Rewards of Crowdsourcing Marketplaces
Chandler, J. J., Paolacci, G., & Mueller, P. A. (2013) · Handbook of Human Computation · doi:10.1007/978-1-4614-8806-4_30 · Cited by 79 · see citing works
Self-citationEffects of deceiving and debriefing psychological subjects on performance in later experiments.
SILVERMAN, I. W., Shulman, A. D., & Wiesenthal, D. L. (1970) · Journal of Personality and Social Psychology · doi:10.1037/h0028852 · Cited by 69 · see citing works
Selectively Recruiting Participants from Amazon Mechanical Turk Using QualtricsPreprint
Péer, E., Paolacci, G., Chandler, J., & Mueller, P. A. (2012) · SSRN Electronic Journal · doi:10.2139/ssrn.2100631 · Cited by 67 · see citing works
Self-citationEdlund, J. E., Sagarin, B. J., Skowronski, J. J., Johnson, S. J., & Kutter, J. (2009) · Personality and Social Psychology Bulletin · doi:10.1177/0146167208331255 · Cited by 63 · see citing works
Pseudo-dyadic “interaction” on Amazon’s Mechanical Turk
Summerville, A. L., & Chartier, C. R. (2012) · Behavior Research Methods · doi:10.3758/s13428-012-0250-9 · Cited by 50 · see citing works
Nonnaivety contamination in conformity experiments: Sources, effects, and implications for control.
Glinski, R. J., Glinski, B. C., & Slatin, G. T. (1970) · Journal of Personality and Social Psychology · doi:10.1037/h0030073 · Cited by 41 · see citing works
Mediation of artifacts in behavioral research
Rosnow, R. L., & Aiken, L. S. (1973) · Journal of Experimental Social Psychology · doi:10.1016/0022-1031(73)90009-7 · Cited by 38 · see citing works
Reasoning about others' reasoning
Mata, A., Fiedler, K., Ferreira, M. B., & Almeida, T. (2013) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2013.01.010 · Cited by 37 · see citing works
Gaggioli, A., & Riva, G. (2008) · Science · doi:10.1126/science.321.5895.1443a · Cited by 28 · see citing works
Questions and Comparisons: Methods of Research in Social Psychology
Ellsworth, P. C., & Gonzalez, R. (2007) · SAGE Publications Ltd eBooks · doi:10.4135/9781848608221.n2 · Cited by 23 · see citing works
Emailing Workers Using PythonPreprint
Mueller, P. A., & Chandler, J. (2012) · SSRN Electronic Journal · doi:10.2139/ssrn.2100601 · Cited by 8 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Prolific.ac—A subject pool for online experiments
Palan, S., & Schitter, C. (2018) · Journal of Behavioral and Experimental Finance · doi:10.1016/j.jbef.2017.12.004 · Cited by 3,450 · see citing works
Beyond the Turk: Alternative platforms for crowdsourcing behavioral research
Peer, E., Brandimarte, L., Samat, S., & Acquisti, A. (2017) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2017.01.006 · Cited by 2,957 · see citing works
Inside the Turk: Understanding Mechanical Turk as a Participant Pool
Paolacci, G., & Chandler, J. (2014) · Current Directions in Psychological Science · doi:10.1177/0963721414531598 · Cited by 2,136 · see citing works
Self-citationHauser, D. J., & Schwarz, N. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0578-z · Cited by 1,739 · see citing works
Data quality of platforms and panels for online behavioral research
Peer, E., Rothschild, D., Gordon, A., Evernden, Z., & Damer, E. (2021) · Behavior Research Methods · doi:10.3758/s13428-021-01694-3 · Cited by 1,718 · see citing works
Reputation as a sufficient condition for data quality on Amazon Mechanical Turk
Peer, E., Vosgerau, J., & Acquisti, A. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0434-y · Cited by 1,618 · see citing works
Conducting Clinical Research Using Crowdsourced Convenience Samples
Chandler, J., & Shapiro, D. (2016) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-021815-093623 · Cited by 1,072 · see citing works
Self-citationMany Labs 2: Investigating variation in replicability across samples and settings
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., Aveyard, M., Axt, J. R., Babalola, M. T., Bahník, Š., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918810225 · Cited by 944 · see citing works
Self-citationHuff, C., & Tingley, D. (2015) · Research & Politics · doi:10.1177/2053168015604648 · Cited by 903 · see citing works
Abbey, J. D., & Meloy, M. G. (2017) · Journal of Operations Management · doi:10.1016/j.jom.2017.06.001 · Cited by 522 · see citing works
Internet Research in Psychology
Gosling, S. D., & Mason, W. (2015) · Annual Review of Psychology · doi:10.1146/annurev-psych-010814-015321 · Cited by 458 · see citing works
Amazon mechanical Turk in organizational psychology: an evaluation and practical recommendations
Cheung, J. H., Burns, D. K., Sinclair, R. R., & Sliter, M. (2017) · Journal of Business and Psychology · doi:10.1007/s10869-016-9458-5 · Cited by 474 · see citing works
Does Counter-Attitudinal Information Cause Backlash? Results from Three Large Survey Experiments
Guess, A., & Coppock, A. (2020) · British Journal of Political Science · doi:10.1017/s0007123418000327 · Cited by 372 · see citing works
Ideology, Motivated Reasoning, and Cognitive Reflection: An Experimental Study
Kahan, D. M. (2012) · doi:10.2139/ssrn.2182588 · Cited by 331 · see citing works
Keith, M. G., Tay, L., & Harms, P. D. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.01359 · Cited by 319 · see citing works
The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers
Stewart, N., Ungemach, C., Harris, A. J. L., Bartels, D. M., Newell, B. R., Paolacci, G., & Chandler, J. (2015) · Judgment and Decision Making · doi:10.1017/s1930297500005611 · Cited by 304 · see citing works
“Experienced workers show practice effects which may inflate measures of ability or attentiveness to trick questions (Chandler et al., 2013; Glinski, Glinski, & Slatin, 1970; Hauser & Schwarz, 2015).”
InfluentialSelf-citationConducting perception research over the internet: A tutorial review
Woods, A. T., Velasco, C., Levitan, C. A., Wan, X., & Spence, C. (2015) · PeerJ · doi:10.7717/peerj.1058 · Cited by 301 · see citing works
Income inequality influences perceptions of legitimate income differences
Trump, K.-S. (2018) · British Journal of Political Science · doi:10.1017/s0007123416000326 · Cited by 236 · see citing works
Planning and Conducting Experimental Advertising Research and Questionnaire Design
Geuens, M., & De Pelsmacker, P. (2017) · Journal of Advertising · doi:10.1080/00913367.2016.1225233 · Cited by 234 · see citing works
Hong, J.-W., & Curran, N. M. (2019) · ACM Transactions on Multimedia Computing, Communications, and Applications · doi:10.1145/3326337 · Cited by 217 · see citing works
Using Nonnaive Participants Can Reduce Effect Sizes
Chandler, J., Paolacci, G., Peer, E., Mueller, P., & Ratliff, K. A. (2015) · Psychological Science · doi:10.1177/0956797615585115 · Cited by 185 · see citing works
Miller, J. D., Crowe, M., Weiss, B., Maples-Keller, J. L., & Lynam, D. R. (2017) · Personality Disorders Theory Research and Treatment · doi:10.1037/per0000191 · Cited by 179 · see citing works
Brühlmann, F., Petralito, S., Aeschbach, L. F., & Opwis, K. (2020) · Methods in Psychology · doi:10.1016/j.metip.2020.100022 · Cited by 168 · see citing works
Crowdsourcing hypothesis tests: Making transparent how design choices shape research results.
Landy, J. F., Jia, M. (., Ding, I. L., Viganola, D., Tierney, W., Dreber, A., Johannesson, M., Pfeiffer, T., Ebersole, C. R., Gronau, Q. F., et al. (2020) · Psychological Bulletin · doi:10.1037/bul0000220 · Cited by 168 · see citing works
Forming global estimates of self-performance from local confidence
Rouault, M., Dayan, P., & Fleming, S. M. (2019) · Nature Communications · doi:10.1038/s41467-019-09075-3 · Cited by 134 · see citing works
Crowdsourcing for Cognitive Science – The Utility of Smartphones
Brown, H. R., Zeidman, P., Smittenaar, P., Adams, R. A., McNab, F., Rutledge, R. B., & Dolan, R. J. (2014) · PLoS ONE · doi:10.1371/journal.pone.0100662 · Cited by 129 · see citing works
Barends, A. J., & de Vries, R. E. (2019) · Personality and Individual Differences · doi:10.1016/j.paid.2019.02.015 · Cited by 124 · see citing works
Sanders, W., Parent, J., Forehand, R., Sullivan, A. D. W., & Jones, D. J. (2016) · Journal of Applied Developmental Psychology · doi:10.1016/j.appdev.2016.02.005 · Cited by 116 · see citing works
Williams, M. T., Davis, A. K., Xin, Y., Sepeda, N. D., Grigas, P. C., Sinnott, S., & Haeny, A. M. (2021) · Drugs: Education, Prevention and Policy · doi:10.1080/09687637.2020.1854688 · Cited by 101 · see citing works
Arechar, A. A., & Rand, D. G. (2021) · Behavior Research Methods · doi:10.3758/s13428-021-01588-4 · Cited by 105 · see citing works
Gendered public support for criminalizing “revenge porn
Lageson, S. E., McElrath, S., & Palmer, K. E. (2019) · Feminist Criminology · doi:10.1177/1557085118773398 · Cited by 79 · see citing works
The true trigger of shame: social devaluation is sufficient, wrongdoing is unnecessary
Robertson, T. E., Sznycer, D., Delton, A. W., Tooby, J., & Cosmides, L. (2018) · Evolution and Human Behavior · doi:10.1016/j.evolhumbehav.2018.05.010 · Cited by 77 · see citing works
Mindfulness in Parenting and Coparenting
Parent, J., McKee, L. G., Anton, M., Gonzalez, M., Jones, D. J., & Forehand, R. (2016) · Mindfulness · doi:10.1007/s12671-015-0485-5 · Cited by 72 · see citing works
Perceived partner responsiveness promotes intellectual humility
Reis, H. T., Lee, K. Y., O'Keefe, S. D., & Clark, M. S. (2018) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2018.05.006 · Cited by 70 · see citing works
Treger, S., Benau, E. M., & Timko, C. A. (2023) · PLOS ONE · doi:10.1371/journal.pone.0285267 · Cited by 25 · see citing works
Reynolds, C. J., Makhanova, A., Ng, B. K. L., & Conway, P. (2020) · Personality and Individual Differences · doi:10.1016/j.paid.2019.109632 · Cited by 22 · see citing works
Hu, S., Zheng, X., Zhang, N., & Zhu, J. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.01399 · Cited by 19 · see citing works
Comparing In-Person, Sona, and Mechanical Turk Measurements of Three Prejudice-Relevant Constructs
Gamblin, B. W., Winslow, M. P., Lindsay, B., Newsom, A. W., & Kehn, A. (2017) · Current Psychology · doi:10.1007/s12144-015-9403-1 · Cited by 19 · see citing works
Does Power Reduce Temporal Discounting? Commentary on Joshi and Fast (2013)
Zhang, M., & Smith, P. K. (2018) · Psychological Science · doi:10.1177/0956797617754219 · Cited by 13 · see citing works
On the State of Reporting in Crowdsourcing Experiments and a Checklist to Aid Current Practices
Ramírez, J., Sayin, B., Baez, M., Casati, F., Cernuzzi, L., Benatallah, B., & Demartini, G. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479531 · Cited by 4 · see citing works
- Authors
- Chandler, J., Mueller, P., & Paolacci, G.
- Affiliations
- Princeton University (US)
- Erasmus University Rotterdam (NL)
- Corresponding author
- Chandler, Jesse
- Venue
- Behavior Research Methods
- Published
- 2013-07-09
- Volume
- 46
- Issue
- 1
- Pages
- 112-130
- ISSN
- 1554-351X (ISSN-L)
- 1554-3528
- OpenAlex
- W1975450585
- Cited by
- 950
- References
- 57
- Citations by year
- 2026: 18 · 2025: 29 · 2024: 50 · 2023: 54 · 2022: 61 · 2021: 84 · 2020: 120 · 2019: 116 · 2018: 101 · 2017: 111 · 2016: 85 · 2015: 79 · 2014: 32 · 2013: 7 · 2012: 1
- FWCI
- 102.24
- Type
- journal article
- Language
- en
- Open access
- green
- pure.eur.nl
- Licence
- Locations
- Behavior Research Methods (publishedVersion)
- PubMed (publishedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (publishedVersion, other-oa)
- EUR Research Repository (Erasmus University Rotterdam) (submittedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (submittedVersion, other-oa)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Social Sciences › Safety Research › Experimental Behavioral Economics Studies
- Keywords
- Amazon Mechanical Turk, crowdsourcing, participant recruitment, prescreening, longitudinal data collection, behavioral research
- MeSH terms
- Analysis of Variance, Data Collection, Humans, Longitudinal Studies, Male, Population Surveillance, Practice, Psychological, Research Personnel, Task Performance and Analysis, Video Games, Internet, Information Dissemination, Behavioral Research, Research Subjects, Health Literacy, Crowdsourcing, Healthy Volunteers
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review