Using Nonnaive Participants Can Reduce Effect Sizes
Abstract
Although researchers often assume their participants are naive to experimental materials, this is not always the case. We investigated how prior exposure to a task affects subsequent experimental results. Participants in this study completed the same set of 12 experimental tasks at two points in time, first as a part of the Many Labs replication project and again a few days, a week, or a month later. Effect sizes were markedly lower in the second wave than in the first. The reduction was most pronounced when participants were assigned to a different condition in the second wave. We discuss the methodological implications of these findings.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Repeated exposure to stimuli can blunt effect sizes in various experimental paradigms.
“Repeated exposure to stimuli can blunt effect sizes in various experimental paradigms [103].”
From Not so terrifying after all? A set of failed replications of the mortality salience effects of Terror Management Theory (2023) · cites this paper as [103]
MTurk has growing participant non-naivety.
“The high rate of non-naivety among MTurk participants has recently been shown to have the potential to significantly reduce the effect sizes of known research findings (Chandler et al., 2015).”
From Beyond the Turk: Alternative platforms for crowdsourcing behavioral research (2017) · cites this paper as Chandler et al., 2015
Cites
Statistical power analyses using G∗Power 3.1: tests for correlation and regression analyses
Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (2009) · Behavior Research Methods · doi:10.3758/brm.41.4.1149 · Cited by 37,615 · see citing works
The Framing of Decisions and the Psychology of Choice
Tversky, A., & Kahneman, D. (1981) · Science · doi:10.1126/science.7455683 · Cited by 17,604 · cited 1× · see citing works
“In the gain-versus-loss-framing task (Tversky & Kahneman, 1981), participants read a disease scenario framed in terms of either gains or losses.”
MethodMeasuring individual differences in implicit cognition: the implicit association test
Greenwald, A. G., McGhee, D. E., & Schwartz, J. L. K. (1998) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.74.6.1464 · Cited by 9,210 · see citing works
Instructional manipulation checks: Detecting satisficing to increase statistical power
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2009.03.009 · Cited by 3,493 · see citing works
Johnson, M. K., Hashtroudi, S., & Lindsay, D. S. (1993) · Psychological Bulletin · doi:10.1037/0033-2909.114.1.3 · Cited by 3,392 · see citing works
Inside the Turk: Understanding Mechanical Turk as a Participant Pool
Paolacci, G., & Chandler, J. (2014) · Current Directions in Psychological Science · doi:10.1177/0963721414531598 · Cited by 2,136 · see citing works
One Hundred Years of Social Psychology Quantitatively Described
Richard, F. D., Bond, C. F., & Stokes-Zoota, J. J. (2003) · Review of General Psychology · doi:10.1037/1089-2680.7.4.331 · Cited by 1,558 · see citing works
Experimental methods: Between-subject and within-subject design
Charness, G., Gneezy, U., & Kuhn, M. A. (2012) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2011.08.009 · Cited by 1,469 · see citing works
Investigating Variation in Replicability: A “Many Labs” Replication Project
Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Bahník, Š., Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Brumbaugh, C. C., et al. (2014) · Social Psychology · doi:10.1027/1864-9335/a000178 · Cited by 1,208 · see citing works
Ambiguity Aversion and Comparative Ignorance
Fox, C. R., & Tversky, A. (1995) · The Quarterly Journal of Economics · doi:10.2307/2946693 · Cited by 1,039 · see citing works
Chandler, J., Mueller, P., & Paolacci, G. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0365-7 · Cited by 950 · see citing works
Measures of Anchoring in Estimation Tasks
Jacowitz, K. E., & Kahneman, D. J. (1995) · Personality and Social Psychology Bulletin · doi:10.1177/01461672952111004 · Cited by 753 · see citing works
Intuitive Cooperation and the Social Heuristics Hypothesis: Evidence from 15 Time Constraint Studies
Rand, D. G., Peysakhovich, A., Kraft‐Todd, G., Newman, G. E., Wurzbacher, O., Nowak, M. A., & Greene, J. D. (2013) · Nature Communications · doi:10.1038/ncomms4677 · Cited by 730 · see citing works
Within-subjects designs: To use or not to use?
Greenwald, A. G. (1976) · Psychological Bulletin · doi:10.1037/0033-2909.83.2.314 · Cited by 529 · see citing works
On the self-erasing nature of errors of prediction.
Sherman, S. J. (1980) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.39.2.211 · Cited by 525 · see citing works
Health of the Implicit Association Test at age 3.
Greenwald, A., & Nosek, B. A. (2001) · Experimental Psychology · doi:10.1026//0949-3946.48.2.85 · Cited by 488 · see citing works
Response Scales: Effects of Category Range on Reported Behavior and Comparative Judgments
Schwarz, N., Hippler, H., Deutsch, B., & Strack, F. (1985) · Public Opinion Quarterly · doi:10.1086/268936 · Cited by 372 · see citing works
Weber, S. J., & Cook, T. D. (1972) · Psychological Bulletin · doi:10.1037/h0032351 · Cited by 368 · see citing works
How to show that 9 > 221: Collect judgments in a between-subjects design.
Birnbaum, M. H. (1999) · Psychological Methods · doi:10.1037/1082-989x.4.3.243 · Cited by 230 · see citing works
Elaboration enhances the imagined contact effect
Husnu, S., & Crisp, R. J. (2010) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2010.05.014 · Cited by 219 · see citing works
Prospect theory's reflection hypothesis: A critical examination
Hershey, J. C., & Schoemaker, P. J. H. (1980) · Organizational Behavior and Human Performance · doi:10.1016/0030-5073(80)90037-9 · Cited by 215 · see citing works
Experiments in Wording Questions: II
Rugg, D. (1941) · Public Opinion Quarterly · doi:10.1086/265467 · Cited by 153 · see citing works
Professional respondents in nonprobability online panels
Hillygus, D. S., Jackson, N., & Young, M. (2014) · Wiley eBooks · doi:10.1002/9781118763520.ch10 · Cited by 138 · see citing works
Surveys on surveys: limitations and potentialities
Goyder, J. (1986) · Public Opinion Quarterly · doi:10.1086/268957 · Cited by 123 · see citing works
Prestige, Suggestion, and Attitudes
Lorge, I. D., & Curtiss, C. (1936) · The Journal of Social Psychology · doi:10.1080/00224545.1936.9919891 · Cited by 114 · see citing works
Attitudes Over Time: The Psychology of Panel Conditioning
Sturgis, P., Allum, N., & Brunton‐Smith, I. (2009) · Wiley eBooks · doi:10.1002/9780470743874.ch7 · Cited by 110 · see citing works
The retrospective gambler’s fallacy: Unlikely events, constructing the past, and multiple universes
Oppenheimer, D. M., & Monin, B. (2009) · Judgment and Decision Making · doi:10.1017/s1930297500001170 · Cited by 83 · see citing works
The role of prior knowledge in explaining hypothetical events
Hirt, E. R., & Sherman, S. J. (1985) · Journal of Experimental Social Psychology · doi:10.1016/0022-1031(85)90023-x · Cited by 56 · see citing works
Long-run learning in games of cooperation
Mason, W., Suri, S., & Watts, D. J. (2014) · ACM Conference on Economics and Computation (EC) · doi:10.1145/2600057.2602892 · Cited by 14 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Beyond the Turk: Alternative platforms for crowdsourcing behavioral research
Peer, E., Brandimarte, L., Samat, S., & Acquisti, A. (2017) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2017.01.006 · Cited by 2,957 · see citing works
TurkPrime.com: A versatile crowdsourcing data acquisition platform for the behavioral sciences
Litman, L., Robinson, J., & Abberbock, T. (2016) · Behavior Research Methods · doi:10.3758/s13428-016-0727-z · Cited by 1,978 · see citing works
Data quality of platforms and panels for online behavioral research
Peer, E., Rothschild, D., Gordon, A., Evernden, Z., & Damer, E. (2021) · Behavior Research Methods · doi:10.3758/s13428-021-01694-3 · Cited by 1,718 · see citing works
Understanding and managing the threat of common method bias: Detection, prevention and control
Kock, F., Berbekova, A., & Assaf, A. G. (2021) · Tourism Management · doi:10.1016/j.tourman.2021.104330 · Cited by 1,692 · see citing works
Conducting Clinical Research Using Crowdsourced Convenience Samples
Chandler, J., & Shapiro, D. (2016) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-021815-093623 · Cited by 1,072 · see citing works
Self-citationMTurk Research: Review and Recommendations
Aguinis, H., Villamor, I., & Ramani, R. S. (2020) · Journal of Management · doi:10.1177/0149206320969787 · Cited by 942 · see citing works
Coppock, A., & McClellan, O. A. (2019) · Research & Politics · doi:10.1177/2053168018822174 · Cited by 987 · see citing works
Online panels in social science research: Expanding sampling methods beyond Mechanical Turk
Chandler, J., Rosenzweig, C., Moss, A. J., Robinson, J., & Litman, L. (2019) · Behavior Research Methods · doi:10.3758/s13428-019-01273-7 · Cited by 668 · see citing works
Investigating an alternate form of the cognitive reflection test
Thomson, K. S., & Oppenheimer, D. M. (2016) · Judgment and Decision Making · doi:10.1017/s1930297500007622 · Cited by 612 · see citing works
Crowdsourcing Consumer Research
Goodman, J. K., & Paolacci, G. (2017) · Journal of Consumer Research · doi:10.1093/jcr/ucx047 · Cited by 597 · see citing works
Demand Effects in Survey Experiments: An Empirical Assessment
MUMMOLO, J., & PETERSON, E. (2019) · American Political Science Review · doi:10.1017/s0003055418000837 · Cited by 548 · see citing works
Generalizing from Survey Experiments Conducted on Mechanical Turk: A Replication Approach
Coppock, A. (2019) · Political Science Research and Methods · doi:10.1017/psrm.2018.10 · Cited by 515 · see citing works
Abbey, J. D., & Meloy, M. G. (2017) · Journal of Operations Management · doi:10.1016/j.jom.2017.06.001 · Cited by 522 · see citing works
The shape of and solutions to the MTurk quality crisis
Kennedy, R., Clifford, S., Burleigh, T., Waggoner, P. D., Jewell, R., & Winter, N. J. G. (2020) · Political Science Research and Methods · doi:10.1017/psrm.2020.6 · Cited by 506 · see citing works
Amazon mechanical Turk in organizational psychology: an evaluation and practical recommendations
Cheung, J. H., Burns, D. K., Sinclair, R. R., & Sliter, M. (2017) · Journal of Business and Psychology · doi:10.1007/s10869-016-9458-5 · Cited by 474 · see citing works
Crowdsourcing research: Data collection with Amazon’s Mechanical Turk
Sheehan, K. B. (2018) · Communication Monographs · doi:10.1080/03637751.2017.1342043 · Cited by 408 · see citing works
Rand, D. G. (2016) · Psychological Science · doi:10.1177/0956797616654455 · Cited by 444 · see citing works
Conducting interactive experiments online
Arechar, A. A., Gächter, S., & Molleman, L. (2017) · Experimental Economics · doi:10.1007/s10683-017-9527-2 · Cited by 362 · see citing works
Implications of the Credibility Revolution for Productivity, Creativity, and Progress
Vazire, S. (2018) · Perspectives on Psychological Science · doi:10.1177/1745691617751884 · Cited by 361 · see citing works
The Use of Online Panel Data in Management Research: A Review and Recommendations
Porter, C. O. L. H., Outlaw, R., Gale, J., & Cho, T. S. (2018) · Journal of Management · doi:10.1177/0149206318811569 · Cited by 346 · see citing works
MTurk Character Misrepresentation: Assessment and Solutions
Sharpe Wessling, K., Huber, J., & Netzer, O. (2017) · Journal of Consumer Research · doi:10.1093/jcr/ucx053 · Cited by 330 · see citing works
Keith, M. G., Tay, L., & Harms, P. D. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.01359 · Cited by 319 · see citing works
The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers
Stewart, N., Ungemach, C., Harris, A. J. L., Bartels, D. M., Newell, B. R., Paolacci, G., & Chandler, J. (2015) · Judgment and Decision Making · doi:10.1017/s1930297500005611 · Cited by 304 · see citing works
“Participants previously exposed to an experiment tend to show smaller effect sizes the second time (Chandler, Paolacci, Peer, Mueller, & Ratliff, 2015).”
Data Collection via Online Platforms: Challenges and Recommendations for Future Research
Newman, A., Bavik, Y. L., Mount, M. P., & Shao, B. (2020) · Applied Psychology · doi:10.1111/apps.12302 · Cited by 307 · see citing works
Chandler, J., & Paolacci, G. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550617698203 · Cited by 279 · see citing works
Crowdsourcing Samples in Cognitive Science
Stewart, N., Chandler, J., & Paolacci, G. (2017) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2017.06.007 · Cited by 216 · see citing works
Robinson, J., Rosenzweig, C., Moss, A. J., & Litman, L. (2019) · PLoS ONE · doi:10.1371/journal.pone.0226394 · Cited by 203 · see citing works
Brühlmann, F., Petralito, S., Aeschbach, L. F., & Opwis, K. (2020) · Methods in Psychology · doi:10.1016/j.metip.2020.100022 · Cited by 168 · see citing works
Comparing attentional disengagement between prolific and MTurk samples
Albert, D. A., & Smilek, D. (2023) · Scientific Reports · doi:10.1038/s41598-023-46048-5 · Cited by 166 · see citing works
Necka, E. A., Cacioppo, S., Norman, G. J., & Cacioppo, J. T. (2016) · PLoS ONE · doi:10.1371/journal.pone.0157732 · Cited by 161 · see citing works
Gerpott, F. H., Quaquebeke, N. V., Schlamp, S., & Voelpel, S. C. (2017) · Journal of Business Ethics · doi:10.1007/s10551-017-3625-0 · Cited by 145 · see citing works
Correcting misperceptions of exponential coronavirus growth increases support for social distancing
Lammers, J., Crusius, J., & Gast, A. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2006048117 · Cited by 137 · see citing works
Increasing altruistic and cooperative behaviour with simple moral nudges
Capraro, V., Jagfeld, G., Klein, R., Mul, M., & de Pol, I. V. (2019) · Scientific Reports · doi:10.1038/s41598-019-48094-4 · Cited by 135 · see citing works
Statistical Discrimination and the Rationalization of Stereotypes
Tilcsik, A. (2020) · American Sociological Review · doi:10.1177/0003122420969399 · Cited by 127 · see citing works
Casey, L. S., Chandler, J., Levine, A. S., Proctor, A., & Strolovitch, D. Z. (2017) · SAGE Open · doi:10.1177/2158244017712774 · Cited by 122 · see citing works
Past-focused environmental comparisons promote proenvironmental outcomes for conservatives
Baldwin, M., & Lammers, J. (2016) · Proceedings of the National Academy of Sciences of the United States of America · doi:10.1073/pnas.1610834113 · Cited by 97 · see citing works
Moss, A. J., Rosenzweig, C., Robinson, J., Jaffe, S. N., & Litman, L. (2023) · Behavior Research Methods · doi:10.3758/s13428-022-02005-0 · Cited by 109 · see citing works
Hulland, J. S., & Miller, J. (2018) · Journal of the Academy of Marketing Science · doi:10.1007/s11747-018-0587-4 · Cited by 107 · see citing works
Consumers care and firms should too: On the benefits of disclosing supplier monitoring activities
Duan, Y., Hofer, C., & Aloysius, J. A. (2020) · Journal of Operations Management · doi:10.1002/joom.1129 · Cited by 104 · see citing works
A Note on Increases in Inattentive Online Survey-Takers Since 2020
Ternovski, J., Orr, L., Kalla, J., & Aronow, P. M. (2022) · Journal of Quantitative Description Digital Media · doi:10.51685/jqd.2022.002 · Cited by 103 · see citing works
The Shape of and Solutions to the MTurk Quality CrisisPreprint
Kennedy, R., Clifford, S., Burleigh, T. J., Jewell, R., & Waggoner, P. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3272468 · Cited by 99 · see citing works
Rodd, J. M. (2023) · Journal of Memory and Language · doi:10.1016/j.jml.2023.104472 · Cited by 101 · see citing works
Can we reduce facial biases? Persistent effects of facial trustworthiness on sentencing decisions
Jaeger, B., Todorov, A. T., Evans, A. M., & van Beest, I. (2020) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2020.104004 · Cited by 95 · see citing works
The Political Domain Appears Simpler to the Politically Extreme Than to Political Moderates
Lammers, J., Koch, A. S., Conway, P. J., & Brandt, M. J. (2016) · Social Psychological and Personality Science · doi:10.1177/1948550616678456 · Cited by 92 · see citing works
Savela, N., Kaakinen, M., Ellonen, N., & Oksanen, A. (2021) · Computers in Human Behavior · doi:10.1016/j.chb.2020.106585 · Cited by 76 · see citing works
Arechar, A. A., Kraft‐Todd, G., & Rand, D. G. (2017) · Journal of the Economic Science Association · doi:10.1007/s40881-017-0035-0 · Cited by 90 · see citing works
Çobanoğlu, C., Çavuşoğlu, M., & Türktarhan, G. (2021) · Journal of Global Business Insights · doi:10.5038/2640-6489.6.1.1177 · Cited by 90 · see citing works
Berry, C., Kees, J., & Burton, S. (2022) · Journal of Advertising · doi:10.1080/00913367.2022.2079026 · Cited by 90 · see citing works
Participant Nonnaiveté and the reproducibility of cognitive psychology
Zwaan, R. A., Pecher, D., Paolacci, G., Bouwmeester, S., Verkoeijen, P. P. J. L., Katinka Dijkstra, B., & Zeelenberg, R. (2017) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1348-y · Cited by 89 · see citing works
Wise deliberation sustains cooperation
Grossmann, I., Brienza, J. P., & Bobocel, D. R. (2017) · Nature Human Behaviour · doi:10.1038/s41562-017-0061 · Cited by 87 · see citing works
Capraro, V., & Cococcioni, G. (2016) · Scientific Reports · doi:10.1038/srep27219 · Cited by 87 · see citing works
Time pressure and honesty in a deception game
Capraro, V., Schulz, J., & Rand, D. G. (2019) · Journal of Behavioral and Experimental Economics · doi:10.1016/j.socec.2019.01.007 · Cited by 77 · see citing works
Moving outside the lab: The viability of conducting sensorimotor learning studies online
Tsay, J. S., Lee, A., Ivry, R. B., & Avraham, G. (2021) · Neurons Behavior Data analysis and Theory · doi:10.51628/001c.26985 · Cited by 75 · see citing works
Kan, I. P., & Drummey, A. B. (2018) · Computers in Human Behavior · doi:10.1016/j.chb.2018.02.005 · Cited by 57 · see citing works
Capraro, V., Corgnet, B., Espín, A. M., & Hérnan-Gonzalez, R. (2017) · Royal Society Open Science · doi:10.1098/rsos.160605 · Cited by 53 · see citing works
“Put Yourself in Their Shoes”: Testing Empathy's Ability to Motivate Cosmopolitan Behavior
Faulkner, N. (2017) · Political Psychology · doi:10.1111/pops.12411 · Cited by 52 · see citing works
Exposing omitted moderators: Explaining why effect sizes differ in the social sciences
Krefeld-Schwalb, A., Sugerman, E. R., & Johnson, E. J. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2306281121 · Cited by 60 · see citing works
Conducting Online Behavioral Research Using Crowdsourcing Services in Japan
Majima, Y., Nishiyama, K., Nishihara, A., & Hata, R. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.00378 · Cited by 49 · see citing works
Foundational Tests of the Need-Support Model
Vaughn, L. A. (2017) · Personality and Social Psychology Bulletin · doi:10.1177/0146167216684132 · Cited by 46 · see citing works
Improving the Replicability of Psychological Science Through Pedagogy
Hawkins, R. X. D., Smith, E. N., Au, C., Arias, J. M., Catapano, R., Hermann, E., Keil, M., Lampinen, A. K., Raposo, S., Reynolds, J., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245917740427 · Cited by 46 · see citing works
Media effects on the perceptions of robots
Savela, N., Turja, T., Latikka, R., & Oksanen, A. (2021) · Human Behavior and Emerging Technologies · doi:10.1002/hbe2.296 · Cited by 42 · see citing works
Conducting Web-Based Experiments for Numerical Cognition Research
Kochari, A. R. (2019) · Journal of Cognition · doi:10.5334/joc.85 · Cited by 38 · see citing works
Wingen, T., Berkessel, J. B., & Dohle, S. (2022) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459211070559 · Cited by 39 · see citing works
Biomicroscopy and scanning electron microscopy of early opacities in the aging human lens.
Vrensen, G. F. J. M., & Willekens, B. (1990) · PubMed · Cited by 38 · see citing works
On the priming of risk preferences: The role of fear and general affect
Alempaki, D., Starmer, C., & Tufano, F. (2019) · Journal of Economic Psychology · doi:10.1016/j.joep.2018.12.011 · Cited by 37 · see citing works
Creativity on Paid Crowdsourcing Platforms
Oppenlaender, J., Milland, K., Visuri, A., Ipeirotis, P. G., & Hosio, S. J. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376677 · Cited by 37 · see citing works
Ross, R. M., Hartig, B., & McKay, R. (2016) · Journal of Behavior Therapy and Experimental Psychiatry · doi:10.1016/j.jbtep.2016.08.018 · Cited by 37 · see citing works
Emotional reactions to robot colleagues in a role-playing experiment
Savela, N., Oksanen, A., Pellert, M., & García, D. (2021) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2021.102361 · Cited by 33 · see citing works
Snooping on Mobile Phones: Prevalence and Trends
Marques, D., Muslukhov, I., Guerreiro, T., Beznosov, K., & Carriço, L. M. (2016) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.3264715 · Cited by 30 · see citing works
Krefeld-Schwalb, A., & Scheibehenne, B. (2022) · Marketing Letters · doi:10.1007/s11002-022-09662-3 · Cited by 28 · see citing works
Manago, B. N., Mize, T. D., & Doan, L. (2021) · Sociological Methodology · doi:10.1177/00811750211014242 · Cited by 27 · see citing works
Turkers of the World Unite: Multilevel In-Group Bias Among Crowdworkers on Amazon Mechanical Turk
Almaatouq, A., Krafft, P. M., Dunham, Y. C., Rand, D. G., & Pentland, A. (2019) · Social Psychological and Personality Science · doi:10.1177/1948550619837002 · Cited by 26 · see citing works
Treger, S., Benau, E. M., & Timko, C. A. (2023) · PLOS ONE · doi:10.1371/journal.pone.0285267 · Cited by 25 · see citing works
Gain-loss framing effects in dilemmas of trust and reciprocity
Evans, A. M., & van Beest, I. (2017) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2017.06.012 · Cited by 25 · see citing works
Reducing the number of non-naïve participants in Mechanical Turk samples
Meyers, E. A., Walker, A. C., Fugelsang, J. A., & Koehler, D. J. (2020) · Methods in Psychology · doi:10.1016/j.metip.2020.100032 · Cited by 26 · see citing works
Crowdsourcing in Cognitive and Systems Neuroscience
Johnson, B. P., Dayan, E., Censor, N., & Cohen, L. G. (2021) · The Neuroscientist · doi:10.1177/10738584211017018 · Cited by 23 · see citing works
Bryant, J. S., Gallagher, M. R., Collins, A. C., & Winer, E. S. (2022) · Journal of Behavior Therapy and Experimental Psychiatry · doi:10.1016/j.jbtep.2022.101830 · Cited by 22 · see citing works
A View from the Cockpit: Exploring Pilot Reactions to Attacks on Avionic Systems
Smith, M., Strohmeier, M., Harman, J., Lenders, V., & Martinović, I. (2020) · Network and Distributed System Security Symposium · doi:10.14722/ndss.2020.23022 · Cited by 20 · see citing works
Stressing the Advantages of Female Leadership Can Place Women at a Disadvantage
Lammers, J., & Gast, A. (2017) · Social Psychology · doi:10.1027/1864-9335/a000292 · Cited by 21 · see citing works
Woike, J. K. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02646 · Cited by 19 · see citing works
Ideas Are Cheap: When and Why Adults Value Labor Over Ideas
Burgmer, P., Forstmann, M., & Stavrova, O. (2019) · Journal of experimental psychology. General · doi:10.1037/xge0000473 · Cited by 16 · see citing works
Ambiguity and expectation-neglect in dilemmas of interpersonal trust
Evans, A. M., & Krueger, J. I. (2017) · Judgment and Decision Making · doi:10.1017/s1930297500006719 · Cited by 15 · see citing works
Lukács, G., & Specker, E. (2020) · PLoS ONE · doi:10.1371/journal.pone.0240259 · Cited by 12 · see citing works
Levine, S. S., Reypens, C., & Stark, D. (2021) · Science Advances · doi:10.1126/sciadv.abg9508 · Cited by 16 · see citing works
When the agency wants too much: Experimental evidence on unfair audits and tax compliance
Lancee, B. S., Rossel, L., & Kasper, M. (2023) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2023.08.011 · Cited by 15 · see citing works
Altaf, B., Bianchi, E., Douglas, I. P., Douglas, K., Byers, B. S., Paredes, P. E., Ardoin, N. M., Markus, H. R., Murnane, E. L., Bencharit, L. Z., et al. (2022) · Frontiers in Sustainable Cities · doi:10.3389/frsc.2022.780376 · Cited by 13 · see citing works
Yu, S., & Shea, C. T. (2023) · Organization Science · doi:10.1287/orsc.2020.14640 · Cited by 14 · see citing works
Distinguishing Between Need Support and Regulatory Focus with LIWC
Vaughn, L. A. (2019) · Collabra Psychology · doi:10.1525/collabra.185 · Cited by 14 · see citing works
Why language matters: Shaping public risk tolerance during deterrence crises
Berejikian, J., & Zwald, Z. (2020) · Contemporary Security Policy · doi:10.1080/13523260.2020.1729496 · Cited by 13 · see citing works
Conducting Interactive Experiments OnlinePreprint
Arechar, A. A., Gächter, S., & Molleman, L. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2884409 · Cited by 13 · see citing works
Comprehension in economic games
Koppel, L., Andersson, D., Johannesson, M., Strømland, E., & Tinghög, G. (2025) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2025.107039 · Cited by 12 · see citing works
DeVoe, S. E., & House, J. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.11.004 · Cited by 12 · see citing works
Appreciative Joy Meditation Enhances Acceptance of Unfair Offer in Ultimatum Game
Ng, G. T. T., Lai, D. C. K., Zeng, X., & Oei, T. P. S. (2019) · Mindfulness · doi:10.1007/s12671-019-01139-1 · Cited by 12 · see citing works
Díaz, R., Viciana, H., & Gomila, A. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.00295 · Cited by 11 · see citing works
Why People Hate Congress but Love Their Own Congressperson: An Information Processing Explanation
Lammers, J., Pauels, E., Fleischmann, A., & Galinsky, A. D. (2021) · Personality and Social Psychology Bulletin · doi:10.1177/01461672211002336 · Cited by 11 · see citing works
Wingen, T., Berkessel, J. B., & Dohle, S. (2021) · doi:10.31219/osf.io/7n3mj · Cited by 9 · see citing works
Swaryandini, G., Graham, J., Griffith, S., Grilo, V., Ruzzante, F., Zhang, X., Yeung, S. K., Mangiarulo, M., Basarkod, G., Ng, C., et al. (2025) · Nature Human Behaviour · doi:10.1038/s41562-025-02253-y · Cited by 11 · see citing works
Prolific: Crowdsourcing Academic Online Research
Péer, E. (2024) · Cambridge University Press eBooks · doi:10.1017/9781009000796.005 · Cited by 10 · see citing works
The Daily Lives of Crowdsourced U.S. Respondents: A Time Use Comparison of MTurk, Prolific, and ATUS
Rinderknecht, R. G., Doan, L., & Sayer, L. C. (2025) · Sociological Methodology · doi:10.1177/00811750241312226 · Cited by 10 · see citing works
Haug, M. C. (2017) · Review of Philosophy and Psychology · doi:10.1007/s13164-017-0374-z · Cited by 10 · see citing works
- Authors
- Chandler, J., Paolacci, G., Peer, E., Mueller, P., & Ratliff, K. A.
- Affiliations
- University of Michigan (US)
- Erasmus University Rotterdam (NL)
- Graduate School of Business Administration, Bar-Ilan University
- Princeton University (US)
- University of Florida (US)
- Venue
- Psychological Science
- Publisher
- SAGE Publications
- Published
- 2015-06-10
- Volume
- 26
- Issue
- 7
- Pages
- 1131-1139
- ISSN
- 0956-7976 (ISSN-L)
- 1467-9280
- OpenAlex
- W2162475434
- Cited by
- 185
- References
- 29
- Citations by year
- 2026: 6 · 2025: 20 · 2024: 10 · 2023: 15 · 2022: 10 · 2021: 23 · 2020: 20 · 2019: 21 · 2018: 14 · 2017: 27 · 2016: 15 · 2015: 3
- FWCI
- 29.35
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Science (publishedVersion)
- PubMed (publishedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (publishedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (submittedVersion)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory Processes and Influences
- Keywords
- replication, task familiarity, within-participant design
- MeSH terms
- Adult, Female, Humans, Male, Patient Participation, Patient Selection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
