Contextual sensitivity in scientific reproducibility
Open accessThe DOI matches its Crossref record; 98 of the 98 references it cites were found in a registry or an index.
Abstract
Significance: Scientific progress requires that findings can be reproduced by other scientists. However, there is widespread debate in psychology (and other fields) about how to interpret failed replications. Many have argued that contextual factors might account for several of these failed replications. We analyzed 100 replication attempts in psychology and found that the extent to which the research topic was likely to be contextually sensitive (varying in time, culture, or location) was associated with replication success. This relationship remained a significant predictor of replication success even after adjusting for characteristics of the original and replication studies that previously had been associated with replication success (e.g., effect size, statistical power). We offer recommendations for psychologists and other scientists interested in reproducibility.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“First because, once again, such a low level of reproducibility was not ubiquitous but varied depending on subfield, methodology, and expertise of the authors conducting the replication (23 -25).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [24]
Experimental — accuracy not yet measuredIn the citing paper
“These methodologies have several advantages (40, 41) but can be computationally expensive, create a barrier for adoption for behavioral scientists who lack the required technical expertise, and are most likely context-dependent which can undercut replicability (42).”
From Troll and divide: the language of online polarization (2022) · cites this paper as [42]
In the citing paper
“Therefore, many researchers have begun to push for increased focus on reproducible and open practices in science31,53–55.”
From Guidelines for wrist-worn consumer wearable assessment of heart rate in biobehavioral research (2020) · cites this paper as [53]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 24,532
The weirdest people in the world?
Henrich, J., Heine, S. J., & Norenzayan, A. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0999152x · Cited by 12,641
The Handbook of Social Psychology
Gravatt, A. E., Lindzey, G., & Aronson, F. (1969) · The Family Coordinator · doi:10.2307/582242 · Cited by 12,517
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 1×
“Scientists have speculated that a lack of reproducibility in psychology, as well as in other fields, is the result of a wide range of questionable research practices, including a file-drawer problem (21, 22), low statistical power (23–25), researcher’s degrees of freedom (26), presenting post hoc hypotheses as a prior…”
BackgroundEstimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · cited 1×
“Lakens D (2016) The statistical conclusions in Gilbert et al (2016) are completely invalid.”
BackgroundSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 3×
“Unsuccessful attempts to replicate findings in genetics (1), pharmacology (2), oncology (3), biology (4), and economics (5) have given credence to previous speculation that most published research findings are false (6).”
BackgroundResultWEIRD languages have misled us, too
Majid, A., & Levinson, S. C. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x1000018x · Cited by 4,076
Principles of topological psychology.
Lewin, K. (1936) · McGraw-Hill eBooks · doi:10.1037/10019-000 · Cited by 3,927
Cited by
Many Labs 2: Investigating variation in replicability across samples and settings
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., Aveyard, M., Axt, J. R., Babalola, M. T., Bahník, Š., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918810225 · Cited by 944
“…to replicate when there have been no empirical tests of whether such moderators are operative (Baumeister & Vohs, 2016; Crisp, Miles, & Husnu, 2014; Gilbert, King, Pettigrew, & Wilson, 2016; Ramscar, Shaoul, & Baayen, 2015; Schwarz & Clore, 2016; Stroebe & Strack, 2014; Van Bavel et al., 2016).”
Transparency and replicability in qualitative research: the case of interviews with elite informants
Aguinis, H., & Solarino, A. M. (2019) · Strategic Management Journal · doi:10.1002/smj.3015 · Cited by 791
Replicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770
Psychology, Science, and Knowledge Construction: Broadening Perspectives from the Replication Crisis
Shrout, P. E., & Rodgers, J. L. (2018) · Annual Review of Psychology · doi:10.1146/annurev-psych-122216-011845 · Cited by 725
“…2014), and some have even suggested specific target attention to measurement error (Asendorpf et al. 2013) and to the quality of peer review (Coyne 2016, Nosek & Bar-Anan F o r p e r on a Bar-Anan 2012) and for flexibility regarding the traditional significance level of p < 0.05 (Vazire 2016).”
BackgroundInfluentialConstruct Validation in Social and Personality Research: Current Practice and Recommendations
Flake, J. K., Pek, J., & Hehman, E. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550617693063 · Cited by 697
Yarkoni, T. (2022) · Behavioral and Brain Sciences · doi:10.1017/s0140525x20001685 · Cited by 701
Nelson, L. D., Simmons, J., & Simonsohn, U. (2018) · Annual Review of Psychology · doi:10.1146/annurev-psych-122216-011836 · Cited by 548
Behavioural science is unlikely to change the world without a heterogeneity revolution
Bryan, C. J., Tipton, E., & Yeager, D. S. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01143-3 · Cited by 537
Do Student Samples Provide an Accurate Estimate of the General Public?
Hanel, P. H. P., & Vione, K. C. (2016) · PLOS ONE · doi:10.1371/journal.pone.0168354 · Cited by 510
“However, this reproducibility project was criticized for not taking contextual differences into account, as some studies were replicated in another country, which may have partly caused the low reproducibility rate [13].”
BackgroundZwaan, R. A., Etz, A., Lucas, R. E., & Donnellan, M. B. (2018) · Behavioral and Brain Sciences · doi:10.1017/s0140525x17001972 · Cited by 444
Is science really facing a reproducibility crisis, and do we need it to?
Fanelli, D. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708272114 · Cited by 416
Social Media Elements, Ecologies, and Effects
Bayer, J. B., Triệu, P., & Ellison, N. B. (2020) · Annual Review of Psychology · doi:10.1146/annurev-psych-010419-050944 · Cited by 410
Moshontz, H., Campbell, L., Ebersole, C. R., IJzerman, H., Urry, H. L., Forscher, P. S., Grahe, J. E., McCarthy, R. J., Musser, E. D., Antfolk, J., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918797607 · Cited by 377
Meta-assessment of bias in science
Fanelli, D., Costas, R., & Ioannidis, J. P. A. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1618569114 · Cited by 364
Reflections on the 2016 Decade Award: Incorporating Context in Organizational Research
Johns, G. (2017) · Academy of Management Review · doi:10.5465/amr.2017.0044 · Cited by 348
Challenges and opportunities for grounding cognition.
Barsalou, L. W. (2020) · Journal of Cognition · doi:10.5334/joc.116 · Cited by 319
Lack of theory building and testing impedes progress in the factor and network literature
Fried, E. I. (2020) · Psychological Inquiry · doi:10.1080/1047840x.2020.1853461 · Cited by 317
What meta-analyses reveal about the replicability of psychological research.
Stanley, T. D., Carter, E. C., & Doucouliagos, H. (2018) · Psychological Bulletin · doi:10.1037/bul0000169 · Cited by 307
Tackett, J. L., Lilienfeld, S. O., Patrick, C. J., Johnson, S. L., Krueger, R. F., Miller, J. D., Oltmanns, T. F., & Shrout, P. E. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617690042 · Cited by 251
Nguyen, M.-H., La, V.-P., Le, T.-T., & Vuong, Q.-H. (2022) · MethodsX · doi:10.1016/j.mex.2022.101808 · Cited by 277
Yarkoni, T. (2019) · doi:10.31234/osf.io/jqw35 · Cited by 244
Guidelines for wrist-worn consumer wearable assessment of heart rate in biobehavioral research
Nelson, B. W., Low, C. A., Jacobson, N., Areán, P., Torous, J., & Allen, N. B. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0297-4 · Cited by 241
Variability and Consistency in Early Language Learning: The Wordbank Project
Frank, M. C., Braginsky, M., Yurovsky, D., & Marchman, V. A. (2021) · doi:10.7551/mitpress/11577.001.0001 · Cited by 206
The Oxford Handbook of Psychological Situations
(2020) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780190263348.001.0001 · Cited by 203
Johnson, B. T., & Hennessy, E. A. (2019) · Social Science & Medicine · doi:10.1016/j.socscimed.2019.05.035 · Cited by 201
Field Experiments Across the Social Sciences
Baldassarri, D., & Abascal, M. (2017) · Annual Review of Sociology · doi:10.1146/annurev-soc-073014-112445 · Cited by 196
The Research–Practice Dialogue in Second Language Learning and Teaching: Past, Present, and Future
Sato, M., & Loewen, S. (2022) · The Modern Language Journal · doi:10.1111/modl.12791 · Cited by 195
Brady, W. J., Wills, J. A., Burkart, D., Jost, J. T., & Van Bavel, J. J. (2019) · Journal of Experimental Psychology: General · doi:10.1037/xge0000532 · Cited by 188
Self-citationA manifesto for applying behavioural science
Hallsworth, M. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01555-3 · Cited by 182
Context is Everything (in Emotion Research)
Greenaway, K. H., Kalokerinos, E. K., & Williams, L. A. (2018) · Social and Personality Psychology Compass · doi:10.1111/spc3.12393 · Cited by 175
Advancing Models and Theories for Digital Behavior Change Interventions
Hekler, E. B., Michie, S., Pavel, M., Rivera, D. E., Collins, L. M., Jimison, H. B., Garnett, C., Parral, S., & Spruijt-Metz, D. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.06.013 · Cited by 174
“, Van Bavel et al.(30)) before have argued for the need for specification of when, where, and for whom amechanism of action will produce a targeted effect through moderation testing.”
BackgroundToward a Social Psychophysics of Face Communication
Jack, R. E., & Schyns, P. G. (2017) · Annual Review of Psychology · doi:10.1146/annurev-psych-010416-044242 · Cited by 172
Crowdsourcing hypothesis tests: Making transparent how design choices shape research results.
Landy, J. F., Jia, M. (., Ding, I. L., Viganola, D., Tierney, W., Dreber, A., Johannesson, M., Pfeiffer, T., Ebersole, C. R., Gronau, Q. F., et al. (2020) · Psychological Bulletin · doi:10.1037/bul0000220 · Cited by 168
“Because all participants in each study were drawn from the same large online sample and randomly assigned to conditions, it is unlikely that heterogeneity can be attributed to hidden moderators (e.g., different populations being sampled, different study environments, etc., see Van Bavel et al., 2016), and thus is like…”
InfluentialSelf-citationDemographics of physics education research
Kanim, S., & Cid, X. C. (2020) · Physical Review Physics Education Research · doi:10.1103/physrevphyseducres.16.020106 · Cited by 186
Medin, D., Ojalehto, B., Marin, A., & Bang, M. (2017) · Nature Human Behaviour · doi:10.1038/s41562-017-0088 · Cited by 146
Horn, S. R., Long, M. M., Nelson, B. W., Allen, N. B., Fisher, P. A., & Byrne, M. L. (2018) · Brain, Behavior, and Immunity · doi:10.1016/j.bbi.2018.06.016 · Cited by 129
Reproducibility in systems biology modelling
Tiwari, K., Kananathan, S., Roberts, M. G., Meyer, J. P., Sharif Shohan, M. U., Xavier, A., Maire, M., Zyoud, A., Men, J., Ng, S., et al. (2021) · Molecular Systems Biology · doi:10.15252/msb.20209982 · Cited by 131
“…as a positive finding, unrecognized or incomplete experimental variables, data fabrication or bias, publishing premature or incomplete results 6–9 , 5 and inappropriate Systems biology modelling involves mathematical representation of biological processes to investigate complex behaviour of…”
BackgroundConceptualizing and evaluating replication across domains of behavioral research
Tackett, J. L., & McShane, B. B. (2018) · Behavioral and Brain Sciences · doi:10.1017/s0140525x18000882 · Cited by 124
No Replication, No Trust? How Low Replicability Influences Trust in Psychology
Wingen, T., Berkessel, J. B., & Englich, B. (2020) · Social Psychological and Personality Science · doi:10.1177/1948550619877412 · Cited by 119
“Indeed, many new methodological standards in psychology aim at increasing replicability (Cook, Lloyd, Mellor, Nosek, & Therrien, 2018; Van Bavel et al., 2016).”
BackgroundThe context deficit in leadership research
Johns, G. (2023) · The Leadership Quarterly · doi:10.1016/j.leaqua.2023.101755 · Cited by 180
Current Challenges in Translational and Clinical fMRI and Future Directions
Specht, K. (2020) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2019.00924 · Cited by 144
Registered Replication Report on Mazar, Amir, and Ariely (2008)
Verschuere, B., Meijer, E. H., Jim, A., Hoogesteyn, K., Orthey, R., McCarthy, R. J., Skowronski, J. J., Acar, O. A., Aczel, B., Bakos, B. E., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918781032 · Cited by 117
Scardulla, F., Cosoli, G., Spinsante, S., Poli, A., Iadarola, G., Pernice, R., Busacca, A., Pasta, S., Scalise, L., & D’Acquisto, L. (2023) · Measurement · doi:10.1016/j.measurement.2023.113150 · Cited by 151
Replicability Crisis in Social Psychology: Looking at the Past to Find New Pathways for the Future
Świątkowski, W., & Dompnier, B. (2017) · International Review of Social Psychology · doi:10.5334/irsp.66 · Cited by 114
“Finally, we also pointed out that the neglect of possible auxiliary assumptions relative to context-dependency variables could also account for an unexpectedly low rate of replicability in the domain of social psychology (Van Bavel et al., 2016b).”
BackgroundA synthesis of evidence for policy from behavioural science during COVID-19
Ruggeri, K., Stock, F., Haslam, S. A., Capraro, V., Boggio, P. S., Ellemers, N., Cichocka, A. K., Douglas, K. M., Rand, D. G., van der Linden, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06840-9 · Cited by 144
Motyl, M., Demos, A. P., Carsel, T. S., Hanson, B. E., Melton, Z. J., Mueller, A. B., Prims, J. P., Sun, J., Washburn, A. N., Wong, K. M., et al. (2017) · Journal of Personality and Social Psychology · doi:10.1037/pspa0000084 · Cited by 110
The Flatland Fallacy: Moving Beyond Low–Dimensional Thinking
Jolly, E., & Chang, L. J. (2018) · Topics in Cognitive Science · doi:10.1111/tops.12404 · Cited by 118
Noah, T., Schul, Y., & Mayo, R. (2018) · Journal of Personality and Social Psychology · doi:10.1037/pspa0000121 · Cited by 109
Estimating the deep replicability of scientific findings using human and artificial intelligence
Yang, Y., Youyou, W., & Uzzi, B. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1909046117 · Cited by 101
Kinds of Replication: Examining the Meanings of “Conceptual Replication” and “Direct Replication”
Derksen, M., & Morawski, J. G. (2022) · Perspectives on Psychological Science · doi:10.1177/17456916211041116 · Cited by 120
Research Ethics for Students in the Social Sciences
Bos, J. (2020) · Springer eBooks · doi:10.1007/978-3-030-48415-6 · Cited by 256
Predicting the replicability of social science lab experiments
Altmejd, A., Dreber, A., Forsell, E., Huber, J., Imai, T., Johannesson, M., Kirchler, M., Nave, G., & Camerer, C. (2019) · PLOS ONE · doi:10.1371/journal.pone.0225826 · Cited by 95
“In [40], the author argues that the association between contextual sensitivity (as measured on a scale from 0-5) and replicability found by [8] is spuriously identified from the difference in replication rates between fields.”
BackgroundGurven, M. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1720433115 · Cited by 108
“Thus, while the Reproducibility Project found that only 39% of 100 studies published in top psychology journals could be unambiguously reproduced, a reexamination revealed that studies dealing with topics that are contextually sensitive (to time, place and culture) were less likely to replicate successfully (89).”
BackgroundTheorizing Moral Cognition: Culture in Action, Situations, and Relationships
Luft, A. (2020) · Socius Sociological Research for a Dynamic World · doi:10.1177/2378023120916125 · Cited by 105
“Given the sensitivity of replicability to context (Van Bavel et al. 2016) and priming (Bargh and Chartrand 2000), it is easy to imagine how such simple and seemingly insignificant variations could in fact be very significant for shaping judgments about the targets of self-driving cars.16 The moral…”
BackgroundMachine Learning for Wireless Link Quality Estimation: A Survey
Cerar, G., Yetgin, H., Mohorčič, M., & Fortuna, C. (2021) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2021.3053615 · Cited by 107
Almaatouq, A., Griffiths, T. L., Suchow, J. W., Whiting, M. E., Evans, J. A., & Watts, D. J. (2022) · Behavioral and Brain Sciences · doi:10.1017/s0140525x22002874 · Cited by 181
A Validity-Based Framework for Understanding Replication in Psychology
Fabrigar, L. R., Wegener, D. T., & Petty, R. E. (2020) · Personality and Social Psychology Review · doi:10.1177/1088868320931366 · Cited by 123
“…note that failure to obtain previously demonstrated effects could have resulted from the complexity of the original phenomena under investigation and/or potentially problematic methods used in the replication efforts (e.g., Gilbert et al., 2016; Stroebe & Strack, 2014; Van Bavel et al., 2016).”
BackgroundMethodLaraway, S., Snycerski, S., Pradhan, S., & Huitema, B. E. (2019) · Perspectives on Behavior Science · doi:10.1007/s40614-019-00193-3 · Cited by 103
Troll and divide: the language of online polarization
Simchon, A., Brady, W. J., & Van Bavel, J. J. (2022) · PNAS Nexus · doi:10.1093/pnasnexus/pgac019 · Cited by 85
“Here, we sought to validate a dictionary of polarized language that would be applicable across numerous contexts.”
Self-citationFacets of trust in simulation studies
Harper, A. L., Mustafee, N., & Yearworth, M. (2020) · European Journal of Operational Research · doi:10.1016/j.ejor.2020.06.043 · Cited by 97
Many Labs 2: Investigating Variation in Replicability Across Sample and Setting
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., Aveyard, M., Axt, J., Babalola, M. T., Bahník, Š., et al. (2018) · doi:10.31234/osf.io/9654g · Cited by 75
“…as counterclaims to failures to replicate without empirically 77 testing whether such moderators are operative (Baumeister & Vohs, 2016; Crisp, Miles, & Husnu, 2014; Gilbert et al., 2016; Ramscar, Shaoul, & Baayen, 2015; Schwarz & Clore, 2016; Stroebe & Strack, 2014; van Bavel et al., 2016).”
BackgroundMaking Replication MainstreamPreprint
Zwaan, R. A., Etz, A. J., Lucas, R. E., & Donnellan, B. M. (2017) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/4tg9c · Cited by 92
Kline, A. C., Cooper, A. A., Rytwinski, N. K., & Feeny, N. C. (2020) · Behavior Therapy · doi:10.1016/j.beth.2020.04.015 · Cited by 79
Context matters: Learner beliefs and interactional behaviors in an EFL vs. ESL context
Sato, M., & Storch, N. (2020) · Language Teaching Research · doi:10.1177/1362168820923582 · Cited by 96
Wölfer, R., Christ, O., Schmid, K., Tausch, N., Buchallik, F., Vertovec, S., & Hewstone, M. (2019) · Journal of Personality and Social Psychology · doi:10.1037/pspi0000146 · Cited by 85
Stigma by Prejudice Transfer: Racism Threatens White Women and Sexism Threatens Men of Color
Sanchez, D. T., Chaney, K. E., Manuel, S. K., Wilton, L. S., & Remedios, J. D. (2017) · Psychological Science · doi:10.1177/0956797616686218 · Cited by 76
The replication crisis: How might philosophy and theory of psychology be of use?
Morawski, J. G. (2019) · Journal of Theoretical and Philosophical Psychology · doi:10.1037/teo0000129 · Cited by 78
Point estimates, Simpson’s paradox, and nonergodicity in biological sciences
Mangalam, M., & Kelty‐Stephen, D. G. (2021) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2021.02.017 · Cited by 79
Evolutionary psychology in marketing: Deep, debated, but fancier with fieldwork
Otterbring, T. (2021) · Psychology and Marketing · doi:10.1002/mar.21453 · Cited by 73
Replicating and fixing failed replications: The case of need for cognition and argument quality
Luttrell, A., Petty, R. E., & Xu, M. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2016.09.006 · Cited by 72
Replicator degrees of freedom allow publication of misleading failures to replicate
Bryan, C. J., Yeager, D. S., & O’Brien, J. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1910951116 · Cited by 84
The Case for Measuring and Reporting Bilingualism in Developmental Research
Byers‐Heinlein, K., Esposito, A. G., Winsler, A., Marian, V., Castro, D. C., & Luk, G. (2019) · Collabra Psychology · doi:10.1525/collabra.233 · Cited by 80
Leichtmann, B., Nitsch, V., & Mara, M. (2022) · Frontiers in Robotics and AI · doi:10.3389/frobt.2022.838116 · Cited by 66
Elbæk, C. T., Mitkidis, P., Aarøe, L., & Otterbring, T. (2023) · Nature Communications · doi:10.1038/s41467-023-41007-0 · Cited by 86
Cultural Psychology, Diversity, and Representation in Open Science
Syed, M., & Kathawalla, U. (2021) · Oxford University Press eBooks · doi:10.1093/oso/9780190095949.003.0015 · Cited by 73
Implications of the ABCD study for developmental neuroscience
Feldstein Ewing, S. W., Bjork, J. M., & Luciana, M. (2018) · Developmental Cognitive Neuroscience · doi:10.1016/j.dcn.2018.05.003 · Cited by 64
Evolutionary psychological consumer research: Bold, bright, but better with behavior
Otterbring, T., Sundie, J. M., Li, Y. J., & Hill, S. E. (2020) · Journal of Business Research · doi:10.1016/j.jbusres.2020.07.010 · Cited by 74
The Too-Much-Precision Effect: When and Why Precise Anchors Backfire With Experts
Loschelder, D. D., Friese, M., Schaerer, M., & Galinsky, A. D. (2016) · Psychological Science · doi:10.1177/0956797616666074 · Cited by 77
Bracht, E. M., Monzani, L., Boer, D., Haslam, S. A., Kerschreiter, R., Lemoine, J. E., Steffens, N. K., Akfirat, S. A., Avanzi, L., Barghi, B., et al. (2022) · Applied Psychology · doi:10.1111/apps.12381 · Cited by 84
Increasing the Impact of Materials in and beyond Bio-Nano Science
Björnmalm, M., Faria, M., & Caruso, F. (2016) · Journal of the American Chemical Society · doi:10.1021/jacs.6b08673 · Cited by 68
Hensel, P. G. (2021) · European Management Journal · doi:10.1016/j.emj.2021.01.002 · Cited by 78
Greenfield, P. M. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617707314 · Cited by 69
Geography of scientific knowledge: A proximity approach
Frenken, K. (2020) · Quantitative Science Studies · doi:10.1162/qss_a_00058 · Cited by 61
Science as collaborative knowledge generation
Ellemers, N. (2020) · British Journal of Social Psychology · doi:10.1111/bjso.12430 · Cited by 77
Association between contextual dependence and replicability in psychology may be spurious
Inbar, Y. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1608676113 · Cited by 56
Too close for comfort? The impact of salesperson‐customer proximity on consumers' purchase behavior
Otterbring, T., Wu, F., & Kristensson, P. (2021) · Psychology and Marketing · doi:10.1002/mar.21519 · Cited by 59
“Indeed, research on the downstream effects of physical proximity on customers' purchase behavior is likely prone to contextual sensitivity (Van Bavel et al., 2016) and may thus vary not only as a function of culture, but also due to pandemic‐induced societal norms.”
BackgroundExamining the generalizability of research findings from archival data
Delios, A., Clemente, E. G., Wu, T., Tan, H., Wang, Y., Gordon, M., Viganola, D., Chen, Z., Dreber, A., Johannesson, M., et al. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2120377119 · Cited by 84
Almaatouq, A., Griffiths, T. L., Suchow, J. W., Whiting, M. E., Evans, J. A., & Watts, D. J. (2022) · doi:10.31234/osf.io/anjkm · Cited by 57
RETRACTED ARTICLE: High replicability of newly discovered social-behavioural findings is achievable
Protzko, J., Krosnick, J. A., Nelson, L. D., Nosek, B. A., Axt, J. R., Berent, M., Buttrick, N. R., DeBell, M., Ebersole, C. R., Lundmark, S., et al. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01749-9 · Cited by 97
RetractedFusaroli, R., Grossman, R. B., Bilenberg, N., Cantio, C., Jepsen, J. R. M., & Weed, E. (2021) · Autism Research · doi:10.1002/aur.2661 · Cited by 68
Lack of theory building and testing impedes progress in the factor and network literature
Fried, E. I. (2020) · doi:10.31234/osf.io/zg84s · Cited by 53
The role of anchoring in judgments about expert consensus
Goldberg, M. H., van der Linden, S., Ballew, M. T., Rosenthal, S. A., & Leiserowitz, A. (2019) · Journal of Applied Social Psychology · doi:10.1111/jasp.12576 · Cited by 53
Binning, K. R., & Browman, A. S. (2020) · Social Issues and Policy Review · doi:10.1111/sipr.12066 · Cited by 51
Frightened by an Old Scarecrow: The Remarkable Resilience of Demand Characteristics
Sharpe, D., & Whelton, W. J. (2016) · Review of General Psychology · doi:10.1037/gpr0000087 · Cited by 49
Name Norms: A Guide to Casting Your Next Experiment
Newman, L. S., Tan, M., Caldwell, T. L., Duff, K. J., & Winer, E. S. (2018) · Personality and Social Psychology Bulletin · doi:10.1177/0146167218769858 · Cited by 61
Manipulating Belief in Free Will and Its Downstream Consequences: A Meta-Analysis
Genschow, O., Cracco, E., Schneider, J., Protzko, J., Wisniewski, D., Brass, M., & Schooler, J. (2022) · Personality and Social Psychology Review · doi:10.1177/10888683221087527 · Cited by 45
“Indeed, there is a discussion regarding the degree to which hidden moderators and high context sensitivity influence psychological effects and account for the success of replications (e.g., Stroebe & Strack, 2014; Van Bavel et al., 2016).”
BackgroundHigh Replicability of Newly-Discovered Social-behavioral Findings is AchievablePreprint
Protzko, J., Krosnick, J. A., Nelson, L. D., Nosek, B. A., Axt, J. R., Berent, M. K., Buttrick, N., DeBell, M., Ebersole, C. R., Lundmark, S., et al. (2020) · doi:10.31234/osf.io/n2a9x · Cited by 48
Caruso, E. M., Shapira, O., & Landy, J. F. (2017) · Psychological Science · doi:10.1177/0956797617706161 · Cited by 55
What Can We Learn from Many Labs Replications?
Stroebe, W. (2019) · Basic and Applied Social Psychology · doi:10.1080/01973533.2019.1577736 · Cited by 52
Improving the Replicability of Psychological Science Through Pedagogy
Hawkins, R. X. D., Smith, E. N., Au, C., Arias, J. M., Catapano, R., Hermann, E., Keil, M., Lampinen, A. K., Raposo, S., Reynolds, J., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245917740427 · Cited by 48
Metadata
- Authors
- Van Bavel, J. J., Mende-Siedlecki, P., Brady, W. J., & Reinero, D. A.
- Affiliations
- Department of Psychology, New York University, New York, NY 10003
- New York University (US)
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2016-05-23
- Volume
- 113
- Issue
- 23
- Pages
- 6454-6459
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC4988618
- OpenAlex
- W2397691833
- Cited by
- 398
- References
- 92
- Citations by year
- 2026: 17 · 2025: 23 · 2024: 22 · 2023: 29 · 2022: 47 · 2021: 51 · 2020: 49 · 2019: 48 · 2018: 43 · 2017: 43 · 2016: 26
- FWCI
- 75.91
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- OSF Preprints (OSF Preprints) (submittedVersion)
- Funders
- National Science Foundation (1555131)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › General Psychology › Academic and Historical Perspectives in Psychology
- Keywords
- scientific reproducibility, replication success, context sensitivity, replication study, statistical power, degrees of freedom, effect size
- MeSH terms
- Humans, Psychology, Publishing, Research, Science, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.