Contextual sensitivity in scientific reproducibility
Open accessThe DOI matches its Crossref record; 98 of the 98 references it cites were found in a registry or an index.
Abstract
Significance: Scientific progress requires that findings can be reproduced by other scientists. However, there is widespread debate in psychology (and other fields) about how to interpret failed replications. Many have argued that contextual factors might account for several of these failed replications. We analyzed 100 replication attempts in psychology and found that the extent to which the research topic was likely to be contextually sensitive (varying in time, culture, or location) was associated with replication success. This relationship remained a significant predictor of replication success even after adjusting for characteristics of the original and replication studies that previously had been associated with replication success (e.g., effect size, statistical power). We offer recommendations for psychologists and other scientists interested in reproducibility.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“First because, once again, such a low level of reproducibility was not ubiquitous but varied depending on subfield, methodology, and expertise of the authors conducting the replication (23 -25).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [24]
Experimental — accuracy not yet measuredIn the citing paper
“These methodologies have several advantages (40, 41) but can be computationally expensive, create a barrier for adoption for behavioral scientists who lack the required technical expertise, and are most likely context-dependent which can undercut replicability (42).”
From Troll and divide: the language of online polarization (2022) · cites this paper as [42]
In the citing paper
“Therefore, many researchers have begun to push for increased focus on reproducible and open practices in science31,53–55.”
From Guidelines for wrist-worn consumer wearable assessment of heart rate in biobehavioral research (2020) · cites this paper as [53]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 24,532
The weirdest people in the world?
Henrich, J., Heine, S. J., & Norenzayan, A. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0999152x · Cited by 12,641
The Handbook of Social Psychology
Gravatt, A. E., Lindzey, G., & Aronson, F. (1969) · The Family Coordinator · doi:10.2307/582242 · Cited by 12,517
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 1×
“Scientists have speculated that a lack of reproducibility in psychology, as well as in other fields, is the result of a wide range of questionable research practices, including a file-drawer problem (21, 22), low statistical power (23–25), researcher’s degrees of freedom (26), presenting post hoc hypotheses as a prior…”
BackgroundEstimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · cited 1×
“Lakens D (2016) The statistical conclusions in Gilbert et al (2016) are completely invalid.”
BackgroundSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 3×
“Unsuccessful attempts to replicate findings in genetics (1), pharmacology (2), oncology (3), biology (4), and economics (5) have given credence to previous speculation that most published research findings are false (6).”
BackgroundResultWEIRD languages have misled us, too
Majid, A., & Levinson, S. C. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x1000018x · Cited by 4,076
Principles of topological psychology.
Lewin, K. (1936) · McGraw-Hill eBooks · doi:10.1037/10019-000 · Cited by 3,927
Cited by
Items 201–300 of 400
The Crisis in Psychological Science and the Need for a Person-Oriented Approach
Lundh, L. (2019) · Theory and history in the human and social sciences · doi:10.1007/978-3-030-33099-6_12 · Cited by 16
Replicator Degrees of Freedom Allow Publication of Misleading 'Failures to Replicate'Preprint
Bryan, C. J., Yeager, D. S., & O’Brien, J. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3408200 · Cited by 14
Dufner, M., Brümmer, M., Chung, J. M., Drewke, P. M., Blaison, C., & Schmukle, S. C. (2017) · Psychological Science · doi:10.1177/0956797617734315 · Cited by 12
Williams, M. J., Mullane, K., & Curtis, M. J. (2017) · Elsevier eBooks · doi:10.1016/b978-0-12-804725-5.00005-7 · Cited by 13
Retiring Popper: Critical realism, falsificationism, and the crisis of replication
Archer, R. P. (2024) · Theory & Psychology · doi:10.1177/09593543241250079 · Cited by 14
“As Van Bavel et al. (2016) note, “the role of context is frequently overlooked – and even dismissed [emphasis added] – in the evaluation of replication results.”
Eating moderates the link between body mass index and perceived social connection
Jaremka, L. M., Sunami, N., & Nadzan, M. A. (2017) · Appetite · doi:10.1016/j.appet.2017.01.016 · Cited by 12
“Indeed, a recent review concluded that replication success was more likely when the result in question had relatively low contextual sensitivity (van Bavel et al., 2016).”
Khanna, P. (2019) · Springer eBooks · doi:10.1007/978-981-10-5251-4_59 · Cited by 19
Wingen, T., Berkessel, J. B., & Dohle, S. (2021) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/7n3mj · Cited by 9
Lewis, M. L., Mathur, M. B., VanderWeele, T. J., & Frank, M. C. (2020) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/pbrdk · Cited by 16
Lakens, D. (2023) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/dtvs7 · Cited by 17
The Daily Lives of Crowdsourced U.S. Respondents: A Time Use Comparison of MTurk, Prolific, and ATUS
Rinderknecht, R. G., Doan, L., & Sayer, L. C. (2025) · Sociological Methodology · doi:10.1177/00811750241312226 · Cited by 16
“Concern with the “replication crisis” in psychology and other fields has brought to light the importance of hidden moderators shaping the applicability of some theories to specific settings (Stroebe and Strack 2014; Van Bavel et al. 2016; Weinberg, Freese, and McElhattan 2014).”
A transparency statement improves trust in community-police interactions
Dobson, K. S. H., Dittmann, A. G., & Yeager, D. S. (2025) · Nature Communications · doi:10.1038/s41467-024-55709-6 · Cited by 27
Character Trouble: Undisciplined Essays on Moral Agency and Personality
Doris, J. M. (2021) · doi:10.1093/oso/9780198719601.001.0001 · Cited by 18
Estimating the replicability of technology education research
Buckley, J., Hyland, T., & Seery, N. (2022) · International Journal of Technology and Design Education · doi:10.1007/s10798-022-09787-6 · Cited by 12
“…experimental procedure (Stroebe & Strack, 2014) and that a direct replication is often not possible due to contextual sensitivity— that variability in local and/or cultural context can impact an experiment even if proce-durally identical (Fabrigar & Wegener, 2016; Van Bavel et al., 2016a, 2016b).”
MethodSunami, N., Nadzan, M. A., & Jaremka, L. M. (2018) · Social Psychological and Personality Science · doi:10.1177/1948550618762301 · Cited by 14
Jaremka, L. M., Kane, H. S., Sunami, N., Lebed, O., & Austin, K. A. (2020) · Personal Relationships · doi:10.1111/pere.12338 · Cited by 11
Otterbring, T. (2022) · Journal of Financial Services Marketing · doi:10.1057/s41264-021-00131-y · Cited by 12
Prinz, R. J. (2016) · Child Abuse & Neglect · doi:10.1016/j.chiabu.2016.12.005 · Cited by 10
Amir, O., Mažar, N., & Ariely, D. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918769062 · Cited by 10
Brady, W. J., & Bavel, J. J. V. (2021) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/s4w2x · Cited by 17
Self-citationLi, X., Liu, J., Gao, W., & Cohen, G. L. (2024) · PLoS ONE · doi:10.1371/journal.pone.0306911 · Cited by 15
Moving beyond unwise replication practices: The case of romantic motivation.
Sundie, J. M., Beal, D. J., Neuberg, S. L., & Kenrick, D. T. (2019) · Journal of Experimental Psychology General · doi:10.1037/xge0000527 · Cited by 10
Siem, B., Neymeyer, L., & Rohmann, A. (2020) · Social Psychology · doi:10.1027/1864-9335/a000432 · Cited by 11
Saito, T. I., Motoki, K., Nouchi, R., Kawashima, R., & Sugiura, M. (2019) · PLoS ONE · doi:10.1371/journal.pone.0220883 · Cited by 10
Designing optimal behavioral experiments using machine learning
Valentin, S., Kleinegesse, S., Bramley, N. R., Seriès, P., Gutmann, M. U., & Lucas, C. G. (2024) · eLife · doi:10.7554/elife.86224 · Cited by 8
“The methodologies discussed in this tutorial make no assumption, however, on whether or not the likelihood function of the computational model needs be tractable, which opens up the space of scientific theories that can be tested considerably.”
MethodMethods for Studying the Contextual Nature of Implicit Cognition
Shepherd, H. R. (2019) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780190273385.013.20 · Cited by 10
Carter, T. J., Pandey, G., Bolger, N., Hassin, R. R., & Ferguson, M. (2020) · Social Cognition · doi:10.1521/soco.2020.38.6.489 · Cited by 5
Wang, M.-J., Yogeeswaran, K., Sivaram, S., & Nash, K. A. (2021) · Humanities and Social Sciences Communications · doi:10.1057/s41599-021-00987-4 · Cited by 10
On the role of contextual factors in cognitive neuroscience experiments: a mechanistic approach
Paz, A. W., & Rojas‐Líbano, D. (2022) · Synthese · doi:10.1007/s11229-022-03870-0 · Cited by 9
“…and intra-subject variability (Goodhew & Edwards, 2019; Seghier & Price, 2018), characterizing the sources of replicability issues (Gelman, 2015; Van Bavel et al., 2016), addressing the challenges of the ecological validity of experimental results (Adolph, 2020; Miller et al., 2019; 1 3 Page 21…”
BackgroundWhite Lies in Hand: Are Other-Oriented Lies Modified by Hand Gestures? Possibly Not
Cantarero, K., Parzuchowski, M., & Dukała, K. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.00814 · Cited by 10
Towards a Replicable and Relevant Social Psychology
Epstude, K. (2017) · Social Psychology · doi:10.1027/1864-9335/a000303 · Cited by 8
Is the political slant of psychology research related to scientific replicability?Preprint
Reinero, D. A., Wills, J. A., Brady, W. J., Mende‐Siedlecki, P., Crawford, J. T., & Bavel, J. J. V. (2019) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/6k3j5 · Cited by 11
Self-citationGrand challenge: social psychology without hubris
Jost, J. T. (2024) · Frontiers in Social Psychology · doi:10.3389/frsps.2023.1283272 · Cited by 8
Evaluating meta-analysis as a replication success measure
Muradchanian, J., Hoekstra, R., Kiers, H. A. L., & van Ravenzwaaij, D. (2024) · PLoS ONE · doi:10.1371/journal.pone.0308495 · Cited by 9
Generalizability of choice architecture interventions
Szászi, B., Goldstein, D. G., Soman, D., & Michie, S. (2025) · Nature Reviews Psychology · doi:10.1038/s44159-025-00471-9 · Cited by 22
“…53 showed that replication success was negatively associated with the contextual sensitivity (the extent to which independent coders predicted that a finding would vary depending on time, culture or location) 71 , confirming the importance of contextual moderators behind heterogeneous results.”
Brady, W., Rathje, S., Globig, L. K., & Bavel, J. J. V. (2025) · PNAS Nexus · doi:10.1093/pnasnexus/pgaf327 · Cited by 6
“This is likely to be the case in domains where the topics under investigation are sensitive to the social context (22), such as political and moral psychology (23).”
Self-citationWhere’s the tech in sex research? A brief critique and call for research
Shaughnessy, K., & Braham, J. (2021) · The Canadian Journal of Human Sexuality · doi:10.3138/cjhs.2021-0026 · Cited by 7
Wikström, S., Hövel, H., Hansen‐Pupp, I., Fellman, V., Hüppi, P. S., Ley, D., & Hellström-Westas, L. K. (2017) · Neonatology · doi:10.1159/000479423 · Cited by 7
Stereotype Threat and Educational Tracking: A Field Experiment in Chinese Vocational High Schools
Chu, J., Loyalka, P. K., Li, G., Gao, L., & Song, Y. (2018) · Socius Sociological Research for a Dynamic World · doi:10.1177/2378023118782011 · Cited by 10
“Scholars have identified several important conditions that must be present before stereotypes are threatening (Cesario 2014; Lamont, Swift, and Abrams 2015; Van Bavel et al. 2016).”
BackgroundDubèl, R., Schumacher, G., Homan, M. D., Peterson, D., & Bakker, B. N. (2024) · Media and Communication · doi:10.17645/mac.7807 · Cited by 9
“SFN found little indication that country characteristics explained variation in the negativity bias, thereby addressing concerns that the negativity bias is conditional upon context (e.g., Van Bavel et al., 2016).”
BackgroundDo individuation instructions reduce the cross-race effect? A registered replication of
Cruz, F., Palma, T. A., Bansemer, E., Correll, J., Fonseca, S., Gonçalves, P., & Santos, A. S. (2022) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2022.104423 · Cited by 11
Open science initiatives: challenges for public health promotion
Holzmeyer, C. (2018) · Health Promotion International · doi:10.1093/heapro/day002 · Cited by 7
The Flatland Fallacy: Moving Beyond Low Dimensional ThinkingPreprint
Jolly, E., & Chang, L. J. (2017) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/zdrfg · Cited by 8
A Tale of Open Science: Emergence of a New Normal
Soliman, M., Sarstedt, M., Adler, S. J., Siegfried, D., Genschow, O., & Imschloß, M. (2025) · Schmalenbach Journal of Business Research · doi:10.1007/s41471-025-00218-5 · Cited by 9
“At least partial support for this notion comes from research showing that the success rate of replicating social psychological findings strongly depends on contextual factors, including time, culture, location, and population (van Bavel et al. 2016).”
The role of generalizability in moral and political psychology
Harris, E. A., Pärnamets, P., Brady, W. J., Robertson, C. E., & Bavel, J. J. V. (2022) · Behavioral and Brain Sciences · doi:10.1017/s0140525x2100042x · Cited by 7
Self-citationPsychology: Where history, culture, and biology meet
Newton, T. (2022) · Theory & Psychology · doi:10.1177/09593543221131782 · Cited by 21
“…a Galilean aspiration may partly explain why there has been a lengthy and continuing denial of differences between psychology and natural science, seemingly motivated by a desire to maintain psychology’s status as a proper science (e.g., Schlenker, 1974; Van Bavel et al., 2016; Zwaan et al., 2018).”
BackgroundOn Methods and Marshmallows: A Roadmap for Science That Is Openly Feminist and Radically Open
Gervais, S. J., Baildon, A. E., & Lorenz, T. K. (2021) · Psychology of Women Quarterly · doi:10.1177/03616843211032632 · Cited by 8
“As feminist social scientists, we know that context—obvious and subtle cues—can powerfully shape how participants respond in behavioral studies (Klein et al., 2012; Van Bavel et al., 2016).”
BackgroundRecruitment Strategies Bias Sampling and Shape Replicability
Vaughan‐Johnston, T. I., Imtiaz, F., Patro, G. A., Shang, S. X., Fabrigar, L. R., & Ji, L.-J. (2024) · Personality and Social Psychology Bulletin · doi:10.1177/01461672241293504 · Cited by 7
“…such explanations may suggest that the methodological change shifted the underlying effect itself (e.g., Gilbert et al., 2016; Kerr et al., 2018; Van Bavel et al., 2016) or that operational-izations may not have captured the same type of variance among the original and replication populations…”
Purwowidodo, A. (2023) · JURNAL IQRA · doi:10.25217/ji.v8i2.3395 · Cited by 7
“…learning content online, offline, and mobile, with a learning model that provides the availability of easily accessible and visually nuanced subject matter on handheld and mobile electronic devices, such as smartphones, tablets, and iPads (Kumar Basak et al., 2018) (Van Bavel et al., 2016).”
BackgroundSchimmelpfennig, R., Spicer, R., White, C., Gervais, W. M., Norenzayan, A., Heine, S. J., Henrich, J., & Muthukrishna, M. (2024) · Communications Psychology · doi:10.1038/s44271-024-00135-z · Cited by 9
Agential realism as an alternative philosophy of science perspective for quantitative psychology
Scholz, J. F. (2024) · Frontiers in Psychology · doi:10.3389/fpsyg.2024.1410047 · Cited by 9
Favorable Evaluations of Black and White Women’s Workplace Anger During the Era of #MeToo
McCormick‐Huhn, K., & Shields, S. A. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.594260 · Cited by 10
“The intersectional positions (i.e., gender and race) we selected for comparison were derived from our research questions (Warner, 2008).”
MethodOn effects that do occur versus effects that can be made to occur
Bräuer, M., & Kennedy, K. R. (2023) · Frontiers in Social Psychology · doi:10.3389/frsps.2023.1193349 · Cited by 9
Sato, M. (2020) · Language Teaching · doi:10.1017/s0261444820000610 · Cited by 5
Pati, S. P., & Kakani, R. K. (2022) · Review of Public Personnel Administration · doi:10.1177/0734371x211062481 · Cited by 7
“In the past, scholars have expressed concern that public administration research is not aligned with the conceptual and theoretical advancement of the mainstream management literature (see Van Wart, 2013).”
BackgroundSternisko, A., Delouvée, S., & Bavel, J. J. V. (2022) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2022.104357 · Cited by 6
Pot la psicologia rescatar-se a si mateixa?. Incentius, biaix i replicabilitat
Fernando, B., César, P. L. J., & Vadillo, M. A. (2018) · Anuari de Psicologia de la Societat Valenciana de Psicologia · doi:10.7203/anuari.psicologia.18.2.231 · Cited by 5
Shanks, D. R., & Vadillo, M. A. (2019) · Journal of Experimental Psychology General · doi:10.1037/xge0000597 · Cited by 5
Gleibs, I. H., & Alvarado, A. L. (2019) · Social Psychological Bulletin · doi:10.32872/spb.v14i3.35320 · Cited by 6
Khanna, P. (2018) · Springer eBooks · doi:10.1007/978-981-10-2779-6_59-1 · Cited by 8
Crawford, J. T., Vodapalli, S., Stingel, R., & Ruscio, J. (2018) · Social Psychological and Personality Science · doi:10.1177/1948550618792057 · Cited by 5
An ideological asymmetry in the diffusion of moralized content among political elites
Brady, W. J., Van Bavel, J. J., Jost, J., & Wills, J. A. (2018) · doi:10.31234/osf.io/43n5e · Cited by 5
Self-citationThe "Tau" of Science - How to Measure, Study, and Integrate Quantitative and Qualitative KnowledgePreprint
Fanelli, D. (2022) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/67sak · Cited by 5
Reproducibility in systems biology modelling
Tiwari, K., Kananathan, S., Roberts, M. G., Meyer, J. P., Sharif Shohan, M. U., Xavier, A., Maire, M., Zyoud, A., Men, J., Ng, S., et al. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.08.07.239855 · Cited by 131
Hamilton, D. G., McKenzie, J. E., Nguyen, P., Rethlefsen, M. L., McDonald, S. J., Brennan, S. E., Fidler, F., Higgins, J. P. T., Kanukula, R., Karunananthan, S., et al. (2026) · Research Synthesis Methods · doi:10.1017/rsm.2025.10064 · Cited by 8
“…contrast to replication work involving participants, where experimental set ups can be sensitive to seemingly innocuous contextual factors (i.e., ‘hidden moderators’) [61], we were able to directly observe, investigate and resolve discrepancies, often with the assistance of the original reviewers.”
Teymoori, A., & Trappes, R. (2026) · Review of General Psychology · doi:10.1177/10892680261421875 · Cited by 7
“Proponents of theory development argue that direct replication of a study is nigh on impossible due to inevitable heterogeneity in the time, context and population of studies (Stroebe & Strack, 2014), factors which are seen as confounding variables in replication studies (Van Bavel et al., 2016).”
Concerns About Replicability Across Two Crises in Social Psychology
Lakens, D. (2025) · International Review of Social Psychology · doi:10.5334/irsp.1036 · Cited by 8
Many Labs 5: Registered Replication of Crosby, Monin, and Richardson (2008)
Rabagliati, H., Corley, M. M. B., Dering, B. R., Hancock, P., King, J. P. J., Levitan, C., Loy, J. E., & Millen, A. E. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919870737 · Cited by 4
Volinn, E. (2023) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.13783 · Cited by 4
The Promise of Experiments for Studying the Presidency
Cohen, J. Е. (2017) · Presidential Studies Quarterly · doi:10.1111/psq.12389 · Cited by 4
Lack of Replication or Generalization? Cultural Values Explain a Question Wording Effect
Silber, H., Tvinnereim, E. M., Stark, T., Blom, A. G., Krosnick, J. A., Bošnjak, M., Clement, S. L., Cornilleau, A., Cousteaux, A.-S., John, M., et al. (2021) · Journal of Survey Statistics and Methodology · doi:10.1093/jssam/smab007 · Cited by 6
Interventions for Rural Caregiver Health: An Integrative Review
Braley, T. L., Eisenhauer, C. M., & Rowland, S. A. (2022) · Research in Gerontological Nursing · doi:10.3928/19404921-20220204-01 · Cited by 4
Pichardo, E. J. R., Salfate, S. V., & Jost, J. T. (2022) · Cambridge University Press eBooks · doi:10.1017/9781108779104.041 · Cited by 6
Kato, T. (2018) · The Japanese Journal of Personality · doi:10.2132/personality.27.2.11 · Cited by 6
Siblings, shopping, and sustainability: Birth-order differences in green consumption
Otterbring, T., Sundgot‐Borgen, J., Bratland‐Sanda, S., & Trangsrud, L. K. J. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1105072 · Cited by 4
OMICtools: a community-driven search engine for biological data analysisPreprint
Perrin, H., Denorme, M., Grosjean, J., Community, O., Dynomant, E., Henry, V., Pichon, F., Darmoni, S. J., Desfeux, A., & Gonzalez, B. J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.03659 · Cited by 6
Muradchanian, J., Hoekstra, R., Kiers, H. A. L., & van Ravenzwaaij, D. (2020) · doi:10.31222/osf.io/wvdjf · Cited by 4
Troll and Divide: The Language of Online PolarizationPreprint
Simchon, A., Brady, W. J., & Bavel, J. J. V. (2020) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/xjd64 · Cited by 7
Self-citationResearch into evidence-based psychological interventions needs a stronger focus on replicability
Niemeyer, H., Knaevelsrud, C., van Aert, R. C. M., & Ehring, T. (2023) · Clinical Psychology in Europe · doi:10.32872/cpe.9997 · Cited by 3
Our brains sense the future through a new quantum-like implicit learning mechanism
Escolà-Gascón, Á. (2024) · Brain Research Bulletin · doi:10.1016/j.brainresbull.2024.111048 · Cited by 3
Hoffmann, J., Twardawski, M., Höhs, J. M., Gast, A., Pohl, S., & Sengewald, M.-A. D.-P. (2025) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459251328273 · Cited by 7
“…error and statistical inference (e.g., Fiedler & Prager, 2018; Lewandowsky & Oberauer, 2020; Loken, & Gelman, 2017), unintended differences between the characteristics of primary and replication studies provide possible explanations (e.g., Nosek & Errington, 2017; Van Bavel et al., 2016).”
Three strategies for claiming research progress
Gal, D., Calder, B. J., & Sternthal, B. (2026) · Journal of Consumer Psychology · doi:10.1002/jcpy.70033 · Cited by 3
Shockvertising: The Effect of Disgust Exposure on Viewers’ Unconscious Behavioral Responses
Fumagalli, E., & Shrum, L. (2023) · Current Research in Ecological and Social Psychology · doi:10.1016/j.cresp.2023.100172 · Cited by 3
Context is Key for Reproducibility of Empirical Studies in Human-Computer Interaction
Vanderdonckt, J. M., & Vatavu, R.-D. (2025) · ACM Conference on Reproducibility and Replicability (ACM REP) · doi:10.1145/3736731.3746142 · Cited by 6
“Instead, it requires ensuring that the research findings remain valid and reliable in different contexts of use [8] involving variations in the team conducting the research, the platforms employed, and the environmental conditions under which experiments are administered [11].”
Context sensitivity and failed replications in linguistics – a reply to Grieve
Roettger, T. B. (2021) · Linguistics · doi:10.1515/ling-2020-0239 · Cited by 4
Bieri, A., & Rossel, T. (2022) · CHIMIA International Journal for Chemistry · doi:10.2533/chimia.2022.368 · Cited by 3
Chan, E. W., Wong, J., Chan, C. S., & Cheung, F. (2022) · Springer eBooks · doi:10.1007/978-3-031-04968-2_18 · Cited by 4
A new rationalist account of the development of false-belief understanding
Antilici, F. (2023) · Philosophical Studies · doi:10.1007/s11098-023-01986-8 · Cited by 3
Giving Away the Science of School Psychology
Ash, T. L. (2024) · School Psychology Review · doi:10.1080/2372966x.2024.2417234 · Cited by 3
“Within this context then, failures to replicate do not necessarily insin-uate that the original findings were not valid (Van Bavel et al., 2016).”
Gawronski, B., Brannon, S. M., & Ng, N. L. (2022) · Social Cognition · doi:10.1521/soco.2022.40.6.580 · Cited by 3
“Although we cannot rule out that the conflicting outcomes in the two studies are both false positives, our proposed post-hoc interpretation aligns with concerns that some effects may be real but difficult to replicate, because they are sensitive to contextual influences (Van Bavel et al., 2016).”
ResultRodela, R., Pérez‐Soba, M., Bregt, A. K., & Verweij, P. (2018) · Computers Environment and Urban Systems · doi:10.1016/j.compenvurbsys.2018.06.011 · Cited by 3
Passmore, D. L., & Chae, C. (2019) · Advances in Developing Human Resources · doi:10.1177/1523422319870173 · Cited by 4
What Predicts Stroop Performance? A Conditional Random Forest ApproachPreprint
IJzerman, H., Pollet, T. V., Ebersole, C. R., & Kun, D. (2017) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/scpe2 · Cited by 4
Clinical Science and the Replicability CrisisPreprint
Tackett, J. L., Lilienfeld, S. O., Patrick, C. J., Johnson, S. L., Krueger, R. F., Miller, J. D., Oltmanns, T. F., & Shrout, P. E. (2016) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/uwus7 · Cited by 3
Replication or Generalizability? How Flexible Inferences Uphold Unfounded Universal ClaimsPreprint
Syed, M. (2023) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/znv5r · Cited by 3
AITurk: Using ChatGPT for Social Science ResearchPreprint
Qin, X., Huang, M., & Ding, J. (2024) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/xkd23 · Cited by 4
Literature Reviews: An Overview of Systematic, Integrated, and Scoping Reviews
Turner, J. R. (2023) · Springer eBooks · doi:10.1007/978-981-99-1284-1_38 · Cited by 3
Making Multimethod Latent State–Trait Models for Random and Fixed Situations Accessible: A Tutorial
Tinhof, D. L., & Mayer, A. (2025) · Journal of Personality · doi:10.1111/jopy.13031 · Cited by 2
“In line with prior findings (Deinzer et al. 1995; van Bavel et al. 2016), this highlights the context sensitivity of psychological constructs like the Big Five and underscores the need to consider fixed situations in personality research.”
Can Blockchain-Based Authenticity Disclosures Increase Consumer Trust in Online Advertising?
Antsipava, D., van Reijmersdal, E. A., Strycharz, J., & van Noort, G. (2025) · Journal of Advertising · doi:10.1080/00913367.2025.2510233 · Cited by 7
“To demonstrate the robustness of the findings in Study 1 and to examine whether these results are reproducible in an alternative context (Van Bavel et al. 2016), we try to replicate these results in Study 2 in the context of Google search ads.”
Meta-regression to explain shrinkage and heterogeneity in large-scale replication projects
Heyard, R., & Held, L. (2025) · PLoS ONE · doi:10.1371/journal.pone.0327799 · Cited by 4
Metadata
- Authors
- Van Bavel, J. J., Mende-Siedlecki, P., Brady, W. J., & Reinero, D. A.
- Affiliations
- Department of Psychology, New York University, New York, NY 10003
- New York University (US)
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2016-05-23
- Volume
- 113
- Issue
- 23
- Pages
- 6454-6459
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC4988618
- OpenAlex
- W2397691833
- Cited by
- 398
- References
- 92
- Citations by year
- 2026: 17 · 2025: 23 · 2024: 22 · 2023: 29 · 2022: 47 · 2021: 51 · 2020: 49 · 2019: 48 · 2018: 43 · 2017: 43 · 2016: 26
- FWCI
- 75.91
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- OSF Preprints (OSF Preprints) (submittedVersion)
- Funders
- National Science Foundation (1555131)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › General Psychology › Academic and Historical Perspectives in Psychology
- Keywords
- scientific reproducibility, replication success, context sensitivity, replication study, statistical power, degrees of freedom, effect size
- MeSH terms
- Humans, Psychology, Publishing, Research, Science, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.