Contextual sensitivity in scientific reproducibility
Open accessThe DOI matches its Crossref record; 98 of the 98 references it cites were found in a registry or an index.
Abstract
Significance: Scientific progress requires that findings can be reproduced by other scientists. However, there is widespread debate in psychology (and other fields) about how to interpret failed replications. Many have argued that contextual factors might account for several of these failed replications. We analyzed 100 replication attempts in psychology and found that the extent to which the research topic was likely to be contextually sensitive (varying in time, culture, or location) was associated with replication success. This relationship remained a significant predictor of replication success even after adjusting for characteristics of the original and replication studies that previously had been associated with replication success (e.g., effect size, statistical power). We offer recommendations for psychologists and other scientists interested in reproducibility.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“First because, once again, such a low level of reproducibility was not ubiquitous but varied depending on subfield, methodology, and expertise of the authors conducting the replication (23 -25).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [24]
Experimental — accuracy not yet measuredIn the citing paper
“These methodologies have several advantages (40, 41) but can be computationally expensive, create a barrier for adoption for behavioral scientists who lack the required technical expertise, and are most likely context-dependent which can undercut replicability (42).”
From Troll and divide: the language of online polarization (2022) · cites this paper as [42]
In the citing paper
“Therefore, many researchers have begun to push for increased focus on reproducible and open practices in science31,53–55.”
From Guidelines for wrist-worn consumer wearable assessment of heart rate in biobehavioral research (2020) · cites this paper as [53]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 24,532
The weirdest people in the world?
Henrich, J., Heine, S. J., & Norenzayan, A. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0999152x · Cited by 12,641
The Handbook of Social Psychology
Gravatt, A. E., Lindzey, G., & Aronson, F. (1969) · The Family Coordinator · doi:10.2307/582242 · Cited by 12,517
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 1×
“Scientists have speculated that a lack of reproducibility in psychology, as well as in other fields, is the result of a wide range of questionable research practices, including a file-drawer problem (21, 22), low statistical power (23–25), researcher’s degrees of freedom (26), presenting post hoc hypotheses as a prior…”
BackgroundEstimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · cited 1×
“Lakens D (2016) The statistical conclusions in Gilbert et al (2016) are completely invalid.”
BackgroundSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 3×
“Unsuccessful attempts to replicate findings in genetics (1), pharmacology (2), oncology (3), biology (4), and economics (5) have given credence to previous speculation that most published research findings are false (6).”
BackgroundResultWEIRD languages have misled us, too
Majid, A., & Levinson, S. C. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x1000018x · Cited by 4,076
Principles of topological psychology.
Lewin, K. (1936) · McGraw-Hill eBooks · doi:10.1037/10019-000 · Cited by 3,927
Cited by
Items 101–200 of 400
Five Myths of School Turnaround Policy and Practice
Meyers, C. V., & Smylie, M. A. (2017) · Leadership and Policy in Schools · doi:10.1080/15700763.2016.1270333 · Cited by 59
Causal Inference in Generalizable Environments: Systematic Representative Design
Miller, L. C., Shaikh, S. J., Jeong, D. C., Wang, L., Gillig, T. K., Godoy, C. G., Appleby, P. R., Corsbie-Massay, C. L., Marsella, S., Christensen, J. L., et al. (2019) · Psychological Inquiry · doi:10.1080/1047840x.2019.1693866 · Cited by 52
Observation, experimentation, and replication in linguistics
Grieve, J. (2021) · Linguistics · doi:10.1515/ling-2021-0094 · Cited by 53
Meyer, D., Schoetter, R., Riechert, M., Verrelle, A., Tewari, M., Dudhia, J., Masson, V., van Reeuwijk, M., & Grimmond, S. (2020) · Journal of Advances in Modeling Earth Systems · doi:10.1029/2019ms001961 · Cited by 49
“Issues of lack of scienti fi c reproducibility have been noted by several authors in various disciplines (e.g., Atmanspacher & Maasen, 2016; Chen et al., 2019; Cohen ‐ Boulakia et al., 2017; Grning et al., 2018; Redish et al., 2018; Sochat et al., 2017; Van Bavel et al., 2016).”
BackgroundBreaking Groupthink: Why Scientific Identity and Norms Mitigate Ideological Epistemology
Bavel, J. J. V., Reinero, D. A., Harris, E., Robertson, C. E., & Pärnamets, P. (2020) · Psychological Inquiry · doi:10.1080/1047840x.2020.1722599 · Cited by 46
“While social psychology, by definition, focuses on the social context, the field has long acknowledged that behavior is a function of both the person and the environment—elegantly captured by Lewin’s equation: B¼ f(P,E) (see Van Bavel et al., 2016 for a recent discussion).”
BackgroundSelf-citationWillemse, C., & Wykowska, A. (2019) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2018.0036 · Cited by 48
Making Every Study Count: Learning From Replication Failure to Improve Intervention Research
Kim, J. S. (2019) · Educational Researcher · doi:10.3102/0013189x19891428 · Cited by 55
“For example, a recent meta-analysis of 100 replication studies in social psychology found that macro-level contextual factors like time, location, and culture affected replication success (Van Bavel et al. 2016).”
BackgroundMethodFrom Contextualizing to Context Theorizing: Assessing Context Effects in Privacy Research
Xu, H., & Zhang, N. (2022) · Management Science · doi:10.1287/mnsc.2021.4249 · Cited by 52
Wingen, T., Berkessel, J. B., & Dohle, S. (2022) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459211070559 · Cited by 44
Cauchoix, M., Chaine, A. S., & Barragan‐Jason, G. (2020) · Frontiers in Ecology and Evolution · doi:10.3389/fevo.2020.00106 · Cited by 45
Mixed-Methods Inquiry in Public Administration: The Interaction of Theory, Methodology, and Praxis
Raimondo, E., & Newcomer, K. E. (2017) · Review of Public Personnel Administration · doi:10.1177/0734371x17697247 · Cited by 38
Moral self-judgment is stronger for future than past actions
Sjåstad, H., & Baumeister, R. F. (2019) · Motivation and Emotion · doi:10.1007/s11031-019-09768-8 · Cited by 37
Context Matters: How an Ecological-Belonging Intervention Can Reduce Inequities in STEM
Hammarlund, S. P., Scott, C., Binning, K. R., & Cotner, S. H. (2022) · BioScience · doi:10.1093/biosci/biab146 · Cited by 37
Replication Studies in Mathematics Education: What Kind of Questions Would Be Productive to Explore?
Aguilar, M. S. (2020) · International Journal of Science and Mathematics Education · doi:10.1007/s10763-020-10069-7 · Cited by 48
Sabik, N. J., Matsick, J. L., McCormick‐Huhn, K., & Cole, E. R. (2021) · Psychology of Women Quarterly · doi:10.1177/03616843211035678 · Cited by 36
The Impact of Complexity on Methods and Findings in Psychological Science
Sanbonmatsu, D. M., Cooley, E. H., & Butner, J. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.580111 · Cited by 50
Cui, Y., Yin, F., Deng, Y., Volinn, E., Chen, F., Ji, K., Zeng, J., Zhao, X., & Li, X. (2016) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph13121225 · Cited by 32
Hooker, E. D., Campos, B., Zoccola, P. M., & Dickerson, S. S. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550617732387 · Cited by 38
Hittner, E. F., Rim, K. L., & Haase, C. M. (2019) · Emotion · doi:10.1037/emo0000539 · Cited by 40
A theory and methodology to quantify knowledge
Fanelli, D. (2019) · Royal Society Open Science · doi:10.1098/rsos.181055 · Cited by 44
“[34–36]), methodological differences between original and 993 replication studies [37], variable expertise of the replicators [38], and variable contextual sensitivity 994 of the phenomena studied [39,40].”
BackgroundPrediction of robust scientific facts from literature
Belikov, A. V., Rzhetsky, A., & Evans, J. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00474-8 · Cited by 30
A research agenda for understanding how social inequality is linked to brain structure and function
Hatzenbuehler, M. L., McLaughlin, K. A., Weissman, D. G., & Cikara, M. (2024) · Nature Human Behaviour · doi:10.1038/s41562-023-01774-8 · Cited by 30
“Although the role of contextual sensitivity has been highlighted in discussions of scientific reproducibility 65 , few studies have provided empirical evidence for it, particularly in cognitive neuroscience.”
Replication and extension of framing effects to compliance with health behaviors during pandemics
Otterbring, T., Festila, A., & Folwarczny, M. (2020) · Safety Science · doi:10.1016/j.ssci.2020.105065 · Cited by 32
“Such mixed re-sults indicate that framing effects are contextually sensitive (Otterbring et al., 2020; Van Bavel et al., 2016).”
BackgroundEguren, J., Antúnez, L., Otterbring, T., Curutchet, M. R., & Ares, G. (2021) · Appetite · doi:10.1016/j.appet.2021.105469 · Cited by 34
Spatiotemporal meta-analysis: reviewing health psychology phenomena over space and time
Johnson, B. T., Cromley, E. K., & Marrouch, N. (2017) · Health Psychology Review · doi:10.1080/17437199.2017.1343679 · Cited by 30
Whiteside, D. B., & Barclay, L. J. (2016) · Journal of Management · doi:10.1177/0149206316672531 · Cited by 38
“We encourage researchers to replicate these effects in other studies, particularly given the possibility of uncovering additional contextual influences that can deepen our understanding of depletion effects (cf. Van Bavel et al., 2016).”
BackgroundCentralized scientific communities are less likely to generate replicable results
Danchev, V., Rzhetsky, A., & Evans, J. A. (2019) · eLife · doi:10.7554/elife.43094 · Cited by 27
The Search for Clinically Useful Biomarkers of Complex Disease: A Data Analysis Perspective
Considine, E. C. (2019) · Metabolites · doi:10.3390/metabo9070126 · Cited by 30
“This view proposes that a large part of the reproducibility crisis in clinical and preclinical research can be ascribed to a failure to account for contextual sensitivity [58], phenotypic plasticity [59], and reaction norms [60].”
BackgroundMatsick, J. L., Kruk, M., Oswald, F., & Palmer, L. (2021) · Psychology of Women Quarterly · doi:10.1177/03616843211026564 · Cited by 33
“It thus stands to reason that researchers cannot overlook context but rather should take context seriously, leading Van Bavel et al. (2016) to conclude that “the lesson here is not that context is too hard to study but rather that context is too important to ignore” (p. 6458).”
BackgroundResultInfluentialImplications of research staff demographics for psychological science.
Does, S., Ellemers, N., Dovidio, J. F., Norman, J. B., Mentovich, A., van der Lee, R., & Goff, P. A. (2018) · American Psychologist · doi:10.1037/amp0000199 · Cited by 33
Harris, E. A., & Van Bavel, J. J. (2020) · doi:10.31234/osf.io/hfuas · Cited by 28
Harris, E. A., & Van Bavel, J. J. (2021) · Psychological Science · doi:10.1177/0956797620968792 · Cited by 27
“It is hard to know whether this new finding stemmed from our larger, more representative sample or differences in the sociopolitical context that have changed over the past decade (see Van Bavel et al., 2016).”
Schmiemann, P., Nehm, R. H., & Tornabene, R. E. (2017) · Science & Education · doi:10.1007/s11191-017-9925-z · Cited by 31
“Especially the latter is considered of importance to prove findings for stability (cf. Van Bavel et al. 2016).”
BackgroundAre we truly special and unique? A replication of Goldenberg et al. (2001)
Rodríguez-Ferreiro, J., Barberia, I., González-Guerra, J., & Vadillo, M. A. (2019) · Royal Society Open Science · doi:10.1098/rsos.191114 · Cited by 19
Best practices in replication: a case study of common information in coordination games
Chen, R., Chen, Y., & Riyanto, Y. E. (2021) · Experimental Economics · doi:10.1007/s10683-020-09658-8 · Cited by 28
“Bavel et al. (2016) find that, among the studies that Open Science Collaboration (2015) examined, replication success was negatively correlated with the contextual sensitivity of the research topic.”
BackgroundMuradchanian, J., Hoekstra, R., Kiers, H. A. L., & van Ravenzwaaij, D. (2021) · Royal Society Open Science · doi:10.1098/rsos.201697 · Cited by 38
Leman, J. K., Lyskov, S., Lewis, S. M., Adolf‐Bryfogle, J., Alford, R. F., Barlow, K. A., Ben-Aharon, Z., Farrell, D. P., Fell, J. S., Hansen, W. A., et al. (2021) · Nature Communications · doi:10.1038/s41467-021-27222-7 · Cited by 28
“[3], Lindorff-Larsen, Kresten [27], Meiler, Jens [17, 19], Schief, William [5, 6], Schueler-”
Assessing and Improving Robustness of Psychological Research Findings in Four Steps
Nuijten, M. B. (2022) · Springer eBooks · doi:10.1007/978-3-031-04968-2_17 · Cited by 30
Bavel, J. J. V., Mende‐Siedlecki, P., Brady, W. J., & Reinero, D. A. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1609700113 · Cited by 26
Self-citationMartín, M. S., Plana, M. P., Gea, A. I. P., & Navarro‐Mateu, F. (2023) · ESPIRAL CUADERNOS DEL PROFESORADO · doi:10.25115/ecp.v16i32.9102 · Cited by 71
What Predicts Stroop Performance? A Conditional Random Forest ApproachPreprint
IJzerman, H. R., Pollet, T. V., & Ebersole, C. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2805205 · Cited by 29
Does expertise matter in replication? An examination of the reproducibility project: Psychology
Bench, S. W., Rivera, G. N., Schlegel, R. J., Hicks, J. A., & Lench, H. C. (2017) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2016.07.003 · Cited by 24
“Van Bavel et al. (2016a) recently found that context sensitivitywas a reliable predictor of replication success (see also Inbar, in press and Van Bavel et al., 2016b for further discussion of this issue) and inferred that the experienced researchers who coded context sensitivity were able to…”
BackgroundLv, L., Liang, Y., Chen, S., Liu, G., & Liao, J. (2024) · Annals of Tourism Research · doi:10.1016/j.annals.2024.103889 · Cited by 25
Mieronkoski, R., Syrjälä, E., Jiang, M., Rahmani, A. M., Pahikkala, T., Liljeberg, P., & Salanterä, S. (2020) · PLoS ONE · doi:10.1371/journal.pone.0235545 · Cited by 36
Kershaw, C., Rast, D. E., Hogg, M. A., & van Knippenberg, D. (2020) · Journal of Applied Social Psychology · doi:10.1111/jasp.12715 · Cited by 24
Does Language Matter? Exploring Chinese–Korean Differences in Holistic Perception
Rhode, A. K., Voyer, B. G., & Gleibs, I. H. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01508 · Cited by 31
“Recent research suggests that contextual factors (e.g., time, location, culture) are associated with reproducibility, even after adjusting for methodological variables of the original research that are linked to replication success (Van Bavel et al., 2016).”
BackgroundSoininen, E. M., Henden, J., Ravolainen, V. T., Yoccoz, N. G., Bråthen, K. A., Killengreen, S. T., & Ims, R. A. (2018) · Ecology and Evolution · doi:10.1002/ece3.4399 · Cited by 24
Maximal positive controls: A method for estimating the largest plausible effect size
Hilgard, J. (2020) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2020.104082 · Cited by 27
Bar, A., & Otterbring, T. (2021) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2021.102506 · Cited by 29
Lewis, M. L., Mathur, M. B., VanderWeele, T. J., & Frank, M. C. (2022) · Royal Society Open Science · doi:10.1098/rsos.211499 · Cited by 30
Human latent inhibition: Problems with the stimulus exposure effect
Byrom, N., Msetfi, R. M., & Murphy, R. A. (2018) · Psychonomic Bulletin & Review · doi:10.3758/s13423-018-1455-4 · Cited by 34
Otterbring, T., Rolschau, K., Furrebøe, E. F., & Nyhus, E. K. (2021) · Food Quality and Preference · doi:10.1016/j.foodqual.2021.104376 · Cited by 27
Otterbring, T., Samuelsson, P., Arsenovic, J. F., Elbæk, C. T., & Folwarczny, M. (2022) · European Journal of Marketing · doi:10.1108/ejm-04-2022-0250 · Cited by 25
Schnell, S. D. (2018) · Bulletin of Mathematical Biology · doi:10.1007/s11538-018-0500-9 · Cited by 28
“…et al. 2002, Ioannidis 2005, Prinz et al. 2011, Begley and Ellis 2012), psychology (Bakker and Wicherts 2011, John et al 2012, Pashler and Wagenmakers 2012, Van Bavel et al. 2016), sociology and economy (Bollen et al 2015, Ioannidis et al. 2017) and computer science (Donoho et al. 2009).”
BackgroundRubin, M. (2019) · Synthese · doi:10.1007/s11229-019-02433-0 · Cited by 27
Genschow, O., Westfal, M., Crusius, J., Bartosch, L., Feikes, K. I., Pallasch, N., & Wozniak, M. (2020) · Journal of Personality and Social Psychology · doi:10.1037/pspa0000261 · Cited by 24
Hoyt, C. L., Moss, A. J., Burnette, J. L., Schieffelin, A., & Goethals, A. (2017) · European Journal of Social Psychology · doi:10.1002/ejsp.2289 · Cited by 46
Précis ofTalking to Our Selves: Reflection, Ignorance, and Agency
Doris, J. M. (2016) · Behavioral and Brain Sciences · doi:10.1017/s0140525x16002016 · Cited by 24
Manipulating belief in free will and its downstream consequences: A meta-analysisPreprint
Genschow, O., Cracco, E., Schneider, J., Protzko, J., Wisniewski, D., Braß, M., & Schooler, J. W. (2021) · doi:10.31234/osf.io/quwgr · Cited by 21
Doing better leadership science via replications and registered reports
Carsten, M. K., Clapp‐Smith, R., Haslam, S. A., Bastardoz, N., Gooty, J., Connelly, S., & Spain, S. M. (2023) · The Leadership Quarterly · doi:10.1016/j.leaqua.2023.101712 · Cited by 24
Crawford, J. T., & Ruscio, J. (2021) · Psychological Science · doi:10.1177/0956797620972367 · Cited by 20
“That said, given that contextual sensitivity is associated with lower rep-licability rates (Van Bavel et al., 2016), we cannot rule out the possibility that changing social contexts may partly explain these replication failures.”
BackgroundSchwarz, N., & Clore, G. L. (2016) · Psychological Science · doi:10.1177/0956797616653102 · Cited by 24
The effect of testosterone on economic risk-taking: A multi-study, multi-method investigation
Stanton, S. J., Welker, K. M., Bonin, P. L., Goldfarb, B., & Carré, J. M. (2021) · Hormones and Behavior · doi:10.1016/j.yhbeh.2021.105014 · Cited by 20
Is the Political Slant of Psychology Research Related to Scientific Replicability?
Reinero, D. A., Wills, J. A., Brady, W. J., Mende‐Siedlecki, P., Crawford, J. T., & Bavel, J. J. V. (2020) · Perspectives on Psychological Science · doi:10.1177/1745691620924463 · Cited by 21
Self-citationXie, W., Cappiello, M., Meng, M., Rosenthal, R., & Zhang, W. (2018) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2018.05.010 · Cited by 24
Brattli, M. B., Egeland, T. B., Nordeide, J. T., & Folstad, I. (2018) · Ecology and Evolution · doi:10.1002/ece3.4277 · Cited by 20
Goldberg, M. H., van der Linden, S., Ballew, M. T., Rosenthal, S. A., & Leiserowitz, A. (2019) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/2h7as · Cited by 25
Emotional Appeals, Climate Change, and Young Adults: A Direct Replication of Skurka et al. (2018)
Skurka, C., Romero‐Canyas, R., Joo, H. H., Acup, D., & Niederdeppe, J. (2021) · Human Communication Research · doi:10.1093/hcr/hqab013 · Cited by 23
Nelson, B. W., Byrne, M. L., Sheeber, L., & Allen, N. B. (2017) · Clinical Psychological Science · doi:10.1177/2167702616680061 · Cited by 21
“As noted earlier, attention to contextual influences has potentially significant implications for the reproducibility of findings (Van Bavel et al., 2016).”
BackgroundEstablishing Generalizable Mechanisms
Barsalou, L. W. (2019) · Psychological Inquiry · doi:10.1080/1047840x.2019.1693857 · Cited by 25
“As Van Bavel et al. (2016) note, replication failures that result from context sensitivity offer opportunities for better understanding an effect.”
BackgroundReplication and the Establishment of Scientific Truth
Iso-Ahola, S. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.02183 · Cited by 13
Reproducibility in Psychology: Theoretical Distinction of Different Types of Replications
Baucal, A., Gillespie, A., Krstić, K., & Zittoun, T. (2019) · Integrative Psychological and Behavioral Science · doi:10.1007/s12124-019-09499-y · Cited by 22
“Etz, A., & Vandekerckhove, J. (2016).”
BackgroundInfluentialPredicting the Replicability of Social Science Lab ExperimentsPreprint
Altmejd, A., Dreber, A., Forsell, E., Ho, T., Huber, J., Imai, T., Johannesson, M., Kirchler, M., Nave, G., & Camerer, C. F. (2019) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/zamry · Cited by 23
Anier, N., Roebroeck, É., Kleinlogel, E. P., Badea, C., Nugier, A., Berthon, M., & Guimond, S. (2019) · International Journal of Intercultural Relations · doi:10.1016/j.ijintrel.2019.07.006 · Cited by 23
A Unified Framework for Understanding and Intervening on False News SharingPreprint
Gollwitzer, A., Tump, A. N., Martel, C., Deffner, D., Sultan, M., Kurvers, R. H. J. M., & Hertwig, R. (2024) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/pxn29 · Cited by 19
A reporting guideline for IS survey research
Hui, W., Lui, S. M., & Lau, W. K. (2019) · Decision Support Systems · doi:10.1016/j.dss.2019.113136 · Cited by 20
Peña, J. F., & Pérez, J. F. H. (2019) · New Media & Society · doi:10.1177/1461444819874472 · Cited by 23
“For example, contextual differences including individualistic-collectivistic culture, urban–rural setting, and racially diverse–predominantly White populations accounted for replication success factors, such as effect size and power of the replication attempt (Van Bavel et al., 2016).”
BackgroundIdiographic Psychology: From Forms of Inference to Levels of Analysis, and Back AgainPreprint
Syed, M. (2022) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/hsp5v · Cited by 20
Psychologists update their beliefs about effect sizes after replication studies
McDiarmid, A. D., Tullett, A. M., Whitt, C. M., Vazire, S., Smaldino, P. E., & Stephens, J. E. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01220-7 · Cited by 16
Testing the reproducibility and robustness of the cancer biology literature by robot
Roper, K., Abdel‐Rehim, A., Hubbard, S. A., Carpenter, M. R., Rzhetsky, A., Soldatova, L., & King, R. D. (2022) · Journal of The Royal Society Interface · doi:10.1098/rsif.2021.0821 · Cited by 23
Generalization and the qualitative–quantitative debate
Norman, G. R. (2017) · Advances in Health Sciences Education · doi:10.1007/s10459-017-9799-5 · Cited by 20
“But a further analysis by van Bavel et al. (2016) took a completely different approach that led to real insights.”
BackgroundIlluminating illumination: Understanding the influence of ambient lighting on prosocial behaviors
Esteky, S., Wooten, D. B., & Bos, M. W. (2020) · Journal of Environmental Psychology · doi:10.1016/j.jenvp.2020.101405 · Cited by 22
Adolph, K. E. (2020) · Infancy · doi:10.1111/infa.12336 · Cited by 14
Optimizing Research Output: How Can Psychological Research Methods Be Improved?
Miller, J. O., & Ulrich, R. (2021) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-094927 · Cited by 21
Psychologically authentic versus inauthentic replication attempts
Bryan, C. J., Walton, G. M., & Dweck, C. S. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1609625113 · Cited by 16
Context Dependency as a Predictor of Replicability
Gollwitzer, M., & Schwabe, J. (2020) · Review of General Psychology · doi:10.1177/10892680211015635 · Cited by 13
“Perspectives on Psychological Science, 9, 59–71. https://doi.org/10.1177/1745691613514450 Van Bavel, J. J., Mende-Siedlecki, P., Brady, W. J., & Reinero, D. A. (2016).”
BackgroundInfluentialBracht, E. M., Bark, A. S. H., She, Z., van Dick, R., & Junker, N. M. (2023) · Leadership & Organization Development Journal · doi:10.1108/lodj-03-2023-0129 · Cited by 21
Confounds in “Failed” Replications
Bressan, P. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.01884 · Cited by 18
Exploring the feasibility of subliminal priming on smartphones
Pinder, C. N., Vermeulen, J., Cowan, B. R., Beale, R., & Hendley, R. (2017) · International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI) · doi:10.1145/3098279.3098531 · Cited by 19
“Replicating subliminal experiments on smartphones may yield different results to those run in psychology labs since research shows replication is sensitive to contextual factors [85].”
BackgroundTheory and the Replication Problem
Willer, D., & Emanuelson, P. E. (2020) · Sociological Methodology · doi:10.1177/0081175020955216 · Cited by 17
So you want to do ESM? Ten Essential Topics for Implementing the Experience Sampling Method (ESM)Preprint
Fritz, J., Piccirillo, M. L., Cohen, Z. D., Frumkin, M. R., Kirtley, O. J., Moeller, J., Neubauer, A. B., Norris, L. A., Schuurman, N. K., Snippe, E., et al. (2023) · doi:10.31219/osf.io/fverx · Cited by 15
Ye, Y., Tong, Y., Chiu, C., & Gawronski, B. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2016.12.002 · Cited by 14
Sharing the Recipe: Reproducibility and Replicability in Research Across Disciplines
Rahal, R.-M., Hamann, H., Brohmer, H., & Pethig, F. (2022) · Research Ideas and Outcomes · doi:10.3897/rio.8.e89980 · Cited by 19
Harnessing heterogeneity in behavioural research using computational social science
Veltri, G. (2023) · Behavioural Public Policy · doi:10.1017/bpp.2023.35 · Cited by 13
“There is a long-standing debate about effect size in psychology that has shaped the research practices of the past years (Kelley and Preacher, 2012, Westfall et al ., 2014, Van Bavel et al. , 2016, Schäfer and Schwarz, 2019).”
Informal Laboratory Practices in Psychology
Brenninkmeijer, J., Derksen, M., & Rietzschel, E. F. (2019) · Collabra Psychology · doi:10.1525/collabra.221 · Cited by 18
Improving the reproducibility of findings by updating research methodology
Klein, J. (2021) · Quality & Quantity · doi:10.1007/s11135-021-01196-6 · Cited by 22
Boyce, V., Mathur, M. B., & Frank, M. C. (2023) · Royal Society Open Science · doi:10.1098/rsos.231240 · Cited by 22
How to True Psychology’s Objects
Morawski, J. G. (2021) · Review of General Psychology · doi:10.1177/10892680211046518 · Cited by 17
“For instance, advocating the reform’s replication agenda yet recognizing objects’ contextsensitivity, Van Bavel et al. (2016) admitted the extensiveness and sometimes amorphous nature of contextual influences.”
BackgroundAITurk: Using ChatGPT for Social Science ResearchPreprint
Qin, X., Huang, M., & Ding, J. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4922861 · Cited by 25
Objectivity for the research worker
van Dongen, N. N. N., & Sikorski, M. (2021) · European Journal for Philosophy of Science · doi:10.1007/s13194-021-00400-6 · Cited by 25
“The possibility of choosing between two or more options in these instances are called researchers’ degrees of freedom (Simmons et al., 2011; Wicherts et al., 2016).”
BackgroundMethodInfluential
Metadata
- Authors
- Van Bavel, J. J., Mende-Siedlecki, P., Brady, W. J., & Reinero, D. A.
- Affiliations
- Department of Psychology, New York University, New York, NY 10003
- New York University (US)
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2016-05-23
- Volume
- 113
- Issue
- 23
- Pages
- 6454-6459
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC4988618
- OpenAlex
- W2397691833
- Cited by
- 398
- References
- 92
- Citations by year
- 2026: 17 · 2025: 23 · 2024: 22 · 2023: 29 · 2022: 47 · 2021: 51 · 2020: 49 · 2019: 48 · 2018: 43 · 2017: 43 · 2016: 26
- FWCI
- 75.91
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- OSF Preprints (OSF Preprints) (submittedVersion)
- Funders
- National Science Foundation (1555131)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › General Psychology › Academic and Historical Perspectives in Psychology
- Keywords
- scientific reproducibility, replication success, context sensitivity, replication study, statistical power, degrees of freedom, effect size
- MeSH terms
- Humans, Psychology, Publishing, Research, Science, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.