Publication bias in the social sciences: Unlocking the file drawer
The DOI matches its Crossref record; 37 of the 38 references it cites were found in a registry or an index.
Abstract
The file drawer is full. Should we worry?: Experiments that produce null results face a higher barrier to publication than those that yield statistically significant differences. Whether this is a problem depends on how many null but otherwise valid results might be trapped in the file drawer. Franco et al. use a Time-sharing Experiments in the Social Sciences archive of nearly 250 peer-reviewed proposals of social science experiments conducted on nationally representative samples. They find that only 10 out of 48 null results were published, whereas 56 out of 91 studies with strongly significant results made it into a journal. Science , this issue p. 1502
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Selective reporting, questionable research practices, and low-powered research all contribute to the unreliability of findings in a range of disciplines”
From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as Franco et al., 2014
In the citing paper
“In particular, negative or non-significant results are far less likely to be published (Franco et al., 2014).”
From What do we know about interventions to improve educator wellbeing? A systematic literature review (2024) · cites this paper as Franco et al., 2014
In the citing paper
“The bias against negative results appears to be a consequence of both editors and reviewers finding them less compelling for publication and from authors themselves not trying to publish them 8-12”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [8]
In the citing paper
“Positive results -- finding that a treatment is effective or that there is a relationship between two variables -- are more likely to be published than negative results 6-12 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [8]
In the citing paper
“A possible interpretation of these results is that the missing effect sizes are mainly a file‐drawer problem (Rosenthal, 1979). This interpretation is consistent with the evidence presented in Franco et al. (2014), which indicates that the file‐drawer problem is the main culprit behind publication bias in the social sciences.”
From Targeted school‐based interventions for improving reading and mathematics for students with or at risk of academic difficulties in Grades K‐6: A systematic review (2021) · cites this paper as Franco et al. (2014)
3 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722 · cited 1×
“In the presence of publication bias smaller studies report larger effects in order to exceed arbitrary significance thresholds (19, 20).”
BackgroundThe file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
Meta-analysis in social research
(1982) · Evaluation and Program Planning · doi:10.1016/0149-7189(82)90011-8 · Cited by 3,853
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109
Publication Bias in Meta‐Analysis: Prevention, Assessment and Adjustments
(2005) · doi:10.1002/0470870168 · Cited by 2,368
The case of the misleading funnel plot
Lau, J., Ioannidis, J. P. A., Terrin, N., Schmid, C. H., & Olkin, I. (2006) · BMJ · doi:10.1136/bmj.333.7568.597 · Cited by 1,766 · cited 1×
“However, these visualization-based approaches are sensitive to using different measures of precision (21, 22) and also assume outcome variables and effect sizes are comparable across studies (23).”
Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias
Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A.-W., Cronin, E., Decullier, E., Easterbrook, P. J., Von Elm, E., Gamble, C., et al. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003081 · Cited by 1,546 · cited 1×
“A different approach involves examining the publication outcomes of a cohort of studies, either prospectively or retrospectively (24, 25).”
MethodSumming up: The Science of Reviewing Research
Wilkinson, I., Light, R. J., & Pillemer, D. B. (1987) · Journal of Educational Statistics · doi:10.2307/1164691 · Cited by 1,471
Summing Up: The Science of Reviewing Research
Light, R. J., & Pillemer, D. B. (1984) · Harvard University Press eBooks · doi:10.4159/9780674040243 · Cited by 1,427
The Existence of Publication Bias and Risk Factors for Its Occurrence
Dickersin, K. (1990) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1990.03440100097014 · Cited by 1,417 · cited 1×
“Publication bias occurs when “publication of study results is based on the direction or significance of the findings” (1).”
Background
Research works printed but not linked: 3
- 10.1037/0735-7028.17.2.136 · doi:10.1037/0735-7028.17.2.136 · Paper page
- K. Dickersin in Publication Bias in Meta-Analysis: Prevention Assessment and Adjustments H. R. Rothstein A. J. Sutton M. Borenstein Eds. (Wiley Chichester UK 2006) ch. 2.
- Dickersin K., Min Y. I., Online J. Curr. Clin. Trials 1993, 50 (1993).
Cited by
Items 501–600 of 1,002
Open Access in Special Education: A Review of Journal and Publisher Policies
Fleming, J. I., & Cook, B. G. (2021) · Remedial and Special Education · doi:10.1177/0741932521996461 · Cited by 28
“Publication bias negatively affects the special education research base, as the lack of null and negative results in the published literature can result in inflated estimates of intervention effects (Cook & Therrien, 2017; Franco et al., 2014; Gage et al., 2017; Sham & Smith, 2014).”
BackgroundRavenscroft, S., & Williams, P. F. (2021) · Accounting Organizations and Society · doi:10.1016/j.aos.2021.101280 · Cited by 25
A Practical Guide to Registered Reports for Economists
Arpinon, T., & Espinosa, R. (2022) · Journal of the Economic Science Association · doi:10.1007/s40881-022-00123-1 · Cited by 39
Football and Public Opinion: A Partial Replication and Extension
Busby, E. C., & Druckman, J. N. (2017) · Journal of Experimental Political Science · doi:10.1017/xps.2017.22 · Cited by 24
“In some sense, our sample size was two – two schools around one event…” 1Another possibility is publication bias such that only significant results enter the published literature (Brown et al. 2017; Franco et al. 2014).”
BackgroundDo Natural Disasters Affect Voting Behavior? Evidence from Croatian Floods
Bovan, K., Banai, B., & Banai, I. P. (2018) · PLoS Currents · doi:10.1371/currents.dis.cbf57c8ac3b239ba51ccc801d3362c07 · Cited by 22
Default-Switching: The Hidden Cost of DefaultsPreprint
Jachimowicz, J. M., Duncan, S., & Weber, E. U. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2727301 · Cited by 23
Incorrect Inferences When Using Generated Regressors in Accounting ResearchPreprint
Chen, W., Hribar, P., & Melessa, S. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3724730 · Cited by 23
A Framework for Evaluating Reproducibility and Replicability in EconomicsPreprint
Dreber, A., & Johannesson, M. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4458153 · Cited by 34
“…Brodeur et al. 2016, 2020; Nelson et al. 2018), low statistical power (Button et al. 2013; Ioannidis et al. 2017), testing hypotheses with low priors (Maniadis et al. 2014; Dreber et al. 2015; Johnson et al 2017), and publication bias (Hedges 1992; Stern & Simes 1997; Franco et al. 2014, 2015).”
Toward open science in marketing research
Deer, L., Adler, S. J., Datta, H., Mizik, N., & Sarstedt, M. (2024) · International Journal of Research in Marketing · doi:10.1016/j.ijresmar.2024.12.005 · Cited by 28
How Not to Fool Ourselves About Heterogeneity of Treatment Effects
von Hippel, P. T., & Schuetze, B. A. (2025) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459241304347 · Cited by 36
“Investigators seeking publication and funding still have strong incentives to claim interactions where none exist (Franco et al., 2014), especially if there is little or no main effect.”
Hasan, N., Agarwal, C., Joshi, A., Rahal, D., Traisa, R., & Sharma, S. (2025) · International Journal of Ethics and Systems · doi:10.1108/ijoes-10-2024-0326 · Cited by 35
Conclusion: The Challenges and Opportunities for Social Media Research
Persily, N., & Tucker, J. A. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108890960.014 · Cited by 24
Janzen, S., & Michler, J. D. (2021) · Applied Economic Perspectives and Policy · doi:10.1002/aepp.13133 · Cited by 23
New Opportunities for Integrated Formal Methods
Gleirscher, M., Foster, S., & Woodcock, J. (2019) · White Rose Research Online (University of Leeds, The University of Sheffield, University of York) · Cited by 22
Split Feelings: Understanding Implicit and Explicit Political Persuasion
Ryan, T. J., & Krupnikov, Y. (2021) · American Political Science Review · doi:10.1017/s0003055421000538 · Cited by 21
“To close, we note that this series of investigations illustrates one underappreciated benefit of the disciplinary trend toward experimental pre-registration (Franco et al. 2014).”
BackgroundAcademic favoritism at work: insider bias in Turkish national journals
Tütüncü, L., Yücedoğru, R., & SARISOY, İ. (2022) · Scientometrics · doi:10.1007/s11192-022-04355-0 · Cited by 24
“Biases can arise in every aspect of the process, from the selfimposed publication bias prior to submission (Franco et al., 2014; Manchikanti et al., 2015; van Lent et al., 2014) to selection bias of editors (Matias-Guiu & Garcia-Ramos, 2011; Moustafa, 2015; Tivina et al.”
BackgroundSolana, P., & Santiago, J. (2022) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2022.104834 · Cited by 28
Zhu, P.-C., Shen, J., Qian, R.-Y., Xu, J., Liu, C., Hu, W., Zhang, Y., & Lv, L.-C. (2023) · Frontiers in Pharmacology · doi:10.3389/fphar.2023.1165212 · Cited by 27
“, 2012), and negative findings are rarely published (Franco et al., 2014).”
BackgroundLee, J.-Y., & Baek, M.-K. (2023) · Sustainability · doi:10.3390/su151411325 · Cited by 46
P ‐hacking in meta‐analyses: A formalization and new meta‐analytic methods
Mathur, M. B. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1701 · Cited by 32
The Impacts of Fiscal Openness: A Review of the EvidencePreprint
de Renzio, P., & Wehner, J. (2015) · SSRN Electronic Journal · doi:10.2139/ssrn.2602439 · Cited by 22
Englis, B. G., & Frederiks, A. J. (2023) · Entrepreneurship Education and Pedagogy · doi:10.1177/25151274231161102 · Cited by 35
Congressional seniority and pork: A pig fat myth?
Fowler, A., & Hall, A. B. (2015) · European Journal of Political Economy · doi:10.1016/j.ejpoleco.2015.07.006 · Cited by 21
“Second, publication bias skews the reporting of empirical results, leading to many false positive results which are rarely overturned (Franco et al., 2014; Gerber and Malhotra, 2008).”
BackgroundImaginary alternatives: The effects of mental simulation on powerless negotiators.
Schaerer, M., Schweinsberg, M., & Swaab, R. I. (2018) · Journal of Personality and Social Psychology · doi:10.1037/pspi0000129 · Cited by 23
Hartgerink, C. H. J., & van Zelst, M. (2018) · Publications · doi:10.3390/publications6020021 · Cited by 21
Lyu, X., Xu, Y., Zhao, X.-F., Zuo, X., & Chuan-Peng, H. (2020) · Journal of Pacific Rim Psychology · doi:10.1017/prp.2019.28 · Cited by 30
Linkages between Globalisation, Carbon Dioxide Emissions and Governance in Sub-Saharan Africa
Asongu, S. A., Nting, R. T., & Nnanna, J. (2019) · International Journal of Public Administration · doi:10.1080/01900692.2019.1663530 · Cited by 23
Elson, M., Huff, M., & Utz, S. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919895419 · Cited by 26
Are Voting Treatments Transformative? Expanding Upon and Meta-Analyzing the Evidence
Holbein, J. B., Rangel, M. A., Moore, R., & Croft, M. (2021) · Political Behavior · doi:10.1007/s11109-021-09746-2 · Cited by 24
Experiments from unfinished Registered Reports in the Reproducibility Project: Cancer Biology
Errington, T. M., Denis, A., Allison, A. B., Araiza, R., Aza-Blanc, P., Bower, L. R., Campos, J., Chu, H., Denson, S., Donham, C., et al. (2021) · eLife · doi:10.7554/elife.73430 · Cited by 23
“However, this requires that all outcomes are shared, which is less common for null or incomplete findings (Franco et al., 2014).”
BackgroundBieg, T., Gerdenitsch, C., Schwaninger, I., Kern, B., & Frauenberger, C. (2022) · Computers in Human Behavior · doi:10.1016/j.chb.2022.107249 · Cited by 25
Power to the People: A Beginner’s Tutorial to Power Analysis using jamovi
Bartlett, J. E., & Charles, S. J. (2022) · Meta-Psychology · doi:10.15626/mp.2021.3078 · Cited by 31
Wess, R., Priemer, B., & Parchmann, I. (2023) · Disciplinary and Interdisciplinary Science Education Research · doi:10.1186/s43031-023-00076-3 · Cited by 44
P-Hacking, Data Type and Data-Sharing Policy
Brodeur, A., Cook, N., & Neisser, C. (2022) · The Economic Journal · doi:10.1093/ej/uead104 · Cited by 41
Conservative Apostles of Objectivity and the Myth of a “Liberal Bias” in Science
Larrègue, J. (2017) · The American Sociologist · doi:10.1007/s12108-017-9366-9 · Cited by 21
Legal Empowerment and Group-Based Inequality
Gisselquist, R. M. (2018) · The Journal of Development Studies · doi:10.1080/00220388.2018.1451636 · Cited by 27
Salandra, R. (2018) · Research Policy · doi:10.1016/j.respol.2018.04.005 · Cited by 24
“Although some evidence comes from studies measuring publication bias, documented in various studies and disciplines within the biomedical and social sciences (Easterbrook et al., 1991; Franco et al., 2014), the drivers of poor reporting practices are not completely clear.”
BackgroundMeaux, L. T., Cox, J. M., & Kopkin, M. R. (2018) · Psychiatry Psychology and Law · doi:10.1080/13218719.2018.1474817 · Cited by 27
“Considering publication bias (Franco et al., 2014), the ‘file-drawer problem’ (Rosenthal, 1979) in scientific research and the tendency for journals to focus on publishing significant results, one may opine that these null findings are more reliable and generalizable than is currently reported in…”
ResultDewa, C. S., Loong, D., Trujillo, A., & Bonato, S. (2018) · PLoS ONE · doi:10.1371/journal.pone.0199368 · Cited by 27
Buckley, J., Adams, L., Aribilola, I., Arshad, I., Azeem, M., Bracken, L., Breheny, C., Buckley, C., Chimello, I., Fagan, A., et al. (2021) · International Journal of Technology and Design Education · doi:10.1007/s10798-021-09695-1 · Cited by 37
“In theory, this prevents QRP’s such as hypothesising after the results are known (HARKing: Kerr, 1998) and publication bias stemming from journals being less accepting of null results (Franco et al., 2014), and there is already evidence indicative of these benefits (Allen & Mehler, 2018).”
ResultScacchia, P., & Pascalis, V. D. (2020) · American Journal of Clinical Hypnosis · doi:10.1080/00029157.2019.1586639 · Cited by 21
Jager, N. W., Newig, J., Challies, E., Kochskämper, E., & von Wehrden, H. (2021) · Research Synthesis Methods · doi:10.1002/jrsm.1514 · Cited by 27
“While this method cannot account for those results that remain in researchers' file drawers without ever being written up and published, 19,54 two factors make this approach the most promising at hand: First, in social science, and in comparative governance research in particular, where a case…”
BackgroundMethodShort of Suspension: How Suspension Warnings Can Reduce Hate Speech on Twitter
Yildirim, M. M., Nagler, J., Bonneau, R., & Tucker, J. A. (2021) · Perspectives on Politics · doi:10.1017/s1537592721002589 · Cited by 28
Swanson, E. P., Young, G. M., & Yust, C. G. (2021) · Journal of Corporate Finance · doi:10.1016/j.jcorpfin.2021.102144 · Cited by 28
Subjective Freedom of Speech: Why Do Citizens Think They Cannot Speak Freely?
Menzner, J., & Traunmüller, R. (2022) · Politische Vierteljahresschrift · doi:10.1007/s11615-022-00414-6 · Cited by 32
“This way, we avoid the problem of confounding the processes of hypothesis generating and hypothesis testing (Nosek et al. 2018) and the problem of publication bias in favor of statistically significant results (Franco et al. 2014).”
BackgroundResultRepresentative Bureaucracy and Social Equity: Bias, Perceived Fairness and Efficacy
Meier, K. J. (2023) · Journal of Social Equity and Public Administration · doi:10.24926/jsepa.v1i1.4814 · Cited by 20
Muradchanian, J., Hoekstra, R., Kiers, H. A. L., & van Ravenzwaaij, D. (2023) · PLoS ONE · doi:10.1371/journal.pone.0292279 · Cited by 26
Globalization in lifelong gender inclusive education for structural transformation in Africa
Asongu, S. A., Bouanza, J. R. F. K., & Agyemang‐Mintah, P. (2024) · Economic Systems · doi:10.1016/j.ecosys.2024.101218 · Cited by 23
Steckler, T., Brose, K., Haas, M., Kas, M. J., Koustova, E., & Bespalov, A. (2015) · European Neuropsychopharmacology · doi:10.1016/j.euroneuro.2015.05.011 · Cited by 22
“…important to realize and acknowledge that low reproducibility is an inherent feature of science especially when highly unexpected fi ndings ( ‘ discoveries ’ ) are made, as the positive predictive value of those studies is often very low (Franco et al., 2014; Ioannidis, 2005; Tsilidis et al., 2013).”
Ulrich, R., Miller, J. O., & Erdfelder, E. (2018) · Zeitschrift für Psychologie · doi:10.1027/2151-2604/a000319 · Cited by 21
Nachtigall, V., Rummel, N., & Serova, K. (2018) · Unterrichtswissenschaft · doi:10.1007/s42010-018-0020-1 · Cited by 25
“Außerdem erscheint es relevant auch Nullergebnisse zu veröffentlichen, um dem Problem des Publikationsbias (z.B. Franco et al. 2014) entgegenzuwirken.”
BackgroundResults-blind review: a masked crusader for science
Woznyj, H. M., Grenier, K., Ross, R. L., Banks, G. C., & Rogelberg, S. G. (2018) · European Journal of Work and Organizational Psychology · doi:10.1080/1359432x.2018.1496081 · Cited by 27
“Third, under traditional peer review methods, reviewer and editor favouritism towards positive outcomes may further exacerbate systematic suppression of research results by influencing how authors report their findings (Bosco, Aguinis, Field, Pierce, & Dalton, 2016; Franco et al., 2014).”
BackgroundResultInfluentialAsongu, S. A., & Odhiambo, N. M. (2020) · Innovation The European Journal of Social Science Research · doi:10.1080/13511610.2020.1745058 · Cited by 24
The limitations of quantitative social science for informing public policy
Jerrim, J., & de Vries, R. (2015) · Evidence & Policy · doi:10.1332/174426415x14431000856662 · Cited by 39
Early-Life Exposures and Social Stratification
Torche, F., & Nobles, J. E. (2024) · Annual Review of Sociology · doi:10.1146/annurev-soc-091523-023313 · Cited by 22
Change in Police Organizations: Perceptions, Experiences, and the Failure to Launch
Schafer, J. A., & Varano, S. P. (2017) · Journal of Contemporary Criminal Justice · doi:10.1177/1043986217724532 · Cited by 19
Stimulus Sampling and Research Integrity
Cummings, J. J., & Reeves, B. B. (2022) · Oxford University Press eBooks · doi:10.1093/oso/9780190938550.003.0008 · Cited by 24
Roubinov, D. S., Don, B. P., Blades, R. B., & Epel, E. S. (2022) · Family Process · doi:10.1111/famp.12810 · Cited by 25
Olsson, T. M., & Sundell, K. (2023) · PLoS ONE · doi:10.1371/journal.pone.0281110 · Cited by 24
Bush, N. R. (2024) · Development and Psychopathology · doi:10.1017/s0954579424000488 · Cited by 24
“Although reviewers and editors sometimes find the lack of significant associations to be uninteresting, or they demand a greater burden of evidence for null than significant findings (Franco et al., 2014), certain strategies can enhance success (Schuengel, 2022).”
BackgroundToward a theory of collaboration for evidence-based management
HakemZadeh, F., & Baba, V. V. (2016) · Management Decision · doi:10.1108/md-06-2015-0243 · Cited by 39
Introduction to Special Issue: Null Effects and Publication Bias in Learning Disabilities Research
Therrien, W. J., & Cook, B. G. (2018) · Learning Disabilities Research and Practice · doi:10.1111/ldrp.12163 · Cited by 22
Is There a Cult of Statistical Significance in Agricultural Economics?
Rommel, J., & Weltin, M. (2020) · Applied Economic Perspectives and Policy · doi:10.1002/aepp.13050 · Cited by 20
“An overemphasis of statistical significance may obstruct scientific knowledge production by introducing publication biases (FRANCO ET AL., 2014), a lock-in and path dependence of journals publishing only statistically significant findings (ALTMAN, 2004), and the manipulation of specifications in…”
BackgroundInhibition Tasks are not Associated with a Variety of Behaviours in College Students
Gunten, C. D. V., Bartholow, B. D., & Martins, J. S. (2020) · European Journal of Personality · doi:10.1002/per.2250 · Cited by 23
Farrar, B. G., Altschul, D., Fischer, J., van der Mescht, J., Placì, S., Troisi, C. A., Vernouillet, A., Clayton, N. S., & Ostojić, L. (2020) · Animal Behavior and Cognition · doi:10.26451/abc.07.03.09.2020 · Cited by 17
“Estimating publication bias will therefore require a mix of research methods, such as survey measures, getting direct access to unpublished data or following-up on registered studies (Franco et al., 2014), in addition to the methods we have used here.”
MethodBuilding peaceful societies: an evidence gap map
(3ie), I. I. F. I. E., Sonnenfeld, A., Chirgwin, H., (3ie), I. I. F. I. E., Berretta, M., Evaluation, I. I. F. I., Longman, K., Consultant, I., Krämer, M., Snilstveit, B., et al. (2020) · doi:10.23846/egm015 · Cited by 21
Zhu, X., Tang, Y., Lu, J., Song, M., Yang, C., & Zhao, X. (2024) · Educational Psychology Review · doi:10.1007/s10648-024-09976-w · Cited by 26
“Publication bias (i.e., statistically significant findings are more likely to be published than non-significant findings) can cause inflated estimates of an effect (Franco et al., 2014).”
A Meta-analysis of the relationship between growth mindset and grit
Lam, K. K. L., & Zhou, M. (2025) · Acta Psychologica · doi:10.1016/j.actpsy.2025.104872 · Cited by 34
Coles, N. A., Larsen, J. T., Kuribayashi, J., & Kuelz, A. (2019) · Emotion Review · doi:10.1177/1754073919868762 · Cited by 24
“First, as is often the case in scientific literatures, results provided some evidence of publication bias (Franco et al., 2014; Rothstein et al., 2006).”
ResultNot Published Is Not Perished: Addressing Publication Bias in Meta-Analytic Studies in Communication
Sun, Y., & Pan, Z. (2020) · Human Communication Research · doi:10.1093/hcr/hqz015 · Cited by 18
Wang, H., Zhuang, Z., Huang, Y.-Y., Zhuang, Z., Jin, Y., Ye, H.-Y., Lin, X., Zheng, Q., & Wang, Y.-L. (2020) · Frontiers in Pharmacology · doi:10.3389/fphar.2020.00988 · Cited by 19
“…of astragaloside IV on blood urea nitrogen vs control. le 988 Wang et al. Astragaloside IV for Diabetic Nephropathy of AS-IV being overestimated (Franco et al., 2014); (4) though animal welfare regulations were observed in 14 studies, no study reported disinfection when the experimenter…”
BackgroundCritical review of uniform color effects in sports
Goldschmied, N. P., Furley, P., & Bush, R. A. (2020) · International Review of Sport and Exercise Psychology · doi:10.1080/1750984x.2020.1855668 · Cited by 25
DiGiovanni, A., & Font, S. A. (2021) · Children and Youth Services Review · doi:10.1016/j.childyouth.2021.105943 · Cited by 19
Nudging debtors to pay their debt: Two randomized controlled trials
Holzmeister, F., Huber, J., Kirchler, M., & Schwaiger, R. (2022) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2022.04.006 · Cited by 25
Scholz, D., Taylor, A. M., & Strelan, P. (2022) · Adolescent Research Review · doi:10.1007/s40894-022-00193-6 · Cited by 31
“Publication bias has been reported to be as much as 40% more likely for studies that produce significant positive results compared to studies confirming the null hypothesis (Franco et al., 2014).”
ResultExamining the predictive validity of the Duolingo English Test: Evidence from a major UK university
Isaacs, T., Hu, R., Trenkic, D., & Varga, J. (2023) · Language Testing · doi:10.1177/02655322231158550 · Cited by 18
“Some studies with null results might also not have been disseminated due to publication bias (Franco et al., 2014).”
ResultA Primer on the ‘Reproducibility Crisis’ and Ways to Fix It
Reed, W. R. (2018) · Australian Economic Review · doi:10.1111/1467-8462.12262 · Cited by 18
Revisiting a Criminological Classic: The Cycle of Violence
Myers, W., Lloyd, K. M., Turanovic, J. J., & Pratt, T. C. (2018) · Journal of Contemporary Criminal Justice · doi:10.1177/1043986218770003 · Cited by 18
Sterck, O. C. (2019) · Journal of Economic Surveys · doi:10.1111/joes.12330 · Cited by 17
“It complements recent works on how to raise standards and increase transparency in applied economics (see e.g., Brodeur et al. (2016); Franco et al. (2014) on p-value hacking and publication bias; Maniadis et al. (2017) or Camerer et al. (2016) on replicability; Pritchett and Sandefur (2015) or…”
The effectiveness of DNA databases in relation to their purpose and content: A systematic review
Struyf, P., Moor, S. D., Vandeviver, C., Renard, B., & Beken, T. V. (2019) · Forensic Science International · doi:10.1016/j.forsciint.2019.05.052 · Cited by 19
Identification of and Correction for Publication Bias
Andrews, I., & Kasy, M. (2017) · National Bureau of Economic Research · doi:10.3386/w23298 · Cited by 33
“Second, we might additionally observe the results X⇤ from unpublished working papers as in Franco et al. (2014).”
Powerless Conservatives or Powerless Findings?
Utych, S. (2020) · PS Political Science & Politics · doi:10.1017/s1049096520000505 · Cited by 17
“Indeed, nullandmixedresultsarebothlesslikely to be published and also less likely to be written up than strong results (Franco, Malhotra, and Simonovits 2014).”
Vlandas, T., & Halikiopoulou, D. (2024) · Perspectives on Politics · doi:10.1017/s1537592723002980 · Cited by 18
“Third, we follow a growing literature that reports null e ff ects (Schaub, Gereke, and Baldassarri 2021; Blair, Christensen, and Rudkin 2021; Masterson and Yasenov 2021) thus contributing to endeavours to balance scien-ti fi c fi ndings and overcome con fi rmation bias in the social sciences (Gerber and Malhotra 2008; Fra…”
Lahiri, S., Bingenheimer, J. B., Evans, W. D., Wang, Y., Cislaghi, B., Dubey, P., & Snowden, B. (2024) · Addiction · doi:10.1111/add.16685 · Cited by 20
Bhatti, Y., Dahlgaard, J. O., Hansen, J. H., & Hansen, K. M. (2017) · West European Politics · doi:10.1080/01402382.2017.1339985 · Cited by 30
“As a field, we need to know both the successes and failures of high quality mobilization campaigns to establish their impact and avoid bias from the publication of only positive results (Franco et al. 2014).”
BackgroundPublication Biases in Replication Studies
Berinsky, A. J., Druckman, J. N., & Yamamoto, T. (2020) · Political Analysis · doi:10.1017/pan.2020.34 · Cited by 21
“…bias stems from authors discontinuing the paper writing process upon discovering null results; when it comes to the set of papers that have been written, they find a roughly 5 percentage point bias toward significant results in the publication process (see Franco et al. 2014, appendix, Table S2).”
BackgroundMethodResultInfluentialTabash, M. I., Abdulkarim, F. M., Akinlaso, I. M., & Dhankar, R. S. (2022) · African Journal of Economic and Management Studies · doi:10.1108/ajems-03-2021-0138 · Cited by 23
Tiller, N. B., & Ekkekakis, P. (2023) · Kinesiology Review · doi:10.1123/kr.2022-0039 · Cited by 19
Measuring the effects of publication bias in political science
Esarey, J., & Wu, A. (2016) · Research & Politics · doi:10.1177/2053168016665856 · Cited by 21
Social Security and Saving: An Update
SLAVOV, S. N., Gorry, D., Gorry, A., & Caliendo, F. (2018) · Public Finance Review · doi:10.1177/1091142118770199 · Cited by 17
The Spread of Policy Diffusion Studies: A Systematic Review and Meta-Analysis, 1990-2018Preprint
Mallinson, D. J. (2020) · doi:10.33774/apsa-2020-csnt6 · Cited by 17
(3ie), I. I. F. I. E., Moore, N., Lane, C. E., (3ie), I. I. F. I. E., Storhaug, I., (3ie), I. I. F. I. E., Franich, A., (3ie), I. I. F. I. E., Rolker, H. B., (IMMANA), I. M. A. M. F. A. A. N. A., et al. (2021) · doi:10.23846/egm016 · Cited by 23
The replication crisis has led to positive structural, procedural, and community changesPreprint
Korbmacher, M., Azevedo, F., Pennington, C. R., Hartmann, H., Pownall, M., Schmidt, K., Elsherif, M. M., Breznau, N., Robertson, O. M., Kalandadze, T., et al. (2023) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/r6cvx · Cited by 23
Delios, M. S., Kehoe, C. E., & Pizarro‐Campagna, E. (2023) · Mental Health & Prevention · doi:10.1016/j.mhp.2023.200299 · Cited by 32
The association between social support and prosocial behavior: A three‐level meta‐analysis
Wang, Y., Ran, G., Zhang, Q., & Zhang, Q. (2024) · PsyCh Journal · doi:10.1002/pchj.792 · Cited by 41
The contribution of scenic views of, and proximity to, lakes and reservoirs to property values
Nicholls, S., & Crompton, J. L. (2018) · Lakes & Reservoirs Science Policy and Management for Sustainable Use · doi:10.1111/lre.12207 · Cited by 17
The "P"-Value: The Primary Alphabet of Research Revisited
Das, D., & Das, T. (2023) · International Journal of Preventive Medicine · doi:10.4103/ijpvm.ijpvm_200_22 · Cited by 23
“Selective reporting is where non‐significant results are not reported as top journals consider them to be less interesting.[23] Head et al.”
BackgroundAre Betting Markets Inefficient? Evidence From Simulations and Real Data
Winkelmann, D., Ötting, M., Deutscher, C., & Makarewicz, T. (2023) · Journal of Sports Economics · doi:10.1177/15270025231204997 · Cited by 23
“Such mechanism, that is, a higher barrier to publication for studies that produce null results, is observed in different fi elds and labeled as publication bias (Franco et al., 2014).”
BackgroundPublication bias in psychology: A closer look at the correlation between sample size and effect size
Linden, A. H., Pollet, T. V., & Hönekopp, J. (2024) · PLoS ONE · doi:10.1371/journal.pone.0297075 · Cited by 22
The Effectiveness of Interventions Addressing Conspiracy Beliefs: A Meta‐Analysis
Stasielowicz, L. (2025) · European Journal of Social Psychology · doi:10.1002/ejsp.70041 · Cited by 16
Metadata
- Authors
- Franco, A., Malhotra, N., & Simonovits, G.
- Affiliations
- Stanford University (US)
- Department of Political Science, Stanford University, Stanford, CA, USA.
- Corresponding author
- Malhotra, Neil
- ORCID
- Malhotra, Neil
- Venue
- Science
- Published
- 2014-09-19
- Volume
- 345
- Issue
- 6203
- Pages
- 1502-1505
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W1997809363
- Cited by
- 1,542
- References
- 38
- Citations by year
- 2026: 88 · 2025: 152 · 2024: 143 · 2023: 162 · 2022: 129 · 2021: 152 · 2020: 212 · 2019: 123 · 2018: 119 · 2017: 110 · 2016: 91 · 2015: 52 · 2014: 9
- FWCI
- 85.56
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- Open Research (Auckland University of Technology) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- publication bias, negative results, file drawer problem, social sciences, survey experiment, peer review
- MeSH terms
- Social Sciences, Publication Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.