Publication bias in the social sciences: Unlocking the file drawer
The DOI matches its Crossref record; 37 of the 38 references it cites were found in a registry or an index.
Abstract
The file drawer is full. Should we worry?: Experiments that produce null results face a higher barrier to publication than those that yield statistically significant differences. Whether this is a problem depends on how many null but otherwise valid results might be trapped in the file drawer. Franco et al. use a Time-sharing Experiments in the Social Sciences archive of nearly 250 peer-reviewed proposals of social science experiments conducted on nationally representative samples. They find that only 10 out of 48 null results were published, whereas 56 out of 91 studies with strongly significant results made it into a journal. Science , this issue p. 1502
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Selective reporting, questionable research practices, and low-powered research all contribute to the unreliability of findings in a range of disciplines”
From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as Franco et al., 2014
In the citing paper
“In particular, negative or non-significant results are far less likely to be published (Franco et al., 2014).”
From What do we know about interventions to improve educator wellbeing? A systematic literature review (2024) · cites this paper as Franco et al., 2014
In the citing paper
“The bias against negative results appears to be a consequence of both editors and reviewers finding them less compelling for publication and from authors themselves not trying to publish them 8-12”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [8]
In the citing paper
“Positive results -- finding that a treatment is effective or that there is a relationship between two variables -- are more likely to be published than negative results 6-12 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [8]
In the citing paper
“A possible interpretation of these results is that the missing effect sizes are mainly a file‐drawer problem (Rosenthal, 1979). This interpretation is consistent with the evidence presented in Franco et al. (2014), which indicates that the file‐drawer problem is the main culprit behind publication bias in the social sciences.”
From Targeted school‐based interventions for improving reading and mathematics for students with or at risk of academic difficulties in Grades K‐6: A systematic review (2021) · cites this paper as Franco et al. (2014)
3 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722 · cited 1×
“In the presence of publication bias smaller studies report larger effects in order to exceed arbitrary significance thresholds (19, 20).”
BackgroundThe file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
Meta-analysis in social research
(1982) · Evaluation and Program Planning · doi:10.1016/0149-7189(82)90011-8 · Cited by 3,853
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109
Publication Bias in Meta‐Analysis: Prevention, Assessment and Adjustments
(2005) · doi:10.1002/0470870168 · Cited by 2,368
The case of the misleading funnel plot
Lau, J., Ioannidis, J. P. A., Terrin, N., Schmid, C. H., & Olkin, I. (2006) · BMJ · doi:10.1136/bmj.333.7568.597 · Cited by 1,766 · cited 1×
“However, these visualization-based approaches are sensitive to using different measures of precision (21, 22) and also assume outcome variables and effect sizes are comparable across studies (23).”
Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias
Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A.-W., Cronin, E., Decullier, E., Easterbrook, P. J., Von Elm, E., Gamble, C., et al. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003081 · Cited by 1,546 · cited 1×
“A different approach involves examining the publication outcomes of a cohort of studies, either prospectively or retrospectively (24, 25).”
MethodSumming up: The Science of Reviewing Research
Wilkinson, I., Light, R. J., & Pillemer, D. B. (1987) · Journal of Educational Statistics · doi:10.2307/1164691 · Cited by 1,471
Summing Up: The Science of Reviewing Research
Light, R. J., & Pillemer, D. B. (1984) · Harvard University Press eBooks · doi:10.4159/9780674040243 · Cited by 1,427
The Existence of Publication Bias and Risk Factors for Its Occurrence
Dickersin, K. (1990) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1990.03440100097014 · Cited by 1,417 · cited 1×
“Publication bias occurs when “publication of study results is based on the direction or significance of the findings” (1).”
Background
Research works printed but not linked: 3
- 10.1037/0735-7028.17.2.136 · doi:10.1037/0735-7028.17.2.136 · Paper page
- K. Dickersin in Publication Bias in Meta-Analysis: Prevention Assessment and Adjustments H. R. Rothstein A. J. Sutton M. Borenstein Eds. (Wiley Chichester UK 2006) ch. 2.
- Dickersin K., Min Y. I., Online J. Curr. Clin. Trials 1993, 50 (1993).
Cited by
Items 701–800 of 1,002
Huckle, K., & Silva, A. (2020) · Religions · doi:10.3390/rel11050249 · Cited by 14
“While there is a strong tendency to omit non-findings from publications, we wish to be a part of a growing body of researchers who acknowledge the importance of non-findings to the scientific process (Franco et al. 2014; Leal et al. 2016; Mervis 2014).”
BackgroundOpen Minds, Open Methods: Transparency and Inclusion in Pursuit of Better Scholarship
Rinke, E. M., & Wuttke, A. (2021) · PS Political Science & Politics · doi:10.1017/s1049096520001729 · Cited by 14
A Systematic Review and Qualitative Assessment of Fraud Detection Methodologies in Health Care
Ai, J., Russomanno, J., Guigou, S., & Allan, R. (2021) · North American Actuarial Journal · doi:10.1080/10920277.2021.1895843 · Cited by 12
Psychological Science's Aversion to the Null, and Why Many of the Things You Think Are True, Aren't
Heene, M., & Ferguson, C. J. (2017) · Wiley eBooks · doi:10.1002/9781119095910.ch3 · Cited by 14
Pre-analysis Plans: A StocktakingPreprint
Ofosu, G. K., & Posner, D. (2019) · doi:10.31222/osf.io/e4pum · Cited by 12
Neumark, D., & Corella, L. F. M. (2019) · National Bureau of Economic Research · doi:10.3386/w26462 · Cited by 12
Mathur, M. B., & VanderWeele, T. J. (2019) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/p3xyd · Cited by 13
What Big Data May Mean for Surveys
Couper, M. P. (2014) · Cited by 11
Virtual Work as a Job Demand? Work Behaviors of Public Servants during Covid-19
Lenz, L., Hattke, F., Kalucza, J., & Redlbacher, F. (2023) · Public Performance & Management Review · doi:10.1080/15309576.2023.2217552 · Cited by 13
“Nonetheless, our null findings relate to theoretical debates and are thus of value for advancing JD-R research, also because they help mitigating the “ file drawer problem ” (Franco et al., 2014).”
BackgroundExploring the scientific impact of negative results
Tian, D., Hu, X., Qian, Y., & Li, J. (2023) · Journal of Informetrics · doi:10.1016/j.joi.2023.101481 · Cited by 12
Rehren, P. (2024) · Frontiers in Psychology · doi:10.3389/fpsyg.2024.1388966 · Cited by 15
Conventional wisdom, meta‐analysis, and research revision in economics
Gechert, S., Mey, B., Opatrný, M., Havránek, T., Stanley, T. D., Bom, P. R. D., Doucouliagos, H., Heimberger, P., Iršová, Z., & Rachinger, H. J. (2024) · Journal of Economic Surveys · doi:10.1111/joes.12630 · Cited by 15
Assessment as learning: Evidence based on meta-analysis and quantitative ethnography research
Liu, Y., Xu, G., Yuan, S., Zhou, C., & Wang, C. (2024) · Studies In Educational Evaluation · doi:10.1016/j.stueduc.2024.101423 · Cited by 16
Balanced examination of positive publication bias impact
Ramos, M. (2025) · Accountability in Research · doi:10.1080/08989621.2025.2538066 · Cited by 12
“These and others (Dowdy, Tincani, and Schneider 2020; Driessen et al. 2015; Dwan et al. 2013; Franco, Malhotra, and Simonovits 2014; Olsson, Sundell, and Rowe 2023; Sham and Smith 2014) show how studies on positive publication bias seem largely focused on demonstrating the overrepresentation of…”
InfluentialAdam, K. C. S., Chang, L., Rangan, N., & Serences, J. T. (2021) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_01665 · Cited by 15
Borissova, A., Soni, S., Aston, E. R., Lees, R., Petrilli, K., Wall, M. B., Bloomfield, M., Mertzani, E., Paksina, A., Freeman, T. P., et al. (2022) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2022.109531 · Cited by 12
Till, C. E., Haverkamp, J., White, D. A., & Bhaduri, B. (2016) · The Journal of Defense Modeling and Simulation Applications Methodology Technology · doi:10.1177/1548512916679038 · Cited by 11
“Recent publications in Science highlighted exactly this frustration and call for a change in publishing culture.(96,97) The acknowledgement that a publication bias toward significant or targeted results exists has been known since the late 1950s, when Sterling investigated publishing decisions.”
BackgroundPreferences for Temporal Sequences of Real Outcomes Differ Across Domains but do not Vary by Age
Löckenhoff, C. E., Rutt, J. L., Samanez-Larkin, G. R., O’Donoghue, T., & Reyna, V. F. (2017) · The Journals of Gerontology Series B · doi:10.1093/geronb/gbx094 · Cited by 12
Resource Sharing to Improve Research Quality
Hamra, G. B., Goldstein, N. D., & Harper, S. B. (2019) · Journal of the American Heart Association · doi:10.1161/jaha.119.012292 · Cited by 12
Values affirmation replication at the University of Illinois
Gutmann, B., & Stelzer, T. J. (2021) · Physical Review Physics Education Research · doi:10.1103/physrevphyseducres.17.020121 · Cited by 12
Bertoldo, G., Callegher, C. Z., & Altoè, G. (2022) · Meta-Psychology · doi:10.15626/mp.2020.2573 · Cited by 14
Who nudges whom? Expert opinions on behavioural field experiments with public partners
Fels, K. (2022) · Behavioural Public Policy · doi:10.1017/bpp.2022.14 · Cited by 13
“’ In the academic literature, publication bias is a well-known phenomenon: If researchers have a greater tendency to submit, and editors a greater tendency to publish studies with significant results, the publicly available evidence will be systematically skewed (Franco et al. , 2014).”
Suter, S., Barrett, B., & Welden, N. A. C. (2023) · Environmental Monitoring and Assessment · doi:10.1007/s10661-022-10887-y · Cited by 19
“…increasing the publication of null findings, increasing citations from preregistrations, improving researcher rights, and heightening collaboration leading to greater research efficiency (Clements, 2017; Franco et al., 2014; Hajjem et al., 2006; Levin et al., 2016; McKiernan et al., 2016).”
BackgroundWormBase: A Model Organism Database
Price, M., & Hurd, D. D. (2019) · Medical Reference Services Quarterly · doi:10.1080/02763869.2019.1548896 · Cited by 12
Improving Transparency in Observational Social Science Research: A Pre-Analysis Plan ApproachPreprint
Burlig, F. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2731920 · Cited by 11
“Several studies from across the social sciences have documented a variety of issues in existing research, including publication bias (Rosenthal, 1979; Franco et al., 2014); specification search (Humphreys et al.”
BackgroundSensitivity analysis for publication bias in meta-analysesPreprint
Mathur, M. B., & VanderWeele, T. J. (2019) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/s9dp6 · Cited by 11
How Powerful is the Evidence in Criminology? On Whether We Should Fear a Coming Crisis of ConfidencePreprint
Barnes, J. C. (2017) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/qmeba · Cited by 11
Syropoulos, S., & Leidner, B. (2023) · Personality and Social Psychology Bulletin · doi:10.1177/01461672231192384 · Cited by 14
“This is primarily the consequence of the field being hesitant to publish findings that do not focus on significant differences (Franco et al., 2014).”
The relationship between school connectedness and depression: A three-level meta-analytic review
Meng, X., Yijing, C. H. E. N., Wang, X., Yuan, J., & Yu, D. (2023) · Advances in Psychological Science · doi:10.3724/sp.j.1042.2024.00246 · Cited by 12
“发表偏倚是指具有统计学意义的研究结果更容易发表(Franco et al., 2014)。发表偏倚是客观存在的, 因此研究者常常采用多种方法检验(Reed et al., 2015)。本研究将分别使用漏斗图(funnel plot)、EggerMLMA 回归法和剪补法对发表偏倚进行定性和定量评估。定性评估时, 若漏斗图呈对称的倒漏斗状, 则 表明发表偏倚较小(Sterne & Harbord, 2004)。在纳入分析的效应量间非相互独立时, 与传统 Egger 回归 法相比, Egger-MLMA 回归法更能有效控制 I 类错误(Rodgers & Pustejovsky,…”
Evaluating the Pedagogical Effectiveness of Study Preregistration in the Undergraduate Dissertation
Pownall, M., Pennington, C. R., Norris, E., Juanchich, M., Smailes, D., Russell, P. S., Gooch, D., Evans, T. R., Persson, S., Mak, M. H., et al. (2023) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459231202724 · Cited by 19
“This is related to the pressures put on academics to publish novel, positive results (Franco et al., 2014) because of the “pub-lish or perish” culture that pervades academia (Grimes et al., 2018), which might filter down to their students.”
BackgroundA meta-analysis of syntactic priming experiments in children
Kumarage, S., Donnelly, S., & Kidd, E. (2024) · Journal of Memory and Language · doi:10.1016/j.jml.2024.104532 · Cited by 14
McDonagh, T., Travers, Á., Armour, C., Cunningham, T., & Hansen, M. (2024) · Trauma Violence & Abuse · doi:10.1177/15248380241284793 · Cited by 13
“…literature in the present review, although done for quality-control reasons, is a factor which may have introduced a degree of bias into the findings, due to the reduced likelihood of nonsignificant findings to be published compared with papers reporting significant results (Franco et al., 2014).”
Conceptual and ethical considerations in invasion science
Haubrock, P. J., Parker, B., Błońska, D., Briski, E., Everts, T., Fernandez, R. D., Kouba, A., Kourantidou, M., Kurtul, I., Mammola, S., et al. (2025) · BioScience · doi:10.1093/biosci/biae138 · Cited by 16
“The so-called file drawer problem (Franco et al. 2014 ) can lead to a publication bias that emphasizes the most dramatic cases of nonnative species effects, underrepresenting studies that show neutral or less impactful results.”
Petrowsky, H. M., Boecker, L., Escher, Y. A., Frech, M.-L., Friese, M., Galinsky, A. D., Gunia, B., Lee, A. J., Schaerer, M., Schweinsberg, M., et al. (2025) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2025.104448 · Cited by 15
“Publication bias refers to non-significant effects being less likely to be published (Franco et al., 2014).”
The Banker’s oath and financial advice
Weitzel, U., & Kirchler, M. (2022) · Journal of Banking & Finance · doi:10.1016/j.jbankfin.2022.106750 · Cited by 13
Davis, K. E., & Maiden, R. (2021) · Studies in Social Science Research · doi:10.22158/sssr.v2n2p1 · Cited by 11
“get published, and authors know this (Rosenthal, 1979; Franco et al., 2014).”
BackgroundPersily, N., & Tucker, J. A. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108890960.002 · Cited by 11
Estimating the replicability of technology education research
Buckley, J., Hyland, T., & Seery, N. (2022) · International Journal of Technology and Design Education · doi:10.1007/s10798-022-09787-6 · Cited by 12
“…is critical to ensuring an accurate literature base which can otherwise present an inaccurate depiction of the validity of an effect and create bias in meta research (Aert et al., 2019; Brodeur et al., 2020; Franco et al., 2014; Friese & Frankenbach, 2020; Lin & Chu, 2018; Murad et al., 2018).”
BackgroundOlschewski, P., Herzmann, P., & Schlüter, K. (2023) · Education Sciences · doi:10.3390/educsci13040401 · Cited by 17
Kravitz, D. J., & Mitroff, S. R. (2023) · Journal for Reproducibility in Neuroscience · doi:10.36850/jrn.2023.e44 · Cited by 12
Attachment Styles and Suicidal Thoughts and Behaviors: A Meta-Analysis
MacNeil, S., Roderbourg, L., Orri, M., Geoffroy, M., McGrath, J. J., Renaud, J., & Gouin, J. (2023) · Journal of Social and Clinical Psychology · doi:10.1521/jscp.2023.42.4.323 · Cited by 13
Kappes, H. B., Toma, M., Balu, R., Burnett, R., Chen, N., Johnson, R., Leight, J., Omer, S. B., Safran, E., Steffel, M., et al. (2023) · Behavioral Science & Policy · doi:10.1177/23794607231192690 · Cited by 14
Use of Technology-Based Tools in Ensuring Quality of Publishable Journal Articles
Magulod, G. C., Capulso, L. B., Tabiolo, C. D. L., Luza, M. N., & Ramada, M. G. C. (2020) · International Journal of Learning Teaching and Educational Research · doi:10.26803/ijlter.19.11.9 · Cited by 22
“…community and the society at large (Wager & Kleinert, 2010), but what is lamenting is that many studies are never published and termed as the file-drawer problem (Dalton et al., 2012; Franco et al., 2014; Iwachiw, Button & Atlas, 2019; Lane et al., 2016; Simonsohn et al., 2014; Song et al., 2010).”
BackgroundPromoting farmers market via information nudges and coupons: A randomized control trial
Didero, N. M., Costanigro, M., & Jablonski, B. B. R. (2021) · Agribusiness · doi:10.1002/agr.21688 · Cited by 13
Turner, A. N., Parmar, N., Jovanovski, A., & Hearne, G. (2021) · Strength and conditioning journal · doi:10.1519/ssc.0000000000000625 · Cited by 16
Zuriaga, A., Kaplan, M. S., Choi, N. G., Hodkinson, A., Storman, D., Brudasca, N. I., Hirani, S. P., & Brini, S. (2021) · Journal of Affective Disorders · doi:10.1016/j.jad.2021.05.005 · Cited by 10
Li, H., Schrode, K. M., & Bee, M. A. (2022) · Animal Behaviour · doi:10.1016/j.anbehav.2022.05.003 · Cited by 10
Five Models of Science, Illustrating How Selection Shapes Methods
Smaldino, P. E. (2022) · University of Pittsburgh Press eBooks · doi:10.2307/j.ctv31djr2f.5 · Cited by 10
Refinetti, R., Sani, M., Jean‐Louis, G., Pandi-Perumal, S. R., Durazo-Arvizú, R., Dugas, L. R., Kafensztok, R., Bovet, P., Forrester, T., Lambert, E. V., et al. (2015) · Annals of Medicine · doi:10.3109/07853890.2015.1085125 · Cited by 12
“This recommendation is reinforced by the fact that publication bias in the form of selective suppression of null findings has been documented also in the social sciences (40).”
BackgroundResearch practices and assessment of research misconduct
Hartgerink, C. H. J., & Wicherts, J. M. (2016) · ScienceOpen Research · doi:10.14293/s2199-1006.1.sor-socsci.arysbi.v1 · Cited by 11
Exposure to suicidal behaviors: A common suicide risk factor or a personal negative life event?
Harris, K. M., & Bettiol, S. S. (2016) · International Journal of Social Psychiatry · doi:10.1177/0020764016682361 · Cited by 12
Deressa, T. K., Stern, D. I., Vangronsveld, J., Minx, J. C., Lizin, S., Malina, R. M., & Bruns, S. B. (2023) · EcoEvoRxiv · doi:10.32942/x24g6z · Cited by 13
Sprawność intelektualna osób o odmiennych chronotypach
Jankowski, K. S. (2021) · Wydawnictwo Liberi Libri eBooks · doi:10.47943/lib.9788363487515.rozdzial10 · Cited by 12
Inteligencja a funkcjonowanie w bliskich związkach
Leniarska, M., & Zajenkowski, M. (2021) · Wydawnictwo Liberi Libri eBooks · doi:10.47943/lib.9788363487515.rozdzial11 · Cited by 8
Inteligencja a agresja. Czy wysokie IQ może uchronić przed wkroczeniem na ścieżkę przestępczą?
Bodecka-Zych, M. (2021) · Wydawnictwo Liberi Libri eBooks · doi:10.47943/lib.9788363487515.rozdzial03 · Cited by 8
Inteligentny sport: o związkach inteligencji z aktywnością fizyczną i poziomem wykonania sportowego
Waleriańczyk, W., & Stolarski, M. (2021) · Wydawnictwo Liberi Libri eBooks · doi:10.47943/lib.9788363487515.rozdzial09 · Cited by 8
Coburn, K. L., Kurtz, M. R., Rivera, D. L., & Kana, R. K. (2022) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2022.104748 · Cited by 14
US school finance: Resources and outcomes
Handel, D., & Hanushek, E. A. (2023) · Handbook of the economics of education · doi:10.1016/bs.hesedu.2023.03.003 · Cited by 24
EviewsR: An R Package for Dynamic and Reproducible Research Using EViews, R, R Markdown and Quarto
Mati, S., Civcir, İ., & Abba, S. I. (2023) · The R Journal · doi:10.32614/rj-2023-045 · Cited by 21
Publication bias is bad for science if not necessarily scientists
Heesen, R., & Bright, L. K. (2025) · Royal Society Open Science · doi:10.1098/rsos.240688 · Cited by 12
“…lived experience to draw on in understanding the more or less formalized procedures that scientists use to decide whether to publish studies, and second, from their being sufficiently tapped into their epistemic community that they read and appropriately understand papers like Franco et al. [43].”
Knox, S., Gök, A., Aydoğdu, E., Carter, S., Levie, J., & Shaw, E. (2025) · Entrepreneurship and Regional Development · doi:10.1080/08985626.2025.2503145 · Cited by 21
“First, documents that provide evidence on negative outcomes tend to be published less and thus negative outcomes are possibly under-represented in our study (Franco, Malhotra, and Simonovits 2014).”
Cai, Y., Kang, D., & Xu, X. (2025) · Frontiers in Psychology · doi:10.3389/fpsyg.2025.1659927 · Cited by 15
“This selective dissemination can result in a published literature that does not comprehensively represent the entire body of research conducted in the field (Franco et al., 2014).”
Yan, J., Cai, K., Zhu, L., Fan, M., Zhao, Y., & Wang, D. (2025) · Brain and Behavior · doi:10.1002/brb3.70883 · Cited by 12
“Publication bias is the tendency for findings that are statistically significant to be published more frequently (Franco et al. 2014; Rodgers and Pustejovsky 2021).”
The role of family relationships in youth mentoring: An ecological perspective
Downey, S. K., Lyons, M. D., & Williams, J. L. (2022) · Children and Youth Services Review · doi:10.1016/j.childyouth.2022.106508 · Cited by 10
do Rêgo, G. G., Gonçalves, Ó. F., & Boggio, P. S. (2022) · Scientific Reports · doi:10.1038/s41598-022-22245-6 · Cited by 9
Guedes, D., Prada, M., Lamy, E., & Garrido, M. V. (2023) · Food Quality and Preference · doi:10.1016/j.foodqual.2023.104964 · Cited by 9
A meta-analysis of the impact of AR and VR technologies on mathematics learning
Cao, L. (2023) · Journal of Education Humanities and Social Sciences · doi:10.54097/ehss.v23i.13133 · Cited by 12
“In systematic review and meta-analyses, there is a serious problem, publication bias, which can have a large extent of negative impact on the validity of the conclusions [39].”
BackgroundKalny, C. S., & Walter, N. (2024) · Science Communication · doi:10.1177/10755470241231289 · Cited by 11
“Pre-registration is undoubtedly a valuable practice designed to enhance methodological transparency and promote safeguards against potential bias in research design and analysis (Franco et al., 2014; Nosek et al., 2018).”
Program and Policy Evaluations in Practice: Highlights from the Federal Perspective
Maynard, R., Goldstein, N., & Nightingale, D. (2016) · New Directions for Evaluation · doi:10.1002/ev.20209 · Cited by 7
Postpublication Metrics of Randomized Clinical Trials With and Without Null Findings
Murray, S. B., Heathers, J. A. J., Schauer, R., Griffiths, S., Mitchison, D., Mond, J., & Nagata, J. M. (2019) · JAMA · doi:10.1001/jama.2019.2994 · Cited by 9
Cooper, C. A. (2020) · Journal of Public and Nonprofit Affairs · doi:10.20899/jpna.6.1.63-78 · Cited by 11
The conditional nature of publication bias: a meta-regression analysis
Owen, E., & Li, Q. (2020) · Political Science Research and Methods · doi:10.1017/psrm.2020.15 · Cited by 9
“…such as medicine (e.g., Easterbrook et al. 1991), psychology (e.g., Ferguson and Heene 2012), economics (e.g., Stanley 2008) and more recently, political science (e.g., Doucouliagos and Ulubaşoglu 2008; Gerber and Malhotra 2008; Ahmadov 2014; Franco et al. 2014; Findley et al. 2016; Philips 2016).”
BackgroundA framework for open policy analysis
de la Guardia, F. H., Grant, S., & Miguel, E. A. (2020) · Science and Public Policy · doi:10.1093/scipol/scaa067 · Cited by 19
Murphy, R., Cunningham, C. J., Harris, B. P., & Brown, C. L. (2020) · Fisheries · doi:10.1002/fsh.10568 · Cited by 8
Is the Psychopathic Brain an Artifact of Coding Bias? A Systematic Review
Jalava, J., Griffiths, S. Y., Larsen, R. R., & Alcott, B. E. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.654336 · Cited by 13
“Research tracking original studies from grant proposal to publication show that most null-findings are not even written up for publication, and that journals—particularly top-tier journals—show amarked preference for strong positive findings (Franco et al., 2014; Ioannidis et al., 2014).”
BackgroundDo naloxone access laws affect perceived risk of heroin use? Evidence from national US data
Kelly, B. C., & Vuolo, M. (2021) · Addiction · doi:10.1111/add.15682 · Cited by 10
Neoh, M. J. Y., Carollo, A., Lee, A., & Esposito, G. (2023) · Royal Society Open Science · doi:10.1098/rsos.230677 · Cited by 17
“…just below the commonly used threshold of 0.05 have been reported to be more prevalent, in part owing to selective publication of positive results [6] but in part, also attributable to false positive results. de Winter & Dodou [72] demonstrate an increase in p -values from 0.041 to 0.049, and…”
BackgroundResultInfluentialCook, B. G. (2016) · Behavioral Disorders · doi:10.17988/0198-7429-41.3.161 · Cited by 9
Krefeld-Schwalb, A., Witte, E. H., & Zenker, F. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.00460 · Cited by 15
Brauer, J. R., Day, J. C., & Hammond, B. M. (2019) · Sociological Methods & Research · doi:10.1177/0049124119826158 · Cited by 9
“Consequently, researchers and editors increasingly are encouraged to report and publish null findings as a remedy to this “file drawer” problem (cf. Franco, Malhotra, and Simonovits 2014; Goodchild van Hilten 2015; Grimes, Bauch, and Ioannidis 2018).”
BackgroundPublication Bias in the Social Sciences: A Threat to Scientific Integrity
Malhotra, N. A., & Zigerell, L. J. (2022) · Oxford University Press eBooks · doi:10.1093/oso/9780190938550.003.0013 · Cited by 11
Self-citationStrengths and Weaknesses of Meta-AnalysesPreprint
Corker, K. S. (2018) · doi:10.31234/osf.io/6gcnm · Cited by 10
The Role of A/B Tests in the Study of Large-Scale Online LearningPreprint
Savi, A. O., Williams, J. J., Maris, G., & van der Maas, H. L. J. (2017) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/83jsg · Cited by 9
How many is enough? Sample Size in Staggered Difference-in-Differences DesignsPreprint
Egerod, B. C. K., & Hollenbach, F. M. (2024) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/ac5ru · Cited by 17
Wu, D., Wu, H., & Li, J. (2024) · Scientometrics · doi:10.1007/s11192-024-05074-4 · Cited by 8
“This situation could perpetuate the ‘file drawer problem,’ where negative or inconclusive results remain unreported, which may contribute to publication bias and a lack of transparency in the scientific literature (Franco et al., 2014; Rosenthal, 1979).”
de Haan, E. (2021) · The Coaching Psychologist · doi:10.53841/bpstcp.2021.17.1.7 · Cited by 23
Karabağ, S. F., Berggren, C., Pielaszkiewicz, J., & Gerdin, B. (2024) · Journal of Academic Ethics · doi:10.1007/s10805-024-09520-z · Cited by 12
“contrast to previous studies (Franco et al., 2014), however, the norm Openness does not emerge as an important factor associated with lower QRP prevalence.”
The effects of rule changes in football-code team sports: a systematic review
Rennie, G., Chesson, L. J., Weaving, D., & Jones, B. L. (2024) · Science and Medicine in Football · doi:10.1080/24733938.2024.2375752 · Cited by 10
“…bias is greater likelihood of statistically significant findings being published than null findings when research quality is of an equal standard (Franco et al. 2014), and is a limitation within biomedical (Song et al. 2010), social (Coursol and Wagner 1986; Gerber and Malhotra 2008), and sports…”
Do Changes in Journal Rank Influence Publication Output? Evidence from China
Sun, Z., Zhang, C., Pang, K. L., Tang, Y., & Li, Y. (2024) · Scientometrics · doi:10.1007/s11192-024-05167-0 · Cited by 8
“This study could be a useful step toward better understanding trends in top journal publications (Franco et al., 2014) induced by journal evaluation system in developing countries.”
Is musical ability related to second-language acquisition? A meta-analysis
Thompson, R. M., Salig, L. K., & Slevc, L. R. (2025) · Royal Society Open Science · doi:10.1098/rsos.241193 · Cited by 7
“Aside from our primary conclusions, this meta-analysis suggests that the literature on music and second-language learning suffers from publication bias (the unfortunately common ‘file-drawer problem’; e.g. [103]).”
Lengersdorff, L., & Lamm, C. (2025) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459241296397 · Cited by 8
“Another phenomenon that severely affects the credibility of research output is publication bias (Franco et al., 2014).”
Martín‐Tapia, I., & Sánchez, R. L. (2025) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2025.145204 · Cited by 10
Organizational-Level Training and Performance: A Meta-Analytic Investigation
Kim, J., Chang, H., & Bell, B. S. (2025) · Journal of Management · doi:10.1177/01492063251327588 · Cited by 24
“Third, we controlled for publication status, accounting for the potential “file drawer” issue, where more pronounced results are more likely to appear in published than unpublished studies (Franco, Malhotra, & Simonovits, 2014).”
The Need for Formal Evidence Synthesis in Food Policy: A Case Study of Willingness-to-Pay
Clark, B., Frewer, L. J., Panzone, L., & Stewart, G. B. (2017) · Animals · doi:10.3390/ani7030023 · Cited by 8
“Despite the majority of evidence coming from the health sciences [20], there is also evidence of a growing body of evidence in the social science literature [22], for example, in economics [17,19] and psychology [23].”
BackgroundThe publication symmetry test: a simple editorial heuristic to combat publication bias
Earp, B. D., & Wilkinson, D. J. C. (2017) · Journal of Clinical and Translational Research · doi:10.18053/jctres.03.2017s2.002 · Cited by 10
“In a recent attempt to estimate the extent of the problem, researchers examined the fate of 221 studies from the social sciences that had been pre-registered in a database between 2002 and 2012 [18].”
BackgroundInfluentialCañedo-Ayala, M., Rice, D. B., Levis, A. W., Chiovitti, M. J., & Thombs, B. D. (2020) · Health Psychology · doi:10.1037/hea0001020 · Cited by 8
Reis, D., & Friese, M. (2022) · Springer eBooks · doi:10.1007/978-3-031-04968-2_5 · Cited by 14
Neuroethics: Agency in the Age of Brain Science
May, J. (2023) · doi:10.1093/oso/9780197648087.001.0001 · Cited by 14
Socioeconomic impacts of land restoration in agriculture: A systematic review
Malan, M., Berkhout, E., Duchoslav, J., Voors, M. J., & van der Esch, S. (2024) · Q Open · doi:10.1093/qopen/qoae022 · Cited by 9
“It is plausible, similar to other systematic reviews, that studies showing significant results are more likely to be published than studies that find null effects (Franco, Malhotra et al. 2014).”
ResultBrenner, P. S. (2020) · Frontiers in sociology and social research · doi:10.1007/978-3-030-47256-6_3 · Cited by 13
Metadata
- Authors
- Franco, A., Malhotra, N., & Simonovits, G.
- Affiliations
- Stanford University (US)
- Department of Political Science, Stanford University, Stanford, CA, USA.
- Corresponding author
- Malhotra, Neil
- ORCID
- Malhotra, Neil
- Venue
- Science
- Published
- 2014-09-19
- Volume
- 345
- Issue
- 6203
- Pages
- 1502-1505
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W1997809363
- Cited by
- 1,542
- References
- 38
- Citations by year
- 2026: 88 · 2025: 152 · 2024: 143 · 2023: 162 · 2022: 129 · 2021: 152 · 2020: 212 · 2019: 123 · 2018: 119 · 2017: 110 · 2016: 91 · 2015: 52 · 2014: 9
- FWCI
- 85.56
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- Open Research (Auckland University of Technology) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- publication bias, negative results, file drawer problem, social sciences, survey experiment, peer review
- MeSH terms
- Social Sciences, Publication Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.