Publication bias in the social sciences: Unlocking the file drawer
The DOI matches its Crossref record; 37 of the 38 references it cites were found in a registry or an index.
Abstract
The file drawer is full. Should we worry?: Experiments that produce null results face a higher barrier to publication than those that yield statistically significant differences. Whether this is a problem depends on how many null but otherwise valid results might be trapped in the file drawer. Franco et al. use a Time-sharing Experiments in the Social Sciences archive of nearly 250 peer-reviewed proposals of social science experiments conducted on nationally representative samples. They find that only 10 out of 48 null results were published, whereas 56 out of 91 studies with strongly significant results made it into a journal. Science , this issue p. 1502
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Selective reporting, questionable research practices, and low-powered research all contribute to the unreliability of findings in a range of disciplines”
From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as Franco et al., 2014
In the citing paper
“In particular, negative or non-significant results are far less likely to be published (Franco et al., 2014).”
From What do we know about interventions to improve educator wellbeing? A systematic literature review (2024) · cites this paper as Franco et al., 2014
In the citing paper
“The bias against negative results appears to be a consequence of both editors and reviewers finding them less compelling for publication and from authors themselves not trying to publish them 8-12”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [8]
In the citing paper
“Positive results -- finding that a treatment is effective or that there is a relationship between two variables -- are more likely to be published than negative results 6-12 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [8]
In the citing paper
“A possible interpretation of these results is that the missing effect sizes are mainly a file‐drawer problem (Rosenthal, 1979). This interpretation is consistent with the evidence presented in Franco et al. (2014), which indicates that the file‐drawer problem is the main culprit behind publication bias in the social sciences.”
From Targeted school‐based interventions for improving reading and mathematics for students with or at risk of academic difficulties in Grades K‐6: A systematic review (2021) · cites this paper as Franco et al. (2014)
3 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722 · cited 1×
“In the presence of publication bias smaller studies report larger effects in order to exceed arbitrary significance thresholds (19, 20).”
BackgroundThe file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
Meta-analysis in social research
(1982) · Evaluation and Program Planning · doi:10.1016/0149-7189(82)90011-8 · Cited by 3,853
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109
Publication Bias in Meta‐Analysis: Prevention, Assessment and Adjustments
(2005) · doi:10.1002/0470870168 · Cited by 2,368
The case of the misleading funnel plot
Lau, J., Ioannidis, J. P. A., Terrin, N., Schmid, C. H., & Olkin, I. (2006) · BMJ · doi:10.1136/bmj.333.7568.597 · Cited by 1,766 · cited 1×
“However, these visualization-based approaches are sensitive to using different measures of precision (21, 22) and also assume outcome variables and effect sizes are comparable across studies (23).”
Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias
Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A.-W., Cronin, E., Decullier, E., Easterbrook, P. J., Von Elm, E., Gamble, C., et al. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003081 · Cited by 1,546 · cited 1×
“A different approach involves examining the publication outcomes of a cohort of studies, either prospectively or retrospectively (24, 25).”
MethodSumming up: The Science of Reviewing Research
Wilkinson, I., Light, R. J., & Pillemer, D. B. (1987) · Journal of Educational Statistics · doi:10.2307/1164691 · Cited by 1,471
Summing Up: The Science of Reviewing Research
Light, R. J., & Pillemer, D. B. (1984) · Harvard University Press eBooks · doi:10.4159/9780674040243 · Cited by 1,427
The Existence of Publication Bias and Risk Factors for Its Occurrence
Dickersin, K. (1990) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1990.03440100097014 · Cited by 1,417 · cited 1×
“Publication bias occurs when “publication of study results is based on the direction or significance of the findings” (1).”
Background
Research works printed but not linked: 3
- 10.1037/0735-7028.17.2.136 · doi:10.1037/0735-7028.17.2.136 · Paper page
- K. Dickersin in Publication Bias in Meta-Analysis: Prevention Assessment and Adjustments H. R. Rothstein A. J. Sutton M. Borenstein Eds. (Wiley Chichester UK 2006) ch. 2.
- Dickersin K., Min Y. I., Online J. Curr. Clin. Trials 1993, 50 (1993).
Cited by
Items 601–700 of 1,002
A Systematic Review of Autohemotherapy as a Treatment for Urticaria and Eczema
Brewer, D. D. (2014) · Cureus · doi:10.7759/cureus.233 · Cited by 16
p-Hacking in Experimental Audit Research
Khan, M. J., & Tronnes, P. C. (2018) · Behavioral Research in Accounting · doi:10.2308/bria-52183 · Cited by 16
“Importantly, it can mask potential implications of non-significant p-values (Franco et al., 2014), which are either manipulated into false positives or remain unpublished.”
BackgroundResultPhysician Empathy Is Not Associated with Laboratory Outcomes in Diabetes: a Cross-sectional Study
Chaitoff, A. M., Rothberg, M. B., Windover, A. K., Calabrese, L. H., Misra‐Hebert, A. D., & Martinez, K. A. (2018) · Journal of General Internal Medicine · doi:10.1007/s11606-018-4731-0 · Cited by 21
Paternal Depressive Symptoms and Marital Quality: A Meta‐Analysis of Cross‐Sectional Studies
Cheung, K., Theule, J., Hiebert‐Murphy, D., & Piotrowski, C. C. (2019) · Journal of Family Theory & Review · doi:10.1111/jftr.12341 · Cited by 21
Vasilev, M. R., Yates, M. J., & Slattery, T. J. (2019) · Journal of Cognition · doi:10.5334/joc.87 · Cited by 18
Wise academic development: learning from the ‘failure’ experiences of retired academic developers
Timmermans, J. A., & Sutherland, K. A. (2020) · The International Journal for Academic Development · doi:10.1080/1360144x.2019.1704291 · Cited by 16
Estimating the effect of spillovers on exports: a meta-analysis
Duan, J., Das, K. K., Meriluoto, L., & Reed, W. R. (2020) · Review of World Economics · doi:10.1007/s10290-020-00377-z · Cited by 20
Putwain, D. W., Symes, W., Coxon, E., & Gallard, D. (2020) · Educational Psychology · doi:10.1080/01443410.2020.1740653 · Cited by 22
Edridge, C., Wolpert, M., Deighton, J., & Edbrooke-Childs, J. H. (2020) · Journal of Medical Internet Research · doi:10.2196/14223 · Cited by 18
van der Heijden, J. (2020) · Regulation & Governance · doi:10.1111/rego.12378 · Cited by 19
Open science reforms: Strengths, challenges, and future directions
Wentzel, K. R. (2021) · Educational Psychologist · doi:10.1080/00461520.2021.1901709 · Cited by 20
“In addition, claims concerning publication bias (e.g., Franco et al., 2014), questionable research practices, and lack of transparency (e.g., Anderson et al., 2007; John et al., 2012) are not based on work in educational psychology.”
BackgroundSamota, E. K., & Davey, R. (2021) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2021.678554 · Cited by 23
“Furthermore, for transparent and reproducible science, both negative (or null) and positive results should be reported for others to examine the evidence (Franco et al., 2014; Prager et al., 2019; Miyakawa, 2020).”
BackgroundAn Experimental Approach to Economic Voting
Simonovits, G. (2015) · Political Behavior · doi:10.1007/s11109-015-9303-y · Cited by 16
Self-citationNull Findings, Replications and Preregistered Studies in Business Ethics Research
Roloff, J., & Zyphur, M. J. (2018) · Journal of Business Ethics · doi:10.1007/s10551-018-3864-8 · Cited by 20
Adult Self-Esteem and Family Relationships: A Literature Review
Ghaziri, N. E., & Darwiche, J. (2018) · Swiss Journal of Psychology · doi:10.1024/1421-0185/a000212 · Cited by 21
Bernard, R., Weissgerber, T. L., Bobrov, E., Winham, S. J., Dirnagl, U., & Riedel, N. (2020) · Clinical Science · doi:10.1042/cs20201125 · Cited by 19
U.S. School Finance: Resources and Outcomes
Handel, D., & Hanushek, E. A. (2022) · National Bureau of Economic Research · doi:10.3386/w30769 · Cited by 23
Coles, N. A., Wyatt, M., & Frank, M. C. (2023) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/uw85a · Cited by 16
da Silva, J. A. T., & Nazarovets, S. (2025) · Naunyn-Schmiedeberg s Archives of Pharmacology · doi:10.1007/s00210-025-04651-5 · Cited by 25
“From another perspective, how much unpublished “file drawer” research that contains a high percentage of non-significant results (Rosenthal 1979), or null findings (Franco et al. 2014), should be published?”
Brosens, L., Robinet, J., Pelckmans, I., Ameijeiras‐Mariño, Y., Govers, G., Opfergelt, S., Minella, J. P. G., & Vanderborght, J. (2021) · CATENA · doi:10.1016/j.catena.2021.105698 · Cited by 17
Meier, M., Lonsdorf, T. B., LUPIEN, S. J., Stalder, T., Laufer, S., Sicorello, M., Linz, R., & Puhlmann, L. M. C. (2022) · Comprehensive Psychoneuroendocrinology · doi:10.1016/j.cpnec.2022.100144 · Cited by 20
“as a direct response to the finding that many published studies – in some fields indeed the majority of studies – cannot be replicated (often referred to as the so-called ‘replication crisis’; [3].”
ResultRibeiro, L. A., Donolato, E., Aguiar, C., Correia, N. E. F. G., & Zachrisson, H. D. (2023) · Journal of Cognition and Development · doi:10.1080/15248372.2023.2248259 · Cited by 17
“…it is more likely that published studies are those with larger and statistically significant findings since studies with null findings are harder to publish – an estimate of 21% - whereas 65% of the studies with significant findings make it into a journal (Franco, Malhotra, & Simonovits, 2014).”
ResultWieczorek, K., DeGroot, M., Madigan, S. L., Pador, P., Ganshorn, H., & Graham, S. A. (2023) · American Journal of Speech-Language Pathology · doi:10.1044/2023_ajslp-22-00406 · Cited by 27
The power of the “be creative” instruction: A meta-analytical evaluation
Wei, X., Shen, W., Long, H., & Lu, F. (2023) · Learning and Motivation · doi:10.1016/j.lmot.2023.101945 · Cited by 18
Revisiting the Income Inequality-Crime Puzzle
Pazzona, M. (2024) · World Development · doi:10.1016/j.worlddev.2023.106520 · Cited by 21
“The extent of publication bias in economics is widespread (Blanco-Perez & Brodeur, 2020; Brodeur et al., 2020) but also in other fields (Franco, Malhotra, & Simonovits, 2014).”
BackgroundREES: A Registry of Efficacy and Effectiveness Studies in Education
Anderson, D., Spybrook, J. K., & Maynard, R. A. (2018) · Educational Researcher · doi:10.3102/0013189x18810513 · Cited by 16
Marques‐Quinteiro, P., Rico, R., Passos, A., & Curral, L. A. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.00847 · Cited by 19
Governance and the capital flight trap in Africa
Asongu, S. A., & Nnanna, J. (2020) · Transnational Corporation Review · doi:10.1080/19186444.2020.1771123 · Cited by 18
Moving Health Literacy Research and Practice towards a Vision of Equity, Precision and Transparency
Guo, S., Yu, X., & Okan, O. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17207650 · Cited by 18
“However, in practice, null results are less frequently published than statistically significant results and are more likely to be inaccessible and lost in the “file drawer” [82].”
BackgroundStability of Experimental Results: Forecasts and Evidence
DellaVigna, S., & Pope, D. G. (2019) · American Economic Journal Microeconomics · doi:10.1257/mic.20200129 · Cited by 23
Registered Reports in Special Education: Introduction to the Special Series
Cook, B. G., Maggin, D. M., & Robertson, R. E. (2021) · Remedial and Special Education · doi:10.1177/0741932521996459 · Cited by 14
“Indeed, Franco et al. (2014) found many researchers in the social sciences simply choose to not write up studies with null findings, perhaps due to the belief they will not be published.”
BackgroundWest, J. J., Stanley, A. M., Bowers, J. P., & Isbell, D. S. (2021) · Bulletin of the Council for Research in Music Education · doi:10.5406/bulcouresmusedu.229.0007 · Cited by 18
“This absence of negative findings could be attributed to publication bias (Fanelli, 2012; Franco et al., 2014), with researchers or reviewers dismissing outcomes from unsuccessful PD programs as aberrant or flawed.”
BackgroundThe Experiences of Students with Disabilities Actually Doing Service Learning: A Systematic Review
Garwood, J. D., Peltier, C. J., Ciullo, S. P., Wissinger, D. R., McKenna, J. W., Giangreco, M. F., & Kervick, C. T. (2022) · Journal of Experiential Education · doi:10.1177/10538259221109374 · Cited by 15
“Dissertations were included to minimize issues related to publication bias (see Franco et al., 2014).”
BackgroundVidafar, P., & Spitschan, M. (2022) · Journal of Biological Rhythms · doi:10.1177/07487304221126785 · Cited by 17
“GnRH stimulates the pituitary gland, which in turn releases a surge of luteinizing hormone (Franco et al., 2014) and FSH (Mihm et al.”
BackgroundResult“Transit makes you short”: On health impact assessment of transportation and the built environment
Ermagun, A., & Levinson, D. M. (2016) · Journal of Transport & Health · doi:10.1016/j.jth.2016.08.004 · Cited by 15
“Samimi et al. (2010) also mentioned that a one percent increase in public transit use increases the chance of having excellent, very good, or good general health by 0.002 percent.”
InfluentialWork Values across Generations among Construction Professionals in the United States
Simmons, D. R., Iorio, J., Taylor, J. E., & Li, D. (2018) · Journal of Construction Engineering and Management · doi:10.1061/(asce)co.1943-7862.0001554 · Cited by 18
Chu, L., & Grafton, R. Q. (2019) · Agricultural Water Management · doi:10.1016/j.agwat.2019.105881 · Cited by 17
Masouleh, S. K., Eickhoff, S. B., Balajoo, S. M., Nicolaisen‐Sobesky, E., Thirion, B., & Genon, S. (2022) · Scientific Reports · doi:10.1038/s41598-022-17556-7 · Cited by 20
Fifty Shades of QE: Conflicts of Interest in Economic ResearchPreprint
Fabo, B., Jančoková, M., Kempf, E., & Pástor, Ľ. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3693196 · Cited by 15
One finding is no finding: Toward a replication culture in family business research
Brinkerink, J., Massis, A. D., & Kellermanns, F. W. (2022) · Journal of Family Business Strategy · doi:10.1016/j.jfbs.2022.100521 · Cited by 18
Zaller, N. D., Gorvine, M. M., Ross, J., Mitchell, S. G., Taxman, F. S., & Farabee, D. J. (2022) · Addiction Science & Clinical Practice · doi:10.1186/s13722-022-00351-0 · Cited by 20
Xie, S., & Mi, C. (2023) · Learned Publishing · doi:10.1002/leap.1515 · Cited by 15
“Second, the inclination to use more positive words in abstracts might be caused by the so-called publication bias and outcome report bias in both hard and soft sciences (Dwan et al., 2013; Franco et al., 2014).”
BackgroundIs Democracy the Answer to Intractable Climate Change?
Chesler, A., Javeline, D., Peh, K., & Scogin, S. (2023) · Global Environmental Politics · doi:10.1162/glep_a_00710 · Cited by 17
“The null fi ndings are especially important considering publication bias that favors statistically signi fi cant fi ndings (Franco et al. 2014) and thus may inadvertently give undue attention to the role of regime type.”
BackgroundAssociation between childhood maltreatment and empathy: A three-level meta-analytic review
Meng, X., Yu, D., Yijing, C. H. E. N., Zhang, L., & Fu, X. (2023) · Acta Psychologica Sinica · doi:10.3724/sp.j.1041.2023.01285 · Cited by 22
Liu, Z., Wen, J., Liu, Y., & Chuan-Peng, H. (2023) · British Journal of Educational Psychology · doi:10.1111/bjep.12636 · Cited by 23
Zhang, X.-E., Pang, Y., Bo, Q., Hu, S., Xiang, J., Yang, Z.-R., Zhang, X., Chen, A., Zeng, J., Ma, X., et al. (2023) · PLoS ONE · doi:10.1371/journal.pone.0282275 · Cited by 18
Decision avoidance and post-decision regret: A systematic review and meta-analysis
Han, Q., Quadflieg, S., & Ludwig, C. J. H. (2023) · PLoS ONE · doi:10.1371/journal.pone.0292857 · Cited by 21
“Publication bias means that the set of effects are a biased sample from a population of effects, where smaller (null) effects are less likely to be published [68].”
BackgroundEngineering Inequality: Informal Coaching, Glass Walls, and Social Closure in Silicon Valley
Luhr, S. W. (2024) · American Journal of Sociology · doi:10.1086/729506 · Cited by 18
The effect of mere presence of smartphone on cognitive functions: A four-level meta-analysis.
Hartanto, A., Lua, V. Y. Q., Kasturiratna, K. T. A. S., Koh, P. S., Tng, G. Y. Q., Kaur, M., Quek, F. Y. X., Chia, J. L., & Majeed, N. M. (2024) · Technology Mind and Behavior · doi:10.1037/tmb0000123 · Cited by 25
Fernández‐Castilla, B., Said‐Metwaly, S., Kreitchmann, R. S., & Noortgate, W. V. D. (2024) · Behavior Research Methods · doi:10.3758/s13428-024-02373-9 · Cited by 17
“Negative findings face a greater publication challenge than their positive counter-parts (Fanelli, 2010; Franco et al., 2014).”
BackgroundAre the Signs of Factor Loadings Arbitrary in Confirmatory Factor Analysis? Problems and Solutions
Tang, D., Boker, S. M., & Tong, X. (2024) · Structural Equation Modeling A Multidisciplinary Journal · doi:10.1080/10705511.2024.2351102 · Cited by 25
“Franco et al. (2014) found that only 20% of the studies published in social sciences journals reported non-significant results, and 60% of the studies discovered non-significant results but never reported them.”
Growth mindsets in academics and academia: a review of influence and interventions
Sousa, B. J., & Clark, A. M. (2024) · Journal of Higher Education Policy and Management · doi:10.1080/1360080x.2024.2384003 · Cited by 23
“Studies in this area may be prone to publication bias – the tendency for affirmative studies to be more likely to be published (Franco et al., 2014).”
Winner and loser effects: a meta-analysis
Yan, J. L., Smith, N. M. T., Filice, D. C. S., & Dukas, R. (2024) · Animal Behaviour · doi:10.1016/j.anbehav.2024.07.014 · Cited by 23
Is planning through the Internet (un)related to trip satisfaction?
Ferrer-Rosell, B., Coenders, G., & Mariné-Roig, E. (2017) · Information Technology & Tourism · doi:10.1007/s40558-017-0082-7 · Cited by 17
Making Research Data Accessible
Kapiszewski, D., & Karcher, S. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.008 · Cited by 16
Salandra, R., Criscuolo, P., & Salter, A. J. (2020) · Research Policy · doi:10.1016/j.respol.2020.104130 · Cited by 16
Shifting Minds: A Quantitative Reappraisal of Cognitive-Intervention Research
Moreau, D. (2020) · Perspectives on Psychological Science · doi:10.1177/1745691620950696 · Cited by 16
Hite, R. L., Shin, S., & Lesley, M. K. (2021) · Journal of Academic Ethics · doi:10.1007/s10805-021-09418-0 · Cited by 21
Sharing is caring: Ethical implications of transparent research in psychology.
Bosma, C. M., & Granger, A. M. (2022) · American Psychologist · doi:10.1037/amp0001002 · Cited by 24
Long, J. S., & Sullivan, C. J. (2016) · Crime & Delinquency · doi:10.1177/0011128716629757 · Cited by 16
How null results can be significant for physics education research
Conlin, L. D., Kuo, E., & Hallinen, N. R. (2019) · Physical Review Physics Education Research · doi:10.1103/physrevphyseducres.15.020104 · Cited by 19
Brown, E., Shrestha, M., & Gray, R. J. (2020) · Brazilian Journal of Psychiatry · doi:10.1590/1516-4446-2020-0948 · Cited by 19
The Memory Labyrinth: Systems, Processes, and Boundaries
Veselis, R. A. (2017) · Springer eBooks · doi:10.1007/978-3-319-47609-4_3 · Cited by 14
Centring individual animals to improve research and citation practices
Volsche, S., Root‐Gutteridge, H., Korzeniowska, A. T., & Horowitz, A. (2022) · Biological reviews/Biological reviews of the Cambridge Philosophical Society · doi:10.1111/brv.12912 · Cited by 16
All-pervading insider bias alters review time in Turkish university journals
Tütüncü, L. (2023) · Scientometrics · doi:10.1007/s11192-023-04724-3 · Cited by 14
Robinson, R. S., Brass, J. N., Shermeyer, A., & Grossman, N. (2023) · Development Policy Review · doi:10.1111/dpr.12737 · Cited by 13
Do ESG factors influence firm valuations? Evidence from the field
Bancel, F., Glavas, D., & Karolyi, G. A. (2025) · Financial Review · doi:10.1111/fire.12444 · Cited by 39
Missing Effect of Party Cues in Japan: Evidence from a Survey Experiment
Kobayashi, T., & Yokoyama, T. (2018) · Japanese Journal of Political Science · doi:10.1017/s1468109917000226 · Cited by 16
“…significant results are more likely to be published than statistically insignificant results, holding research quality fixed, the likelihood that published results reflect type I errors rather than true population parameters is increased, biasing estimated effect sizes upward (Franco et al., 2014).”
ResultLearning from Biased Research Designs
Little, A. T., & Pepinsky, T. B. (2020) · The Journal of Politics · doi:10.1086/710088 · Cited by 16
Preregistration and Registered Reports in Sociology: Strengths, Weaknesses, and Other Considerations
Manago, B. N. (2023) · The American Sociologist · doi:10.1007/s12108-023-09563-6 · Cited by 18
“Specifically, since published research is biased towards larger effect sizes, statistically significant results, and novel findings, the published literature may not provide accurate measurements of effects (Fanelli, 2010; Franco et al. 2014; Gerber and Malhotra 2008).”
BackgroundResultReviewing evidence for evidence-based policing
BOWERS, K. J., Tompson, L., Sidebottom, A., Bullock, K., & Johnson, S. D. (2017) · Routledge eBooks · doi:10.4324/9781315518299-7 · Cited by 13
The Impact of Greenways and Trails on Proximate Property Values: An Updated Review
Crompton, J. L., & Nicholls, S. (2019) · Journal of Park and Recreation Administration · doi:10.18666/jpra-2019-9906 · Cited by 13
Beken, H. V., Bruyne, E. D., & Brysbaert, M. (2020) · Quarterly Journal of Experimental Psychology · doi:10.1177/1747021820910694 · Cited by 18
“This is in line with Gablasova (2014, 2015), but not with Lee (1986). As we argued in the “Introduction,” we think the deviating finding of Lee may be due to the fact that he tested students on an easy text for which the participants arguably had much background knowledge in L1.”
BackgroundResultInfluentialToward a Science of Failure Analysis: A Narrative Review
Allen‐Platt, C., Gerstner, C. E., Boruch, R. F., & Ruby, A. (2021) · Review of Research in Education · doi:10.3102/0091732x20985074 · Cited by 15
“, significant) statistical outcomes (e.g., Bakker et al., 2012; P. Dawson & Dawson, 2016; Franco et al., 2014; Pigott et al., 2013; Rosenthal, 1979).”
BackgroundPublication bias: A “bird's-eye view” of meta-analytic practice in criminology and criminal justice
Kim, B. (2021) · Journal of Criminal Justice · doi:10.1016/j.jcrimjus.2021.101878 · Cited by 16
van Loon, N. M., Bækgaard, M., & Moynihan, D. P. (2019) · Public Administration · doi:10.1111/padm.12639 · Cited by 16
Using Split Samples to Improve Inference about Causal Effects
Fafchamps, M., & Labonne, J. (2016) · National Bureau of Economic Research · doi:10.3386/w21842 · Cited by 21
p-Hacking and Publication Bias Interact to Distort Meta-Analytic Effect Size EstimatesPreprint
Friese, M., & Frankenbach, J. (2020) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/bvm58 · Cited by 15
Fifty Shades of QE: Comparing Findings of Central Bankers and AcademicsPreprint
Fabo, B., Jančoková, M., Kempf, E., & Pástor, Ľ. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3693344 · Cited by 14
Purposeful turns for more equitable and transparent publishing in language testing and assessment
Isaacs, T., & Winke, P. M. (2024) · Language Testing · doi:10.1177/02655322231203234 · Cited by 14
“Another advantage is that authors may struggle to publish statistically non-significant results due to a well-documented phenomenon known as publication bias (Franco et al., 2014).”
BackgroundUsing Split Samples to Improve Inference on Causal Effects
Fafchamps, M., & Labonne, J. (2017) · Political Analysis · doi:10.1017/pan.2017.22 · Cited by 59
“8 Effect sizes of 0.2 standard deviation can be detected with power comfortably above 80 percent as soon 7 Franco, Malhotra and Simonovits (2014) take advantage of an NSF-sponsored program to quantify publication bias.”
BackgroundInfluentialFostering Scientific Meta-analyses with Knowledge Graphs: A Case-Study
Tiddi, I., Balliet, D. P., & ten Teije, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-49461-2_17 · Cited by 12
Positive Psychological Interventions in African Contexts: A Scoping Review
Guse, T. (2022) · Cross-cultural advancements in positive psychology · doi:10.1007/978-3-030-85924-4_16 · Cited by 17
Netra, S., Sørensen, P. F., & Nejstgaard, C. H. (2022) · Public Administration Review · doi:10.1111/puar.13553 · Cited by 19
Trninić, D., Sinha, T., & Kapur, M. (2022) · Learning and Instruction · doi:10.1016/j.learninstruc.2022.101688 · Cited by 21
How effective is community-based management of freshwater resources? A review
Karres, N., Kang, S., Aldous, A. R., Pattison‐Williams, J. K., & Masuda, Y. J. (2022) · Journal of Environmental Management · doi:10.1016/j.jenvman.2022.116161 · Cited by 21
Li, H., Liu, H., Chang, W.-Y., & Wang, C. (2023) · Agriculture · doi:10.3390/agriculture13010150 · Cited by 14
Walker, R. M., Andrews, R., George, B., & Tu, X. (2023) · Asia Pacific Journal of Public Administration · doi:10.1080/23276665.2023.2176333 · Cited by 19
“Publication bias in the scholarly community – also called the “file drawer problem” – is assumed to mean that more statistically significant research findings are being published than null results (Franco et al., 2014; Ringquist, 2013).”
BackgroundThe winner's curse and related perils of low statistical power − spelled out and illustrated
Lindstromberg, S. (2023) · Research Methods in Applied Linguistics · doi:10.1016/j.rmal.2023.100059 · Cited by 16
Lahn, L. C., & Berntsen, S. K. (2023) · Nordic Journal of Comparative and International Education (NJCIE) · doi:10.7577/njcie.5322 · Cited by 31
Mathur, M. B. (2023) · Research Synthesis Methods · doi:10.1002/jrsm.1667 · Cited by 19
“15,39,40 However, these estimates refer to individual studies rather than meta-analyses.”
BackgroundSmartphone Apps for Food Purchase Choices: Scoping Review of Designs, Opportunities, and Challenges
de Grave, R. B., Bull, C., Monteiro, D. M. S., Margariti, E., McMurchy, G., Hutchinson, J. W., & Smeddinck, J. D. (2023) · Journal of Medical Internet Research · doi:10.2196/45904 · Cited by 15
Moore, T. R., Lee, S., Freeman, R., Mahmoundi, M., Dimian, A. F., Riegelman, A., & Simacek, J. J. (2024) · Behavior Modification · doi:10.1177/01454455231218742 · Cited by 14
“…effective interventions in this review, which highlights the potential role for publishing intervention research that does not demonstrate clinically-or statistically-significant effects (Franco et al., 2014) while adding to the knowledge base for tailoring treatments to people with complex needs.”
BackgroundZhu, L., Zhou, B., Hou, J., Wang, J., & Zhou, Y. (2024) · Psychology of sport and exercise · doi:10.1016/j.psychsport.2024.102762 · Cited by 31
Spirling, A., & Stewart, B. (2024) · The Journal of Politics · doi:10.1086/734280 · Cited by 21
Tincani, M., Travers, J. C., Dowdy, A., Slocum, T. A., & Deitrich, R. (2025) · Perspectives on Behavior Science · doi:10.1007/s40614-025-00441-9 · Cited by 17
“Such Selectively reporting positive/statistically significant results while omitting negative/insignificant results (Franco et al., 2014; Rosenthal, 1978) p -hacking Selectively conducting data analyses to produce and/or enhance positive/statistically significant outcomes (Head et al., 2015)…”
Information markets and nonmarkets
Bergemann, D., & Ottaviani, M. (2021) · Handbook of Industrial Organization · doi:10.1016/bs.hesind.2021.11.008 · Cited by 24
Positivity bias in higher education research
Tight, M. (2022) · Higher Education Quarterly · doi:10.1111/hequ.12388 · Cited by 25
Does schooling promote democracy? A meta-analysis
Ahmadov, A., & Holstege, F. (2022) · Democratization · doi:10.1080/13510347.2022.2109016 · Cited by 19
Political psychophysiology: A primer for interested researchers and consumers
Settle, J. E., Hibbing, M. V., Anspach, N. M., Carlson, T. N., Coe, C. M., Hernandez, E., Peterson, J. C., Stuart, J., & Arceneaux, K. (2020) · Politics and the Life Sciences · doi:10.1017/pls.2020.5 · Cited by 30
“Although publication bias against null results exists in the social sciences (Franco et al., 2014), we encourage researchers to submit their null results, in the main text or the appendix as appropriate.”
Background
Metadata
- Authors
- Franco, A., Malhotra, N., & Simonovits, G.
- Affiliations
- Stanford University (US)
- Department of Political Science, Stanford University, Stanford, CA, USA.
- Corresponding author
- Malhotra, Neil
- ORCID
- Malhotra, Neil
- Venue
- Science
- Published
- 2014-09-19
- Volume
- 345
- Issue
- 6203
- Pages
- 1502-1505
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W1997809363
- Cited by
- 1,542
- References
- 38
- Citations by year
- 2026: 88 · 2025: 152 · 2024: 143 · 2023: 162 · 2022: 129 · 2021: 152 · 2020: 212 · 2019: 123 · 2018: 119 · 2017: 110 · 2016: 91 · 2015: 52 · 2014: 9
- FWCI
- 85.56
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- Open Research (Auckland University of Technology) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- publication bias, negative results, file drawer problem, social sciences, survey experiment, peer review
- MeSH terms
- Social Sciences, Publication Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.