Publication bias in the social sciences: Unlocking the file drawer
The DOI matches its Crossref record; 37 of the 38 references it cites were found in a registry or an index.
Abstract
The file drawer is full. Should we worry?: Experiments that produce null results face a higher barrier to publication than those that yield statistically significant differences. Whether this is a problem depends on how many null but otherwise valid results might be trapped in the file drawer. Franco et al. use a Time-sharing Experiments in the Social Sciences archive of nearly 250 peer-reviewed proposals of social science experiments conducted on nationally representative samples. They find that only 10 out of 48 null results were published, whereas 56 out of 91 studies with strongly significant results made it into a journal. Science , this issue p. 1502
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Selective reporting, questionable research practices, and low-powered research all contribute to the unreliability of findings in a range of disciplines”
From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as Franco et al., 2014
In the citing paper
“In particular, negative or non-significant results are far less likely to be published (Franco et al., 2014).”
From What do we know about interventions to improve educator wellbeing? A systematic literature review (2024) · cites this paper as Franco et al., 2014
In the citing paper
“The bias against negative results appears to be a consequence of both editors and reviewers finding them less compelling for publication and from authors themselves not trying to publish them 8-12”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [8]
In the citing paper
“Positive results -- finding that a treatment is effective or that there is a relationship between two variables -- are more likely to be published than negative results 6-12 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [8]
In the citing paper
“A possible interpretation of these results is that the missing effect sizes are mainly a file‐drawer problem (Rosenthal, 1979). This interpretation is consistent with the evidence presented in Franco et al. (2014), which indicates that the file‐drawer problem is the main culprit behind publication bias in the social sciences.”
From Targeted school‐based interventions for improving reading and mathematics for students with or at risk of academic difficulties in Grades K‐6: A systematic review (2021) · cites this paper as Franco et al. (2014)
3 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722 · cited 1×
“In the presence of publication bias smaller studies report larger effects in order to exceed arbitrary significance thresholds (19, 20).”
BackgroundThe file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
Meta-analysis in social research
(1982) · Evaluation and Program Planning · doi:10.1016/0149-7189(82)90011-8 · Cited by 3,853
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109
Publication Bias in Meta‐Analysis: Prevention, Assessment and Adjustments
(2005) · doi:10.1002/0470870168 · Cited by 2,368
The case of the misleading funnel plot
Lau, J., Ioannidis, J. P. A., Terrin, N., Schmid, C. H., & Olkin, I. (2006) · BMJ · doi:10.1136/bmj.333.7568.597 · Cited by 1,766 · cited 1×
“However, these visualization-based approaches are sensitive to using different measures of precision (21, 22) and also assume outcome variables and effect sizes are comparable across studies (23).”
Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias
Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A.-W., Cronin, E., Decullier, E., Easterbrook, P. J., Von Elm, E., Gamble, C., et al. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003081 · Cited by 1,546 · cited 1×
“A different approach involves examining the publication outcomes of a cohort of studies, either prospectively or retrospectively (24, 25).”
MethodSumming up: The Science of Reviewing Research
Wilkinson, I., Light, R. J., & Pillemer, D. B. (1987) · Journal of Educational Statistics · doi:10.2307/1164691 · Cited by 1,471
Summing Up: The Science of Reviewing Research
Light, R. J., & Pillemer, D. B. (1984) · Harvard University Press eBooks · doi:10.4159/9780674040243 · Cited by 1,427
The Existence of Publication Bias and Risk Factors for Its Occurrence
Dickersin, K. (1990) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1990.03440100097014 · Cited by 1,417 · cited 1×
“Publication bias occurs when “publication of study results is based on the direction or significance of the findings” (1).”
Background
Research works printed but not linked: 3
- 10.1037/0735-7028.17.2.136 · doi:10.1037/0735-7028.17.2.136 · Paper page
- K. Dickersin in Publication Bias in Meta-Analysis: Prevention Assessment and Adjustments H. R. Rothstein A. J. Sutton M. Borenstein Eds. (Wiley Chichester UK 2006) ch. 2.
- Dickersin K., Min Y. I., Online J. Curr. Clin. Trials 1993, 50 (1993).
Cited by
Items 901–1,000 of 1,002
Misinformation Effect in Aging: A New Light with Equivalence Testing
Tessoulin, M., Galharret, J., Gilet, A.-L., & Colombel, F. (2019) · The Journals of Gerontology Series B · doi:10.1093/geronb/gbz057 · Cited by 8
Crossman, M. K., & Herzog, H. (2019) · Elsevier eBooks · doi:10.1016/b978-0-12-815395-6.00031-6 · Cited by 5
Sealed Envelope Submissions Foster Research Integrity
Dufwenberg, M. A., & Martinsson, P. (2019) · Revue économique · doi:10.3917/reco.706.0919 · Cited by 5
Reliability of Inference: Analogs of Replication in Qualitative Research
Fairfield, T., & Charman, A. E. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.012 · Cited by 6
Lessons learned from two randomized controlled trials: CITIES and STOP-DKD
Zullig, L. L., Oakes, M. M., McCant, F., & Bosworth, H. B. (2020) · Contemporary Clinical Trials Communications · doi:10.1016/j.conctc.2020.100612 · Cited by 5
Botella, J., Suero, M., Durán, J. I., & Blazquez, D. (2021) · Anales de Psicología · doi:10.6018/analesps.433051 · Cited by 6
“…the probability that the study provides non-significant results is .607 with N = 15, and, for example, 40% of them are censored, approximately one in every four studies carried out will remain in the file drawer (a very realistic situation; see, for example, the estimates of Franco et al., 2014).”
BackgroundHuntington, C., Owen, J. J., Stanley, S. M., Knopp, K. C., & RHOADES, G. K. (2021) · Family Process · doi:10.1111/famp.12734 · Cited by 8
Gerring, J., Mahoney, J., & Elman, C. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.001 · Cited by 6
Questionable Research Practices in Clinical Psychology
O’Donohue, W., & Masuda, A. (2022) · Springer eBooks · doi:10.1007/978-3-031-04968-2_1 · Cited by 5
Measuring transparency in the social sciences: political science and international relations
Scoggins, B., & Robertson, M. P. (2024) · Royal Society Open Science · doi:10.1098/rsos.240313 · Cited by 12
Leal, D. L., Patterson, J. T., & Tafoya, J. R. (2016) · RSF The Russell Sage Foundation Journal of the Social Sciences · doi:10.7758/rsf.2016.2.3.07 · Cited by 5
Garamszegi, L. Z., & de Villemereuil, P. (2017) · Trends in Neuroscience and Education · doi:10.1016/j.tine.2017.10.001 · Cited by 6
“There is an ongoing debate regarding negative results (Knight, 2003; Fanelli, 2011; Forstmeier et al, 2016, see also negative-results.org) and publication bias in favour of significant result (Easter-brook et al, 1991; Fanelli, 2010; Franco et al, 2014).”
Pot la psicologia rescatar-se a si mateixa?. Incentius, biaix i replicabilitat
Fernando, B., César, P. L. J., & Vadillo, M. A. (2018) · Anuari de Psicologia de la Societat Valenciana de Psicologia · doi:10.7203/anuari.psicologia.18.2.231 · Cited by 5
Use of Repeated Within-Subject Measures to Assess Infants’ Preference for Similar Others
Cruz‐Khalili, A., Bettencourt, K., Kohn, C. S., Normand, M. P., & Schlinger, H. D. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02239 · Cited by 6
“…and Extension Infants as young as 5 months (Hamlin et al., 2007, 2010, 2011; Hamlin and Wynn, 2011, 2012) and older infants and toddlers (Geraci and Surian, 2011; Buon et al., 2014; Scola et al., 2015; Woo et al., 2017; Chae and Song, 2018) seem capable of socially evaluating the…”
BackgroundDoes Voting Have Upstream and Downstream Consequences? Evidence from Compulsory Voting in BrazilPreprint
Holbein, J. B., & Rangel, M. A. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2809382 · Cited by 5
Veldkamp, C. L. S. (2017) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/g8cjq · Cited by 6
Editorial: Evidential Statistics, Model Identification, and Science
Taper, M. L., Ponciano, J. M., & Toquenaga, Y. (2022) · Frontiers in Ecology and Evolution · doi:10.3389/fevo.2022.883456 · Cited by 5
Schindler, S., Hilgard, J., Fritsche, I., Burke, B., & Pfattheicher, S. (2022) · doi:10.31234/osf.io/8y6me · Cited by 5
Fifty Shades of QE: Comparing Findings of Central Bankers and Academics
Fabo, B., Jančoková, M., Kempf, E., & Pástor, Ľ. (2020) · National Bureau of Economic Research · doi:10.3386/w27849 · Cited by 7
Yang, Y., Lagisz, M., Williams, C., Pan, J., Noble, D., & Nakagawa, S. (2023) · doi:10.32942/x20g6q · Cited by 2
Fellnhofer, K., Wennberg, K. J., Allison, T. H., Arenius, P. M., Lévesque, M., Gish, J. J., & Pollack, J. M. (2026) · Entrepreneurship Theory and Practice · doi:10.1177/10422587251401175 · Cited by 6
“A foundational study by Franco et al. (2014) demonstrates that null results are systematically underreported in the social sciences, underscoring the importance of Registered Reports (and Registered Revisions) in addressing publication bias and promoting more balanced scientific reporting.”
P-hacking in clinical trials and how incentives shape the distribution of results across phases
Adda, J., Decker, C., & Ottaviani, M. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1919906117 · Cited by 67
“Our analysis builds on and develops the methods proposed in the literature that investigates “p-hacking,” publication bias, and the “file-drawer problem” (22, 23) for academic journal publications in a number of fields, ranging from life sciences (24) to psychology (25, 26), political science (27, 28), and economics (…”
Gugushvili, N., Masciantonio, A., Nitschinsk, L., Rozgonjuk, D., Täht, K., Kross, E. F., Ruiter, R. A. C., & Verduyn, P. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2531288 · Cited by 7
“First, we only included published articles in English, which may have led to the inclusion of papers affected by publication bias (Franco et al., 2014) and overrepresentation of White, educated samples from the Global North (Ghai et al., 2023).”
Toward a Philosophy of Error in Science
Allchin, D. (2025) · doi:10.1093/9780197827703.001.0001 · Cited by 10
An audit of social science survey experiments
Rauf, T., Voelkel, J. G., Druckman, J., & Freese, J. (2025) · Public Opinion Quarterly · doi:10.1093/poq/nfaf052 · Cited by 4
Correcting for publication bias in a meta-analysis with the p-uniform* method
van Aert, R. C. M., & van Assen, M. A. L. M. (2026) · Psychonomic Bulletin & Review · doi:10.3758/s13423-025-02812-4 · Cited by 134
“Franco et al. (2014) observed evidence that experiments with null or mixed results were less often published compared to strong results.”
Learning flow in educational contexts: A meta‐analysis of its antecedents and outcomes
Xie, M., Milani, A., Wang, S., Kou, T., Mao, Z., Ma, Y., & Mao, Y. (2026) · British Journal of Educational Psychology · doi:10.1111/bjep.70075 · Cited by 6
Schabram, K., Myers, C. G., & Hardin, A. E. (2024) · Organizational Research Methods · doi:10.1177/10944281241300952 · Cited by 9
“…to contemplate one of several sub-optional options including abandoning the project, thereby contributing to our fi eld ’ s fi le drawer problem (Franco et al., 2014; Rosenthal, 1979), or exercising concerning practices including re-running the study until it “ works ” (exacerbating the issue…”
Replication for Quantitative Research
Freese, J., & Peterson, D. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.010 · Cited by 4
Al‐Fayez, D., & Goodman, D. (2023) · Public Personnel Management · doi:10.1177/00910260231173082 · Cited by 7
“Indeed, there is a publication bias against null results in social sciences (Franco et al., 2014).”
BackgroundNudging accurate scientific communication
Allard, A., & Clavien, C. (2023) · PLoS ONE · doi:10.1371/journal.pone.0290225 · Cited by 6
“4% of experiments returning strong evidence in support of the stated hypotheses [10].”
BackgroundResultContexts of Convenience: Generalizing from Published Evaluations of School Finance Policies
Handel, D., & Hanushek, E. A. (2024) · Evaluation Review · doi:10.1177/0193841x241228335 · Cited by 7
“As another example, Franco et al. (2014) analyze a cohort of NSF-sponsored projects in the social sciences and fi nd that “ Strong results are 40 percentage points more likely to be published than are null results and 60 percentage points more likely to be written up.”
BackgroundMethodEstimating the extent of selective reporting: An application to economics
Bruns, S. B., Deressa, T. K., Stanley, T. D., Doucouliagos, H., & Ioannidis, J. P. A. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1711 · Cited by 15
“17 Franco et al. 29 show that non-significant findings have a substantially smaller probability of being written up by authors.”
Factors Influencing Aggression Among Adolescents in Indonesia: A Literature Review
Wangsa, P. R. P., & Tobing, D. H. (2024) · Nusantara Journal of Behavioral and Social Sciences · doi:10.47679/202448 · Cited by 4
Crediting or blaming theories or auxiliary assumptions for empirical victories or defeats
Trafimow, D., Quinton, T. S., & Weber, M. (2024) · Theory & Psychology · doi:10.1177/09593543241253548 · Cited by 4
“But publication frequencies are overbalanced, in the extreme, in the direction of publishing empirical victories (Fanelli, 2012; Franco et al., 2014; Joober et al., 2012).”
Zhai, J., & Aryadoust, V. (2024) · Metacognition and Learning · doi:10.1007/s11409-024-09392-z · Cited by 7
“…findings are selectively published over non-significant or small-effect ones, potentially leading to an overestimation of overall effect size in meta-analysis results (e.g., Borenstein et al., 2009; Dickersin, 2005; Field & Gillett, 2010; Franco et al., 2014; Marks-Anglin & Chen, 2020).”
Koers, G. J., Raaphorst, K., van der Wal, M. M., Ellen, G. J., & Meijerink, S. V. (2024) · Wiley Interdisciplinary Reviews Water · doi:10.1002/wat2.1763 · Cited by 4
“Additionally, academic research also has a success-fulness bias as failures often fail to be published (Franco, Malhotra, and Simonovits 2014).”
In Utero Shocks and Health at Birth: The Distorting Effect of Fetal Losses
Hajdu, T. (2024) · Oxford Bulletin of Economics and Statistics · doi:10.1111/obes.12649 · Cited by 6
An Inception-Cohort Study Quantifying How Many Registered Studies Are Publicly Shared
Ensinck, E. N. F., & Lakens, D. (2025) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459241296031 · Cited by 15
“…to quantify how many studies remain unpublished, the existence of the proverbial file drawer (Rosenthal, 1979) has been demonstrated in disciplines such as psychology, sociology, economics, and medicine (Dickersin et al., 1992; Franco et al., 2014; Gerber et al., 2010; Scheel et al., 2021).”
InfluentialProminence and exclusivity: Identity and opposition to immigration in a multinational context
Creighton, M. J. (2015) · Journal of Ethnic and Migration Studies · doi:10.1080/1369183x.2015.1063992 · Cited by 5
“Franco, Malhotra, and Simonovits (2014) find that in a sample of social science experiments, null findings are 60% less likely to even appear in written form.”
BackgroundResultReproducibility from the Perspective of Meta-Analysis
Ehm, W. (2016) · John Wiley & Sons, Inc. eBooks · doi:10.1002/9781118865064.ch7 · Cited by 6
At the Crossroads: Lessons and Challenges in Computational Social Science
Borge-Holthoefer, J., Moreno, Y., & Yasseri, T. (2016) · Frontiers research topics · doi:10.3389/978-2-88945-021-3 · Cited by 4
A Text Analytic Approach to Classifying Document Types
Walczak, S. (2017) · The Journal of Writing Analytics · doi:10.37514/jwa-j.2017.1.1.06 · Cited by 5
Transparency and Reproducibility: Conceptualizing the Problem
Christensen, G., & Miguel, E. A. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.006 · Cited by 4
Measurement Replication in Qualitative and Quantitative Studies
Reiter, D. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.011 · Cited by 4
Ayorinde, A., Williams, I., Mannion, R., Song, F., Skrybant, M., Lilford, R. J., & Chen, Y. (2020) · Health Services and Delivery Research · doi:10.3310/hsdr08330 · Cited by 6
Shields, A. N., Brandes, C. M., Reardon, K. W., España, R. A., & Tackett, J. L. (2021) · Adaptive Human Behavior and Physiology · doi:10.1007/s40750-021-00167-3 · Cited by 4
Situating Linguistics in the Social Science Data Movement
Gawne, L., & Styles, S. J. (2022) · The MIT Press eBooks · doi:10.7551/mitpress/12200.003.0006 · Cited by 11
What’s in Your File Drawer? The Case of the Missing Null in Criminology and Criminal Justice
Crow, M. S., Smykla, J. O., O’Brien, H., Cerna, T., Johnson, A., Pisaris, S., Suarez, M., & Wilder, J. (2022) · Crime & Delinquency · doi:10.1177/00111287221090959 · Cited by 4
Transparency and reproducibility in the journal of contextual behavioral science: An audit study
Lear, M. K., Spata, A., Tittler, M. V., Fishbein, J. N., Arch, J. J., & Luoma, J. B. (2023) · Journal of Contextual Behavioral Science · doi:10.1016/j.jcbs.2023.03.017 · Cited by 7
Chandrashekar, S. P., Adelina, N., Zeng, S., Chiu, Y. Y. E., Leung, G. Y. S., Henne, P., Cheng, B., & Feldman, G. (2023) · Meta-Psychology · doi:10.15626/mp.2022.3108 · Cited by 5
Hello, You’ve been hacked: a study of victim notification preferences
Muniz, C. N., Fisher, T., Smith, K. V., Ali, R., Howell, C. J., & Maimon, D. (2024) · Journal of Crime and Justice · doi:10.1080/0735648x.2024.2340554 · Cited by 5
“…that results from an incomplete evidence-base that is unrepresentative of all results of research in that area (Banks et al. 2012; Crow et al. 2022; Franco et al. 2014). to gather threat intelligence, which is then used to test social scientific theory, build profiles of active cyber-offenders,…”
Lavrenteva, E., & Orland‐Barak, L. (2024) · Frontiers in Education · doi:10.3389/feduc.2024.1234871 · Cited by 6
Social media and electoral prediction: ten years after
Gayo-Avello, D., Jungherr, A. M., & Rivero, G. (2025) · Edward Elgar Publishing eBooks · doi:10.4337/9781802207309.00040 · Cited by 3
In the Spirit of What Might Be Lost: Troubling the Boundaries of Good Fit
Korcuska, J. S. (2016) · Counselor Education and Supervision · doi:10.1002/ceas.12042 · Cited by 4
Kato, T. (2018) · The Japanese Journal of Personality · doi:10.2132/personality.27.2.11 · Cited by 6
Dangerous Science: Science Policy and Risk Analysis for Scientists and Engineers
Rozell, D. J. (2020) · Ubiquity Press eBooks · doi:10.5334/bci · Cited by 5
Watt, J. A., Campitelli, M. A., Maxwell, C. J., Guan, J., Maclagan, L. C., Gomes, T., Bokhari, M., Straus, S. E., & Bronskill, S. E. (2020) · Journal of the American Geriatrics Society · doi:10.1111/jgs.16710 · Cited by 5
Literatur am Bildschirm: Zum Stand der empirischen Leseforschung
Schwabe, A., Brandl, L., Boomgaarden, H. G., & Stocker, G. (2020) · Orbis Litterarum · doi:10.1111/oli.12271 · Cited by 8
Akhtar, P. (2017) · Palgrave Macmillan UK eBooks · doi:10.1057/978-1-137-59099-2_4 · Cited by 4
Theories of Public Opinion Change Versus Stability and their Implications for Null Findings
Wozniak, K. H., Drakulich, K., & Calfano, B. (2020) · Frontiers in sociology and social research · doi:10.1007/978-3-030-47256-6_13 · Cited by 5
Disgust and political attitudes
Petersen, M. B., Tybur, J. M., & Stewart, P. A. (2020) · Politics and the Life Sciences · doi:10.1017/pls.2020.23 · Cited by 4
Wright, L., Head, J. A., & Jivraj, S. (2021) · Occupational and Environmental Medicine · doi:10.1136/oemed-2021-107473 · Cited by 5
Anvari, F., & Marchiori, D. (2021) · Royal Society Open Science · doi:10.1098/rsos.201944 · Cited by 4
Troublesome Dependency Modeling: Causality, Inference, Statistical LearningPreprint
Mandel, I. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2984045 · Cited by 5
Tree-based Synthetic Control Methods: Consequences of moving the US EmbassyPreprint
Mühlbach, N. S., & Nielsen, M. S. (2019) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.1909.03968 · Cited by 8
“Running several regressions may lead to publication bias (Franco, Malhotra, & Simonovits, 2014).”
BackgroundSamota, E. K., & Davey, R. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/581033 · Cited by 4
Estimating the Reproducibility of Experimental PhilosophyPreprint
Cova, F., & Abatista, A. (2024) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/hr4zs · Cited by 7
Princípios de Hong Kong para a Avaliação de Pesquisadores: Promovendo a Integridade em Pesquisa
Moher, D., Bouter, L. M., Kleinert, S., Glasziou, P., Sham, M. H., Barbour, V., Coriat, A., Foeger, N., & Dirnagl, U. (2020) · doi:10.21452/abec.2021.abec.001 · Cited by 5
“Essa revisão identificou que quase três quartos dos pacientes incluídos estavam em estudos que não tinham sido publicados [22]; há outros exemplos em diferentes áreas do conhecimento [23,24].”
BackgroundBrekelmans, G., Evans, B. G., & Wonnacott, E. (2020) · doi:10.31234/osf.io/63dhn · Cited by 4
Fifty Shades of QE: Comparing Findings of Central Bankers and AcademicsPreprint
Fabo, B., Jančoková, M., Kempf, E., & Pástor, Ľ. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3779630 · Cited by 17
Pownall, M., Pennington, C. R., Norris, E., Juanchich, M., Smailes, D., Russell, P. S., Gooch, D., Evans, T. R., Persson, S., Mak, M. H., et al. (2023) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/xg2ah · Cited by 3
Estimating the extent of inflated significance in economicsPreprint
Bruns, S. B., Deressa, T. K., Stanley, T. D., Doucouliagos, H., & Ioannidis, J. P. A. (2022) · doi:10.31222/osf.io/h29xn · Cited by 4
An Inception Cohort Study Quantifying How Many Registered Studies are PublishedPreprint
Ensinck, E. N. F., & Lakens, D. (2023) · doi:10.31234/osf.io/5hkjz · Cited by 4
Survey Experiments in Public Policy: A Systematic Literature Review
Kovács, R., & Veselý, A. (2025) · Review of Policy Research · doi:10.1111/ropr.70040 · Cited by 5
Liang, Y., Zhang, Q., Ran, G., & Niu, X. (2025) · The Journal of Genetic Psychology · doi:10.1080/00221325.2025.2532439 · Cited by 4
Dapica, P., Santiago, J., & Solana, P. (2025) · Cognitive Science · doi:10.1111/cogs.70127 · Cited by 4
Predicting replication outcomes and generalizability of results
Dreber, A. (2025) · European Review of Agricultural Economics · doi:10.1093/erae/jbaf035 · Cited by 3
Liu, G. H. W., Chua, C. E. H., Lee, N. C.-A., & Wu, J. H.-J. (2025) · Information & Management · doi:10.1016/j.im.2025.104271 · Cited by 4
The adaptive significance of polyandry: a meta-analysis
Yan, J. L., Dobbin, M. L., Chen, J. M., & Dukas, R. (2026) · Evolution · doi:10.1093/evolut/qpag013 · Cited by 2
Baelen, R. N., Lovett, J. M., Bourke, K. T., Galloway, C., Parker, A., Baghdady, A., & Schonert-Reichl, K. A. (2026) · Review of Educational Research · doi:10.3102/00346543251413248 · Cited by 4
Looking at the bigger picture: on the relevance of null effects of model differences across time
Ehrlich, J. S., Kreienbaum, C., & Ringle, C. M. (2026) · Journal of Marketing Analytics · doi:10.1057/s41270-026-00497-9 · Cited by 4
Predicting the impact of American Economic Review articles by author characteristics
Yuret, T. (2022) · Quantitative Science Studies · doi:10.1162/qss_a_00180 · Cited by 4
“Franco, Malhotra, and Simonovits (2014) analyze all the projects in a prestigious sponsored program in social sciences, and they show that if the results are not significant and positive, the results are not likely to be written up, and the papers are not likely to be published if written.”
BackgroundThe Challenges of Identifying Significant Epistemic Failure in Science
Lehmann, T., Borggräfe, M., & Gläser, J. (2022) · Brill | mentis eBooks · doi:10.30965/9783969752487_012 · Cited by 4
The rocky road to translational science: An analysis of Clinical and Translational Science Awards
Kim, Y. H., Lee, Y.-N., & Woo, S. (2023) · Research Evaluation · doi:10.1093/reseval/rvac048 · Cited by 3
Signal Smoothing and Syntactic Choices: A Critical Reflection on the UID Hypothesis
Juzek, T. S. (2024) · Open Mind · doi:10.1162/opmi_a_00125 · Cited by 4
“Franco et al. (2014) estimated that in the social sciences, only one in five negative results get published, while three out of five positive results get published.”
BackgroundSlow Science But Fast Forward: The Political Economy of Rock Art Research in A Globalized World
Conkey, M. W. (2024) · Interdisciplinary Contributions to Archaeology/Interdisciplinary contributions to archaeology · doi:10.1007/978-3-031-54638-9_21 · Cited by 6
Berens, S., & Brady, D. (2024) · Sociology of Development · doi:10.1525/sod.2023.0018 · Cited by 4
International remittances and productivity in Sub‐Saharan Africa
Asongu, S. A., & Nnanna, J. (2024) · World Affairs · doi:10.1002/waf2.12036 · Cited by 4
Open science needs a standardized data format: Suggestions for the field of psychoneuroendocrinology
Meier, M., Vinkers, C. H., Pruessner, J. C., & Sep, M. S. C. (2024) · Psychoneuroendocrinology · doi:10.1016/j.psyneuen.2024.107170 · Cited by 3
Komatsu, T., Yamanaka, S., Oshima, H., & Hayashi, K. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2492806 · Cited by 3
“Such habits are widely recognized by researchers and in academic research areas; cf., the desk drawer effect and publication bias (Franco et al., 2014; Rosenthal, 1979).”
How bad gatekeepers undermine good science
Brooks, P. (2025) · Synthese · doi:10.1007/s11229-025-05346-3 · Cited by 5
“As a result of this, “significant” findings are overrepresented while null results are often consigned to the file drawer, especially among early career researchers (Ioannidis, 2005; Fanelli, 2010; Franco et al., 2014 ).”
Cognitive reflection, 2D:4D and social value orientation
Millet, K., & Aydinli, A. (2019) · PLoS ONE · doi:10.1371/journal.pone.0212767 · Cited by 3
Hill, C. R., Samendinger, S., & Rymal, A. M. (2020) · Annals of Behavioral Medicine · doi:10.1093/abm/kaaa080 · Cited by 4
Predictors of dissertation publication in clinical and counseling psychology.
Herbert, R. S., Evans, S. C., Guler, J., & Roberts, M. C. (2021) · Training and Education in Professional Psychology · doi:10.1037/tep0000371 · Cited by 3
“Dissertation nonpublica-tion contributes to the “ fi le drawer problem, ” where studies yielding nonsigni fi cant results are less likely to be published (Franco et al., 2014).”
Targeted Learning for Pre-Analysis Plans in Public Health and Health Policy Research
Rose, S. (2015) · Observational Studies · doi:10.1353/obs.2015.0008 · Cited by 3
Bias and Careers: Evidence from the Aid Effectiveness Literature
Doucouliagos, H., Hinz, T., & Zigová, K. (2020) · European Journal of Political Economy · doi:10.1016/j.ejpoleco.2021.102056 · Cited by 5
Chen, C., MacInnis, B., Waltman, M., & Krosnick, J. A. (2021) · Climatic Change · doi:10.1007/s10584-021-03194-x · Cited by 6
Marital Status, Gender, and Race in The U.S.: Perceptions of Middle-Aged Men and Women
Munsch, C. L., & Davis, S. M. (2021) · Journal of Comparative Family Studies · doi:10.3138/jcfs.52.4.04 · Cited by 4
“Null findings are often relegated to the file drawer; significant results are more likely to be written up and published than null results (Franco, Malhotra, and Simonovits, 2014).”
Lotsberg, M. L., & Peters, S. D. (2022) · Human perspectives in health sciences and technology · doi:10.1007/978-3-030-92612-0_10 · Cited by 4
Metadata
- Authors
- Franco, A., Malhotra, N., & Simonovits, G.
- Affiliations
- Stanford University (US)
- Department of Political Science, Stanford University, Stanford, CA, USA.
- Corresponding author
- Malhotra, Neil
- ORCID
- Malhotra, Neil
- Venue
- Science
- Published
- 2014-09-19
- Volume
- 345
- Issue
- 6203
- Pages
- 1502-1505
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W1997809363
- Cited by
- 1,542
- References
- 38
- Citations by year
- 2026: 88 · 2025: 152 · 2024: 143 · 2023: 162 · 2022: 129 · 2021: 152 · 2020: 212 · 2019: 123 · 2018: 119 · 2017: 110 · 2016: 91 · 2015: 52 · 2014: 9
- FWCI
- 85.56
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- Open Research (Auckland University of Technology) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- publication bias, negative results, file drawer problem, social sciences, survey experiment, peer review
- MeSH terms
- Social Sciences, Publication Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.