Publication bias in the social sciences: Unlocking the file drawer
The DOI matches its Crossref record; 37 of the 38 references it cites were found in a registry or an index.
Abstract
The file drawer is full. Should we worry?: Experiments that produce null results face a higher barrier to publication than those that yield statistically significant differences. Whether this is a problem depends on how many null but otherwise valid results might be trapped in the file drawer. Franco et al. use a Time-sharing Experiments in the Social Sciences archive of nearly 250 peer-reviewed proposals of social science experiments conducted on nationally representative samples. They find that only 10 out of 48 null results were published, whereas 56 out of 91 studies with strongly significant results made it into a journal. Science , this issue p. 1502
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Selective reporting, questionable research practices, and low-powered research all contribute to the unreliability of findings in a range of disciplines”
From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as Franco et al., 2014
In the citing paper
“In particular, negative or non-significant results are far less likely to be published (Franco et al., 2014).”
From What do we know about interventions to improve educator wellbeing? A systematic literature review (2024) · cites this paper as Franco et al., 2014
In the citing paper
“The bias against negative results appears to be a consequence of both editors and reviewers finding them less compelling for publication and from authors themselves not trying to publish them 8-12”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [8]
In the citing paper
“Positive results -- finding that a treatment is effective or that there is a relationship between two variables -- are more likely to be published than negative results 6-12 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [8]
In the citing paper
“A possible interpretation of these results is that the missing effect sizes are mainly a file‐drawer problem (Rosenthal, 1979). This interpretation is consistent with the evidence presented in Franco et al. (2014), which indicates that the file‐drawer problem is the main culprit behind publication bias in the social sciences.”
From Targeted school‐based interventions for improving reading and mathematics for students with or at risk of academic difficulties in Grades K‐6: A systematic review (2021) · cites this paper as Franco et al. (2014)
3 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722 · cited 1×
“In the presence of publication bias smaller studies report larger effects in order to exceed arbitrary significance thresholds (19, 20).”
BackgroundThe file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
Meta-analysis in social research
(1982) · Evaluation and Program Planning · doi:10.1016/0149-7189(82)90011-8 · Cited by 3,853
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109
Publication Bias in Meta‐Analysis: Prevention, Assessment and Adjustments
(2005) · doi:10.1002/0470870168 · Cited by 2,368
The case of the misleading funnel plot
Lau, J., Ioannidis, J. P. A., Terrin, N., Schmid, C. H., & Olkin, I. (2006) · BMJ · doi:10.1136/bmj.333.7568.597 · Cited by 1,766 · cited 1×
“However, these visualization-based approaches are sensitive to using different measures of precision (21, 22) and also assume outcome variables and effect sizes are comparable across studies (23).”
Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias
Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A.-W., Cronin, E., Decullier, E., Easterbrook, P. J., Von Elm, E., Gamble, C., et al. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003081 · Cited by 1,546 · cited 1×
“A different approach involves examining the publication outcomes of a cohort of studies, either prospectively or retrospectively (24, 25).”
MethodSumming up: The Science of Reviewing Research
Wilkinson, I., Light, R. J., & Pillemer, D. B. (1987) · Journal of Educational Statistics · doi:10.2307/1164691 · Cited by 1,471
Summing Up: The Science of Reviewing Research
Light, R. J., & Pillemer, D. B. (1984) · Harvard University Press eBooks · doi:10.4159/9780674040243 · Cited by 1,427
The Existence of Publication Bias and Risk Factors for Its Occurrence
Dickersin, K. (1990) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1990.03440100097014 · Cited by 1,417 · cited 1×
“Publication bias occurs when “publication of study results is based on the direction or significance of the findings” (1).”
Background
Research works printed but not linked: 3
- 10.1037/0735-7028.17.2.136 · doi:10.1037/0735-7028.17.2.136 · Paper page
- K. Dickersin in Publication Bias in Meta-Analysis: Prevention Assessment and Adjustments H. R. Rothstein A. J. Sutton M. Borenstein Eds. (Wiley Chichester UK 2006) ch. 2.
- Dickersin K., Min Y. I., Online J. Curr. Clin. Trials 1993, 50 (1993).
Cited by
Items 801–900 of 1,002
No Strong Evidence of Stereotype Threat in Females: A Reassessment of the Meta-Analysis
Warne, R. T. (2021) · Journal of Advanced Academics · doi:10.1177/1932202x211061517 · Cited by 11
“Treating the methodological quality and p-values as being related is a major contributor to publication bias (Dickersin & Min, 1993; Franco et al., 2014).”
BackgroundLahiri, S., Bingenheimer, J. B., Evans, W. D., Wang, Y., Dubey, P., & Snowden, B. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph182212186 · Cited by 9
“Studies are more likely to be written up and published if they contain statistically significant results than if they do not contain statistically significant results [62].”
Backgroundvan Aert, R. C. M., & Niemeyer, H. (2022) · Springer eBooks · doi:10.1007/978-3-031-04968-2_10 · Cited by 12
Governance quality and trade performance in Sub‐Saharan Africa
Asongu, S. A., & Odhiambo, N. M. (2024) · World Affairs · doi:10.1002/waf2.12003 · Cited by 9
Nudging transparent behavioural science and policy
Maynard, O., & Munafo, M. R. (2018) · Behavioural Public Policy · doi:10.1017/bpp.2018.10 · Cited by 9
“The reasons for publication bias have been described elsewhere (Rosenthal, 1979; Joober et al., 2012; Franco et al., 2014; Ioannidis et al., 2014).”
BackgroundLöckenhoff, C. E., Rutt, J. L., Samanez-Larkin, G. R., Gallagher, C., O’Donoghue, T., & Reyna, V. F. (2018) · The Journals of Gerontology Series B · doi:10.1093/geronb/gby090 · Cited by 9
Observational Data for Heterogeneous Treatment Effects with Application to Recommender Systems
Lada, A., Peysakhovich, A., Aparicio, D., & Bailey, M. (2019) · ACM Conference on Economics and Computation (EC) · doi:10.1145/3328526.3329558 · Cited by 19
“…be specified before launching a controlled testing evaluation, e.g. time-stamping a pre-analysis plan, to avoid concerns that researchers might engage in p -hacking, or data snooping, that is to iteratively search across features or subgroups and present those that are significant [18, 23, 34, 42].”
BackgroundKogut, A., Foster, M., Ramírez, D., & Xiao, D. Y. (2019) · College & Research Libraries · doi:10.5860/crl.80.7.973 · Cited by 12
On Minimizing the Risk of Bias in Randomized Controlled Trials in Economics
Eble, A., Boone, P., & Elbourne, D. R. (2016) · The World Bank Economic Review · doi:10.1093/wber/lhw034 · Cited by 11
“Franco et al. (2014) find that significant results were 40 percentage points more likely to be published than null results among a set of 221 National Science Foundation-funded studies in the social sciences spanning political science, economics, and psychology.”
Fifty Shades of QE: Conflicts of Interest in Economic ResearchPreprint
Fabo, B., Jančoková, M., Kempf, E., & Pástor, Ľ. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3693174 · Cited by 17
Moral judgement and decision-making: theoretical predictions and null results
Hertz, U., Jia, F., & Francis, K. (2023) · Scientific Reports · doi:10.1038/s41598-023-34899-x · Cited by 8
The association between childhood maltreatment and internet addiction: A three-level meta-analysis
Xianxin, M., Chen, Y., Delin, Y., Shuling, G., & Fu, X. (2024) · Advances in Psychological Science · doi:10.3724/sp.j.1042.2024.01087 · Cited by 10
Interstate environmental peacebuilding: A systematic review
Balinskaia, A. (2024) · Environment and Security · doi:10.1177/27538796241287366 · Cited by 10
“While the study acknowledges the potential impact of publication bias (Franco et al., 2014), a deliberate decision was made to set a lower threshold for the reviewed articles, intending to offer an overview of the white literature specifically.”
The Effects of Climate Change on Mesocarnivores: A Global Review and Meta‐Analysis
Valdez, V., Ferreras, P., & Rosalino, L. M. (2025) · Global Change Biology · doi:10.1111/gcb.70302 · Cited by 10
“Studies with null results—those that show no predicted effect of climate change on distribution—are of high conservation importance but may be underrepresented in the literature (Franco et al. 2014).”
Meng, X., Chen, Y., Rudasill, K. M., Xu, L., Yu, D., & Harrell, D. R. (2025) · Journal of Youth and Adolescence · doi:10.1007/s10964-025-02212-w · Cited by 9
“So meta-analyses including the results of those studies may result in higher meta-analytic effect values than true values (Franco et al., 2014; Kuppens et al., 2013).”
Wang, X., Li, L., Wang, Q., Zhong, B., & Xu, Y. (2025) · Educational Research Review · doi:10.1016/j.edurev.2025.100702 · Cited by 10
Can augmented reality technology reduce learners' cognitive load? A meta-analysis
Zhu, X., Peng, K., Yu, S., & Wang, G. P. (2026) · Smart Learning Environments · doi:10.1186/s40561-025-00429-7 · Cited by 21
“Publication bias refers to the fact that statistically significant findings are more likely to be published (Franco et al., 2014).”
Strengths and Weaknesses of Meta-Analyses
Corker, K. S. (2022) · Oxford University Press eBooks · doi:10.1093/oso/9780190938550.003.0006 · Cited by 18
Energy-Saving Occupant-Feedback Control Method Under Preferred Air-Conditioner Settings of Occupants
Yano, T., & Sako, M. (2024) · IEEE Access · doi:10.1109/access.2024.3367954 · Cited by 7
“These difficulties prevent researchers from reporting their results and thus obstruct publication [50].”
BackgroundReducing Political Bias in Political Science Estimates
Zigerell, L. J. (2017) · PS Political Science & Politics · doi:10.1017/s1049096516002389 · Cited by 7
Blacklist or Short List: Do Employers Discriminate against Union Supporter Job Applicants?
Kreisberg, N., & Wilmers, N. (2021) · Industrial and Labor Relations Review · doi:10.1177/00197939211036444 · Cited by 23
“Null results are difficult to interpret, because they may result from an underpowered experiment, and researchers frequently do not submit them for publication (Franco, Malhotra, and Simonovits 2014).”
BackgroundToward Open Science: Contributing to Research Culture Change
Shaw, L. C., Errington, T. M., & Mellor, D. T. (2022) · Science Editor · doi:10.36591/se-d-4501-14 · Cited by 12
Brückner, T., Wieschowski, S., Heider, M., Deutsch, S., Drude, N., Toelch, U., Bleich, A., Tolba, R. H., & Strech, D. (2022) · PLoS ONE · doi:10.1371/journal.pone.0271976 · Cited by 8
Walker‐Bolton, A. D. (2022) · American Journal of Primatology · doi:10.1002/ajp.23459 · Cited by 8
“Many researchers have stated that it is of primary importance to publish not only our conservation successes but also our failures (Ardoin et al., 2020; Catalano et al., 2018, 2019; Franco et al., 2014; Polanin et al., 2016; Wood, 2020).”
BackgroundBehavioural Insights and Public Policy in Canada
Hopkins, V. C., & Lawlor, A. (2023) · Canadian Journal of Political Science · doi:10.1017/s0008423923000100 · Cited by 9
“Consistent with other fields (see, for example, Franco et al., 2014; Esarey and Wu, 2016), meta-analysis shows evidence of publication bias in peer-reviewed behavioural science (Hummel and Maedche, 2019).”
ResultCritical Values Robust to P-Hacking
McCloskey, A., & Michaillat, P. (2024) · The Review of Economics and Statistics · doi:10.1162/rest_a_01456 · Cited by 17
“In the next step, we apply the correction discussed in the literature.”
BackgroundMethodInfluentialExperimental Investigations of Judicial Decision-Making
Holste, L., & Spamann, H. (2025) · Cambridge University Press eBooks · doi:10.1017/9781009170901.022 · Cited by 10
Regulatory Forum Opinion Piece*: The Value of Publishing Negative Scientific Study Data
Boorman, G. A., Foster, J. R., Laast, V. A., & Francke, S. (2015) · Toxicologic Pathology · doi:10.1177/0192623315595884 · Cited by 10
“The National Science Foundation funded 221 similar time-sharing experiments in social science (Franco, Malhotra, and Simonovitz 2014).”
BackgroundMethodInfluentialSitaram, R., Yu, T., Halsband, U., Vogel, D., Müller, F., Lang, S., Birbaumer, N., & Kotchoubey, B. (2018) · Brain and Cognition · doi:10.1016/j.bandc.2018.11.007 · Cited by 7
Campaigns and the Selection of Policy‐Seeking Representatives
Bowler, S., McElroy, G., & Müller, S. (2019) · Legislative Studies Quarterly · doi:10.1111/lsq.12260 · Cited by 7
“…it is, of course, possible to read too much into weak findings. despite the interest within several literatures on the value of nonresults (e.g., Findley et al. 2016; Franco, Malhotra, and simonovits 2014), there is always the suspicion that weak results could be made stronger by better modeling.”
BackgroundReteig, L. C., Newman, L. A., Ridderinkhof, K. R., & Slagter, H. A. (2022) · PLoS ONE · doi:10.1371/journal.pone.0262718 · Cited by 8
Now Is the Time: How to Increase the Value of Social Science
Lupia, A. (2017) · Social research · doi:10.1353/sor.2017.0041 · Cited by 7
Keeping Researchers Honest: The Case for Sealed-Envelope-SubmissionsPreprint
Dufwenberg, M. A., & Martinsson, P. (2014) · RePEc: Research Papers in Economics · Cited by 8
Swanson, E. P., & Young, G. M. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2823067 · Cited by 5
Social Security and Saving: An Update
SLAVOV, S. N., Gorry, D., Gorry, A., & Caliendo, F. (2017) · National Bureau of Economic Research · doi:10.3386/w23506 · Cited by 7
Masouleh, S. K., Eickhoff, S. B., & Genon, S. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.01.10.901181 · Cited by 7
“, 2009), psychological (Franco et al., 2014; Lindsay, 2015; Open Science Collaboration, 2015), biomedical and neuroimaging-based neuroscience (Button et al.”
BackgroundInfluentialStatistics education in undergraduate psychology: A survey of UK curriculaPreprint
TARG Meta-Research Group (2020) · doi:10.31234/osf.io/jv8x3 · Cited by 7
Herbst, T. D., & Hausberg, J. P. (2023) · International Journal of Innovation and Technology Management · doi:10.1142/s0219877024500184 · Cited by 9
Beyond Predictive Algorithms in Child Welfare
Moon, E. S.-Y., Saxena, D., Maharaj, T., & Guha, S. (2024) · Graphics Interface · doi:10.1145/3670947.3670976 · Cited by 11
“The random predictor predicts ‘R’ outcomes 80% of the time and ‘NR’ 20% of the time. negative results [47, 94, 114] to evaluate and audit the impact of algorithmic systems through multiple methodological lenses.”
BackgroundLyons, M. A., & Delgadillo, J. (2024) · British Journal of Clinical Psychology · doi:10.1111/bjc.12513 · Cited by 10
“Therefore, the reviewed literature may be subject to publication bias, given that significant findings are more likely to be published than non-significant findings (Franco et al., 2014).”
Jerke, J., Velicu, A., Winter, F., & Rauhut, H. (2025) · PLoS ONE · doi:10.1371/journal.pone.0305666 · Cited by 8
Pooled perspectives: Critical considerations for the modern meta-analysis
Justo, M., Askin, H., Mehta, D., & Benharash, P. (2025) · Surgery Open Science · doi:10.1016/j.sopen.2025.03.005 · Cited by 9
“This selective publication process raises the probability that reported findings represent type I errors rather than true population parameters, leading to inflated effect size [7].”
Milojević, S. (2025) · Scientometrics · doi:10.1007/s11192-025-05322-1 · Cited by 12
“However, “a large share of failed research remains unpublished” (Franzoni & Stephan, 2021, p. 26), a phenomenon known as “the file drawer problem” (Franco et al., 2014; Rosenthal, 1979).”
Wei, C., & Chen, L. (2025) · Journal of Intelligence · doi:10.3390/jintelligence13100130 · Cited by 17
“Publication bias refers to the phenomenon that studies reporting statistically significant findings are more likely to be published (Franco et al. 2014).”
Xia, Y. Q., He, S., Sun, Z., & Wang, F. (2025) · Journal of Affective Disorders · doi:10.1016/j.jad.2025.120481 · Cited by 10
Publication bias and low power in field studies on goal priming
Shanks, D. R., & Vadillo, M. A. (2021) · Royal Society Open Science · doi:10.1098/rsos.210544 · Cited by 7
Make abandoned research publicly available
Lakens, D., & Ensinck, E. N. F. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-01829-4 · Cited by 5
The association between trait mindfulness and sleep problems: A three-level meta-analysis
Lu, L., & Ran, G. (2024) · Journal of Health Psychology · doi:10.1177/13591053241253483 · Cited by 7
“The emergence of publication bias indicated the fact that the included studies were not well representative of the field (Franco et al., 2014).”
Quintero, M. J., López, F. J., Vadillo, M. A., & Morís, J. (2024) · Cognitive Therapy and Research · doi:10.1007/s10608-024-10497-7 · Cited by 7
“Lastly, we only included published articles (see Eligibility criteria ), but there may be laboratory and clinical studies that have not been published yet due to publication bias (Dwan et al., 2013; Franco et al., 2014), and, therefore, were not included in this review.”
Origin, destination, or mobility? A systematic review of studies using diagonal reference models
Shi, S., & Gugushvili, A. (2025) · Research in Social Stratification and Mobility · doi:10.1016/j.rssm.2025.101047 · Cited by 7
THE (NON‐)SIGNIFICANCE OF REPORTING ERRORS IN ECONOMICS: EVIDENCE FROM THREE TOP JOURNALS
Pütz, P., & Bruns, S. B. (2020) · Journal of Economic Surveys · doi:10.1111/joes.12397 · Cited by 7
Shi, F. (2021) · Education and Information Technologies · doi:10.1007/s10639-021-10444-z · Cited by 12
Changing Pro-Environmental Behavior: Evidence from (Un)Successful Intervention Studies
Lange, F., & Brick, C. (2021) · Sustainability · doi:10.3390/su13147748 · Cited by 11
Roth, A. M., & Singhal, V. R. (2022) · Production and Operations Management · doi:10.1111/poms.13847 · Cited by 9
“Franco et al. (2014, p. 1502) describe the adverse impacts of publication bias in social sciences and report the following: “[W]e provide direct evidence of publication bias and identify the stage of research production at which publication bias occurs: Authors do not write up and submit null…”
BackgroundSay where you sample: Increasing site selection transparency in urban ecology
Dyson, K., Dawwas, E. B., Poulton‐Kamakura, R., Alberti, M., & Fuentes, T. L. (2023) · Ecosphere · doi:10.1002/ecs2.4466 · Cited by 9
Segerstrom, S. C., Diefenbach, M. A., Hamilton, K., O'Connor, D. B., Tomiyama, A. J., Bacon, S., Bennett, G. G., Brondolo, E. N., Czajkowski, S. M., Davidson, K. W., et al. (2023) · Annals of Behavioral Medicine · doi:10.1093/abm/kaac044 · Cited by 14
Segerstrom, S. C., Diefenbach, M. A., Hamilton, K., O'Connor, D. B., & Tomiyama, A. J. (2023) · Health Psychology · doi:10.1037/hea0001236 · Cited by 7
Fowler, C., Chu, A. W., Guo, N., Ansari, J. R., Shafer, S. L., & Flood, P. D. (2023) · Anesthesia & Analgesia · doi:10.1213/ane.0000000000006429 · Cited by 7
Hu, H.-F., Moody, G. D., & Galletta, D. F. (2023) · Communications of the Association for Information Systems · doi:10.17705/1cais.05241 · Cited by 9
How Cognitive Biases Influence Problematic Research Methods Practices
Andrieux, P., Leonard, S. R., Simmering, V. R., Simmering, M. J., & Fuller, C. M. (2024) · The Electronic Journal of Business Research Methods · doi:10.34190/ejbrm.22.1.3212 · Cited by 8
“QRPs have been commonly employed to overcome publication bias, which occurs when the probability that a result is reported depends on statistical significance (Franco, Malhotra, and Simonovits, 2014; Simonsohn, Simmons, and Nelson, 2015; Sutton, 2009).”
Accounting and Public Policy: The Importance of Credible Research
Denison, C. A., Ravenscroft, S., & Williams, P. F. (2014) · Accounting and the Public Interest · doi:10.2308/apin-51158 · Cited by 7
Anglin, S. M., & Edlund, J. E. (2019) · Psychology Learning & Teaching · doi:10.1177/1475725719859453 · Cited by 5
Hwang, T. (2017) · Cambridge University Press eBooks · doi:10.1017/9781108890960.012 · Cited by 13
Using Rational Models to Interpret the Results of Experiments on Accent Adaptation
Tan, M., Xie, X., & Jaeger, T. F. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.676271 · Cited by 5
The Limitations of Social Science as the Arbiter of Blame: An Argument for Abandoning Retribution
Tullett, A. M. (2022) · Perspectives on Psychological Science · doi:10.1177/17456916211033284 · Cited by 8
Fanelli, D. (2022) · Oxford University Press eBooks · doi:10.1093/oso/9780190938550.003.0004 · Cited by 7
Do minimum wages crowd out union density?
Kozák, M., Picot, G., & Starke, P. (2024) · British Journal of Industrial Relations · doi:10.1111/bjir.12809 · Cited by 8
Publishing Findings that are Not Significant: Can Non-significant Findings Be Significant?
Lederman, N. G., & Lederman, J. S. (2016) · Journal of Science Teacher Education · doi:10.1007/s10972-016-9475-2 · Cited by 11
Psychopathy: Neurohype and Its Consequences
Jalava, J., & Griffiths, S. Y. (2021) · History, philosophy and theory of the life sciences · doi:10.1007/978-3-030-82454-9_6 · Cited by 8
Smaldino, P. E., Turner, M. A., & Kallens, P. C. (2019) · Royal Society Open Science · doi:10.1098/rsos.191249 · Cited by 6
Open Science and Educational Technology Research
Lodge, J. M., Corrin, L., Hwang, G., & Thompson, K. (2021) · Australasian Journal of Educational Technology · doi:10.14742/ajet.7565 · Cited by 6
Segerstrom, S. C., Diefenbach, M. A., Hamilton, K., O'Connor, D. B., & Tomiyama, A. J. (2022) · doi:10.31234/osf.io/dj3xe · Cited by 6
Improving transparency in observational social science research: A pre-analysis plan approachPreprint
Burlig, F. (2017) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/qemkz · Cited by 9
The Influence of Market Support Interventions on Household Food Security: An evidence synthesis
Juillard, H., Mohiddin, L., Péchayre, M., Smith, G., & Lewin, R. (2017) · doi:10.21201/2017.8951 · Cited by 6
The Benefits of Preregistration and Registered ReportsPreprint
Lakens, D., Mesquida, C., Rasti, S., & Ditroilo, M. (2024) · doi:10.31234/osf.io/dqap7 · Cited by 6
Chan, S. W. K., Lu, S., Cheng, N. H. C., CHUI, C. H.-K., & Lum, T. Y. S. (2024) · Nonprofit and Voluntary Sector Quarterly · doi:10.1177/08997640241258140 · Cited by 9
A scoping review of gendered academic career outcomes: An ecosystem perspective
Gould, J. A., Kulik, C. T., & Platt, A. L. M. (2025) · European Management Journal · doi:10.1016/j.emj.2025.02.006 · Cited by 6
Kühner, C., Verdaasdonk, I., Christ, C., Goudriaan, A. E., Thomaes, K., & de Waal, M. M. (2025) · Frontiers in Psychology · doi:10.3389/fpsyg.2025.1610030 · Cited by 8
“Reporting non-significant factors is valuable given the inflation of effect estimates when the literature reports only on those studies that found a significant effect for a specific risk/protective factor (Franco et al., 2014).”
Increasing Take‐Up of Social Benefits: A Meta‐Analysis of Field Experiments
Bendtsen, K. (2026) · Journal of Policy Analysis and Management · doi:10.1002/pam.70085 · Cited by 6
Degradation of lithium-ion batteries: A meta-analysis
Ligorati, M., Geyer‐Klingeberg, J., Günther, T. E., & Rathgeber, A. (2026) · Chemical Engineering Journal · doi:10.1016/j.cej.2026.174528 · Cited by 5
A First Experience of Using Failure Report as a Reflective Tool in Engineering Education
Harithuddin, A. S. M. (2021) · International Journal of Emerging Technologies in Learning (iJET) · doi:10.3991/ijet.v16i18.24271 · Cited by 8
Do US State Breach Notification Laws Decrease Firm Data Breaches?
Vaaler, P. M., & Greenwood, B. N. (2023) · Review of Law & Economics · doi:10.1515/rle-2023-0038 · Cited by 7
“…allows us to avoid the “file-drawer” problem, wherein researchers gravitate towards significant and publishable results rather than insignificant ones often failing to reach the public through publication (Dynes and Holbein 2020; Franco, Malhotra, and Simonovits, 2014; Goldfarb and King 2015).”
BackgroundMethods for assessing inverse publication bias of adverse events
Xing, X., Xu, C. Q., Amer, F. M. A., Shi, L., Zhu, J., & Lin, L. (2024) · Contemporary Clinical Trials · doi:10.1016/j.cct.2024.107646 · Cited by 6
Characterisation and calibration of multiversal methods
Cantone, G. G., & Tomaselli, V. (2024) · Advances in Data Analysis and Classification · doi:10.1007/s11634-024-00610-9 · Cited by 8
“…suffer from a publication bias, which is empirically demonstrable bias of published studies: studies that reject the null hypothesis are more likely to be researched, and published after processes of peer review (Franco et al. 2014; Dwan et al. 2008; van Zwet and Cator 2021; Bartoš et al. 2024).”
The file drawer problem in social science survey experiments
Moniz, P., Druckman, J. N., & Freese, J. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2426937122 · Cited by 13
Bertin, P., & Nera, K. (2025) · International Review of Social Psychology · doi:10.5334/irsp.979 · Cited by 7
Collective Victimhood Beliefs and Conflict-Related Attitudes: A Meta-Analysis
Kljajić, M., Shelef, N. G., & vanderWilden, E. (2025) · Comparative Political Studies · doi:10.1177/00104140251381739 · Cited by 8
“Since this biases publications towards research featuring larger effects (Franco et al., 2014; Gerber et al., 2001; Gerber & Malhotra, 2008), it may also bias the magnitude of meta-estimates upwards (away from zero).”
How to Identify and How to Conduct Research that Is Informative and Reproducible
Zickfeld, J., & Schubert, T. W. (2019) · Theory and history in the human and social sciences · doi:10.1007/978-3-030-33099-6_9 · Cited by 5
Aggregate context effects in music processing
Schmuckler, M. A., Vuvan, D. T., & Lewandowska, O. P. (2020) · Attention Perception & Psychophysics · doi:10.3758/s13414-020-02003-4 · Cited by 6
Klecka, H., johnston, I. A., Bowman, N. D., & Green, C. S. (2021) · Entertainment Computing · doi:10.1016/j.entcom.2021.100406 · Cited by 7
Does relative age affect fame? Ask Wikipedia
Peña, P. A., & Stephens-Davidowitz, S. I. (2021) · Education Economics · doi:10.1080/09645292.2021.1880548 · Cited by 5
McIlhiney, P., Gignac, G. E., Weinborn, M., & Ecker, U. K. H. (2021) · Quarterly Journal of Experimental Psychology · doi:10.1177/17470218211048986 · Cited by 10
“…in this investigation were conducted only to try help identify the issues with the task, so that future research may be better informed; for this reason, and for the sake of transparency (e.g., see Franco et al., 2014), we report this investigation in the Online Supplement at https://osf.io/ nj9kr.”
BackgroundResponse Decoupling and Partisans' Evaluations of Politicians' Transgressions
Yair, O., & Schaffner, B. F. (2022) · Political Behavior · doi:10.1007/s11109-022-09796-0 · Cited by 6
Segerstrom, S. C., Diefenbach, M. A., Hamilton, K., O'Connor, D. B., & Tomiyama, A. J. (2023) · Psychosomatic Medicine · doi:10.1097/psy.0000000000001186 · Cited by 5
Ciubotariu, I. I., & Bosch, G. (2023) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1011160 · Cited by 7
“For instance, a study of social science experiments found that majority of null results, in contrast to “strong” results which are usually published in top journals, remain unwritten [60,64].”
ResultXu, Y., Wei, S., Liu, E., & Yang, Y. (2023) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2023.139462 · Cited by 5
Rehkopf, D. H., Wojcicki, J. M., Haydel, K. F., Lin, J., Smith, D. L., Kapphahn, K., & Robinson, T. N. (2023) · Pediatric Obesity · doi:10.1111/ijpo.13082 · Cited by 5
“…studies conducted with pre-pubertal children and adolescents have shown observed loss of 0.02 T/S units/year 21 and loss of 0.005 T/S units/year, 22 although these values cannot be compared directly with the findings in this study because T/S ratios can vary by study and measurement method.”
ResultInfluentialBlake, K., Kubo, T., & Veríssimo, D. (2023) · Global Environmental Psychology · doi:10.5964/gep.11181 · Cited by 7
“…of message framing on outcome variables has been overstated due to publication bias—where social science experiments with statistically significant results can be as much as three times more likely to be published in peer-reviewed journals than those with null results (Franco et al., 2014).”
Too Small to Succeed: Small Samples and the p-Value Problem
Aguirre-Urreta, M. I., Rönkkö, M., & McIntosh, C. N. (2024) · ACM SIGMIS Database the DATABASE for Advances in Information Systems · doi:10.1145/3685235.3685238 · Cited by 9
“The vast majority of these results, in turn, will not make it into a publication (Franco et al., 2014).”
The association between cyber victimization and social anxiety: A three‐level meta‐analysis
Yang, S., Zhang, Q., Ran, G., Zhang, Q., & Ren, J. (2024) · Psychology in the Schools · doi:10.1002/pits.23336 · Cited by 7
Metadata
- Authors
- Franco, A., Malhotra, N., & Simonovits, G.
- Affiliations
- Stanford University (US)
- Department of Political Science, Stanford University, Stanford, CA, USA.
- Corresponding author
- Malhotra, Neil
- ORCID
- Malhotra, Neil
- Venue
- Science
- Published
- 2014-09-19
- Volume
- 345
- Issue
- 6203
- Pages
- 1502-1505
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W1997809363
- Cited by
- 1,542
- References
- 38
- Citations by year
- 2026: 88 · 2025: 152 · 2024: 143 · 2023: 162 · 2022: 129 · 2021: 152 · 2020: 212 · 2019: 123 · 2018: 119 · 2017: 110 · 2016: 91 · 2015: 52 · 2014: 9
- FWCI
- 85.56
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- Open Research (Auckland University of Technology) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- publication bias, negative results, file drawer problem, social sciences, survey experiment, peer review
- MeSH terms
- Social Sciences, Publication Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.