Publication bias in the social sciences: Unlocking the file drawer
The DOI matches its Crossref record; 37 of the 38 references it cites were found in a registry or an index.
Abstract
The file drawer is full. Should we worry?: Experiments that produce null results face a higher barrier to publication than those that yield statistically significant differences. Whether this is a problem depends on how many null but otherwise valid results might be trapped in the file drawer. Franco et al. use a Time-sharing Experiments in the Social Sciences archive of nearly 250 peer-reviewed proposals of social science experiments conducted on nationally representative samples. They find that only 10 out of 48 null results were published, whereas 56 out of 91 studies with strongly significant results made it into a journal. Science , this issue p. 1502
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Selective reporting, questionable research practices, and low-powered research all contribute to the unreliability of findings in a range of disciplines”
From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as Franco et al., 2014
In the citing paper
“In particular, negative or non-significant results are far less likely to be published (Franco et al., 2014).”
From What do we know about interventions to improve educator wellbeing? A systematic literature review (2024) · cites this paper as Franco et al., 2014
In the citing paper
“The bias against negative results appears to be a consequence of both editors and reviewers finding them less compelling for publication and from authors themselves not trying to publish them 8-12”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [8]
In the citing paper
“Positive results -- finding that a treatment is effective or that there is a relationship between two variables -- are more likely to be published than negative results 6-12 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [8]
In the citing paper
“A possible interpretation of these results is that the missing effect sizes are mainly a file‐drawer problem (Rosenthal, 1979). This interpretation is consistent with the evidence presented in Franco et al. (2014), which indicates that the file‐drawer problem is the main culprit behind publication bias in the social sciences.”
From Targeted school‐based interventions for improving reading and mathematics for students with or at risk of academic difficulties in Grades K‐6: A systematic review (2021) · cites this paper as Franco et al. (2014)
3 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722 · cited 1×
“In the presence of publication bias smaller studies report larger effects in order to exceed arbitrary significance thresholds (19, 20).”
BackgroundThe file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
Meta-analysis in social research
(1982) · Evaluation and Program Planning · doi:10.1016/0149-7189(82)90011-8 · Cited by 3,853
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109
Publication Bias in Meta‐Analysis: Prevention, Assessment and Adjustments
(2005) · doi:10.1002/0470870168 · Cited by 2,368
The case of the misleading funnel plot
Lau, J., Ioannidis, J. P. A., Terrin, N., Schmid, C. H., & Olkin, I. (2006) · BMJ · doi:10.1136/bmj.333.7568.597 · Cited by 1,766 · cited 1×
“However, these visualization-based approaches are sensitive to using different measures of precision (21, 22) and also assume outcome variables and effect sizes are comparable across studies (23).”
Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias
Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A.-W., Cronin, E., Decullier, E., Easterbrook, P. J., Von Elm, E., Gamble, C., et al. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003081 · Cited by 1,546 · cited 1×
“A different approach involves examining the publication outcomes of a cohort of studies, either prospectively or retrospectively (24, 25).”
MethodSumming up: The Science of Reviewing Research
Wilkinson, I., Light, R. J., & Pillemer, D. B. (1987) · Journal of Educational Statistics · doi:10.2307/1164691 · Cited by 1,471
Summing Up: The Science of Reviewing Research
Light, R. J., & Pillemer, D. B. (1984) · Harvard University Press eBooks · doi:10.4159/9780674040243 · Cited by 1,427
The Existence of Publication Bias and Risk Factors for Its Occurrence
Dickersin, K. (1990) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1990.03440100097014 · Cited by 1,417 · cited 1×
“Publication bias occurs when “publication of study results is based on the direction or significance of the findings” (1).”
Background
Research works printed but not linked: 3
- 10.1037/0735-7028.17.2.136 · doi:10.1037/0735-7028.17.2.136 · Paper page
- K. Dickersin in Publication Bias in Meta-Analysis: Prevention Assessment and Adjustments H. R. Rothstein A. J. Sutton M. Borenstein Eds. (Wiley Chichester UK 2006) ch. 2.
- Dickersin K., Min Y. I., Online J. Curr. Clin. Trials 1993, 50 (1993).
Cited by
Items 401–500 of 1,002
The effect of rivers, streams, and canals on property values
Nicholls, S., & Crompton, J. L. (2017) · River Research and Applications · doi:10.1002/rra.3197 · Cited by 34
Identification of and correction for publication biasPreprint
Kasy, M., & Andrews, I. (2018) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/49yst · Cited by 42
The Impact of Complexity on Methods and Findings in Psychological Science
Sanbonmatsu, D. M., Cooley, E. H., & Butner, J. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.580111 · Cited by 50
Evidence for, and challenges to, sensory recruitment models of visual working memory
Adam, K. C. S., Rademaker, R. L., & Serences, J. T. (2022) · Routledge eBooks · doi:10.4324/9781003158134-2 · Cited by 57
Motivations behind problematic short video use: A three-level meta-analysis
Chen, Y., Zhang, W., Zhong, N., & Zhao, M. (2024) · Telematics and Informatics · doi:10.1016/j.tele.2024.102196 · Cited by 35
The Validity of Conscientiousness Is Overestimated in the Prediction of Job Performance
Kepes, S., & McDaniel, M. A. (2015) · PLOS ONE · doi:10.1371/journal.pone.0141468 · Cited by 33
“In both the medical sciences [21–23] and the social sciences [24], publication bias appears to be primarily driven by authors who do not submit null or otherwise undesirable findings [16, 25].”
Dietrichson, J., Filges, T., Klokker, R. H., Viinholt, B. C. A., Bøg, M., & Jensen, U. H. (2020) · Campbell Systematic Reviews: Better Evidence for a Better World · doi:10.1002/cl2.1081 · Cited by 33
“This interpretation is consistent with the evidence presented in Franco, Malhotra, and Simonovits (2014), which indicates that the file‐drawer problem is the main culprit behind publication bias in the social sciences.”
Type M Error Might Explain Weisburd’s Paradox
Gelman, A., Skarðhamar, T., & Aaltonen, M. (2017) · Journal of Quantitative Criminology · doi:10.1007/s10940-017-9374-5 · Cited by 32
“Comparing effects in published vs. unpublished studies is not the same as significant vs. not significant, because the latter are often never even written up (Franco, Malhotra, and Simonovits, 2014).”
ResultA Call for Open Science in Giftedness Research
McBee, M. T., Makel, M. C., Peters, S. J., & Matthews, M. S. (2018) · Gifted Child Quarterly · doi:10.1177/0016986218784178 · Cited by 40
Democratic Creative Destruction? The Effect of a Changing Media Landscape on Democracy
Nielsen, R. K., & Fletcher, R. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108890960.008 · Cited by 62
Growth and gaps: a meta-review of policy diffusion studies in the American states
Mallinson, D. J. (2021) · Policy & Politics · doi:10.1332/030557321x16119271286848 · Cited by 33
Wuttke, A. (2018) · Politische Vierteljahresschrift · doi:10.1007/s11615-018-0131-7 · Cited by 46
Wagner, J. A. (2021) · International Journal of Social Research Methodology · doi:10.1080/13645579.2021.1922805 · Cited by 37
Turi, E. R., McMenamin, A. L., Kueakomoldej, S., Kurtzman, E. T., & Poghosyan, L. (2023) · Nursing Outlook · doi:10.1016/j.outlook.2023.101995 · Cited by 41
Greenlee, J. L., Winter, M. A., & Diehl, J. J. (2018) · Research in Autism · doi:10.1016/j.rasd.2018.06.002 · Cited by 36
When Digital Trace Data Meet Traditional Communication Theory: Theoretical/Methodological Directions
Choi, S. (2018) · Social Science Computer Review · doi:10.1177/0894439318788618 · Cited by 38
“…in the file drawer and only publishing significant results—can distort the literature because the continuous publication of statistically significant results may make relevant findings appear to have a strong effect, when they in fact have the opposite effect (Franco, Malhotra, & Simonovits, 2014).”
ResultLevesque, A. R., MacDonald, S., Berg, S. A., & Reka, R. (2021) · Adolescent Research Review · doi:10.1007/s40894-021-00151-8 · Cited by 48
Placebo Selection in Survey Experiments: An Agnostic Approach
Porter, E. V., & Velez, Y. R. (2021) · Political Analysis · doi:10.1017/pan.2021.16 · Cited by 32
“Finally, in light of evidence about publication bias (Dickersin et al. 1987; Franco, Malhotra, and Simonovits 2014), researchers may be tempted to use placebos that shift outcomes in a negative direction relative to a pure control to heighten their probability of detecting significant effects, even…”
BackgroundAdvocacy planning: were expectations fulfilled?
Sager, T. Ø. (2022) · Planning Perspectives · doi:10.1080/02665433.2022.2040189 · Cited by 32
Open Science and Research Reproducibility
Munafo, M. R. (2016) · ecancermedicalscience · doi:10.3332/ecancer.2016.ed56 · Cited by 37
Denial of service to same-sex and interracial couples: Evidence from a national survey experiment
Powell, B., Schnabel, L., & Apgar, L. (2017) · Science Advances · doi:10.1126/sciadv.aao5834 · Cited by 37
Irreproducible text-book “knowledge”: The effects of color bands on zebra finch fitness
Wang, D., Forstmeier, W., Ihle, M., Khadraoui, M., Jerónimo, S., Martin, K., & Kempenaers, B. (2018) · Evolution · doi:10.1111/evo.13459 · Cited by 35
Yikes! Are we disgusted by politicians?
Bakker, B. N., Schumacher, G., & Homan, M. D. (2020) · Politics and the Life Sciences · doi:10.1017/pls.2020.16 · Cited by 39
“Null findings tend to be much less likely to be written and/or published (Franco et al., 2014).”
BackgroundBenton, J. S., Cotterill, S., Anderson, J., Macintyre, V. G., Gittins, M., Dennis, M., Lindley, S. J., & French, D. (2021) · Wellbeing Space and Society · doi:10.1016/j.wss.2021.100029 · Cited by 37
Shim, Y., Jebb, A. T., Tay, L., & Pawelski, J. O. (2021) · Review of General Psychology · doi:10.1177/10892680211021350 · Cited by 28
“This type of review may be susceptible to publication bias, typically called the “file drawer” problem, which is known to show a preference toward more substantial (larger) results in published results than those that go unpublished (Franco et al., 2014).”
ResultCairo, A. H., Green, J. D., Forsyth, D. R., Behler, A. M. C., & Raldiris, T. L. (2020) · Personality and Social Psychology Bulletin · doi:10.1177/0146167220903896 · Cited by 34
Burke, M., Hsiang, S., & Miguel, E. A. (2014) · National Bureau of Economic Research · doi:10.3386/w20598 · Cited by 42
Transparency, Reproducibility, and the Credibility of Economics Research
Christensen, G., & Miguel, E. A. (2016) · National Bureau of Economic Research · doi:10.3386/w22989 · Cited by 48
Bernecker, S. L., Constantino, M. J., Atkinson, L. R., Bagby, R. M., Ravitz, P., & McBride, C. (2016) · Psychotherapy · doi:10.1037/pst0000036 · Cited by 37
Ego Depletion May Disappear by 2020
Vadillo, M. A. (2019) · Social Psychology · doi:10.1027/1864-9335/a000375 · Cited by 33
What Causes Social Marketing Programs to Fail? A Qualitative Study
Akbar, M. B., Foote, L., Soraghan, C., Millard, R., & Spotswood, F. (2021) · Social Marketing Quarterly · doi:10.1177/15245004211010202 · Cited by 34
“This publication bias favoring strong results over null results has been documented in multiple disciplines including the social and biomedical sciences (Franco et al., 2014) and occurs because researchers perceive their findings as having “no publication potential” (Franco et al., 2014, p. 1504).”
ResultEvans, T. R., Branney, P., Clements, A. J., & Hatton, E. (2021) · Accountability in Research · doi:10.1080/08989621.2021.1969233 · Cited by 39
“…(Haddaway and Bayliss 2015) where the decision to publish is disproportionately biased by the outcomes of the study rather than driven by the quality of design, thereby distorting the literature to reflect larger and more statistically significant effects (Franco, Malhotra, and Simonovits 2014).”
Reporting all results efficiently: A RARE proposal to open up the file drawer
Laitin, D. D., Miguel, E. A., Alrababa’h, A., Bogdanoski, A., Grant, S., Hoeberling, K., Mo, C. H., Moore, D. A., Vazire, S., Weinstein, J. M., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2106178118 · Cited by 40
Level of Attention to Motherese Speech as an Early Marker of Autism Spectrum Disorder
Pierce, K., Wen, T. H., Zahiri, J., Andreason, C., Courchesne, E., Barnes, C. C., Lopez, L., Arias, S. J., Esquivel, A., & Cheng, A. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2022.55125 · Cited by 36
Ramaiya, A., Chandra‐Mouli, V., Both, R., Gottert, A., Guglielmi, S., Beckwith, S., Li, M., & Blum, R. W. (2023) · Sexual and Reproductive Health Matters · doi:10.1080/26410397.2023.2187170 · Cited by 41
A Phenome-Wide Mendelian Randomization Study of Pancreatic Cancer Using Summary Genetic Data
Langdon, R., Richmond, R. C., Hemani, G., Zheng, J., Wade, K. H., Carreras‐Torres, R., Johansson, M., Brennan, P., Wootton, R. E., Munafo, M. R., et al. (2019) · Cancer Epidemiology Biomarkers & Prevention · doi:10.1158/1055-9965.epi-19-0036 · Cited by 32
Yeager, D. S., Krosnick, J. A., Visser, P. S., Holbrook, A. L., & Tahk, A. M. (2019) · Journal of Personality and Social Psychology · doi:10.1037/pspa0000171 · Cited by 33
“For more than 15 years, TESS has allowed researchers to conduct experiments with representative samples of American adults, and hundreds of such experiments have been conducted since 2003 by psychologists, sociologists, political scientists, and economists (Franco et al., 2014; TESS, 2018).”
BackgroundMethodInfluentialEvaluating Registered Reports: A Naturalistic Comparative Study of Article Impact
Hopun, L. T., Singleton Thorn, F., Nosek, B. A., & Errington, T. M. (2017) · doi:10.31219/osf.io/5y8w7 · Cited by 29
Titiunik, R., & Feher, A. (2017) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/rssa.12293 · Cited by 30
Ideology Between the Lines: Lay Inferences About Scientists’ Values and Motives
Hannikainen, I. R. (2018) · Social Psychological and Personality Science · doi:10.1177/1948550618790230 · Cited by 31
A systematic literature review of the relationship between work hours and sickness absence
Bernstrøm, V. H., & Houkes, I. (2017) · Work & Stress · doi:10.1080/02678373.2017.1394926 · Cited by 32
Negative Results: Conceptual and Methodological Dimensions in Single-Case Intervention Research
Kratochwill, T. R., Levin, J. R., & Horner, R. H. (2017) · Remedial and Special Education · doi:10.1177/0741932517741721 · Cited by 35
Pedersen, R. L., & Robinson, Z. (2018) · Local Environment · doi:10.1080/13549839.2018.1463210 · Cited by 33
“Publication bias against papers with negative outcomes is a well-documented, however unresolved, phenomenon in medical (Easterbrook et al. 1991) and social sciences (Franco, Malhotra, and Simonovits 2014).”
BackgroundBrunjes, B. M., & Kellough, J. E. (2018) · Journal of Public Administration Research and Theory · doi:10.1093/jopart/muy022 · Cited by 29
A Future for Observational Epidemiology: Clarity, Credibility, Transparency
Harper, S. B. (2019) · American Journal of Epidemiology · doi:10.1093/aje/kwy280 · Cited by 33
Lee, S., Allison, T. A., O’Neill, D. J., Punch, P., Helitzer, E., & Moss, H. (2022) · Health Promotion International · doi:10.1093/heapro/daac024 · Cited by 40
Generalizing Survey Experiments Using Topic Sampling: An Application to Party Cues
Clifford, S., Leeper, T. J., & Rainey, C. (2023) · Political Behavior · doi:10.1007/s11109-023-09870-1 · Cited by 40
“For example, an analysis of the Time-sharing Experiments in the Social Sciences (TESS) database revealed that statistically significant findings are dramatically more likely to be written up and published than null results (Franco, Malhotra, and Simonovits 2014).”
An Introduction to Conducting Email Audit Studies
Crabtree, C. D. (2018) · Springer eBooks · doi:10.1007/978-3-319-71153-9_5 · Cited by 35
Pre-registration and Results-Free Review in Observational and Qualitative Research
Jacobs, A. M. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.009 · Cited by 34
On the Need for a Replication Journal
Zimmermann, C. (2015) · doi:10.20955/wp.2015.016 · Cited by 31
Riedel, A. S., Beatson, A., Worsteling, A., Mulcahy, R. F., & Keating, B. W. (2023) · Journal of service management · doi:10.1108/josm-08-2022-0250 · Cited by 36
Yang, C., Zhao, W., Yuan, B., Luo, L., & Shanks, D. R. (2022) · Review of Educational Research · doi:10.3102/00346543221094083 · Cited by 38
Adapting open science and pre‐registration to longitudinal research
Petersen, I. T., Apfelbaum, K. S., & McMurray, B. (2022) · Infant and Child Development · doi:10.1002/icd.2315 · Cited by 30
“…(d) lack of standardization (Frank et al., 2017), (e) publication biases that suppress non-significant results, that is, the file drawer problem (Franco et al., 2014; Rosenthal, 1979), (f) statistical reporting errors (Bakker & Wicherts, 2011; Nuijten et al., 2016), (g) measurement error…”
BackgroundHenry, A., Alphonse, J., Tynan, D., & Welsh, A. W. (2017) · Ultrasound in Obstetrics and Gynecology · doi:10.1002/uog.17476 · Cited by 30
Bao, X., Zheng, Q., Tong, Q., Zhu, P.-C., Zhuang, Z., Zheng, G., & Wang, Y. (2018) · Frontiers in Pharmacology · doi:10.3389/fphar.2018.01445 · Cited by 28
Does News Frame Affect Free Movement Attitudes? A Comparative Analysis
Theorin, N., Meltzer, C. E., Galyga, S., Strömbäck, J., Schemer, C., Eberl, J., Lind, F., Heidenreich, T., & Boomgaarden, H. G. (2021) · Journalism & Mass Communication Quarterly · doi:10.1177/10776990211006793 · Cited by 30
“One possible explanation might be publication bias, meaning that the publication of studies is influenced by the significance or directions of the findings (Franco et al., 2014; Gerber et al., 2010) and that studies showing no effects have a harder time getting published.”
BackgroundExamining publication bias—a simulation-based evaluation of statistical tests on publication bias
Schneck, A. (2017) · PeerJ · doi:10.7717/peerj.4115 · Cited by 32
“Franco et al. (2014) also note 100 that most non-significant findings go to the file-drawer right after the analysis and are not even 101 written up.”
BackgroundHildebrand, M., Wibbelink, C. J. M., & Verschuère, B. (2018) · International Journal of Law and Psychiatry · doi:10.1016/j.ijlp.2018.02.013 · Cited by 36
Dissociation and disasters: A systematic review
Canan, F., & North, C. S. (2019) · World Journal of Psychiatry · doi:10.5498/wjp.v9.i6.83 · Cited by 32
Do Practice Tests (Quizzes) Reduce or Provoke Test Anxiety? A Meta-Analytic Review
Yang, C., Li, J., Zhao, W., Luo, L., & Shanks, D. R. (2023) · Educational Psychology Review · doi:10.1007/s10648-023-09801-w · Cited by 52
Bierman, D. J., Spottiswoode, J., & Aron, B. (2016) · PLoS ONE · doi:10.1371/journal.pone.0153049 · Cited by 43
Löckenhoff, C. E., & Samanez-Larkin, G. R. (2019) · The Journals of Gerontology Series B · doi:10.1093/geronb/gbz097 · Cited by 26
Should the Randomistas (Continue to) Rule?
Ravallion, M. D. (2020) · National Bureau of Economic Research · doi:10.3386/w27554 · Cited by 53
“Franco et al. (2014) found that experiments in the social and political sciences reporting statistically significant effects are more likely to be published.”
NOAA, Busch, D. S., O’Donnell, M. J., Hauri, C., Mach, K. J., Poach, M. E., Doney, S. C., & Signorini, S. R. (2015) · Oceanography · doi:10.5670/oceanog.2015.29 · Cited by 27
“Studies of CO2 vent systems can elucidate some indirect effects of acidification on community members (Garrard et al., 2014).”
BackgroundThe Research Reproducibility Crisis and Economics of Science
Maniadis, Z., & Tufano, F. (2017) · The Economic Journal · doi:10.1111/ecoj.12526 · Cited by 32
“…whether similar policies should be imposed (e.g., Nyhan, 2015; Maniadis et al., 2014, 2015; Cook, 2016; Byington and Felps, 2017), but the relative deficit in quantitative research synthesis makes this assessment difficult (but, in social sciences, note the seminal work of Franco et al., 2014).”
Zhuang, Z., Wang, Z.-H., Huang, Y.-Y., Zheng, Q., & Pan, X. (2020) · Journal of Ethnopharmacology · doi:10.1016/j.jep.2020.112568 · Cited by 29
Comparative Media Regulation in the United States and Europe
Fukuyama, F., & Grotto, A. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108890960.010 · Cited by 42
Blanco, B., Molnar, M. T., Carreiras, M., Collins-Jones, L. H., Vidal-Rosas, E. E., Cooper, R. J., & Caballero‐Gaudes, C. (2021) · Neurophotonics · doi:10.1117/1.nph.8.2.025011 · Cited by 32
“Negative results can help filling the gaps of current scientific knowledge and are essential for improving research transparency and reproducibility.(92,93)”
Zhao, F., Guo, Z., Ma, Z., Ma, L., & Zhao, J. (2021) · Journal of Ethnopharmacology · doi:10.1016/j.jep.2021.114395 · Cited by 31
Farayibi, A. O., & Folarin, O. (2021) · African Development Review · doi:10.1111/1467-8268.12588 · Cited by 36
Which Findings Should Be Published?
Frankel, A. P., & Kasy, M. (2022) · American Economic Journal Microeconomics · doi:10.1257/mic.20190133 · Cited by 55
The pathway from sensory processing sensitivity to physical health: Stress as a mediator
Kenemore, J., Chavez, J., & Benham, G. (2023) · Stress and Health · doi:10.1002/smi.3250 · Cited by 30
Devolution of Researcher Care in Organization Studies and the Moderation of Organizational Knowledge
Schwarz, G. M., Cummings, C., & Cummings, T. G. (2015) · Academy of Management Learning and Education · doi:10.5465/amle.2014.0167 · Cited by 28
“For example, Franco et al. (2014) recently indicated in their study of publication outcomes funded by the National Science Foundation that nearly two thirds of the social science studies that produced null results were not even written up, but remained as working papers in the “file drawer,” based…”
BackgroundBayesian data analysis as a tool for behavior analysts
Young, M. E. (2019) · Journal of the Experimental Analysis of Behavior · doi:10.1002/jeab.512 · Cited by 30
Spellman, B. A., Gilbert, E. A., & Corker, K. S. (2018) · Wiley eBooks · doi:10.1002/9781119170174.epcn519 · Cited by 92
Designing safety interventions for specific contexts: Results from a literature review
Karanikas, N., Khan, S. R., Baker, P. R. A., & Pilbeam, C. (2022) · Safety Science · doi:10.1016/j.ssci.2022.105906 · Cited by 12
Preclinical data reproducibility for R&D—the challenge for neuroscience
Steckler, T. (2015) · Psychopharmacology · doi:10.1007/s00213-014-3836-3 · Cited by 25
“2013) and to abandon negative results that they believe journals are unlikely to publish (the filedrawer phenomenon; Franco et al. 2014).”
BackgroundCueing effect in multimedia learning: A meta-analysis
Xie, H., Wang, F., Zhou, Z., & Peng, W. U. (2016) · Acta Psychologica Sinica · doi:10.3724/sp.j.1041.2016.00540 · Cited by 28
Grounds for Ambiguity: Justifiable Bases for Engaging in Questionable Research Practices
Sacco, D. F., Brown, M., & Bruton, S. V. (2018) · Science and Engineering Ethics · doi:10.1007/s11948-018-0065-x · Cited by 33
Forecasting the Results of Experiments: Piloting an Elicitation Strategy
DellaVigna, S., Otis, N. G., & Vivalt, E. (2020) · AEA Papers and Proceedings · doi:10.1257/pandp.20201080 · Cited by 25
Bor, A., & Simonovits, G. (2021) · Political Behavior · doi:10.1007/s11109-021-09728-4 · Cited by 28
Self-citationOnline Political Advertising in the United States
Fowler, E. F., Franz, M. M., & Ridout, T. N. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108890960.007 · Cited by 40
On Minimizing the Risk of Bias in Randomized Controlled Trials in Economics
Eble, A., Boone, P., & Elbourne, D. R. (2016) · World Bank, Washington, DC eBooks · doi:10.1596/1813-9450-7746 · Cited by 29
Seo, C., Kim, B., & Kruis, N. E. (2021) · International Criminal Justice Review · doi:10.1177/1057567720979184 · Cited by 24
“There exists a criticism of meta-analysis studies that rely on published work (Ellis, 2010; Franco et al., 2014; Kruis et al., 2020).”
BackgroundMethodHorton, M., Dixon, J., Turi, E. R., Balusu, C., Paikoff, R., Maier, C. B., & Poghosyan, L. (2024) · Journal of Clinical Nursing · doi:10.1111/jocn.17614 · Cited by 30
Sorting the File Drawer: A Typology for Describing Unpublished Studies
Lishner, D. A. (2021) · Perspectives on Psychological Science · doi:10.1177/1745691620979831 · Cited by 25
Emerging diasporas: Exploring mobilization outside the homeland
Kopchick, C., Cunningham, K. G., Jenne, E. K., & Saideman, S. M. (2021) · Journal of Peace Research · doi:10.1177/0022343320980803 · Cited by 37
Learning from Null Effects: A Bottom-Up Approach
Alrababa’h, A., Williamson, S., Dillon, A., Hainmueller, J., Hangartner, D., Hotard, M., Laitin, D. D., Lawrence, D., & Weinstein, J. M. (2022) · Political Analysis · doi:10.1017/pan.2021.51 · Cited by 35
“Part of this reflects a perception that such papers will be rejected, but it also appears to be the case that researchers “lose interest in ‘unsuccessful’ projects” (Franco et al. 2014).”
A systematic review on the current conceptualisations of successful psychopathy
Wallace, L. M., Fido, D., Sumich, A. L., & Heym, N. (2022) · Forensic Science International Mind and Law · doi:10.1016/j.fsiml.2022.100076 · Cited by 51
Forest Therapy Trails: A Conceptual Framework and Scoping Review of Research
Gobster, P. H., Schultz, C. L., Kruger, L. E., & Henderson, J. R. (2022) · Forests · doi:10.3390/f13101613 · Cited by 35
Zigerell, L. J. (2018) · Research & Politics · doi:10.1177/2053168017753862 · Cited by 26
Building knowledge economies in Africa: an introduction
Asongu, S. A., & Kuada, J. (2020) · Contemporary Social Science · doi:10.1080/21582041.2020.1722213 · Cited by 25
Do Pre-analysis Plans Hamper Publication?
Ofosu, G. K., & Posner, D. (2020) · AEA Papers and Proceedings · doi:10.1257/pandp.20201079 · Cited by 22
HARK Side of Deep Learning -- From Grad Student Descent to Automated Machine LearningPreprint
Gencoglu, O., van Gils, M., Guldogan, E., Morikawa, C., Süzen, M., Gruber, M., Leinonen, J., & Huttunen, H. J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.07633 · Cited by 44
“As stated by Nissen et al., even if authors’ behavior is the main contributor to publication bias (there is evidence supporting this in other fields [27, 28]), they may simply be responding to the editorial preferences for positive results [29].”
BackgroundConservatives and Liberals have Similar Physiological Responses to ThreatsPreprint
Bakker, B. N., Schumacher, G., Gothreau, C., & Arceneaux, K. (2019) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/vdpyt · Cited by 33
The Two-step Model of Clustered Democratization
Houle, C., & Kayser, M. A. L. A. D. (2019) · Journal of Conflict Resolution · doi:10.1177/0022002719875565 · Cited by 26
“…the literature that democratization diffuses and raises the prospect that the history of findings in support of diffusion in the preceding literature was a high-profile example of the “file-drawer problem”—bias against null results in the publication process (Franco, Malhotra, and Simonovits 2014).”
ResultConnecting Language Abilities and Social Competence in Children: A Meta-Analytic Review
Wieczorek, K., DeGroot, M., Ganshorn, H., & Graham, S. A. (2025) · Child Development · doi:10.1111/cdev.14218 · Cited by 55
“To address the possibility of a file drawer problem (i.e., the tendency for published literature to embody stronger and significant effects; Franco, Malhotra, and Simonovits 2014), we include both published and unpublished literature in our search strategy.”
Linder, C., & Farahbakhsh, S. (2020) · Business Ethics Quarterly · doi:10.1017/beq.2019.52 · Cited by 26
Understanding the greater diffusion of mobile money innovations in Africa
Asongu, S. A., Biekpe, N. B., & Cassimon, D. (2020) · Telecommunications Policy · doi:10.1016/j.telpol.2020.102000 · Cited by 92
McClain, M. B., Callan, G. L., Harris, B., Floyd, R. G., Haverkamp, C. R., Golson, M. E., Longhurst, D. N., & Benallie, K. J. (2021) · Journal of School Psychology · doi:10.1016/j.jsp.2020.11.002 · Cited by 25
Metadata
- Authors
- Franco, A., Malhotra, N., & Simonovits, G.
- Affiliations
- Stanford University (US)
- Department of Political Science, Stanford University, Stanford, CA, USA.
- Corresponding author
- Malhotra, Neil
- ORCID
- Malhotra, Neil
- Venue
- Science
- Published
- 2014-09-19
- Volume
- 345
- Issue
- 6203
- Pages
- 1502-1505
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W1997809363
- Cited by
- 1,542
- References
- 38
- Citations by year
- 2026: 88 · 2025: 152 · 2024: 143 · 2023: 162 · 2022: 129 · 2021: 152 · 2020: 212 · 2019: 123 · 2018: 119 · 2017: 110 · 2016: 91 · 2015: 52 · 2014: 9
- FWCI
- 85.56
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- Open Research (Auckland University of Technology) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- publication bias, negative results, file drawer problem, social sciences, survey experiment, peer review
- MeSH terms
- Social Sciences, Publication Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.