Publication bias in the social sciences: Unlocking the file drawer
The DOI matches its Crossref record; 37 of the 38 references it cites were found in a registry or an index.
Abstract
The file drawer is full. Should we worry?: Experiments that produce null results face a higher barrier to publication than those that yield statistically significant differences. Whether this is a problem depends on how many null but otherwise valid results might be trapped in the file drawer. Franco et al. use a Time-sharing Experiments in the Social Sciences archive of nearly 250 peer-reviewed proposals of social science experiments conducted on nationally representative samples. They find that only 10 out of 48 null results were published, whereas 56 out of 91 studies with strongly significant results made it into a journal. Science , this issue p. 1502
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Selective reporting, questionable research practices, and low-powered research all contribute to the unreliability of findings in a range of disciplines”
From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as Franco et al., 2014
In the citing paper
“In particular, negative or non-significant results are far less likely to be published (Franco et al., 2014).”
From What do we know about interventions to improve educator wellbeing? A systematic literature review (2024) · cites this paper as Franco et al., 2014
In the citing paper
“The bias against negative results appears to be a consequence of both editors and reviewers finding them less compelling for publication and from authors themselves not trying to publish them 8-12”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [8]
In the citing paper
“Positive results -- finding that a treatment is effective or that there is a relationship between two variables -- are more likely to be published than negative results 6-12 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [8]
In the citing paper
“A possible interpretation of these results is that the missing effect sizes are mainly a file‐drawer problem (Rosenthal, 1979). This interpretation is consistent with the evidence presented in Franco et al. (2014), which indicates that the file‐drawer problem is the main culprit behind publication bias in the social sciences.”
From Targeted school‐based interventions for improving reading and mathematics for students with or at risk of academic difficulties in Grades K‐6: A systematic review (2021) · cites this paper as Franco et al. (2014)
3 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722 · cited 1×
“In the presence of publication bias smaller studies report larger effects in order to exceed arbitrary significance thresholds (19, 20).”
BackgroundThe file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
Meta-analysis in social research
(1982) · Evaluation and Program Planning · doi:10.1016/0149-7189(82)90011-8 · Cited by 3,853
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109
Publication Bias in Meta‐Analysis: Prevention, Assessment and Adjustments
(2005) · doi:10.1002/0470870168 · Cited by 2,368
The case of the misleading funnel plot
Lau, J., Ioannidis, J. P. A., Terrin, N., Schmid, C. H., & Olkin, I. (2006) · BMJ · doi:10.1136/bmj.333.7568.597 · Cited by 1,766 · cited 1×
“However, these visualization-based approaches are sensitive to using different measures of precision (21, 22) and also assume outcome variables and effect sizes are comparable across studies (23).”
Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias
Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A.-W., Cronin, E., Decullier, E., Easterbrook, P. J., Von Elm, E., Gamble, C., et al. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003081 · Cited by 1,546 · cited 1×
“A different approach involves examining the publication outcomes of a cohort of studies, either prospectively or retrospectively (24, 25).”
MethodSumming up: The Science of Reviewing Research
Wilkinson, I., Light, R. J., & Pillemer, D. B. (1987) · Journal of Educational Statistics · doi:10.2307/1164691 · Cited by 1,471
Summing Up: The Science of Reviewing Research
Light, R. J., & Pillemer, D. B. (1984) · Harvard University Press eBooks · doi:10.4159/9780674040243 · Cited by 1,427
The Existence of Publication Bias and Risk Factors for Its Occurrence
Dickersin, K. (1990) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1990.03440100097014 · Cited by 1,417 · cited 1×
“Publication bias occurs when “publication of study results is based on the direction or significance of the findings” (1).”
Background
Research works printed but not linked: 3
- 10.1037/0735-7028.17.2.136 · doi:10.1037/0735-7028.17.2.136 · Paper page
- K. Dickersin in Publication Bias in Meta-Analysis: Prevention Assessment and Adjustments H. R. Rothstein A. J. Sutton M. Borenstein Eds. (Wiley Chichester UK 2006) ch. 2.
- Dickersin K., Min Y. I., Online J. Curr. Clin. Trials 1993, 50 (1993).
Cited by
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397
“Studies that obtain positive and novel results are more likely to be published than studies that obtain negative results or report replications of prior results 47,49,50 .”
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708
Self-citationThe preregistration revolution
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708274114 · Cited by 1,998
Power Analysis and Effect Size in Mixed Effects Models: A Tutorial
Brysbaert, M., & Stevens, M. (2018) · Journal of Cognition · doi:10.5334/joc.10 · Cited by 1,420
The Generalizability of Survey Experiments
Mullinix, K. J., Leeper, T. J., Druckman, J. N., & Freese, J. (2015) · Journal of Experimental Political Science · doi:10.1017/xps.2015.19 · Cited by 1,228
“STUDY 1 For both studies, the source of our population-based sample is the National Science Foundation funded Time-sharing Experiments for the Social Sciences (TESS) program (http://tessexperiments.org/; also see Franco et al. 2014).”
A New Paradigm for Systematic Literature Reviews in Supply Chain Management
Durach, C. F., Kembro, J., & Wieland, A. (2017) · Journal of Supply Chain Management · doi:10.1111/jscm.12145 · Cited by 924
The natural selection of bad science
Smaldino, P. E., & McElreath, R. (2016) · Royal Society Open Science · doi:10.1098/rsos.160384 · Cited by 860
Environmental education outcomes for conservation: A systematic review
Ardoin, N. M., Bowers, A. W., & Gaillard, E. (2020) · Biological Conservation · doi:10.1016/j.biocon.2019.108224 · Cited by 924
Harvey, S. B., Modini, M., Joyce, S., Milligan-Saville, J. S., Tan, L., Mykletun, A., Bryant, R. A., Christensen, H., & Mitchell, P. B. (2017) · Occupational and Environmental Medicine · doi:10.1136/oemed-2016-104015 · Cited by 799
Replicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770
Burke, M., Hsiang, S. M., & Miguel, E. (2015) · Annual Review of Economics · doi:10.1146/annurev-economics-080614-115430 · Cited by 655
Carter, E. C., Kofler, L. M., Forster, D. E., & McCullough, M. E. (2015) · Journal of Experimental Psychology: General · doi:10.1037/xge0000083 · Cited by 594
Mindfulness mediates the physiological markers of stress: Systematic review and meta-analysis
Pascoe, M. C., Thompson, D. R., Jenkins, Z. M., & Ski, C. F. (2017) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2017.08.004 · Cited by 579
Cooper, C., Booth, A., Varley-Campbell, J., Britten, N., & Garside, R. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0545-3 · Cited by 561
Yoga, mindfulness-based stress reduction and stress-related physiological measures: A meta-analysis.
Pascoe, M. C., Thompson, D. R., & Ski, C. F. (2017) · Psychoneuroendocrinology · doi:10.1016/j.psyneuen.2017.08.008 · Cited by 570
“Indeed, previous work has demonstrated significant psychological effects of yoga practice, including decreased depressive and anxious symptoms, perceived stress (Cramer et al., 2013; Li and Goldsmith, 2012) and improved emotional regulation (Gard et al., 2014).”
Heisig, J. P., & Schaeffer, M. (2019) · European Sociological Review · doi:10.1093/esr/jcy053 · Cited by 552
“Proving this is difficult, since about 60% of null-results are never written up (Franco et al., 2014).”
BackgroundVerhage, M. L., Schuengel, C., Madigan, S., Fearon, R. M. P., Oosterman, M., Cassibba, R., Bakermans-Kranenburg, M. J., & van IJzendoorn, M. H. (2016) · Psychological Bulletin · doi:10.1037/bul0000038 · Cited by 550
“George & Solomon (1996) 32 AAI, SRP Both No Gloger-Tippelt et al. (2002) 27 AAI, SSP Both No Gomille & Gloger-Tippelt (1999) 28 Gojman et al. (2012; urban) 35 AAI, SSP, Ainsworth scales Both Yes Gojman et al. (2012; rural) 31 AAI, SSP, Ainsworth scales Both Yes Grossmann et al. (1988; Sample a) 20 AAI, SSP Forced No G…”
BackgroundInfluentialThe Hong Kong Principles for assessing researchers: Fostering research integrity
Moher, D., Bouter, L., Kleinert, S., Glasziou, P., Sham, M. H., Barbour, V., Coriat, A.-M., Foeger, N., & Dirnagl, U. (2020) · PLOS Biology · doi:10.1371/journal.pbio.3000737 · Cited by 540
“For example, a systematic review of trials of reboxetine for treating depression found that almost three-quarters of included patients were in unpublished trials [22]; other examples across different disciplines also exist [23,24].”
BackgroundEffects of Neonicotinoid Pesticide Exposure on Human Health: A Systematic Review
Cimino, A. M., Boyles, A. L., Thayer, K. A., & Perry, M. J. (2017) · Environmental Health Perspectives · doi:10.1289/ehp515 · Cited by 536
“The distribution of age varied significantly among the studies, with children <19 years comprising 37% of the cases in Forrester (2009) compared to no children <14 years of age enrolled in Mohamed et al. (2009) and only two children included among the cases in Phua et al.”
BackgroundResultInfluentialGeneralizing from Survey Experiments Conducted on Mechanical Turk: A Replication Approach
Coppock, A. (2019) · Political Science Research and Methods · doi:10.1017/psrm.2018.10 · Cited by 515
RCTs to Scale: Comprehensive Evidence From Two Nudge Units
DellaVigna, S., & Linos, E. (2022) · Econometrica · doi:10.3982/ecta18709 · Cited by 480
The past, present and future of Registered Reports
Chambers, C. D., & Tzavella, L. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01193-7 · Cited by 499
Have Instrumental Variables Brought Us Closer to the Truth
Jiang, W. (2017) · The Review of Corporate Finance Studies · doi:10.1093/rcfs/cfx015 · Cited by 489
When and why defaults influence decisions: a meta-analysis of default effects
JACHIMOWICZ, J. M., DUNCAN, S., WEBER, E. U., & JOHNSON, E. J. (2019) · Behavioural Public Policy · doi:10.1017/bpp.2018.43 · Cited by 481
Zwaan, R. A., Etz, A., Lucas, R. E., & Donnellan, M. B. (2018) · Behavioral and Brain Sciences · doi:10.1017/s0140525x17001972 · Cited by 444
Why, When and How to Adjust Your P Values?
Jafari, M., & Ansari‐Pour, N. (2019) · PubMed · doi:10.22074/cellj.2019.5992 · Cited by 468
On doing better science: From thrill of discovery to policy implications
Antonakis, J. (2017) · The Leadership Quarterly · doi:10.1016/j.leaqua.2017.01.006 · Cited by 475
Soto, C. J. (2019) · Psychological Science · doi:10.1177/0956797619831612 · Cited by 470
Yeager, D. S., Fong, C. J., Lee, H. Y., & Espelage, D. L. (2015) · Journal of Applied Developmental Psychology · doi:10.1016/j.appdev.2014.11.005 · Cited by 472
Detecting and avoiding likely false‐positive findings – a practical guide
Forstmeier, W., Wagenmakers, E., & Parker, T. H. (2017) · Biological Reviews · doi:10.1111/brv.12315 · Cited by 436
KALLA, J. L., & BROOCKMAN, D. E. (2018) · American Political Science Review · doi:10.1017/s0003055417000363 · Cited by 454
“…to authors of previous experiments and listservs yielded at least five additional experiments with null effects that have not been written up (Franco, Malhotra, and Simonovits 2014). are clearly positive on average.17 However, Panel 1(c) shows that in the two existing studies that examined…”
ResultTransparency, Reproducibility, and the Credibility of Economics Research
Christensen, G., & Miguel, E. (2018) · Journal of Economic Literature · doi:10.1257/jel.20171350 · Cited by 446
“Figure 1 reproduces some of the main patterns from Franco, Malhotra, and Simonovits (2014), as described in (Mervis 2014b).”
InfluentialCan emotional intelligence be trained? A meta-analytical investigation
Mattingly, V., & Kraiger, K. (2019) · Human Resource Management Review · doi:10.1016/j.hrmr.2018.03.002 · Cited by 420
Boundary Work among Groups, Occupations, and Organizations: From Cartography to Process
Langley, A., Lindberg, K., Mørk, B. E., Nicolini, D., Raviola, E., & Walter, L. (2019) · Academy of Management Annals · doi:10.5465/annals.2017.0089 · Cited by 421
Macnamara, B. N., & Burgoyne, A. P. (2023) · Psychological Bulletin · doi:10.1037/bul0000352 · Cited by 415
Early childhood environmental education: A systematic review of the research literature
Ardoin, N. M., & Bowers, A. W. (2020) · Educational Research Review · doi:10.1016/j.edurev.2020.100353 · Cited by 405
Identification of and correction for publication bias
Andrews, I., & Kasy, M. (2019) · American Economic Review · doi:10.1257/aer.20180310 · Cited by 388
Is Ego Depletion Real? An Analysis of Arguments
Friese, M., Loschelder, D. D., Gieseler, K., Frankenbach, J., & Inzlicht, M. (2019) · Personality and Social Psychology Review · doi:10.1177/1088868318762183 · Cited by 393
Swedberg, R. (2020) · The Production of Knowledge · doi:10.1017/9781108762519.002 · Cited by 293
Investigating the replicability of preclinical cancer biology
Errington, T. M., Mathur, M., Soderberg, C. K., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.71601 · Cited by 462
Meta-analysis of pro-environmental behaviour spillover
Maki, A., Carrico, A. R., Raimi, K. T., Truelove, H. B., Araujo, B., & Yeung, K. L. (2019) · Nature Sustainability · doi:10.1038/s41893-019-0263-9 · Cited by 377
Social Media, Echo Chambers, and Political Polarization
Barberá, P. (2020) · Social Media and Democracy · doi:10.1017/9781108890960.004 · Cited by 373
An Excess of Positive Results: Comparing the Standard Psychology Literature With Registered Reports
Scheel, A. M., Schijen, M. R. M. J., & Lakens, D. (2021) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459211007467 · Cited by 370
“…and in longitudinal data from a National Science Foundation grant program that found a file-drawering effect for studies with negative results (Franco et al., 2014, 2016), and QRPs have been admitted by scientists in several survey studies (Agnoli et al., 2017; Fiedler & Schwarz, 2016; Fraser…”
BackgroundThe Effects of Youth Mentoring Programs: A Meta-analysis of Outcome Studies
Raposa, E. B., Rhodes, J., Stams, G. J. J. M., Card, N., Burton, S., Schwartz, S., Sykes, L. A. Y., Kanchewa, S., Kupersmidt, J., & Hussain, S. (2019) · Journal of Youth and Adolescence · doi:10.1007/s10964-019-00982-8 · Cited by 366
“One recent study showed that strong, statistically significant results are 40 percent more likely to be published than null results (Franco et al. 2014).”
ResultSensitivity Analysis for Publication Bias in Meta-Analyses
Mathur, M. B., & VanderWeele, T. J. (2020) · Journal of the Royal Statistical Society Series C: Applied Statistics · doi:10.1111/rssc.12440 · Cited by 354
“, 2004; Coursol and Wagner, 1986) and selective submission of papers (Franco et al., 2014; Greenwald, 1975), as well as editors’ and reviewers’ biases (although some empirical work suggests that the latter may be a weaker influence than author decision making; see Lee et al.”
BackgroundSimonsohn, U., Simmons, J. P., & Nelson, L. D. (2015) · Journal of Experimental Psychology General · doi:10.1037/xge0000104 · Cited by 343
“Franco, Malhotra, and Simonovits (2014) obtained information on all studies conducted using an online platform funded by the National Science Foundation (NSF), and examined which subset of studies was written up, submitted, and published.”
Publication bias examined in meta-analyses from psychology and medicine: A meta-meta-analysis
van Aert, R. C. M., Wicherts, J. M., & van Assen, M. A. L. M. (2019) · PLOS ONE · doi:10.1371/journal.pone.0215052 · Cited by 334
“Franco, Malhotra, and Simonovits [11] examined publication bias in studies that received a grant within the social sciences and found that 64.”
BackgroundMethodInfluentialDoes Self-Control Training Improve Self-Control? A Meta-Analysis
Friese, M., Frankenbach, J., Job, V., & Loschelder, D. D. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617697076 · Cited by 334
Multinationals’ profit response to tax differentials: Effect size and shifting channels
Heckemeyer, J. H., & Overesch, M. (2017) · Canadian Journal of Economics/Revue canadienne d'économique · doi:10.1111/caje.12283 · Cited by 322
The Impact of Performance Management on Performance in Public Organizations: A Meta‐Analysis
Gerrish, E. (2016) · Public Administration Review · doi:10.1111/puar.12433 · Cited by 319
Beland, L.-P., Brodeur, A., & Wright, T. (2023) · PLOS ONE · doi:10.1371/journal.pone.0270341 · Cited by 321
Veli Korkmaz, A., van Engen, M. L., Knappert, L., & Schalk, R. (2022) · Human Resource Management Review · doi:10.1016/j.hrmr.2022.100894 · Cited by 314
Partial least squares path modeling: Time for some serious second thoughts
Rönkkö, M., McIntosh, C. N., Antonakis, J., & Edwards, J. R. (2016) · Journal of Operations Management · doi:10.1016/j.jom.2016.05.002 · Cited by 303
“…methodological shortcomings of PLS may actually contribute to its staying power; that is, given the pervasive phenomenon of publication bias (Franco et al., 2014), both the bias away from zero (owing to capitalization on chance) and the absence of evidence against the model (due to a lack…”
BackgroundSocial Media and Democracy: The State of the Field, Prospects for Reform
(2020) · Cambridge University Press eBooks · doi:10.1017/9781108890960 · Cited by 295
Empirical failures of the claim that autistic people lack a theory of mind.
Gernsbacher, M. A., & Yergeau, M. (2019) · Archives of Scientific Psychology · doi:10.1037/arc0000067 · Cited by 295
Information asymmetry and market power in the African banking industry
Boateng, A., Asongu, S., Akamavi, R., & Tchamyou, V. (2018) · Journal of Multinational Financial Management · doi:10.1016/j.mulfin.2017.11.002 · Cited by 292
“results are discarded in favour of strong findings (Rosenberg, 2005; Franco et al., 2014).”
ResultSoderberg, C. K., Errington, T. M., Schiavone, S. R., Bottesini, J., Thorn, F. S., Vazire, S., Esterling, K. M., & Nosek, B. A. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01142-4 · Cited by 295
Detecting Selection Bias in Meta-Analyses with Multiple Outcomes: A Simulation Study
Fernández-Castilla, B., Declercq, L., Jamshidi, L., Beretvas, S. N., Onghena, P., & Van den Noortgate, W. (2021) · The Journal of Experimental Education · doi:10.1080/00220973.2019.1582470 · Cited by 285
Publication bias and the canonization of false facts
Nissen, S. B., Magidson, T., Gross, K., & Bergstrom, C. T. (2016) · eLife · doi:10.7554/elife.21451 · Cited by 266
Comparing meta-analyses and preregistered multiple-laboratory replication projects
Kvarven, A., Strømland, E., & Johannesson, M. (2019) · Nature Human Behaviour · doi:10.1038/s41562-019-0787-z · Cited by 271
Quintana, D. S., Lischke, A., Grace, S., Scheele, D., Ma, Y., & Becker, B. (2021) · Molecular Psychiatry · doi:10.1038/s41380-020-00864-7 · Cited by 263
Efficacy of Command-and-Control and Market-Based Environmental Regulation in Developing Countries
Blackman, A., Li, Z., & Liu, A. A. (2018) · Annual Review of Resource Economics · doi:10.1146/annurev-resource-100517-023144 · Cited by 258
Considering bias and conflicts of interest among the included studies
Boutron, I., Page, M. J., Higgins, J. P., Altman, D. G., Lundh, A., Hróbjartsson, A., & on behalf of the Cochrane Bias Methods Group (2019) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9781119536604.ch7 · Cited by 258
Models Are Stupid, and We Need More of Them
Smaldino, P. E. (2017) · Computational Social Psychology · doi:10.4324/9781315173726-14 · Cited by 236
“Some propose that all results should be published, to ensure that a "file drawer effect" doesn't lead to over-representation of positive results (Franco et al. 2014), while others are skeptical of the value of failed replications because replication studies may have diminished power (Kahneman 2011; Bissell 2013; Schna…”
BackgroundHow Much Can We Generalize From Impact Evaluations?
Vivalt, E. (2020) · Journal of the European Economic Association · doi:10.1093/jeea/jvaa019 · Cited by 237
Reviewing the Assumptions About Men’s Mental Health: An Exploration of the Gender Binary
Smith, D. T., Mouzon, D. M., & Elliott, M. (2018) · American Journal of Men's Health · doi:10.1177/1557988316630953 · Cited by 243
Guidelines for wrist-worn consumer wearable assessment of heart rate in biobehavioral research
Nelson, B. W., Low, C. A., Jacobson, N., Areán, P., Torous, J., & Allen, N. B. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0297-4 · Cited by 241
Trends in sensitivity analysis practice in the last decade
Ferretti, F., Saltelli, A., & Tarantola, S. (2016) · Science of The Total Environment · doi:10.1016/j.scitotenv.2016.02.133 · Cited by 241
Lomas, T., Medina, J. C., Ivtzan, I., Rupprecht, S., & Eiroa-Orosa, F. J. (2019) · The Journal of Positive Psychology · doi:10.1080/17439760.2018.1519588 · Cited by 236
Do fiscal rules constrain fiscal policy? A meta-regression-analysis
Heinemann, F., Moessinger, M.-D., & Yeter, M. (2018) · European Journal of Political Economy · doi:10.1016/j.ejpoleco.2017.03.008 · Cited by 231
Hutton, C., Wagener, T., Freer, J., Han, D., Duffy, C., & Arheimer, B. (2016) · Water Resources Research · doi:10.1002/2016wr019285 · Cited by 233
Mock juror sampling issues in jury simulation research: a meta-analysis.
Bornstein, B. H., Golding, J. M., Neuschatz, J., Kimbrough, C., Reed, K., Magyarics, C., & Luecht, K. (2017) · Law and Human Behavior · doi:10.1037/lhb0000223 · Cited by 227
The effectiveness of forest conservation policies and programs
Börner, J., Schulz, D., Wunder, S., & Pfaff, A. (2020) · Annual Review of Resource Economics · doi:10.1146/annurev-resource-110119-025703 · Cited by 225
Optimal Inference After Model Selection
Fithian, W., Sun, D. L., & Taylor, J. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1410.2597 · Cited by 205
“However, only the large estimates are ever published—a caricature which may not be too far from the truth, as recently demonstrated by Franco et al. (2014).”
BackgroundAuspurg, K., Schneck, A., & Hinz, T. (2019) · Journal of Ethnic and Migration Studies · doi:10.1080/1369183x.2018.1489223 · Cited by 225
Modelling science trustworthiness under publish or perish pressure
Grimes, D. R., Bauch, C. T., & Ioannidis, J. P. A. (2018) · Royal Society Open Science · doi:10.1098/rsos.171511 · Cited by 225
An Agenda for Open Science in Communication
Dienlin, T., Johannes, N., Bowman, N. D., Masur, P. K., Engesser, S., Kümpel, A. S., Lukito, J., Bier, L. M., Zhang, R., Johnson, B. K., et al. (2021) · Journal of Communication · doi:10.1093/joc/jqz052 · Cited by 209
“Similarly, there exist several accounts of publication bias in Communication (Franco, Malhotra, & Simonovits, 2014; Keating & Totzkay, 2019), which indicates a preference for novel and statistically significant results.”
BackgroundBurgess, G. C., Kandala, S., Nolan, D., Laumann, T. O., Power, J. D., Adeyemo, B., Harms, M. P., Petersen, S. E., & Barch, D. M. (2016) · Brain Connectivity · doi:10.1089/brain.2016.0435 · Cited by 216
“Unfortunately, the publication bias in scientific literature (Franco et al., 2014) may make it relatively easy to report significant correlations resulting from motion-correlated artifacts, but harder to re-fute such findings with correlations that are not significant after denoising.”
van den Noort, M., Struys, E., Bosch, P., Jaswetz, L., Perriard, B., Yeo, S., Barisch, P., Vermeire, K., Lee, S.-H., & Lim, S. (2019) · Behavioral Sciences · doi:10.3390/bs9030027 · Cited by 211
Misinformation, Disinformation, and Online Propaganda
Guess, A. M., & Lyons, B. A. (2020) · Social Media and Democracy · doi:10.1017/9781108890960.003 · Cited by 284
Walker, D. W., Smigaj, M., & Tani, M. (2021) · WIREs Water · doi:10.1002/wat2.1488 · Cited by 205
Johnson, B. T., & Hennessy, E. A. (2019) · Social Science & Medicine · doi:10.1016/j.socscimed.2019.05.035 · Cited by 201
“…biases such that studies with significant and positive effects are more likely to be submitted for publication and eventually published, and, as a result, the synthesis outcome is somewhat inflated (Dwan et al., 2008; Easterbrook et al., 1991; Fanelli, 2012; Franco et al., 2014; Soc Sci Med.”
NGOs and international development: A review of thirty-five years of scholarship
Brass, J. N., Longhofer, W., Robinson, R. S., & Schnable, A. (2018) · World Development · doi:10.1016/j.worlddev.2018.07.016 · Cited by 200
Star Wars: The Empirics Strike Back
Brodeur, A., Lé, M., Sangnier, M., & Zylberberg, Y. (2016) · American Economic Journal: Applied Economics · doi:10.1257/app.20150044 · Cited by 430
Hermsen, S., Frost, J., Renes, R. J., & Kerkhof, P. (2016) · Computers in Human Behavior · doi:10.1016/j.chb.2015.12.023 · Cited by 194
Hardwicke, T. E., Wallach, J. D., Kidwell, M. C., Bendixen, T., Crüwell, S., & Ioannidis, J. P. A. (2020) · Royal Society Open Science · doi:10.1098/rsos.190806 · Cited by 217
The Tao of open science for ecology
Hampton, S. E., Anderson, S. S., Bagby, S. C., Gries, C., Han, X., Hart, E. M., Jones, M. B., Lenhardt, W. C., MacDonald, A., Michener, W. K., et al. (2015) · Ecosphere · doi:10.1890/es14-00402.1 · Cited by 187
“…example, authors’ reluctance to submit null results for publication leads to a "file-drawer" effect that can not only systematically bias the published literature (Iyengar and Greenhouse 1988, Franco et al. 2014), but also allows independent scientists to go repeatedly down the same blind alleys.”
BackgroundA literature review of technical analysis on stock markets
Farias Nazário, R. T., e Silva, J. L., Sobreiro, V. A., & Kimura, H. (2017) · The Quarterly Review of Economics and Finance · doi:10.1016/j.qref.2017.01.014 · Cited by 192
Participant Outcomes of Biodiversity Citizen Science Projects: A Systematic Literature Review
Peter, M., Diekötter, T., & Kremer, K. (2019) · Sustainability · doi:10.3390/su11102780 · Cited by 190
RCTs to Scale: Comprehensive Evidence from Two Nudge Units
DellaVigna, S., & Linos, E. (2020) · National Bureau of Economic Research · doi:10.3386/w27594 · Cited by 189
Burnout in medical students: a systematic review of experiences in Chinese medical schools
Chunming, W. M., Harrison, R., MacIntyre, R., Travaglia, J., & Balasooriya, C. (2017) · BMC Medical Education · doi:10.1186/s12909-017-1064-3 · Cited by 187
Economic Benefits of Empowering Women in Agriculture: Assumptions and Evidence
Anderson, C. L., Reynolds, T. W., Biscaye, P., Patwardhan, V., & Schmidt, C. (2021) · The Journal of Development Studies · doi:10.1080/00220388.2020.1769071 · Cited by 185
Trevethan, R. (2017) · Health Services and Outcomes Research Methodology · doi:10.1007/s10742-016-0156-6 · Cited by 181
“…inappropriately choosing the averaged output makes results look more impressive for both researchers and their colleagues, and journals are more likely to accept impressive results—the well-known positive publication bias (see Franco et al. 2014)—so it might be an attractive course of action.”
Reducing bias, increasing transparency and calibrating confidence with preregistration
Hardwicke, T. E., & Wagenmakers, E.-J. (2023) · Nature Human Behaviour · doi:10.1038/s41562-022-01497-2 · Cited by 179
The effects of educational robotics in STEM education: a multilevel meta-analysis
Ouyang, F., & Xu, W. (2024) · International Journal of STEM Education · doi:10.1186/s40594-024-00469-4 · Cited by 171
“Since studies with positive findings were more likely to be published, we checked if there was any publication bias in this meta-analysis (Franco et al., 2014; Thornton & Lee, 2000).”
BackgroundEstimating the Reproducibility of Experimental Philosophy
Cova, F., Strickland, B., Abatista, A., Allard, A., Andow, J., Attie, M., Beebe, J., Berniūnas, R., Boudesseul, J., Colombo, M., et al. (2021) · Review of Philosophy and Psychology · doi:10.1007/s13164-018-0400-9 · Cited by 179
“As many have argued, the systematic non-publication of null results – which contributes to the so-called filedrawer effect – is a leading factor in increasing the proportion of false positives in the literature and thus of non-replicable effects (Earp 2017; Franco et al. 2014; Rosenthal 1979).”
BackgroundProtecting against researcher bias in secondary data analysis: challenges and potential solutions
Baldwin, J. R., Pingault, J.-B., Schoeler, T., Sallis, H. M., & Munafò, M. R. (2022) · European Journal of Epidemiology · doi:10.1007/s10654-021-00839-0 · Cited by 178
“While this is a positive outcome for research integrity, null results are less likely to be published [13, 41, 42] and cited [11], which could disadvantage researchers’ careers.”
BackgroundResultSmithers, L. G., Sawyer, A. C. P., Chittleborough, C. R., Davies, N. M., Davey Smith, G., & Lynch, J. W. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0461-x · Cited by 175
The Preregistration Revolution
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2017) · doi:10.31219/osf.io/2dxu5 · Cited by 168
Conservatives and liberals have similar physiological responses to threats
Bakker, B. N., Schumacher, G., Gothreau, C., & Arceneaux, K. (2020) · Nature Human Behaviour · doi:10.1038/s41562-020-0823-z · Cited by 165
Metadata
- Authors
- Franco, A., Malhotra, N., & Simonovits, G.
- Affiliations
- Stanford University (US)
- Department of Political Science, Stanford University, Stanford, CA, USA.
- Corresponding author
- Malhotra, Neil
- ORCID
- Malhotra, Neil
- Venue
- Science
- Published
- 2014-09-19
- Volume
- 345
- Issue
- 6203
- Pages
- 1502-1505
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W1997809363
- Cited by
- 1,542
- References
- 38
- Citations by year
- 2026: 88 · 2025: 152 · 2024: 143 · 2023: 162 · 2022: 129 · 2021: 152 · 2020: 212 · 2019: 123 · 2018: 119 · 2017: 110 · 2016: 91 · 2015: 52 · 2014: 9
- FWCI
- 85.56
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- Open Research (Auckland University of Technology) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- publication bias, negative results, file drawer problem, social sciences, survey experiment, peer review
- MeSH terms
- Social Sciences, Publication Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.