Publication bias in the social sciences: Unlocking the file drawer
The DOI matches its Crossref record; 37 of the 38 references it cites were found in a registry or an index.
Abstract
The file drawer is full. Should we worry?: Experiments that produce null results face a higher barrier to publication than those that yield statistically significant differences. Whether this is a problem depends on how many null but otherwise valid results might be trapped in the file drawer. Franco et al. use a Time-sharing Experiments in the Social Sciences archive of nearly 250 peer-reviewed proposals of social science experiments conducted on nationally representative samples. They find that only 10 out of 48 null results were published, whereas 56 out of 91 studies with strongly significant results made it into a journal. Science , this issue p. 1502
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Selective reporting, questionable research practices, and low-powered research all contribute to the unreliability of findings in a range of disciplines”
From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as Franco et al., 2014
In the citing paper
“In particular, negative or non-significant results are far less likely to be published (Franco et al., 2014).”
From What do we know about interventions to improve educator wellbeing? A systematic literature review (2024) · cites this paper as Franco et al., 2014
In the citing paper
“The bias against negative results appears to be a consequence of both editors and reviewers finding them less compelling for publication and from authors themselves not trying to publish them 8-12”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [8]
In the citing paper
“Positive results -- finding that a treatment is effective or that there is a relationship between two variables -- are more likely to be published than negative results 6-12 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [8]
In the citing paper
“A possible interpretation of these results is that the missing effect sizes are mainly a file‐drawer problem (Rosenthal, 1979). This interpretation is consistent with the evidence presented in Franco et al. (2014), which indicates that the file‐drawer problem is the main culprit behind publication bias in the social sciences.”
From Targeted school‐based interventions for improving reading and mathematics for students with or at risk of academic difficulties in Grades K‐6: A systematic review (2021) · cites this paper as Franco et al. (2014)
3 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722 · cited 1×
“In the presence of publication bias smaller studies report larger effects in order to exceed arbitrary significance thresholds (19, 20).”
BackgroundThe file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
Meta-analysis in social research
(1982) · Evaluation and Program Planning · doi:10.1016/0149-7189(82)90011-8 · Cited by 3,853
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109
Publication Bias in Meta‐Analysis: Prevention, Assessment and Adjustments
(2005) · doi:10.1002/0470870168 · Cited by 2,368
The case of the misleading funnel plot
Lau, J., Ioannidis, J. P. A., Terrin, N., Schmid, C. H., & Olkin, I. (2006) · BMJ · doi:10.1136/bmj.333.7568.597 · Cited by 1,766 · cited 1×
“However, these visualization-based approaches are sensitive to using different measures of precision (21, 22) and also assume outcome variables and effect sizes are comparable across studies (23).”
Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias
Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A.-W., Cronin, E., Decullier, E., Easterbrook, P. J., Von Elm, E., Gamble, C., et al. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003081 · Cited by 1,546 · cited 1×
“A different approach involves examining the publication outcomes of a cohort of studies, either prospectively or retrospectively (24, 25).”
MethodSumming up: The Science of Reviewing Research
Wilkinson, I., Light, R. J., & Pillemer, D. B. (1987) · Journal of Educational Statistics · doi:10.2307/1164691 · Cited by 1,471
Summing Up: The Science of Reviewing Research
Light, R. J., & Pillemer, D. B. (1984) · Harvard University Press eBooks · doi:10.4159/9780674040243 · Cited by 1,427
The Existence of Publication Bias and Risk Factors for Its Occurrence
Dickersin, K. (1990) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1990.03440100097014 · Cited by 1,417 · cited 1×
“Publication bias occurs when “publication of study results is based on the direction or significance of the findings” (1).”
Background
Research works printed but not linked: 3
- 10.1037/0735-7028.17.2.136 · doi:10.1037/0735-7028.17.2.136 · Paper page
- K. Dickersin in Publication Bias in Meta-Analysis: Prevention Assessment and Adjustments H. R. Rothstein A. J. Sutton M. Borenstein Eds. (Wiley Chichester UK 2006) ch. 2.
- Dickersin K., Min Y. I., Online J. Curr. Clin. Trials 1993, 50 (1993).
Cited by
Items 101–200 of 1,002
A historical review of publication bias
Marks‐Anglin, A., & Chen, Y. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1452 · Cited by 165
Meditation, Mindfulness, and Attention: a Meta-analysis
Sumantry, D., & Stewart, K. E. (2021) · Mindfulness · doi:10.1007/s12671-021-01593-w · Cited by 165
Questions About Questionable Research Practices in the Field of Management: A Guest Commentary
Banks, G. C., O’Boyle, E. H., Pollack, J. M., White, C. D., Batchelor, J. H., Whelpley, C. E., Abston, K. A., Bennett, A. A., & Adkins, C. L. (2016) · Journal of Management · doi:10.1177/0149206315619011 · Cited by 163
Fisher, Neyman-Pearson or NHST? A tutorial for teaching data testing
Perezgonzalez, J. D. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.00223 · Cited by 162
“A sample of helpful readings on this controversy are Christensen (2005), Hubbard (2004), Gigerenzer (2004), Goodman (1999), Louçã (2008, https://www.”
Effectiveness of Multimedia Pedagogical Agents Predicted by Diverse Theories: a Meta-Analysis
Castro-Alonso, J. C., Wong, R. M., Adesope, O. O., & Paas, F. (2021) · Educational Psychology Review · doi:10.1007/s10648-020-09587-1 · Cited by 162
Threats of a replication crisis in empirical computer science
Cockburn, A., Dragicevic, P., Besançon, L., & Gutwin, C. (2020) · Communications of the ACM · doi:10.1145/3360311 · Cited by 155
“16 Publication bias and the file drawer effect combine to propagate the dissemination and maintenance of false knowledge: through the file drawer ef-could not do so in 47 of 53 cases, 3 and psychology researchers similarly failed to replicate 39 out of 100 studies.”
ResultA meta-analysis of signaling principle in multimedia learning environments
Alpizar, D., Adesope, O. O., & Wong, R. M. (2020) · Educational Technology Research and Development · doi:10.1007/s11423-020-09748-7 · Cited by 157
“The literature has reported that studies with non-significant findings do not get published as much as those with significant results (Franco et al. 2014).”
BackgroundKlar, S., Krupnikov, Y., Ryan, J. B., Searles, K., & Shmargad, Y. (2020) · PLoS ONE · doi:10.1371/journal.pone.0229446 · Cited by 157
Pre-Analysis Plans Have Limited Upside, Especially Where Replications Are Feasible
Coffman, L. C., & Niederle, M. (2015) · Journal of Economic Perspectives · doi:10.1257/jep.29.3.81 · Cited by 156
“Franco et al. (2014) show that strong results have a 60 percentage point higher likelihood of being written up, and about a 40 percentage point higher chance to be published than null results.”
ResultOn the Reproducibility of Psychological Science
Johnson, V. E., Payne, R. D., Wang, T., Asher, A., & Mandal, S. (2017) · Journal of the American Statistical Association · doi:10.1080/01621459.2016.1240079 · Cited by 149
“We note that publication bias played a prominent role in the American Statistical Association’s recent statement on statistical significance and p-values (Wasserstein and Lazar 2016), and alarm over its effects on the reproducibility of science is increasing (e.g., Fanelli 2010; Franco et al. 2014; Peplow 2014).”
BackgroundReplication in Criminology and the Social Sciences
Pridemore, W. A., Makel, M. C., & Plucker, J. A. (2017) · Annual Review of Criminology · doi:10.1146/annurev-criminol-032317-091849 · Cited by 169
Meta-Analysis of Emotional Designs in Multimedia Learning: A Replication and Extension Study
Wong, R. M., & Adesope, O. (2020) · Educational Psychology Review · doi:10.1007/s10648-020-09545-x · Cited by 190
“Studies with non-significant findings are either not published as frequently as studies with significant findings (Franco et al. 2014; Rosenthal 1979).”
BackgroundMedina, J., & Cason, S. (2017) · Cortex · doi:10.1016/j.cortex.2017.06.021 · Cited by 168
Unbiased, reliable, and valid student evaluations can still be unfair
Esarey, J., & Valdes, N. (2020) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2020.1724875 · Cited by 164
“…Cohen (1981) and Feldman (1989) and find that their results are influenced by the tendency for small studies with exaggeratedly positive results to be published while null results are ignored (Schwarzer et al., 2015, Chapter 5; see also Sterling, 1959; Sterling et al., 1995; Franco et al., 2014).”
BackgroundGoldschmidt, A. B. (2017) · Obesity Reviews · doi:10.1111/obr.12491 · Cited by 141
“There were limited data addressing the discriminant validity of LOC and overeating, which may partially reflect a bias against publishing null findings (93).”
BackgroundPredict science to improve science
DellaVigna, S., Pope, D., & Vivalt, E. (2019) · Science · doi:10.1126/science.aaz1704 · Cited by 136
Editorial: preclinical data reproducibility for R&D - the challenge for neuroscience
Steckler, T. (2015) · SpringerPlus · doi:10.1186/2193-1801-4-1 · Cited by 140
Social capital and health: a meta-analysis
Xue, X., Reed, W. R., & Menclova, A. (2020) · Journal of Health Economics · doi:10.1016/j.jhealeco.2020.102317 · Cited by 161
The Replication Crisis Has Led to Positive Structural, Procedural, and Community Changes
Korbmacher, M., Azevedo, F., Pennington, C. R., Hartmann, H., Pownall, M., Schmidt, K., Elsherif, M., Breznau, N., Robertson, O., Kalandadze, T., et al. (2023) · Communications Psychology · doi:10.1038/s44271-023-00003-2 · Cited by 137
“Negative fi ndings can often end in the ‘ fi le-drawer ’ and are never published 223 .”
BackgroundConsumer interaction with sustainability labelling on food products: a narrative literature review
Cook, B., Costa Leite, J., Rayner, M., Stoffel, S., van Rijn, E., & Wollgast, J. (2023) · Nutrients · doi:10.3390/nu15173837 · Cited by 134
“In a further step, to address the publication bias in academic research whereby studies with null results are less likely to be written up [12], the research team developed a short online questionnaire and distributed it among 192 European sustainability-related label owners and third-party certification agencies, ide…”
BackgroundA kinked meta‐regression model for publication bias correction
Bom, P. R. D., & Rachinger, H. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1352 · Cited by 133
The own-wage elasticity of labor demand: A meta-regression analysis
Lichter, A., Peichl, A., & Siegloch, S. (2015) · European Economic Review · doi:10.1016/j.euroecorev.2015.08.007 · Cited by 133
“In addition to identifying sources of heterogeneity, we further explicitly test for publication selection (or reporting) bias given that journals’ preference to publish statistically significant results (DeLong and Lang, 1992) and economists’ strong beliefs in particular economic relationships might prompt researchers…”
The Production of Knowledge: Enhancing Progress in Social Science
(2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519 · Cited by 241
Strengthening the Practice of Exercise and Sport-Science Research
Halperin, I., Vigotsky, A. D., Foster, C., & Pyne, D. B. (2018) · International Journal of Sports Physiology and Performance · doi:10.1123/ijspp.2017-0322 · Cited by 130
“38,41 Authors may also decide against attempting to publish nonsigni fi cant results because leading journals have a high rejection rate of negative results.”
BackgroundImpacts and Determinants of Health Levels in Low-Income Countries
Dupas, P., & Miguel, E. A. (2016) · Handbook of economic field experiments · doi:10.1016/bs.hefe.2016.09.003 · Cited by 202
“It also helps scholars locate studies that are delayed in publication, or are never published, helping to fill in gaps in the literature and thus resolving some of the problems of “disappearing” null results identified in Franco et al (2014).”
BackgroundGaskin, C. J., Taylor, D., Kinnear, S. H. W., Mann, J., Hillman, W., & Moran, M. (2017) · Weather Climate and Society · doi:10.1175/wcas-d-16-0126.1 · Cited by 153
“With research showing that there is a bias toward publishing studies with stronger, statistically significant effects (Franco et al. 2014; McAuley et al. 2000), the incorporation of gray literature into systematic reviews represents an attempt to assess the whole evidence base. b. Selection…”
BackgroundDoes self-control improve with practice? Evidence from a six-week training program.
Miles, E., Sheeran, P., Baird, H., Macdonald, I., Webb, T. L., & Harris, P. R. (2016) · Journal of Experimental Psychology: General · doi:10.1037/xge0000185 · Cited by 128
Promoting Open Science to Increase the Trustworthiness of Evidence in Special Education
Adelson, J. L., Barton, E., Bradshaw, C., Bryant, B. R., Bryant, D., Cook, B. G., Coyne, M., DeBettencourt, L., DeHaven, A. C., Dymond, S., et al. (2018) · Exceptional Children · doi:10.1177/0014402918793138 · Cited by 143
Community policing does not build citizen trust in police or reduce crime in the Global South
Blair, G., Weinstein, J. M., Christia, F., Arias, E., Badran, E., Blair, R., Cheema, A., Farooqui, A. Z., Fetzer, T., Grossman, G., et al. (2021) · Science · doi:10.1126/science.abd3446 · Cited by 222
“Third, we designed the six studies jointly, preregis-tered them, and implemented them during the sameperiod,thusincreasingourconfidencein thecomparabilityoftheresultsandavoiding thewidespreadchallengeofpublicationbias ( 36 ).”
Asongu, S. A., Nnanna, J., & Acha-Anyi, P. N. (2020) · Economic Analysis and Policy · doi:10.1016/j.eap.2020.01.002 · Cited by 150
Tipping point detection and early warnings in climate, ecological, and human systems
Dakos, V., Boulton, C. A., Buxton, J. E., Abrams, J. F., Nava, B. A., McKay, D. I. A., Bathiany, S., Blaschke, L. L., Boers, N., Dylewsky, D., et al. (2024) · Earth System Dynamics · doi:10.5194/esd-15-1117-2024 · Cited by 160
Conceptualizing and evaluating replication across domains of behavioral research
Tackett, J. L., & McShane, B. B. (2018) · Behavioral and Brain Sciences · doi:10.1017/s0140525x18000882 · Cited by 124
Replication Research, Publication Bias, and Applied Behavior Analysis
Tincani, M., & Travers, J. C. (2019) · Perspectives on Behavior Science · doi:10.1007/s40614-019-00191-5 · Cited by 144
Stepenuck, K. F., & Green, L. T. (2015) · Ecology and Society · doi:10.5751/es-07329-200319 · Cited by 150
P-hacking and publication bias interact to distort meta-analytic effect size estimates
Friese, M., & Frankenbach, J. (2020) · Psychological Methods · doi:10.1037/met0000246 · Cited by 122
Replication studies in economics—How many and which papers are chosen for replication, and why?
Mueller‐Langer, F., Fecher, B., Harhoff, D., & Wagner, G. G. (2018) · Research Policy · doi:10.1016/j.respol.2018.07.019 · Cited by 145
“Due to the large number of publications and researchers' bias toward finishing and publishing positive findings, the problem of false positives is likely to persist (Coffman and Niederle, 2015; Franco et al., 2014; Ioannidis, 2005).”
Publication Bias in Special Education Meta-Analyses
Gage, N. A., Cook, B. G., & Reichow, B. (2017) · Exceptional Children · doi:10.1177/0014402917691016 · Cited by 119
“…review studies with small effects or null findings (Neuliep & Crandall, 1993; Shadish et al., 2016), other studies have shown that the larger problem lies with authors choosing not to submit their research with small or null findings for publication (Franco et al., 2014; Olson et al., 2002).”
BackgroundInfluentialSiegel, A. A. (2020) · Social Media and Democracy · doi:10.1017/9781108890960.005 · Cited by 99
Data, measurement and empirical methods in the science of science
Liu, L., Jones, B. F., Uzzi, B., & Wang, D. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01562-4 · Cited by 182
“The publication of negative results is also rare (the file drawer problem) 72,73 .”
BackgroundUnderreporting in Psychology Experiments: Evidence From a Study Registry
Franco, A., Malhotra, N., & Simonovits, G. (2016) · Social Psychological and Personality Science · doi:10.1177/1948550615598377 · Cited by 117
“Although the existence of publication bias has been documented in psychology (Cooper, DeNeve, & Charlton, 1997) and the social sciences in general (Franco et al., 2014), direct evidence of underreporting in published research is scarce (see Franco, Malhotra, & Simonovits, 2015, for a study of…”
BackgroundSelf-citationReplication Research and Special Education
Travers, J. C., Cook, B. G., Therrien, W. J., & Coyne, M. D. (2016) · Remedial and Special Education · doi:10.1177/0741932516648462 · Cited by 125
“Franco, Malhotra, and Simonovits (2014), for example, reported rigorous studies that fail to find an effect are much less likely to be published than studies that find effects with less rigorous methods.”
BackgroundReplication, Communication, and the Population Dynamics of Scientific Discovery
McElreath, R., & Smaldino, P. E. (2015) · PLOS ONE · doi:10.1371/journal.pone.0136088 · Cited by 117
“And it is not yet easy to acquire empirical evidence about these factors, as even the best empirical studies of publication bias still rely upon researcher self-report [3].”
BackgroundAre Psychology Journals Anti-replication? A Snapshot of Editorial Practices
Martin, G. N., & Clarke, R. M. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.00523 · Cited by 116
“, 2011) and null findings are less likely to be submitted or accepted for publication as a result (Rosenthal, 1979; Franco et al., 2014).”
BackgroundThe diversity Wave:A meta-analysis of the native-born white response to ethnic diversity
Kaufmann, E., & Goodwin, M. (2018) · Social Science Research · doi:10.1016/j.ssresearch.2018.07.008 · Cited by 138
Consortium for Transparency in Exercise Science (COTES) Collaborators, Caldwell, A. R., Vigotsky, A. D., Tenan, M. S., Radel, R., Mellor, D. T., Kreutzer, A., Lahart, I. M., Mills, J. P., & Boisgontier, M. P. (2020) · Sports Medicine · doi:10.1007/s40279-019-01227-1 · Cited by 114
Norris, J. M. (2015) · Language Learning · doi:10.1111/lang.12114 · Cited by 137
Brain volume and intelligence: The moderating role of intelligence measurement quality
Gignac, G. E., & Bates, T. C. (2017) · Intelligence · doi:10.1016/j.intell.2017.06.004 · Cited by 155
Kluve, J., Puerto, S., Robalino, D., Romero, J. M., Rother, F., Stöterau, J., Weidenkaff, F., & Witte, M. (2017) · Campbell Systematic Reviews: Better Evidence for a Better World · doi:10.4073/csr.2017.12 · Cited by 106
Executive functions in trauma-exposed youth: a meta-analysis
Op den Kelder, R., Van den Akker, A. L., Geurts, H. M., Lindauer, R. J. L., & Overbeek, G. (2018) · European Journal of Psychotraumatology · doi:10.1080/20008198.2018.1450595 · Cited by 110
Quantifying Support for the Null Hypothesis in Psychology: An Empirical Investigation
Aczél, B., Pálfi, B., Szollosi, A., Kovács, M., Szászi, B., Szécsi, P., Zrubka, M., Gronau, Q. F., van den Bergh, D., & Wagenmakers, E. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918773742 · Cited by 132
Huang, K., Beckman, E. M., Ng, N., Dingle, G. A., Han, R., James, K. E., WINKLER, E. A. H., Stylianou, M., & Gomersall, S. R. (2024) · Health Promotion International · doi:10.1093/heapro/daae054 · Cited by 204
“Publication bias, where studies with little to no effect tend to remain unpub-lished (Franco et al ., 2014), was considered by testing for small-study effects, and providing corrected estimates where needed, but these approaches have limitations, and it is still possible that the inclusion of…”
BackgroundP-Values: Misunderstood and Misused
Vidgen, B., & Yasseri, T. (2016) · Frontiers in Physics · doi:10.3389/fphy.2016.00006 · Cited by 143
Edelsbrunner, P. A., Simonsmeier, B. A., & Schneider, M. (2025) · Educational Psychology Review · doi:10.1007/s10648-024-09982-y · Cited by 104
“…not reach standard cut-offs such as 0.70 or 0.80, it can lead to bias in several ways: Researchers might not submit the study for publication or get rejected from journals, contributing to file drawer bias (Franco et al., 2014), or they might remove, select, or re-write items to increase Alpha.”
InfluentialCorrecting for Publication Bias in a Meta-Analysis with the P-uniform* Method
van Aert, R. C. M., & van Assen, M. A. L. M. (2018) · doi:10.31222/osf.io/zqjr9 · Cited by 105
Improving natural product research translation: From source to clinical trial
Sorkin, B. C., Kuszak, A. J., Bloss, G., Fukagawa, N. K., Hoffman, F. A., Jafari, M., Barrett, B. P., Brown, P. N., Bushman, F. D., Casper, S. J., et al. (2019) · The FASEB Journal · doi:10.1096/fj.201902143r · Cited by 121
Standards for Experimental Research: Encouraging a Better Understanding of Experimental Methods
Mutz, D. C., & Pemantle, R. (2015) · Journal of Experimental Political Science · doi:10.1017/xps.2015.4 · Cited by 120
Predicting replication outcomes in the Many Labs 2 study
Forsell, E., Viganola, D., Pfeiffer, T., Almenberg, J., Wilson, B., Chen, Y., Nosek, B. A., Johannesson, M., & Dreber, A. (2019) · Journal of Economic Psychology · doi:10.1016/j.joep.2018.10.009 · Cited by 104
Calibrating the Scientific Ecosystem Through Meta-Research
Hardwicke, T. E., Serghiou, S., Janiaud, P., Danchev, V., Crüwell, S., Goodman, S. N., & Ioannidis, J. P. A. (2020) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-031219-041104 · Cited by 98
“…(e.g., only reporting experiments, outcomes, or analyses that achieved statistical significance) Chan et al. 2004; Dwan et al. 2011, 2013, 2014; Franco et al. 2014, 2016; Goldacre et al. 2019; John et al. 2012; Easterbrook et al. 1991 Presenting post hoc hypotheses as if they were specified a…”
Effects of Plastic Waste Materials on Geotechnical Properties of Clayey Soil
Hassan, H. J. A., Rasul, J. M., & Samin, M. (2021) · Transportation Infrastructure Geotechnology · doi:10.1007/s40515-020-00145-4 · Cited by 128
“The numbers in parenthesis are representing the percentage of improvement in UCS Pal et al. 2015; Sharma 2017; Franco et al. 2014; Louzada et al. 2019; Sai and Srinivas 2019).”
BackgroundMartins, L. L., & Sohn, W. (2021) · Academy of Management Annals · doi:10.5465/annals.2019.0109 · Cited by 125
Global synthesis reveals heterogeneous changes in connection of humans to nature
Soga, M., & Gaston, K. J. (2023) · One Earth · doi:10.1016/j.oneear.2023.01.007 · Cited by 132
Autism and Employment: A Review of the “New Frontier” of Diversity Research
Ezerins, M. E., Simon, L. S., Vogus, T. J., Gabriel, A. S., Calderwood, C., & Rosen, C. C. (2024) · Journal of Management · doi:10.1177/01492063231193362 · Cited by 98
How is extraversion related to social media use? A literature review
Bowden-Green, T., Hinds, J. M., & Joinson, A. (2020) · Personality and Individual Differences · doi:10.1016/j.paid.2020.110040 · Cited by 119
Kasdan, A. V., Burgess, A. N., Pizzagalli, F., Scartozzi, A., Chern, A., Kotz, S. A., Wilson, S. M., & Gordon, R. L. (2022) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2022.104588 · Cited by 121
“Studies reporting null or unexpected results often are not published (though this is rapidly changing, see: Franco et al., 2014; The importance of no evidence, 2019).”
Fifty Shades of QE: Conflicts of Interest in Economic Research
Fabo, B., Jančoková, M., Kempf, E., & Pástor, Ľ. (2020) · Journal of Monetary Economics · doi:10.1016/j.jmoneco.2021.04.001 · Cited by 120
Dunning, T., Grossman, G., Humphreys, M., Hyde, S., McIntosh, C., Nellis, G., Adida, C. L., Arias, E., Bicalho, C., Boas, T. C., et al. (2019) · Science Advances · doi:10.1126/sciadv.aaw2612 · Cited by 127
Research Ethics for Students in the Social Sciences
Bos, J. (2020) · Springer eBooks · doi:10.1007/978-3-030-48415-6 · Cited by 254
Research self-efficacy: A meta-analysis
Livinți, R., Gunnesch‐Luca, G., & Iliescu, D. (2021) · Educational Psychologist · doi:10.1080/00461520.2021.1886103 · Cited by 141
“A first possible moderator is publication status, which could have an impact on our results due to the unfortunate but welldocumented fact that statistically significant results are more likely to be published than those that are not (Franco et al., 2014).”
BackgroundHARK No More: On the Preregistration of CHI Experiments
Cockburn, A., Gutwin, C., & Dix, A. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173715 · Cited by 95
“Publication bias also causes a ‘file drawer’ effect [35, 14], in which studies that do not reach the threshold α level are rejected for publication (or are not submitted).”
BackgroundInfluentialReproducibility and Replicability in Geographical Analysis
Kedron, P., Frazier, A. E., Trgovac, A. B., Nelson, T., & Fotheringham, A. S. (2021) · Geographical Analysis · doi:10.1111/gean.12221 · Cited by 94
Paid online convenience samples in gambling studies: questionable data quality
Pickering, D., & Blaszczynski, A. P. (2021) · International Gambling Studies · doi:10.1080/14459795.2021.1884735 · Cited by 106
“Study pre-registration offers a partial solution to questionable research practices such as file-drawering and p-hacking; and is especially relevant given the ease of data collection using paid online samples (Franco et al., 2014).”
BackgroundWomen on the Run: Gender, Media, and Political Campaigns in a Polarized Era
Hayes, D., & Lawless, J. L. (2016) · Cited by 93
Z-curve 2.0: Estimating Replication Rates and Discovery Rates
Bartos̆, F., & Schimmack, U. (2022) · Meta-Psychology · doi:10.15626/mp.2021.2720 · Cited by 126
Mesquida, C., Murphy, J., Lakens, D., & Warne, J. P. (2022) · Royal Society Open Science · doi:10.1098/rsos.220946 · Cited by 116
“While the above assumes an unbiased p-value distribution, one explanation for an excess of significant findings in a literature body that has been raised is publication bias and p-hacking [13,48,49].”
BackgroundPeña, P. A. (2016) · Economics of Education Review · doi:10.1016/j.econedurev.2016.12.001 · Cited by 91
Distributions of p -values smaller than .05 in psychology: what is going on?
Hartgerink, C. H. J., van Aert, R. C. M., Nuijten, M. B., Wicherts, J. M., & van Assen, M. A. L. M. (2016) · PeerJ · doi:10.7717/peerj.1935 · Cited by 89
“…Publication bias refers to the selective publication of102 disproportionate amounts of statistically significant outcomes (Gerber et al., 2010; Franco et al., 2014).103 Publication bias contributes to a higher frequency of p-values just below .05 relative to the frequency104 of p-values just…”
BackgroundResultSex as an important factor in nanomedicine
Hajipour, M. J., Aghaverdi, H., Serpooshan, V., Vali, H., Sheibani, S., & Mahmoudi, M. (2021) · Nature Communications · doi:10.1038/s41467-021-23230-9 · Cited by 96
The Distribution of School Spending Impacts
Jackson, C. K., & Mackevicius, C. (2021) · doi:10.3386/w28517 · Cited by 89
“Studies that find null results may be less likely to be published than studies that find signif- icant effects (Franco et al. (2014), Christensen and Miguel (2018)).”
BackgroundPorru, F., Robroek, S. J., Bültmann, U., Portoghese, I., Campagna, M., & Burdorf, A. (2020) · Journal of Affective Disorders · doi:10.1016/j.jad.2020.12.183 · Cited by 126
The misuse and abuse of statistics in biomedical research
Thiese, M. S., Arnold, Z. C., & Walker, S. D. (2015) · Biochemia Medica · doi:10.11613/bm.2015.001 · Cited by 87
“There is a proposal for a registry of unpublished social science data that has statistically insignificant results (32).”
BackgroundThe Anatomy of Failure: An Ethnography of a Randomized Trial to Deepen Democracy in Rural India
Rao, V., Ananthpur, K., & Malik, K. (2017) · World Development · doi:10.1016/j.worlddev.2017.05.037 · Cited by 91
Lifestyle Interventions in Preschool Children: A Meta-analysis of Effectiveness
Ling, J., Robbins, L. B., Wen, F., & Zhang, N. (2017) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2017.01.018 · Cited by 91
Unconscious or underpowered? Probabilistic cuing of visual attention.
Vadillo, M. A., Linssen, D., Orgaz, C., Parsons, S., & Shanks, D. R. (2020) · Journal of Experimental Psychology: General · doi:10.1037/xge0000632 · Cited by 87
Harrison, R., Cohen, A., & Walton, M. (2015) · International Journal for Quality in Health Care · doi:10.1093/intqhc/mzv041 · Cited by 101
“published material, important perspectives from non-published and non-empirical workmay have been omitted [62].”
BackgroundManiadis, Z., Tufano, F., & List, J. A. (2017) · The Economic Journal · doi:10.1111/ecoj.12527 · Cited by 105
“But, notice that hard evidence – for example in the form of a study such as Franco et al. (2014), who utilise a unique preregistration pool to derive an estimate of publication bias – has not been provided yet in economics. Finally, is there evidence for the degree to which economic researchers are biased at the indiv…”
Mathur, M. B., & VanderWeele, T. J. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1464 · Cited by 96
What Do Books in the Home Proxy For? A Cautionary Tale
Engzell, P. (2019) · Sociological Methods & Research · doi:10.1177/0049124119826143 · Cited by 105
“…underpowered research designs, flexibility in data collection and analysis, misuse of statistical tests, hypothesizing after results are known, and so on (Bernardi, Chakhaia, and Leopold 2017, Franco, Malhotra, and Simonovits 2014, Silberzahn et al. 2017, Simmons, Nelson, and Simonsohn 2011).”
BackgroundNo consistent cooling of the real hand in the rubber hand illusion
de Haan, A. M., van Stralen, H. E., Smit, M., Keizer, A., der Stigchel, S. V., & Dijkerman, H. C. (2017) · Acta Psychologica · doi:10.1016/j.actpsy.2017.07.003 · Cited by 92
Incorrect Inferences When Using Generated Regressors in Accounting Research
CHEN, W., Hribar, P., & Melessa, S. (2020) · Journal of Accounting Research · doi:10.1111/1475-679x.12470 · Cited by 109
Vasilev, M. R., & Angele, B. (2016) · Psychonomic Bulletin & Review · doi:10.3758/s13423-016-1147-x · Cited by 100
Nagtegaal, R., de Boer, N., van Berkel, R., Derks, B., & Tummers, L. G. (2023) · Journal of Occupational Rehabilitation · doi:10.1007/s10926-022-10076-1 · Cited by 136
“Other reviews have focused on either the negative side [18] or the positive side [19] of hiring people with disabilities, or both [20, 21].”
BackgroundThe impact of sample size on the reproducibility of voxel-based lesion-deficit mappings
Lorca-Puls, D. L., Gajardo-Vidal, A., White, J., Seghier, M. L., Leff, A. P., Green, D. W., Crinion, J. T., Ludersdorfer, P., Hope, T. M. H., Bowman, H., et al. (2018) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2018.03.014 · Cited by 82
“This is known as “the file drawer problem” (Franco et al., 2014; Simonsohn et al., 2014b).”
BackgroundWhere Are the Self-Correcting Mechanisms in Science?
Vazire, S., & Holcombe, A. O. (2022) · Review of General Psychology · doi:10.1177/10892680211033912 · Cited by 81
“Multiple fields of science have a very poor record of publishing negative and null results (Sterling et al., 1995; Fanelli, 2010, 2012; Greenwald, 1975; Hubbard & Armstrong, 1992; Franco et al., 2014; Sterling, 1959; Sterling, Rosenbaum, & Wenkam, 1995), suggesting that the published literature is biased and incomplet…”
BackgroundResultFood parenting practices and children's weight outcomes: A systematic review of prospective studies
Beckers, D., Karssen, L. T., Vink, J. M., Burk, W. J., & Larsen, J. K. (2020) · Appetite · doi:10.1016/j.appet.2020.105010 · Cited by 103
Newman, B., Joseph, K., Chauhan, A., Seale, H., Li, J., Manias, E., Walton, M., Mears, S. R., Jones, B., & Harrison, R. (2021) · Health Expectations · doi:10.1111/hex.13343 · Cited by 108
Unemployment and health: A meta‐analysis
Picchio, M., & Ubaldi, M. (2023) · Journal of Economic Surveys · doi:10.1111/joes.12588 · Cited by 121
Williamson, O., Swann, C., Bennett, K. J. M., Bird, M. D., Goddard, S. G., Schweickle, M. J., & Jackman, P. C. (2024) · International Review of Sport and Exercise Psychology · doi:10.1080/1750984x.2022.2116723 · Cited by 79
“Second, the review only included published studies, which are more likely to contain signi fi cant results (Franco et al., 2014).”
Participation in payments for ecosystem services programs in the Global South: A systematic review
Jones, K. W., Powlen, K. A., Roberts, R. M., & Shinbrot, X. A. (2020) · Ecosystem Services · doi:10.1016/j.ecoser.2020.101159 · Cited by 96
Zhang, W., Zhu, B., Li, Y., & Yan, D. (2024) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-02671-9 · Cited by 105
Successful performance and cardiovascular markers of challenge and threat: A meta-analysis
Behnke, M., & Kaczmarek, L. D. (2018) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2018.04.007 · Cited by 99
“This publication gap might result from authors who decided not to publish all of their findings or they may stem from the decisions of reviewers and journal editors who might be more likely to decide against publication of such findings, or they may be a combination of both (Franco et al., 2014; Rosenthal, 1979).”
Background
Metadata
- Authors
- Franco, A., Malhotra, N., & Simonovits, G.
- Affiliations
- Stanford University (US)
- Department of Political Science, Stanford University, Stanford, CA, USA.
- Corresponding author
- Malhotra, Neil
- ORCID
- Malhotra, Neil
- Venue
- Science
- Published
- 2014-09-19
- Volume
- 345
- Issue
- 6203
- Pages
- 1502-1505
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W1997809363
- Cited by
- 1,542
- References
- 38
- Citations by year
- 2026: 88 · 2025: 152 · 2024: 143 · 2023: 162 · 2022: 129 · 2021: 152 · 2020: 212 · 2019: 123 · 2018: 119 · 2017: 110 · 2016: 91 · 2015: 52 · 2014: 9
- FWCI
- 85.56
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- Open Research (Auckland University of Technology) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- publication bias, negative results, file drawer problem, social sciences, survey experiment, peer review
- MeSH terms
- Social Sciences, Publication Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.