Publication bias in the social sciences: Unlocking the file drawer
The DOI matches its Crossref record; 37 of the 38 references it cites were found in a registry or an index.
Abstract
The file drawer is full. Should we worry?: Experiments that produce null results face a higher barrier to publication than those that yield statistically significant differences. Whether this is a problem depends on how many null but otherwise valid results might be trapped in the file drawer. Franco et al. use a Time-sharing Experiments in the Social Sciences archive of nearly 250 peer-reviewed proposals of social science experiments conducted on nationally representative samples. They find that only 10 out of 48 null results were published, whereas 56 out of 91 studies with strongly significant results made it into a journal. Science , this issue p. 1502
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Selective reporting, questionable research practices, and low-powered research all contribute to the unreliability of findings in a range of disciplines”
From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as Franco et al., 2014
In the citing paper
“In particular, negative or non-significant results are far less likely to be published (Franco et al., 2014).”
From What do we know about interventions to improve educator wellbeing? A systematic literature review (2024) · cites this paper as Franco et al., 2014
In the citing paper
“The bias against negative results appears to be a consequence of both editors and reviewers finding them less compelling for publication and from authors themselves not trying to publish them 8-12”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [8]
In the citing paper
“Positive results -- finding that a treatment is effective or that there is a relationship between two variables -- are more likely to be published than negative results 6-12 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [8]
In the citing paper
“A possible interpretation of these results is that the missing effect sizes are mainly a file‐drawer problem (Rosenthal, 1979). This interpretation is consistent with the evidence presented in Franco et al. (2014), which indicates that the file‐drawer problem is the main culprit behind publication bias in the social sciences.”
From Targeted school‐based interventions for improving reading and mathematics for students with or at risk of academic difficulties in Grades K‐6: A systematic review (2021) · cites this paper as Franco et al. (2014)
3 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722 · cited 1×
“In the presence of publication bias smaller studies report larger effects in order to exceed arbitrary significance thresholds (19, 20).”
BackgroundThe file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
Meta-analysis in social research
(1982) · Evaluation and Program Planning · doi:10.1016/0149-7189(82)90011-8 · Cited by 3,853
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109
Publication Bias in Meta‐Analysis: Prevention, Assessment and Adjustments
(2005) · doi:10.1002/0470870168 · Cited by 2,368
The case of the misleading funnel plot
Lau, J., Ioannidis, J. P. A., Terrin, N., Schmid, C. H., & Olkin, I. (2006) · BMJ · doi:10.1136/bmj.333.7568.597 · Cited by 1,766 · cited 1×
“However, these visualization-based approaches are sensitive to using different measures of precision (21, 22) and also assume outcome variables and effect sizes are comparable across studies (23).”
Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias
Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A.-W., Cronin, E., Decullier, E., Easterbrook, P. J., Von Elm, E., Gamble, C., et al. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003081 · Cited by 1,546 · cited 1×
“A different approach involves examining the publication outcomes of a cohort of studies, either prospectively or retrospectively (24, 25).”
MethodSumming up: The Science of Reviewing Research
Wilkinson, I., Light, R. J., & Pillemer, D. B. (1987) · Journal of Educational Statistics · doi:10.2307/1164691 · Cited by 1,471
Summing Up: The Science of Reviewing Research
Light, R. J., & Pillemer, D. B. (1984) · Harvard University Press eBooks · doi:10.4159/9780674040243 · Cited by 1,427
The Existence of Publication Bias and Risk Factors for Its Occurrence
Dickersin, K. (1990) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1990.03440100097014 · Cited by 1,417 · cited 1×
“Publication bias occurs when “publication of study results is based on the direction or significance of the findings” (1).”
Background
Research works printed but not linked: 3
- 10.1037/0735-7028.17.2.136 · doi:10.1037/0735-7028.17.2.136 · Paper page
- K. Dickersin in Publication Bias in Meta-Analysis: Prevention Assessment and Adjustments H. R. Rothstein A. J. Sutton M. Borenstein Eds. (Wiley Chichester UK 2006) ch. 2.
- Dickersin K., Min Y. I., Online J. Curr. Clin. Trials 1993, 50 (1993).
Cited by
Items 201–300 of 1,002
Heterogeneity in direct replications in psychology and its association with effect size.
Olsson-Collentine, A., Wicherts, J. M., & van Assen, M. A. L. M. (2020) · Psychological Bulletin · doi:10.1037/bul0000294 · Cited by 75
We Ran 9 Billion Regressions: Eliminating False Positives through Computational Model Robustness
Muñoz, J., & Young, C. (2018) · Sociological Methodology · doi:10.1177/0081175018777988 · Cited by 84
“However, researchers must openly acknowledge the risk of confirmation bias and the pressure to publish significant findings (Reason 1995; see reviews in Franco, Malhotra, and Simonovits 2014; Nickerson 1998).”
BackgroundPublication Bias and Editorial Statement on Negative Findings
Blanco-Perez, C., & Brodeur, A. (2020) · The Economic Journal · doi:10.1093/ej/ueaa011 · Cited by 76
The Past, Present, and Future of Experimental Methods in the Social Sciences
Mize, T. D., & Manago, B. (2022) · Social Science Research · doi:10.1016/j.ssresearch.2022.102799 · Cited by 76
Ehret, P. J., Hodges, H., Kuehl, C., Brick, C., Mueller, S., & Anderson, S. E. (2020) · Environment and Behavior · doi:10.1177/0013916519896868 · Cited by 91
“Due to publication bias (Fanelli & Ioannidis, 2013; Franco et al., 2014), these effect size estimates are likely biased and possibly inflated.”
Researcher degrees of freedom in phonetic research
Roettger, T. B. (2019) · Laboratory Phonology Journal of the Association for Laboratory Phonology · doi:10.5334/labphon.147 · Cited by 80
The role of replication studies in ecology
Fraser, H. S., Barnett, A., Parker, T. H., & Fidler, F. M. (2020) · Ecology and Evolution · doi:10.1002/ece3.6330 · Cited by 91
Questionable science and reproducibility in electrical brain stimulation research
Héroux, M. E., Loo, C. K. A., Taylor, J. L., & Gandevia, S. C. (2017) · PLoS ONE · doi:10.1371/journal.pone.0175635 · Cited by 80
Factorial Designs, Model Selection, and (Incorrect) Inference in Randomized Experiments
Muralidharan, K., Romero, M., & Wüthrich, K. (2019) · National Bureau of Economic Research · doi:10.3386/w26562 · Cited by 100
“Third, there is well-documented publication bias towards significant findings (e.g., Franco et al., 2014; Andrews & Kasy, 2018; Christensen & Miguel, 2018; Abadie, 2020).”
BackgroundResultInfluentialCan Results-Free Review Reduce Publication Bias? The Results and Implications of a Pilot Study
Findley, M. G., Jensen, N. M., Malesky, E. J., & Pepinsky, T. B. (2016) · Comparative Political Studies · doi:10.1177/0010414016655539 · Cited by 85
“Job placement and tenure decisions can generate incentives for (a) prioritizing work in the pipeline that has “significance stars” next to key coefficients, which was the key lesson of Franco et al. (2014); (b) choking unresponsive data with a barrage of specification choices and subsample analyses until the data conf…”
BackgroundInfluentialExperimental research into teaching innovations: responding to methodological and ethical challenges
Taber, K. S. (2019) · Studies in Science Education · doi:10.1080/03057267.2019.1658058 · Cited by 94
Finance, inequality and inclusive education in Sub-Saharan Africa
Asongu, S. A., Nnanna, J., & Acha-Anyi, P. N. (2020) · Economic Analysis and Policy · doi:10.1016/j.eap.2020.07.006 · Cited by 85
The Role of Expert Judgment in Statistical Inference and Evidence-Based Decision-Making
Brownstein, N. C., Louis, T. A., O’Hagan, A., & Pendergast, J. (2019) · The American Statistician · doi:10.1080/00031305.2018.1529623 · Cited by 99
“To guard against publication bias, null results, in particular, must be communicated, despite disincentives for doing so (Franco, Malhotra, and Simonovits 2014; Easterbrook et al. 1991).”
BackgroundJackson, C. K., & Mackevicius, C. L. (2023) · American Economic Journal Applied Economics · doi:10.1257/app.20220279 · Cited by 161
Heterogeneity in effect size estimates
Holzmeister, F., Johannesson, M., BOHM, R. M., Dreber, A., Huber, J., & Kirchler, M. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2403490121 · Cited by 103
Making Replication MainstreamPreprint
Zwaan, R. A., Etz, A. J., Lucas, R. E., & Donnellan, B. M. (2017) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/4tg9c · Cited by 92
A genetically informed Registered Report on adverse childhood experiences and mental health
Baldwin, J. R., Sallis, H., Schoeler, T., Taylor, M. J., Kwong, A. S. F., Tielbeek, J. J., Barkhuizen, W., Warrier, V., Howe, L. D., Danese, A., et al. (2022) · Nature Human Behaviour · doi:10.1038/s41562-022-01482-9 · Cited by 92
Pickett, J. T., & Roche, S. P. (2016) · Science and Engineering Ethics · doi:10.1007/s11948-017-9886-2 · Cited by 82
“While roughly 2% of scientists commit data fraud (Fanelli 2009), a much larger number engage (or have engaged) in selective reporting (Franco et al. 2014, 2015; O’Boyle et al. 2014).”
Null Effects and Publication Bias in Special Education Research
Cook, B. G., & Therrien, W. J. (2017) · Behavioral Disorders · doi:10.1177/0198742917709473 · Cited by 82
“Research findings also demonstrate the existence of publication bias in the social scientific (e.g., Driessen, Hollon, Bockting, Cuijpers, & Turner, 2015; Franco et al., 2014), psychological (e.”
BackgroundInfluentialStandard Operating Procedures: A Safety Net for Pre-Analysis Plans
Lin, W., & Green, D. P. (2016) · Political Science and Politics · doi:10.1017/s1049096516000810 · Cited by 71
“A growing body of evidence suggests that fishing and file drawer problems are widespread in the social sciences (Brodeur et al. 2016 ; Franco, Malhotra, and Simonovits 2014 , 2016 ; Gerber and Malhotra 2008 ), undercutting the reliability of published research findings (Open Science Collaboration…”
BackgroundThe Impacts of Fiscal Openness
de Renzio, P., & Wehner, J. (2015) · The World Bank Research Observer · doi:10.1093/wbro/lkx004 · Cited by 133
“This is of course a wider problem, irrespective of methodological approach (Franco, Malhotra, and Simonovits 2014).”
Neural correlates of socioeconomic status in early childhood: a systematic review of the literature
Olson, L., Chen, B., & Fishman, I. (2021) · Child Neuropsychology · doi:10.1080/09297049.2021.1879766 · Cited by 81
“While this pattern of results may reflect a “file drawer” bias (with null findings more likely to be unpublished; Franco et al., 2014), the findings reviewed above provide compelling evidence for relationships between socioeconomic context and brain development in early childhood, prior to entry…”
ResultSystematic review of conservation interventions to promote voluntary behavior change
Thomas‐Walters, L., McCallum, J., Montgomery, R., Petros, C., Wan, A. K. Y., & Veríssimo, D. (2022) · Conservation Biology · doi:10.1111/cobi.14000 · Cited by 85
Milani, A., Dessi, F., & Bonaiuto, M. (2024) · Energy Research & Social Science · doi:10.1016/j.erss.2024.103624 · Cited by 91
The Replication Paradox: Combining Studies can Decrease Accuracy of Effect Size Estimates
Nuijten, M. B., van Assen, M. A. L. M., Veldkamp, C. L. S., & Wicherts, J. M. (2015) · Review of General Psychology · doi:10.1037/gpr0000034 · Cited by 69
de Winter, J. C., & Dodou, D. (2015) · PeerJ · doi:10.7717/peerj.733 · Cited by 78
Pre-Analysis Plans: An Early Stocktaking
Ofosu, G. K., & Posner, D. (2021) · Perspectives on Politics · doi:10.1017/s1537592721000931 · Cited by 100
Preregistration in Single-Case Design Research
Johnson, A. H., & Cook, B. G. (2019) · Exceptional Children · doi:10.1177/0014402919868529 · Cited by 68
“Franco, Malhotra, and Simonovits (2014) examined how the strength of observed results affected the reporting and publication of 221 high-quality National Science Foundation– sponsored group experiments in the social sciences conducted from 2002 to 2012.”
BackgroundWagge, J., Brandt, M. J., Lazarević, L. B., Legate, N., Christopherson, C. D., Wiggins, B., & Grahe, J. E. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.00247 · Cited by 84
“There is pressure for faculty to publish original research in high-impact journals and report significant effects (Franco et al., 2014), and so replication work often does not get the attention that it requires or deserves (Martin and Clarke, 2017).”
BackgroundCarrico, A. R. (2021) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2021.03.025 · Cited by 90
Message Framing and Climate Change Communication: A Meta-Analytical Review
Li, N., & Su, L. Y.-F. (2018) · Journal of Applied Communications · doi:10.4148/1051-0834.2189 · Cited by 90
Democratic Transparency in the Platform Society
Gorwa, R., & Ash, T. G. (2020) · Social Media and Democracy · doi:10.1017/9781108890960.013 · Cited by 64
Mastery-approach goals: A large-scale cross-cultural analysis of antecedents and consequences.
Guo, J., Hu, X., Elliot, A. J., Marsh, H. W., Murayama, K., Basarkod, G., Parker, P. D., & Dicke, T. (2023) · Journal of Personality and Social Psychology · doi:10.1037/pspp0000436 · Cited by 87
A framework for learning from null results
Jacob, R. T., Doolittle, F., Kemple, J., & Somers, M.-A. (2019) · Educational Researcher · doi:10.3102/0013189x19891955 · Cited by 65
“This is both because studies of interventions that do not work are less likely to get published and program developers are more likely to try to publish evaluations that show that their program is effective (Franco et al., 2014; Rosenthal, 1979).”
BackgroundVadillo, M. A., Hardwicke, T. E., & Shanks, D. R. (2016) · Journal of Experimental Psychology: General · doi:10.1037/xge0000157 · Cited by 65
Misinformation and Its Correction
Wittenberg, C., & Berinsky, A. J. (2020) · Social Media and Democracy · doi:10.1017/9781108890960.009 · Cited by 58
Buhl-Wiggers, J., Kjærgaard, A., & Munk, K. (2022) · Studies in Higher Education · doi:10.1080/03075079.2022.2123911 · Cited by 109
“One should also be aware that the studies reviewed tended to report positive e ff ects from changes to the face-to-face activities, but this tendency might, at least in part, be due to a publication bias favoring studies that fi nd positive e ff ects (Franco, Malhotra, and Simonovits 2014).”
ResultWollburg, C., Steinert, J., Reeves, A., & Nye, E. (2023) · PLoS ONE · doi:10.1371/journal.pone.0281283 · Cited by 77
Transparent Social Inquiry: Implications for Political Science
Elman, C., Kapiszewski, D., & Lupia, A. (2017) · Annual Review of Political Science · doi:10.1146/annurev-polisci-091515-025429 · Cited by 83
Crime and Fear in Public Places: Towards Safe, Inclusive and Sustainable Cities
Ceccato, V. A., & Nalla, M. K. (2020) · Routledge eBooks · doi:10.4324/9780429352775 · Cited by 97
Improving Transparency in Observational Social Science Research: A Pre-Analysis Plan Approach
Burlig, F. (2016) · Economics Letters · doi:10.1016/j.econlet.2018.03.036 · Cited by 79
Alt, M., Bruns, H., Valle, N. D., & Murauskaite-Bull, I. (2024) · Global Environmental Change · doi:10.1016/j.gloenvcha.2023.102776 · Cited by 84
The benefits of preregistration and Registered Reports
Lakens, D., Mesquida, C., Rasti, S., & Ditroilo, M. (2024) · Evidence-Based Toxicology · doi:10.1080/2833373x.2024.2376046 · Cited by 104
Should we use single items? Better not
Sarstedt, M., Diamantopoulos, A., & Salzberger, T. (2016) · Journal of Business Research · doi:10.1016/j.jbusres.2016.02.040 · Cited by 63
What Do We Know about Legal Empowerment? Mapping the Evidence
Goodwin, L., & Maru, V. (2017) · Hague Journal on the Rule of Law · doi:10.1007/s40803-016-0047-5 · Cited by 93
“Franco et al. (2014), pp. 1502–1504.”
Split-Sample Strategies for Avoiding False Discoveries
Anderson, M., & Magruder, J. (2017) · National Bureau of Economic Research · doi:10.3386/w23544 · Cited by 61
“Due to the likelihood of publication bias and p-hacking (Franco et al. 2014), we interpret this distribution as an overestimate of the ex ante t-statistic distribution that a researcher should expect when beginning a typical field experiment.”
BackgroundEthical considerations in social work research
Sobočan, A. M., Bertotti, T., & Strom-Gottfried, K. (2018) · European Journal of Social Work · doi:10.1080/13691457.2018.1544117 · Cited by 95
van den Akker, O. R., van Assen, M. A. L. M., Bakker, M., Elsherif, M. M., Wong, T. K., & Wicherts, J. M. (2023) · Behavior Research Methods · doi:10.3758/s13428-023-02277-0 · Cited by 99
“Although finding a positive result is not necessarily the result of better scholarship, positive results are more likely to be published (Dickersin, 1990; Ferguson & Brannick, 2012; Franco et al., 2014) and are more often cited (Duyx et al., 2017) than negative results.”
ResultSustainability through institutional failure and decline? Archetypes of productive pathways
Newig, J., Derwort, P., & Jager, N. W. (2019) · Ecology and Society · doi:10.5751/es-10700-240118 · Cited by 72
Bakker, B. N., Jaidka, K., Dörr, T., Fasching, N., & Lelkes, Y. (2021) · Journal of Communication · doi:10.1093/joc/jqab031 · Cited by 60
Measuring Accounting Fraud and Irregularities Using Public and Private Enforcement
Donelson, D. C., Kartapanis, A., McInnis, J. M., & Yust, C. G. (2021) · The Accounting Review · doi:10.2308/tar-2018-0592 · Cited by 68
Héroux, M. E., Taylor, J. L., & Gandevia, S. C. (2015) · PLoS ONE · doi:10.1371/journal.pone.0144151 · Cited by 66
“This behaviour is well documented across branches of science and has been well analysed in sociology [19, 39], psychology [40, 41] and medicine [42].”
BackgroundFleming, J. I., Wilson, S. E., Hart, S. A., Therrien, W. J., & Cook, B. G. (2021) · Educational Psychologist · doi:10.1080/00461520.2021.1897593 · Cited by 64
Transparency and Openness Promotion (TOP) Guidelines
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2016) · doi:10.31219/osf.io/vj54c · Cited by 60
Self-citationHeinz, N., & Koessler, A. (2020) · Ecological Economics · doi:10.1016/j.ecolecon.2021.106987 · Cited by 70
“6 Considering the difficulty to publish studies that do not find effects, unsuccessful other-regarding interventions can be assumed to be underreported (Franco et al., 2014).”
BackgroundRaghav, K. P. S., Mahajan, S., Yao, J. C., Hobbs, B. P., Berry, D. A., Pentz, R. D., Tam, A., Hong, W. K., Ellis, L. M., Abbruzzese, J., et al. (2015) · Journal of Clinical Oncology · doi:10.1200/jco.2015.62.4148 · Cited by 59
How Powerful is the Evidence in Criminology? On Whether We Should Fear a Coming Crisis of Confidence
Barnes, J. C., TenEyck, M. F., Pratt, T. C., & Cullen, F. T. (2019) · Justice Quarterly · doi:10.1080/07418825.2018.1495252 · Cited by 69
Specification Searching and Significance Inflation Across Time, Methods and Disciplines
Vivalt, E. (2019) · Oxford Bulletin of Economics and Statistics · doi:10.1111/obes.12289 · Cited by 80
Stewart, R. J., Langer, L., Silva, N. R. D., Muchiri, E., Zaranyika, H., Erasmus, Y., Randall, N., Rafferty, S. L., Korth, M., Madinga, N., et al. (2015) · Campbell Systematic Reviews · doi:10.4073/csr.2015.16 · Cited by 95
Kruis, N. E., Seo, C., & Kim, B. (2019) · Substance Use & Misuse · doi:10.1080/10826084.2019.1696821 · Cited by 67
“Investigating publication bias in social sciences by analyzing both published and unpublished studies, Franco, Malhotra, and Simonovits (2014) showed that null findings are 40 percent less likely to be published than statistically significant findings, and as a result, many authors do not submit…”
BackgroundBrini, S., Sohrabi, H. R., Hébert, J. J., Forrest, M. R. L., Laine, M., Hämäläinen, H., Karrasch, M., Peiffer, J. J., Martins, R. N., & Fairchild, T. J. (2020) · Neuropsychology Review · doi:10.1007/s11065-020-09426-8 · Cited by 71
Kane, J. (2024) · Journal of Experimental Political Science · doi:10.1017/xps.2024.1 · Cited by 77
A Bayesian approach to mitigation of publication bias
Guan, M., & Vandekerckhove, J. (2015) · Psychonomic Bulletin & Review · doi:10.3758/s13423-015-0868-6 · Cited by 68
Button, K. S., Bal, L., Clark, A., & Shipley, T. (2016) · BMC Psychology · doi:10.1186/s40359-016-0167-7 · Cited by 58
Neumark, D., & Corella, L. F. M. (2020) · World Development · doi:10.1016/j.worlddev.2020.105165 · Cited by 68
A meta-analysis of Libet-style experiments
Braun, M. N., Wessler, J., & Friese, M. (2021) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2021.06.018 · Cited by 70
“Publication bias Publication bias occurs when studies that did not produce the desired outcomes are published less often than studies that did produce the desired outcomes (Fanelli, 2012; Franco et al., 2014).”
BackgroundMeta-Analytic Methods to Detect Publication Bias in Behavior Science Research
Dowdy, A., Hantula, D. A., Travers, J. C., & Tincani, M. (2021) · Perspectives on Behavior Science · doi:10.1007/s40614-021-00303-0 · Cited by 67
“This overrepresentation of positive or confirmatory results and exclusion of negative results in the research literature is known as publication bias (Franco et al., 2014).”
BackgroundBrown, A. W., Aslibekyan, S. W., Bier, D. M., da Silva, R. F., Hoover, A. W., Klurfeld, D. M., Loken, E., Mayo‐Wilson, E., Menachemi, N., Pavela, G., et al. (2021) · Critical Reviews in Food Science and Nutrition · doi:10.1080/10408398.2021.1985427 · Cited by 65
Crossmodal interactions between audition and taste: A systematic review and narrative synthesis
Guedes, D., Garrido, M. V., Lamy, E., Cavalheiro, B. P., & Prada, M. (2023) · Food Quality and Preference · doi:10.1016/j.foodqual.2023.104856 · Cited by 65
Evans, S. C., Amaro, C. M., Herbert, R. S., Blossom, J. B., & Roberts, M. C. (2018) · PLoS ONE · doi:10.1371/journal.pone.0192219 · Cited by 71
Taxes and Economic Growth in OECD Countries: A Meta-analysis
Alinaghi, N., & Reed, W. R. (2020) · Public Finance Review · doi:10.1177/1091142120961775 · Cited by 99
Arrested Development: Misguided Directions in Deterrence Theory and Policy
Pickett, J. T., & Roche, S. P. (2016) · Criminology & Public Policy · doi:10.1111/1745-9133.12217 · Cited by 62
Open science and modified funding lotteries can impede the natural selection of bad science
Smaldino, P. E., Turner, M. A., & Kallens, P. C. (2019) · Royal Society Open Science · doi:10.1098/rsos.190194 · Cited by 66
Bias in science: natural and social
May, J. (2020) · Synthese · doi:10.1007/s11229-020-02937-0 · Cited by 66
Promoting Female Economic Inclusion for Tax Performance in Sub-Saharan Africa
Asongu, S. A., Adegboye, A., & Nnanna, J. (2020) · Economic Analysis and Policy · doi:10.1016/j.eap.2020.11.010 · Cited by 65
The Weak Spots in Contemporary Science (and How to Fix Them)
Wicherts, J. M. (2017) · Animals · doi:10.3390/ani7120090 · Cited by 59
“Because publication bias is often caused by selection on significance [70,71], fields that involve weaker effects and smaller samples (key ingredients for non-significance even if an effect is genuine) are particularly vulnerable to it.”
BackgroundOpen and Reproducible Research on Open Science Framework
Sullivan, I., DeHaven, A. C., & Mellor, D. T. (2019) · Current Protocols Essential Laboratory Techniques · doi:10.1002/cpet.32 · Cited by 57
Dietrichson, J., Filges, T., Seerup, J. K., Klokker, R. H., Viinholt, B. C. A., Bøg, M., & Eiberg, M. (2021) · Campbell Systematic Reviews: Better Evidence for a Better World · doi:10.1002/cl2.1152 · Cited by 54
Mierop, A., Mikolajczak, M., Stahl, C., Béna, J., Luminet, O., Lane, A., & Corneille, O. (2020) · Perspectives on Psychological Science · doi:10.1177/1745691620921525 · Cited by 53
May I have your Attention, please? An eye tracking study on emotional social media comments
Kohout, S., Kruikemeier, S., & Bakker, B. N. (2022) · Computers in Human Behavior · doi:10.1016/j.chb.2022.107495 · Cited by 71
Yu, L.-J., Zhang, K., Zhu, J., Zheng, Q., Bao, X., Thapa, S., Wang, Y., & Chu, M. (2017) · Oxidative Medicine and Cellular Longevity · doi:10.1155/2017/8192383 · Cited by 56
“Second, there are no negative studies, which may overestimate the effect to a certain degree [40].”
BackgroundUNIVERSAL PRESCHOOL PROGRAMS AND LONG‐TERM CHILD OUTCOMES: A SYSTEMATIC REVIEW
Dietrichson, J., Kristiansen, I. L., & Viinholt, B. C. A. (2020) · Journal of Economic Surveys · doi:10.1111/joes.12382 · Cited by 75
“However, Franco, Malhotra, and Simonovits (2014) find that publication bias in the social sciences is more driven by researchers not writing up null results than by journals not publishing them.”
BackgroundDoes hippocampal volume explain performance differences on hippocampal-dependant tasks?
Clark, I. A., Monk, A. M., Hotchin, V., Pizzamiglio, G., Liefgreen, A., Callaghan, M. F., & Maguire, E. A. (2020) · NeuroImage · doi:10.1016/j.neuroimage.2020.117211 · Cited by 53
ICT, information asymmetry and market power in African banking industry
Asongu, S. A., & Biekpe, N. B. (2017) · Research in International Business and Finance · doi:10.1016/j.ribaf.2017.07.121 · Cited by 59
A Systems-Based Approach to Fostering Robust Science in Industrial-Organizational Psychology
Grand, J. A., Rogelberg, S. G., Allen, T. D., Landis, R. S., Reynolds, D. H., Scott, J. C., Tonidandel, S., & Truxillo, D. M. (2018) · Industrial and Organizational Psychology · doi:10.1017/iop.2017.55 · Cited by 52
Xu, C., Bi, S., Zhang, W., & Luo, L. (2024) · Frontiers in Nutrition · doi:10.3389/fnut.2024.1424972 · Cited by 54
Tielbeek, J. J., Linnér, R. K., Beers, K., Posthuma, D., Popma, A., & Polderman, T. J. C. (2016) · American Journal of Medical Genetics Part B Neuropsychiatric Genetics · doi:10.1002/ajmg.b.32442 · Cited by 61
“…if null results are missing from the published body of literature because of publication bias then the results of the Stouffer’s and Liptak–Stouffer’s methods are biased towards showing a larger average effect than if the non-published null results would have been included [Franco et al., 2014].”
ResultThe neural and behavioral correlates of social evaluation in childhood
Achterberg, M., van Duijvenvoorde, A. C. K., van der Meulen, M., Euser, S., Bakermans-Kranenburg, M. J., & Crone, E. A. (2017) · Developmental Cognitive Neuroscience · doi:10.1016/j.dcn.2017.02.007 · Cited by 52
“, the bias resulting from selective publication of significant results (Franco et al., 2014)).”
BackgroundInvention Versus Direct Instruction: For Some Content, It’s a Tie
Chase, C. C., & Klahr, D. (2017) · Journal of Science Education and Technology · doi:10.1007/s10956-017-9700-6 · Cited by 52
Kepes, S., Keener, S. K., McDaniel, M. A., & Hartman, N. S. (2022) · Journal of Organizational Behavior · doi:10.1002/job.2623 · Cited by 71
Ordering effects in discrete choice experiments: A systematic literature review across domains
Boxebeld, S. (2024) · Journal of Choice Modelling · doi:10.1016/j.jocm.2024.100489 · Cited by 54
Preventing Sexual Violence: A Behavioral Problem Without a Behaviorally Informed Solution
Porat, R., Gantman, A. P., Green, S. A., Pezzuto, J.-H., & Paluck, E. L. (2024) · Gothic.net · doi:10.1177/15291006231221978 · Cited by 70
“This problem has been highly prevalent in psychology and other social-science literature (Open Science Collaboration, 2015; Franco et al., 2014; Paluck et al., 2021).”
Identifying Key Antecedents of Quiet Quitting Among Nurses: A Cross‐Profession Meta‐Analytic Review
Geng, R., Geng, X., & Geng, S. (2025) · Journal of Advanced Nursing · doi:10.1111/jan.16934 · Cited by 71
Kelly, J., & Knottenbelt, W. J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.00962 · Cited by 51
“How many negative results exist unpublished? Perhaps academics fear that reviewers would reject a null result? Might companies fear that customers or shareholders would be driven away? A study on publication bias in the social sciences found that positive results are 60% more likely to be written up than null results…”
BackgroundA randomized controlled design reveals barriers to citizenship for low-income immigrants
Hainmueller, J., Lawrence, D., Gest, J., Hotard, M., Koslowski, R., & Laitin, D. D. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1714254115 · Cited by 51
“However, it stands to reason that, given publication bias (31), failed nudges do not easily find their way into the published literature.”
BackgroundQuestionable Research Practices and Open Science in Quantitative Criminology
Chin, J. M., Pickett, J. T., Vazire, S., & Holcombe, A. O. (2021) · Journal of Quantitative Criminology · doi:10.1007/s10940-021-09525-6 · Cited by 82
Burns, E. J., Gaunt, E., Kidane, B., Hunter, L., & Pulford, J. (2022) · Behavior Research Methods · doi:10.3758/s13428-022-02017-w · Cited by 61
“By contrast, null results have been commonly ignored, or difficult to publish (Franco et al., 2014; Mervis, 2014; Rosenthal, 1979).”
ResultYan, V. X., & Schuetze, B. A. (2023) · Psychological Bulletin · doi:10.1037/bul0000370 · Cited by 68
Pan, X., Wong, C. W. Y., Wong, C. Y., Boon‐itt, S., & Li, C. (2024) · International Journal of Production Economics · doi:10.1016/j.ijpe.2024.109267 · Cited by 73
Estimating the evidential value of significant results in psychological science
Aczél, B., Pálfi, B., & Szászi, B. (2017) · PLoS ONE · doi:10.1371/journal.pone.0182651 · Cited by 56
Metadata
- Authors
- Franco, A., Malhotra, N., & Simonovits, G.
- Affiliations
- Stanford University (US)
- Department of Political Science, Stanford University, Stanford, CA, USA.
- Corresponding author
- Malhotra, Neil
- ORCID
- Malhotra, Neil
- Venue
- Science
- Published
- 2014-09-19
- Volume
- 345
- Issue
- 6203
- Pages
- 1502-1505
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W1997809363
- Cited by
- 1,542
- References
- 38
- Citations by year
- 2026: 88 · 2025: 152 · 2024: 143 · 2023: 162 · 2022: 129 · 2021: 152 · 2020: 212 · 2019: 123 · 2018: 119 · 2017: 110 · 2016: 91 · 2015: 52 · 2014: 9
- FWCI
- 85.56
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- Open Research (Auckland University of Technology) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- publication bias, negative results, file drawer problem, social sciences, survey experiment, peer review
- MeSH terms
- Social Sciences, Publication Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.