Publication bias in the social sciences: Unlocking the file drawer
The DOI matches its Crossref record; 37 of the 38 references it cites were found in a registry or an index.
Abstract
The file drawer is full. Should we worry?: Experiments that produce null results face a higher barrier to publication than those that yield statistically significant differences. Whether this is a problem depends on how many null but otherwise valid results might be trapped in the file drawer. Franco et al. use a Time-sharing Experiments in the Social Sciences archive of nearly 250 peer-reviewed proposals of social science experiments conducted on nationally representative samples. They find that only 10 out of 48 null results were published, whereas 56 out of 91 studies with strongly significant results made it into a journal. Science , this issue p. 1502
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Selective reporting, questionable research practices, and low-powered research all contribute to the unreliability of findings in a range of disciplines”
From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as Franco et al., 2014
In the citing paper
“In particular, negative or non-significant results are far less likely to be published (Franco et al., 2014).”
From What do we know about interventions to improve educator wellbeing? A systematic literature review (2024) · cites this paper as Franco et al., 2014
In the citing paper
“The bias against negative results appears to be a consequence of both editors and reviewers finding them less compelling for publication and from authors themselves not trying to publish them 8-12”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [8]
In the citing paper
“Positive results -- finding that a treatment is effective or that there is a relationship between two variables -- are more likely to be published than negative results 6-12 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [8]
In the citing paper
“A possible interpretation of these results is that the missing effect sizes are mainly a file‐drawer problem (Rosenthal, 1979). This interpretation is consistent with the evidence presented in Franco et al. (2014), which indicates that the file‐drawer problem is the main culprit behind publication bias in the social sciences.”
From Targeted school‐based interventions for improving reading and mathematics for students with or at risk of academic difficulties in Grades K‐6: A systematic review (2021) · cites this paper as Franco et al. (2014)
3 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722 · cited 1×
“In the presence of publication bias smaller studies report larger effects in order to exceed arbitrary significance thresholds (19, 20).”
BackgroundThe file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068
Meta-analysis in social research
(1982) · Evaluation and Program Planning · doi:10.1016/0149-7189(82)90011-8 · Cited by 3,853
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109
Publication Bias in Meta‐Analysis: Prevention, Assessment and Adjustments
(2005) · doi:10.1002/0470870168 · Cited by 2,368
The case of the misleading funnel plot
Lau, J., Ioannidis, J. P. A., Terrin, N., Schmid, C. H., & Olkin, I. (2006) · BMJ · doi:10.1136/bmj.333.7568.597 · Cited by 1,766 · cited 1×
“However, these visualization-based approaches are sensitive to using different measures of precision (21, 22) and also assume outcome variables and effect sizes are comparable across studies (23).”
Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias
Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A.-W., Cronin, E., Decullier, E., Easterbrook, P. J., Von Elm, E., Gamble, C., et al. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003081 · Cited by 1,546 · cited 1×
“A different approach involves examining the publication outcomes of a cohort of studies, either prospectively or retrospectively (24, 25).”
MethodSumming up: The Science of Reviewing Research
Wilkinson, I., Light, R. J., & Pillemer, D. B. (1987) · Journal of Educational Statistics · doi:10.2307/1164691 · Cited by 1,471
Summing Up: The Science of Reviewing Research
Light, R. J., & Pillemer, D. B. (1984) · Harvard University Press eBooks · doi:10.4159/9780674040243 · Cited by 1,427
The Existence of Publication Bias and Risk Factors for Its Occurrence
Dickersin, K. (1990) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1990.03440100097014 · Cited by 1,417 · cited 1×
“Publication bias occurs when “publication of study results is based on the direction or significance of the findings” (1).”
Background
Research works printed but not linked: 3
- 10.1037/0735-7028.17.2.136 · doi:10.1037/0735-7028.17.2.136 · Paper page
- K. Dickersin in Publication Bias in Meta-Analysis: Prevention Assessment and Adjustments H. R. Rothstein A. J. Sutton M. Borenstein Eds. (Wiley Chichester UK 2006) ch. 2.
- Dickersin K., Min Y. I., Online J. Curr. Clin. Trials 1993, 50 (1993).
Cited by
Items 301–400 of 1,002
Crompton, J. L., & Nicholls, S. (2019) · Journal of Leisure Research · doi:10.1080/00222216.2019.1637704 · Cited by 50
Transparency, Reproducibility, and the Credibility of Economics Research
Christensen, G., & Miguel, E. (2017) · doi:10.31222/osf.io/9a3rw · Cited by 50
What do we know about interventions to improve educator wellbeing? A systematic literature review
Cann, R., Sinnema, C., Rodway, J., & Daly, A. J. (2024) · Journal of Educational Change · doi:10.1007/s10833-023-09490-w · Cited by 54
“In particular, negative or non-significant results are far less likely to be published (Franco et al., 2014).”
Quality of Growth Empirics: Comparative gaps, benchmarking and policy syndromes
Asongu, S. A., & Nwachukwu, J. C. (2017) · Journal of Policy Modeling · doi:10.1016/j.jpolmod.2017.08.005 · Cited by 49
“in social sciences: of strong results against null results (Franco et al., 1991; Rosenberg, 2005).”
ResultNeglect of publication bias compromises meta-analyses of educational research
Ropovik, I., Adamkovic, M., & Greger, D. (2021) · PLoS ONE · doi:10.1371/journal.pone.0252415 · Cited by 61
“Even searching for grey literature or increasing the number of included studies cannot solve the problem if the authors’ decision to write up the study is being affected by the results [43].”
Discovery of Treatments from Text Corpora
Fong, C., & Grimmer, J. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1151 · Cited by 49
“The use of the training and test set ensures that this discovery does not come at the expense of credibly inferring causal effects, insulating the research design from concerns about “p-hacking” and overfitting (Ioanni-dis, 2005; Humphreys et al., 2013; Franco et al., 2014).”
BackgroundPreregistration of Analyses of Preexisting Data
Mertens, G., & Krypotos, A.-M. (2019) · Psychologica Belgica · doi:10.5334/pb.493 · Cited by 48
Working Out What Works: The Case of the Education Endowment Foundation in England
Edovald, T., & Nevill, C. (2020) · ECNU Review of Education · doi:10.1177/2096531120913039 · Cited by 76
“An analysis of social science research found that those with strong results are 40 percentage points more likely to be published than null results (Franco et al., 2014).”
BackgroundBloomfield, R. J., Rennekamp, K. M., & Steenhoven, B. A. (2018) · Journal of Accounting Research · doi:10.1111/1475-679x.12208 · Cited by 54
Open Science: What, Why, and HowPreprint
Spellman, B., Gilbert, E. A., & Corker, K. S. (2017) · doi:10.31234/osf.io/ak6jr · Cited by 50
Sharing successes and hiding failures: ‘reporting bias’ in learning and teaching research
Dawson, P., & Dawson, S. L. (2018) · Studies in Higher Education · doi:10.1080/03075079.2016.1258052 · Cited by 48
Bhatti, Y., Dahlgaard, J. O., Hansen, J. H., & Hansen, K. M. (2016) · British Journal of Political Science · doi:10.1017/s0007123416000521 · Cited by 48
Diagnosis of crop secondary and micro-nutrient deficiencies in sub-Saharan Africa
Wortmann, C. S., Kaizzi, K. C., Maman, N., Cyamweshi, A. R., Dicko, M., Garba, M., Milner, M., Senkoro, C. J., Tarfa, B. D., Tettah, F., et al. (2019) · Nutrient Cycling in Agroecosystems · doi:10.1007/s10705-018-09968-7 · Cited by 48
Factorial Designs, Model Selection, and (Incorrect) Inference in Randomized Experiments
Muralidharan, K., Romero, M., & Wüthrich, K. (2020) · The Review of Economics and Statistics · doi:10.1162/rest_a_01317 · Cited by 122
Three Decades of Climate Mitigation Policy: What Has It Delivered?
Hoppe, J., Hinder, B., Rafaty, R., Patt, A., & Grubb, M. J. (2023) · Annual Review of Environment and Resources · doi:10.1146/annurev-environ-112321-103821 · Cited by 85
Krishna, A., & Peter, S. M. (2018) · PLoS ONE · doi:10.1371/journal.pone.0203470 · Cited by 53
Reporting errors and biases in published empirical findings: Evidence from innovation research
Bruns, S. B., Asanov, I., Bode, R., Dunger, M., Funk, C., Hassan, S. M., Hauschildt, J., Heinisch, D. P., Kempa, K., König, J., et al. (2019) · Research Policy · doi:10.1016/j.respol.2019.05.005 · Cited by 61
Xie, H., Zhou, Z., & Liu, Q. (2018) · Educational Psychology Review · doi:10.1007/s10648-018-9442-x · Cited by 46
“Studies have empirically confirmed the existence of publication bias (Franco et al. 2014; Rosenthal 1979).”
Financial Access and Productivity Dynamics in Sub-Saharan Africa
Asongu, S. A. (2019) · International Journal of Public Administration · doi:10.1080/01900692.2019.1664570 · Cited by 49
Chopra, F., Haaland, I., Roth, C., & Stegmann, A. (2023) · The Economic Journal · doi:10.1093/ej/uead060 · Cited by 85
Unpacking p-Hacking and Publication Bias
Brodeur, A., Carrell, S., Figlio, D., & Lusher, L. R. (2023) · American Economic Review · doi:10.1257/aer.20210795 · Cited by 81
Toews, I., Glenton, C., Lewin, S. A., Berg, R. C., Noyes, J., Booth, A., Marušić, A., Malički, M., Munthe‐Kaas, H. M., & Meerpohl, J. J. (2016) · PLoS ONE · doi:10.1371/journal.pone.0159290 · Cited by 62
“Specifically, studies showing that an intervention had little or no effect are less likely to be made available than studies showing beneficial or harmful effects, leading to systematic biases in estimates of effectiveness [10, 12].”
BackgroundStanton, S. J. (2016) · Hormones and Behavior · doi:10.1016/j.yhbeh.2016.11.006 · Cited by 51
“Given this mixed set of reported results, it is worth noting that the “file drawer” problem, while prevalent in the social sciences, should be discussed here as well (Franco et al. 2014).”
Fish, J. N., & Russell, S. T. (2018) · Archives of Sexual Behavior · doi:10.1007/s10508-017-0993-6 · Cited by 46
Mason, S. (2018) · Higher Education Research & Development · doi:10.1080/07294360.2018.1462307 · Cited by 46
Social sciences suffer from severe publication bias
Peplow, M. (2014) · Nature · doi:10.1038/nature.2014.15787 · Cited by 46
Impacts and Determinants of Health Levels in Low-Income Countries
Pascaline Dupas, Y., & Miguel, E. A. (2016) · National Bureau of Economic Research · doi:10.3386/w22235 · Cited by 46
Conservative Tests under Satisficing Models of Publication Bias
McCrary, J., Christensen, G., & Fanelli, D. (2016) · PLoS ONE · doi:10.1371/journal.pone.0149590 · Cited by 49
“In recent years the 11 issue has received renewed attention in finance [14–18], in statistics [19–22], in political 12 science [23,24], in psychology [1, 25–27], in medicine [28–30], and in other fields.”
BackgroundPreregistration in experimental linguistics: applications, challenges, and limitations
Roettger, T. B. (2021) · Linguistics · doi:10.1515/ling-2019-0048 · Cited by 52
Mesquida, C., Murphy, J., Lakens, D., & Warne, J. P. (2023) · Journal of Sports Sciences · doi:10.1080/02640414.2023.2269357 · Cited by 57
“Despite the relevance of publication bias to the non-replication of studies and cumulative research (Carter & McCullough, 2014; Francis, 2012; Franco et al., 2014), it has been overlooked in the field of sports and exercise science.”
BackgroundDeJoseph, M. L., Ellwood-Lowe, M. E., Miller‐Cotto, D., Silverman, D. M., Shannon, K. A., Reyes, G., Rakesh, D., & Frankenhuis, W. E. (2024) · Developmental Cognitive Neuroscience · doi:10.1016/j.dcn.2024.101375 · Cited by 64
“Likely even more researchers have found null effects and not published them (Franco et al., 2014).”
Ross, R. M., Pennycook, G., McKay, R., Gervais, W. M., Langdon, R. A., & Coltheart, M. (2016) · Cognitive Neuropsychiatry · doi:10.1080/13546805.2016.1192025 · Cited by 50
Asongu, S. A., & Odhiambo, N. M. (2019) · Sustainable Development · doi:10.1002/sd.1936 · Cited by 53
New Opportunities for Integrated Formal Methods
Gleirscher, M., Foster, S., & Woodcock, J. (2019) · ACM Computing Surveys · doi:10.1145/3357231 · Cited by 66
Franco, A., Malhotra, N., & Simonovits, G. (2015) · Political Analysis · doi:10.1093/pan/mpv006 · Cited by 44
Self-citationCredibility crisis in agricultural economics
Ferraro, P. J., & Shukla, P. (2022) · Applied Economic Perspectives and Policy · doi:10.1002/aepp.13323 · Cited by 59
Antecedents of proactive customer service performance in hospitality: a meta-analysis
Yan, Z., Mansor, Z. D., & Choo, W. C. (2023) · Journal of Hospitality Marketing & Management · doi:10.1080/19368623.2023.2184896 · Cited by 48
“Publication bias exists based on the fact that individual studies that are of statistical significance are more likely to be published, while those whose findings are insignificant are usually shelved Franco et al. (2014).”
BackgroundAsongu, S. A., Batuo, M. E., Nwachukwu, J. C., & Tchamyou, V. S. (2018) · Journal of Multinational Financial Management · doi:10.1016/j.mulfin.2018.04.005 · Cited by 52
“Hence, the insignificant findings should not be observed in the light of publication bias or the file drawer concern in social sciences, where compared to strong results that are more likely to be published, insignificant and null findings are not published and discarded (Rosenberg, 2005; Franco et al., 2014).”
A Comprehensive Review of the Evidence of the Impact of Surface Water Quality on Property Values
Nicholls, S., & Crompton, J. L. (2018) · Sustainability · doi:10.3390/su10020500 · Cited by 44
“research projects with significant results are substantially more likely to be written up and published than those with null results [67,68].”
Paternal Depressive Symptoms and Parenting Behaviors: An Updated Meta-Analysis
Cheung, K., & Theule, J. (2019) · Journal of Child and Family Studies · doi:10.1007/s10826-018-01316-1 · Cited by 78
Moral and religious convictions: Are they the same or different things?
Skitka, L. J., Hanson, B. E., Washburn, A. N., & Mueller, A. B. (2018) · PLoS ONE · doi:10.1371/journal.pone.0199311 · Cited by 45
A Proposal to Organize and Promote Replications
Coffman, L. C., Niederle, M., & Wilson, A. J. (2017) · American Economic Review · doi:10.1257/aer.p20171122 · Cited by 50
Effect Sizes, Power, and Biases in Intelligence Research: A Meta-Meta-Analysis
Nuijten, M. B., van Assen, M. A. L. M., Augusteijn, H. E. M., Crompvoets, E. A. V., & Wicherts, J. M. (2020) · Journal of Intelligence · doi:10.3390/jintelligence8040036 · Cited by 45
Button, K. S., Chambers, C. D., Lawrence, N., & Munafò, M. R. (2020) · Psychology Learning & Teaching · doi:10.1177/1475725719857659 · Cited by 42
“Studies that obtain positive and novel results are more likely to be published than studies that obtain negative results or report replications of prior results (Franco, Malhotra, & Simonovits, 2014; Sterling, 1959, 1975).”
ResultA Systematic Review and Meta-analysis of Income Inequality and Crime in Europe: Do Places Matter?
Kim, B., Seo, C., & Hong, Y.-O. (2020) · European Journal on Criminal Policy and Research · doi:10.1007/s10610-020-09450-7 · Cited by 44
Zeraatkar, D., Pitre, T., Leung, G., Cusano, E., Agarwal, A., Khalid, F., Escamilla, Z., Cooper, M. A., Ghadimi, M., Wang, Y., et al. (2022) · BMJ Medicine · doi:10.1136/bmjmed-2022-000309 · Cited by 44
Ross, A. B., Shertukde, S. P., Livingston Staffier, K., Chung, M., Jacques, P. F., & McKeown, N. M. (2023) · Advances in Nutrition · doi:10.1016/j.advnut.2023.04.003 · Cited by 41
“The problem of nonreporting of null results biasing literature reviews is known across scientific research [93].”
BackgroundGlobal synthesis indicates widespread occurrence of shifting baseline syndrome
Soga, M., & Gaston, K. J. (2024) · BioScience · doi:10.1093/biosci/biae068 · Cited by 46
“Third, the variation in reports indicating a decline in perceived environmental baselines among younger generations across the four environmental contexts might be caused partly by publication bias (Franco et al. 2014 ).”
BackgroundVermeulen, I., Beukeboom, C. J., Batenburg, A., Avrămiea, A.-E., Stoyanov, D., van de Velde, B., & Oegema, D. (2015) · Communication Methods and Measures · doi:10.1080/19312458.2015.1096333 · Cited by 39
“05 (significance) level seems to be a given (Dickersin, 1990; Fanelli, 2010; Franco et al., 2014; Mervis, 2014; Rosenthal, 1979).”
BackgroundInfluentialToews, I., Booth, A., Berg, R. C., Lewin, S. A., Glenton, C., Munthe‐Kaas, H. M., Noyes, J., Schroter, S., & Meerpohl, J. J. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.04.010 · Cited by 60
Differential identification of females and males with reading difficulties: A meta-analysis
Quinn, J. M. (2018) · Reading and Writing · doi:10.1007/s11145-018-9827-8 · Cited by 54
Lawrence, C. N., & Lester, J. (2018) · Journal of Political Science Education · doi:10.1080/15512169.2017.1422739 · Cited by 45
Noisy Retrospection: The Effect of Party Control on Policy Outcomes
Dynes, A. M., & Holbein, J. B. (2019) · American Political Science Review · doi:10.1017/s0003055419000649 · Cited by 47
“The second issue,multiple comparisons, follows from our need to examinemultiple outcomes,which is due, in part, to avoiding a file-drawer problem (Franco, Malhotra, and Simonovits 2014).”
BackgroundResultInfluentialBurgers, C. F., & Brugman, B. C. (2021) · Communication Research · doi:10.1177/00936502211032100 · Cited by 57
“, 2015) and the social sciences in general (Franco et al., 2014) may suffer from publication bias.”
BackgroundGut microbiome and amyotrophic lateral sclerosis: a systematic review of current evidence
Sun, J., Huang, T., Debelius, J. W., & Fang, F. (2021) · Journal of Internal Medicine · doi:10.1111/joim.13336 · Cited by 43
Hewitt, L., Broockman, D., Coppock, A., Tappin, B. M., SLEZAK, J., COFFMAN, V., Lubin, N., & HAMIDIAN, M. (2024) · American Political Science Review · doi:10.1017/s0003055423001387 · Cited by 52
“…publication bias, either because campaigns might not agree to release null or positive findings (to avoid embarrassment or, alternatively, to avoid sharing successful tactics with the opposition), or because researchers may not bother to publish null results (Franco, Malhotra, and Simonovits 2014).”
Kepes, S., & Thomas, M. A. (2018) · European Journal of Information Systems · doi:10.1080/0960085x.2017.1390188 · Cited by 53
“Recently, research in the social and behavioural sciences, including publications in Science , have started to examine the selective reporting of scientific findings (i.e., publication bias; e.g., Franco, Malhotra, & Simonovits, 2014; Mervis, 2014).”
BackgroundHas the evidence for moral licensing been inflated by publication bias?
Kuper, N., & Bott, A. (2019) · Meta-Psychology · doi:10.15626/mp.2018.878 · Cited by 30
Stracke, C. M. (2020) · Lecture notes in educational technology · doi:10.1007/978-981-15-4276-3_2 · Cited by 54
Schönbrodt, F. D., Gärtner, A., Frank, M., Gollwitzer, M., Ihle, M., Mischkowski, D., Phan, L. V., Schmitt, M., Scheel, A. M., Schubert, A., et al. (2022) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/rgh5b · Cited by 45
A framework for evaluating reproducibility and replicability in economics
Dreber, A., & Johannesson, M. (2024) · Economic Inquiry · doi:10.1111/ecin.13244 · Cited by 67
Miller, J. O., & Ulrich, R. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616649170 · Cited by 43
Conditional equivalence testing: An alternative remedy for publication bias
Campbell, H., & Gustafson, P. (2018) · PLoS ONE · doi:10.1371/journal.pone.0195145 · Cited by 46
“Whether due to a reluctance of journals to publish null results or to a reluctance of investigators to submit their null research (Dickersin et al., 1992; Reysen, 2006), the consequence is severe publication bias; see Franco et al. (2014) and Doshi et al. (2013).”
BackgroundA 20-Year Review of Outcome Reporting Bias in Moderated Multiple Regression
O’Boyle, E. H., Banks, G. C., Carter, K., Walter, S. L., & Yuan, Z. (2018) · Journal of Business and Psychology · doi:10.1007/s10869-018-9539-8 · Cited by 43
Is Healthy Neuroticism Associated with Chronic Conditions? A Coordinated Integrative Data Analysis
Weston, S. J., Graham, E. K., Turiano, N. A., Aschwanden, D., Booth, T., Harrison, F., James, B. D., Lewis, N. A., Makkar, S. R., Mueller, S., et al. (2020) · Collabra Psychology · doi:10.1525/collabra.267 · Cited by 44
Bots and Computational Propaganda: Automation for Communication and Control
Woolley, S. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108890960.006 · Cited by 59
Is COVID-19 a Threat to Liberal Democracy?Preprint
Arceneaux, K., Bakker, B. N., HOBOLT, S. B., & Vries, C. E. D. (2020) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/8e4pa · Cited by 42
Niu, X., Li, J.-Y., King, D. L., Rost, D. H., Wang, H., & Wang, J. (2023) · Journal of Behavioral Addictions · doi:10.1556/2006.2023.00043 · Cited by 53
Facts and Where to Find Them: Empirical Research on Internet Platforms and Content Moderation
Keller, D., & Leerssen, P. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108890960.011 · Cited by 61
Abadi, B., Yazdanpanah, N., Nokhodchi, A., & Rezaei, N. (2021) · Advanced Drug Delivery Reviews · doi:10.1016/j.addr.2021.114035 · Cited by 38
Bryce, S., Sloan, E., Lee, S. J., Ponsford, J. L., & Rossell, S. L. (2016) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2016.01.004 · Cited by 44
“This is an area of growing concern for reviews in the social sciences (Franco et al., 2014) and those regarding therapeutic interventions for schizophrenia specifically (Niemeyer et al., 2012), as it may result in non-representative interpretations of scientific literature.”
BackgroundInterpreting t-Statistics Under Publication Bias: Rough Rules of Thumb
Winship, C., & Zhuo, X. (2018) · Journal of Quantitative Criminology · doi:10.1007/s10940-018-9387-8 · Cited by 46
Empathic accuracy in design: Exploring design outcomes through empathic performance and physiology
Chang-Arana, Á. M., Piispanen, M., Himberg, T., Surma‐aho, A., ALHO, J. S., Sams, M. E., & Hölttä‐Otto, K. (2020) · Design Science · doi:10.1017/dsj.2020.14 · Cited by 49
Competition for priority harms the reliability of science, but reforms can help
Tiokhin, L., Yan, M., & Morgan, T. J. H. (2021) · Nature Human Behaviour · doi:10.1038/s41562-020-01040-1 · Cited by 53
The Practicalities of Running Randomized Evaluations
Glennerster, R. (2017) · Handbook of economic field experiments · doi:10.1016/bs.hefe.2016.10.002 · Cited by 56
The importance of open science for biological assessment of aquatic environments
Beck, M. W., O’Hara, C. C., Lowndes, J. S., Mazor, R. D., Theroux, S. M., Gillett, D. J., Lane, B., & Gearheart, G. (2020) · PeerJ · doi:10.7717/peerj.9539 · Cited by 47
Negative results provide valuable evidence for conservation
Wood, K. A. (2020) · Perspectives in Ecology and Conservation · doi:10.1016/j.pecon.2020.10.007 · Cited by 42
Alhabash, S., McAlister, A. R., Kim, W., Lou, C., Cunningham, C., Quilliam, E. T., & Richards, J. I. (2016) · Journal of Interactive Advertising · doi:10.1080/15252019.2016.1160330 · Cited by 43
The More (Social Group Memberships), the Merrier: Is This the Case for Asians?
Chang, M. X., Jetten, J., Cruwys, T., Haslam, C., & Praharso, N. F. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01001 · Cited by 42
“The reason for this is not clear, but publication bias against null effects may play a role (Franco et al., 2014).”
BackgroundStudy strategies of first-year undergraduates with and without dyslexia and the effect of gender
Tops, W., Glatz, T., Premchand, A., Callens, M., & Brysbaert, M. (2019) · European Journal of Special Needs Education · doi:10.1080/08856257.2019.1703580 · Cited by 38
“…and older ones can be attributed to national di ff erences in educational systems, or to the fact that underpowered studies producing signi fi cant results are more likely to be accepted for publication – as opposed to studies with null-e ff ects (Fanelli 2011; Franco, Malhotra, and Simonovits 2014).”
ResultAsongu, S. A., le Roux, S., Nwachukwu, J. C., & Pyke, C. (2019) · International Journal of Managerial Finance · doi:10.1108/ijmf-01-2018-0027 · Cited by 55
Duckworth, A. L., & Milkman, K. L. (2022) · PNAS Nexus · doi:10.1093/pnasnexus/pgac214 · Cited by 44
“Likewise, megastudies make it straightforward to publish null results, reducing the file drawer problem (13), and clarifying the frequency at which null results occur so accurate conclusions about overall effectiveness can be drawn (Beshears and Kosowsky (14); Mertens et al.”
BackgroundSherry, A. D., Miller, A. M., Parlapalli, J. P., Kupferman, G. S., Beck, E. J., McDonald, J. A., Kouzy, R., Jaoude, J. A., Lin, T. A., Sanford, N. N., et al. (2025) · JAMA Oncology · doi:10.1001/jamaoncol.2025.1002 · Cited by 43
Internal conceptual replications do not increase independent replication success
Kunert, R. (2016) · Psychonomic Bulletin & Review · doi:10.3758/s13423-016-1030-9 · Cited by 36
Basic statistical considerations for physiology: The journal Temperature toolbox
Caldwell, A. R., & Cheuvront, S. N. (2019) · Temperature · doi:10.1080/23328940.2019.1624131 · Cited by 40
Improving norms in research culture to incentivize transparency and rigor
Mellor, D. T. (2021) · Educational Psychologist · doi:10.1080/00461520.2021.1902329 · Cited by 41
“And that preference for such results does indeed exist in the form of publication bias, which has been well documented for decades (Atkinson et al., 1982; Franco et al., 2014; Mahoney, 1977; Sterling, 1959).”
BackgroundENIGMA‐Sleep: Challenges, opportunities, and the road map
Tahmasian, M., Aleman, A., Andreassen, O. A., Arab, Z., Baillet, M., Benedetti, F., Bresser, T., Bright, J. K., Chee, M. W. L., Chylinski, D., et al. (2021) · Journal of Sleep Research · doi:10.1111/jsr.13347 · Cited by 35
Family functioning and adaptation following pediatric brain tumor: A systematic review
Moscato, E. L., Patronick, J., & Wade, S. L. (2021) · Pediatric Blood & Cancer · doi:10.1002/pbc.29470 · Cited by 42
Radical Solutions and Open Science: An Open Approach to Boost Higher Education
Burgos, D. (2020) · Lecture notes in educational technology · doi:10.1007/978-981-15-4276-3 · Cited by 51
Transparent, Open, and Reproducible Prevention Science
Grant, S., Wendt, K. E., Leadbeater, B. J. R., Supplee, L. H., Mayo‐Wilson, E., Gardner, F. E. M., & Bradshaw, C. P. (2022) · Prevention Science · doi:10.1007/s11121-022-01336-w · Cited by 46
“There is ample and long-standing evidence across disciplines that “statistically significant” or “positive” results are more likely to be published than results that are “not statistically significant,” “negative,” “null,” “inconclusive,” or otherwise countervailing (Axford et al., 2020; Dwan et al., 2013; Fanelli, 20…”
BackgroundA manifesto for reproducible science.
Munafo, M. R., Nosek, B. A., Bishop, D., Button, K. S., Chambers, C., du Sert, N. P., Simonsohn, U., Wagenmakers, E., Ware, J. J., & Ioannidis, J. P. A. (2024) · American Psychological Association eBooks · doi:10.1037/0000409-041 · Cited by 42
Practical tools and strategies for researchers to increase replicability
Nuijten, M. B. (2018) · Developmental Medicine & Child Neurology · doi:10.1111/dmcn.14054 · Cited by 45
“Studies have found that manuscripts without significant results are both less likely to be submitted and less likely to be accepted for publication.(14,15) Furthermore, published studies seem to have systematically larger effects than unpublished ones.”
BackgroundEstimating the Reproducibility of Experimental PhilosophyPreprint
Cova, F., Strickland, B., Abatista, A., Allard, A., Andow, J., Attie, M., Beebe, J. R., Berniûnas, R., Boudesseul, J., Colombo, M., et al. (2018) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/sxdah · Cited by 38
Schindler, S., Hilgard, J., Fritsche, I., Burke, B., & Pfattheicher, S. (2023) · Personality and Social Psychology Review · doi:10.1177/10888683221107267 · Cited by 43
“Given that publication bias and researcher degrees of freedom are understood to be pervasive problems in psychological science (Bakker et al., 2012; Franco et al., 2014, 2016; Hesse, 2018; John et al., 2012), unadjusted effect sizes are likely to be overestimated (Kvarven et al., 2020).”
BackgroundThe Association Between Child Abuse and Aggressive Behavior: A Three-Level Meta-Analysis
Ran, G., Zhang, Q., Zhang, Q., Li, J., & Chen, J. (2022) · Trauma Violence & Abuse · doi:10.1177/15248380221129596 · Cited by 47
“Publication bias was attributed to the fact that the included studies were not well representative of the field (Franco et al., 2014).”
BackgroundDoes sentiment affect stock returns? A meta-analysis across survey-based measures
Gric, Z., Bajzík, J., & Badura, O. (2023) · International Review of Financial Analysis · doi:10.1016/j.irfa.2023.102773 · Cited by 42
Heyman, R. E., Slep, A. M. S., Lorber, M. F., Mitnick, D. M., Xu, S., Baucom, K. J. W., Halford, W. K., & Niolon, P. H. (2018) · Prevention Science · doi:10.1007/s11121-018-0961-y · Cited by 42
Sensory evaluation of seafood freshness using the quality index method: A meta-analysis
Esteves, E., & Aníbal, J. (2020) · International Journal of Food Microbiology · doi:10.1016/j.ijfoodmicro.2020.108934 · Cited by 41
Law, mobile money drivers and mobile money innovations in developing countries
Asongu, S. A., Agyemang‐Mintah, P., & Nting, R. T. (2021) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2021.120776 · Cited by 39
Targeted Museum Programs for Older Adults: A Research and Program Review
Smiraglia, C. (2016) · Curator The Museum Journal · doi:10.1111/cura.12144 · Cited by 42
Metadata
- Authors
- Franco, A., Malhotra, N., & Simonovits, G.
- Affiliations
- Stanford University (US)
- Department of Political Science, Stanford University, Stanford, CA, USA.
- Corresponding author
- Malhotra, Neil
- ORCID
- Malhotra, Neil
- Venue
- Science
- Published
- 2014-09-19
- Volume
- 345
- Issue
- 6203
- Pages
- 1502-1505
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W1997809363
- Cited by
- 1,542
- References
- 38
- Citations by year
- 2026: 88 · 2025: 152 · 2024: 143 · 2023: 162 · 2022: 129 · 2021: 152 · 2020: 212 · 2019: 123 · 2018: 119 · 2017: 110 · 2016: 91 · 2015: 52 · 2014: 9
- FWCI
- 85.56
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- Open Research (Auckland University of Technology) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- publication bias, negative results, file drawer problem, social sciences, survey experiment, peer review
- MeSH terms
- Social Sciences, Publication Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.