Why null results rarely see the light of day
Abstract
“File drawer” study proposes registry for unpublished social science data.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Some results may have been influenced by publication bias.
“some of our results may have been influenced by publication bias”
From The interpersonal theory of suicide: A systematic review and meta-analysis of a decade of cross-national research. (2017) · cites this paper as Mervis, 2014
Publication bias could have introduced an overestimation of the actual effect of trainings and manipulations on DD.
“Finally, publication bias could have introduced an overestimation of the actual effect of trainings and manipulations on DD”
From Behavioral trainings and manipulations to reduce delay discounting: A systematic review (2019) · cites this paper as Mervis, 2014
Cited by
Sorted by citations
Chu, C., Buchman-Schmitt, J. M., Stanley, I. H., Hom, M. A., Tucker, R. P., Hagan, C. R., Rogers, M. L., Podlogar, M. C., Chiurliza, B., Ringer, F. B., et al. (2017) · Psychological Bulletin · doi:10.1037/bul0000123 · Cited by 1,231 · see citing works
Transparency, Reproducibility, and the Credibility of Economics Research
Christensen, G., & Miguel, E. A. (2018) · Journal of Economic Literature · doi:10.1257/jel.20171350 · Cited by 448 · see citing works
Swedberg, R. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.002 · Cited by 408 · see citing works
van den Noort, M., Struys, E., Bosch, P., Jaswetz, L., Perriard, B., Yeo, S., Barisch, P., Vermeire, K., Lee, S., & Lim, S. (2019) · Behavioral Sciences · doi:10.3390/bs9030027 · Cited by 211 · see citing works
“A need exists in science for good-quality journals willing to publish non-effects [122].”
BackgroundBehavioral trainings and manipulations to reduce delay discounting: A systematic review
Scholten, H., Scheres, A., de Water, E., Graf, U., Granic, I., & Luijten, M. (2019) · Psychonomic Bulletin & Review · doi:10.3758/s13423-019-01629-2 · Cited by 135 · see citing works
“Finally, publication bias could have introduced an overestimation of the actual effect of trainings and manipulations on DD (Hopewell, Clarke, Stewart, & Tierney, 2007; Mervis, 2014).”
Background(2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519 · Cited by 135 · see citing works
Héroux, M. E., Taylor, J. L., & Gandevia, S. C. (2015) · PLoS ONE · doi:10.1371/journal.pone.0144151 · Cited by 58 · see citing works
Transparency, Reproducibility, and the Credibility of Economics ResearchPreprint
Christensen, G., & Miguel, E. A. (2017) · doi:10.31222/osf.io/9a3rw · Cited by 57 · see citing works
Burns, E. J., Gaunt, E., Kidane, B., Hunter, L., & Pulford, J. (2022) · Behavior Research Methods · doi:10.3758/s13428-022-02017-w · Cited by 50 · see citing works
“By contrast, null results have been commonly ignored, or difficult to publish (Franco et al., 2014; Mervis, 2014; Rosenthal, 1979).”
BackgroundResultVermeulen, I., Beukeboom, C. J., Batenburg, A., Avrămiea, A.-E., Stoyanov, D. S., van de Velde, B., & Oegema, D. (2015) · Communication Methods and Measures · doi:10.1080/19312458.2015.1096333 · Cited by 41 · see citing works
“This not only leads to a severe publication bias, leaving only “significant” findings publicly available (Dickersin, 1990; Franco, Malhotra, & Simonovits, 2014; Mervis, 2014; Rosenthal, 1979), but it also incentivizes researchers to report significant findings.”
BackgroundInfluentialKepes, S., & Thomas, M. A. (2018) · European Journal of Information Systems · doi:10.1080/0960085x.2017.1390188 · Cited by 40 · see citing works
“Recently, research in the social and behavioural sciences, including publications in Science , have started to examine the selective reporting of scientific findings (i.e., publication bias; e.g., Franco, Malhotra, & Simonovits, 2014; Mervis, 2014).”
BackgroundMethodTransparency, Reproducibility, and the Credibility of Economics Research
Christensen, G., & Miguel, E. A. (2016) · National Bureau of Economic Research · doi:10.3386/w22989 · Cited by 30 · see citing works
Pre-registration and Results-Free Review in Observational and Qualitative Research
Jacobs, A. M. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.009 · Cited by 30 · see citing works
McClain, M. B., Callan, G. L., Harris, B., Floyd, R. G., Haverkamp, C. R., Golson, M. E., Longhurst, D. N., & Benallie, K. J. (2021) · Journal of School Psychology · doi:10.1016/j.jsp.2020.11.002 · Cited by 23 · see citing works
Mood effects on semantic processes: Behavioural and electrophysiological evidence
Naranowicz, M. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.1014706 · Cited by 22 · see citing works
“Speculatively, a potential cause of the frequently reported dichotomous mood-dependent processing styles may also be rooted in the “file drawer” phenomenon – a common scientific practice of not publishing research producing null results (Mervis, 2014).”
Harnessing Chemical Ecology for Environment-Friendly Crop Protection
Kang, S., Lumactud, R. A., Li, N., Bell, T. H., Kim, H.-S., Park, S., & Lee, Y. (2021) · Phytopathology · doi:10.1094/phyto-01-21-0035-rvw · Cited by 21 · see citing works
“…(Krause et al. 2006), most peptaibols cataloged in Peptaibiotics Database (Zeilinger et al. 2016) are produced by Trichoderma spp. Alamethicin, which was first isolated from T. viride (Meyer and Reusser 1967), kills bacteria and fungi by forming pores in their plasmamembrane (Pieta et al. 2012).”
BackgroundKhodadadifar, T., Soltaninejad, Z., Ebneabbasi, A., Eickhoff, C., Sorg, C., van Eimeren, T., Vogeley, K., Zarei, M., Eickhoff, S., & Tahmasian, M. (2021) · Human Brain Mapping · doi:10.1002/hbm.25722 · Cited by 13 · see citing works
“Accordingly, we had to exclude 22 eligible studies because of their null findings (Table S1), despite knowing the importance of their valuable results (Mervis, 2014).”
BackgroundAction observation effects reflect the modular organization of the human motor system
Hilt, P. M., Bartoli, E., Ferrari, E., Jacono, M., Fadiga, L., & D’Ausilio, A. (2017) · Cortex · doi:10.1016/j.cortex.2017.07.020 · Cited by 18 · see citing works
“Our findings, together with the known difficulty in publishing negative results (Matosin, Frank, Engel, Lum, & Newell, 2014; Mervis, 2014), suggest that a quite significant number of unpublished studies did not find AOEs using classical CSE measures (i.e., MEPs).”
ResultParent–child attachment as a mechanism of intergenerational (dis)advantage
Moullin, S., Waldfogel, J., & Washbrook, E. (2017) · Families Relationships and Societies · doi:10.1332/204674317x15071998786492 · Cited by 16 · see citing works
“As in any review, null findings are likely to be under-represented due to publication bias (Mervis, 2014).”
BackgroundMaking Research Data Accessible
Kapiszewski, D., & Karcher, S. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.008 · Cited by 15 · see citing works
How null results can be significant for physics education research
Conlin, L. D., Kuo, E., & Hallinen, N. R. (2019) · Physical Review Physics Education Research · doi:10.1103/physrevphyseducres.15.020104 · Cited by 14 · see citing works
McCormack, M. (2020) · Sexualities · doi:10.1177/1363460720905519 · Cited by 12 · see citing works
“Publishing bias; it might be easier to publish qualitative research that reports harms than positive trends (similar to the difficulty quantitative researchers have in publishing studies which do not find significant effects (Mervis, 2014)).”
BackgroundHuckle, K., & Silva, A. (2020) · Religions · doi:10.3390/rel11050249 · Cited by 12 · see citing works
“While there is a strong tendency to omit non-findings from publications, we wish to be a part of a growing body of researchers who acknowledge the importance of non-findings to the scientific process (Franco et al. 2014; Leal et al. 2016; Mervis 2014).”
BackgroundFranke, H., Kwasnitschka, D., Schmutz, J. B., & Netland, T. H. (2024) · Journal of Operations Management · doi:10.1002/joom.1300 · Cited by 11 · see citing works
Till, C., Haverkamp, J., White, D., & Bhaduri, B. (2018) · The Journal of Defense Modeling and Simulation Applications Methodology Technology · doi:10.1177/1548512916679038 · Cited by 9 · see citing works
“Recent publications in Science highlighted exactly this frustration and call for a change in publishing culture.(96,97) The acknowledgement that a publication bias toward significant or targeted results exists has been known since the late 1950s, when Sterling investigated publishing decisions.”
BackgroundValues affirmation replication at the University of Illinois
Gutmann, B., & Stelzer, T. J. (2021) · Physical Review Physics Education Research · doi:10.1103/physrevphyseducres.17.020121 · Cited by 11 · see citing works
Chavarnakul, T., Lin, Y.-C., Khan, A., & Chen, S. (2024) · Emerging Science Journal · doi:10.28991/esj-2024-08-01-06 · Cited by 11 · see citing works
Krause, M. S. (2016) · Psychotherapy Research · doi:10.1080/10503307.2016.1222459 · Cited by 7 · see citing works
“…to discard a Tj that failed its t-test but produced an Ojk > 0, were such RCTjk even published (see, e.g., Hunter & Schmidt, 2004, pp. 493–509; Mervis, 2014; Sutton, 2009), is to risk discarding a Tj that may have benefitted some clients more than this Tk did (see Krause & Lutz, 2009a, on…”
BackgroundMethodReliability of Inference: Analogs of Replication in Qualitative Research
Fairfield, T., & Charman, A. E. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.012 · Cited by 5 · see citing works
Measurement Replication in Qualitative and Quantitative Studies
Reiter, D. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.011 · Cited by 4 · see citing works
Gerring, J., Mahoney, J., & Elman, C. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.001 · Cited by 5 · see citing works
Leal, D. L., Patterson, J. T., & Tafoya, J. R. (2016) · RSF The Russell Sage Foundation Journal of the Social Sciences · doi:10.7758/rsf.2016.2.3.07 · Cited by 5 · see citing works
“Annie Franco, Neil Malhotra, and Gabor Simonovits (2014) explain how null findings are an important part of the scientific process (see also Mervis 2014).”
BackgroundIdeological bias in the production of research findings
Borjas, G. J., & Breznau, N. (2026) · Science Advances · doi:10.1126/sciadv.adz7173 · Cited by 5 · see citing works
“…reasons that are not purely scientific ( 10 , 11 ); the “file drawer problem,” where some research findings never get submitted for publication ( 12 ); the use of questionable data and faking of results ( 13 ); and the noise and errors that enter the research production function because of…”
Replication for Quantitative Research
Jeremy Freese, A., & Peterson, D. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.010 · Cited by 4 · see citing works
Ciubotariu, I. I., & Bosch, G. (2023) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1011160 · Cited by 5 · see citing works
Transparency and Reproducibility: Conceptualizing the Problem
Christensen, G., & Miguel, E. A. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.006 · Cited by 4 · see citing works
Bègue, L., Zerhouni, O., & Jobard, F. (2020) · International Criminal Justice Review · doi:10.1177/1057567720953874 · Cited by 4 · see citing works
“For example, a study of 221 survey-based experiments funded by the National Science Foundation found that almost two thirds of studies yielding null findings were not submitted for publication (Mervis, 2014).”
BackgroundTargeted Learning for Pre-Analysis Plans in Public Health and Health Policy Research
Rose, S. (2015) · Observational Studies · doi:10.1353/obs.2015.0008 · Cited by 3 · see citing works
“Other arguments against mandatory registration of studies with pre-analysis plans include giving undue weight to less important unpublished findings and creating a culture where scientists are not trusted (Mervis, 2014).”
BackgroundTransparency and Reproducibility: Potential Solutions
Christensen, G., & Miguel, E. A. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.007 · Cited by 2 · see citing works
Gong, F., & Xu, J. (2020) · Sociological Spectrum · doi:10.1080/02732173.2020.1753133 · Cited by 2 · see citing works
“…in the field of health, because of the well-known “file drawer” phenomenon that confirmatory, positive, and significant results are more likely to be embraced, while alternative, non-confirmatory, or null findings are often disregarded or simply wiped off the table (Mervis 2014; Rosenthal 1979).”
BackgroundWhat’s Wrong with Replicating the Old Boys’ Networks?
Teele, D. L. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.016 · Cited by 2 · see citing works
Gerring, J. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.013 · Cited by 2 · see citing works
Gerring, J., & Cojocaru, L. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.005 · Cited by 1 · see citing works
Deep pockets needed? Financial health and organic certification adoption in the European Union
Tennhardt, L. M., Bozzola, M., & Grovermann, C. (2026) · European Review of Agricultural Economics · doi:10.1093/erae/jbag013 · Cited by 2 · see citing works
Lieberman, E. S. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.003 · Cited by 2 · see citing works
Gerring, J., Mahoney, J., & Elman, C. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.018 · Cited by 1 · see citing works
Gerring, J., Karcher, S., & Apfeld, B. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.015 · Cited by 2 · see citing works
Gerring, J. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.014 · Cited by 1 · see citing works
Gross, N., & Robertson, C. T. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.017 · Cited by 1 · see citing works
Analysis of Variance and hypothesis testing
McIntosh, M. S. (2018) · ASSA, CSSA and SSSA · doi:10.2134/appliedstatistics.2016.0009.c2 · Cited by 1 · see citing works
Ideological Bias in Estimates of the Impact of ImmigrationPreprint
Borjas, G. J., & Breznau, N. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.5069454 · Cited by 1 · see citing works
Ideological Bias in Estimates of the Impact of ImmigrationPreprint
Borjas, G. J., & Breznau, N. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.5069989 · Cited by 1 · see citing works
The effect of stock market indexing on corporate tax avoidancePreprint
Young, A. (2015) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.1509.00136
“ms just outside. 1E-mail: alex.young.1@ndsu.edu. 1 1 Introduction The “file drawer” problem refers to the practice of researchers abandoning results that they believe journals are unlikely to publish [Mervis, 2015]. The belief is supported by evidence of publication bias in, for example, the social sciences: studies wi…”
BackgroundResultInfluentialNuanced negative result reporting
Lhachimi, S. K., Lehrer, D., Leschke, J., Nelson, M., & Weiffen, B. (2014) · Science · doi:10.1126/science.346.6207.308-b
Attachment Style, Tinder Use, and Permissive Sexual Attitude
Borges, T. A. (2017)
Chapter 2: Analysis of Variance and hypothesis testing
McIntosh, M. S., Glaz, B., & Yeater, K. M. (2018) · ACSESS publications · doi:10.2134/appliedstatistics.2016.0009
Nothing's plenty: The significance of null results in physics education researchPreprint
Conlin, L. D., Kuo, E., & Hallinen, N. R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.10071
“This points to a root cause of publication bias against null results -- an overemphasis on hypothesis testing using a pvalue cutoff [2].”
BackgroundResultOn the Measurement of ImportancePreprint
Sterck, O. C. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3386218
Liao, T. F. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.004
Elman, C., Gerring, J., & Mahoney, J. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.019
Use of population indices in cancer research: a scoping review
Hughes, M. C., Afrin, S., & Vaezi, M. (2022) · Journal of Global Health Economics and Policy · doi:10.52872/001c.38056
Breznau, N., Auspurg, K., Brüderl, J., Holzmeister, F., Nilsonne, G., Aczél, B., Clark, C. J., Nguyen, H. H. V., Varga, M. A., & Uhlmann, E. L. (2023) · doi:10.31222/osf.io/h9t5c
Association of Private Equity Investment in Health Care With Medical Education Disruptions
Caverzagie, K. J., Ryan, A. L., Brotherton, S. E., et al. (2025) · Academic Medicine · doi:10.1097/acm.0000000000006126
Use of population indices in cancer research: a scoping review
Hughes, M. C., Afrin, S., & Vaezi, M. (2022) · Journal of Global Health Economics and Policy · doi:10.7189/001c.38056
- Authors
- Mervis, J.
- Corresponding author
- Mervis, Jeffrey
- Venue
- Science
- Published
- 2014-08-29
- Volume
- 345
- Issue
- 6200
- Pages
- 992-992
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W2050398713
- Cited by
- 65
- References
- 0
- Citations by year
- 2026: 2 · 2025: 1 · 2024: 4 · 2023: 3 · 2022: 4 · 2021: 4 · 2020: 24 · 2019: 4 · 2018: 5 · 2017: 5 · 2016: 4 · 2015: 4 · 2014: 1
- Type
- other
- Language
- en
- Open access
- closed
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Keywords
- negative results, file drawer problem, unpublished data, research registry, social science research, publication bias, data sharing
- MeSH terms
- Data Interpretation, Statistical, Periodicals as Topic, Social Sciences, Research Report
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
