“Positive” results increase down the Hierarchy of the Sciences
Open accessAbstract
The hypothesis of a Hierarchy of the Sciences with physical sciences at the top, social sciences at the bottom, and biological sciences in-between is nearly 200 years old. This order is intuitive and reflected in many features of academic life, but whether it reflects the "hardness" of scientific research--i.e., the extent to which research questions and results are determined by data and theories as opposed to non-cognitive factors--is controversial. This study analysed 2434 papers published in all disciplines and that declared to have tested a hypothesis. It was determined how many papers reported a "positive" (full or partial) or "negative" support for the tested hypothesis. If the hierarchy hypothesis is correct, then researchers in "softer" sciences should have fewer constraints to their conscious and unconscious biases, and therefore report more positive outcomes. Results confirmed the predictions at all levels considered: discipline, domain and methodology broadly defined. Controlling for observed differences between pure and applied disciplines, and between papers testing one or several hypotheses, the odds of reporting a positive result were around 5 times higher among papers in the disciplines of Psychology and Psychiatry and Economics and Business compared to Space Science, 2.3 times higher in the domain of social sciences compared to the physical sciences, and 3.4 times higher in studies applying behavioural and social methodologies on people compared to physical and chemical studies on non-biological material. In all comparisons, biological studies had intermediate values. These results suggest that the nature of hypotheses tested and the logical and methodological rigour employed to test them vary systematically across disciplines and fields, depending on the complexity of the subject matter and possibly other factors (e.g., a field's level of historical and/or intellectual development). On the other hand, these results support the scientific status of the social sciences against claims that they are completely subjective, by showing that, when they adopt a scientific approach to discovery, they differ from the natural sciences only by a matter of degree.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
More than 80% of scientific publications in various sciences report positive results.
“which is in line with a study by Fanelli showing that more than 80% of scientific publications in various sciences report positive results (22).”
From Statistical and Methodological Considerations for the Interpretation of Intranasal Oxytocin Studies (2016) · cites this paper as [22]
The Results report that 2045 of 2434 papers (84%) across 20 disciplines gave positive or partial support for a tested hypothesis. But the study included only papers that explicitly tested a hypothesis, and the Discussion notes its focus on English-language scientific journals; it does not establish that more than 80% of scientific publications generally report positive results. (LLM, conf=0.85)
Experimental — accuracy not yet measuredThere is a problem of positive bias in reviews toward or in favor of the testing hypothesis.
“Several authors alert for the problem of a positive bias in reviews toward or in favor of the testing hypothesis (e.g., Fanelli, 2010).”
From Contextual Cognitive-Behavioral Therapies Across the Psychosis Continuum: A Review of Evidence for Schizophrenia, Schizoaffective and Bipolar Disorders (2017) · cites this paper as Fanelli, 2010
The Introduction (page unavailable) says confirmation bias can affect how scientists collect, interpret, and publish data, producing an excess of results supporting tested hypotheses. The Discussion section on non-publication (page unavailable) also describes a bias against negative results. Neither passage establishes that this bias occurs specifically in reviews; the paper discusses research papers and publication decisions. (LLM, conf=0.83)
Experimental — accuracy not yet measuredPotentially problematic practices include selective reporting, selective analysis, and insufficient specification of the conditions necessary or sufficient to obtain the results.
“Potentially problematic practices include selective reporting, selective analysis, and insufficient specification of the conditions necessary or sufficient to obtain the results ( 1223 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [15]
The Discussion sections on non-publication (2C), flexibility in analysis (2A), and manipulation of results (2D) describe keeping negative results unpublished, discretion in analyzing data, and dropping outliers. These support selective reporting and analysis as potentially problematic practices. The copy does not specifically address insufficient specification of the necessary or sufficient conditions for obtaining results. (LLM, conf=0.85)
Experimental — accuracy not yet measuredThe original studies may have inflated effect sizes due to publication, selection, reporting, or other biases.
“An important qualification about this result is the possibility that the original studies have inflated effect sizes due to publication, selection, reporting, or other biases ( 9, 1223 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [15]
The Discussion section on non-publication of negative or nonsignificant results describes publication and selection biases, while the Introduction says confirmation bias can affect data collection, analysis, interpretation, and publication. These passages support the possibility of bias in published studies, but they do not establish that the original studies cited here had inflated effect-size estimates. Our copy provides no page numbers for these passages. (LLM, conf=0.82)
Experimental — accuracy not yet measuredConcerns about reproducibility are widespread across disciplines.
“Concerns about reproducibility are widespread across disciplines ( 921 ).”
From Estimating the reproducibility of psychological science (2015) · cites this paper as [15]
The Introduction (page unavailable) says scientific controversies and cases of 'pathological science' occur across fields. The Discussion (page unavailable) examines cross-disciplinary differences in positive results and notes possible consequences for the reliability of scientific literature. These passages support broad concerns about research reliability, but they do not establish that concerns specifically about reproducibility are widespread across disciplines. (LLM, conf=0.77)
Experimental — accuracy not yet measured
Most papers in behavioral intranasal oxytocin studies report positive findings.
“When reading the literature on behavioral IN-OT studies in humans it is obvious that most papers report positive findings, which is in line with a study by Fanelli showing that more than 80% of scientific publications in various sciences report positive results (22).”
From Statistical and Methodological Considerations for the Interpretation of Intranasal Oxytocin Studies (2016) · cites this paper as [22]
Positive results are much more likely to be published than negative results.
“Positive results are much more likely to be published than negative results 5-7”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [7]
Positive results are more likely to be published than negative results.
“Positive results -- finding that a treatment is effective or that there is a relationship between two variables -- are more likely to be published than negative results 6-12 .”
From Initial evidence of research quality of registered reports compared with the standard publishing model (2021) · cites this paper as [7]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Statistical power analyses using G∗Power 3.1: tests for correlation and regression analyses
Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (2009) · Behavior Research Methods · doi:10.3758/brm.41.4.1149 · Cited by 37,615 · see citing works
<i>The Structure of Scientific Revolutions</i>
Kuhn, T. S., & Schlegel, R. (1963) · Physics Today · doi:10.1063/1.3050879 · Cited by 35,538 · see citing works
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505 · see citing works
The Structure of Scientific Revolutions.
Böhm, D., & Kühn, T. (1964) · The Philosophical Quarterly · doi:10.2307/2217783 · Cited by 9,193 · see citing works
The Logic of Scientific Discovery
Bhaskar, R. (2013) · Routledge eBooks · doi:10.4324/9780203090732-9 · Cited by 7,765 · see citing works
Confirmation Bias: A Ubiquitous Phenomenon in Many Guises
Nickerson, R. S. (1998) · Review of General Psychology · doi:10.1037/1089-2680.2.2.175 · Cited by 7,267 · see citing works
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · see citing works
Academic Tribes and Territories: Intellectual Enquiry and the Cultures of Disciplines
Bayer, A. E., Becher, T., & Keynes, M. (1991) · The Journal of Higher Education · doi:10.2307/1982146 · Cited by 2,730 · see citing works
<i>The Logic of Scientific Discovery</i>
Popper, K. R., & Weiss, G. (1959) · Physics Today · doi:10.1063/1.3060577 · Cited by 2,717 · see citing works
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989 · see citing works
Meehl, P. E. (1978) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.46.4.806 · Cited by 1,952 · see citing works
Fanelli, D. (2009) · PLoS ONE · doi:10.1371/journal.pone.0005738 · Cited by 1,911 · cited 1× · see citing works
“Survey data suggests that outright scientific misconduct is relatively rare compared to more subtle forms of bias, although it is probably higher than commonly assumed, particularly in medical/clinical research [69].”
BackgroundSelf-citationThe characteristics of subject matter in different academic areas.
Biglan, A. (1973) · Journal of Applied Psychology · doi:10.1037/h0034701 · Cited by 1,761 · see citing works
Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias
Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A.-W., Cronin, E., Decullier, E., Easterbrook, P. J., Von Elm, E., Gamble, C., et al. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003081 · Cited by 1,546 · see citing works
Academic tribes and territories: Intellectual enquiry and the culture of the disciplines
Bazerman, C. (1990) · English for Specific Purposes · doi:10.1016/0889-4906(90)90017-7 · Cited by 1,352 · see citing works
ON OUR KNOWLEDGE OF THE EXTERNAL WORLD
Russell, B. (2009) · Routledge eBooks · doi:10.4324/9780203875360-5 · Cited by 850 · see citing works
Comte, A. (2002) · Classiques des sciences sociales. · doi:10.1522/cla.coa.cou2 · Cited by 756 · see citing works
Coefficients of Determination for Multiple Logistic Regression Analysis
Menard, S. (2000) · The American Statistician · doi:10.1080/00031305.2000.10474502 · Cited by 739 · see citing works
Econometrics: Alchemy or Science?
Hendry, D. F. (2000) · doi:10.1093/0198293542.001.0001 · Cited by 635 · see citing works
When things strike back: a possible contribution of ‘science studies’ to the social sciences
Latour, B. (2000) · British Journal of Sociology · doi:10.1111/j.1468-4446.2000.00107.x · Cited by 580 · see citing works
How much variance can be explained by ecologists and evolutionary biologists?
Møller, A., & Jennions, M. D. (2002) · Oecologia · doi:10.1007/s00442-002-0952-2 · Cited by 553 · see citing works
When things strike back: a possible contribution of 'science studies' to the social sciences
Latour, B. (2000) · British Journal of Sociology · doi:10.1080/000713100358453 · Cited by 545 · see citing works
The golem: what everyone should know about science
(1994) · Choice Reviews Online · doi:10.5860/choice.31-2650 · Cited by 532 · see citing works
The Hierarchy of the Sciences?
Cole, S. (1983) · American Journal of Sociology · doi:10.1086/227835 · Cited by 485 · see citing works
Publication and Related Biases
Song, F., Eastwood, A. J., Gilbody, S. M., Duley, L., & Sutton, A. J. (2001) · SAGE Publications Ltd eBooks · doi:10.4135/9781848608344.n21 · Cited by 470 · see citing works
Sterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995) · The American Statistician · doi:10.1080/00031305.1995.10476125 · Cited by 453 · see citing works
Jary, D., & Jary, J. (1991) · Cited by 448 · see citing works
How hard is hard science, how soft is soft science? The empirical cumulativeness of research.
Hedges, L. V. (1987) · American Psychologist · doi:10.1037/0003-066x.42.5.443 · Cited by 435 · see citing works
Publication and related biases.
Song, Eastwood, Gilbody, Duley, & Sutton (2000) · Health Technology Assessment · doi:10.3310/hta4100 · Cited by 427 · see citing works
The Philosophy of the Inductive Sciences
Whewell, W. (2014) · Cambridge University Press eBooks · doi:10.1017/cbo9781139644679 · Cited by 425 · see citing works
The Effects of Discipline on Deep Approaches to Student Learning and College Outcomes
Laird, T. F. N., Shoup, R., Kuh, G. D., & Schwarz, M. J. (2008) · Research in Higher Education · doi:10.1007/s11162-008-9088-5 · Cited by 387 · see citing works
THE PHILOSOPHY OF THE INDUCTIVE SCIENCES
Whewell, W. (2012) · Cambridge University Press eBooks · doi:10.1017/cbo9781139105415.012 · Cited by 362 · see citing works
Our Knowledge of the External World
Russell, B. (2009) · Routledge eBooks · doi:10.4324/9780203875360 · Cited by 337 · see citing works
Collins (2009) · Cited by 332 · see citing works
Hypothesis Testing in Ecology: Psychological Aspects and the Importance of Theory Maturation
Loehle, C. (1987) · The Quarterly Review of Biology · doi:10.1086/415619 · Cited by 288 · see citing works
Publication bias in ecology and evolution: an empirical assessment using the ‘trim and fill’ method
Jennions, M. D., & Møller, A. P. (2002) · Biological reviews/Biological reviews of the Cambridge Philosophical Society · doi:10.1017/s1464793101005875 · Cited by 279 · see citing works
A survey of the statistical power of research in behavioral ecology and animal behavior
Jennions, M. D. (2003) · Behavioral Ecology · doi:10.1093/beheco/14.3.438 · Cited by 278 · see citing works
Palmer, A. R. (1999) · The American Naturalist · doi:10.1086/303223 · Cited by 264 · see citing works
Age, aging, and age structure in science
Zuckerman, H., & Merton, R. K. (1968) · Cited by 252 · see citing works
Almost all articles on cancer prognostic markers report statistically significant results
Kyzas, P., Denaxa-Kyza, D., & Ioannidis, J. P. A. (2007) · European Journal of Cancer · doi:10.1016/j.ejca.2007.08.030 · Cited by 231 · see citing works
The Limitations of Social Research
Shipman, M. (2014) · Routledge eBooks · doi:10.4324/9781315840727 · Cited by 205 · see citing works
A nice derangement of epistemes: Post-positivism in the study of science from quine to latour
Turner, S. L. (2005) · Science Education · doi:10.1002/sce.20080 · Cited by 204 · see citing works
A nice derangement of epistemes: post-positivism in the study of science from Quine to Latour
(2004) · Choice Reviews Online · doi:10.5860/choice.42-1488 · Cited by 187 · see citing works
BLIND ANALYSIS IN NUCLEAR AND PARTICLE PHYSICS
Klein, J. R., & Roodman, A. J. (2005) · Annual Review of Nuclear and Particle Science · doi:10.1146/annurev.nucl.55.090704.151521 · Cited by 162 · see citing works
Scientific Graphs and the Hierarchy of the Sciences:
Smith, L. D., Best, L. A., Stubbs, D. A., et al. (2000) · Social Studies of Science · doi:10.1177/030631200030001003 · Cited by 140 · see citing works
The Biglan classification revisited
Stoecker, J. L. (1993) · Research in Higher Education · doi:10.1007/bf00991854 · Cited by 137 · see citing works
The golem at large: what you should know about technology
(1999) · Choice Reviews Online · doi:10.5860/choice.36-5640 · Cited by 127 · see citing works
Why sociology doesn't make progress like the natural sciences
Cole, S. (1994) · Sociological Forum · doi:10.1007/bf01476359 · Cited by 123 · see citing works
Langmuir, I., & Hall, R. N. (1989) · Physics Today · doi:10.1063/1.881205 · Cited by 119 · see citing works
The hard sciences and the soft: some sociological observations.
(1967) · Cited by 117 · see citing works
The Subjectivity of Scientists and the Bayesian Approach
Press, S. J., & Tanur, J. M. (2001) · Wiley series in probability and statistics · doi:10.1002/9781118150634 · Cited by 116 · see citing works
The Golem at Large: What you Should Know about Technology
Collins, H. M., Pinch, T. J. (. J., & Cambridge University Press (1999) · Information Technology and People · doi:10.1108/itp.1999.12.2.5.2 · Cited by 113 · see citing works
Simonton, D. K. (2004) · Review of General Psychology · doi:10.1037/1089-2680.8.1.59 · Cited by 107 · see citing works
Validation of the Biglan model
Smart, J. C., & Elton, C. F. (1982) · Research in Higher Education · doi:10.1007/bf00976699 · Cited by 100 · see citing works
Comte's philosophy of the sciences
Lewes, G. H., & Pyle, A. (1887) · Medical Entomology and Zoology · Cited by 88 · see citing works
Heritabilities and paradigm shifts
Alatalo, R. V., Mappes, J., & Elgar, M. A. (1997) · Nature · doi:10.1038/385402a0 · Cited by 81 · see citing works
Researcher Incentives and Empirical MethodsPreprint
Glaeser, E. (2006) · SSRN Electronic Journal · doi:10.2139/ssrn.934557 · Cited by 73 · see citing works
Physics of the Photographic Latent Image
Tani, T. (1989) · Physics Today · doi:10.1063/1.881181 · Cited by 69 · see citing works
We cannot live by scepticism alone
Collins, H. M. (2009) · Nature · doi:10.1038/458030a · Cited by 69 · see citing works
Experimenter effects in behavioral research, Enlarged ed.
Rosenthal, R. (1976) · Cited by 67 · see citing works
Econometrics – Alchemy or Science?
Hendry, D. F. (2000) · doi:10.1093/0198293542.003.0002 · Cited by 60 · see citing works
Cole, S. (2018) · Routledge eBooks · doi:10.4324/9781351326360 · Cited by 57 · see citing works
Seeing What We Want to See: Confirmation Bias in Animal Behavior Research
Marsh, D. M., & Hanlon, T. J. (2007) · Ethology · doi:10.1111/j.1439-0310.2007.01406.x · Cited by 53 · see citing works
Microparadigms: Chains of collective reasoning in publications about molecular interactions
Rzhetsky, A., Iossifov, I. H., Loh, J. M., & White, K. P. (2006) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0600591103 · Cited by 52 · see citing works
Woods, S. P., Rippeth, J. D., Conover, E., Carey, C. L., Parsons, T. D., & Tröster, A. I. (2006) · The Clinical Neuropsychologist · doi:10.1080/13854040500203290 · Cited by 50 · see citing works
Statistical power of articles published in three health-psychology related journals.
Maddock, J. E., & Rossi, J. S. (2001) · Health Psychology · doi:10.1037/0278-6133.20.1.76 · Cited by 46 · see citing works
Simonton, D. K. (2006) · Review of General Psychology · doi:10.1037/1089-2680.10.2.98 · Cited by 45 · see citing works
Inadequate Statistical Power of Negative Clinical Trials in Urological Literature
Breau, R. H., Carnat, T. A., & Gaboury, I. (2006) · The Journal of Urology · doi:10.1016/s0022-5347(06)00505-2 · Cited by 43 · see citing works
Biomechanical testing in experimental bone interventions—May the power be with you
Leppänen, O., Sievänen, H. T., & Järvinen, T. L. N. (2008) · Journal of Biomechanics · doi:10.1016/j.jbiomech.2008.03.017 · Cited by 42 · see citing works
Adams, D., & Pimple, K. D. (2005) · Accountability in Research · doi:10.1080/08989620500217495 · Cited by 40 · see citing works
Graph use in psychology and other sciences
Best, L. A., Smith, L. D., & Stubbs, D. A. (2001) · Behavioural Processes · doi:10.1016/s0376-6357(01)00156-5 · Cited by 40 · see citing works
Whitehead, R. (2003) · Cited by 31 · see citing works
(1976) · Elsevier eBooks · doi:10.1016/c2013-0-02792-2 · Cited by 30 · see citing works
Differentiation in graduate education
(1986) · doi:10.1007/bf00991880 · Cited by 23 · see citing works
Fraud in Science An Economic Approach
Wible, J. R. (1992) · Philosophy of the Social Sciences · doi:10.1177/004839319202200101 · Cited by 21 · see citing works
Uk, C., & Forsyth, M. (2005) · Cited by 19 · see citing works
(1968) · Studies in logic and the foundations of mathematics · doi:10.1016/s0049-237x(08)70500-7 · Cited by 18 · see citing works
An initial survey of fractional graph and table area in behavioral journals
Kubina, R. M., Kostewicz, D. E., & Datchuk, S. M. (2008) · The Behavior Analyst · doi:10.1007/bf03392161 · Cited by 6 · see citing works
Fan, Y., Lu, H., Liang, Z., et al. (2006) · Nuclear Instruments and Methods in Physics Research Section A Accelerators Spectrometers Detectors and Associated Equipment · doi:10.1016/j.nima.2006.10.061 · Cited by 6 · see citing works
Bandwagon effects and error bars in particle physics
Jeng, M. (2006) · Nuclear Instruments and Methods in Physics Research Section A Accelerators Spectrometers Detectors and Associated Equipment · doi:10.1016/j.nima.2006.11.024 · Cited by 6 · see citing works
A conjecture concerning the ranking of the sciences
Humphreys, P. W. (1990) · Topoi · doi:10.1007/bf00135895 · Cited by 6 · see citing works
The pretense of psychology as science: The myth of mental illness in Statu Nascendi
SZASZ, T. S. (2006) · Current Psychology · doi:10.1007/s12144-006-1015-3 · Cited by 4 · see citing works
Why the Con Hasn't Been Taken Out of Econometrics
Zelder, M. (2007) · Eastern Economic Journal · doi:10.1057/palgrave.eej.9050006 · Cited by 3 · see citing works
Research bias: Some preliminary findings
Feigenbaum, S. K., & Levy, D. M. (1996) · Knowledge, technology & policy/Knowledge in society/Knowledge, technology, & policy · doi:10.1007/bf02696303 · Cited by 2 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- [8]Conant J 1950 Harvard case studies in experimental science. Cambridge, Mass. Harvard University Press
- [23]Rosenberg A 2006 Philosphy of science: A contemporary introduction. Moser PK Oxon, UK Routledge
- [61]Rosenthal R 1976 Experimenter effects in behavioural research. Enlarged edition. New York Irvington Publishers, Inc.
Cited by
Sorted by citations
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · see citing works
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397 · see citing works
Van Dam, N. T., van Vugt, M. K., Vago, D. R., Schmalzl, L., Saron, C. D., Olendzki, A., Meissner, T., Lazar, S. W., Kerr, C. E., Gorchov, J., et al. (2018) · Perspectives on Psychological Science · doi:10.1177/1745691617709589 · Cited by 1,506 · see citing works
Nosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301 · see citing works
Negative results are disappearing from most disciplines and countries
Fanelli, D. (2012) · Scientometrics · doi:10.1007/s11192-011-0494-7 · Cited by 1,248 · see citing works
Self-citationA call for transparent reporting to optimize the predictive value of preclinical research
Landis, S. C., Amara, S. G., Asadullah, K., Austin, C. P., Blumenstein, R., Bradley, E. W., Crystal, R. G., Darnell, R. B., Ferrante, R. J., Fillit, H., et al. (2012) · Nature · doi:10.1038/nature11556 · Cited by 1,196 · see citing works
Reproducibility in Science: Improving the Standard for Basic and Preclinical Research
Begley, C. G., & Ioannidis, J. P. A. (2015) · Circulation Research · doi:10.1161/circresaha.114.303819 · Cited by 1,035 · see citing works
The rules of the game called psychological science
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459060 · Cited by 800 · see citing works
Do Pressures to Publish Increase Scientists' Bias? An Empirical Support from US States Data
Fanelli, D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010271 · Cited by 805 · see citing works
Self-citationHow to Make More Published Research True
Ioannidis, J. P. A. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001747 · Cited by 760 · see citing works
Replications in Psychology Research: How Often Do They Really Occur?
Makel, M. C., Plucker, J. A., & Hegarty, B. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612460688 · Cited by 750 · see citing works
Replicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770 · see citing works
Registered Reports: A Method to Increase the Credibility of Published Results
Nosek, B. A., & Lakens, D. (2014) · Social Psychology · doi:10.1027/1864-9335/a000192 · Cited by 716 · see citing works
The Value of Direct Replication
Simons, D. J. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691613514755 · Cited by 630 · see citing works
Ferguson, C. J., & Heene, M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459059 · Cited by 603 · see citing works
Facts Are More Important Than Novelty: Replication in the Education Sciences
Makel, M. C., & Plucker, J. A. (2014) · Educational Researcher · doi:10.3102/0013189x14545513 · Cited by 609 · see citing works
Why Science Is Not Necessarily Self-Correcting
Ioannidis, J. P. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612464056 · Cited by 563 · see citing works
Verhage, M. L., Schuengel, C., Madigan, S., Fearon, R. M. P., Oosterman, M., Cassibba, R., Bakermans-Kranenburg, M. J., & van IJzendoorn, M. H. (2016) · Psychological Bulletin · doi:10.1037/bul0000038 · Cited by 550 · see citing works
Detecting and avoiding likely false‐positive findings – a practical guide
Forstmeier, W., Wagenmakers, E., & Parker, T. H. (2017) · Biological Reviews · doi:10.1111/brv.12315 · Cited by 436 · see citing works
The ironic effect of significant results on the credibility of multiple-study articles.
Schimmack, U. (2012) · Psychological Methods · doi:10.1037/a0029487 · Cited by 455 · see citing works
Kühberger, A., Fritz, A., & Scherndl, T. (2014) · PLoS ONE · doi:10.1371/journal.pone.0105825 · Cited by 399 · see citing works
McShane, B. B., Böckenholt, U., & Hansen, K. T. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616662243 · Cited by 398 · see citing works
Hakulinen, C., Elovainio, M., Pulkki-Råback, L., Virtanen, M., Kivimäki, M., & Jokela, M. (2015) · Depression and Anxiety · doi:10.1002/da.22376 · Cited by 418 · see citing works
Evolution of Reporting <i>P</i> Values in the Biomedical Literature, 1990-2015
Chavalarias, D., Wallach, J. D., Li, A. H. T., & Ioannidis, J. P. A. (2016) · JAMA · doi:10.1001/jama.2016.1952 · Cited by 370 · see citing works
Meta-assessment of bias in science
Fanelli, D., Costas, R., & Ioannidis, J. P. A. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1618569114 · Cited by 364 · see citing works
Self-citationQuestionable research practices in ecology and evolution
Fraser, H., Parker, T., Nakagawa, S., Barnett, A., & Fidler, F. (2018) · PLOS ONE · doi:10.1371/journal.pone.0200303 · Cited by 346 · see citing works
Statistical and Methodological Considerations for the Interpretation of Intranasal Oxytocin Studies
Walum, H., Waldman, I. D., & Young, L. J. (2016) · Biological Psychiatry · doi:10.1016/j.biopsych.2015.06.016 · Cited by 345 · see citing works
“When reading the literature on behavioral IN-OT studies in humans it is obvious that most papers report positive findings, which is in line with a study by Fanelli showing that more than 80% of scientific publications in various sciences report positive results (22).”
ResultImplementing Reproducible Research
Stodden, V. (2014) · doi:10.1201/b16868 · Cited by 314 · see citing works
Kostyrka-Allchorne, K., Cooper, N. R., & Simpson, A. (2017) · Developmental Review · doi:10.1016/j.dr.2016.12.002 · Cited by 302 · see citing works
Soderberg, C. K., Errington, T. M., Schiavone, S. R., Bottesini, J., Thorn, F. S., Vazire, S., Esterling, K. M., & Nosek, B. A. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01142-4 · Cited by 289 · see citing works
Fanelli, D., Costas, R., & Larivière, V. (2015) · PLOS ONE · doi:10.1371/journal.pone.0127556 · Cited by 268 · see citing works
Self-citationA Short (Personal) Future History of Revolution 2.0
Spellman, B. A. (2015) · Perspectives on Psychological Science · doi:10.1177/1745691615609918 · Cited by 213 · see citing works
Transparency in Ecology and Evolution: Real Problems, Real Solutions
Parker, T. H., Forstmeier, W., Koricheva, J., Fidler, F., Hadfield, J. D., Chee, Y. E., Kelly, C. D., Gurevitch, J., & Nakagawa, S. (2016) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2016.07.002 · Cited by 201 · see citing works
D. Chambers, C., 1 Cardiff University Brain Research Imaging Centre, S. O. P., Feredoes, E., D. Muthukumaraswamy, S., & J. Etchells, P. (2014) · AIMS Neuroscience · doi:10.3934/neuroscience.2014.1.4 · Cited by 213 · see citing works
Dammhahn, M., & Almeling, L. (2012) · Animal Behaviour · doi:10.1016/j.anbehav.2012.08.014 · Cited by 208 · see citing works
The erring brain: Error‐related negativity as an endophenotype for OCD—A review and meta‐analysis
Riesel, A. (2019) · Psychophysiology · doi:10.1111/psyp.13348 · Cited by 170 · see citing works
Teixeira da Silva, J. A., & Dobránszki, J. (2015) · Accountability in Research · doi:10.1080/08989621.2014.899909 · Cited by 158 · see citing works
P-hacking and publication bias interact to distort meta-analytic effect size estimates
Friese, M., & Frankenbach, J. (2020) · Psychological Methods · doi:10.1037/met0000246 · Cited by 122 · see citing works
Coronado-Montoya, S., Levis, A. W., Kwakkenbos, L., Steele, R. J., Turner, E. H., & Thombs, B. D. (2016) · PLOS ONE · doi:10.1371/journal.pone.0153220 · Cited by 107 · see citing works
Commensuration Bias in Peer Review
Lee, C. J. (2015) · Philosophy of Science · doi:10.1086/683652 · Cited by 90 · see citing works
The Past, Present, and Future of Experimental Methods in the Social Sciences
Mize, T. D., & Manago, B. (2022) · Social Science Research · doi:10.1016/j.ssresearch.2022.102799 · Cited by 76 · see citing works
de Winter, J. C., & Dodou, D. (2015) · PeerJ · doi:10.7717/peerj.733 · Cited by 78 · see citing works
Where Are the Self-Correcting Mechanisms in Science?
Vazire, S., & Holcombe, A. O. (2022) · Review of General Psychology · doi:10.1177/10892680211033912 · Cited by 81 · see citing works
Novelty versus Replicability: Virtues and Vices in the Reward System of Science
Romero, F. (2017) · Philosophy of Science · doi:10.1086/694005 · Cited by 61 · see citing works
A Systems-Based Approach to Fostering Robust Science in Industrial-Organizational Psychology
Grand, J. A., Rogelberg, S. G., Allen, T. D., Landis, R. S., Reynolds, D. H., Scott, J. C., Tonidandel, S., & Truxillo, D. M. (2018) · Industrial and Organizational Psychology · doi:10.1017/iop.2017.55 · Cited by 52 · see citing works
Treanor, L., Frank, R. A., Cherpak, L. A., Dehmoobad Sharifabadi, A., Salameh, J.-P., Hallgrimson, Z., Fabiano, N., McGrath, T. A., Kraaijpoel, N., Yao, J., et al. (2020) · European Radiology · doi:10.1007/s00330-019-06568-z · Cited by 38 · see citing works
A retrospective analysis of dissemination biases in the brief alcohol intervention literature.
Tanner-Smith, E. E., & Polanin, J. R. (2015) · Psychology of Addictive Behaviors · doi:10.1037/adb0000014 · Cited by 26 · see citing works
“Whereas some empirical studies have reported that significant and/or positive results are more likely to be published in higher impact factor journals (De Oliveira, Chang, Kendall, Fitzgerald, & McCarthy, 2012; Etter & Stapleton, 2009; Jannot, Agoritsas, Gayet-Ageron, & Perneger, 2013; Timmer, Hilsden, Cole, Hailey, &…”
Research accomplishments that are too good to be true
Ioannidis, J. P. A. (2014) · Intensive Care Medicine · doi:10.1007/s00134-013-3100-z · Cited by 12 · see citing works
Martins, M. J. R. V., Castilho, P., Carvalho, C. B., Pereira, A. T., Santos, V., Gumley, A., & de Macedo, A. F. (2017) · European Psychologist · doi:10.1027/1016-9040/a000283 · Cited by 11 · see citing works
- Authors
- Fanelli, D.
- Affiliations
- University of Edinburgh (GB)
- Corresponding author
- Fanelli, Daniele
- ORCID
- Fanelli, Daniele
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2010-04-07
- Volume
- 5
- Issue
- 4
- Pages
- e10068
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC2850928
- OpenAlex
- W2018443796
- Cited by
- 734
- References
- 84
- Citations by year
- 2026: 24 · 2025: 35 · 2024: 50 · 2023: 47 · 2022: 41 · 2021: 64 · 2020: 74 · 2019: 61 · 2018: 52 · 2017: 74 · 2016: 43 · 2015: 53 · 2014: 47 · 2013: 35 · 2012: 21
- FWCI
- 22.31
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS ONE (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
- Funders
- European Commission (221441)
- Economic and Social Research Council (ES/F028180/1)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- positive results, hypothesis testing, research bias, publication bias, scientific rigor, physical sciences, methodology, scientific status of social sciences
- MeSH terms
- Biological Science Disciplines, Methods, Natural Science Disciplines, Publications, Research, Science, Social Sciences
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
