CoChat
Sign Up
PreprintCited by 313Meta-analysis and systematic reviewsStatistical Methods in EpidemiologyPsychology Research and Bibliometrics

Publication Bias (The "File-Drawer Problem") in Scientific Inference

Open access
Scargle, Jeffrey D.·1999·arXiv
DOI 10.48550/arxiv.physics/9909033arXiv physics/9909033
Open this paper in CoChat — free accountOpen PDF

Abstract

Publication bias arises whenever the probability that a study is published depends on the statistical significance of its results. This bias, often called the file-drawer effect since the unpublished results are imagined to be tucked away in researchers' file cabinets, is potentially a severe impediment to combining the statistical results of studies collected from the literature. With almost any reasonable quantitative model for publication bias, only a small number of studies lost in the file-drawer will produce a significant bias. This result contradicts the well known Fail Safe File Drawer (FSFD) method for setting limits on the potential harm of publication bias, widely used in social, medical and psychic research. This method incorrectly treats the file drawer as unbiased, and almost always misestimates the seriousness of publication bias. A large body of not only psychic research, but medical and social science studies, has mistakenly relied on this method to validate claimed discoveries. Statistical combination can be trusted only if it is known with certainty that all studies that have been carried out are included. Such certainty is virtually impossible to achieve in literature surveys.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.physics/9909033

Cites

40 of 40 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • If only significant findings are published, especially if based on underpowered experiments, this can lead to systematic distortions in the literature.

    “This problem can lead to systematic distortions in the literature if only significant findings are published, especially if these results are based on underpowered experiments (Scargle, 1999).”

    From With Little Power Comes Great Responsibility (2020) · cites this paper as Scargle, 1999

    Section 4.1 models a setting in which studies meeting a significance threshold are published and those that do not are excluded. The Conclusions state that publication bias can produce apparently significant but spurious results and bias analyses of the published literature. The cited passages do not establish the added point that this distortion is especially pronounced for underpowered experiments. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 5

  1. The treatment of the file-drawer effect, in Chapter 5, Section II.B, is essentially identical to that in (Rosenthal, 1979).
  2. Sturrock, P. A. (1994). Applied Scientific Inference. Journal of Scientific Exploration , Vol. 8 , No. 4, pp. 491-508.
  3. Sturrock, P. A. (1997). A Bayesian Maximum-Entropy Approach to Hypothesis Testing, for Application to RNG and Similar Experiments. Journal of Scientific Exploration , Vol. 11 , No. 2, pp. x-xx.
  4. Taylor, S., and Tweedie, R. (1998). A non-parametric 'trim and fill' method of assessing publication bias in meta-analysis, U. Colorado preprint?
  5. Taylor, S., and Tweedie, R. (1998). Trim and fill: A simple method of assessing publication bias in meta-analysis, U. Colorado preprint?

Cited by

Sorted by citations

Related

Authors
Scargle, J. D.
Affiliations
Ames Research Center (US)
Corresponding author
Scargle, Jeffrey D.
ORCID
Scargle, Jeffrey D.
Venue
arXiv
Publisher
arXiv
Published
1999
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.physics/9909033
arXiv
physics/9909033
OpenAlex
W2127635577
Cited by
313
References
40
Citations by year
2026: 1 · 2025: 3 · 2024: 5 · 2023: 8 · 2022: 8 · 2021: 35 · 2020: 11 · 2019: 28 · 2018: 23 · 2017: 15 · 2016: 19 · 2015: 20 · 2014: 18 · 2013: 19 · 2012: 15
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/assumed-1991-2003/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • CERN Document Server (European Organization for Nuclear Research) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
Keywords
publication bias, meta-analysis, statistical significance, unpublished research, psychical research, medical research, social science research

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review