CoChat
Sign Up
PreprintCited by 313Meta-analysis and systematic reviewsStatistical Methods in EpidemiologyPsychology Research and Bibliometrics

The DOI matches its DataCite record; 40 of the 40 references it cites were found in a registry or an index.

Publication Bias (The "File-Drawer Problem") in Scientific Inference

Open access
Scargle, Jeffrey D.·1999·arXiv
DOI 10.48550/arxiv.physics/9909033arXiv physics/9909033
Open this paper in CoChat — free accountOpen PDF

Abstract

Publication bias arises whenever the probability that a study is published depends on the statistical significance of its results. This bias, often called the file-drawer effect since the unpublished results are imagined to be tucked away in researchers' file cabinets, is potentially a severe impediment to combining the statistical results of studies collected from the literature. With almost any reasonable quantitative model for publication bias, only a small number of studies lost in the file-drawer will produce a significant bias. This result contradicts the well known Fail Safe File Drawer (FSFD) method for setting limits on the potential harm of publication bias, widely used in social, medical and psychic research. This method incorrectly treats the file drawer as unbiased, and almost always misestimates the seriousness of publication bias. A large body of not only psychic research, but medical and social science studies, has mistakenly relied on this method to validate claimed discoveries. Statistical combination can be trusted only if it is known with certainty that all studies that have been carried out are included. Such certainty is virtually impossible to achieve in literature surveys.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.physics/9909033

Cites

40 of 40 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 40 references

Research works printed but not linked: 5

  1. The treatment of the file-drawer effect, in Chapter 5, Section II.B, is essentially identical to that in (Rosenthal, 1979).
  2. Sturrock, P. A. (1994). Applied Scientific Inference. Journal of Scientific Exploration , Vol. 8 , No. 4, pp. 491-508.
  3. Sturrock, P. A. (1997). A Bayesian Maximum-Entropy Approach to Hypothesis Testing, for Application to RNG and Similar Experiments. Journal of Scientific Exploration , Vol. 11 , No. 2, pp. x-xx.
  4. Taylor, S., and Tweedie, R. (1998). A non-parametric 'trim and fill' method of assessing publication bias in meta-analysis, U. Colorado preprint?
  5. Taylor, S., and Tweedie, R. (1998). Trim and fill: A simple method of assessing publication bias in meta-analysis, U. Colorado preprint?

Cited by

Sorted by citations

See all 313

Related

Metadata

Authors
Scargle, J. D.
Affiliations
Ames Research Center (US)
Corresponding author
Scargle, Jeffrey D.
ORCID
Scargle, Jeffrey D.
Venue
arXiv
Publisher
arXiv
Published
1999
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.physics/9909033
arXiv
physics/9909033
OpenAlex
W2127635577
Cited by
313
References
40
Citations by year
2026: 1 · 2025: 3 · 2024: 5 · 2023: 8 · 2022: 8 · 2021: 35 · 2020: 11 · 2019: 28 · 2018: 23 · 2017: 15 · 2016: 19 · 2015: 20 · 2014: 18 · 2013: 19 · 2012: 15
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/assumed-1991-2003/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • CERN Document Server (European Organization for Nuclear Research) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
Keywords
publication bias, meta-analysis, statistical significance, unpublished research, psychical research, medical research, social science research

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review