CoChat

Current Incentives for Scientists Lead to Underpowered Studies with Erroneous Conclusions

Open access
Higginson, Andrew D., Munafò, Marcus R. · 2016 · PLOS Biology

The DOI matches its Crossref record; the references it cites are not checked yet.

Abstract

We can regard the wider incentive structures that operate across science, such as the priority given to novel findings, as an ecosystem within which scientists strive to maximise their fitness (i.e., publication record and career success). Here, we develop an optimality model that predicts the most rational research strategy, in terms of the proportion of research effort spent on seeking novel results rather than on confirmatory studies, and the amount of research effort per exploratory study. We show that, for parameter values derived from the scientific literature, researchers acting to maximise their fitness should spend most of their effort seeking novel results and conduct small studies that have only 10%-40% statistical power. As a result, half of the studies they publish will report erroneous conclusions. Current incentive structures are in conflict with maximising the scientific value of research; we suggest ways that the scientific ecosystem could be improved.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1371/journal.pbio.2000995

Cites

Statistics

Figures

Fig 1 Fitness landscape for an individual researcher. An individual researcher is able to choose the parameters θ (y-axis) and SE; the x-axis shows the resultant power of exploratory studies, WE. White indicates high fitness, black low fitness. For small values of SE, few papers are accepted, while for high values of SE, few studies are carried out. For low values of θ, few novel studies are carried out. (A) γ = 0.09, ϕ = 0.9. The optimal strategy that maximises individual fitness is therefore to carry out many small exploratory studies with a power of around 15%. (B) γ = 0.055, ϕ = 0.55. A mixture of exploratory and confirmatory work should be carried out with slightly higher power (20%).
Fig 1 Fitness landscape for an individual researcher. An individual researcher is able to choose the parameters θ (y-axis) and SE; the x-axis shows the resultant power of exploratory studies, WE. White indicates high fitness, black low fitness. For small values of SE, few papers are accepted, while for high values of SE, few studies are carried out. For low values of θ, few novel studies are carried out. (A) γ = 0.09, ϕ = 0.9. The optimal strategy that maximises individual fitness is therefore to carry out many small exploratory studies with a power of around 15%. (B) γ = 0.055, ϕ = 0.55. A mixture of exploratory and confirmatory work should be carried out with slightly higher power (20%).CC BY · PubMed Central
Fig 2 Effect of varying the weighting given to published exploratory studies (γ). Parameter γ reflects the relative importance of published exploratory studies. The lines show predictions for two values of the probability that an effect is real (fE) and two values of the effect sizes rC and rE (solid: fE = 0.2, rC = rE = 0.21; dotted: fE = 0.3, rC = rE = 0.21; dashed: fE = 0.2, rC = rE = 0.32). The panels show (A) the optimal proportion of total sampling to spend on exploratory studies θ*, (B) the optimal sample size of exploratory studies SE*, (C) the resultant total number of published studies NE + NC, (D) the proportion of published studies that are confirmatory NC / (NE + NC), (E) the statistical power of exploratory studies WE, and (F) the proportion of published studies that draw incorrect conclusions (PF). Other values: SC = 120, T = 2,000, k = 20, α = 0.05, σ2 = 1, m = 3, and ϕ = 0.8. The chosen values for rC = rE reflect data reported by Richard and colleagues [14], where a correlation coefficient mode of 0.09 and a mean of 0.21 were observed. These values are in the middle of the range of effect sizes observed in meta-analyses across a number of biomedical research domains (range r ~ 0.15 to 0.50) [13].
Fig 2 Effect of varying the weighting given to published exploratory studies (γ). Parameter γ reflects the relative importance of published exploratory studies. The lines show predictions for two values of the probability that an effect is real (fE) and two values of the effect sizes rC and rE (solid: fE = 0.2, rC = rE = 0.21; dotted: fE = 0.3, rC = rE = 0.21; dashed: fE = 0.2, rC = rE = 0.32). The panels show (A) the optimal proportion of total sampling to spend on exploratory studies θ*, (B) the optimal sample size of exploratory studies SE*, (C) the resultant total number of published studies NE + NC, (D) the proportion of published studies that are confirmatory NC / (NE + NC), (E) the statistical power of exploratory studies WE, and (F) the proportion of published studies that draw incorrect conclusions (PF). Other values: SC = 120, T = 2,000, k = 20, α = 0.05, σ2 = 1, m = 3, and ϕ = 0.8. The chosen values for rC = rE reflect data reported by Richard and colleagues [14], where a correlation coefficient mode of 0.09 and a mean of 0.21 were observed. These values are in the middle of the range of effect sizes observed in meta-analyses across a number of biomedical research domains (range r ~ 0.15 to 0.50) [13].CC BY · PubMed Central
Fig 3 Effect of γ and ϕ on a hypothetical measure of the total scientific value of research (VS). The figure shows the product of the number of published confirmatory studies, the number of published exploratory studies, and the proportion of published studies that are correct (red = high, blue = low). This measure is calculated for when all researchers are following the rational strategy given the values of γ and ϕ. The current emphasis on a small number of publications that report novel findings is characterised by high γ and high ϕ (top right). To improve scientific output according to this measure, we could reduce ϕ (i.e., make more published studies count for researchers’ careers) or reduce γ (i.e., reduce weighting of published exploratory studies). Interestingly, the ridge is flat, so any point along it has equal fitness. Therefore, a pragmatic compromise would be to reduce both γ and ϕ by a lesser amount. The panels show the VT for two values of the dependence of acceptance on sample size m and the Type I error rate α: (A) α = 0.05, m = 3, colour range: 2.0–3.18; (B) α = 0.05, m = 6, colour range: 2.0–2.82; (C) α = 0.03, m = 6, colour range: 2.0–2.065. Other values: SC = 120, T = 2000, k = 20, fE = 0.2, rC = rE = 0.21, and σ2 = 1.
Fig 3 Effect of γ and ϕ on a hypothetical measure of the total scientific value of research (VS). The figure shows the product of the number of published confirmatory studies, the number of published exploratory studies, and the proportion of published studies that are correct (red = high, blue = low). This measure is calculated for when all researchers are following the rational strategy given the values of γ and ϕ. The current emphasis on a small number of publications that report novel findings is characterised by high γ and high ϕ (top right). To improve scientific output according to this measure, we could reduce ϕ (i.e., make more published studies count for researchers’ careers) or reduce γ (i.e., reduce weighting of published exploratory studies). Interestingly, the ridge is flat, so any point along it has equal fitness. Therefore, a pragmatic compromise would be to reduce both γ and ϕ by a lesser amount. The panels show the VT for two values of the dependence of acceptance on sample size m and the Type I error rate α: (A) α = 0.05, m = 3, colour range: 2.0–3.18; (B) α = 0.05, m = 6, colour range: 2.0–2.82; (C) α = 0.03, m = 6, colour range: 2.0–2.065. Other values: SC = 120, T = 2000, k = 20, fE = 0.2, rC = rE = 0.21, and σ2 = 1.CC BY · PubMed Central
Fig 4 Effect of editorial stringency on total scientific output for current incentive structures. The figure shows the proportion of published findings that are correct 1-PF (A, B), the total number of published studies NC + NC (C, D), and the total scientific value of research VT (E, F). We varied the following parameters: the probability of a Type I error α (A, C, E), and the dependence of acceptance on sample size m (B, D, F). The lines show predictions for two values of the probability that an effect is real (fE) and two values of the effect size (solid: fE = 0.2, rC = rE = 0.21; dotted: fE = 0.3, rC = rE = 0.21; and dashed: fE = 0.2, rC = rE = 0.32). Other values: NC = 120, T = 2,000, k = 20, α = 0.05, σ2 = 1, m = 3, ϕ = 0.9, and γ = 0.09. The steps occur where there is discontinuity in the effect of α on SE*.
Fig 4 Effect of editorial stringency on total scientific output for current incentive structures. The figure shows the proportion of published findings that are correct 1-PF (A, B), the total number of published studies NC + NC (C, D), and the total scientific value of research VT (E, F). We varied the following parameters: the probability of a Type I error α (A, C, E), and the dependence of acceptance on sample size m (B, D, F). The lines show predictions for two values of the probability that an effect is real (fE) and two values of the effect size (solid: fE = 0.2, rC = rE = 0.21; dotted: fE = 0.3, rC = rE = 0.21; and dashed: fE = 0.2, rC = rE = 0.32). Other values: NC = 120, T = 2,000, k = 20, α = 0.05, σ2 = 1, m = 3, ϕ = 0.9, and γ = 0.09. The steps occur where there is discontinuity in the effect of α on SE*.CC BY · PubMed Central
Fig 5 Effect of editorial stringency on total scientific output for ideal incentive structures. The figure shows the proportion of published studies that are correct 1 − PT (A, B), the total number of published studies NC + NC (C, D), and the total scientific value of research VS (E, F). We varied the following parameters: the probability of a Type I error α (A, C, E), and the dependence of acceptance on sample size m (B, D, F). The lines show predictions for two values of the probability that an effect is real (fE) and two values of the effect size (solid: fE = 0.2, rC = rE = 0.21; dotted: fE = 0.3, rC = rE = 0.21; dashed: fE = 0.2, rC = rE = 0.32). Other values: NC = 120, T = 2,000, k = 20, α = 0.05, σ2 = 1, m = 3, ϕ = 0.55, and γ = 0.055.
Fig 5 Effect of editorial stringency on total scientific output for ideal incentive structures. The figure shows the proportion of published studies that are correct 1 − PT (A, B), the total number of published studies NC + NC (C, D), and the total scientific value of research VS (E, F). We varied the following parameters: the probability of a Type I error α (A, C, E), and the dependence of acceptance on sample size m (B, D, F). The lines show predictions for two values of the probability that an effect is real (fE) and two values of the effect size (solid: fE = 0.2, rC = rE = 0.21; dotted: fE = 0.3, rC = rE = 0.21; dashed: fE = 0.2, rC = rE = 0.32). Other values: NC = 120, T = 2,000, k = 20, α = 0.05, σ2 = 1, m = 3, ϕ = 0.55, and γ = 0.055.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

The citation lists are queued

They are fetched later and appear here then.

References 25 · Cited by 192

Metadata

Authors
Higginson, A. D., & Munafò, M. R.
Affiliations
University of Exeter (GB)
Corresponding author
Higginson, Andrew D.
Venue
PLOS Biology
Published
2016-11-10
Volume
14
Issue
11
Pages
e2000995
ISSN
  • 1544-9173 (ISSN-L)
  • 1545-7885
OpenAlex
W2549633770
Cited by
192
References
25
Citations by year
2026: 6 · 2025: 12 · 2024: 8 · 2023: 8 · 2022: 28 · 2021: 27 · 2020: 18 · 2019: 27 · 2018: 30 · 2017: 25 · 2016: 3
FWCI
53.73
Type
journal article
Language
en
Open access
Locations
  • PLoS Biology (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Explore Bristol Research (submittedVersion, cc-by)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Figshare (submittedVersion)
  • Figshare (submittedVersion, cc-by)
  • Open Research Exeter (University of Exeter) (submittedVersion, cc-by)
  • Bristol Research (University of Bristol) (submittedVersion, cc-by)
Funders
  • United Kingdom Clinical Research Collaboration
  • Cancer Research UK
  • National Institute for Health and Care Research
  • British Heart Foundation
  • University of Bristol (MC_UU_12013, MC_UU_12013/6)
  • Medical Research Council (MC_UU_12013, MC_UU_12013/6)
  • Economic and Social Research Council
  • Natural Environment Research Council (NE/L011921/1, NE/L011921/2)
Topics
  • Social Sciences › Business, Management and Accounting › Management of Technology and Innovation › Research, Science, and Academia
  • Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
  • Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
Keywords
research incentives, publication incentives, statistical power, underpowered studies, optimality models, research ecosystem
MeSH terms
Workforce, Humans, Motivation, Publishing, Research, Science

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Current Incentives for Scientists Lead to Underpowered Studies with Erroneous Conclusions. https://app.cochat.ai/papers/10.1371/journal.pbio.2000995
The paper
Higginson, A. D., & Munafò, M. R. (2016). Current Incentives for Scientists Lead to Underpowered Studies with Erroneous Conclusions. PLOS Biology. https://doi.org/10.1371/journal.pbio.2000995

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.