CoChat
Sign Up
Cited by 220Meta-analysis and systematic reviewsPsychology Research and Bibliometricsscientometrics and bibliometrics research

The DOI matches its Crossref record; 54 of the 54 references it cites were found in a registry or an index.

Conducting Meta-Analyses Based on p Values: Reservations and Recommendations for Applying p -Uniform and p -Curve

Open access
van Aert, Robbie C. M., Wicherts, Jelte M., van Assen, Marcel A. L. M.·2016·Perspectives on Psychological Science
DOI 10.1177/1745691616650874PMC PMC5117126
Open this paper in CoChat — free accountOpen PDF

Abstract

Because of overwhelming evidence of publication bias in psychology, techniques to correct meta-analytic estimates for such bias are greatly needed. The methodology on which the p-uniform and p-curve methods are based has great promise for providing accurate meta-analytic estimates in the presence of publication bias. However, in this article, we show that in some situations, p-curve behaves erratically, whereas p-uniform may yield implausible estimates of negative effect size. Moreover, we show that (and explain why) p-curve and p-uniform result in overestimation of effect size under moderate-to-large heterogeneity and may yield unpredictable bias when researchers employ p-hacking. We offer hands-on recommendations on applying and interpreting results of meta-analyses in general and p-uniform and p-curve in particular. Both methods as well as traditional methods are applied to a meta-analysis on the effect of weight on judgments of importance. We offer guidance for applying p-uniform or p-curve using R and a user-friendly web application for applying p-uniform.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1177/1745691616650874

Cites

54 of 54 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1. Probability-probability (P-P) plot for a meta-analysis of 20 studies with large heterogeneity.
Fig. 1. Probability-probability (P-P) plot for a meta-analysis of 20 studies with large heterogeneity.CC BY · PubMed Central
Fig. 2. Values for Kolmogorov-Smirnov (KS) test statistics in implementation of p-curve and p-uniform for the example with three observed effect sizes and p values close to .05. D stat = test statistics of KS test.
Fig. 2. Values for Kolmogorov-Smirnov (KS) test statistics in implementation of p-curve and p-uniform for the example with three observed effect sizes and p values close to .05. D stat = test statistics of KS test.CC BY · PubMed Central
Fig. 3. Effect size estimates in p-uniform and fixed-effect meta-analysis in case of four types of p-hacking. FE = fixed-effect; DV = dependent variable.
Fig. 3. Effect size estimates in p-uniform and fixed-effect meta-analysis in case of four types of p-hacking. FE = fixed-effect; DV = dependent variable.CC BY · PubMed Central
Fig. A1. Computation of conditional p values for Effect 3 (q3) for three effect sizes: δ = 0; δ = 0.5 (true effect size); and δ = 0.748 (estimate of fixed-effect meta-analysis). Critical value on Cohen’s d scale is denoted by dcv and observed effect size is denoted by dobs.
Fig. A1. Computation of conditional p values for Effect 3 (q3) for three effect sizes: δ = 0; δ = 0.5 (true effect size); and δ = 0.748 (estimate of fixed-effect meta-analysis). Critical value on Cohen’s d scale is denoted by dcv and observed effect size is denoted by dobs.CC BY · PubMed Central
Fig. A2. Observed conditional p values (solid black lines) and conditional p values under uniformity (dashed gray lines) for the example with three observed effect sizes. The three panels refer to the conditional p values for the p-uniform hypothesis test of no effect (δ = 0), p-uniform effect size estimate (δ = 0.5), and effect size obtained by fixed-effect (FE) meta-analysis (δ = 0.748).
Fig. A2. Observed conditional p values (solid black lines) and conditional p values under uniformity (dashed gray lines) for the example with three observed effect sizes. The three panels refer to the conditional p values for the p-uniform hypothesis test of no effect (δ = 0), p-uniform effect size estimate (δ = 0.5), and effect size obtained by fixed-effect (FE) meta-analysis (δ = 0.748).CC BY · PubMed Central
Fig. A3. Conditional p values as a function of true effect size (x axis) for each of the three observed effect sizes in the example. Effect sizes zero (δ = 0), true effect size (δ = 0.5), and estimated by fixed-effect meta-analysis (δ = 0.748) are indicated by vertical lines.
Fig. A3. Conditional p values as a function of true effect size (x axis) for each of the three observed effect sizes in the example. Effect sizes zero (δ = 0), true effect size (δ = 0.5), and estimated by fixed-effect meta-analysis (δ = 0.748) are indicated by vertical lines.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 54 references

Research works printed but not linked: 4

  1. American Psychological Association, 2010, Publication manual
  2. Augusteijn H. E. M. (2015). The effect of publication bias on the Q test and assessment of heterogeneity (Unpublished master’s thesis). Tilburg University, Tilburg, the Netherlands.
  3. Raudenbush S. W., 2009, The handbook of research synthesis and meta-analysis, 295
  4. van Aert R. C. M., 2016, Estimating effect size in meta-analysis in the presence of publication bias and questionable research practices

Cited by

Sorted by citations

Related

See all

Metadata

Authors
van Aert, R. C. M., Wicherts, J. M., & van Assen, M. A. L. M.
Affiliations
  • Tilburg University (NL)
  • Utrecht University (NL)
Corresponding author
van Aert, Robbie C. M.
ORCID
Venue
Perspectives on Psychological Science
Publisher
SAGE Publications
Published
2016-09
Volume
11
Issue
5
Pages
713-729
ISSN
  • 1745-6916 (ISSN-L)
  • 1745-6924
DOI
10.1177/1745691616650874
PMC
PMC5117126
OpenAlex
W2526603989
Cited by
219
References
54
Citations by year
2026: 12 · 2025: 13 · 2024: 24 · 2023: 22 · 2022: 28 · 2021: 22 · 2020: 25 · 2019: 23 · 2018: 20 · 2017: 27 · 2016: 3
FWCI
22.73
Type
journal article
Language
en
Open access
Licence
Locations
  • Perspectives on Psychological Science (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • Research portal (Tilburg University) (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
  • Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
  • OSF Preprints (OSF Preprints) (submittedVersion)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
  • Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
Keywords
p-curve, publication bias, meta-analysis, effect size estimation, p-hacking, heterogeneity, effect size overestimation, R package, web application
MeSH terms
Data Interpretation, Statistical, Humans, Psychology, Software, Meta-Analysis as Topic, Models, Statistical, Publication Bias, Internet

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review