CoChat

Correlation among baseline variables yields non-uniformity of p-values

Open access
Betensky, Rebecca A., Chiou, Sy Han · 2017 · PLOS ONE

The DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index; none of the 9 sources its claims cite is checked for support yet.

Abstract

A recent paper in Neurology used statistical techniques to investigate the integrity of the randomization in 33 clinical trials conducted by a group of investigators. Without justification, the approach assumed that there would be no impact of correlation among baseline variables. We investigated the impact of correlation on the conclusions of the approach in several large-scale simulation studies that replicated the sample sizes and baseline variables of the clinical trials in question and utilized proper randomization. Additionally, we considered scenarios with larger numbers of baseline variables. We found that, with even moderate correlation, there can be substantial inflation of the type I error of statistical tests of randomization integrity. This is also the case under no correlation, in the presence of some discrete baseline variables, with a large number of variables. Thus, statistical techniques for assessing randomization integrity should be applied with extreme caution given that very low p-values, which are taken as evidence against valid randomization, can arise even in the case of valid randomization, in the presence of correlation. More generally, the use of tests of goodness of fit to uniformity for the purpose of testing a global null hypothesis is not advisable in the presence of correlation.

Overview

CoChat Summary

TL;DR

The paper shows that correlation among baseline variables can substantially inflate type I error when testing uniformity of p-values to assess randomization integrity, even when correlation is consistent across trials. It concludes that such goodness-of-fit tests for a global null are misleading in the presence of correlation and should be used with extreme caution.

Key findings

  • Under valid randomization, correlation among baseline variables inflated rejection rates of uniformity tests. For 500 variables with 100 binary variables (Model 2), the chi-square rejection proportion was 0.772 at rho = 0.4 and 0.950 when rho was drawn from Uniform(0.4,0.9); the KS test gave 0.866 and 0.971, respectively.
  • With 17 variables and 4 binary variables (Model 2), rejection proportions were 0.153 (chi-square) and 0.175 (KS) at rho = 0.4, rising to 0.356 and 0.387 when rho was Uniform(0.4,0.9). With 60 variables and 15 binary variables (Model 2), the corresponding proportions were 0.367 and 0.419 at rho = 0.4, and 0.672 and 0.708 under Uniform(0.4,0.9).
  • Even with no correlation, binary variables alone could inflate type I error: for 500 variables with 100 binary variables, the chi-square rejection proportion was 0.149 and the KS test approximately 0.62. For 60 variables with 15 binary variables, these were 0.078 and 0.089.
  • The paper concludes that Bolland et al.’s results are reliable only under zero correlation among the 17 baseline variables, which the author calls “not a plausible scenario.”

Methods

  • Simulation study replicating the sample sizes and baseline variable counts from the 33 clinical trials in Bolland et al.: three settings with 500 variables (100 or 0 binary), 17 variables (4 or 0 binary), and 60 variables (14 or 0 binary; the table labels 15 Bernoulli variables).
  • Continuous variables were generated from equally correlated multivariate normal distributions (mean 0, variance 1); binary variables were either independent Bernoulli variables (Model 1) or dichotomized normals (Model 2); Model 3 used only continuous variables. Correlation rho was fixed at 0, 0.1, 0.2, 0.3, 0.4, or 0.5, or drawn from Uniform(0.4,0.9), either fixed or randomly generated across repetitions.
  • For each of 100,000 repetitions, 31×N p-values were calculated using t-tests for continuous variables and Fisher’s exact tests for binary variables, then tested for uniformity using a multinomial chi-square goodness-of-fit test and a Kolmogorov–Smirnov test; rejection was defined as p < 0.05.

Limitations

  • The paper does not explicitly enumerate limitations. It states that no data other than the simulation R code are included, so the work is simulation-based rather than an empirical reanalysis of the original trial data.
  • Evident from the text: the simulations assume equally correlated multivariate normal continuous variables and specific binary-variable generation schemes, so results may not generalize to all real baseline variable structures. The findings are also tied to the specific sample sizes and variable counts modeled from Bolland et al. and to the chosen correlation scenarios.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1371/journal.pone.0184531

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

0 Valid · 3 Not valid · 9 Not checked · 22 Own results and interpretations

14 more claims

Create a free account to see all 34 claims and to verify the 9 references still open.

See all claims — free account
  • It may state two separate claims as one. 9
  • Probably not a substantial claim. 9

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 4 references

Cited by

See all 8

Metadata

Authors
Betensky, R. A., & Chiou, S. H.
Affiliations
Harvard University (US)
Corresponding author
Betensky, Rebecca A.
Venue
PLOS ONE
Published
2017-09-08
Volume
12
Issue
9
Pages
e0184531
ISSN
1932-6203 (ISSN-L)
OpenAlex
W2753394018
Cited by
8
References
4
Citations by year
2021: 1 · 2020: 5 · 2019: 1 · 2018: 1
FWCI
1.33
Type
journal article
Language
en
Open access
Locations
  • PLoS ONE (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Funders
  • National Institute of Neurological Disorders and Stroke (NS094610, NS048005)
  • National Center for Advancing Translational Sciences (TR001102)
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
  • Physical Sciences › Mathematics › Statistics and Probability › Benford’s Law and Fraud Detection
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
Keywords
type I error, goodness-of-fit tests, uniformity testing, clinical trials, simulation study
MeSH terms
Clinical Trials as Topic, Computer Simulation, Data Interpretation, Statistical, Humans, Research Design, Models, Statistical

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Correlation among baseline variables yields non-uniformity of p-values. https://app.cochat.ai/papers/10.1371/journal.pone.0184531
The paper
Betensky, R. A., & Chiou, S. H. (2017). Correlation among baseline variables yields non-uniformity of p-values. PLOS ONE. https://doi.org/10.1371/journal.pone.0184531

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.