Correlation among baseline variables yields non-uniformity of p-values
Open accessThe DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index; none of the 9 sources its claims cite is checked for support yet.
Abstract
A recent paper in Neurology used statistical techniques to investigate the integrity of the randomization in 33 clinical trials conducted by a group of investigators. Without justification, the approach assumed that there would be no impact of correlation among baseline variables. We investigated the impact of correlation on the conclusions of the approach in several large-scale simulation studies that replicated the sample sizes and baseline variables of the clinical trials in question and utilized proper randomization. Additionally, we considered scenarios with larger numbers of baseline variables. We found that, with even moderate correlation, there can be substantial inflation of the type I error of statistical tests of randomization integrity. This is also the case under no correlation, in the presence of some discrete baseline variables, with a large number of variables. Thus, statistical techniques for assessing randomization integrity should be applied with extreme caution given that very low p-values, which are taken as evidence against valid randomization, can arise even in the case of valid randomization, in the presence of correlation. More generally, the use of tests of goodness of fit to uniformity for the purpose of testing a global null hypothesis is not advisable in the presence of correlation.
Overview
CoChat Summary
TL;DR
The paper shows that correlation among baseline variables can substantially inflate type I error when testing uniformity of p-values to assess randomization integrity, even when correlation is consistent across trials. It concludes that such goodness-of-fit tests for a global null are misleading in the presence of correlation and should be used with extreme caution.
Key findings
- Under valid randomization, correlation among baseline variables inflated rejection rates of uniformity tests. For 500 variables with 100 binary variables (Model 2), the chi-square rejection proportion was 0.772 at rho = 0.4 and 0.950 when rho was drawn from Uniform(0.4,0.9); the KS test gave 0.866 and 0.971, respectively.
- With 17 variables and 4 binary variables (Model 2), rejection proportions were 0.153 (chi-square) and 0.175 (KS) at rho = 0.4, rising to 0.356 and 0.387 when rho was Uniform(0.4,0.9). With 60 variables and 15 binary variables (Model 2), the corresponding proportions were 0.367 and 0.419 at rho = 0.4, and 0.672 and 0.708 under Uniform(0.4,0.9).
- Even with no correlation, binary variables alone could inflate type I error: for 500 variables with 100 binary variables, the chi-square rejection proportion was 0.149 and the KS test approximately 0.62. For 60 variables with 15 binary variables, these were 0.078 and 0.089.
- The paper concludes that Bolland et al.’s results are reliable only under zero correlation among the 17 baseline variables, which the author calls “not a plausible scenario.”
Methods
- Simulation study replicating the sample sizes and baseline variable counts from the 33 clinical trials in Bolland et al.: three settings with 500 variables (100 or 0 binary), 17 variables (4 or 0 binary), and 60 variables (14 or 0 binary; the table labels 15 Bernoulli variables).
- Continuous variables were generated from equally correlated multivariate normal distributions (mean 0, variance 1); binary variables were either independent Bernoulli variables (Model 1) or dichotomized normals (Model 2); Model 3 used only continuous variables. Correlation rho was fixed at 0, 0.1, 0.2, 0.3, 0.4, or 0.5, or drawn from Uniform(0.4,0.9), either fixed or randomly generated across repetitions.
- For each of 100,000 repetitions, 31×N p-values were calculated using t-tests for continuous variables and Fisher’s exact tests for binary variables, then tested for uniformity using a multinomial chi-square goodness-of-fit test and a Kolmogorov–Smirnov test; rejection was defined as p < 0.05.
Limitations
- The paper does not explicitly enumerate limitations. It states that no data other than the simulation R code are included, so the work is simulation-based rather than an empirical reanalysis of the original trial data.
- Evident from the text: the simulations assume equally correlated multivariate normal continuous variables and specific binary-variable generation schemes, so results may not generalize to all real baseline variable structures. The findings are also tied to the specific sample sizes and variable counts modeled from Bolland et al. and to the chosen correlation scenarios.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 3 Not valid · 9 Not checked · 22 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For any number of variables (500, 17, 60), this is the case when rho = 0 and when there are no binary variables included (Model 3).”
§ Correlation among baseline variables yields non-uniformity of p-values > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We conclude that such statistical techniques should be applied with extreme caution given that very low p-values, which are taken as evidence against the null of valid randomization, can result even under the null.”
- Also stated in: § Abstract · plain
“very low p-values, which are taken as evidence against valid randomization, can arise even in the case of valid randomization, in the presence of correlation.”
§ Correlation among baseline variables yields non-uniformity of p-values > Discussion
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Abstract · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This is also the case under no correlation, in the presence of some discrete baseline variables, with a large number of variables.”
- Also stated in: § Discussion · plain
“Our simulation study shows that this happens even in the presence of a low level of correlation that is consistent across trials, and even with no correlation in the presence of binary variables.”
§ Correlation among baseline variables yields non-uniformity of p-values > Abstract
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Discussion · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, Table 1 indicates that in the setting of 500 variables per experiment and under the null of no group differences, when the correlation is 0.1 for each clinical trial, uniformity of p-values is rejected with probability greater than 0.25.”
§ Correlation among baseline variables yields non-uniformity of p-values > Discussion
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When the correlation varies across experiments and ranges from 0.4 to 0.9, this rejection probability is greater than 0.95.”
§ Correlation among baseline variables yields non-uniformity of p-values > Discussion
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
14 more claims
Create a free account to see all 34 claims and to verify the 9 references still open.
- It may state two separate claims as one. 9
- Probably not a substantial claim. 9
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The distribution of baseline p-values from simulations of variables with highly positively skewed lognormal distributions, and simulations in which all variables had fixed or high levels of correlation was not uniform [8, 9].”
From Rounding, but not randomization method, non-normality, or correlation, affected baseline p-value distributions in randomized trials. (2019) · cites this paper as [9]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Correlation among baseline variables yields non-uniformity of p-values > Results
“The rejection proportions for the KS test for Model 3 are lower than those for Models 1 and 2 due to its consideration of continuous variables only.”
§ Correlation among baseline variables yields non-uniformity of p-values > Results
“| 0.5 | 0.175 | 0.199 | 0.211 | 0.238 | 0.241 | 0.278 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Correlation among baseline variables yields non-uniformity of p-values > Materials and methods
“in the third we included 60 baseline variables with either 14 or zero binary variables.”
§ Correlation among baseline variables yields non-uniformity of p-values > Results
“(N,B) = (500,100), (17,4), and (60,15), respectively.”
Citations
Cites
Systematic review and statistical analysis of the integrity of 33 randomized controlled trials
Bolland, M. J., Avenell, A., Gamble, G. D., & Grey, A. (2016) · Neurology · doi:10.1212/wnl.0000000000003387 · Cited by 106 · cited 10×
“[1]: their assessment of the uniformity of the p-values comparing baseline characteristics between randomized treatment groups.”
BackgroundMethodInfluentialWeber, A., Hochmann, S., Cimalla, P., Gaertner, M., Kuscha, V., Hans, S., Geffarth, M., Kaslin, J., Koch, E., & Brand, M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0080483 · Cited by 65
Do baseline p-values follow a uniform distribution in randomised trials?
Bland, M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0076010 · Cited by 47 · cited 6×
“These latter two points were studied by Bland [3], who showed that even under the null of equality of randomized groups, Table 3.”
BackgroundMethodResultInfluentialStatistics and the detection of scientific misconduct
Gross, R. A., & Editor-in-Chief (2016) · Neurology · doi:10.1212/wnl.0000000000003390 · Cited by 18
Cited by
Data integrity of 35 randomised controlled trials in women' health
Bordewijk, E. M., Wang, R., Askie, L. M., Gurrin, L. C., Thornton, J. G., van Wely, M., Li, W., & Mol, B. W. (2020) · European Journal of Obstetrics & Gynecology and Reproductive Biology · doi:10.1016/j.ejogrb.2020.04.016 · Cited by 66
Methods to assess research misconduct in health-related research: A scoping review
Bordewijk, E. M., Li, W., van Eekelen, R., Wang, R., Showell, M., Mol, B. W., & van Wely, M. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.05.012 · Cited by 68
Integrity of randomized controlled trials: challenges and solutions
Li, W., van Wely, M., Gurrin, L., & Mol, B. W. (2020) · Fertility and Sterility · doi:10.1016/j.fertnstert.2020.04.018 · Cited by 49
Bolland, M. J., Gamble, G. D., Avenell, A., & Grey, A. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.03.001 · Cited by 31
Bordewijk, E. M., Wang, R., van Wely, M., Costello, M. F., Norman, R. J., Teede, H., Gurrin, L. C., Mol, B. W., & Li, W. (2020) · Human Reproduction Update · doi:10.1093/humupd/dmaa031 · Cited by 18
Metadata
- Authors
- Betensky, R. A., & Chiou, S. H.
- Affiliations
- Harvard University (US)
- Corresponding author
- Betensky, Rebecca A.
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2017-09-08
- Volume
- 12
- Issue
- 9
- Pages
- e0184531
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC5590981
- OpenAlex
- W2753394018
- Semantic Scholar
- 84a316f74433a16da0a4804eb01b87a8820b6cb6
- Cited by
- 8
- References
- 4
- Citations by year
- 2021: 1 · 2020: 5 · 2019: 1 · 2018: 1
- FWCI
- 1.33
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- National Institute of Neurological Disorders and Stroke (NS094610, NS048005)
- National Center for Advancing Translational Sciences (TR001102)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Physical Sciences › Mathematics › Statistics and Probability › Benford’s Law and Fraud Detection
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- type I error, goodness-of-fit tests, uniformity testing, clinical trials, simulation study
- MeSH terms
- Clinical Trials as Topic, Computer Simulation, Data Interpretation, Statistical, Humans, Research Design, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Correlation among baseline variables yields non-uniformity of p-values. https://app.cochat.ai/papers/10.1371/journal.pone.0184531
- The paper
- Betensky, R. A., & Chiou, S. H. (2017). Correlation among baseline variables yields non-uniformity of p-values. PLOS ONE. https://doi.org/10.1371/journal.pone.0184531
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.