Randomization Does Not Help Much, Comparability Does
Open accessThe DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
According to R.A. Fisher, randomization "relieves the experimenter from the anxiety of considering innumerable causes by which the data may be disturbed."Since, in particular, it is said to control for known and unknown nuisance factors that may considerably challenge the validity of a result, it has become very popular.This contribution challenges the received view.First, looking for quantitative support, we study a number of straightforward, mathematically simple models.They all demonstrate that the optimism surrounding randomization is questionable: In small to medium-sized samples, random allocation of units to treatments typically yields a considerable imbalance between the groups, i.e., confounding due to randomization is the rule rather than the exception.In the second part of this contribution, the reasoning is extended to a number of traditional arguments in favour of randomization.This discussion is rather non-technical, and sometimes touches on the rather fundamental Frequentist/Bayesian debate.However, the result of this analysis turns out to be quite similar: While the contribution of randomization remains doubtful, comparability contributes much to a compelling conclusion.Summing up, classical experimentation based on sound background theory and the systematic construction of exchangeable groups seems to be advisable. The logic of the experimentRandomization, the allocation of subjects to experimental conditions via a random procedure, was introduced by eminent statistician R.A. Fisher [1].Arguably, it has since become the most important statistical technique.In particular, statistical experiments are defined by the use of randomization [2,3], and many applied fields, such as evidence based medicine, draw a basic distinction between randomized and non-randomized evidence.In order to explain randomization's eminent role, one may refer to the logic of the experiment, largely based on J. S. Mill's method of difference[4]: If one compares two groups of subjects (Treatment T versus Control C, say) and observes a salient contrast in the end (e.g.X À T > X À C ), that difference must be due to the experimental manipulation-IF the groups were equivalent at the very beginning of the experiment.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Probability Theory: The Logic of Science
Jaynes, E. T. (2003) · doi:10.1017/cbo9780511790423 · Cited by 4,045
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979
The interpretation of interaction in contingency tables
Simpson, E. H. (1951) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1951.tb00088.x · Cited by 2,301 · cited 1×
“tment is unconfounded if the treated and untreated groups resemble each other in all relevant features” (Pearl 2009: 196). Pearl shows that this problem is tightly linked to Simpson’s famous paradox (Simpson 1951) and devotes a major part of his book to the development of a formal calculus that is able to cope with it…”
BackgroundBelieve it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888
Contradicted and Initially Stronger Effects in Highly Cited Clinical Research
Ioannidis, J. P. A. (2005) · JAMA · doi:10.1001/jama.294.2.218 · Cited by 1,465
Student (1908) · Biometrika · doi:10.1093/biomet/6.1.1 · Cited by 1,363
Pashler, H., & Wagenmakers, E. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612465253 · Cited by 1,321
Counterfactuals and Causal Inference: Methods and Principles for Social Research
Morgan, S. L., & Winship, C. (2007) · doi:10.1017/cbo9780511804564 · Cited by 1,146
Confounding and Collapsibility in Causal Inference
Greenland, S., Pearl, J., & Robins, J. M. (1999) · Statistical Science · doi:10.1214/ss/1009211805 · Cited by 933 · cited 1×
“es between the groups in one or more of the prognostic factors can occur by chance. [...] At this point the primary objective of randomisation — exclusion of confounding factors — has failed.” 9 (iv) Greenland et al. (1999: 35): [...] random imbalances may be severe, especially if the study size is small [...] (v) Ros…”
BackgroundIn All Likelihood: Statistical Modelling and Inference Using Likelihood
Pawitan, Y. (2001) · doi:10.1093/oso/9780198507659.001.0001 · Cited by 874
Cited by
Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0
Percie du Sert, N., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., Emerson, M., et al. (2020) · PLOS Biology · doi:10.1371/journal.pbio.3000411 · Cited by 2,677
Reporting animal research: Explanation and Elaboration for the ARRIVE guidelines 2019
Percie du Sert, N., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., Emerson, M., et al. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/703355 · Cited by 559
Single-subject research in psychiatry: facts and fictions
Zuidersma, M., Riese, H., Snippe, E., Booij, S. H., Wichers, M., & Bos, E. H. (2020) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2020.539777 · Cited by 54
“This latter assumed similarity is a stronger assumption, and generally not true (69) than the assumption that an individual is similar to him- or herself somewhat earlier in time (66).”
BackgroundCardiovascular effects of brief mindfulness meditation among perfectionists experiencing failure
Koerten, H. R., Watford, T. S., Dubow, E. F., & O’Brien, W. H. (2020) · Psychophysiology · doi:10.1111/psyp.13517 · Cited by 18
Sponchiado-Júnior, E. C., Vieira, W. D. A., Frozoni, M., Herkrath, F. J., & de-Jesus-Soares, A. (2021) · Journal of Endodontics · doi:10.1016/j.joen.2021.07.017 · Cited by 11
Metadata
- Authors
- Saint-Mont, U.
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2015-07-20
- Volume
- 10
- Issue
- 7
- Pages
- e0132102
- ISSN
- 1932-6203 (ISSN-L)
- OpenAlex
- W1912205883
- Cited by
- 55
- References
- 14
- FWCI
- 2.92
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Randomization Does Not Help Much, Comparability Does. https://app.cochat.ai/papers/10.1371/journal.pone.0132102
- The paper
- Saint-Mont, U. (2015). Randomization Does Not Help Much, Comparability Does. PLOS ONE. https://doi.org/10.1371/journal.pone.0132102
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.