The DOI matches its Crossref record; 6 of the 6 references it cites were found in a registry or an index.
Repeated Significance Tests on Accumulating Data
Abstract
THE general effect of performing repeated significance tests at different stages during the accumulation of a body of data is well known. If the null hypothesis is true and if each significance test is performed at the same nominal level, the probability that at some stage or another the test criterion is may be substantially greater than the nominal value. Feller (1940) discussed the possibility that some of the more results in card-guessing experiments in extra-sensory perception might be attributed to at particularly favourable stages during an investigation. The law of the iterated logarithm shows that a test criterion which takes the form of a standardized cumulative sum of deviations from expectation divided by its standard error will, with probability one, eventually reach any preassigned value. Thus, in many common situations a result as highly as one desires can be obtained by sufficiently extensive sampling. Robbins (1952) and Anscombe (1954) provide further discussion of this point. The desire to control the error of the first kind, as well as the power of a test procedure, was of course one of the motivations of sequential analysis (Wald, 1947). More recently the practical relevance of this phenomenon has been called into question. Anscombe (1954) had pointed out that inferences based on likelihoods or, through likelihoods, on posterior probabilities were unaffected by stopping rules. The contrast between this property and the extreme sensitivity of frequency-type inferences to the stopping rule explains why sequential analysis is a topic of such contention between adherents of different viewpoints (Birnbaum, 1964; Cornfield, 1966; Armitage, 1967). The exchanges of opinion on these matters have been remarkable for the lack of quantitative information about the optional stopping effect. It has not, for example, been possible to answer questions such as the following. (a) What is the probability of obtaining a result significant at a certain nominal level, within the first 50 tests? (b) Does the enhancement of the probability of obtaining a result reach a noticeably high level only after a very large number of tests? (c) What is the effect of repeated tests when the null hypothesis is not true? The purpose of the present paper is to repair gome of these gaps in our knowledge without indulging in further discussion of inferential problems. We consider sequential observations of three different distributional forms: binomial, normal and exponential. In the binomial case exact results are obtained by direct calculation of
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Some aspects of the sequential design of experiments
Robbins, H. (1952) · Bulletin of the American Mathematical Society · doi:10.1090/s0002-9904-1952-09620-8 · Cited by 2,184
Flehinger, B. J. (1976) · Technometrics · doi:10.1080/00401706.1976.10489459 · Cited by 494
Hull, T. E., & Dobell, A. R. (1962) · SIAM Review · doi:10.1137/1004061 · Cited by 333
Uniform Random Number Generators
MacLaren, M. D., & Marsaglia, G. (1965) · Journal of the ACM · doi:10.1145/321250.321257 · Cited by 242
A Bayesian Test of Some Classical Hypotheses, with Applications to Sequential Clinical Trials
Cornfield, J. (1966) · Journal of the American Statistical Association · doi:10.1080/01621459.1966.10480890 · Cited by 130
Fixed-Sample-Size Analysis of Sequential Observations
Anscombe, F. J. (1954) · Biometrics · doi:10.2307/3001665 · Cited by 78
Cited by
A practical solution to the pervasive problems ofp values
Wagenmakers, E.-J. (2007) · Psychonomic Bulletin & Review · doi:10.3758/bf03194105 · Cited by 2,490
Discrete sequential boundaries for clinical trials
GORDON LAN, K. K., & DEMETS, D. L. (1983) · Biometrika · doi:10.1093/biomet/70.3.659 · Cited by 2,310
Using Bayes to get the most out of non-significant results
Dienes, Z. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00781 · Cited by 1,998
The preregistration revolution
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708274114 · Cited by 1,998
Group sequential methods in the design and analysis of clinical trials
POCOCK, S. J. (1977) · Biometrika · doi:10.1093/biomet/64.2.191 · Cited by 1,831
Bayesian Approaches to Clinical Trials and Health‐Care Evaluation
Spiegelhalter, D. J., Abrams, K. R., & Myles, J. P. (2003) · Wiley eBooks · doi:10.1002/0470092602 · Cited by 1,673
Delmas, P. D., Bjarnason, N. H., Mitlak, B. H., Ravoux, A.-C., Shah, A. S., Huster, W. J., Draper, M., & Christiansen, C. (1997) · New England Journal of Medicine · doi:10.1056/nejm199712043372301 · Cited by 1,637
Trial Sequential Analysis in systematic reviews with meta-analysis
Wetterslev, J., Jakobsen, J. C., & Gluud, C. (2017) · BMC Medical Research Methodology · doi:10.1186/s12874-017-0315-7 · Cited by 1,189
Iyengar, P., Wardak, Z., Gerber, D. E., Tumati, V., Ahn, C., Hughes, R. S., Dowell, J. E., Cheedella, N., Nedzi, L., Westover, K. D., et al. (2018) · JAMA Oncology · doi:10.1001/jamaoncol.2017.3501 · Cited by 1,047
One-Sample Multiple Testing Procedure for Phase II Clinical Trials
Fleming, T. R. (1982) · Biometrics · doi:10.2307/2530297 · Cited by 1,003
Bayesian Versus Orthodox Statistics: Which Side Are You On?
Dienes, Z. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691611406920 · Cited by 938
Publication bias—a problem in interpreting medical data
Begg, C. B., & Berlin, J. A. (1988) · Journal of the Royal Statistical Society. Series A (Statistics in Society) · doi:10.2307/2982993 · Cited by 814
Bayes factor design analysis: Planning for compelling evidence
Schönbrodt, F. D., & Wagenmakers, E.-J. (2018) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1230-y · Cited by 729
Bayesian approaches to randomized trials
Spiegelhalter, D. J., Freedman, L. S., & Parmar, M. K. B. (1994) · Journal of the Royal Statistical Society. Series A (Statistics in Society) · doi:10.2307/2983527 · Cited by 720
The Coronary Drug Project Research Group (1980) · New England Journal of Medicine · doi:10.1056/nejm198010303031804 · Cited by 659
KEMENY, N., DALY, J., REICHMAN, B., GELLER, N., BOTET, J., & ODERMAN, P. (1987) · Annals of Internal Medicine · doi:10.7326/0003-4819-107-4-459 · Cited by 649
Sequential hypothesis testing with Bayes factors: Efficiently testing mean differences.
Schönbrodt, F. D., Wagenmakers, E.-J., Zehetleitner, M., & Perugini, M. (2017) · Psychological Methods · doi:10.1037/met0000061 · Cited by 567
Interim analysis: The alpha spending function approach
Demets, D. L., & Lan, K. K. G. (1994) · Statistics in Medicine · doi:10.1002/sim.4780131308 · Cited by 564
Repeated assessment of results in clinical trials of cancer treatment
Haybittle, J. L. (1971) · The British Journal of Radiology · doi:10.1259/0007-1285-44-526-793 · Cited by 517
Some Thoughts on Clinical Trials, Especially Problems of Multiplicity
Tukey, J. W. (1977) · Science · doi:10.1126/science.333584 · Cited by 479
Detecting and avoiding likely false‐positive findings – a practical guide
Forstmeier, W., Wagenmakers, E., & Parker, T. H. (2017) · Biological Reviews · doi:10.1111/brv.12315 · Cited by 436
Performing high‐powered studies efficiently with sequential analyses
Lakens, D. (2014) · European Journal of Social Psychology · doi:10.1002/ejsp.2023 · Cited by 414
Cohn, J. N., Pfeffer, M. A., Rouleau, J., Sharpe, N., Swedberg, K., Straub, M., Wiltse, C., Wright, T. J., & for the MOXCON Investigators (2003) · European Journal of Heart Failure · doi:10.1016/s1388-9842(03)00163-6 · Cited by 384
Müller, H.-H., & Schäfer, H. (2001) · Biometrics · doi:10.1111/j.0006-341x.2001.00886.x · Cited by 372
Adaptive increase in sample size when interim results are promising: a practical guide with examples
Mehta, C. R., & Pocock, S. J. (2011) · Statistics in Medicine · doi:10.1002/sim.4102 · Cited by 367
Combining mortality and longitudinal measures in clinical trials
Finkelstein, D. M., & Schoenfeld, D. A. (1999) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19990615)18:11<1341::aid-sim129>3.0.co;2-7 · Cited by 363
Bayesian methods in health technology assessment: a review.
Spiegelhalter, Myles, Jones, & Abrams (2000) · Health Technology Assessment · doi:10.3310/hta4380 · Cited by 358
Streiner, D. L. (2015) · The American Journal of Clinical Nutrition · doi:10.3945/ajcn.115.113548 · Cited by 343
Approximately Optimal One-Parameter Boundaries for Group Sequential Trials
Wang, S. K., & Tsiatis, A. A. (1987) · Biometrics · doi:10.2307/2531959 · Cited by 341
Validation of Software for Bayesian Models Using Posterior Quantiles
Cook, S. R., Gelman, A., & Rubin, D. B. (2006) · Journal of Computational and Graphical Statistics · doi:10.1198/106186006x136976 · Cited by 319
Myocardial Viability and Long-Term Outcomes in Ischemic Cardiomyopathy
Panza, J. A., Ellis, A. M., Al-Khalidi, H. R., Holly, T. A., Berman, D. S., Oh, J. K., Pohost, G. M., Sopko, G., Chrzanowski, L., Mark, D. B., et al. (2019) · New England Journal of Medicine · doi:10.1056/nejmoa1807365 · Cited by 302
Design and analysis of group sequential tests based on the type I error spending rate function
KIM, K., & DEMETS, D. L. (1987) · Biometrika · doi:10.1093/biomet/74.1.149 · Cited by 295
Marsman, M., & Wagenmakers, E.-J. (2017) · European Journal of Developmental Psychology · doi:10.1080/17405629.2016.1259614 · Cited by 267
Interim Analyses for Randomized Clinical Trials: The Group Sequential Approach
Pocock, S. J. (1982) · Biometrics · doi:10.2307/2530298 · Cited by 264
Multiplicity Considerations in the Design and Analysis of Clinical Trials
Cook, R. J., & Farewell, V. T. (1996) · Journal of the Royal Statistical Society. Series A (Statistics in Society) · doi:10.2307/2983471 · Cited by 255
Time-uniform, nonparametric, nonasymptotic confidence sequences
Howard, S. R., Ramdas, A., McAuliffe, J., & Sekhon, J. (2021) · The Annals of Statistics · doi:10.1214/20-aos1991 · Cited by 190
Calibrating the Scientific Ecosystem Through Meta-Research
Hardwicke, T. E., Serghiou, S., Janiaud, P., Danchev, V., Crüwell, S., Goodman, S. N., & Ioannidis, J. P. A. (2020) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-031219-041104 · Cited by 98
The alpha spending function approach to interim data analyses
DeMets, D. L., & Lan, G. (1995) · Cancer Treatment and Research · doi:10.1007/978-1-4615-2009-2_1 · Cited by 58
Revisiting non-significant effects of intranasal oxytocin using equivalence testing
Quintana, D. S. (2018) · Psychoneuroendocrinology · doi:10.1016/j.psyneuen.2017.10.010 · Cited by 38
Independent data monitoring committees: rationale, operations and controversies
Ellenberg, S. S. (2001) · Statistics in Medicine · doi:10.1002/sim.730 · Cited by 28
Floriani, I., Rotmensz, N., Albertazzi, E., Torri, V., De Rosa, M., Tomino, C., & de Braud, F. (2008) · Trials · doi:10.1186/1745-6215-9-46 · Cited by 8
“On the other hand, conducting an interim analysis may also have drawbacks, since immature results on small numbers of patients will provide imprecise or even biased point and interval estimates of the treatment effect, increasing the error in inferential process [3]: when a clinical trial is closed because a treatment…”
Background
Metadata
- Authors
- Armitage, P., McPherson, C. K., & Rowe, B. C.
- Venue
- Journal of the Royal Statistical Society. Series A (General)
- Publisher
- JSTOR
- Published
- 1969
- Volume
- 132
- Issue
- 2
- Pages
- 235
- ISSN
- 0035-9238 (ISSN-L)
- OpenAlex
- W1480884888
- Cited by
- 761
- References
- 6
- FWCI
- 2.06
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.