Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Cited by 203

The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.

The thresholds for statistical and clinical significance – a five-step procedure for evaluation of intervention effects in randomised clinical trials

Open access
Jakobsen, Janus Christian, Gluud, Christian, Winkel, Per, Lange, Theis, Wetterslev, Jørn · 2014 · BMC Medical Research Methodology

Abstract

BACKGROUND: Thresholds for statistical significance are insufficiently demonstrated by 95% confidence intervals or P-values when assessing results from randomised clinical trials. First, a P-value only shows the probability of getting a result assuming that the null hypothesis is true and does not reflect the probability of getting a result assuming an alternative hypothesis to the null hypothesis is true. Second, a confidence interval or a P-value showing significance may be caused by multiplicity. Third, statistical significance does not necessarily result in clinical significance. Therefore, assessment of intervention effects in randomised clinical trials deserves more rigour in order to become more valid. METHODS: Several methodologies for assessing the statistical and clinical significance of intervention effects in randomised clinical trials were considered. Balancing simplicity and comprehensiveness, a simple five-step procedure was developed. RESULTS: For a more valid assessment of results from a randomised clinical trial we propose the following five-steps: (1) report the confidence intervals and the exact P-values; (2) report Bayes factor for the primary outcome, being the ratio of the probability that a given trial result is compatible with a 'null' effect (corresponding to the P-value) divided by the probability that the trial result is compatible with the intervention effect hypothesised in the sample size calculation; (3) adjust the confidence intervals and the statistical significance threshold if the trial is stopped early or if interim analyses have been conducted; (4) adjust the confidence intervals and the P-values for multiplicity due to number of outcome comparisons; and (5) assess clinical significance of the trial results. CONCLUSIONS: If the proposed five-step procedure is followed, this may increase the validity of assessments of intervention effects in randomised clinical trials.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/1471-2288-14-34

Cites

21 of 21 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 21 references

Research works printed but not linked: 68

  1. Jakobsen JC, Gluud C: The necessity of randomized clinical trials. Br J Med Res. 2013, 3 (4): 1453-1468. · DOI
  2. Fisher R: Statistical methods and scientific induction. J R Stat Soc Ser B. 1955, 17 (1): 69-78. · DOI · Paper page
  3. Hald A: A history of mathematical statistics from 1750 to 1930. 1998, New York: John Wiley & Sons
  4. Goodman S: A dirty dozen: twelve p-value misconceptions. Semin Hematol. 2008, 45: 135-140. 10.1053/j.seminhematol.2008.04.003. · DOI · Paper page
  5. Oliveri RS, Gluud C, Wille-Jørgensen PA: Hospital doctors' self-rated skills in and use of evidence-based medicine - a questionnaire survey. J Eval Clin Pract. 2004, 10 (2): 219-226. 10.1111/j.1365-2753.2003.00477.x. · DOI
  6. Bassler D, Briel M, Montori VM, Lane M, Glasziou P, Zhou Q, Heels-Ansdell D, Walter SD, Guyatt GH, Flynn DN, Elamin MB, Murad MH, Abu Elnour NO, Lampropulos JF, Sood A, Mullan RJ, Erwin PJ, Bankhead CR, Perera R, Ruiz Culebro C, You JJ, Mulla SM, Kaur J, Nerenberg KA, Schunemann H, Cook DJ, Lutz K, Ribic CM, Vale N, Malaga G, Akl EA, et al: Stopping randomized trials early for benefit and estimation of treatment effects: systematic review and meta-regression analysis. JAMA. 2010, 303: 1180-1187. 10.1001/jama.2010.310. · DOI · Paper page
  7. Thorlund K, Devereaux PJ, Wetterslev J, Guyatt G, Ioannidis JP, Thabane L, Gluud LL, Als-Nielsen B, Gluud C: Can trial sequential monitoring boundaries reduce spurious inferences from meta-analyses?. Int J Epidemiol. 2009, 38 (1): 276-286. 10.1093/ije/dyn179. · DOI · Paper page
  8. Garattini S, Bertele V: Non-inferiority trials are unethical because they disregard patients' interests. Lancet. 2007, 370 (9602): 1875-1877. 10.1016/S0140-6736(07)61604-3. · DOI · Paper page
  9. Sterne JA: Teaching hypothesis tests–time for significant change?. Stat Med. 2002, 21: 985-999. 10.1002/sim.1129. · DOI
  10. Ranstam J: Why the P-value culture is bad and confidence intervals a better alternative. Osteoarthritis Cartilage. 2012, 20: 805-808. 10.1016/j.joca.2012.04.001. · DOI
  11. Williamson PR, Altman DG, Blazeby JM, Clarke M, Gargon E: The COMET (Core Outcome Measures in Effectiveness Trials) Initiative. Trials. 2011, 12 (Suppl 1): A70-10.1186/1745-6215-12-S1-A70. · DOI · Paper page
  12. Altman DG, Bland JM: How to obtain the confidence interval from a P value. BMJ. 2011, 343: d2090-10.1136/bmj.d2090. · DOI · Paper page
  13. Chow S-C, Shao J, Wang H: Sample Size Calculations in Clinical Research, Second Edition. 2008, Boca Raton, Florida: Chapman and Hall/CRC
  14. Scales DC, Rubenfeld GD: Estimating sample size in critical care clinical trials. J Crit Care. 2005, 20 (1): 6-11. 10.1016/j.jcrc.2005.02.002. · DOI
  15. Myles DJS, Keith RA, Jonathan P: Bayesian approaches to clinical trials and health-care evaluation (Statistics in Practice). 2004, West Sussex, England: John Wiley & Sons
  16. Roloff V, Higgins JP, Sutton AJ: Planning future studies based on the conditional power of a meta-analysis. Stat Med. 2013, 32 (1): 11-24. 10.1002/sim.5524. · DOI
  17. Goodman SN: Introduction to Bayesian methods I: measuring the strength of evidence. Clin Trials. 2005, 2: 282-378. 10.1191/1740774505cn098oa. · DOI
  18. Goodman SN: Toward evidence-based medical statistics. 2: The Bayes factor. Ann Int Med. 1999, 130 (12): 1005-1013. 10.7326/0003-4819-130-12-199906150-00019. · DOI · Paper page
  19. Higgins JP, Whitehead A: Borrowing strength from external trials in a meta-analysis. Stat Med. 1996, 15 (24): 2733-2749. 10.1002/(SICI)1097-0258(19961230)15:24<2733::AID-SIM562>3.0.CO;2-0. · DOI · Paper page
  20. Fayers PM, Cuschieri A, Fielding J, Craven J, Uscinska B, Freedman LS: Sample size calculation for clinical trials: the impact of clinician beliefs. Br J Cancer. 2000, 82 (1): 213-219. 10.1054/bjoc.1999.0902. · DOI
  21. Thorlund K, Imberger G, Walsh M, Chu R, Gluud C, Wetterslev J, Guyatt G, Devereaux PJ, Thabane L: The number of patients and events required to limit the risk of overestimation of intervention effects in meta-analysis-a simulation study. PLoS One. 2011, 6: e25491-10.1371/journal.pone.0025491. · DOI · Paper page
  22. Pereira TV, Horwitz RI, Ioannidis JP: Empirical evaluation of very large treatment effects of medical interventions. JAMA. 2012, 308: 1676-1684. 10.1001/jama.2012.13444. · DOI · Paper page
  23. Mehta CR, Pocock SJ: Adaptive increase in sample size when interim results are promising: a practical guide with examples. Stat Med. 2011, 30 (28): 3267-3284. 10.1002/sim.4102. · DOI · Paper page
  24. Jennison C, Turnbull BW: Efficient group sequential designs when there are several effect sizes under consideration. Stat Med. 2005, 25: 917-932. · DOI
  25. O'Hagan A, Stevens JW, Campbell MJ: Assurance in clinical trial design. Pharm Stat. 2005, 4 (3): 187-201. 10.1002/pst.175. · DOI · Paper page
  26. Levin GP, Emerson SC, Emerson SS: Adaptive clinical trial designs with pre-specified rules for modifying the sample size: understanding efficient types of adaptation. Stat Med. 2012, 32 (8): 1259-1275. · DOI
  27. DeMets DL, Lan KK: Interim analysis: the alpha spending function approach. Stat Med. 1994, 13 (13–14): 1341-1356. · DOI · Paper page
  28. Bassler D, Montori VM, Briel M, Glasziou P, Walter SD, Ramsay T, Guyatt G: Reflections on meta-analyses involving trials stopped early for benefit: is there a problem and if so, what is it?. Stat Methods Med Res. 2013, 22 (2): 159-168. 10.1177/0962280211432211. · DOI
  29. Lindley DV: A statistical paradox. Biometrika. 1957, 44 (1/2): 187-192. 10.2307/2333251. · doi:10.2307/2333251
  30. Wald A: Sequential tests of statistical hypotheses. Ann Math Stat. 1945, 16: 117-186. 10.1214/aoms/1177731118. · DOI · Paper page
  31. Zhang J, Quan H, Ng J, Stepanavage ME: Some statistical methods for multiple endpoints in clinical trials. Control Clin Trials. 1997, 18: 204-221. 10.1016/S0197-2456(96)00129-8. · DOI · Paper page
  32. Imberger G, Vejlby AD, Hansen SB, Møller AM, Wetterslev J: Statistical multiplicity in systematic reviews of anaesthesia interventions: a quantification and comparison between Cochrane and non-Cochrane reviews. PLoS One. 2011, 6: e28422-10.1371/journal.pone.0028422. · DOI
  33. Jennison C, Turnbull BW: Repeated confidence intervals for group sequential clinical trials. Control Clin Trials. 1984, 5 (1): 33-45. 10.1016/0197-2456(84)90148-X. · DOI
  34. Todd S, Whitehead J, Facey KM: Point and interval estimation following a sequential clinical trial. Biometrika. 1996, 83 (2): 453-461. 10.1093/biomet/83.2.453. · DOI
  35. Jennison C, Turnbull BW: Group Sequential Methods with Applications to Clinical Trials (Chapman & Hall/CRC Interdisciplinary Statistics). 1999, : Chapman and Hall/CRC · DOI · Paper page
  36. Thorlund K, Engstrøm J, Wetterslev J, Brok J, Imberger G, Gluud C: User manual for trial sequential analysis (TSA). 2011, Copenhagen, Denmark: Copenhagen Trial Unit, Centre for Clinical Intervention Research, 1-115. Available from http://www.ctu.dk/tsa
  37. Yang Q, Cui J, Chazaro I, Cupples LA, Demissie S: Power and type I error rate of false discovery rate approaches in genome-wide association studies. BMC Genet. 2005, 6 (Suppl 1): S134-10.1186/1471-2156-6-S1-S134. · DOI
  38. Altman DG, Bland JM: How to obtain the P value from a confidence interval. BMJ. 2011, 343: d2304-10.1136/bmj.d2304. · DOI · Paper page
  39. Abdi H: Encyclopedia of Measurement and Statistics. The Bonferonni and Šidák corrections for multiple comparisons. In N.J. Salkind (Ed.) page 103–107. 2007, Thousand Oaks (CA): Sage
  40. Holm S: A simple sequentially rejective multipletestprocedure. Scand J Statist. 1979, 6: 65-70.
  41. Dmitrienko A, Ajit C, Tamhane AC, Bretz F: Multiple testing problems in pharmaceutical statistics (Chapman & Hall/CRC Biostatistics Series). 2009, Boca Raton, Florida: Chapman and Hall/CRC · DOI · Paper page
  42. Tu YH, Cheng B, Cheung YK: A note on confidence bounds after fixed-sequence multiple tests. J Stat Plan Inference. 2012, 142 (11): 2993-2998. 10.1016/j.jspi.2012.05.002. · doi:10.1016/j.jspi.2012.05.002
  43. Wiens BL, Dmitrienko A: The fallback procedure for evaluating a single family of hypotheses. J Biopharm Stat. 2005, 15 (6): 929-942. 10.1080/10543400500265660. · DOI
  44. Korn EL, Li MC, McShane LM, Simon R: An investigation of two multivariate permutation methods for controlling the false discovery proportion. Stat Med. 2007, 26 (24): 4428-4440. 10.1002/sim.2865. · DOI
  45. Westfall PH, Young S: Resampling-Based Multiple Testing: Examples and Methods for p-Value Adjustment (Wiley Series in Probability and Statistics). 1993, New York: Wiley-Interscience
  46. Yu J, Hutson AD, Siddiqui AH, Kedron MA: Group sequential control of overall toxicity incidents in clinical trials - non-Bayesian and Bayesian approaches. Stat Methods Med Res. 2012, Epub ahead of print
  47. Thall PF, Simon RM, Shen Y: Approximate Bayesian evaluation of multiple treatment effects. Biometrics. 2000, 56: 213-219. 10.1111/j.0006-341X.2000.00213.x. · DOI
  48. Zhang X, Cutter G: Bayesian interim analysis in clinical trials. Contemp Clin Trials. 2008, 29: 751-755. 10.1016/j.cct.2008.05.007. · DOI
  49. Jakobsen JC, Lindschou Hansen J, Storebø OJ, Simonsen E, Gluud C: The effects of cognitive therapy versus 'treatment as usual' in patients with major depressive disorder. PLoS One. 2011, 6 (8): e22890-10.1371/journal.pone.0022890. · DOI
  50. Stovring H, Harmsen CG, Wisloff T, Jarbol DE, Nexoe J, Nielsen JB, Kristiansen IS: A competing risk approach for the European Heart SCORE model based on cause-specific and all-cause mortality. Eur J Prev Cardiol. 2012, 20 (5): 827-836. · doi:10.1177/2047487312445425
  51. Prasad V, Vandross A: Cardiovascular primary prevention: how high should we set the bar?. Arch Int Med. 2012, 172: 656-659. 10.1001/archinternmed.2012.812. · DOI
  52. Guyatt G, Oxman AD, Sultan S, Brozek J, Glasziou P, Alonso-Coello P, Atkins D, Kunz R, Montori V, Jaeschke R, Rind D, Dahm P, Akl EA, Meerpohl J, Vist G, Berliner E, Norris S, Falck-Ytter Y, Schunemann HJ: GRADE guidelines: 11. Making an overall rating of confidence in effect estimates for a single outcome and for all outcomes. J Clin Epidemiol. 2013, 66 (2): 151-157. 10.1016/j.jclinepi.2012.01.006. · DOI · Paper page
  53. Higgins JPT, Green S: The Cochrane Handbook for Systematic Reviews of Interventions, Version 5.1.0. 2011, The Cochrane Collaboration, Available from http://www.cochrane-handbook.org
  54. Johnston BC, Thorlund K, Schunemann HJ, Xie F, Murad MH, Montori VM, Guyatt GH: Improving the interpretation of quality of life evidence in meta-analyses: the application of minimal important difference units. Health Qual Life Outcomes. 2010, 8: 116-10.1186/1477-7525-8-116. · DOI · Paper page
  55. Halvorsen PA, Kristiansen IS: Decisions on drug therapies by numbers needed to treat: a randomized trial. Arch Int Med. 2005, 165: 1140-1146. 10.1001/archinte.165.10.1140. · DOI
  56. Chalmers I, Milne I, Trohler U, Vandenbroucke J, Morabia A, Tait G, Dukan E: The James Lind Library: explaining and illustrating the evolution of fair tests of medical treatments. J R Coll Physicians Edinb. 2008, 38 (3): 259-264. · DOI
  57. Garthwaite P, Kadane JB, O'Hagan A: Statistical Methods for Eliciting Probability Distributions. J Am Stat Assoc. 2012, 100 (470): · DOI · Paper page
  58. Higgins JP, Whitehead A, Simmonds M: Sequential methods for random-effects meta-analysis. Stat Med. 2011, 30 (9): 903-921. 10.1002/sim.4088. · DOI · Paper page
  59. Keus F, Wetterslev J, Gluud C, van Laarhoven CJ: Evidence at a glance: error matrix approach for overviewing available evidence. BMC Med Res Methodol. 2010, 10: 90-10.1186/1471-2288-10-90. · DOI · Paper page
  60. Johnson VE: Uniformly most powerful Bayesian tests. Ann Stat. 2013, 41: 1716-1741. 10.1214/13-AOS1123. · DOI
  61. Higgins JP, Spiegelhalter DJ: Being sceptical about meta-analyses: a Bayesian perspective on magnesium trials in myocardial infarction. Int J Epidemiol. 2002, 31 (1): 96-104. 10.1093/ije/31.1.96. · DOI
  62. Korn EL, Freidlin B: The likelihood as statistical evidence in multiple comparisons in clinical trials: no free lunch. Biom J. 2006, 48 (3): 346-355. 10.1002/bimj.200510216. · DOI
  63. Gaziano JM, Sesso HD, Christen WG, Bubes V, Smith JP, MacFadyen J, Schvartz M, Manson JE, Glynn RJ, Buring JE: Multivitamins in the prevention of cancer in men: the Physicians' Health Study II. JAMA. 2012, 308 (18): 1871-1880. 10.1001/jama.2012.14641. · DOI · Paper page
  64. Christen WG, Gaziano JM, Hennekens CH: Design of Physicians' Health Study II–a randomized trial of beta-carotene. Ann Epidemiol. 2000, 10 (2): 125-134. 10.1016/S1047-2797(99)00042-3. · DOI · Paper page
  65. Bjelakovic G, Nikolova D, Gluud LL, Simonetti RG, Gluud C: Antioxidant supplements for prevention of mortality in healthy participants and patients with various diseases. Cochrane Database Syst Rev. 2012, 3: CD007176
  66. Shakur H, Roberts I, Bautista R, Caballero J, Coats T, Dewan Y, El-Sayed H, Gogichaishvili T, Gupta S, Herrera J, Hunt B, Iribhogbe P, Izurieta M, Khamis H, Komolafe E, Marrero MA, Mejia-Mantilla J, Miranda J, Morales C, Olaomi O, Olldashi F, Perel P, Peto R, Ramana PV, Ravi RR, Yutthakasemsunt S: Effects of tranexamic acid on death, vascular occlusive events, and blood transfusion in trauma patients with significant haemorrhage (CRASH-2): a randomised, placebo-controlled trial. Lancet. 2010, 376 (9734): 23-32. · DOI · Paper page
  67. Perner A, Haase N, Guttormsen AB, Tenhunen J, Klemenzson G, Aneman A, Madsen KR, Moller MH, Elkjaer JM, Poulsen LM, Bendtsen A, Winding R, Steensen M, Berezowicz P, Soe-Jensen P, Bestle M, Strand K, Wiis J, White JO, Thornberg KJ, Quist L, Nielsen J, Andersen LH, Holst LB, Thormar K, Kjaeldgaard AL, Fabritius ML, Mondrup F, Pott FC, Moller T, et al: Hydroxyethyl starch 130/0.42 versus Ringer's acetate in severe sepsis. N Eng J Med. 2012, 367 (2): 124-134. 10.1056/NEJMoa1204242. · DOI · Paper page
  68. Perner A, Haase N, Wetterslev J, Aneman A, Tenhunen J, Guttormsen AB, Klemenzson G, Pott F, Bodker KD, Badstolokken PM, Bendtsen A, Soe-Jensen P, Tousi H, Bestle M, Pawlowicz M, Winding R, Bulow HH, Kancir C, Steensen M, Nielsen J, Fogh B, Madsen KR, Larsen NH, Carlsson M, Wiis J, Petersen JA, Iversen S, Schoidt O, Leivdal S, Berezowicz P, et al: Comparing the effect of hydroxyethyl starch 130/0.4 with balanced crystalloid solution on mortality and kidney failure in patients with severe sepsis (6S--Scandinavian Starch for Severe Sepsis/Septic Shock trial): study protocol, design and rationale for a double-blinded, randomised clinical trial. Trials. 2011, 12 (1): 24-10.1186/1745-6215-12-24. · DOI

Cited by

See all 203

Metadata

Authors
Jakobsen, J. C., Gluud, C., Winkel, P., Lange, T., & Wetterslev, J.
Venue
BMC Medical Research Methodology
Published
2014-03-04
Volume
14
Issue
1
Pages
34-34
ISSN
1471-2288 (ISSN-L)
OpenAlex
W2097687553
Cited by
203
References
21
FWCI
16.50
Type
journal article
Language
en
Open access
gold

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: The thresholds for statistical and clinical significance – a five-step procedure for evaluation of intervention effects in randomised clinical trials. https://app.cochat.ai/papers/10.1186/1471-2288-14-34
The paper
Jakobsen, J. C., Gluud, C., Winkel, P., Lange, T., & Wetterslev, J. (2014). The thresholds for statistical and clinical significance – a five-step procedure for evaluation of intervention effects in randomised clinical trials. BMC Medical Research Methodology. https://doi.org/10.1186/1471-2288-14-34

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.