Statistics notes: Absence of evidence is not evidence of absence
Open accessAbstract
Randomized controlled clinical trials are conducted to determine whether differences of clinical importance exist between selected treatment regimens. When statistical analysis of the study data finds a P value greater than 5%, it is convention to deem the assessed difference nonsignificant. Just because convention dictates that such study findings be termed nonsignificant, or negative, however, it does not necessarily follow that the study found nothing of clinical importance. Subject samples used in controlled trials tend to be too small. The studies therefore lack the necessary power to detect real, and clinically worthwhile, differences in treatment. Freiman et al. found that only 30% of a sample of 71 trials published in the New England Journal of Medicine in 1978-79 with a P value greater than 10% were large enough to have a 90% chance of detecting even a 50% difference in the effectiveness of the treatments being compared, and they found no improvement in a similar sample of trials published in 1988. It is therefore wrong and unwise to interpret so many negative trials as providing evidence of the ineffectiveness of new treatments. One must instead seriously question whether the absence of evidence is a valid justification for inaction. Efforts must be made to look for quantification of an association rather than just a P value, especially when the risks under investigation are small. The authors cite a recent trial comparing octreotide and sclerotherapy in patients with variceal bleeding, as well as the overview of clinical trials evaluating fibrinolytic treatment for preventing reinfarction after acute myocardial infarction as examples.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Null findings represent the absence of evidence rather than evidence of an absence (i.e., evidence of no effect).
“null findings represent the absence of evidence rather than evidence of an absence (i.e., evidence of no effect; Altman & Bland, 1995).”
From Does preregistration improve the credibility of research findings? (2020) · cites this paper as Altman & Bland, 1995
With classic null-hypothesis significance tests, a nonsignificant result does not provide support for the null hypothesis but only adds evidence that H0 cannot be rejected.
“It should be pointed out, however, that with classic null-hypothesis significance tests (NHSTs), finding a nonsignificant result does not provide support for the null hypothesis (H0), but only adds evidence that H0 cannot be rejected (Altman & Bland, 1995).”
From Impact of Police Academy Training on Recruits’ Integrity (2016) · cites this paper as Altman & Bland, 1995
Interpreting negative findings is challenging because absence of evidence is not evidence of absence.
“In fact, interpreting negative findings is particularly challenging, because absence of evidence is not evidence of absence 93 .”
From Lack of evidence to favor specific preventive interventions in psychosis: a network meta‐analysis (2018) · cites this paper as [93]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Freiman, J. A., Chalmers, T. C., Smith, H., & Kuebler, R. R. (1978) · New England Journal of Medicine · doi:10.1056/nejm197809282991304 · Cited by 1,628 · see citing works
Yusuf, S., Collins, R., Peto, R., Furberg, C. D., Stampfer, M. J., Goldhaber, S. Z., & Hennekens, C. H. (1985) · European Heart Journal · doi:10.1093/oxfordjournals.eurheartj.a061905 · Cited by 568 · see citing works
Cornell, R. G., Wolf, F. M., Bailar, J. C., & Mosteller, F. C. (1989) · Journal of the American Statistical Association · doi:10.2307/2289981 · Cited by 295 · see citing works
Octreotide infusion or emergency sclerotherapy for variceal haemorrhage
Sung, J. J. �., Lai, C., Chan, F. K. L., et al. (1993) · The Lancet · doi:10.1016/0140-6736(93)91758-e · Cited by 199 · see citing works
Proposal to outlaw the term “negative trial”
Chalmers, I. (1985) · BMJ · doi:10.1136/bmj.290.6473.1002 · Cited by 36 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Moher, D., Hopewell, S., Schulz, K. F., Montori, V., Gotzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2010) · BMJ · doi:10.1136/bmj.c869 · Cited by 8,883 · see citing works
Self-citationThe ASA Statement on <i>p</i> -Values: Context, Process, and Purpose
Wasserstein, R. L., & Lazar, N. A. (2016) · The American Statistician · doi:10.1080/00031305.2016.1154108 · Cited by 6,306 · see citing works
The Revised CONSORT Statement for Reporting Randomized Trials: Explanation and Elaboration
Altman, D. G., Schulz, K. F., Moher, D., Egger, M., Davidoff, F., Elbourne, D., Gøtzsche, P. C., Lang, T., & for the CONSORT Group (2001) · Annals of Internal Medicine · doi:10.7326/0003-4819-134-8-200104170-00012 · Cited by 3,835 · see citing works
Self-citationA tutorial on pilot studies: the what, why and how
Thabane, L., Ma, J., Chu, R., Cheng, J., Ismaila, A., Rios, L. P., Robson, R., Thabane, M., Giangregorio, L., & Goldsmith, C. H. (2010) · BMC Medical Research Methodology · doi:10.1186/1471-2288-10-1 · Cited by 3,384 · see citing works
Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations
Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016) · European Journal of Epidemiology · doi:10.1007/s10654-016-0149-3 · Cited by 3,200 · see citing works
Self-citationMoher, D., Hopewell, S., Schulz, K. F., Montori, V., Gøtzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2012) · International Journal of Surgery · doi:10.1016/j.ijsu.2011.10.001 · Cited by 2,345 · see citing works
Self-citationMoher, D., Hopewell, S., Schulz, K. F., Montori, V., Gøtzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.03.004 · Cited by 1,858 · see citing works
Self-citationUsing Bayes factor hypothesis testing in neuroscience to establish evidence of absence
Keysers, C., Gazzola, V., & Wagenmakers, E.-J. (2020) · Nature Neuroscience · doi:10.1038/s41593-020-0660-4 · Cited by 654 · see citing works
Bias in clinical intervention research.
Gluud, L. L. (2006) · American Journal of Epidemiology · doi:10.1093/aje/kwj069 · Cited by 636 · see citing works
Small studies: strengths and limitations
Hackshaw, A. (2008) · European Respiratory Journal · doi:10.1183/09031936.00136408 · Cited by 496 · see citing works
Fusar-Poli, P., Cappucciati, M., Borgwardt, S., Woods, S. W., Addington, J., Nelson, B., Nieman, D. H., Stahl, D. R., Rutigliano, G., Riecher-Rössler, A., et al. (2016) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2015.2324 · Cited by 440 · see citing works
Strategies to improve recruitment to randomised controlled trials
Treweek, S., Mitchell, E., Pitkethly, M., Cook, J., Kjeldstrøm, M., Johansen, M., Taskila, T. K., Sullivan, F., Wilson, S., Jackson, C., et al. (2010) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000013.pub5 · Cited by 378 · see citing works
Andermann, A. (2018) · Public Health Reviews · doi:10.1186/s40985-018-0094-7 · Cited by 323 · see citing works
Improving inferences about null effects with Bayes factors and equivalence tests.
Lakens, D., McLatchie, N., Isager, P. M., Scheel, A. M., & Dienes, Z. (2020) · The Journals of Gerontology: Series B · doi:10.1093/geronb/gby065 · Cited by 279 · see citing works
Dillard, J. P., Weber, K. M., & Vail, R. G. (2007) · Journal of Communication · doi:10.1111/j.1460-2466.2007.00360.x · Cited by 276 · see citing works
A systematic review of non-pharmacological therapies for sleep problems in later life
Montgomery, P. (2004) · Sleep Medicine Reviews · doi:10.1016/s1087-0792(03)00026-1 · Cited by 260 · see citing works
Sample size determination for image classification accuracy assessment and comparison
Foody, G. M. (2009) · International Journal of Remote Sensing · doi:10.1080/01431160903130937 · Cited by 248 · see citing works
Rafi, Z., & Greenland, S. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-01105-9 · Cited by 243 · see citing works
Ten common statistical mistakes to watch out for when writing or reviewing a manuscript
Makin, T. R., & Orban de Xivry, J.-J. (2019) · eLife · doi:10.7554/elife.48175 · Cited by 238 · see citing works
Lack of evidence to favor specific preventive interventions in psychosis: a network meta‐analysis
Davies, C., Cipriani, A., Ioannidis, J. P. A., Radua, J., Stahl, D., Provenzani, U., McGuire, P., & Fusar‐Poli, P. (2018) · World Psychiatry · doi:10.1002/wps.20526 · Cited by 229 · see citing works
Analysis and reporting of adverse events in randomised controlled trials: a review
Phillips, R., Hazell, L., Sauzet, O., & Cornelius, V. (2019) · BMJ Open · doi:10.1136/bmjopen-2018-024537 · Cited by 228 · see citing works
Interpretation of chronic pain clinical trial outcomes: IMMPACT recommended considerations
Smith, S. M., Dworkin, R. H., Turk, D. C., McDermott, M. P., Eccleston, C., Farrar, J. T., Rowbotham, M. C., Bhagwagar, Z., Burke, L. B., Cowan, P., et al. (2020) · Pain · doi:10.1097/j.pain.0000000000001952 · Cited by 194 · see citing works
Karkou, V., Aithal, S., Zubala, A., & Meekums, B. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.00936 · Cited by 186 · see citing works
Ensuring trial validity by data quality assurance and diversification of monitoring methods
Baigent, C., Harrell, F. E., Buyse, M., Emberson, J. R., & Altman, D. G. (2008) · Clinical Trials · doi:10.1177/1740774507087554 · Cited by 163 · see citing works
Self-citationPrognosis of Brief Psychotic Episodes: A Meta-analysis
Fusar-Poli, P., Cappucciati, M., Bonoldi, I., Hui, L. M. C., Rutigliano, G., Stahl, D. R., Borgwardt, S., Politi, P., Mishara, A. L., Lawrie, S. M., et al. (2016) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2015.2313 · Cited by 151 · see citing works
Meyners, M. (2012) · Food Quality and Preference · doi:10.1016/j.foodqual.2012.05.003 · Cited by 130 · see citing works
Oliver, D., Davies, C., Crossland, G., Lim, S., Gifford, G., McGuire, P., & Fusar-Poli, P. (2018) · Schizophrenia Bulletin · doi:10.1093/schbul/sbx166 · Cited by 126 · see citing works
Hooten, M. W., Qu, W., Townsend, C. O., & Judd, J. W. (2012) · Pain · doi:10.1016/j.pain.2012.01.020 · Cited by 123 · see citing works
Management of multiple myeloma: a systematic review and critical appraisal of published studies
Kumar, A., Loughran, T., Alsina, M., Durie, B. G., & Djulbegovic, B. (2003) · The Lancet Oncology · doi:10.1016/s1470-2045(03)01077-5 · Cited by 113 · see citing works
Ormhaug, S. M., Jensen, T. K., Wentzel-Larsen, T., & Shirk, S. R. (2014) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0033884 · Cited by 113 · see citing works
Preventive Treatments for Psychosis: Umbrella Review (Just the Evidence)
Fusar-Poli, P., Davies, C., Solmi, M., Brondino, N., De Micheli, A., Kotlicka-Antczak, M., Shin, J. I., & Radua, J. (2019) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2019.00764 · Cited by 108 · see citing works
Davies, C., Radua, J., Cipriani, A., Stahl, D., Provenzani, U., McGuire, P., & Fusar-Poli, P. (2018) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2018.00187 · Cited by 107 · see citing works
Yildiz, A., Siafis, S., Mavridis, D., Vieta, E., & Leucht, S. (2023) · The Lancet Psychiatry · doi:10.1016/s2215-0366(23)00199-2 · Cited by 104 · see citing works
Why we need confidence intervals
Altman, D. G. (2005) · World Journal of Surgery · doi:10.1007/s00268-005-7911-0 · Cited by 88 · see citing works
Self-citationPrevalence and risk of depressive disorders in women with polycystic ovary syndrome (PCOS)
Bhattacharya, S. M., & Jha, A. (2010) · Fertility and Sterility · doi:10.1016/j.fertnstert.2009.09.025 · Cited by 88 · see citing works
Comparing midwife-led and doctor-led maternity care: a systematic review of reviews
Sutcliffe, K., Caird, J., Kavanagh, J., Rees, R., Oliver, K., Dickson, K., Woodman, J., Barnett‐Paige, E., & Thomas, J. (2012) · Journal of Advanced Nursing · doi:10.1111/j.1365-2648.2012.05998.x · Cited by 79 · see citing works
Hengartner, M. P. (2020) · Therapeutic Advances in Psychopharmacology · doi:10.1177/2045125320921694 · Cited by 72 · see citing works
Use of an interactive computer agent to support breastfeeding
Edwards, R. A., Bickmore, T., Jenkins, L., Foley, M., & Manjourides, J. (2013) · Maternal and Child Health Journal · doi:10.1007/s10995-013-1222-0 · Cited by 63 · see citing works
Does preregistration improve the credibility of research findings?
Rubin, M. (2020) · The Quantitative Methods for Psychology · doi:10.20982/tqmp.16.4.p376 · Cited by 61 · see citing works
“However, null findings represent the absence of evidence rather than evidence of an absence (i.e., evidence of no effect; Altman & Bland, 1995).”
BackgroundGiromini, L., Ales, F., de Campora, G., Zennaro, A., & Pignolo, C. (2017) · Journal of Psychopathology and Behavioral Assessment · doi:10.1007/s10862-017-9611-0 · Cited by 56 · see citing works
White matter hyperintensities are associated with subthreshold amyloid accumulation
Moscoso, A., Rey-Bretal, D., Silva-Rodríguez, J., Aldrey, J. M., Cortés, J., Pías-Peleteiro, J., Ruibal, Á., & Aguiar, P. (2020) · NeuroImage · doi:10.1016/j.neuroimage.2020.116944 · Cited by 49 · see citing works
Impact of Police Academy Training on Recruits’ Integrity
Blumberg, D. M., Giromini, L., & Jacobson, L. B. (2016) · Police Quarterly · doi:10.1177/1098611115608322 · Cited by 47 · see citing works
Mathieu, S., Giraudeau, B., Soubrier, M., & Ravaud, P. (2012) · Joint Bone Spine · doi:10.1016/j.jbspin.2011.05.008 · Cited by 37 · see citing works
Normal gray matter volumes in women recovered from anorexia nervosa: a voxel-based morphometry study
Bang, L., Rø, Ø., & Endestad, T. (2016) · BMC Psychiatry · doi:10.1186/s12888-016-0856-z · Cited by 36 · see citing works
KRYSAN, D. J., & KEMPER, A. R. (2002) · The Pediatric Infectious Disease Journal · doi:10.1097/00006454-200208000-00011 · Cited by 27 · see citing works
Perry, A., Gardener, C., Oliver, J. E., Taş, Ç., & Özenç, C. (2019) · The Cognitive Behaviour Therapist · doi:10.1017/s1754470x18000041 · Cited by 16 · see citing works
Genetic evidence for serum amyloid P component as a drug target in neurodegenerative disorders
Schmidt, A. F., Finan, C., Chopade, S., Ellmerich, S., Rossor, M. N., Hingorani, A. D., & Pepys, M. B. (2024) · Open Biology · doi:10.1098/rsob.230419 · Cited by 11 · see citing works
“This, however, also implies that non-significant findings should not be over-interpreted as providing proof of absence [57].”
The isolated lateral malleolar fracture: where are we and how did we get here?
Smith, G. (2013) · The Surgeon · doi:10.1016/j.surge.2012.02.008 · Cited by 9 · see citing works
- Authors
- Altman, D. G., & Bland, J. M.
- Affiliations
- St George's, University of London (GB)
- ORCID
- Altman, Douglas G
- Venue
- BMJ
- Publisher
- BMJ
- Published
- 1995-08-19
- Volume
- 311
- Issue
- 7003
- Pages
- 485
- ISSN
- 0007-1447 (ISSN-L)
- 0959-8138
- 1468-5833
- PMC
- PMC2550545
- OpenAlex
- W1525161423
- Semantic Scholar
- aad97125a6159182b0be330f6d3ac45ec8992f1e
- Cited by
- 1,946
- References
- 5
- Citations by year
- 2026: 195 · 2025: 82 · 2024: 88 · 2023: 108 · 2022: 105 · 2021: 107 · 2020: 106 · 2019: 104 · 2018: 78 · 2017: 74 · 2016: 70 · 2015: 60 · 2014: 76 · 2013: 73 · 2012: 68
- FWCI
- 15.94
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- www.bmj.com/company/legal-information/terms-conditions/legal-information/tdm-licencepolicy (tdm)
- Locations
- BMJ (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- HighWire Press Open Archive (submittedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- statistical power, non-significant results, negative clinical trials, treatment ineffectiveness, clinical significance, p-value interpretation, sample size, octreotide, sclerotherapy, variceal bleeding, thrombolysis, acute myocardial infarction, reinfarction prevention
- MeSH terms
- Data Interpretation, Statistical, Statistics as Topic, Randomized Controlled Trials as Topic, Sample Size, Controlled Clinical Trials as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
