Interpreting Null Findings from Trials of Alcohol Brief Interventions
Open accessThe DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Abstract
The effectiveness of alcohol brief intervention (ABI) has been established by a succession of meta-analyses but, because the effects of ABI are small, null findings from randomized controlled trials are often reported and can sometimes lead to skepticism regarding the benefits of ABI in routine practice. This article first explains why null findings are likely to occur under null hypothesis significance testing (NHST) due to the phenomenon known as "the dance of the p-values." A number of misconceptions about null findings are then described, using as an example the way in which the results of the primary care arm of a recent cluster-randomized trial of ABI in England (the SIPS project) have been misunderstood. These misinterpretations include the fallacy of "proving the null hypothesis" that lack of a significant difference between the means of sample groups can be taken as evidence of no difference between their population means, and the possible effects of this and related misunderstandings of the SIPS findings are examined. The mistaken inference that reductions in alcohol consumption seen in control groups from baseline to follow-up are evidence of real effects of control group procedures is then discussed and other possible reasons for such reductions, including regression to the mean, research participation effects, historical trends, and assessment reactivity, are described. From the standpoint of scientific progress, the chief problem about null findings under the conventional NHST approach is that it is not possible to distinguish "evidence of absence" from "absence of evidence." By contrast, under a Bayesian approach, such a distinction is possible and it is explained how this approach could classify ABIs in particular settings or among particular populations as either truly ineffective or as of unknown effectiveness, thus accelerating progress in the field of ABI research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007) · Behavior Research Methods · doi:10.3758/bf03193146 · Cited by 65,971
SAUNDERS, J. B., AASLAND, O. G., BABOR, T. F., DE LA FUENTE, J. R., & GRANT, M. (1993) · Addiction · doi:10.1111/j.1360-0443.1993.tb02093.x · Cited by 13,597
McCambridge, J., Witton, J., & Elbourne, D. R. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.08.015 · Cited by 2,748
The Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis
Hoenig, J. M., & Heisey, D. M. (2001) · The American Statistician · doi:10.1198/000313001300339897 · Cited by 2,046 · cited 1×
“Note, however, that it is also fallacious to believe that the null hypothesis can eventually be “proved” by increasing the sample size and statistical power (25).”
BackgroundUsing Bayes to get the most out of non-significant results
Dienes, Z. (2014) · Frontiers in Psychology · doi:10.3389/fpsyg.2014.00781 · Cited by 1,998 · cited 7×
“As Dienes (63) points out, however, one way in which they do clearly disagree is in the interpretation of nonsignificant results.”
BackgroundResultInfluentialEffectiveness of brief alcohol interventions in primary care populations
Kaner, E. F., Beyer, F. R., Muirhead, C., Campbell, F., Pienaar, E. D., Bertholet, N., Daeppen, J. B., Saunders, J. B., & Burnand, B. (2018) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004148.pub4 · Cited by 1,665
Moyer, A., Finney, J. W., Swearingen, C. E., & Vergun, P. (2002) · Addiction · doi:10.1046/j.1360-0443.2002.00018.x · Cited by 1,092 · cited 1×
“5 and is better estimated as small to medium (14), say δ = 0.”
BackgroundBeyond significance testing: Reforming data analysis methods in behavioral research.
Kline, R. B. (2004) · doi:10.1037/10693-000 · Cited by 1,034
Meta‐Analysis Of Controlled Clinical Trials
Whitehead, A. (2002) · doi:10.1002/0470854200 · Cited by 827
Effectiveness of brief alcohol interventions in primary care populations
Kaner, E. F. S., Dickinson, H. O., Beyer, F. R., Campbell, F., Schlesinger, C., Heather, N., Saunders, J. B., Burnand, B., & Pienaar, E. D. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004148.pub3 · Cited by 746
Self-citationParsons, H. M. (1974) · Science · doi:10.1126/science.183.4128.922 · Cited by 615
The impact of brief alcohol interventions in primary healthcare: a systematic review of reviews
O'Donnell, A., Anderson, P., Newbury-Birch, D., Schulte, B., Schmidt, C., Reimer, J., & Kaner, E. (2014) · Alcohol and Alcoholism · doi:10.1093/alcalc/agt170 · Cited by 613 · cited 1×
“The effectiveness of alcohol brief intervention (ABI) in reducing alcohol consumption among hazardous and harmful drinkers is generally considered to have been demonstrated by a succession of systematic reviews with meta-analysis (1).”
BackgroundSAUNDERS, J. B., AASLAND, O. G., AMUNDSEN, A., & GRANT, M. (1993) · Addiction · doi:10.1111/j.1360-0443.1993.tb00822.x · Cited by 480
Cumming, G. (2008) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2008.00079.x · Cited by 451
Publication Bias (The "File-Drawer Problem") in Scientific Inference
Scargle, J. D. (1999) · arXiv · doi:10.48550/arxiv.physics/9909033 · Cited by 313
Kaner, E., Bland, M., Cassidy, P., Coulton, S., Dale, V., Deluca, P., Gilvarry, E., Godfrey, C., Heather, N., Myles, J., et al. (2013) · BMJ · doi:10.1136/bmj.e8501 · Cited by 277 · cited 4×
“At the time of writing, only the results for the PHC trial have been published (20) and the other two trials will not be covered here.”
BackgroundInfluentialSelf-citationMcCambridge, J., & Kypri, K. (2011) · PLoS ONE · doi:10.1371/journal.pone.0023748 · Cited by 242 · cited 3×
“, by deliberately seeking to foster a discrepancy between the person’s actual selfconcept in relation to drinking behavior and the drinking of their ideal self, by asking explicitly about intentions to cut down or quit, or by enquiring about the perceived benefits of more moderate drinking (51, 58).”
BackgroundA systematic review of emergency care brief alcohol interventions for injury patients.
Nilsen, P., Baird, J., Mello, M. J., Nirenberg, T., Woolard, R., Bendtsen, P., & Longabaugh, R. (2008) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2007.09.008 · Cited by 207
A meta‐analysis of the effectiveness of bibliotherapy for alcohol problems
Apodaca, T. R., & Miller, W. R. (2003) · Journal of Clinical Psychology · doi:10.1002/jclp.10130 · Cited by 162 · cited 1×
“In relation to the provision of a PIL, and depending on how much information of what kind it contained, bibliotherapy in general has been shown to be an effective means of decreasing alcohol problems (62).”
BackgroundWilliams, E. C., Johnson, M. L., Lapham, G. T., Caldeiro, R. M., Chew, L., Fletcher, G. S., McCormick, K. A., Weppner, W. G., & Bradley, K. A. (2011) · Psychology of Addictive Behaviors · doi:10.1037/a0022102 · Cited by 138 · cited 1×
“It is well known that there have been considerable difficulties in persuading GPs, nurses, and other healthcare professionals to implement ABI routinely in their practices; there is a copious literature on this problem (30) and how it may be redressed (31).”
BackgroundAALTO, M., PEKURI, P., & SEPPÄ, K. (2003) · Drug and Alcohol Review · doi:10.1080/09595230100100606 · Cited by 119
Wilson, G. B., Lock, C. A., Heather, N., Cassidy, P., Christie, M. M., & Kaner, E. F. S. (2011) · Alcohol and Alcoholism · doi:10.1093/alcalc/agr067 · Cited by 112 · cited 1×
“In surveys of health professionals’ attitudes to this work, one of the most commonly encountered obstacles is “lack of time” or “too busy” (32, 33).”
BackgroundSelf-citationJenkins, R. J., McAlaney, J., & McCambridge, J. (2009) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2008.09.016 · Cited by 103 · cited 1×
“reduce their drinking by approximately 20% (35, 36).”
BackgroundControlled evaluation of a general practice-based brief intervention for excessive drinking
RICHMOND, R., HEATHER, N., WODAK, A., KEHOE, L., & WEBSTER, I. (1995) · Addiction · doi:10.1111/j.1360-0443.1995.tb01016.x · Cited by 89
Self-citationRegression to the mean in substance use disorder treatment research
Finney, J. W. (2008) · Addiction · doi:10.1111/j.1360-0443.2007.02032.x · Cited by 78
BERNSTEIN, J. A., BERNSTEIN, E., & HEEREN, T. C. (2010) · Drug and Alcohol Review · doi:10.1111/j.1465-3362.2010.00174.x · Cited by 68
McCambridge, J., Kypri, K., & McElduff, P. (2014) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2013.11.017 · Cited by 54 · cited 1×
“The possible effects of regression to the mean on control group participants in brief intervention trials were studied empirically by McCambridge and colleagues (40).”
BackgroundMdege, N. D., Fayter, D., Watson, J. M., Stirk, L., Sowden, A., & Godfrey, C. (2013) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2013.01.023 · Cited by 54 · cited 1×
“While apparently strong in the primary health care (PHC) setting, the evidence to support ABI in emergency (6) and general hospital (7) settings is more equivocal.”
BackgroundThe efficacy-effectiveness distinction in trials of alcohol brief intervention
Heather, N. (2014) · Addiction Science & Clinical Practice · doi:10.1186/1940-0640-9-13 · Cited by 48
Self-citationCherpitel, C. J., Korcha, R. A., Moskalewicz, J., Swiatkiewicz, G., Ye, Y., & Bond, J. (2010) · Alcoholism: Clinical and Experimental Research · doi:10.1111/j.1530-0277.2010.01281.x · Cited by 45
Research works printed but not linked: 15
- Jonas, 2012, Screening, Behavioral Counseling, and Referral in Primary Care to Reduce Alcohol Misuse
- Ballesteros, 2004, Brief interventions for hazardous drinkers delivered in primary care are equally effective in men and women, Addiction, 99, 103, 10.1111/j.1360-0443.2004.00499.x · DOI · Paper page
- 2011, Appraisal for Screening for Alcohol Misuse
- Rosenthal, 1979, The ‘file drawer problem’ and tolerance for null results, Psychol Bull, 86, 638, 10.1037/0033-2909.86.3.638 · DOI · Paper page
- Heather, 2012, Can screening and brief intervention lead to population-level reductions in alcohol-related harm?, Addict Sci Clin Pract, 7, 15, 10.1186/1940-0640-7-15 · DOI · Paper page
- Cumming, 2012, Understanding the New Statistics: Effect Sizes, Confidence Intervals, and Meta-analysis
- Cohen, 1988, Statistical Power Analysis for the Behavioural Sciences
- 2010, Publication Manual of the American Psychological Association
- Schmidt, 1997, Eight common but false objections to the discontinuation of significance testing in the analysis of research data, 37
- 2004, Alcohol Harm Reduction Strategy for England
- Kaner, 2009, Screening and brief interventions for hazardous and harmful alcohol use in primary care: a randomised controlled trial protocol, BMC Public Health, 9, 287, 10.1186/1471-2458-9-287 · DOI · Paper page
- Drummond, 1990, The relationship between alcohol dependence and alcohol-related problems in a clinical population, Br J Addict, 85, 357, 10.1111/j.1360-0443.1990.tb00652.x · DOI · Paper page
- Fisher, 1971, The Design of Experiments
- Saitz, 2013, SIPS trial findings most consistent with a lack of effectiveness of brief intervention in real clinical practice (letter), Br Med J, 346, e8501, 10.1136/bmj.e8501 · DOI · Paper page
- 2013, Clarifying Alcohol Brief Interventions: 2013 Update
Cited by
Tear, M. J., Reader, T. W., Shorrock, S., & Kirwan, B. (2020) · Safety Science · doi:10.1016/j.ssci.2018.10.014 · Cited by 116
“Or more simply, we should not confuse ‘absence of evidence’ with ‘evidence of absence’ (Heather, 2014).”
BackgroundBrown, J., West, R., Angus, C., Beard, E., Brennan, A., Drummond, C., Hickman, M., Holmes, J., Kaner, E., & Michie, S. (2016) · British Journal of General Practice · doi:10.3399/bjgp16x683149 · Cited by 96
Beyer, F., Lynch, E., & Kaner, E. (2018) · Current Addiction Reports · doi:10.1007/s40429-018-0198-7 · Cited by 66
Tanner‐Smith, E. E., Parr, N. J., Schweer‐Collins, M., & Saitz, R. (2022) · Addiction · doi:10.1111/add.15674 · Cited by 54
Diestelkamp, S., Drechsel, M., Baldus, C., Wartberg, L., Arnaud, N., & Thomasius, R. (2016) · European Addiction Research · doi:10.1159/000435877 · Cited by 32
Metadata
- Authors
- Heather, N.
- Venue
- Frontiers in Psychiatry
- Publisher
- Frontiers Media SA
- Published
- 2014-07-16
- Volume
- 5
- Pages
- 85-85
- ISSN
- 1664-0640 (ISSN-L)
- OpenAlex
- W2112943852
- Cited by
- 32
- References
- 30
- FWCI
- 3.60
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.