Departing from the essential features of a high quality systematic review of psychotherapy: A response to Öst (2014) and recommendations for improvement
Open accessThe DOI matches its Crossref record; 74 of the 74 references it cites were found in a registry or an index.
Abstract
Öst's (2014) systematic review and meta-analysis of Acceptance and Commitment Therapy (ACT) has received wide attention. On the basis of his review, Öst argued that ACT research was not increasing in its quality and that, in contradiction to the views of Division 12 of the American Psychological Association (APA), ACT is "not yet well-established for any disorder" (2014, p. 105). We conducted a careful examination of the methods, approach, and data used in the meta-analysis. Based in part on examinations by the authors of the studies involved, which were then independently checked, 91 factual or interpretive errors were documented, touching upon 80% of the studies reviewed. Comparisons of Öst's quality ratings with independent teams rating the same studies with the same scale suggest that Ost's ratings were unreliable. In all of these areas (factual errors; interpretive errors; quality ratings) mistakes and differences were not random: Ost's data were dominantly more negative toward ACT. The seriousness, range, and distribution of errors, and a wider pattern of misinterpreting the purpose of studies and ignoring positive results, suggest that Öst's review should be set aside in future considerations of the evidence base for ACT. We argue that future published reviews and meta-analyses should rely upon diverse groups of scholars rather than a single individual; that resulting raw data should be made available for inspection and independent analysis; that well-crafted committees rather than individuals should design, apply and interpret quality criteria; that the intent of transdiagnostic studies need to be more seriously considered as the field shifts away from a purely syndromal approach; and that data that demonstrate theoretically consistent mediating processes should be given greater weight in evaluating specific interventions. Finally, in order to examine substantive progress since Öst's review, recent outcome and process evidence was briefly examined.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
1 retracted
Psychotherapy for depression in children and adolescents
Watanabe, N., Churchill, R., Hunot, V., & Furukawa, T. A. (2004) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd005334 · Cited by 24
RetractedAcceptance and commitment therapy: model, processes and outcomes
Hayes, S. C., Luoma, J. B., Bond, F. W., Masuda, A., & Lillis, J. (2006) · Behaviour Research and Therapy · doi:10.1016/j.brat.2005.06.006 · Cited by 6,587
Self-citationPain: a review of three commonly used pain rating scales
Williamson, A., & Hoggart, B. (2005) · Journal of Clinical Nursing · doi:10.1111/j.1365-2702.2005.01121.x · Cited by 3,083
Defining empirically supported therapies.
Chambless, D. L., & Hollon, S. D. (1998) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.66.1.7 · Cited by 3,067
Mediators and Mechanisms of Change in Psychotherapy Research
Kazdin, A. E. (2007) · Annual Review of Clinical Psychology · doi:10.1146/annurev.clinpsy.3.022806.091432 · Cited by 2,504
Validity of four pain intensity rating scales
Ferreira-Valente, M. A., Pais-Ribeiro, J. L., & Jensen, M. P. (2011) · Pain · doi:10.1016/j.pain.2011.07.005 · Cited by 1,872
Goodman, R., Ford, T., Richards, H., Gatward, R., & Meltzer, H. (2000) · Journal of Child Psychology and Psychiatry · doi:10.1111/j.1469-7610.2000.tb02345.x · Cited by 1,755
Kazdin, A. E. (2008) · American Psychologist · doi:10.1037/0003-066x.63.3.146 · Cited by 1,112 · cited 1×
“advancing evidence-based therapy (Borkovec & Sibrava, 2005; David & Montgomery, 2011; Herbert & Gaudiano, 2005; Kazdin, 2008; Lohr, 2011) and have pointed with concern to weaknesses in these areas in mainstream CBT (Gaudiano, 2005; Kazdin, 2007; Longmore &”
BackgroundVeehof, M. M., Trompetter, H. R., Bohlmeijer, E. T., & Schreurs, K. M. G. (2016) · Cognitive Behaviour Therapy · doi:10.1080/16506073.2015.1098724 · Cited by 737
Efficacy of the third wave of behavioral therapies: A systematic review and meta- analysis
Öst, L.-G. (2008) · Behaviour Research and Therapy · doi:10.1016/j.brat.2007.12.005 · Cited by 727
Research works printed but not linked: 19
- A-Tjak, 2015, A meta-analysis of the efficacy of acceptance and commitment therapy for clinically relevant mental and physical health problems, Psychotherapy and Psychosomatics, 84, 30, 10.1159/000365764 · DOI · Paper page
- Aebi, 2012, The use of the development and well-being assessment (DAWBA) in clinical practice: A randomized trial, European Child & Adolescent Psychiatry, 21, 559, 10.1007/s00787-012-0293-6 · DOI · Paper page
- Arch, 2012, Randomized clinical trial of cognitive behavioral therapy (CBT) versus acceptance and commitment therapy (ACT) for mixed anxiety disorders, Journal of Consulting and Clinical Psychology, 80, 750, 10.1037/a0028310 · DOI · Paper page
- Benowitz, 2003, Nicotine metabolite ratio as a predictor of cigarette consumption, Nicotine & Tobacco Research, 5, 621, 10.1080/1462220031000158717 · DOI · Paper page
- Bethay, 2013, A controlled pilot evaluation of acceptance and commitment training for intellectual disability staff, Mindfulness, 4, 113, 10.1007/s12671-012-0103-8 · DOI
- David, 2011, The scientific status of psychotherapies: A new evaluative framework for evidence-based psychosocial interventions, Clinical Psychology: Science and Practice, 18, 89
- Forman, 2013, Comparison of acceptance-based and standard cognitive-based coping strategies for craving sweets in overweight and obese women, Eating Behaviors, 14, 64, 10.1016/j.eatbeh.2012.10.016 · DOI
- Han, 2015, Relationship between methamphetamine use history and segmental hair analysis findings of MA users, Forensic Science International, 254, 59, 10.1016/j.forsciint.2015.06.029 · DOI
- Insel, 2010, Research domain criteria (RDoC): Toward a new classification framework for research on mental disorders, American Journal of Psychiatry, 167, 748, 10.1176/appi.ajp.2010.09091379 · DOI · Paper page
- Landis, 1977, The measurement of observer agreement for categorical data, Biometrics, 33, 159, 10.2307/2529310 · DOI · Paper page
- Lillis, 2009, Teaching acceptance and mindfulness to improve the lives of the obese: A preliminary test of a theoretical model, Annals of Behavioral Medicine, 37, 58, 10.1007/s12160-009-9083-x · DOI · Paper page
- Lohr, 2011, What is (and what is not) the meaning of evidence-based psychosocial intervention?, Clinical Psychology: Science and Practice, 18, 100
- Overall, 1962, The brief psychiatric rating scale, Psychological Reports, 10, 799, 10.2466/pr0.1962.10.3.799 · DOI · Paper page
- Patrick, 1994, The validity of self-reported smoking: A review and meta-analysis, American Journal of Public Health, 84, 1086, 10.2105/AJPH.84.7.1086 · DOI · Paper page
- Ruiz, 2012, Acceptance and commitment therapy versus traditional cognitive behavioral therapy: A systematic review and meta-analysis of current empirical evidence, International Journal of Psychology and Psychological Therapy, 12, 333
- Silverman, 2008, The second special issue on evidence-based psychosocial treatments for children and adolescents: A 10-year update, Journal of Clinical Child & Adolescent Psychology, 37, 1, 10.1080/15374410701817725 · DOI · Paper page
- Stotts, 2012, A stage I pilot study of acceptance and commitment therapy for methadone detoxification, Drug Alcohol Depend, 125, 215, 10.1016/j.drugalcdep.2012.02.015 · DOI · Paper page
- Turk, 2003, Core outcome domains for chronic pain clinical trials: IMMPACT recommendations, PAIN, 106, 337, 10.1016/j.pain.2003.08.001 · DOI · Paper page
- Wolfe, 1990, The American College of Rheumatology 1990 criteria for the classification of fibromyalgia, Arthritis & Rheumatology, 33, 160, 10.1002/art.1780330203 · DOI · Paper page
Cited by
Hayes, S. C., Ciarrochi, J., Hofmann, S. G., Chin, F., & Sahdra, B. (2022) · Behaviour Research and Therapy · doi:10.1016/j.brat.2022.104155 · Cited by 234
Self-citationCurrent status of acceptance and commitment therapy for chronic pain: A narrative review
Feliu Soler, A., Montesinos, F., Gutiérrez-Martínez, O., Scott, W., McCracken, L., & Luciano, J. (2018) · Journal of Pain Research · doi:10.2147/jpr.s144631 · Cited by 198
O'Neill, L., Latchford, G., McCracken, L. M., & Graham, C. D. (2019) · Journal of Contextual Behavioral Science · doi:10.1016/j.jcbs.2019.08.008 · Cited by 87
Prudenzi, A., Graham, C. D., Clancy, F., Hill, D., O'Driscoll, R., Day, F., & O'Connor, D. B. (2021) · Journal of Affective Disorders · doi:10.1016/j.jad.2021.07.084 · Cited by 64
Dochat, C., Wooldridge, J. S., Herbert, M. S., Lee, M. W., & Afari, N. (2021) · Journal of Contextual Behavioral Science · doi:10.1016/j.jcbs.2021.03.003 · Cited by 60
Kyllönen, H. M., Muotka, J., Puolakanaho, A., Astikainen, P., Keinonen, K., & Lappalainen, R. (2018) · Journal of Contextual Behavioral Science · doi:10.1016/j.jcbs.2018.08.009 · Cited by 38
Hayes, S. C., Hofmann, S. G., & Ciarrochi, J. (2023) · Behavior Therapy · doi:10.1016/j.beth.2023.07.011 · Cited by 37
Self-citationHarris, E., & Samuel, V. (2020) · Journal of Cognitive Psychotherapy · doi:10.1891/jcpsy-d-20-00001 · Cited by 32
Atefi, G. L., de Vugt, M. E., van Knippenberg, R. J. M., Levin, M. E., Verhey, F. R. J., & Bartels, S. L. (2023) · Clinical Psychology Review · doi:10.1016/j.cpr.2023.102341 · Cited by 33
The utility of the Valuing Questionnaire in Chronic Pain
Carvalho, S. A., Palmeira, L., Pinto-Gouveia, J., Gillanders, D., & Castilho, P. (2018) · Journal of Contextual Behavioral Science · doi:10.1016/j.jcbs.2018.06.002 · Cited by 30
Juncos, D. G., & de Paiva e Pona, E. (2018) · Music & Science · doi:10.1177/2059204317748807 · Cited by 25
“However, Öst’s reviews have since been shown to contain numerous factual and interpretive errors by both ACT researchers and independent scholars, leading many to conclude that they are largely biased against ACT (Atkins et al., 2017).”
BackgroundNagatsu, T. (2020) · Journal of Neural Transmission · doi:10.1007/s00702-019-02111-y · Cited by 8
Sandery, P., Baranoff, J., & Dorstyn, D. (2024) · Journal of Contextual Behavioral Science · doi:10.1016/j.jcbs.2024.100841 · Cited by 4
Metadata
- Authors
- Atkins, P. W. B., Ciarrochi, J., Gaudiano, B. A., Bricker, J. B., Donald, J., Rovner, G., Smout, M., Livheim, F., Lundgren, T., & Hayes, S. C.
- Venue
- Behaviour Research and Therapy
- Publisher
- Elsevier BV
- Published
- 2017-10
- Volume
- 97
- Pages
- 259-272
- ISSN
- 0005-7967 (ISSN-L)
- OpenAlex
- W2617641004
- Cited by
- 41
- References
- 74
- FWCI
- 1.58
- Type
- journal article
- Language
- en
- Open access
- green
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.