Öst's (2008) methodological comparison of clinical trials of acceptance and commitment therapy versus cognitive behavior therapy: Matching Apples with Oranges?

Probably not a substantial claim: may not pass S3 (load-bearing).

Prior reviews by Hayes and colleagues have noted similar methodological weaknesses in ACT research.

“Hayes and colleagues have described similar methodological weaknesses in their previous reviews of ACT research (Hayes et al., 2006; Hayes, Masuda, Bissett, Luoma, & Guerrero, 2004; Hayes, Strosahl, & Wilson, 1999).”

• cites Hayes et al., 2006 sourceSupported
• cites Hayes, Masuda, Bissett, Luoma, & Guerrero, 2004 sourceNot checked — no full text of the source
• cites Hayes, Strosahl, & Wilson, 1999Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Reanalysis found coding errors in Öst's grant funding data: one ACT study was incorrectly coded as funded, one CBT study was incorrectly listed as directly funded, and another CBT study was omitted but confirmed funded.

“One of these studies was incorrectly coded and not in actuality grant funded (i.e., Bach & Hayes, 2002). In addition, one of the CBT studies was incorrectly listed as being directly supported by a grant (i.e., Herbert et al., 2005), and another was not included as grant funded, but the primary author confirmed that it was (Barlow, Craske, Cerny, & Klosko, 1989).”

• cites Bach & Hayes, 2002 sourceNot checked — no full text of the source
• cites Herbert et al., 2005Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Barlow, Craske, Cerny, & Klosko, 1989 sourceNot checked — no full text of the source

May combine two claims.

The ACT study was labeled a pilot and was a preliminary investigation to support future larger studies.

“the ACT study was clearly labeled as a “pilot” study in the title, and was considered a preliminary investigation to support the need for future larger scale studies to verify initial positive findings.”

May combine two claims.

All CBT trials in Öst's matched sample studied emotional disorders, while most ACT trials studied more difficult-to-treat, often treatment-resistant populations.

“All of the purportedly “matched” CBT trials were studies of emotional disorders, whereas the majority of the ACT studies were conducted in more difficult-to-treat and often treatment-resistant populations.”

May combine two claims.

Öst's grant funding analysis used only the 13 matched CBT studies, resulting in low statistical power despite an apparent difference (77% CBT vs 38% ACT).

“Originally, Öst (2008) conducted a statistical analysis of grant funding in the 13 ACT studies compared with the 13 “matched” CBT study subsample only. This use of the subsample of CBT studies led to low statistical power in the analysis, even though the differences appeared meaningful (77% for CBT vs 38% for ACT).”

May combine two claims.

The ACT and CBT samples compared by Öst were mismatched in the clinical populations treated, which led to differences in study design and methodology.

“The samples were clearly mismatched in terms of the populations being treated, leading to differences in study design and methodology.”

May combine two claims.

Reanalysis of Öst's data revealed that CBT studies had significantly more grant support than ACT studies, a difference not reported in Öst's original article.

“reanalysis showed clear differences in grant support favoring CBT compared with ACT studies that were not reported in the original article.”

May combine two claims.

Only one of five grant-supported ACT studies had a large NIH grant, while many CBT studies appeared to have large grants such as R01s.

“Review of the study funding data showed that only 1 of the 5 grant-supported ACT studies was supported by a large grant from the National Institutes of Health, whereas many of the CBT studies appeared to be supported by large grants (e.g., R01s).”

May combine two claims.

In the full CBT sample, 80% (24) were grant funded versus 38% (5) of ACT studies, a significant difference by Fisher's exact test (p = .013).

“In the full CBT sample, 80% (n = 24) of studies were grant funded, compared with only 38% (n = 5) of studies in the ACT sample. A Fisher's exact test showed that significantly more CBT studies were grant funded compared with ACT studies (p = .013).”

May combine two claims.

The 13 ACT studies received a total of $1,448,570 (average $111,428 each), with only one of five funded studies exceeding $150,000; the 13 CBT studies received $6,438,157 (average $495,242 each), with 9 of 10 funded studies exceeding $150,000.

“A total of $1,448,570 went to the 13 ACT studies or an average of $111,428 each. Only one of the five funded studies exceeded $150,000. A total of $6,438,157 went to the 13 CBT studies or $495,242 each, and 9 of the 10 funded studies exceeded $150,000.”

May combine two claims.

The Gaudiano and Herbert trial's findings were consistent with an earlier independent trial and showed additional clinically significant symptom reductions.

“Our findings were consistent with those obtained from an earlier independent trial (Bach & Hayes, 2002), and extended these initial findings by showing additional benefits in terms of clinically significant symptom reductions.”

• cites Bach & Hayes, 2002 sourceNot checked — no full text of the source

Classified as probably not a substantial claim. May combine two claims.

Levin and Hayes reanalyzed the Powers et al. data, corrected problems such as mislabeled outcomes, and found ACT significantly more efficacious than established treatment comparisons (ES = .27-.49).

“Levin and Hayes (in press) had the opportunity to reanalyze the Powers et al. data, and after correcting a number of problems (e.g., process or secondary variables mislabeled as primary outcomes) reported that that ACT was significantly more efficacious than established treatment comparisons in these clinical trials (ES = .27–.49, depending on the studies included as using “established treatments””

• cites Levin and Hayes (in press)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

A subsequent independent meta-analysis of 18 ACT trials found ACT significantly better than waiting lists/placebos (ES = .68) and treatment as usual (ES = .42), but not more efficacious than established treatments (ES = .18).

“Powers, Zum Vörde Sive Vörding, and Emmelkamp (2009) meta-analyzed 18 clinical trials of ACT and reported that ACT was significantly better than waiting lists/psychological placebos (ES = .68) and treatment as usual (ES = .42), but not more efficacious than established treatments (ES = .18).”

• cites Powers, Zum Vörde Sive Vörding, and Emmelkamp (2009) sourceNot checked — no full text of the source

May combine two claims.

After Öst's paper, APA Division 12 listed ACT for depression as having moderately strong empirical support, based on meeting criteria for a probably efficacious treatment.

“ACT for depression is now listed as having “moderately strong” empirical support.4 This is based on the evaluation that ACT meets the criteria for a “probably efficacious” treatment for depression”

• cites [4] Borkovec, 2005, Problems with the use of placebo conditions in psychotherapy research, suggested alternatives, and some strategies for the pursuit of the placebo phenomenon, Journal of Clinical Psychology sourceNot checked — no full text of the source

May combine two claims.

Öst matched an ACT trial for treatment-resistant epilepsy with a CBT social phobia trial, a comparison that makes little sense.

“Öst matched the ACT epilepsy trial with a CBT social phobia trial (Clark et al., 2006). Such comparisons make little sense.”

• cites Lundgren, Dahl, Melin, & Kies, 2006 sourceNot checked — no full text of the source
• cites Clark et al., 2006 sourceNot checked — no full text of the source

May combine two claims.

Öst's matching criterion overlooked that ACT and CBT were at different stages of clinical trial testing, and ACT studies historically had less funding.

“This sole criterion ignored the fact that ACT and CBT interventions are at very different stages of clinical trial testing, and thus ACT studies have historically had less funding to support this early research.”

• AssumptionRests on an assumption: ACT and CBT were at different stages of clinical trial testing at the time of Öst's review.

May combine two claims.

Öst matched the authors' ACT psychosis study with a CBT study for acute stress disorder, which would require a very different design.

“Öst “matched” our ACT study with a CBT study for acute stress disorder (Bryant et al., 2006), which would require a very different type of study design.”

• cites Bryant et al., 2006 sourceNot checked — no full text of the source

May combine two claims.

Öst's review noted weaknesses in some methodological areas but ignored potential strengths in early ACT versus CBT research.

“Thus, Öst's review pointed out weaknesses in some methodological areas, while ignoring the potential strengths in early ACT versus CBT research.”

Probably not a substantial claim: may not pass S2 (checkable).

The appropriateness of using DSM criteria to define a sample should be based on the investigation's aims, not on whether ACT researchers use the DSM.

“the appropriateness of the use of the DSM for defining a sample should to be based on the actual aims of the investigation.”

Classified as probably not a substantial claim.

Öst's statement implies that a study is valid only if it investigates a DSM diagnostic group.

“Öst's statement implies that a study is not valid unless it investigates a diagnostic group included in the current version of the DSM.”

May combine two claims.

Öst's matching method assumed that ACT and CBT studies published in the same journal and period would have similar methodologies and could be judged by identical standards, an assumption the authors consider questionable.

“Öst's methods relied on the rather questionable assumption that ACT studies published around the same time period in the same journal as CBT studies should have similar methodologies and could be judged by the exact same standards.”

May combine two claims.

Methodological evaluations are context-dependent, and Öst's matched CBT comparison using a summary quality index should have used justifiable criteria.

“methodological evaluations do not occur in a vacuum as Öst's analysis assumed, and attempts to create a matched CBT sample for comparison using a summary index of methodological quality should have been based on justifiable criteria.”

May combine two claims.

Öst's comparison drew sweeping conclusions about methodology while ignoring context.

“Öst's (2008) comparison of CBT and ACT studies attempted to draw sweeping conclusions about study methodology while ignoring context.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

It is unclear why Öst focused only on horse-race trial results comparing two treatments on outcome.

“it is unclear why Öst (2008) chose to focus his attention only on “horse race” trial results, in which two treatments are pitted against one another to look for differences principally in outcome.”

Classified as probably not a substantial claim. May combine two claims.

Many researchers advocate focusing on mechanisms of action in effective treatments because most psychotherapies share components and often produce similar outcomes.

“Given that most psychotherapies share components and frequently have been found to produce similar outcomes, many researchers have begun to advocate that more attention be paid to the actual mechanisms of action in effective treatments (Borkovec & Sibrava, 2005; Herbert & Gaudiano, 2005; Kazdin, 2008; Lohr, DeMaio, & McGlynn, 2003; Rosen & Davison, 2003).”

• cites Borkovec & Sibrava, 2005 sourceNot checked — no full text of the source
• cites Herbert & Gaudiano, 2005Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Kazdin, 2008 sourceNot checked — no full text of the source

Classified as probably not a substantial claim.

Öst highlighted important limitations in published ACT trials that warrant careful consideration.

“Öst (2008) highlighted some important limitations in the currently published clinical trials of ACT that must be carefully considered.”

May combine two claims.

Because the samples were mismatched, Öst's methodological ratings are hard to interpret and add little beyond a qualitative review of ACT limitations.

“Given the actual mismatch between the samples, Öst's methodological ratings are difficult to interpret and provide little useful information beyond what could already be gathered by a qualitative review of ACT study limitations.”

Classified as probably not a substantial claim.

For missing funding data, conservative estimates were used based on similar grants or a portion of a known total.

“In the two cases where data were unavailable, a conservative estimate was used based on amounts of similar size grants; in a third case with multiple projects totaling a known amount, a conservative portion for the published study was estimated.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Comparing early ACT and early CBT studies could have its own problems because they were conducted in different eras.

“Of course, this could have caused its own problems as these studies were conducted in different eras.”

Classified as probably not a substantial claim.

The author independently verified grant funding status for the ACT and CBT samples.

“I independently verified grant funding status for the ACT and CBT study samples.”

Probably not a substantial claim: may not pass S4 (based).

ACT and CBT studies funded by similar grant mechanisms, such as NIH R01 grants, are likely to have similar methodological features.

“In the end, ACT studies and CBT studies funded by similar grant mechanisms (e.g., National Institutes of Health R01 grants) are likely to have similar methodological features.”

May combine two claims.

Early ACT studies have methodological limitations that require independent replication with larger samples to confirm.

“Early studies of ACT have methodological limitations that will require independent replication with increasingly larger samples to confirm.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A potentially fairer approach, which Öst rejected, would compare early ACT studies with early CBT studies.

“One potentially fairer approach that Öst rejected was to compare early ACT studies with early CBT studies.”

May combine two claims.

Psychotherapy research has advanced over the years, with increased emphasis on mechanisms of action, which early ACT studies investigated but CBT studies largely did not.

“There have been many advances in psychotherapy research over the years, including an increased emphasis on mechanisms of action, investigations of which have been present even in early ACT studies, but largely missing from CBT studies.”

Classified as probably not a substantial claim.

Funding amount estimates were obtained from public data or directly from principal investigators.

“I obtained estimates of the amount of funding for these studies either from publicly available data or directly from the principal investigators.”