The evidence for CBT in any condition, population or context... A meta-review... and panoramic meta-analysis, 2021, Fordham et al.

[Kalliope]

The evidence for cognitive behavioural therapy in any condition, population or context: a meta-review of systematic review and panoramic meta-analysis by Fordham et al.

Abstract
The majority of psychological treatment research is dedicated to investigating the effectiveness of cognitive behavioural therapy (CBT) across different conditions, population and contexts. We aimed to summarise the current systematic review evidence and evaluate the consistency of CBT's effect across different conditions.

We included reviews of CBT randomised controlled trials in any: population, condition, format, context, with any type of comparator and published in English. We searched DARE, Cochrane, MEDLINE, EMBASE, PsycINFO, CINAHL, CDAS, and OpenGrey between 1992 and January 2019. Reviews were quality assessed, their data extracted and summarised.

The effects upon health-related quality of life (HRQoL) were pooled, within-condition groups. If the across-condition heterogeneity was I2 < 75%, we pooled effects using a random-effect panoramic meta-analysis. We summarised 494 reviews (221 128 participants), representing 14/20 physical and 13/20 mental conditions (World Health Organisation's International Classification of Diseases). Most reviews were lower-quality (351/494), investigated face-to-face CBT (397/494), and in adults (378/494). Few reviews included trials conducted in Asia, South America or Africa (45/494).

CBT produced a modest benefit across-conditions on HRQoL (standardised mean difference 0.23; 95% confidence intervals 0.14–0.33, I2 = 32%). The effect's associated prediction interval −0.05 to 0.50 suggested CBT will remain effective in conditions for which we do not currently have available evidence. While there remain some gaps in the completeness of the evidence base, we need to recognise the consistent evidence for the general benefit which CBT offers.

 

[Kalliope]

 

[Hutan]

You would think, given the extremely high rate of low quality reviews, that there might have been some comment about what was wrong with the reviews, or some comment about how future reviews might be done. But, no.

There was nothing at all said about the reliance on subjective outcomes with interventions that are extremely hard to blind in the trials.
But this is telling:
When CBT is compared to something like a wait list, the benefit looks quite good - SMD 0.31
When CBT is compared to anything else that might include a sympathetic therapist, an expectation of benefit, and/or group support, the benefit looks much smaller - SMD 0.09

In the HRQoL analyses, we found a statistically significant interaction between reviews which compared CBT to an active as opposed to a non-active comparator (p = 0.04). The effect size was larger in the non-active comparator subgroup SMD 0.31 (95% CI 0.18–0.45) than the effect comparing CBT to an active comparator SMD 0.09 (95% CI −0.01 to 0.19). The active comparator interventions were psychoeducation, relaxation, psychotherapy, counselling and physical exercise.

It doesn't seem to matter what sort of CBT is done - face to face sessions or online DIY courses. It all miraculously produces the same effect in the aggregate.

Our CBT intensity subgroup analyses suggested no difference in effectiveness between using high- and low-intensity CBT.

The only mention of adherence and dropouts is when they are talking about doing more research to increase the benefit of CBT. There's no discussion about how high rates of dropouts might invalidate results.

For example, identifying alternative delivery formats to increase adherence and reduce dropout, and pursuing novel methods to assess intervention fidelity and quality.

There was no comment about harms, no mention of the possibility that the very modest reported benefit for those participants who stuck with the treatment long enough to fill out the questionnaires might be more than outweighed by the harm done to those people who dropped out, scarred by the experience of being told that their way of looking at the world is flawed, or feeling like failures because they could not think their way out of their condition.

 

[Jonathan Edwards]

A pity I had not been able to see this before answering questions at NICE committee - i.e. did I think CBT trials were just as rubbish in other conditions (shock horror).

The fact that this review is unable to identify good quality evidence in anything and pick it out and has to bundle everything into a 'works a bit although data are not that good' conclusion makes me pretty confident that CBT is a scam from start to finish. Yes, talking to helpful people may be helpful but CBT as a concept looks to be a complete scam.

 

[cassava7]

As our extraction and analysis were conducted at the review level, we could not account for the risk of bias in the RCTs included within the reviews.

But strangely they do not mention risk of bias when they discuss the limitation of assessing RCTs at the review level:

A limitation of our methodology was the extraction of data only at the review level. This meant that we excluded many reviews which included relevant RCTs, but had combined these with trials of other interventions. If we had extended our methods to extract data from individual RCTs which had been identified by reviews this may have been a more comprehensive picture of the CBT evidence base. Another weakness was the exclusion of reviews published in languages other than English (237 reviews). These reviews may not have met all the inclusion criteria but if they did, they might have addressed the evidence gap we identified of few trials having been conducted in Africa, Asia or South America.

This is unfortunate. According to the article, the systematic reviews of RCTs for CBT that met the eligibility criteria were rated using AMSTAR 2 (tool, guidance). The published article on AMSTAR 2 [1] suggests classifying the quality of a review as follows:

Box 2 Rating overall confidence in the results of the review
High

No or one non-critical weakness: the systematic review provides an accurate and comprehensive summary of the results of the available studies that address the question of interest

Moderate

More than one non-critical weakness*: the systematic review has more than one weakness but no critical flaws. It may provide an accurate summary of the results of the available studies that were included in the review

Low

One critical flaw with or without non-critical weaknesses: the review has a critical flaw and may not provide an accurate and comprehensive summary of the available studies that address the question of interest

Critically low

More than one critical flaw with or without non-critical weaknesses: the review has more than one critical flaw and should not be relied on to provide an accurate and comprehensive summary of the available studies

*Multiple non-critical weaknesses may diminish confidence in the review and it may be appropriate to move the overall appraisal down from moderate to low confidence

With critical domains being those:

Box 1 AMSTAR 2 critical domains

• Protocol registered before commencement of the review (item 2)
  
  
• Adequacy of the literature search (item 4)
  
  
• Justification for excluding individual studies (item 7)
  
  
• Risk of bias from individual studies being included in the review (item 9)
  
  
• Appropriateness of meta-analytical methods (item 11)
  
  
• Consideration of risk of bias when interpreting the results of the review (item 13)
  
  
• Assessment of presence and likely impact of publication bias (item 15)

Hopefully all of the 143 reviews classified as "higher" quality (AMSTAR 2 moderate or high quality) did employ a tool for analyzing risk of bias from individual studies, and correctly so, as they should per the above. Some, but I suspect many, of the 351 "lower" quality reviews (AMSTAR 2 low and critically low quality) may be suffering from high risk of bias and more issues, at which point one can ask whether they can legitimately be included. As an example, a French meta-analysis of studies (not reviews of RCTs) on hydroxychloroquine for COVID-19 excluded those with a critical risk of bias according to the Cochrane RoB2 tool. [2] Here, unfortunately, the individual AMSTAR 2 ratings are not given, even in the supplementary material, so we can't know more about the critical flaws in the included reviews.

As @Hutan pointed out, biases around CBT are not discussed in the article and the non-superiority to other interventions (active comparators) is not addressed. There is no mention of spin, even though it has been shown to be a problem in psychotherapy trials whether they are registered prospectively or not [3]. Overall, the discussion doesn't show critical thinking with respect to CBT...

Taking these issues together, this meta-review arguably spins its findings:

While there remain some gaps in the completeness of the evidence base, we need to recognise the consistent evidence for the general benefit which CBT offers.

There are more than just "some gaps" in the evidence base for CBT: the low quality evidence does not support the authors' conclusions. If we consider the AMSTAR 2 critical domains, this review scores low on "Consideration of risk of bias when interpreting the results of the review" and could be considered as low quality.

[1] Shea BJ, Reeves BC, Wells G, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:j4008. Published 2017 Sep 21. doi:10.1136/bmj.j4008

[2] Fiolet T, Guihur A, Rebeaud ME, Mulot M, Peiffer-Smadja N, Mahamat-Saleh Y. Effect of hydroxychloroquine with or without azithromycin on the mortality of coronavirus disease 2019 (COVID-19) patients: a systematic review and meta-analysis. Clin Microbiol Infect. 2021;27(1):19-27. doi:10.1016/j.cmi.2020.08.022

[3] Stoll M, Mancini A, Hubenschmid L, et al. Discrepancies from registered protocols and spin occurred frequently in randomized psychotherapy trials-A meta-epidemiologic study. J Clin Epidemiol. 2020;128:49-56. doi:10.1016/j.jclinepi.2020.08.013

 

[Kalliope]

 

[Sly Saint]

we need to recognise the consistent evidence for the general benefit which CBT offers.

hmm what is happening in practice (IAPT) would suggest otherwise

"No trial of CBT has ever been conducted with the average dosage delivered in IAPT."

https://www.s4me.info/threads/uk-im...ies-iapt-articles-blogs-and-discussion.14318/

 

[Caroline Struthers]

A pity I had not been able to see this before answering questions at NICE committee - i.e. did I think CBT trials were just as rubbish in other conditions (shock horror).

The fact that this review is unable to identify good quality evidence in anything and pick it out and has to bundle everything into a 'works a bit although data are not that good' conclusion makes me pretty confident that CBT is a scam from start to finish. Yes, talking to helpful people may be helpful but CBT as a concept looks to be a complete scam.

I might write to Edzard Ernst and ask him why he doesn't add it to his SCAM (so-called alternative medicines) list as the evidence for CBT is just as bad as for any of the treatments he bangs on about. He has ignored my hints to that effect so far...and I think claims lack of expertise in non-SCAM (ie. mainstream medicine) means he can't possibly comment. Not sure. It's annoying.

 

[Jonathan Edwards]

Yes, it's a good thing that there are all these things to get annoyed about. Otherwise lockdown would get pretty dull.

 

[Caroline Struthers]

Yes, it's a good thing that there are all these things to get annoyed about. Otherwise lockdown would get pretty dull.

And another thing....what the hell is a "panoramic meta-analysis"?

 

[Jonathan Edwards]

And another thing....what the hell is a "panoramic meta-analysis"?

It's what greater adjutant storks do on the rubbish tip at Guwahati.

 

[ME/CFS Skeptic]

Rather than providing evidence to support the effectiveness of CBT, I think the authors have accidentally done just the opposite.

 

[Hoopoe]

For some patients, a treatment that does nothing might very well be far better than the other treatments.

Maybe even better would be to stop expecting patients who have health problems that nobody understands to do everything to recover.

 

[rvallee]

Confirms what has been obvious since... forever: that CBT is an instrument that can be used to slightly change answers on questionnaires, with greater "efficacy" on changing answers by crafting deliberately vague, non-specific questions and the more rigorous methodology is, the smaller the effect, all the way down to indistinguishable from the natural noise one gets when using subjective ratings on questionnaires that are deliberately ambiguous. After all, how precise and accurate is a 7/10 rating on a tomato sauce? Very open to manipulation by underheating it, microwaving it at max temp for too long or adding too much salt or simply not adding any. There is no right answer, so no way to ever actually confirm anything, make-believe built on more make-believe.

In addition to doing obvious manipulation of results by arbitrarily using larger numbers to give the illusion of a great effect, an example of which is the touting of the 50-point increase on the IBS scale for the CBT for IBS study, never hinting at the fact that on a 500 point scale it means a less than 10% impact, something obviously not significant that was still touted as such.

It's basically institutionally-backed snake oil, built on quantity of evidence with no concern for quality, deliberate attempts to use bias in a way to amplify false effects and a general state of denial that this is all obviously an illusion caused by the imprecision that comes from not using valid measurements.

Fitting that while this review obviously destroys all pretense that any of this is legitimate, of course people like Howard actually pretend it says the opposite. Escalation of commitment is unforgiving, building up an obvious lie means the lying has to keep on lying. And of course those are the same people, like Greenhalgh and Sharpe, who routinely write and tweet about not falling for exactly this bias, which they selectively support or not depending entirely on their own perception.

This is anti-science, the textbook opposite of science. It will eventually be recognized as such, which makes it all that much more ridiculous.

 

[Hutan]

Rather than providing evidence to support the effectiveness of CBT, I think the authors have accidentally done just the opposite.

Actually I could be quite easily persuaded to believe that CBT does have some uses - in conditions where the world might in fact be being viewed unhelpfully e.g. aggression, anxiety disorders and phobias. And, in Figure 2 of this meta-review, reviews of studies of anxiety disorders do appear to report slightly more positive outcomes than the rest. Phobias aren't mentioned and CBT was reported to have no impact on aggression.

Yes, it's quite remarkable that, even with all of the biases and poor quality, the benefits for virtually all of the conditions appear so trivial.

 

[Hoopoe]

Yes, it's quite remarkable that, even with all of the biases and poor quality, the benefits for virtually all of the conditions appear so trivial.

In a way, these studies test certain assumptions common in CBT, for example that the problems of the patient are due to faulty beliefs. Maybe beliefs have simply little to do with whatever problems patients have.

 

[Jonathan Edwards]

Maybe beliefs have simply little to do with whatever problems patients have.

That's a very good point. I can think of lots of people with faulty beliefs who seem to do pretty well.

Luciano Pavarotti had a faulty belief that he was not a very good singer, for instance.

 

[Hoopoe]

Maybe he became a good singer because he often felt inadequate.

 

[shak8]

I have a close relative with anxiety who has attended CBT group therapies (which is the first line of treatment offered by a major US health plan) and she has completed workbooks in CBT and has also diligently attended groups in one of CBT's contemporary offshoots.

She has spent several years doing this, and her anxiety levels continue to be high (on onset of several years past now).

If CBT were effective on anxiety, I'm sure I would know. The phone calls would diminish and her decision-making would cease to be a world crisis.

 

[DigitalDrifter]

I might write to Edzard Ernst and ask him why he doesn't add it to his SCAM (so-called alternative medicines) list as the evidence for CBT is just as bad as for any of the treatments he bangs on about.

It annoys me that skeptics like https://sciencebasedmedicine.org/make lists of fake illness / fake diagnosis yet go on to claim illnesses like ME are somatization or psychosomatic. It's as if as soon as psychogenic attribution rears it's ugly head, science and skepticism go out the window leaving different standards to be applied.

How is somatizing any less fake than Chronic Lyme or Electromagnetic Hypersensitivity? Where is the evidence for it? Why do they never weigh up the pros and cons of psychogenic attribution?