Independent advisory group for the full update of the Cochrane review on exercise therapy and ME/CFS (2020), led by Hilda Bastian

I'm thinking of replying to my "friend" (in NICE) Katy's email along these lines*; grateful for your views.

*"Katy,
Thank you for your email.

Re your comment that "The wording of the enquiry suggests that the Cochrane review mentioned is of psychological interventions.". My understanding is that, in this case, exercise is being used as a psychological intervention: "The 'deconditioning model' assumes that the syndrome is perpetuated by reversible physiological changes of deconditioning and avoidance of activity, and that therefore physical activity (exercise) should reduce deconditioning and facilitate recovery".

In my email I stated that:
"NICE's use of Cochrane reviews also creates a risk for those with Long covid, and Lyme disease, i.e. since "low and very low quality" evidence will be considered "moderate" quality - suitable evidence to support the use of psychological interventions like CBT."

Cochrane uses GRADE and my email should have stated that the use of GRADE by either Cochrane, or NICE, creates a risk that unreliable evidence will be used to recommend the use of treatments which are, in practice, unevidenced. The study you refer to illustrates the issue [https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD003200.pub8/full]. The study states that "Exercise therapy probably reduces fatigue at end of treatment --- moderate‐certainty evidence---".

While the refers to "randomised controlled trials" in practice the studies referred to are unblinded and use subjective outcome criteria i.e. unreliable evidence. One of the studies was supposed to include objective outcome criteria (actimetry) i.e. White 2011 [https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(11)60096-2/fulltext - PACE trial]. However, actimetry was, in practice, only used for the baseline data i.e. not the post intervention assessment.

Regarding the reliability of subjective outcome criteria versus objective outcome criteria, in assessing physical activity, this study, and others, demonstrates that subjective outcome criteria are unreliable [https://pubmed.ncbi.nlm.nih.gov/30670036/].

Another issue, which compounds the problem of using GRADE, is that once NICE has adopted a guideline then it is very difficult for the patient community to challenge it i.e. even if it is based on unreliable GRADE assessments. In theory the guideline can be challenged by judicial review; however, the very high bar in proving that the guideline is "unreasonable" means that this is unlikely to succeed.

Regards and happy to discuss further,
----"

 

More information about the agreement between cochrane and NICE can be found here: https://www.cochrane.org/news/interview-cochrane-and-nice-collaborate-improve-health-guidelines

 

"Cochranes systematic reviews have a well-established reputation for producing high quality reviews of evidence"
Who wrote this? A basic grasp of grammar is not much to ask from Cochrane is it?

I will write to Toby Lasserson tomorrow, on the third anniversary of my first complaint to Cochrane about the Exercise review (https://healthycontrol.org/2018/11/...iew-of-exercise-for-chronic-fatigue-syndrome/). I will ask him to reconsider Cochrane's decision not to withdraw the two reviews on CFS. They contradict the NICE review findings and the now published recommendations, and could cause further confusion...to put it diplomatically.

They also both state they are used in the NICE Clinical Knowledge Summary for Tiredness and Fatigue (https://cks.nice.org.uk/topics/tiredness-fatigue-in-adults/). However, their collaborators NICE have just removed them from the list of references. The list of "related reviews and protocols" on these two reviews is also highly dubious.

https://www.cochranelibrary.com/cds...CD003200.pub8/related-content#guidelines_data
https://www.cochranelibrary.com/cds...CD001027.pub2/related-content#guidelines_data

It would be so much simpler to withdraw the reviews. But then I would have to stop writing to them, and what would I do with my time?

 

carolines kitchen disco?

 

Regarding the problem; I think it's summed up by this exchange*:
@strategist "From what I understood with GRADE such studies would generally count as valid evidence. With the NICE guidelines the studies reached the appropriate low grade only when other factors were also taken into consideration.

GRADE seems to be a system that allows even homeopathy to reach the status of evidence based treatment for almost any condition (if sufficiently subjective endpoints are chosen)."
@Jonathan Edwards "Indeed, @strategist, it is no more complicated than that.
The last thing we want is an argument about who is best at doing GRADE right."

So that's the problem and I need to have another look at my reply to see if I can reflect that.

In terms of NICE; they're responsible for producing guidelines (I know that's obvious) and NICE can use GRADE (directly or indirectly via Cochrane) i.e. provided that any inappropriate evidence is removed - unblinded studies which use subjective outcome indicators ++++. Strategist points out that the NICE guidelines the studies reached the appropriate low grade only when other factors were also taken into consideration." - worrying.
I suppose one purpose of the email is to highlight that there is an awareness of the problem, i.e. of using GRADE, and that NICE should, at the very least, vet the GRADE assessments and ensure that they are defensible.

Luckily (NICE) Katy, in her email/reply, has provided a reference to a Cochrane review which refers to "moderate" quality evidence but is based on the usual crap i.e. an unblinded study with subjective outcome criteria**. So I might just weave that into the reply.

I'd be grateful for a link to @Jonathan Edwards expert testimony.

I'm not actually sure I want to try to understand GRADE, or that I could if I tried; however the scam seems pretty basic - flawed science (unblinded studies with subjective outcomes) labelled as "moderate" quality evidence. I think @Caroline Struthers highlighted that there is a further layer of complexity - i.e. a complicated system of (mathematical?) analysis used to produce systemic reviews - I'm not planning on trying to understand that.

One possible benefit of challenging GRADE is that public bodies like MRC & NIHR might be encouraged to fund studies which provide good quality evidence - a +ve for scientists wishing to do sound research.

I felt that mentioning judicial review was a bit tangential; however, you can have a system that makes lots of poor quality decisions and those decisions are rectified by an appeal system. I find the UK system frustrating because it often relies on judicial review (JR) and JR doesn't look at the facts and make the decision afresh - JR asks the question is this decision so unreasonable that no reasonable person could have arrived at that decision --- / the Courts don't like to interfere with the lawful authority of NICE to make guidelines. So, as with the 2007 guideline, you're stuck with a flawed guideline.

I'll try to have another look at it - thanks you .


*
https://www.s4me.info/threads/indep...ed-by-hilda-bastian.13645/page-80#post-390311

**
"Exercise therapy compared with 'passive' control

Exercise therapy probably reduces fatigue at end of treatment (SMD −0.66, 95% CI −1.01 to −0.31; 7 studies, 840 participants; moderate‐certainty evidence; re‐expressed MD −3.4, 95% CI −5.3 to −1.6; scale 0 to 33). We are uncertain if fatigue is reduced in the long term because the certainty of the evidence is very low (SMD −0.62, 95 % CI −1.32 to 0.07; 4 studies, 670 participants; re‐expressed MD −3.2, 95% CI −6.9 to 0.4; scale 0 to 33).

We are uncertain about the risk of serious adverse reactions because the certainty of the evidence is very low (RR 0.99, 95% CI 0.14 to 6.97; 1 study, 319 participants).

Exercise therapy may moderately improve physical functioning at end of treatment, but the long‐term effect is uncertain because the certainty of the evidence is very low. Exercise therapy may also slightly improve sleep at end of treatment and at long term. The effect of exercise therapy on pain, quality of life and depression is uncertain because evidence is missing or of very low certainty."
https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD003200.pub8/full

 

Here's a revised draft email to NICE*; I've mentioned @Jonathan Edwards expert testimony. I meant to send this today but I didn't get around to it; no point sending it now since they won't be reading emails over the weekend.

*"Katy,
Thank you for your email and apologies for my delay in replying.

Re your comment that "The wording of the enquiry suggests that the Cochrane review mentioned is of psychological interventions.". My understanding is that, the use of exercise therapy in ME/CFS is based on the concept that it is a psychological intervention; the following extract may help to illustrate: "The 'deconditioning model' assumes that the syndrome is perpetuated by reversible physiological changes of deconditioning and avoidance of activity, and that therefore physical activity (exercise) should reduce deconditioning and facilitate recovery".

In my email I stated that:
"NICE's use of Cochrane reviews also creates a risk for those with Long covid, and Lyme disease, i.e. since "low and very low quality" evidence will be considered "moderate" quality - suitable evidence to support the use of psychological interventions like CBT."

Cochrane uses GRADE and my email should have stated that "the use of GRADE by either NICE, or Cochrane (NICE uses Cochrane reviews) creates a risk that unreliable evidence will be used to recommend treatments like CBT and exercise". The Cochrane study you refer to illustrates the issue [Note 1]; it states that "Exercise therapy probably reduces fatigue at end of treatment --- moderate‐certainty evidence---". However, the studies evaluated in this Cochrane study are unblinded and use subjective outcome criteria i.e. they are unreliable as an evidence base for treatment recommendations. One of the studies was supposed to include objective outcome criteria (actimetry) i.e. White 2011 - PACE trial [Note 2]. However, actimetry was, in practice, only used to collect baseline data i.e. the post intervention assessment relied on subjective outcomes.

Studies of the use of subjective outcome criteria, versus objective outcome criteria, in assessing physical activity, have shown that subjective outcome criteria are unreliable [Note 3].

In his expert witness testimony to NICE Professor Jonathan Edwards set out the issues with the available evidence base re CBT & exercise [Note 4].

Another issue, which compounds the problem of using GRADE, is that once NICE has adopted a guideline then it is very difficult for the patient community to challenge it i.e. even if the evidence base is unreliable. In theory a guideline can be challenged by judicial review (JR); however, the very high bar in proving that a guideline is "unreasonable" means that JR is unlikely to succeed.

Finally I would like to thank the NICE Committee which produced the recently published ME/CFS guideline and indeed others in NICE who contributed.

Regards and happy to discuss further,
---

Note 1 - https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD003200.pub8/full
Note 2 - [https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(11)60096-2/fulltext
Note 3 - e.g. [https://pubmed.ncbi.nlm.nih.gov/30670036/].
Note 4- [https://www.nice.org.uk/guidance/ng206/documents/supporting-documentation-3].

 

I've tweaked it a bit* ---- probably going to get a reply along the lines that NICE wouldn't base guidelines on unreliable evidence i.e. they would vet Cochrane reviews and remove any of poor quality. OK, if that's what they do then the patient community will be reassured.

*Katy,
Thank you for your email and apologies for my delay in replying.

Re your comment that "The wording of the enquiry suggests that the Cochrane review mentioned is of psychological interventions.". Yes, i.e. the deconditioning and exercise intolerance (psychological) theories of ME/CFS.

In my email I stated that:
"NICE's use of Cochrane reviews also creates a risk for those with Long covid, and Lyme disease, i.e. since "low and very low quality" evidence will be considered "moderate" quality - suitable evidence to support the use of psychological interventions like CBT."

Cochrane uses GRADE and my email should have stated that "the use of GRADE either directly by NICE, or indirectly through the use of Cochrane, creates a risk that unreliable evidence will be used to recommend treatments like CBT and exercise". The Cochrane review you refer to illustrates the issue [Note 1]; it states that "Exercise therapy probably reduces fatigue at end of treatment --- moderate‐certainty evidence---" --"The estimate remains consistent with a non‐zero effect size". Firstly, the studies evaluated in this Cochrane/GRADE review are unblinded and use subjective outcome criteria i.e. they are unreliable as an evidence base for treatment recommendations. One of the studies was supposed to include objective outcome criteria (actimetry) i.e. White 2011 - PACE trial [Note 2]. However, actimetry was, in practice, only used to collect baseline data i.e. the post intervention assessment relied on subjective outcomes. In addition this Cochrane/GRADE review has no clinical relevance since it examined whether there was a "non‐zero effect" from exercise therapy - not whether there was a clinically meaningful effect.

Regarding the use of subjective outcome criteria, versus objective outcome criteria, in assessing physical activity, studies have shown that subjective outcome criteria are unreliable [Note 3].

In his expert witness testimony to NICE Professor Jonathan Edwards set out the issues with the available evidence base re CBT & exercise [Note 4].

Another issue, which compounds the problem of using GRADE, is that once NICE has adopted a guideline then it is very difficult for the patient community to challenge it i.e. even if the evidence base is unreliable. In theory a guideline can be challenged by judicial review (JR); however, the very high bar in proving that a guideline is "unreasonable" means that JR is unlikely to succeed.

Finally I would like to thank the NICE Committee which produced the recently published ME/CFS guideline and indeed others in NICE who contributed to the delivery of this guideline.

Regards and happy to discuss further,
----

Note 1 - https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD003200.pub8/full
Note 2 - [https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(11)60096-2/fulltext
Note 3 - e.g. [https://pubmed.ncbi.nlm.nih.gov/30670036/].
Note 4- [https://www.nice.org.uk/guidance/ng206/documents/supporting-documentation-3].

 

Sent at last!

 

Really well done!!

I am considering writing to Cochrane (cc NICE) about the meaninglessness of the rating of moderate quality evidence for a non-zero effect. And how this has been unfairly used by critics to say that NICE's application of GRADE to judge the evidence quality as low or very low is "peverse", including Cochrane's own Paul Garner. I am struggling to formulate the letter though...

 

Does anyone have any updates on the Cochrane review on exercise therapy for ME/CFS? It was mentioned quite a few times in stakeholders' comments who wanted NICE to change its draft guideline.

 

Thank you.

I think it would be very good to cc NICE - I should have cc Cochrane but I didn't think of that. I did copy to the APPGs and ME Action & Action for ME ----.

Andrew Dilnot gave a talk in Belfast a few years ago and he mentioned that you cannot ascribe a value/cost to some things --- like a delay in fixing a broken leg or whatever. He used the description provided by his 5 year old daughter --- a pile of poo. So, yes it is difficult to capture the value of a subjectively measured non-zero effect in an unblinded study of a psychological intervention.

A long time ago I worked in analytical chemistry laboratories and there was the signal to noise ratio and lowest detectable amount ----. I can see a point in those relatively well controlled conditions.

I wonder if there's someone on this forum who knows about things like the Hawthorne effect* I think it explains the small +ve effect you find in these type of studies. The fact that it's such a small ("non-zero") +ve effect is pretty damming --- after you've coached the audience ----. I think @Jonathan Edwards ** was suggesting that some folks may be less likely to be influenced by the coaching --- maybe they should build that into the selection process --- boost the scores and all that.

You've previously set out ideas re how to improve Cochrane; they reminded me of the process of developing policy/legislation. White paper (initial draft proposals) which is consulted on --- so the initial Cochrane review would be consulted on; revised draft ----. Really good ideas but they'd require a willing Cochrane --- funny how those who go on about evidence don't seem to want to submit it to scrutiny.

*https://en.wikipedia.org/wiki/Hawthorne_effect
**https://www.s4me.info/threads/uk-ni...lication-discussion.22996/page-31#post-392910

 

Do you mean the old one, or the new one? I am planning to write this week to Toby Lasserson and ask Cochrane to remove the statement on the review saying it is used in the NICE Clinical Knowledge Summary for tiredness, because NICE have confirmed they have (finally) removed both the Exercise and CBT reviews from the references.

 

Right, but of course this didn't work in the case of the Health Research Authority report on the PACE trial because their purview appears to be limited. They couldn't fault PACE because they decided all the T's were crossed and i's dotted as per the standard process (even though this wasn't really the case), but they had no authority to judge whether any of the changes made sense.

 

The new one.

It has been two years since Cochrane made the statement that it is "committing to the production of a full update of this Cochrane Review". That was in October 2019.
https://www.cochrane.org/news/appointment-lead-independent-advisory-group

 

What do you mean by this?

 

One possibility is that Cochrane is the new battleground and lots is happening but none that we will hear about. Clearly far easier to influence than NICE. It would be nice to know what's happening behind the scenes, but I assume a lot is. Especially with Paul "Scuba" Garner on board.

Both are likely happening at the same time, with vastly different resources. I frankly assume this is what will happen, probably a competing review with aligned publications. NICE is an official body, there are limits to what they can do to influence them. Far easier to sidestep NICE entirely by simply working to keep Cochrane in their pockets.

I expect nothing out of Cochrane. Nothing at all.

 

As time has gone on and I've learnt a little bit more about some of the people on the author team and the IAG (and thought about some of the other options) I actually think they're better than I expected, and they could well be some of the best possible people that could have been chosen for this (none particularly ideological, some of them maintain interests and general skepticism about methods and the ways things are done in evidence based medicine). So I'm relatively optimistic. What they do still needs to be within the limits of the cochrane handbook and GRADE of course