Independent advisory group for the full update of the Cochrane review on exercise therapy and ME/CFS (2020), led by Hilda Bastian

[rvallee]

How things work is that patients and their supporters have huge power to change things and have changed things. Maybe in reality the patients have won out with this re-review. The people at Cochrane have realised that they are not going to be able to re-do a review saying exercise works and to say it doesn't would just further tarnish the brand - in their eyes. They seem unaware that the way to untarnished their brand and serve patients' interests is to learn how to do things properly.

Here is where I think Cochrane's unearned reputation as the "gold standard" of EBM is causing problems: to them, the Cochrane way is the right way. A variation on Nixon's "when the president does it, that means it's not illegal", which challenged the entire principle of a republican system of government but is technically correct unless you can actually force compliance onto someone with tremendous powers.

Horton at the Lancet seems to have this same attitude. He is famously an "activist" who said almost half of all published research is flawed and should be retracted, but will call anyone who suggests the same is true of research he published as editor-in-chief an activist, somehow a concept now turned bad, because The Lancet way is right and therefore if they do it it's right.

Frankly there's been such a high level of in-breeding of ideas in EBM that it makes the European royal families look like genetic diversity incarnate. It's all about eminence and kings- and barons-of-a-small-hill.

 

[rvallee]

If we are now being told that regardless of any patient and outside expert input, exactly the same processes and decision making procedures as failed before have to be applied, and may as a result reach the same flawed conclusions as before, then what's the point?

Somehow I'm thinking of unnamed individual trying to convince NICE to publish the revised guidelines without the evidence review, since the patient activists would be happy regardless, and how widespread that kind of thinking is. Performative mollification works against people who are so powerless that no one will care when we say otherwise. Works in the sense of "checking a few boxes on a form somewhere" anyway.

 

[MSEsperanza]

But we were led to believe that Hilda Bastion had some idea of the realities involved and might at least make people at Cochrane aware that things needed to change.

Also, in the beginning Hilda seemed to listen and provide ways also for people outside the IAG zu engage. I had hope when she came here to discuss things with us:

There will be a communication channel with groups that aren't on the IAG - and how that will work will be discussed with them: including S4ME.

I'll add your suggestion to the list of ideas for consideration, thanks.

2) The rough distillation - no, the rough distillation is too messy for that, and there is already feedback from the IAG members pointing out things that need work. The only reason it was even shared internally as it is currently is, was as a shortcut for time. One of the purposes of going public with the next version is to get just that kind of feedback you're talking about. There'll be more on this in the June monthly report.

There'll be several opportunities in the coming months to participate.

Again, I'm aware that there can be good and sadly also tragic reasons for delays. But it's hard now to believe that these are really only delays if nobody bothers to even just inform us about delays.

(Edited for clarity.)

 

[Snowdrop]

But they refuse to do either of these things, because that is not how their reviews are currently formatted. They are obsessed with their own manualised process for producing the same unreadable and unusable format for reviews

This actually gives me hope.

Occasionally an initially small and obscure force that goes unnoticed, in reality, over time sneaks up on all of the main players in a drama and changes things all while everyone is unsuspecting.

In other words. When a thing truly IS useless and the people who know this seem to be getting nowhere and the useless thing carries on that lack of acknowledgement and incapacity to change provides the seeds of it's own destruction.

Of course that sort of 'time will out' never comes fast enough. And maybe this is just a little to philosophical. But people tend to think of things like this sort of business as a lasting thing (without really giving it any thought) and are surprised at it's demise all of a sudden.

Caveat Venditor. Let the seller beware.

All very ominous. Maybe I just need some sleep.

 

[petrichor]

To clarify, I'm not even defending Cochrane. I agree with most of the criticisms of Cochrane, and I am also deeply disappointed with how this review has gone so far.

I am just trying to explain what the purpose of Cochrane is and how they work. Their entire purpose is to conduct systematic reviews according to explicit (allegedly) defined methods. That still allows some flexibility, but it doesn't mean that it is possible for them to just do anything they want.

 

[FMMM1]

This includes where there are no trials at all and/or a preponderance of poor trials like PACE. They have told me that that is beyond their remit, their (volunteer) authors would not want to or have time do it, and so its a job for another agency yet to emerge.

Fair enough i.e. "that is beyond their [Cochrane's] remit, their (volunteer) authors would not want to or have time do it" but why are NICE funding them?

 

[Sean]

But that should be balanced with an awareness of how this works.

Yes, and no.

Demands for change are usually declared unreasonable and impractical by those with a vested interest in things not changing.

 

[ME/CFS Skeptic]

I am just trying to explain what the purpose of Cochrane is and how they work. Their entire purpose is to conduct systematic reviews according to explicit (allegedly) defined methods. That still allows some flexibility, but it doesn't mean that it is possible for them to just do anything they want.

But I don't think the whole process they started with setting up an advisory board with representation from patient organisations is described in that rulebook. And it has never been the main issue that critics of the review asked for (they simply wanted errors to be corrected).

I think that makes clear that the Cochrane rulebook isn't what is preventing them from doing the right thing here.

 

[petrichor]

But I don't the whole process they started with setting up an advisory board with representation from patient organisations is described in that rulebook. And it has never been the main issue that critics of the review asked for (they simply wanted errors to be corrected).

I think that makes clear that the Cochrane rulebook isn't what is preventing them from doing the right thing here.

The IAG or anything related to the IAG isn't specified in the cochrane handbook as far as I'm aware. So I think it would be viewed as something that falls outside the purview of the handbook. The existence of the IAG still isn't going against their basic function as an organisation that produces systematic reviews according to set methods, and I don't believe it's contradicting those methods.

It depends on what is being asked of Cochrane. If they are being asked to do something/things that blatantly violate and don't follow at all what is in the handbook and which doesn't follow their basic function as an organisation that produces systematic reviews that follow set methods, then I don't see how that can happen. Because that contradicts the entire raison d'être of cochrane. That's what cochrane does.

But there are definitely things they can do, I am absolutely not saying there aren't things they can do. It depends on what is meant by "doing the right thing".

 

[petrichor]

And it has never been the main issue that critics of the review asked for (they simply wanted errors to be corrected).

To be clear, I'm not making excuses for cochrane on things like this. They have no excuse for not correcting those errors.

What I'm saying shouldn't be interpreted as any kind of defence of cochrane. I'm trying to explain the function of cochrane, and what I'm saying shouldn't be interpreted as extending further than that.

 

[Caroline Struthers]

I predict they will use the obvious failure of this new "made up as they go along process" (the IAG etc). which they are calling a "pilot" for involving patients in "high profile reviews" as an excuse never to involve patients in future reviews. It diverts attention from the fact that they still refuse either to fix the errors in the published review or withdraw it.

 

[Trish]

I am still hoping that the review will be done sensibly and the obvious conclusion reached that there is no valid evidence that increasing exercise improves the health or function of people with ME in any significant way.

NICE used GRADE and concluded the studies were all poor or very poor and the guideline committee concluded that on this basis GET should not be recommended. The review group appointed by Cochrane have the same evidence to look at and should come to the same conclusion.

 

[Jonathan Edwards]

I think this is a useful discussion because it relates to how Cochrane might defend itself in the face of the sort of criticism being made. I think two different things are getting confused.

I think most people would regard the central purpose of Cochrane as providing reviews that accurately reflect the reliability of all evidence available.

An important part of that is being seen to be systematic in the sense of comprehensive and having, at any one time, a transparent and explicit methodology against which the reviews can be audited.

But that does not in any way imply the methods are fixed.

In fact Cochrane methods are constantly changing. GRADE was introduced, RoB was introduced and then changed to RoB2 (which is worse). This discussion started with my suggestion that Cochrane could avoid a lengthy process by using the trawling work done by NICE. As Caroline has pointed out, this is now explicit Cochrane policy - that it can make use of trawls done by NICE and vice versa.

So I am not asking for anything that is not already part of the Cochrane rule book.

For the exercise review a new review is easy if the NICE trawl is used. There isn't even any need to make use of the NICE grading - just the literature search. As I pointed out when I peered reviewed, the review process explicitly limited itself to controlled trials (NICE doesn't do that and GRADE doesn't but it seems Cochrane does or did). There are no meaningfully controlled trials in this area - something you can see from the abstracts. So there is nothing to review.

Note that this raises an inconsistency with Hilda's point that assessing all outcomes independently allows you to gather useful information even if the primary outcome is negative. It doesn't if you don't include the trials. NICE did things properly in terms of looking at all information - including surveys of harms. Cochrane made a mess of the whole thing.

Nothing is fixed at Cochrane. It keeps changing to suit the interests of those involved. Calling for more positive change is entirely reasonable.

I don't think this exercise is about getting a new valid review. It is about sending Cochrane off home to write out 100 times by Tuesday 'I must improve my handwriting.'

 

[Hutan]

NICE did things properly in terms of looking at all information - including surveys of harms. Cochrane made a mess of the whole thing.

Just as a side note - NICE recently made a mess of the Chronic Pain guideline, and, even more recently has twice made a mess of the Long Covid Rapid Response Guideline. The success of the NICE ME/CFS Guideline is the result of the good work by particular staff and the appointed committee, and the experts (including patients) that they listened to. The success wasn't because of any safeguarding protocols of the organisation.

Not that any one has, but it would be wrong to have a 'Cochrane bad, NICE good' mindset. I think it's worth keeping in mind that NICE has recently done some very poor work, and probably will again.

 

[FMMM1]

I understand all that, CRG. So why not a brief update saying that instead of silence.

I compare this to DecodeME which has had delays too, some due to the pandemic and some to the team including pwME. All this has been explained with regular updates and none of us are complaining that it's taking a bit longer than originally hoped.

Tweeted vague reassurance that something with the Cochrane review is happening is not adequate.

I think the key thing here is that the lead on DecodME actually cares.
Chris Ponting has spoken about the driver - he knows Simon and was keen to try to help.

Apologies but those running Cochrane don't care. If they did then there are ample opportunities identified on this site about how to drive research standards up.

E.g. a leading figure here has laid out that once you're retired then you're free to review honestly (I'm definite that they did that before they retired).

So actually trying to evaluate evidence, the supposed role of Cochrane, may be achievable - it's just that Cochrane isn't motivated to stick to that goal ---- Cochrane is about egos etc.

 

[Jonathan Edwards]

Cochrane bad, NICE good' mindset.

Absolutely. That was not intended even if it might seem so!

 

[Jonathan Edwards]

NICE used GRADE and concluded the studies were all poor or very poor and the guideline committee concluded that on this basis GET should not be recommended. The review group appointed by Cochrane have the same evidence to look at and should come to the same conclusion.

Not according to Mr GRADE - Gordon Guyatt. His handbook for GRADE indicates that different people can come to different conclusions and it is not objective. On the other hand he has also said in the BMJ that there is a right way to do GRADE and it would disagree with NICE. I fear this is not a secure foothold!

 

[Caroline Struthers]

Fair enough i.e. "that is beyond their [Cochrane's] remit, their (volunteer) authors would not want to or have time do it" but why are NICE funding them?

They aren't. I made an assumption about that which turned out to be wrong. When i read the actual collaboration agreement (obtained via FOI...and shared on this forum) there is no money changing hands and the data sharing can go both ways. I had wrongly assumed Cochrane were teaming up with NICE to earn money because they are losing their funding from the NIHR.

 

[CRG]

They aren't. I made an assumption about that which turned out to be wrong. When i read the actual collaboration agreement (obtained via FOI...and shared on this forum) there is no money changing hands and the data sharing can go both ways. I had wrongly assumed Cochrane were teaming up with NICE to earn money because they are losing their funding from the NIHR.

The decision about NIHR's long term funding of/ relationship with Cochrane is due this month - the current arrangement ends in 2023. Presumably this has implications for both the advisory group and the planned substantive review ?

If Cochrane doesn't yet know how it will operate in 12 months time, and the next 12 months are, whatever the outcome of NIHR's deliberations, going to mean radical changes for Cochrane what prospect is there of any ongoing work actually arriving at a conclusion ?

 

[rvallee]

Not that any one has, but it would be wrong to have a 'Cochrane bad, NICE good' mindset. I think it's worth keeping in mind that NICE has recently done some very poor work, and probably will again.

Definitely. NICE produced the 2007 guidelines (and many atrocious ones since, the chronic pain one is awful), the update does not undo the harm they did back then. There was no acknowledgement of harm done other than a meek sentence in the guidelines. Which is better than nothing but clearly makes a mockery of the "do no harm" slogan. Or else what? Or else nothing. The update was clearly improved because of the involvement of people biased for the patients' interests, rather than having people promoting their personal agenda.

Clearly methodology has little to do with the outcome. It's the people involved, since all rules are interpreted anyway. Here having the BPS ideologues fond of GRADE when it goes their way and raging when it doesn't says it all. So in this case there is still a chance for the IAG to do something better, as who is involved, and what their agenda is, is clearly the only relevant factor.