Independent advisory group for the full update of the Cochrane review on exercise therapy and ME/CFS (2020), led by Hilda Bastian

Here is where I think Cochrane's unearned reputation as the "gold standard" of EBM is causing problems: to them, the Cochrane way is the right way. A variation on Nixon's "when the president does it, that means it's not illegal", which challenged the entire principle of a republican system of government but is technically correct unless you can actually force compliance onto someone with tremendous powers.

Horton at the Lancet seems to have this same attitude. He is famously an "activist" who said almost half of all published research is flawed and should be retracted, but will call anyone who suggests the same is true of research he published as editor-in-chief an activist, somehow a concept now turned bad, because The Lancet way is right and therefore if they do it it's right.

Frankly there's been such a high level of in-breeding of ideas in EBM that it makes the European royal families look like genetic diversity incarnate. It's all about eminence and kings- and barons-of-a-small-hill.

 

Somehow I'm thinking of unnamed individual trying to convince NICE to publish the revised guidelines without the evidence review, since the patient activists would be happy regardless, and how widespread that kind of thinking is. Performative mollification works against people who are so powerless that no one will care when we say otherwise. Works in the sense of "checking a few boxes on a form somewhere" anyway.

 

Also, in the beginning Hilda seemed to listen and provide ways also for people outside the IAG zu engage. I had hope when she came here to discuss things with us:

Again, I'm aware that there can be good and sadly also tragic reasons for delays. But it's hard now to believe that these are really only delays if nobody bothers to even just inform us about delays.

(Edited for clarity.)

 

This actually gives me hope.

Occasionally an initially small and obscure force that goes unnoticed, in reality, over time sneaks up on all of the main players in a drama and changes things all while everyone is unsuspecting.

In other words. When a thing truly IS useless and the people who know this seem to be getting nowhere and the useless thing carries on that lack of acknowledgement and incapacity to change provides the seeds of it's own destruction.

Of course that sort of 'time will out' never comes fast enough. And maybe this is just a little to philosophical. But people tend to think of things like this sort of business as a lasting thing (without really giving it any thought) and are surprised at it's demise all of a sudden.

Caveat Venditor. Let the seller beware.

All very ominous. Maybe I just need some sleep.

 

To clarify, I'm not even defending Cochrane. I agree with most of the criticisms of Cochrane, and I am also deeply disappointed with how this review has gone so far.

I am just trying to explain what the purpose of Cochrane is and how they work. Their entire purpose is to conduct systematic reviews according to explicit (allegedly) defined methods. That still allows some flexibility, but it doesn't mean that it is possible for them to just do anything they want.

 

Fair enough i.e. "that is beyond their [Cochrane's] remit, their (volunteer) authors would not want to or have time do it" but why are NICE funding them?

 

But I don't think the whole process they started with setting up an advisory board with representation from patient organisations is described in that rulebook. And it has never been the main issue that critics of the review asked for (they simply wanted errors to be corrected).

I think that makes clear that the Cochrane rulebook isn't what is preventing them from doing the right thing here.

 

The IAG or anything related to the IAG isn't specified in the cochrane handbook as far as I'm aware. So I think it would be viewed as something that falls outside the purview of the handbook. The existence of the IAG still isn't going against their basic function as an organisation that produces systematic reviews according to set methods, and I don't believe it's contradicting those methods.

It depends on what is being asked of Cochrane. If they are being asked to do something/things that blatantly violate and don't follow at all what is in the handbook and which doesn't follow their basic function as an organisation that produces systematic reviews that follow set methods, then I don't see how that can happen. Because that contradicts the entire raison d'être of cochrane. That's what cochrane does.

But there are definitely things they can do, I am absolutely not saying there aren't things they can do. It depends on what is meant by "doing the right thing".

 

To be clear, I'm not making excuses for cochrane on things like this. They have no excuse for not correcting those errors.

What I'm saying shouldn't be interpreted as any kind of defence of cochrane. I'm trying to explain the function of cochrane, and what I'm saying shouldn't be interpreted as extending further than that.

 

I predict they will use the obvious failure of this new "made up as they go along process" (the IAG etc). which they are calling a "pilot" for involving patients in "high profile reviews" as an excuse never to involve patients in future reviews. It diverts attention from the fact that they still refuse either to fix the errors in the published review or withdraw it.

 

I think this is a useful discussion because it relates to how Cochrane might defend itself in the face of the sort of criticism being made. I think two different things are getting confused.

I think most people would regard the central purpose of Cochrane as providing reviews that accurately reflect the reliability of all evidence available.

An important part of that is being seen to be systematic in the sense of comprehensive and having, at any one time, a transparent and explicit methodology against which the reviews can be audited.

But that does not in any way imply the methods are fixed.

In fact Cochrane methods are constantly changing. GRADE was introduced, RoB was introduced and then changed to RoB2 (which is worse). This discussion started with my suggestion that Cochrane could avoid a lengthy process by using the trawling work done by NICE. As Caroline has pointed out, this is now explicit Cochrane policy - that it can make use of trawls done by NICE and vice versa.

So I am not asking for anything that is not already part of the Cochrane rule book.

For the exercise review a new review is easy if the NICE trawl is used. There isn't even any need to make use of the NICE grading - just the literature search. As I pointed out when I peered reviewed, the review process explicitly limited itself to controlled trials (NICE doesn't do that and GRADE doesn't but it seems Cochrane does or did). There are no meaningfully controlled trials in this area - something you can see from the abstracts. So there is nothing to review.

Note that this raises an inconsistency with Hilda's point that assessing all outcomes independently allows you to gather useful information even if the primary outcome is negative. It doesn't if you don't include the trials. NICE did things properly in terms of looking at all information - including surveys of harms. Cochrane made a mess of the whole thing.

Nothing is fixed at Cochrane. It keeps changing to suit the interests of those involved. Calling for more positive change is entirely reasonable.

I don't think this exercise is about getting a new valid review. It is about sending Cochrane off home to write out 100 times by Tuesday 'I must improve my handwriting.'

 

I think the key thing here is that the lead on DecodME actually cares.
Chris Ponting has spoken about the driver - he knows Simon and was keen to try to help.

Apologies but those running Cochrane don't care. If they did then there are ample opportunities identified on this site about how to drive research standards up.

E.g. a leading figure here has laid out that once you're retired then you're free to review honestly (I'm definite that they did that before they retired).

So actually trying to evaluate evidence, the supposed role of Cochrane, may be achievable - it's just that Cochrane isn't motivated to stick to that goal ---- Cochrane is about egos etc.

 

Absolutely. That was not intended even if it might seem so!

 

Not according to Mr GRADE - Gordon Guyatt. His handbook for GRADE indicates that different people can come to different conclusions and it is not objective. On the other hand he has also said in the BMJ that there is a right way to do GRADE and it would disagree with NICE. I fear this is not a secure foothold!

 

They aren't. I made an assumption about that which turned out to be wrong. When i read the actual collaboration agreement (obtained via FOI...and shared on this forum) there is no money changing hands and the data sharing can go both ways. I had wrongly assumed Cochrane were teaming up with NICE to earn money because they are losing their funding from the NIHR.

 

The decision about NIHR's long term funding of/ relationship with Cochrane is due this month - the current arrangement ends in 2023. Presumably this has implications for both the advisory group and the planned substantive review ?

If Cochrane doesn't yet know how it will operate in 12 months time, and the next 12 months are, whatever the outcome of NIHR's deliberations, going to mean radical changes for Cochrane what prospect is there of any ongoing work actually arriving at a conclusion ?

 

Definitely. NICE produced the 2007 guidelines (and many atrocious ones since, the chronic pain one is awful), the update does not undo the harm they did back then. There was no acknowledgement of harm done other than a meek sentence in the guidelines. Which is better than nothing but clearly makes a mockery of the "do no harm" slogan. Or else what? Or else nothing. The update was clearly improved because of the involvement of people biased for the patients' interests, rather than having people promoting their personal agenda.

Clearly methodology has little to do with the outcome. It's the people involved, since all rules are interpreted anyway. Here having the BPS ideologues fond of GRADE when it goes their way and raging when it doesn't says it all. So in this case there is still a chance for the IAG to do something better, as who is involved, and what their agenda is, is clearly the only relevant factor.