Independent advisory group for the full update of the Cochrane review on exercise therapy and ME/CFS (2020), led by Hilda Bastian

[Hutan]

As far as I can tell the current standard is that subjective outcomes in unblinded trials are basically accepted

No. The current standard, as exemplified by most literature on the subject and most recently and relevantly by the NICE ME/CFS Guideline evidence review, is that subjective outcomes in unblinded trials constitute low or very low quality evidence of treatment efficacy. If the reviewers in the exercise therapy review are capable assessors of trials, then we won't have a problem. And that improvement in the assessment of trials in Cochrane can start to flow through to other reviews, to the benefit of Cochrane's reputation and many vulnerable people.

The question I'm trying to ask, is that given they don't seem likely to do that

What makes you think that the reviewers on this particular review won't adhere to basic scientific standards? I do share your concerns about many of the appointees to this review. But surely the solution is not to try to come up with a workaround solution that steps around the fundamental flaw in GET and CBT trials? Any such solution will not stop the flow of mediocre trials. Surely the solution is to make sure that the reviewers have the required level of scientific literacy?

 

[petrichor]

No. The current standard, as exemplified by most literature on the subject and most recently and relevantly by the NICE ME/CFS Guideline evidence review, is that subjective outcomes in unblinded trials constitute low or very low quality evidence of treatment efficacy. If the reviewers in the exercise therapy review are capable assessors of trials, then we won't have a problem. And that improvement in the assessment of trials in Cochrane can start to flow through to other reviews, to the benefit of Cochrane's reputation and many vulnerable people.


What makes you think that the reviewers on this particular review won't adhere to basic scientific standards? I do share your concerns about many of the appointees to this review. But surely the solution is not to try to come up with a workaround solution that steps around the fundamental flaw in GET and CBT trials? Any such solution will not stop the flow of mediocre trials. Surely the solution is to make sure that the reviewers have the required level of scientific literacy?

Perhaps cochrane will significantly downgrade unblinded trials that rely on subjective outcomes. Based on NICE, that does seem like a possibility. However that's not the point I'm trying to discuss here.

 

[Jonathan Edwards]

Identifying whether trials use subjective questionnaire outcomes and whether they're unblinded is a relatively objective fact, although not necessarily that simple. However wholely dismissing the results of unblinded trials that rely on a subjective questionnaire outcome is a pretty radical step,

There is nothing the least bit 'radical' about this. it is a matter of known fact that such trials are unreliable so there inso problem discounting them. Every medical student is taught this. such trials are routinely discounted for drug treatments so why not for non-drug treatments?

 

[Jonathan Edwards]

Perhaps cochrane will significantly downgrade unblinded trials that rely on subjective outcomes. Based on NICE, that does seem like a possibility. However that's not the point I'm trying to discuss here.

The problems that Cochrane is locked in to the GRADE system which has a set of rules that do not reflect the facts about reliability so is bogus.

Not sure what point you are wanting to discuss if not this?

 

[Hutan]

Perhaps cochrane will significantly downgrade unblinded trials that rely on subjective outcomes. Based on NICE, that does seem like a possibility. However that's not the point I'm trying to discuss here.

My understanding is that these are your questions:

So if 1 isn't taken account of properly, because that would be too radical and inconsistent with other systematic reviews, and 2 isn't taken account of because it simply isn't possible to do in a systematic replicable manner, does this significantly limit a cochrane review of GET? And what possible ways are there to take account of those two problems, whilst still being consistent with the limitations of a systematic review?

where 1 = the bias created by relying on a subjective outcome in an unblinded trial, and 2 = the bias created by cognitive manipulation inherent in the treatment acting on an outcome that does not necessarily reflect any real treatment benefit.

The answer to your first question is 'yes, of course not taking into account important biases would significantly limit a Cochrane review'. And if the biases aren't taken 'account of properly', then surely the only possibility is to take account of the biases improperly?

Are you asking for the other less fundamental problems with the GET trials to be listed and focused on because you don't think the reviewers will properly deal with these two biases?

 

[Jonathan Edwards]

Sorry, I had not read all the comments before responding initially.i agree with Hutan.

Systematic reviews are supposed to have rules that make them comprehensive and consistent. That makes sense to everyone. But unfortunately the grADE system used since around 2010 also has rules about how you judge reliability and I think that is something quite different. judging reliability should not be based on rules.it should be based on the facts we have about reliability. we know that unblinded trials with subjective outcomes are unreliable - that is why blinded trials were invented. So having a system that does not discount them is just bad practice.

the central problem with Cochrane is that it arose from a group people biased towards non-drug treatments. They repeatedly admit this. The question is whether they are prepared to overcome it in the context of showing themselves capable being consistent when it comes to application in reviews.

 

[petrichor]

So I think there's a bit of miscommunication going on here, where my specific points and query may not be being completely understood. My basic question is, assuming that cochrane doesn't completely discount unblinded trials with subjective outcomes, how can those two key related problems with the PACE trial still be taken account of? Those two key problems being 1, that it is an unblinded trial with subjective outcomes, which is substantially made worse by the fact that 2: the treatment in PACE and other trials entails an "treatment-inherent cognitive bias" ie. patients are told to do things like ignore and dismiss their symptoms, which is likely to influence how they answer subjective questionnaires.

Given that there is a possibility that cochrane may choose to still give some weight to unblinded trials with subjective outcomes, I feel like this is something worth discussing.

I was a bit vague about the kind of solution I might be looking for. So I'll suggest one, in which those two problems could still be taken account of to some extent, even if cochrane decides not to discount unblinded trials with subjective outcomes, and being consistent with the use of GRADE and the limitations of systematic reviews.

In the GRADE guidelines, one of the guidelines on rating the quality of evidence relates to inconsistency. That is, inconsistency of relative measures of effect across studies and outcomes. When there is a large inconsistency GRADE suggests searching for an explanation. One of the strong indicators that there is this effect of "treatment-inherent cognitive bias" in PACE and other trials, is that the measures of effect in the subjective outcomes is very inconsistent with the lack of significant difference in objective outcomes, and long term followup. So, in accordance with GRADE, an explanation can be sought for that, and that explanation would be something along the lines of "the treatment may lead to a change in subjective reporting, but not real world improvement". And then PACE and other similar trials can be substantially downgraded on the basis of that. Whilst still being systematic and replicable, in accordance with GRADE, and taking account of those two key problems.

 

[Jonathan Edwards]

Yes, you can use GRADE to shift your decision in whatever direction you like- which is why it is so ridiculous. NICE decided to downgrade on the basis of selection criteria I think. You could think up a reason under almost any of the categories. But that needs people who want to do that. If people want to get positive analysis they Willard if not they won't. We can think of ways to get a negative analysis with GRaDE but that doesn't really help. If you stick with GRADE you sendup with what the assessors like. The alternative is to abandon GRADE but the people involved are much too obsessive todo that I think.

 

[petrichor]

Yes, you can use GRADE to shift your decision in whatever direction you like- which is why it is so ridiculous. NICE decided to downgrade on the basis of selection criteria I think. You could think up a reason under almost any of the categories. But that needs people who want to do that. If people want to get positive analysis they Willard if not they won't. We can think of ways to get a negative analysis with GRaDE but that doesn't really help. If you stick with GRADE you sendup with what the assessors like. The alternative is to abandon GRADE but the people involved are much too obsessive todo that I think.

I'm honestly not in a position to really fully understand how systematic reviews and GRADE work, and what's acceptable, hence why I was/am trying to start a discussion. I think your response here partially answers what I was thinking about. NICE says in this document: https://www.nice.org.uk/guidance/gid-ng10091/documents/evidence-review-7 that the main reasons for downgrading were bias (from lack of blinding and subjective outcomes), indirectness and imprecision, by the way.

 

[Trish]

Perhaps one of the key things to move towards a better outcome for reviews of GET trials is to ensure that the protocol for the new Cochrane review insists on the inclusion of objective outcomes, of which there were several in PACE, and that if subjective outcomes are included, greater weight is given to the longest term follow up results. It is the short term subjective effects of hope, therapist effect, placebo effect and persuasion, that are most likely to induce a transient subjective 'improvement' that is not sustained.

 

[Sly Saint]

don't know if this has been posted(?) I haven't read it yet.
Report for June
https://community.cochrane.org/orga...igh-profile-reviews-pilot/cochrane-exercise-0

 

[Snow Leopard]

don't know if this has been posted(?) I haven't read it yet.
Report for June
https://community.cochrane.org/orga...igh-profile-reviews-pilot/cochrane-exercise-0

Thanks, I was curious about the selection of the last two IAG members.

Independent Advisory Group (IAG) membership
The membership of the IAG is almost complete. I’m pleased to introduce our ninth member, Miranda Cumpston. Miranda has 15 years of wide-ranging experience in systematic reviews and the methods of evidence synthesis – with expertise in several methods that have been contentious in this review. Miranda is a senior methods editor for Cochrane Public Health. She is one of the co-editors of Cochrane’s Handbook on methods for Cochrane reviews, as well as a member of the GRADE Working Group. Her bio and disclosures are here.

The IAG will be deciding on our tenth member in the coming weeks. We’ve had expressions of interest from both individuals and groups, as well as suggestions of individuals and areas of expertise and experience: many thanks to you all! At the same time, the IAG and the editorial group will be considering possible peer reviewers for the protocol and review.

The areas of expertise that have been suggested are wide-ranging, from people who have severe ME/CFS or care for people with ME/CFS, to people who have recovered, as well as suggestions about diversity, and valuable research disciplines/experience. The wide range of possible expertise underlines how critical it will be for the IAG to establish ongoing communication with stakeholders who are not directly involved with the IAG.

 

[rvallee]

Hmmm... unlikely that someone for whom GRADE is so central to her work will be able to acknowledge the massive flaws that lead to very low quality "research" to be judged arbitrarily to fit a pre-sought outcome by teams biased for that purpose, how easy it has been to manipulate it simply based on who is present to apply or not its principles. Because frankly the very basic premise of EBM is what has failed massively here, where most of the flaws are, in just how arbitrary this all is, turning this way and that depending entirely on who is present and their personal and professional agenda.

But that's just based on precedent, hoping to be pleasantly surprised. Could be a teaching moment, although those are very far and few in this field.

 

[Ariel]

"The areas of expertise that have been suggested are wide-ranging, from people who have severe ME/CFS or care for people with ME/CFS, to people who have recovered"

Strange placement of "people who have recovered" here. People with severe ME/CFS are not on another side of an argument or contrast with "people who have recovered". What is the expertise or experience of "people who have recovered"? Anecdotal recovery stories? Are they to be taken to have "expertise" on recovery, or did they simply happen to recover? Reminiscent of Paul Garner's talk of listening to those patients who recovered in order to get well.

I feel this contrast of severely ill patients with "people who have recovered" diminishes severe ME/CFS. Would the same framing be used with cancer?

 

[Adrian]

However wholely dismissing the results of unblinded trials that rely on a subjective questionnaire outcome is a pretty radical step, and for consistency that should also mean all other such trials in other systematic reviews should also be dismissed.

I don't see why it is radical to dismiss unreliable results and yes they should be removed from other reviews as well.

I think from a review perspective there is much more that is wrong with the GET/CBT trials

does this significantly limit a cochrane review of GET?

There is not point to the review since there is no reliable data - but I guess writing a protocol and finding that nothing is up to standard could be worth while

 

[Adrian]

Requiring good science doesn't seem like a radical stance.

For some in the UK it is since the requirement seems to be to keep certain eminent people happy.

 

[Adrian]

and if the patient reported outcome measures aren't good quality either (like the chalder fatigue questionnaire), then that doesn't help either.

Its not just that they are poor quality its that they aren't linear hence the maths that cochrane is applying is simply wrong. It doesn't make sense to quote means and mean differences. Its an issue because they don't review and think about what happens but simply apply a menu of techniques in a way that should fail a student if they were to do that in a stats/data science course.

 

[Adrian]

Independent Advisory Group (IAG) membership
The membership of the IAG is almost complete. I’m pleased to introduce our ninth member, Miranda Cumpston. Miranda has 15 years of wide-ranging experience in systematic reviews and the methods of evidence synthesis – with expertise in several methods that have been contentious in this review. Miranda is a senior methods editor for Cochrane Public Health. She is one of the co-editors of Cochrane’s Handbook on methods for Cochrane reviews, as well as a member of the GRADE Working Group. Her bio and disclosures are here.

Putting a cochrane loyalist on an independent advisory group doesn't make it sound very independent. But given this seems to basically be people Hilda chooses/likes I'm not surprised. Also what is this about contentious methods the review is about the basic competence of cochrane not about contentious methods.

 

[Caroline Struthers]

Putting a cochrane loyalist on an independent advisory group doesn't make it sound very independent. But given this seems to basically be people Hilda chooses/likes I'm not surprised. Also what is this about contentious methods the review is about the basic competence of cochrane not about contentious methods.

I know Miranda quite well - she used to be my boss.

 

[FMMM1]

As far as I can tell the current standard is that subjective outcomes in unblinded trials are basically accepted, although they get marked down a bit or a moderate amount to take account of possible bias, and if the patient reported outcome measures aren't good quality either (like the chalder fatigue questionnaire), then that doesn't help either. I'm not trying to open the whole subject of trying to make cochrane disregard unblinded trials that rely on subjective outcomes. The question I'm trying to ask, is that given they don't seem likely to do that, what ways are there to take account of those two key problems (unblinded with subjective outcomes + treatment-inherent cognitive bias) with the PACE trial while still being consistent with the limitations of how systematic reviews are done?

My view is that there have been clues to how you assess the outcome of interventions for some time. E.g., as someone with a slight interest, I'd come across a study of American soldiers, who'd been in one of the Gulf wars, which monitored their phones --- moving around/not moving around --- whatever --- to objectively assess how they were.
Fluge and Mella used actimetry in the rituximab study.

I just don't grasp how come someone can use poor quality outcome indicators when better ones are available - presumably cost [EDIT "of e.g. actimetry"] is a factor , but if the evidence gathered using questionnaires is so poor then why do/publish the study?

Also, I think that dropping actimetry monitoring in the PACE study, with the support of the embedded charity Action for ME, was pretty much a low point in terms of the Government's treatment of people with ME. A study aimed at assessing CBT and GET was turned into a charade and the losers were people with ME/CFS and their families.

My question is how can you get this s--t published but unfortunately posts here indicate that you can get a lot of poor quality research published.