1. Click here to sign the "Publish the NICE ME/CFS Guideline Now" petition.
    Dismiss Notice
  2. Guest, the 'News in Brief' for the week beginning 13th September 2021 is here.
    Dismiss Notice
  3. Welcome! To read the Core Purpose and Values of our forum, click here.
    Dismiss Notice

Independent advisory group for the full update of the Cochrane review on exercise therapy and ME/CFS (2020), led by Hilda Bastian

Discussion in '2021 Cochrane Exercise Therapy Review' started by Lucibee, Feb 13, 2020.

  1. Hutan

    Hutan Moderator Staff Member

    Messages:
    15,998
    Location:
    New Zealand
    No. The current standard, as exemplified by most literature on the subject and most recently and relevantly by the NICE ME/CFS Guideline evidence review, is that subjective outcomes in unblinded trials constitute low or very low quality evidence of treatment efficacy. If the reviewers in the exercise therapy review are capable assessors of trials, then we won't have a problem. And that improvement in the assessment of trials in Cochrane can start to flow through to other reviews, to the benefit of Cochrane's reputation and many vulnerable people.

    What makes you think that the reviewers on this particular review won't adhere to basic scientific standards? I do share your concerns about many of the appointees to this review. But surely the solution is not to try to come up with a workaround solution that steps around the fundamental flaw in GET and CBT trials? Any such solution will not stop the flow of mediocre trials. Surely the solution is to make sure that the reviewers have the required level of scientific literacy?
     
    Kirsten, alktipping, Michelle and 8 others like this.
  2. petrichor

    petrichor Senior Member (Voting Rights)

    Messages:
    221
    Perhaps cochrane will significantly downgrade unblinded trials that rely on subjective outcomes. Based on NICE, that does seem like a possibility. However that's not the point I'm trying to discuss here.
     
    ukxmrv, Michelle and Trish like this.
  3. Jonathan Edwards

    Jonathan Edwards Senior Member (Voting Rights)

    Messages:
    9,786
    There is nothing the least bit 'radical' about this. it is a matter of known fact that such trials are unreliable so there inso problem discounting them. Every medical student is taught this. such trials are routinely discounted for drug treatments so why not for non-drug treatments?
     
    MEMarge, ukxmrv, alktipping and 10 others like this.
  4. Jonathan Edwards

    Jonathan Edwards Senior Member (Voting Rights)

    Messages:
    9,786
    The problems that Cochrane is locked in to the GRADE system which has a set of rules that do not reflect the facts about reliability so is bogus.

    Not sure what point you are wanting to discuss if not this?
     
  5. Hutan

    Hutan Moderator Staff Member

    Messages:
    15,998
    Location:
    New Zealand
    My understanding is that these are your questions:
    where 1 = the bias created by relying on a subjective outcome in an unblinded trial, and 2 = the bias created by cognitive manipulation inherent in the treatment acting on an outcome that does not necessarily reflect any real treatment benefit.

    The answer to your first question is 'yes, of course not taking into account important biases would significantly limit a Cochrane review'. And if the biases aren't taken 'account of properly', then surely the only possibility is to take account of the biases improperly?

    Are you asking for the other less fundamental problems with the GET trials to be listed and focused on because you don't think the reviewers will properly deal with these two biases?
     
  6. Jonathan Edwards

    Jonathan Edwards Senior Member (Voting Rights)

    Messages:
    9,786
    Sorry, I had not read all the comments before responding initially.i agree with Hutan.

    Systematic reviews are supposed to have rules that make them comprehensive and consistent. That makes sense to everyone. But unfortunately the grADE system used since around 2010 also has rules about how you judge reliability and I think that is something quite different. judging reliability should not be based on rules.it should be based on the facts we have about reliability. we know that unblinded trials with subjective outcomes are unreliable - that is why blinded trials were invented. So having a system that does not discount them is just bad practice.

    the central problem with Cochrane is that it arose from a group people biased towards non-drug treatments. They repeatedly admit this. The question is whether they are prepared to overcome it in the context of showing themselves capable being consistent when it comes to application in reviews.
     
  7. petrichor

    petrichor Senior Member (Voting Rights)

    Messages:
    221
    So I think there's a bit of miscommunication going on here, where my specific points and query may not be being completely understood. My basic question is, assuming that cochrane doesn't completely discount unblinded trials with subjective outcomes, how can those two key related problems with the PACE trial still be taken account of? Those two key problems being 1, that it is an unblinded trial with subjective outcomes, which is substantially made worse by the fact that 2: the treatment in PACE and other trials entails an "treatment-inherent cognitive bias" ie. patients are told to do things like ignore and dismiss their symptoms, which is likely to influence how they answer subjective questionnaires.

    Given that there is a possibility that cochrane may choose to still give some weight to unblinded trials with subjective outcomes, I feel like this is something worth discussing.

    I was a bit vague about the kind of solution I might be looking for. So I'll suggest one, in which those two problems could still be taken account of to some extent, even if cochrane decides not to discount unblinded trials with subjective outcomes, and being consistent with the use of GRADE and the limitations of systematic reviews.

    In the GRADE guidelines, one of the guidelines on rating the quality of evidence relates to inconsistency. That is, inconsistency of relative measures of effect across studies and outcomes. When there is a large inconsistency GRADE suggests searching for an explanation. One of the strong indicators that there is this effect of "treatment-inherent cognitive bias" in PACE and other trials, is that the measures of effect in the subjective outcomes is very inconsistent with the lack of significant difference in objective outcomes, and long term followup. So, in accordance with GRADE, an explanation can be sought for that, and that explanation would be something along the lines of "the treatment may lead to a change in subjective reporting, but not real world improvement". And then PACE and other similar trials can be substantially downgraded on the basis of that. Whilst still being systematic and replicable, in accordance with GRADE, and taking account of those two key problems.
     
  8. Jonathan Edwards

    Jonathan Edwards Senior Member (Voting Rights)

    Messages:
    9,786
    Yes, you can use GRADE to shift your decision in whatever direction you like- which is why it is so ridiculous. NICE decided to downgrade on the basis of selection criteria I think. You could think up a reason under almost any of the categories. But that needs people who want to do that. If people want to get positive analysis they Willard if not they won't. We can think of ways to get a negative analysis with GRaDE but that doesn't really help. If you stick with GRADE you sendup with what the assessors like. The alternative is to abandon GRADE but the people involved are much too obsessive todo that I think.
     
  9. petrichor

    petrichor Senior Member (Voting Rights)

    Messages:
    221
    I'm honestly not in a position to really fully understand how systematic reviews and GRADE work, and what's acceptable, hence why I was/am trying to start a discussion. I think your response here partially answers what I was thinking about. NICE says in this document: https://www.nice.org.uk/guidance/gid-ng10091/documents/evidence-review-7 that the main reasons for downgrading were bias (from lack of blinding and subjective outcomes), indirectness and imprecision, by the way.
     
  10. Trish

    Trish Moderator Staff Member

    Messages:
    35,721
    Location:
    UK
    Perhaps one of the key things to move towards a better outcome for reviews of GET trials is to ensure that the protocol for the new Cochrane review insists on the inclusion of objective outcomes, of which there were several in PACE, and that if subjective outcomes are included, greater weight is given to the longest term follow up results. It is the short term subjective effects of hope, therapist effect, placebo effect and persuasion, that are most likely to induce a transient subjective 'improvement' that is not sustained.
     
  11. Sly Saint

    Sly Saint Senior Member (Voting Rights)

    Messages:
    6,662
    Location:
    UK
    Last edited: Jul 2, 2021
  12. Snow Leopard

    Snow Leopard Senior Member (Voting Rights)

    Messages:
    3,174
    Location:
    Australia
    Thanks, I was curious about the selection of the last two IAG members.

     
    Last edited by a moderator: Jul 2, 2021
    MSEsperanza, MEMarge, ukxmrv and 9 others like this.
  13. rvallee

    rvallee Senior Member (Voting Rights)

    Messages:
    7,280
    Location:
    Canada
    Hmmm... unlikely that someone for whom GRADE is so central to her work will be able to acknowledge the massive flaws that lead to very low quality "research" to be judged arbitrarily to fit a pre-sought outcome by teams biased for that purpose, how easy it has been to manipulate it simply based on who is present to apply or not its principles. Because frankly the very basic premise of EBM is what has failed massively here, where most of the flaws are, in just how arbitrary this all is, turning this way and that depending entirely on who is present and their personal and professional agenda.

    But that's just based on precedent, hoping to be pleasantly surprised. Could be a teaching moment, although those are very far and few in this field.
     
    MSEsperanza, MEMarge, ukxmrv and 5 others like this.
  14. Ariel

    Ariel Senior Member (Voting Rights)

    Messages:
    452
    "The areas of expertise that have been suggested are wide-ranging, from people who have severe ME/CFS or care for people with ME/CFS, to people who have recovered"

    Strange placement of "people who have recovered" here. People with severe ME/CFS are not on another side of an argument or contrast with "people who have recovered". What is the expertise or experience of "people who have recovered"? Anecdotal recovery stories? Are they to be taken to have "expertise" on recovery, or did they simply happen to recover? Reminiscent of Paul Garner's talk of listening to those patients who recovered in order to get well.

    I feel this contrast of severely ill patients with "people who have recovered" diminishes severe ME/CFS. Would the same framing be used with cancer?
     
  15. Adrian

    Adrian Administrator Staff Member

    Messages:
    5,674
    Location:
    UK
    I don't see why it is radical to dismiss unreliable results and yes they should be removed from other reviews as well.

    I think from a review perspective there is much more that is wrong with the GET/CBT trials
    There is not point to the review since there is no reliable data - but I guess writing a protocol and finding that nothing is up to standard could be worth while
     
  16. Adrian

    Adrian Administrator Staff Member

    Messages:
    5,674
    Location:
    UK
    For some in the UK it is since the requirement seems to be to keep certain eminent people happy.
     
  17. Adrian

    Adrian Administrator Staff Member

    Messages:
    5,674
    Location:
    UK
    Its not just that they are poor quality its that they aren't linear hence the maths that cochrane is applying is simply wrong. It doesn't make sense to quote means and mean differences. Its an issue because they don't review and think about what happens but simply apply a menu of techniques in a way that should fail a student if they were to do that in a stats/data science course.
     
  18. Adrian

    Adrian Administrator Staff Member

    Messages:
    5,674
    Location:
    UK
    Putting a cochrane loyalist on an independent advisory group doesn't make it sound very independent. But given this seems to basically be people Hilda chooses/likes I'm not surprised. Also what is this about contentious methods the review is about the basic competence of cochrane not about contentious methods.
     
  19. Caroline Struthers

    Caroline Struthers Senior Member (Voting Rights)

    Messages:
    417
    Location:
    Oxford UK
    I know Miranda quite well - she used to be my boss.
     
  20. FMMM1

    FMMM1 Senior Member (Voting Rights)

    Messages:
    1,176
    My view is that there have been clues to how you assess the outcome of interventions for some time. E.g., as someone with a slight interest, I'd come across a study of American soldiers, who'd been in one of the Gulf wars, which monitored their phones --- moving around/not moving around --- whatever --- to objectively assess how they were.
    Fluge and Mella used actimetry in the rituximab study.

    I just don't grasp how come someone can use poor quality outcome indicators when better ones are available - presumably cost [EDIT "of e.g. actimetry"] is a factor , but if the evidence gathered using questionnaires is so poor then why do/publish the study?

    Also, I think that dropping actimetry monitoring in the PACE study, with the support of the embedded charity Action for ME, was pretty much a low point in terms of the Government's treatment of people with ME. A study aimed at assessing CBT and GET was turned into a charade and the losers were people with ME/CFS and their families.

    My question is how can you get this s--t published but unfortunately posts here indicate that you can get a lot of poor quality research published.
     
    Missense, MEMarge, Michelle and 8 others like this.

Share This Page