1. Sign our petition calling on Cochrane to withdraw their review of Exercise Therapy for CFS here.
    Dismiss Notice
  2. Guest, the 'News in Brief' for the week beginning 18th March 2024 is here.
    Dismiss Notice
  3. Welcome! To read the Core Purpose and Values of our forum, click here.
    Dismiss Notice

Who Agrees That GRADE is (a) unjustified in theory and (b) wrong in practice?

Discussion in 'Other research methodology topics' started by Jonathan Edwards, Mar 4, 2021.

  1. FMMM1

    FMMM1 Senior Member (Voting Rights)

    Messages:
    2,593
    Well to be fair, if you were in NICE, and you had to answer for the mess, then it would be reasonable to ask for the money to do the job properly.
     
    Invisible Woman, Kitty and alktipping like this.
  2. FMMM1

    FMMM1 Senior Member (Voting Rights)

    Messages:
    2,593
    They might have wanted to show they had looked at all of the crap - even if you're correct i.e. they should have said it's crap and here's why--- would have raised a few hackles --- disgruntled X---
     
    Invisible Woman, Kitty and alktipping like this.
  3. FMMM1

    FMMM1 Senior Member (Voting Rights)

    Messages:
    2,593
    I think the problem with GRADE is that it's not that easy to define the dependent variables i.e. measure X, Y, Z --- and if the study is flawed then the number is above "n" --- bit like the meaning of life the universe and everything 42! Theoretically you should be able to produce the model but the fact that GRADE rates things that are useless as +ve means that it has no real world use.
    The way these things are supposed to work is that you decide on a course of action and then monitor the outcome --- if it works then OK if not you review/revise. So the fact that a lot of people have had CBT/GET and the likelihood that they are able to participate in work, study --- is no different from those who haven't had CBT/GET, means that objectively it doesn't work. Hint - you don't measure the outcome of your intervention that way you don't have to own up that your pet idea didn't work! Bear in mind that you guys are invested in a positive outcome, the folks who benefit from this are too [CBT/GET] - it's just that they benefit from the failed intervention!

    EDIT - It's expensive to build in "monitoring" re an intervention; also, if politically your sold on an intervention then your servants might "lose" the monitoring. I'm with Jonathan re it's generally cock up not conspiracy. Just don't think those in power care!
     
    Last edited: Mar 4, 2021
    Invisible Woman, Kitty and alktipping like this.
  4. Barry

    Barry Senior Member (Voting Rights)

    Messages:
    8,385
    I started to read the GRADE manual, but must confess I lost the will to live soon into it.

    I'm not averse to trying to bringing order out of chaos, far from it, but all too often such attempts just replace one kind of chaos with another.

    The notion of "weighing the evidence" is long established, and well founded. But it is useless if you impose artificial limits on how much weighting you can apply to the various components; the process itself then imposes artificial bias. Especially if something, or some combination of somethings, means their weight should eclipse all the other things out of the equation. Which is where logical operations come into play. There is no sense giving something a weight of infinity to artificially achieve that logic, you just need ... logic.

    If you are assessing confidence in the marketability of a car - upholstery comfort, seating position, controls ergonomics, heated seats ... but the heated seats are a bit suspect, then:
    Code:
    if (high confidence seats can electrocute driver)
    {
        // Dead in the water. Weights of everything else totally irrelevant, so ignore.
        Confidence car is marketable = 0;
    }
    else
    {
         // blah ...
    }
    
     
  5. Sly Saint

    Sly Saint Senior Member (Voting Rights)

    Messages:
    9,574
    Location:
    UK
    I would not have relied on Cochranes searches one bit given the way they were done for their exercise reviews and the poor upkeep of the Cochrane Central db by the Cochrane Common Mental Disorders Group in relation to ME/CFS research.

    But for me the issue is being approached from the wrong end; ie they should be coming up with a better, more standardised, more transparent way of presenting/writing research papers which has for so long been a rather vague set of guidelines largely open to interpretation.

    eg: certain information should have to be included in the abstract in a concise, structured way not tucked away somewhere in the main body of the paper (eg limitations of the research).
     
  6. Jonathan Edwards

    Jonathan Edwards Senior Member (Voting Rights)

    Messages:
    13,271
    Location:
    London, UK
    Yes, well I suppose another way of looking at it is that if a GRADE assessment is so unreliable that you have to do it again yourself it is not achieving its aim of reproducibility too well!
     
  7. Barry

    Barry Senior Member (Voting Rights)

    Messages:
    8,385
    Yes. They actively don't claim GRADE achieves reproducibility of gradings (a rather tragic irony), just reproducibility of how things are graded. But what is the point of that? It is reproducibility of gradings (within sane variance) that matters, together with those gradings being valid.

    ETA: It basically means that each organisation, NICE in this case, has to home in on their own interpretation of what GRADE means to them, and to comply with that. If the GRADE designers don't like that, then they should have come up with something better.
     
    Last edited: Mar 5, 2021
    FMMM1, Michelle, alktipping and 6 others like this.
  8. Jonathan Edwards

    Jonathan Edwards Senior Member (Voting Rights)

    Messages:
    13,271
    Location:
    London, UK
    I would agree with @Sly Saint that they don't even achieve reproducibility of how, since Cochrane failed to even select the right sorts of trials. In the Larun review the specification was for controlled trials but none of the trials were controlled in any real sense - of having adequate controls.

    It all seems to come down to Eric Morecambe: they are playing all the right notes but not necessarily in the right order.
     
    Michelle, alktipping, FMMM1 and 9 others like this.
  9. Trish

    Trish Moderator Staff Member

    Messages:
    51,868
    Location:
    UK
    I think the first thing any grading system should do is have a list of basic requirements clinical trials have to meet to be included in any review. If a trial does not meet any one of the requirements they would be automatically excluded. Only when these basic requirements are met should the trial be considered for inclusion in grading systems, systematic reviews and clinical guidelines.

    These requirements might include:
    - No unblinded trials with purely subjective primary outcomes.
    - No outcome switching after approval that makes it easier to show 'success'.
    - No rolling preliminary trials into main trials with switching of primary outcomes.
    - No trials of psychological therapies where the 'control' group is not a genuine control where the only difference in patient experience is the active therapy being tested and is presented to patients in an equally or neutral way.
    - No trials in which the active arm has been promoted in recruitment as effective and where the patients can tell whether they are on the active treatment.
    - No data is included on monitoring of short and long term worsening and harms, which should include following up of patients who drop out of the trial.

    And treatment should not be allowed to be claimed to be effective if, for example:
    - The primary outcome measure is meaningless to the patient population, or where the claimed benefit is statistically significant but not clinically significant.
    - The 'benefit' is transient and not maintained at followup at least a year after the trial finished.


    etc etc.
     
  10. Jonathan Edwards

    Jonathan Edwards Senior Member (Voting Rights)

    Messages:
    13,271
    Location:
    London, UK
    That looks a good selection to go in a list. I think we need to be careful about exclusion categories though. There are probably exceptions to every rule that are easy to see the justification for. For instance, for an unblinded trial with subjective outcomes that studied six doses of treatment and found a classical sigmoidal dose-response curve with a four fold increase in successful outcome between dose 2 and 3 and between 3 and 4 but not much difference between 1 and 2 or 4, 5 and 6 then if the treatment was safe and met cost conditions it probably should be recommended. But the conditions are very narrow because once this trial has reported expectation bias relating to dose would colour further studies. Also in practice it is reasonable for the vast majority of conditions to wait until an initial dose ranging study has been replicated blind but there are exceptions like lupus or melanoma where recommendation can be fast-tracked.

    All in all I am against any rules because if you understand the reasons for evaluating a trial you do not need any rules.

    The overarching problem is competing interest. Cochrane seems to have sunk without trace on that one. NICE has had its problems but has passed muster this time - more or less!
     
  11. Adrian

    Adrian Administrator Staff Member

    Messages:
    6,478
    Location:
    UK

    I think rules are often necessary for those who don't understand - and unfortunately its those who really don't understand who feel the rules shouldn't apply to them and they should be allowed to use their intuition. Often guidance can be good but where if people deviate they have to explain why they are ignoring the rules.
     
  12. Caroline Struthers

    Caroline Struthers Senior Member (Voting Rights)

    Messages:
    821
    Location:
    Oxford UK
    Rules are also being applied by people who don't understand and used to justify their poor judgments as scientific. That is as much, if not more of a problem than people who don't understand not applying the rules. Even worse when the rules are invented by people who don't understand either.
     
  13. dave30th

    dave30th Senior Member (Voting Rights)

    Messages:
    2,182
    This parallels another version: "Some prefer to be in the center of a problem than on the periphery of a solution."
     
  14. Jonathan Edwards

    Jonathan Edwards Senior Member (Voting Rights)

    Messages:
    13,271
    Location:
    London, UK
    As Groucho Marx might have said. 'Well why don't we go for the centre of the solution?:nerd:'
     
  15. MSEsperanza

    MSEsperanza Senior Member (Voting Rights)

    Messages:
    2,855
    Location:
    betwixt and between
    I keep forgetting what I read and don't have sufficient knowledge of statistics, so will leave the discussion mainly to others.

    From skimming many points seem to me important and plausible. Especially, I share the criticism about GRADE that it allows downgrading only to very low certainty of evidence and not to zero certainty.

    I was wondering though if one needed to specify the category of outcomes that is of interest first. Depending on the category there might be different requirements to compile relevant evidence ?

    Don't know how those points are addressed in the GRADE handbook.

    Not able to word my thoughts better at the moment, so just leave here my notes:

    Does research on the following outcomes all need the same methodology to get results of relevant evidence?

    1) evidence for benefit of a treatment regarding improvement of the medical condition itself: regain qualitative and quantitative functionality (being able to resume specific physical and cognitive activities per se; increase time being able to do specific activities; ideally: recovery, return to school / work)

    2) evidence for helping not making a progressive condition worse or slow down progression / stabilizing fluctuating conditions

    3) secondary effects of the medical condition -- slow down / prevent deconditioning / counteract negative effects of unavoidable deconditioning

    4) improve quality of life

    5) maintain mental well-being

    6) adverse side-effects/ harm

    7) costs / cost-effectiveness

    For all those outcomes: consider both short term and long term effects/ consequences.

    Maybe rather off-topic but I was thinking that there still are many chronic illnesses for which no or no satisfactory treatment exists. People too ill to work, to go to school or to resume any productive activity deserve evidence-based health care nevertheless.
     
  16. Jonathan Edwards

    Jonathan Edwards Senior Member (Voting Rights)

    Messages:
    13,271
    Location:
    London, UK
    I think what this emphasises is that all sorts of different methods are relevant indifferent situations. There are some commonalities but also an infinite number of variations. We have to be able to make a useful judgment on the reliability of every new method variation. Most of the time that is not too hard you have seen what actually goes on during trials - the people side of things, how they behave, how easily they are tempted to cheat and so on.
     
    Hutan, alktipping, FMMM1 and 4 others like this.
  17. MSEsperanza

    MSEsperanza Senior Member (Voting Rights)

    Messages:
    2,855
    Location:
    betwixt and between
    Related to my post above:

    If a study is designed in a way that it can't provide sufficient evidence for a specific outcome that it aimed to investigate, can it still provide sufficient evidence for another effect (or non-effect)?

    That also would speak against a standardized GRADE-style approach to rate evidence as well as against a general exclusion of studies that don't meet certain requirements.

    One needed to look at each study individually and ask for which outcome it could provide relevant evidence (and for which it doesn't). Also perhaps the more 'hidden' outcomes?
     
    Last edited: Mar 6, 2021
    alktipping and Kitty like this.
  18. Barry

    Barry Senior Member (Voting Rights)

    Messages:
    8,385
    Although I can empathise with wanting to do this, would this not count as post-hoc analysis, given it was not in the original trial protocol. Something we have criticised other trials for. Would it mean that a further trial might be needed to provide valid evidence?
     
  19. Jonathan Edwards

    Jonathan Edwards Senior Member (Voting Rights)

    Messages:
    13,271
    Location:
    London, UK
    Yes we had a discussion about this with Hilda Bastion who argued that you should not throw away any trials just because they do not come up to standard on one particular point. I think she is right in principle but in practice if there is a fatalflaw in the method for extracting a primary outcome everything else is pretty much damage limitation or useful negative information. The fact that PACE shows that the original hypothesis makes no sense however generous you are with the results would be an example.
     
    alktipping, FMMM1, Kitty and 6 others like this.
  20. ME/CFS Skeptic

    ME/CFS Skeptic Senior Member (Voting Rights)

    Messages:
    3,494
    Location:
    Belgium
    GRADE rates each outcome separately. I think there is even an example of an unblinded trial where the subjective outcome was downgraded but the objective one (mortality) isn't.
     
    Hutan, alktipping, Dolphin and 4 others like this.

Share This Page