BMJ: Rapid response to 'Updated NICE guidance on CFS', 2021, Jason Busse et al, Co-chair and members of the GRADE working group

Oh no. It’s even worse. It is more authors. Look who is here:
Per Olav Vandvik, Elie A. Akl, Sue Brennan, Philipp Dahm, Marina Davoli, Signe Agnes Flottorp, Paul Garner, Joerg J. Meerpohl, Reem Mustafa, Maria X. Rojes, Gordon H. Guyatt
McMaster University
1280 Main St. West, Hamilton, ON, Canada

Edit:
Competing interests: GHG is co-chair of the GRADE Working Group; SAF, EAA, PD, SB, MD, MR, JJM, RM, and MR are members of the GRADE guidance and working groups; POV is a member of the GRADE Working Group. This Rapid Response is not an official communication from the GRADE Working Group.

 

Very clever. The one, Busse,who is first named seems to be the only one, apart from PG, not directly associated with GRADE. He seems instead to have an interest in "insurance medicine".

It is however rather confusing. PG does not seem to appear in the competing interests section. However there are twelve authors and eleven COIs, but neither JB or PG are listed. I never was good at maths.

 

https://twitter.com/user/status/1365249758055587840

 

I hope someone (in a prominent position) can counter this publicly and get it published somewhere. I’m worried this is going to hurt NICE and the guidelines

 

@Caroline Struthers , you might be interested in this. If I was into conspiracy theories I might think that Cochrane dragging their feet about their review might have something to do with this.

 

SW must be racking up quite a bar bill.

 

I don't understand the finer details of GRADE but if it's true that it allows demonstratably flawed science to pass as adequate, then it's flawed and should be retired.

 

PG messaged me about the Exercise review on 6 July...

 

About the lead author, Dr Busse (from his McMaster page):

Note that the authors made the clever trick of separating 'lack of blinding' from 'subjective outcomes' into two unlinked paragraphs:

Noone ever said that the lack of blinding alone constitutes a problem, but that the lack of blinding for subjective outcomes is. The meta-epidemiological study on blinding that they cite also mentions conflicting results on the blinding of PROMs (bolding mine):

It is also noted that their meta-epidemiological approach could be unreliable or that their findings are imprecise. They recommend that the risk of bias from lack of blinding should continue to be assessed with currently available tools, such as Cochrane's RoB2.

 

Gordon Guyatt (Godfather of GRADE??) was the person who arbitrated on the wording of the amendment - so saying, I think, that there was still "moderate" evidence, according to GRADE of a "non-zero" effect on fatigue. The correspondence is here

 

Other than lack of blinding, this response mentions the fatigue outcome on the Cochrane review, as measured by the Chalder Fatigue Scale, but completely omits the problems that have been picked up with it. https://www.s4me.info/docs/CFQ-Critique-S4me.pdf

They blame NICE for downgrading the evidence due to indirectness, which of course NICE had to do to be consistent with their diagnostic criteria that require PEM. They say the Cochrane review on exercise therapy for CFS carried out a subgroup analysis for different diagnostic criteria and found no difference between subgroups. Obviously, the only criteria in the studies included in the review were Oxford and Fukuda -- none of which require PEM... But then they go on to blur ME/CFS with other conditions considered as MUS. No wonder they don't know or care about PEM.

Also, it is somewhat ironic that GRADE is used as a mean to justify not rating down the evidence on fatigue in exercise therapy due to imprecision and/or inconsistency, because @Michiel Tack referred to the GRADE handbook in his commentary to Cochrane to justify that it should...

 

NICE did use GRADE, but with a bit more intelligence/common sense

 

I think thesis great, because we have Gordon Guyatt, Mr GRADE himself, weighing in and saying his system would have rated PACE as reliable evidence.

It would have been easy to see the ME/CFS kerfuffle as a backwater in the evidence-baed world but I think this makes it clear that the NICE committee decision is a real threat to the cosy EBM system.

I don't think anyone needs to feel sympathetic to people who switch what they say according to which friends they want to bond with even if they have suffered from Covid for rather a long period. The lack of professionalism here is gobsmacking. These are people claiming to be experts on dispassionate evaluation of evidence and yet they are clearly the opposite.

I am reasonably confident that none of this will affect the NICE decision because, as Caroline says, NICE used GRADE and managed to get a sensible result nonetheless.

The pressure people who do not like the result are putting on NICE is a clear demonstration of the insanity of a system like GRADE. If it generates results that can be open to lobbying it is subjective and if it is subjective then there is no point in having what appears to be an objective numbering system.

 

How would GRADE rate the unblinded Rituximab trial?

 

If you mean the unblinded continuation study following the blinded phase 2 ME study then I don't think it would get as far as being assessed by GRADE because it was not controlled. The controlled trial was blinded.

 

The other thing is that there doesn't appear to a good reason to rely on crap studies/methodology i.e. to assess whether an intervention works. E.g. Fluge, and Mella, used activity monitors to assess rituximab. So rather than defending poor studies they could spend some time looking at objective monitoring [activity monitoring +?] and call for sufficient funding to incorporate this into studies. OK they'd probably have some excuses about assessing other psychological outcomes from interventions. However, if it's Government funded research then it should look at objective outcomes like increased activity, able to return to work, study ---- surely they would indicate "objective psychological" improvement.

Free meal ticket sums them up!

 

Well, it is great for someone looking for an example of how applying the GRADE system can lead to a problematic conclusion.
But for the ME/CFS patient community it's highly concerning that proponents of GET have managed to get Gordon Guyat to become involved and sign a statement such as this one.

Perhaps we should try to contact him, explain how GET actively tried to influence how patients report their symptoms and that GRADE provides no option to take such risk of response bias into account?

 

I'm thinking about submitting the following rapid response - any suggestions before I do so?

-----------------------------------------

I would like to respond to the comment by Jason Busse and colleagues as it includes some remarkable statements. The authors criticize the NICE guidance committee on myalgic encephalomyelitis/chronic fatigue syndrome (ME/CFS) for employing “a disastrous misapplication of GRADE methodology.” In a draft document, the committee rated the quality of evidence for GET as low to very low.

As an example of an “appropriate” application of GRADE, Busse and colleagues refer to a contested Cochrane review on GET for ME/CFS. This review rated the quality of evidence for GET in reducing fatigue as moderate. According to Busse and colleagues, this is “sufficient in GRADE methodology to justify strong favorable recommendations.” These statements are concerning for several reasons.

The Cochrane review cited by Busse and colleagues is currently being updated, following concerns about its methodology. Cochrane’s Editor-in-Chief, Dr. Karla Soares-Weiser explained that “this amended review is still based on a research question and a set of methods from 2002, and reflects evidence from studies that applied definitions of ME/CFS from the 1990s.” [1]

The review is far from an exemplary assessment of the evidence for GET. It highlights fatigue assessments made directly after treatment ended and downplays assessments that were made several months later (the latter formed the primary outcome for the trials that provide most of the data). Follow-up results do not support the recommendation that GET reduces fatigue. In addition, the review compares GET to a passive control condition where patients received less time and attention from healthcare providers. The review also focused on subjective outcomes and ignored negative results on objective measurements such as employment figures, activity levels, and fitness tests which all tend to show no significant improvement in the GET group.

There is reasonable concern that the reduction seen on fatigue questionnaires reflects response bias rather than a genuine reduction in fatigue. Patients in the GET-group, for example, received instructions to interpret their symptoms as less threatening and more benign. According to one therapist manual on GET “participants are encouraged to see symptoms as temporary and reversible, as a result of their current physical weakness, and not as signs of progressive pathology.” Treatment manuals also included strong assertions designed to strengthen patients’ expectations of GET. One patient booklet stated: “You will experience a snowballing effect as increasing fitness leads to increasing confidence in your ability. You will have conquered CFS by your own effort and you will be back in control of your body again.”

These instructions were not given to patients in the control group and result in a high risk of response bias. It was therefore reasonable for the NICE guideline committee to rate the quality of evidence for GET as low. Other reviews had previously come to a similar conclusion. [2, 3]

If the GRADE system was used as Busse and colleagues recommend, there would be a high risk that quack treatments and various forms of pseudo-science also provide reliable evidence of effectiveness in randomized trials. All that is needed is an intervention where therapists actively manipulate how patients interpret and report their symptoms. One example should suffice to clarify this point.

Suppose an intervention based on ‘neurolinguistic programming’ where therapists assume that saying one is fatigued, reinforces neural circuits that perpetuate fatigue. The intervention consists of breaking this vicious cycle by encouraging patients to no longer see or report themselves as fatigued. This example is not that far-fetched as there are already behavioral interventions for ME/CFS that are based on similar principles. [4] According to GRADE methodology, however, such attempts to manipulate how patients report their symptoms, form no reason to downgrade the quality of evidence of randomized trials, even if fatigue questionnaires are used as the primary outcome.

The first and foremost principle of rating quality of evidence should be to understand the specifics of what is being assessed. One has to understand the intervention and the way it impacts patients. By providing a standardized checklist and algorithm to assess quality of evidence, the GRADE methodology discourages researchers from studying the details of what happens in randomized trials. The rapid response by Busse and colleagues is an example of how this approach might result in questionable treatment recommendations.

References

1. Cochrane. Publication of Cochrane Review: ‘Exercise therapy for chronic fatigue syndrome.’ https://www.cochrane.org/news/publication-cochrane-review-exercise-therapy-chronic-fatigue-syndrome. Accessed 26 Nov 2019.

2. Vink M, Vink-Niese A. Graded exercise therapy for myalgic encephalomyelitis/chronic fatigue syndrome is not effective and unsafe. Re-analysis of a Cochrane review. Health Psychol Open. 2018;5:2055102918805187.

3. Tack M, Tuller DM, Struthers C. Bias caused by reliance on patient-reported outcome measures in non-blinded randomized trials: an in-depth look at exercise therapy for chronic fatigue syndrome. Fatigue: Biomedicine, Health & Behavior. 2020;8:181–92.

4. Reme SE, Archer N, Chalder T. Experiences of young people who have undergone the Lightning Process to treat chronic fatigue syndrome/myalgic encephalomyelitis--a qualitative study. Br J Health Psychol. 2013;18:508–25.