BMJ: Rapid response to 'Updated NICE guidance on CFS', 2021, Jason Busse et al, Co-chair and members of the GRADE working group

Ah, thanks.

Do you mean according to the GRADE system or is this your critique of GRADE?

What Busse seems to say in his Tweet and I might confuse now but thought I also read in the handbook is that generally it's better to make a weak recommendation than no recommendation?

But then, if the quality of evidence for a desired treatment effect is very low to low, it perhaps should be a weak remmendation against the treatment and not in favor?

Again, I likely confuse things now, and what Busse says and what the handbook says often seems to differ.

 

I think GRADE simply gives guidance to committees and panels in cases where they would like or are forced to make recommendations. But it doesn't suggest that recommendations should be made.

That seems like the only reasonable explanation.

Does NICE have to make a recommendation for or against all the various treatments that have been tried in ME/CFS? I don't think so: it can simply say that the evidence is too weak in most cases, and focus on the exceptions where current evidence does tell us something. That's what all reports and reviews seem to do.

 

Busse's complaint seems to be non sequitur. He thinks it is unfair to recommend against use of GET when there is some weak evidence of benefit. But surely GRADE makes it clear throughout that where other factors come in to play like concerns about harm that there may be no direct relation between the level of evidence and the recommendation. Busse is trying to impose a naive logic on GRADE that does not stand up under any sensible interpretation.

But what I think has to be kept in mind is that we have no evidence base for the GRADE system itself. GRADE is just what some self-appointed 'experts' think is the way they would make decisions. It is completely contrary to good scientific practice, which is based on evidence. Uniformity is not objectivity, it is just uniformity - which might be good or bad and is almost certainly bad if it is being imposed on a complex decision-making process in a way that presumes to substitute for rational argument itself.

As Robert distilled it, GRADE must be crap because it has been empirically shown to be crap.

 

GRADE has a lot of issues but I don't think it's the main problem here. It's the authors.

Most of what is in the handbook seem reasonable, nothing as absurd or counterintuitive as what Busse et al. are proposing.

GRADE doesn't say that committees should make recommendations for a treatment even if the quality of evidence is very low.
It also doesn't say that blinding is not an issue (quite the opposite: it suggest downgrading the quality of evidence in such cases unless the outcome is very robust, like mortality).

 

I strongly but amicably disagree, @Michiel Tack. GRADE is an obscenity. It replaces rational analysis with pseudo-arithmetic. There is no possible justification for that. And it get things badly wrong. It is an abhorrence. It is totally counterintuitive to replace direct argument from evidence with rules that are supposed to reflect direct argument from evidence. It is potty.

I agree that Busse is trying to squeeze blood out of the GRADE stone but the basic problem is believing that a system like GRADE has any legitimacy whatever. Defending GRADE is a blind alley that I would advise against exploring.

 

It is not just about evidence of benefits, low certainty or otherwise. It is about balance of risks, balancing against the risk of harms. If you have evidence of harms with similar certainty or more than evidence of benefits, then it is not the same. Evidence of benefits is not the be all and end all.

If I know that without treatment X I am going to die within a week, but that with treatment X there is a 10% chance I might recover, then I will happily go with treatment X.

But if for a different ailment I know that without treatment Z I am going to remain moderately incapacitated indefinitely, but that with treatment Z there is a 10% chance I might recover, but that with treatment Z there is also a 10% chance it will render me bedridden indefinitely, that is a very different judgement call. And definitely not to be glossed over or obfuscated by guidelines or professionals, pretending there is no such balance of risks in play.

 

This seems to be what the GRADE handbook says (1.2 Separation of confidence in effect estimates from strength of recommendations, bolding and underlining mine):

And it is exactly what NICE did, in particular by factoring the survey on GET and CBT from Oxford Brookes University into their decision -- which, differences in evidence quality set aside, had more responses (2274) than the total number of participants across the trials included in the Cochrane review (1518).

I don't understand Busse et al's arguments. They contradict what they wrote in the GRADE handbook when they say that NICE listened to patients -- or, as they frame it, "NICE rejected the evidence (...) because of (...) anecdotes from patients" --. No, NICE perfectly followed the GRADE handbook, whereas its authors obviate from their own recommendation of valuing input from (many) patients by arrogantly belittling it as "anecdote".

(In the quote, I kept the point "interactions between the work of implementing the intervention, the intervention itself, and other contexts the patient may be experiencing" in bold: if someone with ME/CFS follows a course of exercise therapy, it uses all of their energy and that negatively interacts with their ability to do other important daily activities.)

 

There isn't that much arithmetic involved, simply rating different levels of evidence from very low to strong.

Could you give an example of something that is not simply wrong but badly wrong/abhorrence? in the GRADE handbook

 

There isn't any real arithmetic but there is enough pseudo-arithmetic to matter. The pseudo-arithmetic consists of giving grades of reliability 1,2,3.. and then saying that you are allowed to shift one or two numbers on the scale - as if this were some sort of 'objective numerical' scale when it is entirely non-objective and non-arithmetical.

What is badly wrong is the idea that instead of looking at evidence and deciding on reliability you should artificially categorise trials as reliable on the basis of one or two features (randomised controlled trials) and then shift your assessment according to some arbitrary rules of numbers. The system is supposed to provide a way to mimic the 'Delphic' level of decision making as defined by fiat by the GRADE inventors. But human beings do not make decisions like this.

When buying a car you do not say that a Mercedes E Class must be a good car but that you are allowed to downrate it if it has some bad features like being old or done a big mileage or has not been serviced regularly. You look over the car in detail and say 'this car does not pass the new emission regulations so it is a write off' or something like that. Nobody uses the sort of Pseudo-decision tree that GRADE provides.

So the relevant example is that if a trial has subjective endpoints and is unblinded it can be assumed to be useless in terms of providing reliable evidence of efficacy unless there is some important mitigating factor - and if there is the implications of that factor will depend entirely on what it is. As GRADE stands there is no means to downgrade a randomised controlled trial to very low quality evidence simply on the basis of being unblinded with subjective outcomes. If I remember rightly you can downgrade by one or two points but not three - for no rational reason whatever. You have to wangle things the way the NICE committee did. So GRADE is badly wrong on what is perhaps the most important point of all.

What is so extraordinary looking at the Busse and Guyatt comment is that not only does Simon Wessely admit to not understanding why trials with subjective outcomes need to be blinded but nor todo the GRADE people. They make comments about subjectivity and blinding separately, exactly as Wessely and co have.

What is abhorrent about GRADE is that it replaces real rational interpretation of evidence with an arbitrary recipe for pseudo-decision making because people think that makes it 'objective'. This thinking is completely wrong and needs to be expunged from biomedical research. You do not make better judgements by making rules.

The clearest example of this is in the construction of rules of logic by people like Leibniz and Russell. The test of whether or not rules of logic are correct is that they make sense. You cannot say something makes sense because it follows the rules. It is the other way around. Within the very austere notation of formal logic you can find rules that seem to work perfectly - if you ignore Russell's paradox and Gödel's theorem that is. Carnap tried to find a way of making logic work within the framework of everyday language and failed completely. It is understood in logic circles that when using ordinary language it is impossible to construct watertight rules that actually reflect the way we use language to decide if things are true or false. The gold standard can only ever be what the most rational person can persuade others is the most rational interpretation. As soon as you try to be Procrustes and force decisions into rules you debase your currency.

The simple upshot of this is that nobody should be arguing that NICE is doing the right or wrong thing for the general public because of its use of rules. It should be judged on whether or not it is making the most rational interpretation of the evidence anyone can come up with. As I pointed out in my testimony the reason why these trials are no use is a matter of simple rational argument from evidence readily available.

 

An additional point is that within medical science formal debate on the most rational interpretation of evidence very rarely occurs and most medical academics are uncomfortable with it - probably because it might expose their inability to think clearly. In contrast it is something we do a lot here and very successfully. It is also something that occurs in a well supervised court of law with an intelligent judge. Courts led by judges (we can forget juries) do not always come to the right decision but very often they do a good job. Medical academics mostly try hard not to.

 

Will try to catch up with this thread later.


Just saw this:

Code:

https://twitter.com/JasonWBusse/status/1367109536478351373

And in addition to Michiel's reply....

Code:

https://twitter.com/MTackCVS/status/1367113830803918856

-- thank you @Michiel Tack for your persistence --

... thought it could be interesting to recall the conflict between the then Cochrane editor-in chief and the review authors:

Form Tovey's last released e-mail from 29.05.2019:

Not sure anymore and not able to check now about how the follwing related to that suggestion:

[My bolding]

Anyone recalls who acted as arbitrator in that conflict?

Edited to add: Michiel wrote about that too -- can't find the link to his blog now, but here's a quote:
https://www.s4me.info/threads/cochr...ersion-october-2019.11564/page-10#post-207196

 

"That would be a disastrous misapplication of GRADE."

But in what senses?

As I understand it all applications of GRADE are misapplications since the tool has no validity - and I think at least Caroline would agree.

(It is far from clear that the example would even be a misapplication within GRADE's system since it acknowledges all sorts of ways you can decide to modify evaluations with context.)

I see no reason to think that a misapplication of GRADE would be disastrous for the public for whom the assessment is being made. That would only be the case if the decision was not in line with evidence (it is in line in this case).

I can certainly see the misapplication being disastrous for GRADE, since the debate has highlighted both the subjective user-dependent nature of GRADE assessments (which it is presumably designed to avoid) and the fact that GRADE fails to deal with unblinded trials with subjective endpoints in a meaningful way.

 

https://twitter.com/user/status/1367149581453123590


Somehow I doubt this person would justify this logic as a generalizable case, that if it's published it must be true. We have reached meta-cherry-picking: cherry-picking what gets cherry-picked. Especially with the new Cochrane policy of giving a veto to authors over retraction, a veto that is itself cherry-picked and would no doubt play differently in a different context. Especially when Cochrane explicitly does not stand behind that decision, but that takes more than a few tweets to explain.

Rules are meaningless when they are arbitrarily enforced. EBM is basically built on the arbitrary application of rules and standards. The precise set of rules is also irrelevant, since exemptions rule the rules.

 

https://bestpractice.bmj.com/info/toolkit/learn-ebm/what-is-grade/


https://gdt.gradepro.org/app/handbook/handbook.html

 

So what on earth is that supposed to mean. What is a reproducible framework, other than one that can be printed on several pieces of paper? And in what sense is it transparent - or is that the paper too!

So it is just a set of hoops to jump through as in Lilliput it seems.

And since the assumption must be that you need rules to follow because the people doing the assessment are not as good at it as the people who invented the rules then the fact that they may come to different conclusions makes it a bit worrying - since they do not fully understand what they are doing by definition.

 

My thoughts also.

And for a process that is supposed to make things easier and more straightforward, the manual is far from concise:

https://gdt.gradepro.org/app/handbook/handbook.html

 

Yes, seems the equivalent of painting by numbers and submitting it to the Tate (albeit they might accept it).

 

Just after I'd been saying how difficult it is for patients to engage in discussions about complicated topics on twitter, here's @Michiel Tack showing how to do it.

I'd just been googling 'Medically Unexplained Syndromes Study Group Guyatt' and reading some of the papers co-authored with Busse. I now have a better understanding of why Larun & co would have suggested Guyatt be involved in assessing their work.

 

I really wish you guys were put in change of reforming science and medical practice. You'd have it whipped into shape in no time. I don't think you would find it difficult. If you have the right intentions everything is obvious. Flim flam, complication and impossibility is created by those with nefarious intentions.

 

Yes, it is revealing. The essence of Busse and Guyatt's beliefs about fibromyalgia appears to be:
'Belief that fibromyalgia is caused by an underlying physical lesion is associated with concern that exercise is harmful for this patient population; both beliefs could be detrimental to fibromyalgia patients.'

It looks as if Turner-Stokes' and Wade's editorial identified the rotten core of the GRADE thinking machine rather well, even if unintentionally.