BMJ: Rapid response to 'Updated NICE guidance on CFS', 2021, Jason Busse et al, Co-chair and members of the GRADE working group

Ah yes. That. Well after a Dutch(?) study using activity monitoring failed to show what they wanted I believe the use of activity monitoring was dropped from PACE (or was it FINE). One of them anyway.

Indeed and in PACE they did. Small negative impact on ability to work and the numbers relying on benefits slightly raised. This was dismissed with some hand waving and vague claims about the economy being at fault in an obvious attempt to fudge that, economy or no, the participants ability to return to work wasn't improved. So if we could just move along. Nothing to see here.

Edit omitted word.

 

One remarkable detail in all of this is that there is absolutely no discussion of the substance of the "treatments", that they explicitly aim to influence and change how participants respond to questionnaires. The entire process of the intervention is to make people perceive themselves better then ask whether they perceive themselves as better than when they came in, after being thrown weeks or months of explicitly positive statements about the product:

Zero discussion of this, that the entire point of the interventions is to explicitly influence the outcome used to determine success. Because the substance here is entirely irrelevant, nobody involved in this actually cares about any of this, the ends justify the means. This is why they don't want objective outcomes or anything that will break the con. Because that's what this all is: a con, a confidence game, simply stating endlessly that you are winning and judging whether you are winning by how often you remind yourself that you are winning, arbitrary targets of no real value.

This is why this ideology is absolutist, it does not allow for even the possibility that it is not universally effective, despite there being zero evidence that it is effective at all. It can't be adjunct or supportive, because if there is any medical alternative at all no one would choose this delusional nonsense. They all know this, that if given a choice patients would never even pay attention to this, there can be no competition or evaluation because the illusion, the con, is the entire thing. Just as true when they speak to us as when they speak to funders, the con is the whole thing.

Honestly I think more attention should be given to the substance of those "interventions", their booklets and the things participants are told. They should be evaluated formally, compared and assessed for what they are, the same way as the design of an electrical device would be pored over with zealous attention to what every part does. Because no middling company would ever pay money to a shyster assuring them that their product will sell like hotcakes based on such biased methodology, it is fully biased. And yet these people are content with forcing this onto millions with zero interest in knowing about the real life outcomes, all they want is for the con to be accepted and for it to go on and on.

 

I see it as great for people who would like to see unfounded opinions out in the open rather than hidden behind dissembling and obfuscation.

And I would not be so concerned about getting Guyatt involved because there are several people clearly more intelligent than Guyatt on the committee who can counter this absurd commentary - some of them members here.

The more this is out in the open the less the chance of a repeat of the 2007 guideline fiasco. I think the two objectives are closely linked.

 

The other point @Michiel Tack is that what we are seeing now is enthusiasts for bad trials fighting amongst themselves. Turner-Stokes and Wade have upset Guyatt and Garner. Bring it on. The more these people argue with each other the more they will expose the idiocy of their analysis. Guyatt and co are to attacking NICE here so much as attacking T-S and W for bad-mouthing their GRADE system.

 

Oh great. We now have to take on all of EBM. And it's never going to stop. Hooray!

Who is paying attention though? All the people with power have an incentive to maintain the pretence that things are better than they are.

I don't think that replying to Garner on twitter is likely to be useful. I worry the patients have created some needless problems here.

 

He believes he cured his ME with exercise. This is the logical next step. I don't think it has anything to do with any patient responses.

 

Couldn't agree more.

Let's face it anyone who designed an electrical device that did not do what it was supposed to in the way it was supposed to would be subject to an investigation by Trading Standards or some similar authority.

An electrical device that not only failed to perform as advertised but actually caused harm would result in compensation claims.

The thing I find striking in all of this is the lack of interest in the poor reporting of harms. The importance of the potential to cause harm seems to rank way below the importance of what angle we need to use to squint at it to make it look vaguely worth the salaries of the people peddling it.

 

At least now it is clear that the ME/CFS community always had to take on all of EBM. Isn't it better for that to be transparent? But the most likely sign that it is going to stop is surely the EBM people starting to complain about what might have seemed something trivial - an ME/CFS NICE review - and bickering amongst themselves?

The EBM people are paying attention. They are ruffled. And who has power in this case - the NICE committee, who as far as I can see understand the reality of the situation. Peter Barry has no incentive to maintain anybody's pretence, nor does Ilora Finlay. Nor do the other substantive members who have concluded that the EBM people have no idea what they are doing so a realistic assessment must be made.

If the substantive NICE guideline is much the same Guyatt and co have blown their cover rather badly and I might write a book to discuss that. If NICE caves in at least we know exactly what has been going on and I might write a book to discuss that.

I spent my life in biomedical science ignoring 'the people in power' and getting on with making progress. I made the progress. The 'people in power' continue to witter on but some people with arthritis are better off. Once I have got shot of some major commitment in immunology and philosophy this spring I am minded to follow Robert's cajoling and write that book after all. And this time it won't just be about PACE.

 

NICE fail with implementing GRADE guideline methods for CFS/ME-this is in plain sight to any guideline methodologist; makes me wonder whether NICE followed standard guideline procedures to assure consensus

Thus speaketh a man from Lilliput. Since when did assuring consensus constitute the right answer?
That foolish consistency again...

 

Your response is excellent @Michiel Tack. Here are some additional points you may want to consider adding, if you feel that these do not go into too much detail and could thus water down the strength of your arguments.

Fatigue outcome in the Cochrane review

Busse et al emphasize on the moderate-to-large SMD for the fatigue outcome following GET:

You can cite your own commentary to the Cochrane review [1] in which you point out that, when removing the trial by Powell for the purpose of heterogeneity, the SMD increases from -0.66 to -0.44 and that this no longer represents a clinically meaningful improvement on the Chalder Fatigue Scale. And that, combined with the issue of imprecision, the evidence for fatigue post-treatment should be rated as 'low' instead of moderate according to the GRADE handbook.

On the topic of 'fatigue questionnaires' you may also want to briefly remind the issues with the Chalder Fatigue Scale, namely interpretation problems (questions that are not relevant to fatigue, focus on change in fatigue versus intensity) and the ceiling effect. [1, 2, 3]

Different diagnostic criteria and their effects on fatigue improvement

You can counter the arguments that, in the Cochrane review subgroup analysis on different diagnostic criteria, there were no differences between subgroups on the fatigue outcome, with the Agency for Healthcare Quality Research's addendum to its review of treatments for ME/CFS. [4] It states (bolding mine apart from paragraph titles):

Further, a recent systematic review on the evidence base for physiotherapy in ME/CFS shows that there is no evidence available when considering PEM. [5] This matters when it comes to directness because NICE now requires PEM in their proposed diagnostic criteria, so it can't recommend an intervention for which there is no evidence of effectiveness based on these new criteria.

The meta-epidemiological study on blinding includes few meta-analyses of behavioural interventions with PROM

Moustgaard et al. [6] found that, for the group "Effect of blinding patients in trials with patient reported outcomes" (Ia):

("An ROR lower than 1 indicated exaggerated effect estimates in trials without blinding")

According to the appendix (table 4), in this group, 3 meta-analyses (out of 18, 16.7%) studied a behavioural intervention, totalling 50 trials (out of 132, 37.9%) and 19.13% of the weight of the analysis:

- Neuropsychological rehabilitation for multiple sclerosis, 3 trials, ROR = 1.37 (95% CI: 0.22 - 8.66)

- Music for stress and anxiety reduction in coronary heart disease patients, 5 trials, ROR = 1.58 (95% CI: 0.42 - 5.85)

- (Not a treatment) Decision aids for people facing health treatment or screening decisions, 42 trials, ROR = 1.93 (95% CI: 1.50 - 2.48)

The rest only included meta-analyses of drug trials. It is not unreasonable to think that, if more blindable behavioural interventions (e.g. as proposed by Busse et al., "attention control in a trial of CBT") where positive expectations can be repeatedly promoted by the therapists (as in your example with the GET manual) had been included for this group, the effects of performance (and detection) bias would have tipped over the balance towards a lower ROR.

For the case of unblindable behavioural interventions, you'll find this as the first example on the page about performance bias in the Catalog of Bias of the University of Oxford's Centre for EBM:

[1] https://www.cochranelibrary.com/cds....pub8/detailed-comment/en?messageId=266353165

[2] Kirke KD. Measuring improvement and deterioration in myalgic encephalomyelitis/chronic fatigue syndrome: the pitfalls of the Chalder Fatigue Questionnaire. J R Soc Med. 2021 Feb;114(2):54. doi: 10.1177/0141076820977843. Epub 2020 Dec 15. PMID: 33319615; PMCID: PMC7879015.

[3] https://huisartsvink.files.wordpress.com/2018/08/wilshire-mcphee-cfq-cde-critique-for-s4me-final.pdf

[4] Beth Smith ME, Nelson HD, Haney E, et al. Diagnosis and Treatment of Myalgic Encephalomyelitis/Chronic Fatigue Syndrome. Rockville (MD): Agency for Healthcare Research and Quality (US); 2014 Dec. (Evidence Reports/Technology Assessments, No. 219.) July 2016 Addendum. Available from: https://www.ncbi.nlm.nih.gov/books/NBK379582/

[5] Wormgoor MEA, Rodenburg SC. The evidence base for physiotherapy in myalgic encephalomyelitis/chronic fatigue syndrome when considering post-exertional malaise: a systematic review and narrative synthesis. J Transl Med. 2021 Jan 4;19(1):1. doi: 10.1186/s12967-020-02683-4. PMID: 33397399; PMCID: PMC7780213.

[6] Moustgaard H, Clayton GL, Jones HE, Boutron I, Jørgensen L, Laursen DRT, Olsen MF, Paludan-Müller A, Ravaud P, Savović J, Sterne JAC, Higgins JPT, Hróbjartsson A. Impact of blinding on estimated treatment effects in randomised clinical trials: meta-epidemiological study. BMJ. 2020 Jan 21;368:l6802. doi: 10.1136/bmj.l6802. Erratum in: BMJ. 2020 Feb 5;368:m358. PMID: 31964641; PMCID: PMC7190062.

 

Would it make sense to spell out that it is the combination of subjective outcomes as primary endpoints in non-blinded trials that allows this problem? And that the authors don't acknowledge that it is this combination that isn't able to produce reliable evidence, and not one of those factors alone?

- Maybe to understand the specifics of what is being assessed is not what is needed but to take the specifics into account?

- Similarly, I think it's not necessary to understand the intervention.

Perhaps something like:

One has to take into account how the intervention is supposed to work and which information and instructions are given to the trial participants and, in case of non-phamacological interventions, also to the therapists.

(Edited to add first question. I see you made the point in your response to Turner-Stokes and Wade, as did Brian Hughes. I think though it would be good to repeat it in this response, because Busse et al seem to have ignored the previous responses and go on to address both points seperately.)

 

Once the NICE guidelines are out, can they be “challenged” by people and forced to re-do the whole process? Or, is it the case that even if there’s disagreement, the guidelines won’t be affected and they’ll stay as they are? this GRADE stuff - in practical terms, how can it affect the guidelines process? Does anyone know? (It’s making me feel stressed and anxious!).

 

Thanky you @Esther12 -- very helpful.

I won't submit a response myself, but if anyone finds the following useful for an additional response, feel free to use, rephrase, add a good closing sentence etc.:

(Latest amendments are bolded)

In their response Jason Busse et al agree with Lynne Turner-Stokes' and Dereck Wade's view that the NICE draft guidelines on ME/CFS applied wrong criteria for the evaluation of the available treatments. Both also agree that contrary to the evaluation by NICE, sufficient evidence exists for benefits from GET for people with ME/CFS. However, Busse et al disagree with Turner-Stokes and Wade with regard to what led NICE to their alleged errors. In the view of the former, not the GRADE methodology is to blame, but its wrong application. Unfortunately, in their defense of the GRADE tools the authors misrepresent the work of both the NICE evaluation team and the guideline committee.

Firstly, the authors point out that GRADE doesn't necessarily downgrade for diverse sources for risk of bias, if they are accompanied by apt additional modalities. In referring, among others, to the lack of blinding and the use of subjective outcomes, they reiterate the misunderstanding that downgrading happened for either of those methodological features per se. As was repeatedly said in the previous comments, e.g. by Brian Hughes and Michiel Tack, it's not one of those features alone that accounts for downgrading the quality of evidence but the combination of non-blinding and the use of subjective outcomes as sole primary endpoints in a clinical trial. That both features, if combined, can't be adequately compensated for risk of bias and thus a trial working with this combination is unable to produce reliable evidence is also the main point made by Jonathan Edwards in his expert testimony provided to the NICE committee. [1],[2]

Secondly, Busse et al state that NICE did "reject the randomized trial evidence focusing on patient-important outcomes on the basis of theoretical arguments and anecdote". However, the quotation they refer to from the introduction to the corresponding evidence review just explains why NICE decided to consider additional evidence from patient surveys. [3] The downgrading of the evidence of GET and other trials had nothing to do with patient surveys and the report of harm. Each were assessed separately. [4]

Finally, Busse et al refer to the Cochrane review [...]*. They fail to mention that this review is currently in the process of being updated because, so the Cochran Editor-in-chief, Karla Soares Weisser, "a new approach to the publication of evidence in this area is needed" [5]. It's interesting that the largest and most-cited trial in this area, the PACE trial, was not even published as a Randomized Controlled Trial but just as a Randomized Trial. [6]



[1] "The main reasons for downgrading were risk of bias, indirectness and imprecision. There was a lack of blinding in the studies due to the nature of the interventions. This, combined with the mostly subjective outcomes, resulted in a high risk of performance bias. The committee considered this an important limitation when interpreting the evidence." [Bolding added.], Evidence review G - Non pharmacological management, p.317
https://www.nice.org.uk/guidance/GID-NG10091/documents/evidence-review-7

[2] [reference expert testimony; see also Tack M, Tuller D, Struthers C 2020...]

[3] .... https://www.nice.org.uk/guidance/gid-ng10091/documents/evidence-review-7, p.5

[4] ... https://www.nice.org.uk/guidance/gid-ng10091/documents/evidence-review-7, pages [???]

[5] ...https://www.cochrane.org/news/publication-cochrane-review-exercise-therapy-chronic-fatigue-syndrome

[6] ...[ reference PACE trial]

[...]* Edit: Removed misleading content, see:

(Edited several times for clarity -- apologies.)

Edit X: Apart from checking the quotes from the NICE evidence review, I wrote this from memory, so not sure about accuracy.

 

Just before looking at Michiel's comment I wanted to re-read the RR, and now I have some questions for other people.

4. Karanicolas PJ, Farrokhyar F, Bhandari M. Practical tips for surgical research: blinding: who, what, when, why, how? Can J Surg. 2010; 53(5): 345-8.

How is an attention control sufficient to claim that a CBT trial has been blinded? That paper they cite doesn't seem to support that point, instead saying things like:

The cited paper concludes:

Are Busse et al trying to redefine what a 'blinded' trial is?

What continuous variables are they referring to? Are they assuming outcomes like the Chalder Fatigue Questionnare or SF36-PF count as coninuous? Aren't they just aggregated scores of discrete variables (answers to questions)? I've got the impression that EBM people can tend to want to act as if more of their data is on continuous variables than is truly the case, but also, I don't know what I'm talking about so feel nervous acting as if world renowned experts are talking bollocks. Anyone else got a view?

Larun just compared results between CDC 1994 criteria and Oxford criteria. NICE downgraded trials that did not use a criteria that required PEM, so that's just a different issue:

https://www.nice.org.uk/guidance/gid-ng10091/documents/supporting-documentation-4

They made the disputable decision to not do so at that point (for fatigue scores), but did at follow up.

You can see the Grade ratings for GET vs control in Table 1 of the Larun review, with more downgrades there: https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD003200.pub8/full

All are rated low or very low except 1 - fatigue scores at end of treatment.

Who are they to decide that this change in fatigue questionnaire scores, which was no longer significant at follow-up, will be crucial to patients - particularly given the problems mentioned by Michiel (and many, many, many others)? Which patients told them that this was most important to patients?

They seem to have the same GRADE ratings tables for exercise interventions as any of the other treatments looked at.

On page 97 they have GRADE ratings for CBT and GET vs usual care. On page 126 they have GRADE ratings for pragmatic rehabilitation (classed as a form of GET in the Larun review) vs supportive listening and page 128 vs usual care. On page 137 they have GRADE ratings fro GET versus standard care (including fatigue), on page 142 vs fexibility/relaxation treatment.. and so on until page 154 when they move on to 'other exercise interventions'. https://www.nice.org.uk/guidance/gid-ng10091/documents/evidence-review-7

I don't know the exact requirements for a 'GRADE evidence summary of findings table'. Does anyone else? Has NICE done anything unusual here?

Are Busse et al misrepresenting something there, before they go on to misrepresent the reasons for it?:

As has been pointed out, this isn't what the authors were telling us. I haven't gone through the evidence review to look at what was said about all the different outcomes, but that would probably be useful to do. There was a lot of tables though.

 

Good point. Does he not then have a competing interest which should have been declared?

 

Again apopogies for multiple edits.
I tried to check the point about subjective outcomes in non-blinded trials and added this reference:

[1] "The main reasons for downgrading were risk of bias, indirectness and imprecision. There was a lack of blinding in the studies due to the nature of the interventions. This, combined with the mostly subjective outcomes, resulted in a high risk of performance bias. The committee considered this an important limitation when interpreting the evidence." [Bolding added.], Evidence review G - Non pharmacological management, p.317
https://www.nice.org.uk/guidance/GID-NG10091/documents/evidence-review-7

Also, the latest amendments I made in my text are now bolded.

 

"First, this is not the reason NICE rejected the evidence – as we have quoted above, it is because of theoretical objections and anecdotes from patients." oh la la

That could just as easily read "First, this is not the reason NICE rejected the evidence – as we have quoted above, it is because of theoretical objections and anecdotes from researchers pushing their thin agendas."