Independent advisory group for the full update of the Cochrane review on exercise therapy and ME/CFS (2020), led by Hilda Bastian

This is all that needs to be said.

Inclusion criteria, statistical analysis and all the other PACE flaws are great topics for a critique in how not to run a clinical trial. But from an overall review perspective either you agree with Jonathan's statement or you don't.

 

You can't reasonably disagree with the position that "Open label trials with subjective primary endpoints (or switched endpoints) are valueless."

We know for sure that in other illnesses open label trials with subjective endpoints generate an illusion of treatment efficacy.

In ME/CFS there is no reason to believe it's any different and we have one open label Rituximab clinical with subjective endpoints that generated what appeared to be good and long lasting positive responses to treatment, that all disappeared in the later double blind placebo controlled clinical trial. The people doing only open label trials with subjective endpoints have never shown that their trials don't suffer from this problem. They simply acted like the problem didn't exist.

Sharpe finally tried to respond to this criticism and appeared to claim that the questionnaires used are not affected by placebo responses (a ridiculous claim) and that placebo responses are too short lived to affect PACE trial outcomes (we know this is false from the aforementioned Rituximab trial).

 

The "compromise" is to simply lump all studies that don't meet basic quality criteria into a "low quality evidence" basket. If all studies fall into this group, then the study must simply conclude all the evidence is of low quality.

(The basic quality criteria would require studies to use outcome measures that are preferred by patients to have relevance (not researchers! - I've never met a patient who would choose to use the Chalder Fatigue Questionnaire in a trial...) and in the case of unblinded trials, this must be a composite of both objective and subjective measures of functioning.)

 

I agree. The only (possible) ethical trial on GET now would be a withdrawal trial. Ie. randomly allocating patients to either carry on with GET or withdraw from the treatment and receive alternative or no treatment. Even that I'm not sure would be ethical now in that it would require half the participants to continue with a potentially dangerous intervention.

 

When anyone who disagrees is labeled either incompetent or having vested interests by virtue of disagreeing with absolutist statements, then I think it's hard to have genuine discussion outside a bubble. According to that absolutist statement, for example, every trial of epidurals for pain relief in labor is "valueless" if women's rating of pain is a primary endpoint, so yes, I disagree with it. As I was a homebirther and never a care provider, you can't tar me with the treatment or professional status vested interest brush on that one. I've never done methodological research on the measurement of pain, so that brush is out. But I guess "or whatever" covers me!

That's not to say that I don't also understand the point about objective outcome measurements in ME/CFS, or that there aren't serious discussion points about how to approach trials where patient-reported outcomes are critical. Just that there can be good reasons people disagree with things that may seem incontrovertible in some discourses.

 

Isn't there another issue here? If studies have themselves already significantly influenced other studies, medical and government policies etc, then maybe they do need to be included, so the implications of that influence are not lost. But the veracity and scientific quality of those studies absolutely must be reviewed with a fresh and wholly unbiased eye, and with great competence. If this were properly done (unfortunately a big 'if'), then it would properly expose their flawed contributions to other studies, public health policies, etc. Cochrane would then be properly and objectively identifying the rightful influence (or not) of these studies.

There seems to be various options, the main ones I can think of being:

1. Include flawed influential studies but without competently identifying the flaws, Cochrane thereby erroneously boosting those studies' supposed credibility. The current situation so far as ME/CFS is concerned.
2. Exclude flawed influential studies, Cochrane competently identifying their flaws and implicitly calling into question those studies' credibility and influence.
3. Include flawed influential studies, Cochrane competently identifying their flaws and explicitly and realistically factoring in their negative impacts on a review's outcomes. In so doing, Cochrane could identify (to some extent quantify maybe) how the poor quality trials had negatively impacted the reviews findings, and conclusions could be drawn regarding their credibility, and maybe eligibility to influence.

I'm not a scientist, so I appreciate the above is likely naive, but it's how it seems to me. '1' is the worst of all worlds. '2' a lot better. '3' better still because it would not brush anything under the carpet, but I suspect very hard to achieve in practice.

ETA: Just to emphasise, I think it is important to consider the negative impact studies might have on a review's outcomes, due to poor quality, flaws, etc.

ETA2: I see @Trish earlier also touched on the point of studies not being excluded and so brushed under the carpet, but instead being more meaningfully included. Apologies if I've missed any others.

 

Dismissing the trial as valueless is the problem: for example, why would the fact that women were asked to rate their own pain mean the data on impact on cesarean section rate was valueless? (Not that I am saying it would be ok to disregard women's reports of their pain, their discomfort during the procedure, their satisfaction with care, etc etc etc.) It would be shocking if an ethics committee approved an injection into the epidural space of women in labor with a placebo in the injection: that's probably why it was the first example to spring to my mind. The injection into the area around a spinal cord alone has the potential to cause major harm, regardless of what's in the fluid injected. Any objective measure (like needing general anesthesia) couldn't possibly get at the question reasonably, and certainly not for questions women want to know like, how does it compare to other methods of pain relief.

Or consider something like heat or ice for pain from osteoarthritis. You can also measure something like how many painkillers did people take, but that is going to be a self-report, too. You can argue of course for how much weight to put on particular subjective outcomes in a trial, but to not dismiss the entire trial as "valueless" because patients were asked about their pain and it was a high-level outcome is not some weird outlier opinion.

 

Surely the trigger threshold for acknowledging the possibility of harms is (and must be) significantly less than that for safety acceptance? A car design must undergo all manner of intensive design reviews, testing and quality assurance, before attaining the very high acceptance threshold needed to be deemed safe.

Nothing like that same evidential threshold has to be met if evidence emerges that the design of car seems prone to spontaneously catching fire, for instance. There simply has to be enough evidence to adequately question if the prevailing high safety acceptance threshold is still being met.

Does the same not apply to medicine?

 

But if a valueless trial has been previously misconstrued as high value, and thereby significantly influenced importance policies and other trials, then it is worse than valueless, because its "value" in terms of knock-on influence is actually negative. If a trial's value is zero, then it makes no difference. But if a trial's value is non-zero, including if it is negative, then the impact of that needs to be addressed and exposed, else its negative impacts will continue to fly under the radar.

 

Yeah, that's a really good point - but there's a lot of diversity opinion about individual trials, and not every review for example only includes randomized trials either. What's more, quality isn't only a feature of the trial as a whole: the quality of evidence varies within a trial. (Here's a post I wrote explaining what I mean by that.) I don't think you can easily reject a trial for forever for any conceivable review. Totally agree that it would be ideal if every reviewer didn't have to re-invent the wheel (and potentially make avoidable mistakes): hopefully at the very least, we're moving to a future where at least it's easy to see what others judged before.

 

I agree with you in terms of being concerned with how some are approaching priority-setting, and that it could lead to a drift towards the priorities of trialists and those of the people who commission and pay for trials. (But just as a point: it's not supposed to be prioritized by the needs of patients alone - it's also meant to be viewpoints of clinicians and policymakers, for example. There are issues that patients mightn't prioritize that might be critical on a daily basis to the person who has to do a procedure, for example.)

 

I've replied elsewhere on the subject of trials being collections of data of unequal quality, so I won't reiterate that I don't think a problem in one area necessarily mean other parts of a trial aren't valuable (although there are problems that can systematically affect everything). But I couldn't agree more that it has huge implications for various parts of a review.

 

Somewhat similar, just more complicated. (That could be my answer to everything, eh?, "it's complicated"!)

 

And it is even worse than that surely, when you have open label trials with subjective primary endpoints, trialling treatments largely based on strongly skewing patients' subjectivity. Seems akin to trialling a treatment for fractured legs, where instead of setting the leg properly you instead give a heavy dose of morphine or something and then ask the patients to try walking and say how they feel.

 

Hopefully not .

 

And in any case the results might be very woolly given the damage from GET doesn't necessarily reverse once you stop.