Independent advisory group for the full update of the Cochrane review on exercise therapy and ME/CFS (2020), led by Hilda Bastian

A million times yes!!! That's what Cochrane cannot seem to wrap their collective review-obsessed heads-in-the-sand around. When they bang on about "advocating for evidence", they don't mean advocating for better primary research, they mean advocating for their own way of doing reviews which is using GRADE, Risk of Bias, exhaustive search strategies, meta-analysis techniques, and all the other things their methodological boffins invented to produce systematic reviews and systematic reviews only. Even CONSORT (guidance for reporting randomized trials) was invented to meet the needs of systematic reviewers only, not to improve how trials are done to make the results actually mean something, and be an ethical use of time and money.

 

Reviewers generally pick Cochrane. So ones with COI can take advantage of Cochrane's volunteer model to do a review on pretty much whatever they like. All they have to do is convince the coordinating editor of the review group that it's a good idea. In the case of the Exercise and CBT for CFS reviews, they were managed by the Common Mental Disorders Group that was led by Simon Wessely and then his protegee Rachel Churchill. At least I think this dysfuntional review group structure is being dismantled with the "New Order" coming in the aftermath of the NIHR pulling the funding. I hope they can find a way to mirror what NICE does - I do not see why that would not be possible, but who would police the review team selection to make sure there was no COI sneaking in there. NICE? I will write and ask them...

 

I've only skimmed this but it reminds me of what @Brian Hughes has been saying i.e. these studies simply replicate previous flawed studies (un/inadequately blinded plus using subjective outcome indicators - questionnaires). Yes, why isn't this designed out at the beginning and how come the ethics committees don't point out that the study can't tell us anything?

I think PACE raises other questions e.g. actimetry/activity monitoring was supposed to be used - thereby dealing with the inevitable lack of blinding. The objective outcome indicator (actimetry/activity) was dropped mid-way through the trial. There are lots of other questions re PACE see @Jonathan Edwards comments here* but yes why didn't the ethics committee deal with these pre/post trial?

*https://www.s4me.info/threads/what-...e-unherd-tom-chivers.22082/page-2#post-367778

 

I suppose if it were used to positively confirm that something does not work, then that might be valuable as evidence of it not working?

 

Indeed some of the best testers are people who find ways of doing things to products no normal person would dream of doing to it. But if it can be done, then someone somewhere will do it.

 

Quite. Moderate quality evidence ... for what?

 

Is there any notion of independent auditing of clinical trials, at various stages of their progress. Manufacturing, engineering, etc has all manner of such things.

So for instance, the company I work for ships large laboratory instruments all over the world, by air. The risks of such goods being tampered with at any point during or after manufacture, and between there and the aircraft, with all the intermediate storage areas, is very non-trivial. So there are highly regulated procedures and regulations in place, for all people involved, in any part of that chain. Auditing and regular training (including refresher training at least every 2 years) is mandatory, and if you do not do the training in time then you are only allowed on site if accompanied, no matter how long you have worked there.

It is all about the risks to human safety. It seems to me that some refresher training for some well known scientists is long overdue, and without it they also should not be allowed anywhere near clinical trials, together with good independent auditing to confirm they apply what they should have learnt.

 

well, ethics boards and trial oversight committees are theoretically supposed to be providing some oversight. but as we know, these processes can be easily subverted.

 

And as with journals so called publication ethics and gatekeeping, once they've published a paper, that's it. Peer reviewers are not held accountable and are usually anonymous, ethics boards and trial oversight committees can be invoked to "prove" everything was done properly, in the latter case, even if the trial oversight committee is stuffed with people with conflicts of interest. Papers might as well be carved on tablets of stone. Once published in a "top" journal, the science can never be corrected, let alone correct itself! Cochrane authors can then smash the tablets of stone into dust add a pinch of GRADE, risk of bias (and water) and mould it to whatever shape suits their purposes. Got a bit carried away with that metaphor...

 

Don't know if I'm answering your question here.

I think moderate quality evidence that the effect of the intervention was non-zero. However, if you look at the fact that the trials were unblinded and used subjective outcome indicators, then the claim that these trials provide ---moderate quality evidence doesn't hold up. @petrichor set out the question was whether the intervention had a non-zero effect --- the unblinded trials, with subjective outcome indicators, indicated that the effect of the intervention was non-zero --- it's bonkers.

Reminds me of Sir Humphrey appearing at the committee

https://www.youtube.com/watch?v=dIto5mwDLxo

"load of meaningless drivel"

 

You're probably right but @Jonathan Edwards set it out very simply* and @Brian Hughes set it out in more blunt terms --- EDIT there is a better way to do studies --- they just don't seem to be motivated to do better work. Maybe the funders (NIHR, MRC ---) should be our target; I think someone mentioned that NIHR had stopped funding Cochrane ---- seems like an unexpected bit of enlightenment.

*https://www.s4me.info/threads/nice-...21-discussion-thread.23066/page-7#post-387365

 

Yes. The processes I'm thinking of would be incredibly difficult to subvert though, and the consequences significant. More independence I suspect, and less chance for cronyism.

 

It would indeed be possible and it would be an good idea since there may be folks who would look at the GRADE/Cochrane analysis and could publicly challenge it.

Yes, if NICE are going to use Cochrane/GRADE then NICE should work with Cochrane to try to improve the Cochrane/GRADE assessments.

I suspect that @Hilda Bastian may support a system of consulting on Cochrane reviews; however, I also expect that getting that revision through is reminiscent of David versus Goliath (and I wouldn't be betting on David).

 

Yea easily subverted by the mates club.

 

This. This. This.

 

When you use subjective outcome criteria (questionnaires) and the study is inadequately blinded then you will get a non-zero +ve effect. @Jonathan Edwards described this as "placebo" and it reminds me of the Hawthorne effect*. However, if you look at the studies of actimetry (objective) versus questionnaires then you'll see that questionnaires are just unreliable - therefore the evidence produced by these studies is unreliable.
*[https://en.wikipedia.org/wiki/Hawthorne_effect]

Yes even Fluge and Mella (Rituximab Phase 3 - 2014) used actimetry, and PACE (2011) was supposed to, so the technology is available and getting better/cheaper all the time. Using this type of outcome indicator you may indeed be able to identify "potentially productive areas of further research". You would of course be able to discount useless interventions --- interventions which these people make their living out of.

 

Yes.
A big general data capture and then analysis might find some interesting things.
Wearable make this eminently doable.


We' ve found oxygen saturation in PEM can dip significantly.