Cochrane Review: 'Exercise therapy for chronic fatigue syndrome', Larun et al. - New version October 2019

Thanks @Dolphin!

I was not seeing this. The pdf I've found stopped after 'Objectives'. It all makes a bit more sense now.

Only a bit because the outcome measure do mention: "timed walking tests and tests of strength or aerobic capacity", "employment status", compliance with the intervention etc.

 

Yes, it is very interesting in terms of Larun et al. claiming they couldn't use most objective outcome measures because of the protocol.

 

I've asked around and it seems that most people have access to the full review and the protocol.

I, however, don't have access. I can only see the abstract of both. For example if I go to the protocol (link: https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD003200/full) I only see this:



I can't unlock the full protocol. I also can't access the full Cochrane review. I have both documents now (they are given in this thread) so that's not the problem. I just want to know if there are others with the same problem or is it just me?

 

Yes, the link I posted was to the 2001 protocol (https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD003200/full). For some reason, I can't access it while others can. I'm looking for people with the same problem as me.

The same problem applies to the 2019 amendment of the review (https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD003200/full). I don't get the full review, only the abstract. Again there's a button called 'Unlock the review'. When I click it, it asks for payment or institutional access. It's all rather frustrating.

If I had access to the full protocol as others have, It would have saved me a lot of time and energy in conditional thinking (what if there's a protocol that I can't access... What if it says A... what if it says B: would that make Larun et al. response sensible or not etc etc...). And to make it even more confusing, if you use Shub to get access it gives you a pdf of the protocol but where the outcome section is left out! (I've attached a pdf-version earlier in the thread).

That's possible. But if you click on 'What's new' (link https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD003200.pub8/information#whatsNew) and look at the history overview, there's a note dated 25 may 2004 that says changes were made to the protocol.

Here's a screenshot:


EDIT: This is speculation but: could it be that the version I got through Shub was in fact the first and original version of the protocol? And that, because it was so vague 'referees told Edmonds et al. that they have to be more specific and specify outcomes and methods? So perhaps the protocol was changed in 2004 from the version I've linked to the version Dolphin linked?

 

From https://consumers.cochrane.org/cochrane-and-systematic-reviews

 

Thanks , You're brilliant @Andy

It was really puzzling for me.

 

As I've said from the start, as someone who *does* have professional statistical training (albeit a while ago), I wouldn't touch any of it with a barge pole - even a very, very long one! Not because I don't feel confident in doing so, but because they have made too many mistakes at the start to warrant combining the results at all.

For that reason, I haven't looked at the review in great detail, because it has so many obvious failings before you get anywhere near any analysis.

But, putting aside that large steaming pile...

Were these the only two studies that looked at 6mwt? Did they use it in the same way? Were the interventions themselves in any way comparable? If not, then the *clinical* heterogeneity rules out even doing a comparison (other than a narrative one). Statistical heterogeneity is about unseen statistical differences between trials once you've established that they are comparable, not obvious differences with big red flags all over them.

As I said above, MD and SMD calcs do not require a baseline comparison (although I agree that here - specifically for 6MWT and not necessarily anything else - it would have been useful), because they simply compare the end result (the group means). But even with the WT, I'm not convinced that a group mean is an adequate summary measure (because of the underlying distribution). Mean difference (ie, difference between baseline and endpoint for each pt) adjusted by baseline, would have been better, but the studies didn't report that and the IPD review is not going to happen. There is also the issue that drop-outs tended to do worse at baseline, so how you take account of drop-outs is also important.

Having said all that, what you've done @Michiel Tack is fine as far as it goes, but the problems with the data itself outweigh everything else.

 

Does this also touch on the issue of Control, within and across trials?

Presumably you could have several trials, each of them a properly designed properly controlled trial, all notionally trialling the same intervention, with notionally the same patient classification. Within the context of each trial there will be confounding factors, some of which are not evident or even known about; the proper use of controls within each trial minimising the effect of such confounding influences.

What I can't get my head around is this: Is it then valid to do statistical analysis across those trials, and if so what would and would not be valid? Would some analyses be invalid simply because the confounding factors within each trial might be different for each trial, and therefore no facility to control for these across trials? Or would it be OK so long as you only analyse data from each trial that already has had control applied to it within the trial. e.g. Only analysing differences between intervention and control arms for each study? Or would there still be an issue there for the unwary?

 

@Barry - Questions, questions! So many questions!

Hilda Bastian has addressed some of the issues here: https://blogs.plos.org/absolutely-maybe/2015/06/30/another-5-things-to-know-about-meta-analysis/

And this article provides quite a nice overview: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3868184/

 

@Barry and anyone else interested (series of videos)
"The talk is about risks of doing multiple analyses, including looking at different subgroups, which can lead to false positive findings. The ways of reducing these risks are discussed"
https://www.s4me.info/threads/assessing-randomised-controlled-trials.5420/#post-101699

 

Thanks for giving your perspective @Lucibee

As someone with no statistical background, I was surprised that baseline differences are ignored in these meta-analyses.

I was messing around with the data on oxygen consumption during an exercise test, to see what the result would be if someone were to pool them. And the analysis was pretty much determined by baseline differences rather than improvements. For example in the Fulcher & White trial, the increase in peak oxygen consumption was 2.4 ml/kg/min larger than the improvement in the control group but the baseline difference was 3.6ml/kg/min. In the trial by Wearden et al. 1998 the improvement in the intervention group was 2.8 ml/kg/min but the baseline difference with the control group was 6.1 ml/kg/min.

I hope that this is an exception but I suspect it's a major problem in meta-analyses with lots of trials with small sample sizes looking for a small treatment effect.

 

8) Lack of blinding
Larun et al. have downgraded the quality of evidence of pretty much all outcomes in their review due to a lack of blinding with one level. I don’t think this adequately addresses the risk of bias. A 2014 review by Hrobjartsson et al. on trials that compared blinded and non-blinded groups found that the average difference in effect size for patient-reported outcomes was 0.56. That’s very similar to the effect sizes in this review. It suggests that bias due to a lack of blinding does not question the quality of evidence, but the evidence itself. Unless the authors can provide a reason why the treatment effects reported in their review cannot be explained by a lack of blinding of trial participants, I think the review should explicitly acknowledge that treatments effects may be due to bias.

Unfortunately the authors do just the opposite. When lack of blinding is briefly mentioned in the discussion section, they use it to minimize the risk of bias. They argue that “many groups representing the interests of those with CFS are opposed to exercise therapy, and this may in contrast reduce the outcome estimate.” So because patient organizations oppose GET, the treatment effects in the randomized trials must be real!

There are so many issues with this argument that I had to split it up into different sections.

• Firstly, in the country where most of the randomized trials have taken place (the UK), GET has been recommended by multiple government reports and healthcare agencies. GET has been advised in the management of ME/CFS patients by (1) the 1997 Report of a joint working group of the Royal Colleges of Physicians, Psychiatrists, and General Practitioners, (2) the 2001 Report to the Chief Medical Officer and (3) the 2007 NICE guidelines. While only a minority of patients come into contact with what patient advocacy organizations are doing, pretty much all are confronted with the official advice of major healthcare agencies in the country. GET has long been considered the standard treatment for ME/CFS patients, even before many of the RCT’s in the Cochrane review were conducted. During the PACE trial patients were even sent a newsletter where it was highlighted that the NICE guideline committee had recommended GET.

• Secondly, in a randomized trial patients are selected. Those who think GET is not safe and effective will simply decline the offer to participate in the trial. Only those who are enthusiastic about the treatment being offered and really think it might work, will agree to participate in an intense 12 week exercise program.

• Thirdly, the expectations patients have before the trial are less important than what happens during the trial. Patients were in these trials for many months and had up to 15 sessions with trained physiotherapists or other healthcare professionals. This commitment to the treatment and relationship with healthcare professional is a much greater potential source of bias. We also know that in these GET trials, therapists not only tried to increase the physical activity level of patients, they explicitly tried to manipulate how patients interpret their symptoms. Encouraging optimism was part of the treatment. A booklet used in the trial by Powell et al. for example told patients: “You will experience a snowballing effect as increasing fitness leads to increasing confidence in your ability. You will have conquered CFS by your own effort and you will be back in control of your body again.” An online description of the exercise therapy used in the FINE Trial said: “Focus on your achievements now. Symptoms and limitations are temporary” and “There is no disease. Go for 100% recovery.” The GET manual for therapists used in the PACE trial had a section on encouraging optimism in patients. It said about patients: “[…] it is important that you encourage optimism about the progress that they may make with this approach. You can explain the previous positive research findings of GET and show in the way you discuss goals and use language that you believe they can get better.” So one should not really be surprised if patients say they do a little better after receiving such instructions.

​

The assertive approach described in these manuals, the lack of a credible control and the fact that healthcare institutions already advised GET as an effective approach before the large trials (FINE and PACE) were conducted, suggests that placebo effects and response bias may be particularly high in the GET studies. I think this is another major flaw of this review: it does not consider the possibility that reported treatment effects were due to bias caused by a lack of blinding, despite evidence supporting this view. The fact that reported improvements were not mediated by fitness in the trials by Fulcher & White, Moss-Morris et al., the FINE trial and the PACE trial, suggests that bias forms a better explanation of the results than reconditioning, fear-avoidance or central sensitization - the hypotheses the Cochrane review mentions.

 

Because the PACE trial is very large, it’s quite easy for small differences to achieve statistical significance. Which is why they used a “clinically useful difference“ threshold for the primary outcomes, of 0.5 SD effect size. This is a generic measure and so it’s reasonable to apply it to secondary outcomes such as walking distance. And the walking distance gain Did not make a clinically useful difference in the GET arm.

 

I don't actually think statistics come in to the concept of a clinically useful difference. A clinically useful difference is defined by clinical usefulness and should have nothing to do with variance or SD as far as I understand it. I had never heard of this way of defining useful difference before. I suspect it is only used by people who have no real idea what clinical usefulness is about!

 

For what it is worth:

https://www.ncbi.nlm.nih.gov/pubmed/17136972

 

Is there some implicit expectation that clinical usefulness leads to, not sure how to put it but the best I can would be actual usefulness? In the sense that if there is some "clinical usefulness" there has to be real-life usefulness as well, as in ability to function outside of a clinical setting as otherwise it would not be clinically useful?

Because absolutely none of the clinically useful measures used in this body of research has any actual tangible usefulness for patients. Maybe rephrasing it would be that clinically useful means useful from the physicians' perspective, which is assumed to align with the patients' but in this exceptional case does not because the physicians involved are out of their damn mind, something that is (incorrectly, clearly) assumed to be impossible.

Because there is a lot of discussion over this clinical usefulness and it means absolutely Jack Shit in real-life terms for us patients. That seems... wrong. Fundamentally, extremely wrong, as in should not happen because it's so self-evident that "clinically useful" means helpful to patients that nobody bothers to check that and here we are talking about something that truly has no actual impact on patient outcome.

Clinically useful white blood count definitely fits that expectation, it has real, tangible impact on outcome. CFQ? None at all. Depression, anxiety and perfectionism questionnaire ratings? None either. SF-36? Depends on whether the response was manipulated, which is precisely the case with a treatment that explicitly aims at changing perception based on an incorrect belief that there is nothing but perception.

I would just assume that something cannot be considered clinically useful if it has exactly zero relevance to patient outcomes. This discussion over mean deviation and such about a completely meaningless questionnaire-based self-assessment seems like a huge exception to that and the whole discussion amounts to the correct number of twirls the angels dancing on hairpins should be performing.

 

Actual usefulness is what clinical usefulness means. Clinical usefulness means relevant the patient rather than a lab or pathology specimen or questionnaire. It has nothing to do with statistical significance - it is defined as not being that, so I cannot see how it can have anything to do with standard deviations.

It isn't a hard concept. In thirty years of rheumatology I never met anyone who had trouble with it. It means useful to the patients.

 

So Wise & Brown (2005) say that "The minimal clinically important difference (MCID) for the 6MWT is conservatively estimated to be 54-80 meters." The difference between the improvements in meters walked in the GET group (67 meters) and SMC (22 meters) was 45 meters, so less than 54-80 meters.

The data for the 6-minute walking test is also no longer statistically significant if the data from the PACE trial is pooled with the data from Jason et al. 2007 (if my attempt at a meta-analysis is correct).

So I don't think there's a case for arguing that GET increases the walking ability of CFS patients.