Independent advisory group for the full update of the Cochrane review on exercise therapy and ME/CFS (2020), led by Hilda Bastian

[Hoopoe]

The problem with the typical clinical trial of CBT/GET research illustrated in this example of various treatments for asthma (except we have no treatment that meaningfully improves objective outcomes, so ignore the albuterol).

 

[Hilda Bastian]

Hi again, @Hilda Bastian. Thank you for engaging in this discussion with us. I'm finding it valuable and illuminating.


It would be easily done by getting women who are going to have a planned epidural for labour anyway to agree to the first hour being a trial while the labour pain is bearable. You would very quickly be able to determine whether the woman could feel her contractions, you could even to pin prick tests to see if she feels them as pressure or pain. Then after an hour it could be unblinded and those who had the placebo given a real dose. Speaking as someone who had an epidural late in labour, I only got as far as the small test dose that did nothing significant for half an hour before realising it was too late and the baby was about to pop out. Obviously you wouldn't test it during a C-section.

It's a bit like the dentist testing whether the tooth is numbed sufficiently to get on with drilling the tooth after an injection. The patient can tell very clearly whether it hurts or not. Acute pain is much easier to gauge whether a treatment that involves numbing is effective or not.

And something like ice or heat for pain is harmless, and in a sense it doesn't matter whether the effect is subjective because the effect is immediate and if it doesn't work, you just stop using it.

The problems we are talking about here are long term life changing effects of talking therapies and therapies that require behaviour changes and can have a long term detrimental effect that is not captured in questionnaires that can be too easily influenced by transient placebo effect, therapist effect (not wanting to disappoint a kind therapist) and the therapy itself telling patients to interpret their symptoms differently.

In this sort of situation, there ideally needs to be a control group given an equally empathic and convincing therapy, and long term objective measures like return to work, and long term activity meters worn before, during and after the trial are needed if the trial is to truly claim recovery or even significant improvement. Before and after 2day CPET would be good too.

Have a look at Graham's short video on the SF-36 to see what I mean about questionnaires.

Discussed on this thread:
https://www.s4me.info/threads/me-analysis-the-3-pace-videos-factsheet.6106/

I'm glad you're finding it helpful - thanks! And I'll check that out - thanks!

Epidurals are done late in labor for a variety of reasons, so that's when it needs to be tested, and it needs to be tested on the type of pain it's meant to relieve. (The C-section aspect was because of the question people had about whether having an epidural increased the risk of C-section because of interfering with the physiological processes of birth.) I totally agree that there are different issues related to the specific issues with this intervention and ME/CFS: but that's not what I was debating. It was about an absolutist statement which I was challenging, on the more general point of things being more complicated than the discourse was allowing for, and that people shouldn't be disparaged because they disagree with a simplistic global position about a very complicated issue.

 

[Hilda Bastian]

As Trish has pointed out, it's the combination of an unblinded treatment and a subjective outcome that is the problem.

There's this study here:
https://www.sciencedaily.com/releases/2017/10/171010224515.htm


I haven't looked at the details of the study, but this blinded study found that the pain scores weren't very different with analgesia or saline given in an epidural (although more women were unhappy with their pain relief in the saline group). Whereas a comparison of epidural vs no epidural would probably find that women reported significantly lower pain scores when given the epidural.

Absolutely, epidurals have the potential to cause harm to the mother and baby in various ways, and that is why it's important to understand exactly how much real benefit there is likely to be from an epidural at a specific dose and how much is a placebo effect - and under what circumstances an epidural is worth the risk.

Edited a bit, sorry

Well, that'll teach me to use an obstetrics example without getting up-to-date on it first - in terms of whether you could have the epidural early you & Trish get to say "OK boomer" about that! But that wasn't a trial of placebo epidurals for pain relief: they all had pain relief until the second stage. And that doesn't change the point I was making.

 

[Hutan]

So, I think it's correct that if you want to determine the utility of a particular treatment, it is important to have a blinded treatment if you have a subjective outcome (or unblinded treatment +objective outcome).

If you want to just find out what people say about the utility of a particular treatment, then a subjective outcome with an open-label treatment is fine. And that might be just what you want and it might even be ethically ok. But it isn't the same thing.


If a trial had an unblinded treatment and a subjective primary outcome, it won't tell you what the utility of the treatment is. It tells you what the utility of the treatment plus a placebo effect is (placebo effect in its broad definition). If the control treatment is a waiting list and the active treatment is heavily promoted, then that placebo effect will be large. If people are told that they are failures if they say the treatment didn't help them, then the placebo effect is likely to be even bigger.

Of course, the trial might still have some value, if the secondary outcomes are objective (and there aren't so many secondary objectives that an orchard for cherry picking is created.) I don't think anyone is suggesting otherwise. Or the trial might still have some value because it's the first one to use some new technology or because it shows that the treatment is feasible and safe. I don't think anyone is saying a trial is automatically worthless in every respect because it has an unblinded treatment and a subjective primary outcome. It's just worthless or close to that in measuring the utility of the treatment, separate from a placebo effect.

 

[Barry]

If you want to just find out what people say about the utility of a particular treatment, then a subjective outcome with an open-label treatment is fine. And that might be just what you want and it might even be ethically ok.

Even that has to come with serious caveats though, because what they say will still likely differ significantly for unblinded versus blinded. So even if wanting to know nothing more than what people say, then you might need to dig deeper into why you are interested in what they say.

I don't think anyone is saying a trial is automatically worthless in every respect because it has an unblinded treatment and a subjective primary outcome.

Not so sure about that? Unless the respects you are interested in are not particularly reliant on good science.

 

[Hutan]

I don't think anyone is saying a trial is automatically worthless in every respect because it has an unblinded treatment and a subjective primary outcome. It's just worthless or close to that in measuring the utility of the treatment, separate from a placebo effect.

Not so sure about that? Unless the respects you are interested in are not particularly reliant on good science.

No. Think of the Stage 2 rituximab trials - open label and subjective reports. Not worthless - they established that Rituximab was relatively safe for ME/CFS patients and helped establish a protocol for the Stage 3 trial. They helped consolidate Fluge and Mella's interest in ME/CFS. They were just worthless in determining the utility of rituximab as a ME/CFS treatment.

 

[Hilda Bastian]

Well, that'll teach me to use an obstetrics example without getting up-to-date on it first - in terms of whether you could have the epidural early you & Trish get to say "OK boomer" about that! But that wasn't a trial of placebo epidurals for pain relief: they all had pain relief until the second stage. And that doesn't change the point I was making.

However, I did just prove that I didn't have an epidural

So, I think it's correct that if you want to determine the utility of a particular treatment, it is important to have a blinded treatment if the outcome related to the aspect of utility you are interested in is a subjective outcome (or unblinded treatment +objective outcome).

If you want to just find out what people say about the utility of a particular treatment, then a subjective outcome with an open-label treatment is fine. And that might be just what you want and it might even be ethically ok. But it isn't the same thing.


If a trial had an unblinded treatment and a subjective primary outcome, it won't tell you what the utility of the treatment is. It tells you what the utility of the treatment plus a placebo effect is (placebo in its broad definition). If the control treatment is a waiting list and the active treatment is heavily promoted, then that placebo effect will be large. If people are told that they are failures if they say the treatment didn't help them, then the placebo effect is likely to be even bigger.

Of course, the trial might still have some value, if the secondary outcomes are objective (and there aren't so many secondary objectives that an orchard for cherry picking is created.) I don't think anyone is suggesting otherwise. Or the trial might still have some value because it's the first one to use some new technology or because it shows that the treatment is feasible and safe. I don't think anyone is saying a trial is automatically worthless in every respect because it has an unblinded treatment and a subjective primary outcome. It's just worthless or close to that in measuring the utility of the treatment, separate from a placebo effect.

"...the trial might still have some value...I don't think anyone is suggesting otherwise": actually, the point I was responding to was someone saying exactly that, and that anyone who disagreed with the statement was either incompetent or conflicted. It seems to be a standard part of a discourse - not that everyone says it, but I've seen it said often, by multiple people.

What people say and the utility of the treatment can be the same thing: eg if the intervention is purely pain relief. Or it can be so integral to whether the treatment is worthwhile, that it rises to the status of a primary endpoint. There can be objective measures that are better than subjective ones, of course, and there can be big reliability differences among methods of measuring subjective outcomes. But even if a specific outcome in a trial is problematic, that doesn't, on its own, necessarily condemn the trial to complete "valueless-ness" if it's not the only outcome. (I'm not talking about any specific trial here.)

 

[cassava7]

Is there any research on exercise therapy for ME/CFS that:

Uses objective outcomes as primary outcome measures (with clinically significant not just statistically significant differences between groups).
Uses currently acceptable diagnostic criteria that includes PEM.
Includes long term follow up (at least 6 months after the end of treatment?)
Properly records adverse effects including significant worsening of symptoms.
Measures patients' adherence to the therapy.

If not, then there are no studies to be included and the only possible outcome of the review is that there are no eligible studies to review.

I would also add monitoring checks to show that the patient isn't just replacing their normal daily activity with the exercise therapy.

@Hilda Bastian Years of clinical trials on ME/CFS point to the above criteria as being the correct ones to look at in ME/CFS. If a clinical trial is truly meant to help patients -- that is, it tests whether a treatment is safe for these patients and has an objectively measurable positive effect over the long term -- can it really be argued that they are not?

I think these criteria apply to other chronic diseases too. It seems to me that they are actually being used in diseases that are well known by the medical field. For example, even though I have no references, I believe that testing drugs for cancer or multiple sclerosis is done that way. But with chronic diseases that are not well understood -- ME/CFS, fibromyalgia, IBS, etc -- we are left with poorly designed clinical trials.

Therefore, the reasonable course of action seems to use these criteria in the review protocol. If the review comes up with an empty result, excluded clinical trials can definitely be discussed, with an indication/explanation of what criteria they failed to meet. In doing so, these trials are being objectively assessed and the discussion still values them.

A silver lining with an empty result is that it can implicitly say: "it's time to do better for these patients". This alone can prove to be very helpful for them -- moreso than piling up trials with low quality evidence --.

ETA: If a subjective outcome is used in a trial, then the authors of the trial should provide robust evidence that the tools used to measure this outcome actually measure it correctly (where 'correctly' is defined according to all of the aspects that the outcome entails). This brings us back to the issue of questionnaires, which, for some (or many), there is no such robust evidence and instead serious flaws can be found. One example is the Chalder Fatigue Questionnaire as previously studied by @Lucibee, @Michiel Tack and others. In this case, the trial should at least be marked as low quality evidence.

 

[Hoopoe]

I think these criteria apply to other chronic diseases too. It seems to me that they are actually being used in diseases that are well known by the medical field. For example, even though I have no references, I believe that testing drugs for cancer or multiple sclerosis is done that way. But with chronic diseases that are not well understood -- ME/CFS, fibromyalgia, IBS, etc -- we are left with poorly designed clinical trials.

The thinking seems to be that it is okay to do open-label clinical trials with subjective outcomes if there is no objective marker of disease severity. A sort of "we will do the best we can given the situation".

This is an emotionally appealing narrative but it doesn't make sense because all the lack of objective marker of disease severity does is making it hard to unambiguously demonstrate that a treatment is worthless. This isn't a serious effort to find treatments for illness, it's about appeasing patients and giving them hope and protecting them and doctors from the feeling of despair and powerlessness that comes with not having a treatment. Dealing with the emotional problems that come with severe untreatable illness is important but it shouldn't be done while pretending to be treating the illness. And I worry about the harm that could come from repeatedly treating patients with placebo therapies, for example disappointment, even worse hopelessness, waste of money and lack of trust.

 

[Hutan]

"...the trial might still have some value...I don't think anyone is suggesting otherwise": actually, the point I was responding to was someone saying exactly that, and that anyone who disagreed with the statement was either incompetent or conflicted. It seems to be a standard part of a discourse - not that everyone says it, but I've seen it said often, by multiple people.

I think you are referring to Jonathan's post. I'm sure he will state his case for himself, but what I understood him to be saying is that it's widely understood that an unblinded treatment with a subjective outcome is valueless for determining whether a treatment works. He knows as well as anyone that the Rituximab Stage 2 trials for example were not blinded but nevertheless had some value (but not for determining whether a treatment worked). Given that we are talking about Cochrane reviews of treatment efficacy, I think it's fair enough to focus on whether a trial structure has validity for determining treatment efficacy.

What people say and the utility of the treatment can be the same thing: eg if the intervention is purely pain relief. Or it can be so integral to whether the treatment is worthwhile, that it rises to the status of a primary endpoint. There can be objective measures that are better than subjective ones, of course, and there can be big reliability differences among methods of measuring subjective outcomes.

I don't think you can assume that what people say and the utility of the treatment are the same thing, if you have an unblinded treatment and a subjective outcome.

In the epidural example the 'purely pain relief' in the second stage of labour wasn't statistically better than a placebo at reducing pain scores. I am sure that if the treatments had been unblinded, this would not have been the case. And that matters, because, as with GET, the results will be used to weigh up the benefits and risks of the treatment protocol. Fatigue, pain, even things as seemingly concrete as how well you think you are breathing in that albuterol trial - reports are enormously influenced by the context. If you want to separate out the actual treatment effect from the placebo effect, unblinded +subjective just does not work.

Edited for clarity

 

[Snow Leopard]

If a trial had an unblinded treatment and a subjective primary outcome, it won't tell you what the utility of the treatment is. It tells you what the utility of the treatment plus a placebo effect is (placebo effect in its broad definition). If the control treatment is a waiting list and the active treatment is heavily promoted, then that placebo effect will be large. If people are told that they are failures if they say the treatment didn't help them, then the placebo effect is likely to be even bigger.

It's also important to consider that there is a difference between placebo effects and reporting biases. The former would be actual reduction in pain (in the case of trials of analgesics/anaesthesia), rather than a change (bias) in pain reporting behaviour. Both placebo effects and reporting biases can be influenced by medical and psychological interventions, and double blinding is a form of minimising the effects of both, to determine the underlying effect of the intervention. Placebo effects (on pain for example) tend to disappear over time for chronic conditions and reporting biases also tend to disappear over time (typically gone by 2 year followups).

 

[Trish]

I think the problem with subjective outcomes for ME/CFS are compounded by different interpretations of the words 'improvement' and 'recovery'. Do they refer to how the patient feels (eg. less of that slippery and subjectively reported concept, fatigue), or do they refer to objectively being able consistently to carry out normal daily activities without relapse?

An illuminating example of a CBT researcher mixing these up is Trudie Chalder, one of the PACE team just yesterday, as reported by psychologist, @Joan Crawford:
https://www.s4me.info/threads/webca...oach-june-8-trudie-chalder.14993/#post-265803

Edit to add:
The report includes this:

- She defines recovery in terms of patients making concrete changes (no matter how minor) – not symptom free or return to previous health. No discussion as to whether these concrete changes made a real world difference to patients’ lives? They feel better – and she believes them. Happy to take patients word for it. Whether they can go to work, make dinner, stand up or walk anywhere is not on her radar. She doesn’t see the relevance of objective, real world measures at all.

 

[Hutan]

Here are the results from a study I think illustrates the importance of not having an 'unblinded+ subjective outcome' structure. The Mendus ME/CFS study was of a CoQ10 supplement (MitoQ) which had two sub trials - one blinded trial with a control group and one open-label trial.

Results
Blinded study: No effects were observed for the MitoQ group on any measures above that observed in the placebo group.

Open label study: Significant (indeed, large) positive effects were observed for many of the measures including decreased pain and increased energy, sleep quality, mental clarity, activity, well being and verbal reasoning. Many of these effects were observed at 6-weeks and continued to grow at two follow-up points of 2 and 3 months. At 6 weeks, energy was reported to have increased by an average of 26% (versus both the blinded placebo and the blinded MitoQ participants reporting decreases in energy at 6 weeks). This is despite the open dosage being probably half of what the blinded study participants were taking.

So, the open label study wasn't a waste of time - it produced a very good example of how the placebo effect distorts results. But it was worthless in terms of determining the efficacy of the supplement.

Edit - I've been using 'placebo effect' in its broadest sense - including reporting biases

 

[Barry]

No. Think of the Stage 2 rituximab trials - open label and subjective reports. Not worthless - they established that Rituximab was relatively safe for ME/CFS patients and helped establish a protocol for the Stage 3 trial. They helped consolidate Fluge and Mella's interest in ME/CFS. They were just worthless in determining the utility of rituximab as a ME/CFS treatment.

So how is that any different from suggesting PACE established that GET is relatively safe for ME/CFS patients? Genuine question, not sarcasm.

 

[Joan Crawford]

I think the problem with subjective outcomes for ME/CFS are compounded by different interpretations of the words 'improvement' and 'recovery'. Do they refer to how the patient feels (eg. less of that slippery and subjectively reported concept, fatigue), or do they refer to objectively being able consistently to carry out normal daily activities without relapse?

An illuminating example of a CBT researcher mixing these up is Trudie Chalder, one of the PACE team just yesterday, as reported by psychologist, @Joan Crawford:
https://www.s4me.info/threads/webca...oach-june-8-trudie-chalder.14993/#post-265803

Edit to add:
The report includes this:

This has developed into is a huge problem as researchers are not using objective measures in unblinded RCTs. And they have not been pulled up for it either via their institutions or via the peer review process.

The researchers into CBT and GET then mix up 'feeling better' (which small/modest improvement is inevitable pretty much after a face to face intervention where someone was nice to you) and 'being better' (recovered/cured/able to do largely as they please with no or few symptoms). The later is 100% what patients want/would like. The former has been used ad nauseum by researchers in RCTs of CBT and GET. Poor.

Small/modest improvements in subjective measures/questionnaires has then been 'sold' as a meaningful result - when all it is, is a demonstration of the placebo effect. And it doesn't get anyone back to work or health or anything close to what the patients would consider recovery. This is a classic example of the psychological process of substitution. Human's are good at it. And slow, poor at identifying it and calling it out.

Joan Crawford
Counselling Psychologist

 

[Snow Leopard]

This has developed into is a huge problem as researchers are not using objective measures in unblinded RCTs. And they have not been pulled up for it either via their institutions or via the peer review process.

No one wants to point out the emperor has no clothes because it means we would necessarily throw out a lot of evidence for non-pharmacological therapies for many different conditions. It means that many people would have to admit they were at least partially wrong (from policy makers, to therapists, to scientists).

Fact is, most non-pharmacological/non-surgical therapies only show efficacy for subjectively reported symptoms and not objectively measured functioning or disease parameters. This is something the evidence base of these therapies shares with the evidence base for alt-med interventions.

 

[Hutan]

So how is that any different from suggesting PACE established that GET is relatively safe for ME/CFS patients? Genuine question, not sarcasm.

I can't recall the details of the Stage 2 rituximab trial but I expect Fluge and Mella had careful protocols for the reporting of harms.

Also, an aim of the Stage 2 trial was determine how the Stage 3 trial would be done and check that rituximab seemed to be safe for ME/CFS trial participants. There was no announcement at the end of it that ME/CFS patients should all rush to their nearest clinic and get rituximab - quite the opposite. Fluge and Mella did not tour around saying that rituximab would cure people or release media statements saying that.

Because the Stage 2 Rituximab trials were open label, it would not be possible to work out if all reports of harm were related to the rituximab, but it was good enough to see if the trial participants did have an unacceptably high amount of problems. I guess previous rituximab trials in other diseases gave them ideas of what to look especially hard for.

 

[Hilda Bastian]

I think you are referring to Jonathan's post. I'm sure he will state his case for himself, but what I understood him to be saying is that it's widely understood that an unblinded treatment with a subjective outcome is valueless for determining whether a treatment works. He knows as well as anyone that the Rituximab Stage 2 trials for example didn't conform to that structure but nevertheless had some value (but not for determining whether a treatment worked). Given that we are talking about Cochrane reviews of treatment efficacy, I think it's fair enough to focus on whether a trial structure has validity for determining treatment efficacy.


I don't think you can assume that what people say and the utility of the treatment are the same thing, if you have an unblinded treatment and a subjective outcome.

In the epidural example the 'purely pain relief' in the second stage of labour wasn't statistically better than a placebo at reducing pain scores. I am sure that if the treatments had been unblinded, this would not have been the case. And that matters, because, as with GET, the results will be used to weigh up the benefits and risks of the treatment protocol. Fatigue, pain, even things as seemingly concrete as how well you think you are breathing in that albuterol trial - reports are enormously influenced by the context. If you want to separate out the actual treatment effect from the placebo effect, unblinded +subjective just does not work.

The thing I've been debating is partly whether subjective outcomes in an unblinded treatment trial can ever justifiably be a primary endpoint, but more importantly this part of what he said - that if it's not, the trial is valueless (not just that particular bit of outcome data):

But the bottom line is not complicated. Open label trials with subjective primary endpoints (or switched endpoints) are valueless.

If Jonathan and the others arguing trials should be discarded so easily don't really mean that literally, that's great.

There are times when the only way of trying to assess whether a treatment is having a beneficial effect is by asking people, e.g. how much pain they are in.

I disagree that the epidural example is making the point you suggest. Since all the women had been on active analgesia until the second stage, when being randomized to placebo meant the analgesia was withdrawn for those in that group, I think it's hard to argue the women were truly "blinded" for that part of the trial. And even if they were on the placebo, they could get analgesia, and the data were analyzed on intention-to-treat, not whether they had analgesia or not. ("During the second stage of labor, participants with excessive pain were allowed be changed to an unblinded infusion of epidural medication.") And even if it that one trial showed what you're arguing for those 50-ish minutes, this wouldn't mean that the dozens (or whatever there are now) of trials of epidurals for pain relief that didn't have placebos were valueless, and epidurals haven't really been shown to reduce pain in labor.

 

[Trish]

@Jonathan Edwards can speak for himself, but I assume he meant the published main PACE papers, for both improvement and recovery, that used only the subjective primary outcomes as the basis for claiming GET and CBT were effective, was worthless.

We have plenty of examples showing that objective and subjective measures produce completely different outcomes in ME/CFS, and in other conditions such as asthma.

If they had published the trial results on the basis of the objective outcomes (step test, 6 minute walk and return to work) that would be different. They didn't do that. They could even have published both subjective and objective outcomes together, and demonstrated that the outcomes were different. They didn't do that either.

Nor did the Cochrane review look at those objective outcomes, as far as I know.

 

[cassava7]

This has developed into is a huge problem as researchers are not using objective measures in unblinded RCTs. And they have not been pulled up for it either via their institutions or via the peer review process.

The researchers into CBT and GET then mix up 'feeling better' (which small/modest improvement is inevitable pretty much after a face to face intervention where someone was nice to you) and 'being better' (recovered/cured/able to do largely as they please with no or few symptoms). The later is 100% what patients want/would like. The former has been used ad nauseum by researchers in RCTs of CBT and GET. Poor.

Small/modest improvements in subjective measures/questionnaires has then been 'sold' as a meaningful result - when all it is, is a demonstration of the placebo effect. And it doesn't get anyone back to work or health or anything close to what the patients would consider recovery. This is a classic example of the psychological process of substitution. Human's are good at it. And slow, poor at identifying it and calling it out.

Joan Crawford
Counselling Psychologist

If these serious issues can be so concisely summed up, one can only hope the new Cochrane review board will quickly reach the same conclusion. It's so desperately needed