Independent advisory group for the full update of the Cochrane review on exercise therapy and ME/CFS (2020), led by Hilda Bastian

[Jonathan Edwards]

This excuse from them really gets up my nose. There is a minimum methodological standard to meet for any study wishing to claim scientific status. Trials that do not meet that minimum standard are not merely a weaker form of evidence, they are non-evidence to start with. They lack the necessary rigour and clarity to be interpreted and applied safely. No amount of hand waving and sophistry can change that.

Yes, I like this idea of non-evidence. A system for assessing usefulness of treatments like Cochrane needs to recognise that some forms or 'evidence' are simply not good enough to be worth even considering' low' or 'poor' or. 'weak'. If a higher score in a test group is actually rather less than what are known to be placebo effects in other trials for instance it is not weak evidence for an effect. If anything it is weak evidence for a negative effect and rather strong evidence for no positive effect.

Basically, trials have to be considered in the real world context of trial psychology.

The central motivation for a Cochrane review is to see if the treatment is useful. You can only get evidence of even minimum value from prespecified primary outcome data. Other measures may be of peripheral interest if the primary outcome is clear but we know that the problem of multiple analyses makes it all too easy to find some other 'evidence' of an effect. We know from past experience that people fiddle their experiments most of the time in all walks of science. We know that in trials of therapist-delivered treatments in ME/CFS they do it all the time - switching outcomes, truncating axes, etc etc. It is reasonable to expect any trial that is to be taken seriously to have a clear primary outcome, and if this is subjective and the trial is unblinded then this is non-evidence.

And I think it is reasonable, in a real life context, to consider all other data relevant to usefulness as non-evidence. For a systematic review to pick over the range of data to try to find some secondary measure that looks a bit positive is simply to commit the same crime that the authors are expected to avoid-post hoc analysis.

I think it is interesting that this discussion probably rarely arises in the context of standard pharmacological treatments or complementary therapies, although for different reasons. For complementary therapies nobody in the establishment system is threatened by being upfront about lack of credibility in real life terms. Everyone is happy that non-evidence is non-evidence. In a sense the same is true for drugs because those who would be threatened are often in industry and in theory outside the establishment system. But in reality this is very blurred. I think the main difference for drugs is that they get evaluated before they are in widespread use (unlike homeopathy etc.).

I see the central problem with therapist-delivered treatments, and also a range of 'medical procedures' like facet joint injections, acupuncture and, strangely, use of radioisotopes which do not require drug licenses, is that they get into routine use before evaluation and there is a large body of people in the establishment system with a strong interest in finding evidence to support continued usage. The salient example is Chalder and Wessely giving detailed instructions on the best method of CBT for ME/CFS in 1989 (if I remember rightly) at a time before any evidence had been gathered from any sort of trial.

There is a strange sense of entitlement shown by people involved in these treatments. It is suggested that they should not be criticised too much because they are doing their best to do trials in difficult circumstances. It is assumed that it is in everyone's interest to do more and better trials to show how well the treatments work. But if, as of now, it is very unclear that these treatments work, or are based on any theory that makes it likely that they will work, then why should these people granted this sort of leeway and support?

If all that can be found is compatible with spurious influences on assessment or random variation or all the other things that mean a PhD lab student can always find a positive result somewhere buried in a week's work then this is non-evidence. And very often the lack of anything more substantial is clear evidence of non-effect.

 

[Joan Crawford]

@Joan Crawford, thank you very much for your ongoing engagement here. I agree with much of what you write but I'd like to discuss a couple of points that I think are relevant to the Cochrane review and ME/CFS research design.

The outcome measure 'ability to work (or equivalent) is quite a difficult one. I've given the example of my son returning to school after a year of home schooling before. So, if my son had been asked in January what his ability to attend school was, he would have said '100%'. He started school at the beginning of February and was attending well at the end of that month. If you had asked him then or had an objective measure based on school attendance, the rating might have been 90%. But by May, he was bed-bound and having difficulty even being awake to answer questions, let alone getting to school - so the rating was 0%. The point I am making is that it is very difficult to work out what a sustainable level of work or school is until you have been doing it for sometime. Any forecasting by the patient at the end of an intervention about what they might be able to sustain is likely to be optimistic. Particularly in the case of work, it takes time to apply for and obtain a job. And any objective measure of work hours or schooling needs to be made after enough time has passed for any cumulative impact to show itself. Enough time is probably at least three months after the increase in activity.

I get what you are saying. So, would it not be more useful to independently monitor school attendance for an academic year or a 12 months post intervention. That takes the patient out of the measurement too. Being asked what do you think you'll be able to do - as in a forecast - is not terribly helpful. That's subjective - even more so than a questionnaire measure of mood - as it involves speculation, forecast and will, I agree with you, often be overly optimistic as what the patent wants (to be well and at school) will be conflated. Need measures - ideally independent - tracking what actually happens pre and post intervention - from school or workplace records. This needs to be follow up over the longer term as there can be a boost effect from an intervention that is not sustainable - so perhaps start the school/workplace measure from 3 months post intervention like you suggest. pwME want to go to school, work etc so it makes sense that they'd keep trying - often to their detriment. Over time that effect is likely to fade if an intervention was not been helpful due to PEM/worsening of symptoms if the patient is overdoing it. 12/24 months follow up is really important too so that those who sadly end up house / bed-bound that outcome is recorded. Lack of longer term follow up has meant a lot of deterioration in pwME has been missed.

Work/ability to work is also a bit fraught. However, even PACE captured data here about benefits, pensions and so forth which demonstrates more patients post intervention were in receipt. That's not demonstration of effectiveness - that's failure. Objective failure.

I saw PDW give a talk about PACE. He brushed this aspect aside along the lines of: 'We're in a recession so less jobs are available.' Giving the impression that largely pwME weren't that bothered about being in employment/work - which took my breath away. Total opposite from my personal and professional experience. The inability to be able to work or equivalent greatly distresses people.

Employment measures need to be asked about in a wider manner/context. I think this often is - it is not necessarily the case that the person is actually in a job - it's whether they consider themselves capable of a job and are well enough to apply/consider this. This also takes into account other responsibilities people have which could keep them out of employment e.g. childcare, caring responsibility and simply not wishing to work and so forth.

The PACE model based on phonic fear avoidance should be easy peasy to treat and demonstrate effectiveness. Large effect sizes and recovery are really common in this type of work. So, if PACE CBT/GET models worked then I'd expect, based on my own clinical experience and my colleagues, and the published data on other phobias - that the majority of patients would be vastly more functional post PACE and, if not in work, be capable of stating that they could work if they had the opportunity, availability or desire. That discrepancy has not been accounted for.

It is possible to measure these things well: to capture a good reflection on what is actually happening/occurring. Getting patients more involved in outcome measures would be really useful so researchers - to help have more of a grasp of the problems patients face over the longer term.

 

[Hilda Bastian]

Hi @Hilda Bastian.

This discussion has stirred a lot of interest because it matters enormously to us what happens with the new Cochrane review. It will have a major impact on our lives and health. I'm pleased you have been appointed to lead the IAG and that you have joined us to engage with our concerns.

I don't know whether you have participated in discussion on patient/carer/scientist forums before. When a discussion provokes a lot of interest among our members, many will want to add their pennyworth to the discussion, even at the risk or repeating what others have said and being emphatic and sometimes not entirely clear in our statements. I am aware I am guilty of doing that and apologise if that felt like I was directing my comments at you personally. It can feel a bit as if one is under siege if a whole lot of people pile in to repeat basically the same point. I hope that won't put you off engaging with us and the points we make.

Thanks - that's nice! Not my first rodeo - and there's Twitter, which is a whole other level, and I do that. I never would have agreed to lead the IAG if I wasn't prepared to be serious about engagement.

 

[Hilda Bastian]

Yes, I also would very much like to endorse that view @Hilda Bastian. Your robust engagement here in seeking to get to the truth of things really heartens me you will do the same within Cochrane. I'm also well aware that no matter how capable you are (and I'm sure that is 'very') there is still a steep hill to climb for you, and not just on the science aspects we mostly witness here. I imagine some of the other things you need to achieve within Cochrane must feel like having to turn the proverbial oil tanker around.

I sometimes think, when in the midst of robust discussions like this one here, that if we were all discussing this in the same room, then things would clarify much more easily.

So your willingness to engage, and above all your determination to do what is right, is hugely appreciated by us here. Sincere thanks.

Thank you - that's nice, and I appreciate it! Oh yes, you can definitely count on me engaging robustly with Cochrane and others, and not being easy to push around or deter.

It's hard when people you don't know, and who don't know you, have so much power in a particular situation that matters more to you than it ever could to them. All I can really tell you is that I am listening, I'm deeply committed to doing the best I can here, and I care.

 

[Hilda Bastian]

I missed that. What do you mean by a valid example, @Medfeb ? Do you mean that investigating pain as an outcome in an unblinded, not adequately controlled trial with self-reported pain relief as the only measure is valid?

I found these statements by @Hilda Bastian:





I am not sure what you mean here, @Hilda Bastian, but since I didn't read all posts I maybe missed it and you said somewhere that it also applies to unblinded trials that...



...and that this will provide reliable evidence?

I don't think you're suggesting only because people don't apply certain criteria in other studies this means it's OK to do studies that way?

Or that, only because it would be helpful to know something there always must be means available to know that?

Apologies for being trivial, but in case it could be helpful for the discussion:

Do you agree, that, if you have well-founded reasons to apply certain treatments but aren't sure about their superiority compared to other treatments, because for whatever reasons this can't be properly assessed, the best thing to do is to honestly say what you know for sure and what you don't know? That you should say what is your reasoning behind suggesting a certain treatment, but at the same time you should be very clear about the fact that there is not sufficient evidence to favor one treatment over another?

I think documenting and registering outcomes of a certain intervention in practice can be of great value. But it's something different than doing an RCT.

For ME research I think treatment documentations and observational studies could be very useful, but these approaches have to be clear about the biases involved, apply adequate protocols and shouldn't be used to exaggerate the evidence.

(Apologies in advance if I'm not able to get back to replies for a while.)

Edited for clarity.

Apologies that my answer won't cover everything you mean. Sometimes, pain is a pivotal issue - when 2 options are roughly equal in all other known respects, for example, if one of the procedures hurts more than the other, it's critical for a lot of people's decision making - and certainly for clinical recommendations that would otherwise claim they are equal.

Yes, totally agree that sometimes you just need to accept something's not known, or the uncertainty around it is great.

 

[adambeyoncelowe]

Yet that is exactly the view of researchers in the field. They cannot personally think of a better way, so they assume their inadequate trials are good enough.







Note: Professor Bentall co-authored the Powell 2001 et al. graded exercise 'education' trial and the FINE trial.

See also (Bentall's double standards with regards to evidence quality):
http://cepuk.org/2020/06/04/guest-b...a-classic-failure-of-evidence-based-medicine/

I think he's missed the point. You may not have objective diagnostic tests for depression or psychosis, but you absolutely can objectively measure their impact with some sensible and creative thinking.

He also mentions therapies being impossible to blind. While it's difficult, I don't think it's impossible to blind to patients. E.g., a sham leg lift test was done by Rowe and patients couldn't tell the difference, apparently.

For CBT, you could have some kind of performative supportive listening and talking exercise. Get empathetic actors in and hand out some biscuits. Avoid therapy techniques but be chummy and make sure you reinforce, instead of challenging, patient thoughts and behaviours.

If the CBT model works for a given condition, then reinforcing negative cognitions and behaviours should lead to a null result.

For physio it's harder to blind, but there should be more objective measures instead so you don't need both subjective and unblinded outcomes.

 

[Hilda Bastian]

I will clarify my original comment in a minute but the suggestion that I hold this view through some sort of prejudice or vested interest is to commit exactly the impugnment that you were complaining of in others!

I don't believe I suggested that.

Let me rephrase my original comment to give the meaning that is transparent to all members here but you seem to have difficulty with. It was I think self-evident in terms of the context in which the comment was made.

The use of subjective outcomes (alone) from unblinded trials to judge the usefulness of a treatment is valueless.

But that is not a sensible reading of my original comment - as I think everyone else here would agree. It makes no sense. I think I did actually use the term 'primary outcome' initially and I am sorry if I did not put it every time. But everyone else here is up to speed on the need for primary outcomes prespecified, the problems of multiple analyses and so on. When I say an unblinded trial with a subjective outcomeI assume that people will understand me to mean a primary outcome or an outcome used to decide whether or not the treatment is useful - that was the problem with the review.

Quite early on, I said I had trouble with the statement as it stood, and explained why people would disagree with it, but thanks for clarifying. I still don't agree, but yes, I disagree with that version a lot less! That said, however, I see that sentiment, without any qualifications, repeatedly elsewhere: and I've seen people dissed for saying they disagree, and I've seen the impact of that.

 

[Barry]

All I can really tell you is that I am listening, I'm deeply committed to doing the best I can here, and I care.

 

[Sean]

I never would have agreed to lead the IAG if I wasn't prepared to be serious about engagement.

 

[Hilda Bastian]

@Hilda Bastian, sorry to keep labouring the point about unblinded treatments + subjective end points not being suitable for assessing treatment efficacy, but it is so fundamental to the problems the ME/CFS community and many others have had with the BPS proponents, and so fundamental to the usefulness of the coming review. And it looks as though we haven't yet convinced you.

Maybe you can give us an example of a treatment efficacy trial that had unblinded treatments and subjective end points where the conclusion about treatment efficacy could not be legitimately questioned as being partly (and even significantly) distorted by factors not strictly related to the treatment? That might help us identify under what, if any, circumstances the combination is ok and what circumstances it is not.

There was a proposal for research recently, for a commercial and expensive intervention that had virtually no scientific evidence underpinning it. The intervention had been talked about a lot by the regional patient support group - there was a lot of excitement about it. The trial structure was unblinded, with a Treatment As Usual control, and with a change in fatigue as the primary outcome (from a survey administered before and after a month of treatment). Can we agree that that trial would be unreasonably biased in favour of the intervention and so the outcome would be worthless?

I'd said I would do this, but when I re-read this carefully, I don't think it's that simple. Even when trials are blinded, and even when the outcomes aren't subjective, results can be distorted by factors not strictly related to the treatment.

In terms of your final question, I'd want more information to have an opinion, but the way you describe it does sound worrisome.

Back to the issue of an example. I don't think a single example can tell you when the combination of impossible-to-blind plus subjective outcome measures is ok and when it's not. It depends on a lot, including how well it's measured. I'll give you an example, though, that raises some of the reasons it can be. I'm not vouching for the particular trial - I didn't spend enough time on this to do that. I just thought of an example of the sort of circumstance where a subjective outcome can be "the" primary endpoint. There are several areas where it's often an important primary outcome - such as health-related quality of life as well as mortality (or another health outcome).

So I quickly looked for an example of a trial in urinary incontinence with a subjective primary endpoint and an unblindable comparison, because that's a condition where subjective outcomes are critical. Here's the protocol, and here's the trial report. Afterwards, it hit me that I should have looked for one where the comparison was with no treatment, but I'm just not sure this exercise is so helpful that I should keep going. I could have looked for unblinded trials that had only subjective endpoints, but I knew I'd then have to spend a lot of time ensuring I was picking a very high quality one.

The underlying point here, for me, was that statements that might make sense in the context of ME/CFS trials of some types of interventions, won't necessarily be valid arguments to people whose frame of reference isn't ME/CFS.

 

[Hoopoe]

I have some issues with describing urinary incontinence as subjective outcome. It think it's an objective outcome. It might be reported in a biased manner by patients but it's observable and isn't a subjective feeling. It's very different from asking patients how fatigued they are.

I would agree that subjectivity/objectivity of an outcome can be thought of as a value on continuum. So maybe we should be saying that the more subjective an outcome, the less reliable it becomes and that some outcomes do not exceed a minimum standard of reliability.

 

[spinoza577]

I'd said I would do this, but when I re-read this carefully, I don't think it's that simple. Even when trials are blinded, and even when the outcomes aren't subjective, results can be distorted by factors not strictly related to the treatment.

E.g. blinding can not be maintained because of side effects of the treatment (so not strictly related to the mechanism of action in its efficiency) or wrong sample.

This brings me to the point that the authors don´t seem to have any near idea of what they are investigating. The paper even comes to the conclusion in regard of pacing that it is not effective. Maybe its not effective in getting better (or maybe only at the very beginning of illness), but it is reported on forums again and again to be effective in not getting worse, or in maintaining the skills one has (at least if not other factors outcompete it).

It is crucial to understand what we are doing, and what we are investigating. Researchers must ask the right questions, or they miss the point. To evaluate what ME might be, I would ask these questions:

• To what normal feeling and its biological mechanism could the hugely exaggerated worsening (PEM) in and/or after exertion be related?
• Why can PEM be delayed?
• Is it possible to pace with different activities, e.g. doing action A the one day, but action B the other day? (In my experience it is.). And why?
• Why is there this wide range of possible symptoms (none of which is specific)?
• Why are there the possible up and downs in the course of the illness (and only in a tendency a worsening)?
• Why is there a feeling of unrest - rather an exhaustion than a fatique / tiredness (cf sleep patterns)?
  
• Why are some ppl more physically affected and other more mentally (PEM vs PEF)?
• What do the known or suspected triggers have in common so that they can trigger the illness?
• Why (if really so) do people in outbreaks often recover to 50% but in sporadic cases only to 5%?
• Can there be a specific reason that no animal model has been convincingly established?

I think it is pretty clear that the pace authors (and even a lot of other ones) don´t have an adequate idea for what they would have to look out. True, they could answer the one or the other question - but for the whole relationship they miss any view, and even worse, they are not interested in the patients.

 

[spinoza577]

I have some issues with describing urinary incontinence as subjective outcome. It think it's an objective outcome. It might be reported in a biased manner by patients but it's observable and isn't a subjective feeling. It's very different from asking patients how fatigued they are.

Easy questions but complicate matter.

I think even pain is thinkable to be able to be measured objectively. Only it is not established yet.

 

[rvallee]

Also good replies from others including Millions Missing France

Major kudos to Millions Missing France. Doing outstanding work lately.

Many of you won't really see it but there is lots of discussion in French twitter, especially with the hashtags #ApresJ20 #ApresJ60 (#AfterD20-60), and this account is very active handing out guidance and support, especially to those gaslit by their GP.

 

[rvallee]

Notably, there has been no studies that actually ask patients which outcome measures are most relevant. Likewise, none of the PROMs have been tested to determine whether patients find them relevant, despite two systematic reviews on PROMs suggesting this is a major problem.

I think a few have been done, Chalder I think, but none that are fit for purpose, they always restrict the options to those they favor and selectively report. This needs a real patient engagement process. Frankly this is something we will probably have to do ourselves in the long run, defining valid outcomes and writing relevant questionnaires that actually consider the whole illness, not some tiny slice of it.

Once we're not too sick to do that anyway. Which is a bit of an obstacle. I love Catch-22's. Well, the band, not the living them part. That part is awful.

 

[rvallee]

one would not expect to see symptoms like POTS or gut issues

Those are definitively prevalent in the acute phase of COVID. I don't know about other viruses but in some COVID cases they are the main or even the only symptoms. It probably depends on where the virus lands. I've seen a few cases where loss of smell was the only notable symptom, probably held up in the olfactory bulb and couldn't progress past it.

 

[Jonathan Edwards]

I have some issues with describing urinary incontinence as subjective outcome. It think it's an objective outcome. It might be reported in a biased manner by patients but it's observable and isn't a subjective feeling. It's very different from asking patients how fatigued they are.

I would agree that subjectivity/objectivity of an outcome can be thought of as a value on continuum. So maybe we should be saying that the more subjective an outcome, the less reliable it becomes and that some outcomes do not exceed a minimum standard of reliability.

My understanding is that for the purposes of discussing trial design 'subjective' is used in the sense of something that can be influenced by spurious contextual factors. It is not 'subjective' in the sense of being a feeling. So if I score histology slides for the degree of inflammation in a joint and use standard graded images of slides as the basis for a score then my assessment is going to be subjective. If I count cells exactly then it is likely to be less so.

On the other hand I think you are absolutely right to say subjectivity is on a spectrum. The assumption has to be made that a measure is called subjective if it thought to be sufficiently at risk of bias in the context being considered.

 

[Jonathan Edwards]

I just thought of an example of the sort of circumstance where a subjective outcome can be "the" primary endpoint. There are several areas where it's often an important primary outcome - such as health-related quality of life as well as mortality (or another health outcome).

An interesting example @Hilda Bastian.

I think we are agreed that a subjective outcome can be "the" primary endpoint. For blinded trials of analgesics that is standard and unproblematic. The question is whether or not it is ever worth setting up an unblinded trial with a subjective outcome as primary endpoint.

The urinary incontinence study looks tricky to me. Reporting of incontinence would be subjective if the account given - maybe a diary - was open to a bit of forgetting or over-remembering. If the patient was constantly wearing a urine sensor with 100% reliability then clearly it would not be subjective. But let's assume it is subjective.

If the diaries are open to subjectivity then I think I would have serious doubts about the reliability of the study. It is clever in that it compares two treatments, and these could be presented as neutral options. The problem is that there is a priori a very high likelihood that those doing the study would not be neutral. (The core of the problem - neutrality in experimentation is a rare thing. Psychology is paramount.) Maybe the physios would be keen to persuade the patients how safe and natural their treatment was. Maybe they wouldn't, if they work for surgeons and are interested in post-operative care. Maybe the surgeons explaining the operation are charismatic, maybe not ...

The bottom line for me is that I would actually put more weight on well designed trials of the individual treatments than a head to head comparison because the psychological difference is likely to be charged. There also seems to be problem in this trial that it is already known that the risk benefit profiles of the treatments are starkly different. One is very safe and not very good and the other can cause bad side effects but often works well. It is not clear to me that a head to head comparison can say which is "most (cost) effective". That sounds like an illusory determination.

I am minded to say this trial was probably not useful. It might have the potential to be useful on the planet Vulcan where everyone was perfectly neutral but a key principle of judging reliability is that if it is perfectly possible that not everyone was neutral then reliability is lost - the probability of a spurious result is real and, being unquantifiable, you have no way of assigning any level of reliability to the result.

One thing that I think is worth throwing in to the discussion is that one way of defining blinding, and the way I tend to think in terms of, is blinding that hides which treatment is 'the test' and which 'the control'. So, for instance, a trial that compares six sessions of treatment with twelve can reconsidered blinded if nobody involved has any prior opinion on what the number should be. In general dose response studies can function as blinded even if everyone knows the dose in each case.

 

[Hoopoe]

I thought about what kind of review of exercise therapy for ME/CFS would be appropriate.

One with roughly the following content:

A conclusion that says exercise therapy cannot be recommended due to an absence (or maybe insufficient number) of studies meeting adequate quality standards and showing benefit, as well as concerns about possible harm.

An explanation of what the problems with existing studies are.

A section on suspicion of harm associated with GET, including patient surveys and results of exercise physiology research, indicating that the reports of harms are not a misinterpretation of ordinary bodily sensations as some argue, but that something abnormal is occurring.

A comment on how view of ME/CFS has evolved over time and that the old view of it being "unexplained fatigue plus any of several other symptoms" and related to lack of fitness is considered outdated by most experts.

That would be an intelligent discussion of the topic.

 

[Lucibee]

why is this coming up as being published yesterday (21 May 2020)??

https://www.cochrane.org/CD003200/DEPRESSN_exercise-treatment-adults-chronic-fatigue-syndrome

eta:

This is appearing under the banner "Coronavirus (COVID-19) resources"

Hmmmm.