Independent advisory group for the full update of the Cochrane review on exercise therapy and ME/CFS (2020), led by Hilda Bastian

Yes, I like this idea of non-evidence. A system for assessing usefulness of treatments like Cochrane needs to recognise that some forms or 'evidence' are simply not good enough to be worth even considering' low' or 'poor' or. 'weak'. If a higher score in a test group is actually rather less than what are known to be placebo effects in other trials for instance it is not weak evidence for an effect. If anything it is weak evidence for a negative effect and rather strong evidence for no positive effect.

Basically, trials have to be considered in the real world context of trial psychology.

The central motivation for a Cochrane review is to see if the treatment is useful. You can only get evidence of even minimum value from prespecified primary outcome data. Other measures may be of peripheral interest if the primary outcome is clear but we know that the problem of multiple analyses makes it all too easy to find some other 'evidence' of an effect. We know from past experience that people fiddle their experiments most of the time in all walks of science. We know that in trials of therapist-delivered treatments in ME/CFS they do it all the time - switching outcomes, truncating axes, etc etc. It is reasonable to expect any trial that is to be taken seriously to have a clear primary outcome, and if this is subjective and the trial is unblinded then this is non-evidence.

And I think it is reasonable, in a real life context, to consider all other data relevant to usefulness as non-evidence. For a systematic review to pick over the range of data to try to find some secondary measure that looks a bit positive is simply to commit the same crime that the authors are expected to avoid-post hoc analysis.

I think it is interesting that this discussion probably rarely arises in the context of standard pharmacological treatments or complementary therapies, although for different reasons. For complementary therapies nobody in the establishment system is threatened by being upfront about lack of credibility in real life terms. Everyone is happy that non-evidence is non-evidence. In a sense the same is true for drugs because those who would be threatened are often in industry and in theory outside the establishment system. But in reality this is very blurred. I think the main difference for drugs is that they get evaluated before they are in widespread use (unlike homeopathy etc.).

I see the central problem with therapist-delivered treatments, and also a range of 'medical procedures' like facet joint injections, acupuncture and, strangely, use of radioisotopes which do not require drug licenses, is that they get into routine use before evaluation and there is a large body of people in the establishment system with a strong interest in finding evidence to support continued usage. The salient example is Chalder and Wessely giving detailed instructions on the best method of CBT for ME/CFS in 1989 (if I remember rightly) at a time before any evidence had been gathered from any sort of trial.

There is a strange sense of entitlement shown by people involved in these treatments. It is suggested that they should not be criticised too much because they are doing their best to do trials in difficult circumstances. It is assumed that it is in everyone's interest to do more and better trials to show how well the treatments work. But if, as of now, it is very unclear that these treatments work, or are based on any theory that makes it likely that they will work, then why should these people granted this sort of leeway and support?

If all that can be found is compatible with spurious influences on assessment or random variation or all the other things that mean a PhD lab student can always find a positive result somewhere buried in a week's work then this is non-evidence. And very often the lack of anything more substantial is clear evidence of non-effect.

 

I get what you are saying. So, would it not be more useful to independently monitor school attendance for an academic year or a 12 months post intervention. That takes the patient out of the measurement too. Being asked what do you think you'll be able to do - as in a forecast - is not terribly helpful. That's subjective - even more so than a questionnaire measure of mood - as it involves speculation, forecast and will, I agree with you, often be overly optimistic as what the patent wants (to be well and at school) will be conflated. Need measures - ideally independent - tracking what actually happens pre and post intervention - from school or workplace records. This needs to be follow up over the longer term as there can be a boost effect from an intervention that is not sustainable - so perhaps start the school/workplace measure from 3 months post intervention like you suggest. pwME want to go to school, work etc so it makes sense that they'd keep trying - often to their detriment. Over time that effect is likely to fade if an intervention was not been helpful due to PEM/worsening of symptoms if the patient is overdoing it. 12/24 months follow up is really important too so that those who sadly end up house / bed-bound that outcome is recorded. Lack of longer term follow up has meant a lot of deterioration in pwME has been missed.

Work/ability to work is also a bit fraught. However, even PACE captured data here about benefits, pensions and so forth which demonstrates more patients post intervention were in receipt. That's not demonstration of effectiveness - that's failure. Objective failure.

I saw PDW give a talk about PACE. He brushed this aspect aside along the lines of: 'We're in a recession so less jobs are available.' Giving the impression that largely pwME weren't that bothered about being in employment/work - which took my breath away. Total opposite from my personal and professional experience. The inability to be able to work or equivalent greatly distresses people.

Employment measures need to be asked about in a wider manner/context. I think this often is - it is not necessarily the case that the person is actually in a job - it's whether they consider themselves capable of a job and are well enough to apply/consider this. This also takes into account other responsibilities people have which could keep them out of employment e.g. childcare, caring responsibility and simply not wishing to work and so forth.

The PACE model based on phonic fear avoidance should be easy peasy to treat and demonstrate effectiveness. Large effect sizes and recovery are really common in this type of work. So, if PACE CBT/GET models worked then I'd expect, based on my own clinical experience and my colleagues, and the published data on other phobias - that the majority of patients would be vastly more functional post PACE and, if not in work, be capable of stating that they could work if they had the opportunity, availability or desire. That discrepancy has not been accounted for.

It is possible to measure these things well: to capture a good reflection on what is actually happening/occurring. Getting patients more involved in outcome measures would be really useful so researchers - to help have more of a grasp of the problems patients face over the longer term.

 

Thanks - that's nice! Not my first rodeo - and there's Twitter, which is a whole other level, and I do that. I never would have agreed to lead the IAG if I wasn't prepared to be serious about engagement.

 

Thank you - that's nice, and I appreciate it! Oh yes, you can definitely count on me engaging robustly with Cochrane and others, and not being easy to push around or deter.

It's hard when people you don't know, and who don't know you, have so much power in a particular situation that matters more to you than it ever could to them. All I can really tell you is that I am listening, I'm deeply committed to doing the best I can here, and I care.

 

Apologies that my answer won't cover everything you mean. Sometimes, pain is a pivotal issue - when 2 options are roughly equal in all other known respects, for example, if one of the procedures hurts more than the other, it's critical for a lot of people's decision making - and certainly for clinical recommendations that would otherwise claim they are equal.

Yes, totally agree that sometimes you just need to accept something's not known, or the uncertainty around it is great.

 

I think he's missed the point. You may not have objective diagnostic tests for depression or psychosis, but you absolutely can objectively measure their impact with some sensible and creative thinking.

He also mentions therapies being impossible to blind. While it's difficult, I don't think it's impossible to blind to patients. E.g., a sham leg lift test was done by Rowe and patients couldn't tell the difference, apparently.

For CBT, you could have some kind of performative supportive listening and talking exercise. Get empathetic actors in and hand out some biscuits. Avoid therapy techniques but be chummy and make sure you reinforce, instead of challenging, patient thoughts and behaviours.

If the CBT model works for a given condition, then reinforcing negative cognitions and behaviours should lead to a null result.

For physio it's harder to blind, but there should be more objective measures instead so you don't need both subjective and unblinded outcomes.

 

I don't believe I suggested that.

Quite early on, I said I had trouble with the statement as it stood, and explained why people would disagree with it, but thanks for clarifying. I still don't agree, but yes, I disagree with that version a lot less! That said, however, I see that sentiment, without any qualifications, repeatedly elsewhere: and I've seen people dissed for saying they disagree, and I've seen the impact of that.

 

I'd said I would do this, but when I re-read this carefully, I don't think it's that simple. Even when trials are blinded, and even when the outcomes aren't subjective, results can be distorted by factors not strictly related to the treatment.

In terms of your final question, I'd want more information to have an opinion, but the way you describe it does sound worrisome.

Back to the issue of an example. I don't think a single example can tell you when the combination of impossible-to-blind plus subjective outcome measures is ok and when it's not. It depends on a lot, including how well it's measured. I'll give you an example, though, that raises some of the reasons it can be. I'm not vouching for the particular trial - I didn't spend enough time on this to do that. I just thought of an example of the sort of circumstance where a subjective outcome can be "the" primary endpoint. There are several areas where it's often an important primary outcome - such as health-related quality of life as well as mortality (or another health outcome).

So I quickly looked for an example of a trial in urinary incontinence with a subjective primary endpoint and an unblindable comparison, because that's a condition where subjective outcomes are critical. Here's the protocol, and here's the trial report. Afterwards, it hit me that I should have looked for one where the comparison was with no treatment, but I'm just not sure this exercise is so helpful that I should keep going. I could have looked for unblinded trials that had only subjective endpoints, but I knew I'd then have to spend a lot of time ensuring I was picking a very high quality one.

The underlying point here, for me, was that statements that might make sense in the context of ME/CFS trials of some types of interventions, won't necessarily be valid arguments to people whose frame of reference isn't ME/CFS.

 

I have some issues with describing urinary incontinence as subjective outcome. It think it's an objective outcome. It might be reported in a biased manner by patients but it's observable and isn't a subjective feeling. It's very different from asking patients how fatigued they are.

I would agree that subjectivity/objectivity of an outcome can be thought of as a value on continuum. So maybe we should be saying that the more subjective an outcome, the less reliable it becomes and that some outcomes do not exceed a minimum standard of reliability.

 

E.g. blinding can not be maintained because of side effects of the treatment (so not strictly related to the mechanism of action in its efficiency) or wrong sample.

This brings me to the point that the authors don´t seem to have any near idea of what they are investigating. The paper even comes to the conclusion in regard of pacing that it is not effective. Maybe its not effective in getting better (or maybe only at the very beginning of illness), but it is reported on forums again and again to be effective in not getting worse, or in maintaining the skills one has (at least if not other factors outcompete it).

It is crucial to understand what we are doing, and what we are investigating. Researchers must ask the right questions, or they miss the point. To evaluate what ME might be, I would ask these questions:

• To what normal feeling and its biological mechanism could the hugely exaggerated worsening (PEM) in and/or after exertion be related?
• Why can PEM be delayed?
• Is it possible to pace with different activities, e.g. doing action A the one day, but action B the other day? (In my experience it is.). And why?
• Why is there this wide range of possible symptoms (none of which is specific)?
• Why are there the possible up and downs in the course of the illness (and only in a tendency a worsening)?
• Why is there a feeling of unrest - rather an exhaustion than a fatique / tiredness (cf sleep patterns)?
  
• Why are some ppl more physically affected and other more mentally (PEM vs PEF)?
• What do the known or suspected triggers have in common so that they can trigger the illness?
• Why (if really so) do people in outbreaks often recover to 50% but in sporadic cases only to 5%?
• Can there be a specific reason that no animal model has been convincingly established?

I think it is pretty clear that the pace authors (and even a lot of other ones) don´t have an adequate idea for what they would have to look out. True, they could answer the one or the other question - but for the whole relationship they miss any view, and even worse, they are not interested in the patients.

 

Easy questions but complicate matter.

I think even pain is thinkable to be able to be measured objectively. Only it is not established yet.

 

Major kudos to Millions Missing France. Doing outstanding work lately.

Many of you won't really see it but there is lots of discussion in French twitter, especially with the hashtags #ApresJ20 #ApresJ60 (#AfterD20-60), and this account is very active handing out guidance and support, especially to those gaslit by their GP.

 

I think a few have been done, Chalder I think, but none that are fit for purpose, they always restrict the options to those they favor and selectively report. This needs a real patient engagement process. Frankly this is something we will probably have to do ourselves in the long run, defining valid outcomes and writing relevant questionnaires that actually consider the whole illness, not some tiny slice of it.

Once we're not too sick to do that anyway. Which is a bit of an obstacle. I love Catch-22's. Well, the band, not the living them part. That part is awful.

 

Those are definitively prevalent in the acute phase of COVID. I don't know about other viruses but in some COVID cases they are the main or even the only symptoms. It probably depends on where the virus lands. I've seen a few cases where loss of smell was the only notable symptom, probably held up in the olfactory bulb and couldn't progress past it.

 

My understanding is that for the purposes of discussing trial design 'subjective' is used in the sense of something that can be influenced by spurious contextual factors. It is not 'subjective' in the sense of being a feeling. So if I score histology slides for the degree of inflammation in a joint and use standard graded images of slides as the basis for a score then my assessment is going to be subjective. If I count cells exactly then it is likely to be less so.

On the other hand I think you are absolutely right to say subjectivity is on a spectrum. The assumption has to be made that a measure is called subjective if it thought to be sufficiently at risk of bias in the context being considered.

 

An interesting example @Hilda Bastian.

I think we are agreed that a subjective outcome can be "the" primary endpoint. For blinded trials of analgesics that is standard and unproblematic. The question is whether or not it is ever worth setting up an unblinded trial with a subjective outcome as primary endpoint.

The urinary incontinence study looks tricky to me. Reporting of incontinence would be subjective if the account given - maybe a diary - was open to a bit of forgetting or over-remembering. If the patient was constantly wearing a urine sensor with 100% reliability then clearly it would not be subjective. But let's assume it is subjective.

If the diaries are open to subjectivity then I think I would have serious doubts about the reliability of the study. It is clever in that it compares two treatments, and these could be presented as neutral options. The problem is that there is a priori a very high likelihood that those doing the study would not be neutral. (The core of the problem - neutrality in experimentation is a rare thing. Psychology is paramount.) Maybe the physios would be keen to persuade the patients how safe and natural their treatment was. Maybe they wouldn't, if they work for surgeons and are interested in post-operative care. Maybe the surgeons explaining the operation are charismatic, maybe not ...

The bottom line for me is that I would actually put more weight on well designed trials of the individual treatments than a head to head comparison because the psychological difference is likely to be charged. There also seems to be problem in this trial that it is already known that the risk benefit profiles of the treatments are starkly different. One is very safe and not very good and the other can cause bad side effects but often works well. It is not clear to me that a head to head comparison can say which is "most (cost) effective". That sounds like an illusory determination.

I am minded to say this trial was probably not useful. It might have the potential to be useful on the planet Vulcan where everyone was perfectly neutral but a key principle of judging reliability is that if it is perfectly possible that not everyone was neutral then reliability is lost - the probability of a spurious result is real and, being unquantifiable, you have no way of assigning any level of reliability to the result.

One thing that I think is worth throwing in to the discussion is that one way of defining blinding, and the way I tend to think in terms of, is blinding that hides which treatment is 'the test' and which 'the control'. So, for instance, a trial that compares six sessions of treatment with twelve can reconsidered blinded if nobody involved has any prior opinion on what the number should be. In general dose response studies can function as blinded even if everyone knows the dose in each case.

 

I thought about what kind of review of exercise therapy for ME/CFS would be appropriate.

One with roughly the following content:

A conclusion that says exercise therapy cannot be recommended due to an absence (or maybe insufficient number) of studies meeting adequate quality standards and showing benefit, as well as concerns about possible harm.

An explanation of what the problems with existing studies are.

A section on suspicion of harm associated with GET, including patient surveys and results of exercise physiology research, indicating that the reports of harms are not a misinterpretation of ordinary bodily sensations as some argue, but that something abnormal is occurring.

A comment on how view of ME/CFS has evolved over time and that the old view of it being "unexplained fatigue plus any of several other symptoms" and related to lack of fitness is considered outdated by most experts.

That would be an intelligent discussion of the topic.

 

This is appearing under the banner "Coronavirus (COVID-19) resources"

Hmmmm.