Independent advisory group for the full update of the Cochrane review on exercise therapy and ME/CFS (2020), led by Hilda Bastian

Hutan said:
I guess because it's such a fundamental issue when determining what studies provide useful evidence. And because there seems to be a surprising level of faith in subjective measures as reliable indicators of treatment utility in unblinded trials.
Click to expand...

I know others will disagree but selecting the wrong cohort of patients is just as fundamental. Even if a body of research is perfectly conducted, its conclusions about efficacy and safety in a particular disease are meaningless if the cohort includes an overrepresentation of patients with some other disease.
 
Medfeb said:
I have admittedly missed parts of this thread but I'm struggling to understand why so much focus and discussion just on the issue of subjective measures in unblinded trials.

@Hilda Bastian's example of surgery and a pain outcome appears to be a valid example of an unblinded trial with a subjective outcome.

But even if that's questioned, PACE isn't problematic just because it used only subjective measures in unblinded trials. It also ignored objective measures,
... switched outcome measures,
... used one recovery measure that meant the patient could worsen from entry but be considered better,
... used selection criteria that selected patients who did not all have ME,
... failed to account for the biomedical evidence that directly discredits its claimed mechanism of action for the therapy,
and I'm sure others can fill in more

Its the whole package that makes that trial such a problem
Click to expand...

I'm not sure why pain outcomes are valid for an unblinded surgery trial. An important question is how you would judge reliability which I can only see being done by correlating with other non-subjective measures (or less subjective measures). I suspect there is also an effect size issue - where there is a big effect size and a likely bias introduced then the effect size may well dominate the bias but if the effect size is small (or non-existent) then the bias could well dominate the effect size causing spurious results.

PACE was problematic due to the way it was run but some of the data is now available and if i remember correctly that shows an improvement (albeit small) in terms of the subjective outcomes. It doesn't show any significant recovery. GET was the only one showing a significant improvement in the 6mwt but that could be due to drop outs and just a slight increase in fitness or practice in walking (which was the activity most selected) - even then the improvement wasn't good. BUT the outcome switching etc doesn't change the more detailed data we now have. It is the subjective outcomes that simply make most of that data unreliable.

With the selection criteria it could be argued that the interventions would work on a subgroup of patients although given the PACE results comparing different criteria (Oxford, intersection(Oxford, CDC), intersection(Oxford, London) giving similar results then I would say that hypothesis isn't good.

So we are back to the subjective measures being unreliable. I said earlier about trying to test the reliability of subjective measures by corrolating with less subjective measures and this is one of the interesting things with PACE because the effect sizes are greatest in the measures which are most subjective (the CFQ) this suggests a lack of reliability. I can't remember if there is correlation. But testing along these lines could be something that a protocol looks at - although it is complex.

Again effect sizes in PACE were quite small and thus I would say the uncontrolled factors of bias have a high risk of dominating the effects from any intervention.
 
strategist said:
Objective outcomes may also be somewhat unreliable, depending on what is measured and how.
Click to expand...
Yes. Another possible loophole in PACE occurred to me, regarding PEM. Without round the clock activity monitoring, PACE almost certainly never captured participants' activity levels during PEM, meaning even the "objective" activity readings were badly clipped at the bottom end, effectively chopping off all the lower readings from the graph; readings that would be very low indeed. So any averaging of activity level readings would be heavily skewed upwards from reality.

If a trial is run without any insights into PEM, then one of the most likely reasons for participants to not attend sessions would be PEM. Who is really going to be attending a session in the midst of a PEM bout?

Moreover, if a pwME knows they have session coming up then they will be tempted to do what they need to avoid PEM at the session. This goes beyond simple activity exchange between session / non-session activity, because participants are likely to "over swap" in order to perform on the day.

I wonder how much of the PACE data actually has information from bouts of PEM? I suspect the whole issue of PEM is brushed under the carpet.
 
MSEsperanza said:
I think this example and all the discussions on other forum threads show that it's more complex than just subjective and objective outcomes; it's also about assessing the adequacy of outcomes in general, as well as the best way to measure and report both subjective and objective outcomes.
Click to expand...

Ideally the outcome measures should relate to things that a relevant to the patient in the real world. Examples, might include: ability to work (or equivalent); actigraphy over a decent time period (a week or two) and followed up on at 12 / 24 months; reduction in medication usage (measured independently via GP records over time); absence or reduced biomarker of disease (independently measured); improvement on VO2Max 2-day testing; neuropsychological testing by independent assessor and so forth. I think this is one area that patients can have more input - where they have been notably absent in the past.

We use the short term measures like the 6 min walking test pre and post NHS pain management programs - where pain reduction/cure is not an expected outcome - it's about helping patients to cope well despite ongoing pain. This measure generally improves modestly over a pain management programme usually - but by a small amount and nothing like compared with age/sex matched controls. So, this type of data can be used to demonstrate an effect (in context) but there is the possibility that it could be used in a misleading / over optimistic way. Occasionally, we will have patients who were enduring through their pain and persisting with a lot of physical activity despite pain (usually those early on in their condition) who will score lower afterwards - but it is because they are now pacing themselves better. Context is important and interpretation.

Having secondary subjective outcomes measures, alongside objective ones, in a trial is very helpful as it gives a wide range of functioning (activity, physical, emotional, mental health, self efficacy) that would also expect to improve. It's informative and provides a backdrop to support an intervention and could help translation into clinical practice from research.
 
Last edited:
Adrian said:
With the selection criteria it could be argued that the interventions would work on a subgroup of patients although given the PACE results comparing different criteria (Oxford, intersection(Oxford, CDC), intersection(Oxford, London) giving similar results then I would say that hypothesis isn't good.
Click to expand...

Dont want to go down this rabbit hole except to say that PACE's pronouncements about comparison across definitions were meaningless because they manipulated those definitions - reportedly for London and definitely for CDC, where PACE only required the "any 4 of 8 symptoms" part for 1 week, not the 6 months required by Fukuda. So who knows what conditions they actually evaluated.
 
Medfeb said:
Dont want to go down this rabbit hole except to say that PACE's pronouncements about comparison across definitions were meaningless because they manipulated those definitions - reportedly for London and definitely for CDC, where PACE only required the "any 4 of 8 symptoms" part for 1 week, not the 6 months required by Fukuda. So who knows what conditions they actually evaluated.
Click to expand...
I was careful in stating the intersection of Oxford and the other criteria as well as this is what they measured and yes they did strange things with the criteria.

But I don't believe they selected a set of patients which their favored interventions would work with my intuition looking at the data is that they are just reporting bias effects. Although there may not be sufficient data (in their private data set or in what is public) to confirm this.
 
Hilda Bastian said:
I realize you hold the position adamantly, and nothing I say will change that, but just for anyone who's seeing this and not reading the whole thread, this is my position: an entire trial is not necessarily valueless even if the data from one, or some, of its outcomes is biased. (I explain that here.) Nor are subjective endpoints always valueless, even on effects, and sometimes there is no endpoint more valid than a patient-reported outcome.
Click to expand...

The innuendo is still there Hilda -'adamantly'. I hold the position because it has been empirically verified thousands of times and is it is in effect a tautology. What we mean by a subjective outcome is one that can suffer spurious influences if people have a prior idea what the answer should be and an unblinded trial is one where they do. I will clarify my original comment in a minute but the suggestion that I hold this view through some sort of prejudice or vested interest is to commit exactly the impugnment that you were complaining of in others!

Thanks for replying and making clear your arguments - I and everyone else here really appreciates that. But the argument here is irrelevant again. As others have said, nobody would be as stupid as to imply that the presence a single subjective endpoint makes a trial valueless - why should anyone be so irrational (that impugnment again?). And I have already dealt with the red herring about subjective endpoints not being valueless. I actually put it in capital letters last time to try to make sure it got read! Of course they are important and valuable - if they are free from bias - which they will be in a blinded trial. Another red herring I get is that blinded trials are hard to do for therapist-delivered treatments. Indeed, that highlights the weakness of the trials we have. It does not mean that it is OK to treat inadequate trials as if they were somehow adequate. Reliability of evidence is a measurable factual matter, not something that shifts with your needs. Of course sometimes the subjective endpoint is more valid. Any fool can see that. But that is not the point.

Let me rephrase my original comment to give the meaning that is transparent to all members here but you seem to have difficulty with. It was I think self-evident in terms of the context in which the comment was made.

The use of subjective outcomes (alone) from unblinded trials to judge the usefulness of a treatment is valueless.

The problem with the exercise review is that the trials included either do not have primary objective endpoints that would be adequate (which relates to the superiority of subjective ones we have agreed on) or they were not meaningfully controlled because the psychological framing of the comparator was notcomparable. That is a different issue and one Ernst has interestingly commented in the context of the Lightning Process recently. I remember noting that one study had fitness as an objective endpoint that improved, but, as we agree, fitness is not a good index of whether or not someone feels less ill, which is what matters. OK the context is complex, but the bit about the combination subjective endpoints and lack of blinding is not. As other members have pointed out all sorts of things about both the trials and the review made things worse but we are discussing this one point.

Hilda Bastian said:
I did not at any time imply that's what you were saying
Click to expand...

I don't understand that comment. If you were not trying to defend the value of subjective endpoints- presumably against a suggestion that they were inferior - why do you keep mentioning it? I am baffled here.

Hilda Bastian said:
It's not the case that the only relevant outcome of a trial of epidural analgesia in labor for pain relief is pain. Even leaving aside other possible maternal outcomes (e.g. whether it increases the risk of forceps or cesarean, and the potential harms of epidurals to women), objective impact on a newborn of drugs in labor are clearly critical. If the first studies had found marginal pain relief and major newborn harm, epidural analgesia would have been dead in the water.
Click to expand...

As someone else has pointed out this is now about harms and clearly falls outside the meaning of my original comment in the relevant context. Yes trials can have value in showing harm. PACE would actually have great value in showing that CBT and GET do not work usefully if it were not for the dreadful mess about recruitment and generalisability. PACE is of value in showing that the theoretical model was wrong - if it was right the objective measures would have paralleled the subjective ones and they definitely did not.

The first paper I ever wrote was a retrospective review of a surgical procedure for pain in the wrist. It was an unblinded study with a primary subjective endpoint. Its value was in the humbling learning experience for a Dr J Edwards who later realised he had made every mistake we are now discussing. But it was valueless as a guide to the usefulness of the treatment.

Hilda Bastian said:
The real examples against the argument that an entire trial is worthless if it's unblinded and it includes even one subjective endpoint?
Click to expand...

But that is not a sensible reading of my original comment - as I think everyone else here would agree. It makes no sense. I think I did actually use the term 'primary outcome' initially and I am sorry if I did not put it every time. But everyone else here is up to speed on the need for primary outcomes prespecified, the problems of multiple analyses and so on. When I say an unblinded trial with a subjective outcomeI assume that people will understand me to mean a primary outcome or an outcome used to decide whether or not the treatment is useful - that was the problem with the review.

I would highly recommend reading many of the threads here.We have had along discussion about combining subjective and objective endpoints into composite prespecified primary outcomes. Rheumatologists have done this for 25 years with the ACR criteria of improvement RA. You have to use a multiple threshold system where improvements in subjective features are combined with objective endpoints that corroborate their validity. It is probably the only way to assess many treatments for ME. It does not completely deal with the problem of bias (especially in very large trials if endpoints are defined statistically) but mitigates quite well.

I am afraid that you are straw-manning here, Hilda. It is a needless diversion from the work in hand - to try to get a Cochrane review that is fit for purpose.
 
Adrian said:
I'm not sure why pain outcomes are valid for an unblinded surgery trial.
Click to expand...

Indeed they are not. Orthopaedic trials have been notorious for recommending procedures based on the sort of flawed follow up I used in 1973.

Another thread that is relevant here but I think would distract is the CCI one. There is no doubt that both doctors and patients may appear to believe in dramatic benefits from operations that almost certainly do nothing specific.

But Hilda was not arguing that pain was a valid measure of success in unblinded trials, rather that other valid measures may be included - as I think we can all agree on.
 
At the risk of distracting with CCI, I just read on Twitter that in Italy, a doctor claimed multiple sclerosis was caused by a problem in the neck vein called called chronic cerebrospinal venous insuffficiency (CCVI). Patients underwent surgery and some claimed to be cured. Only a few specialists were able to make the diagnosis and perform the surgery (just like with CCI). In the end, multiple studies failed to support these claims, like for example this properly controlled clinical trial https://n.neurology.org/content/91/18/e1660.

I think these stories are important because the general public and many researchers do not seem to realize how easily a convincing illusion of treatment efficacy can arise. I also see parallels with the lightning process story unfolding in Norway at the moment.

Unfortunately demanding properly controlled clinical trials for ME/CFS can easily lead to being called an unreasonable extremist.
 
Last edited:
Jonathan Edwards said:
But Hilda was not arguing that pain was a valid measure of success in unblinded trials, rather that other valid measures may be included - as I think we can all agree on.
Click to expand...

Yes I was also arguing that other measures from such a trial may be valid. I also like the idea of looking for correlations between multiple measures if they are essentially trying to measure the same thing then all the measures should correlate. In some instances there may be errors but if the measures are meaningful and essentially measuring the same thing (improvement) then they should correlate (unless they just measure one aspect of improvement which doesn't improve). Very much like a likert scale where the same question is asked multiple times and scores are added up to reduce the error if one question isn't understood by someone.

But I do wonder. If people running a trial choose subjective end points as their primary outcomes should this reflect on their ability to run a trial well and therefore in using other data from the trial should we be asking 'is that data reliable?'. If they can't get the basic things right then we may need to be very careful in taking anything from them and do more quality checking.
 
Hi @Hilda Bastian.

This discussion has stirred a lot of interest because it matters enormously to us what happens with the new Cochrane review. It will have a major impact on our lives and health. I'm pleased you have been appointed to lead the IAG and that you have joined us to engage with our concerns.

I don't know whether you have participated in discussion on patient/carer/scientist forums before. When a discussion provokes a lot of interest among our members, many will want to add their pennyworth to the discussion, even at the risk or repeating what others have said and being emphatic and sometimes not entirely clear in our statements. I am aware I am guilty of doing that and apologise if that felt like I was directing my comments at you personally. It can feel a bit as if one is under siege if a whole lot of people pile in to repeat basically the same point. I hope that won't put you off engaging with us and the points we make.
 
Last edited:
Joan Crawford said:
As the human is largely highly loss averse - approaching the idea that GET is not effective is psychologically a difficult process - if the researcher / clinician has truly and wholly believed in it. The belief that GET for ME/CFS works will be maintained pretty much at all costs - and the human will change the goal posts until the desired outcome for the belief is proven - i.e. persuade themselves, co-researchers, funding agencies, colleagues, peer reviews that switching outcomes and relying on subjective measures etc is ok - unless they are held to account independently (that should happen via peer review...) and by objective evidence. Otherwise it is all belief and wishful thinking - no matter how well intentioned or desired.
Click to expand...
A behavior we saw play out in the FINE trial, which posited that "there is no disease, you can aim for maximum recovery":
'There have been one or two times where I have been worried because they have got angry at the patients...that anger has been communicated to the patients. Their frustration has reached the point where they sort of boiled over... there is sort of feeling that the patient should be grateful and follow your advice, and in actual fact, what happens is the patient is quite resistant and there is this thing like you know, "The bastards don't want to get better"...I think it's a difficult thing for all therapists and I think basically over the time you just basically learn to cope with it, and but they have not had time.' (Supervisor)
Click to expand...
https://implementationscience.biomedcentral.com/articles/10.1186/1748-5908-6-132

Being emotionally committed to an intervention as a purveyor or researcher of that intervention is highly problematic. This is the case for the BPS model of ME, it is an opinion, a belief system, it feels personal, proponents feel it should work because they want it to work and refuse to accept when it doesn't.

Science works by failing to falsify a hypothesis. Here there is no hypothesis, only assumptions held as true and all contradictory evidence is simply dismissed.
 
Joan Crawford said:
From a psychologist's perspective - my understanding of GET and it's underlying rationale.
Click to expand...
Really good summing up.
Joan Crawford said:
It is in essence a behavioural intervention to try and overcome a fear / phobia of movement, activity and exercise. Phobias are straightforward to treat and overcome in many instances and circumstances. Again easy peasey (If it were true).
Click to expand...
Yep. If ME/CFS really was a deconditioning problem and pwME's behaviour needed to be changed in order to fix it, then everyone here would have been running around and jumping up and down again a long time ago. But to coerce and brainwash people into undertaking a physical treatment, for an illness totally different to that on which the trialled treatment was modelled, seems bordering on criminal medical negligence, or worse. When the modelling is that wrong, you cannot have anything like the necessary confidence that harms have been detected or identified, especially if the tell-tale signals are misconstrued as normal aversion behaviour for the illness model presumed. If compensation lawyers ever get involved here, I would think that to be a key part of any argument.
Joan Crawford said:
From a theoretical perspective the GET model should easily show high levels of change if the model was correct and had good validity - including face & construct validity. I would expect large effect size changes which can be measures objectively, subjectively and can be independently verified. Assessors pre and post therapy can be independent of the treating clinicians. That could/should be done to reduce bias too. Small subjective changes should ring large alarms bells. It does to me.
Click to expand...
Yes, as has been said many times, and recently in this thread, PACE actually does show very effectively that the deconditioning model is wrong. If the science had been done and reported honestly then we could have all congratulated the authors for proving that ... because they have. Strangely ... they reported differently.
Joan Crawford said:
Absence or no change in objective measures or the active dismissal / minimisation of the usefulness of objective measures by researchers should be ringing massively large clanging bells of bias.
Click to expand...
Which of course it has been ... it's just that none of the b*ggers are listening who should be! And the bell ringers get told to shut up and stop making a nuisance of themselves. Hopefully if Cochrane manages to now get to grips with all this, then that might change.
 
rvallee said:
A behavior we saw play out in the FINE trial, which posited that "there is no disease, you can aim for maximum recovery":

https://implementationscience.biomedcentral.com/articles/10.1186/1748-5908-6-132

Being emotionally committed to an intervention as a purveyor or researcher of that intervention is highly problematic. This is the case for the BPS model of ME, it is an opinion, a belief system, it feels personal, proponents feel it should work because they want it to work and refuse to accept when it doesn't.

Science works by failing to falsify a hypothesis. Here there is no hypothesis, only assumptions held as true and all contradictory evidence is simply dismissed.
Click to expand...

Spot on.

I use the quote from the FINE trial when i do my critical approach to MUS talk with professionals. Just jaw dropping stuff.
 
Trish said:
Hi @Hilda Bastian.

This discussion has stirred a lot of interest because it matters enormously to us what happens with the new Cochrane review. It will have a major impact on our lives and health. I'm pleased you have been appointed to lead the IAG and that you have joined us to engage with our concerns.
Click to expand...
Yes, I also would very much like to endorse that view @Hilda Bastian. Your robust engagement here in seeking to get to the truth of things really heartens me you will do the same within Cochrane. I'm also well aware that no matter how capable you are (and I'm sure that is 'very') there is still a steep hill to climb for you, and not just on the science aspects we mostly witness here. I imagine some of the other things you need to achieve within Cochrane must feel like having to turn the proverbial oil tanker around.

I sometimes think, when in the midst of robust discussions like this one here, that if we were all discussing this in the same room, then things would clarify much more easily.

So your willingness to engage, and above all your determination to do what is right, is hugely appreciated by us here. Sincere thanks.
 
You must log in or register to reply here.
Back
Top Bottom