Independent advisory group for the full update of the Cochrane review on exercise therapy and ME/CFS (2020), led by Hilda Bastian

OK, but can't do it right now - have to work.

 

I have admittedly missed parts of this thread but I'm struggling to understand why so much focus and discussion just on the issue of subjective measures in unblinded trials.

@Hilda Bastian's example of surgery and a pain outcome appears to be a valid example of an unblinded trial with a subjective outcome.

But even if that's questioned, PACE isn't problematic just because it used only subjective measures in unblinded trials. It also ignored objective measures,
... switched outcome measures,
... used one recovery measure that meant the patient could worsen from entry but be considered better,
... used selection criteria that selected patients who did not all have ME,
... failed to account for the biomedical evidence that directly discredits its claimed mechanism of action for the therapy,
and I'm sure others can fill in more

Its the whole package that makes that trial such a problem

 

Indeed. Instead, they buried the results from these objective outcomes in subsequent papers. It took patients to point out that the results from these objective outcome measures didn't match up with the subjective outcomes:


https://journals.sagepub.com/doi/full/10.1177/1359105316675213

 

Personally I think it most likely because the key players are supremely adept at cultivating high level influential relationships across a wide sphere of influence. The sort of 'scientists' whose mantra is "it's not what you know but who you know". They then, in effect, choreograph the behaviours of other influencers, to achieve the broader outcomes they seek. Of course those seeking to limit NHS funding may well do their own influencing - knighthoods come to mind. And yes, I am a big fan of "Yes Minister" .

 

Did anybody actually say this? I think it's clear for all of us that including relevant subjective endpoints is always highly valuable, also in unblinded trials, but the point some of us repeatedly are trying to make is that if you, in unblinded trials, don't at the same time, in addition to subjective outcomes, apply robust objective outcomes, the evidence on that particular treatment effect is zero.

The examples you provided in the quote above (length of the surgery, blood loss, mortality) all seem to me to be relevant objective outcomes but they measure different treatment (side) effects. I think nobody suggested that objective outcomes are spoiled by also measuring subjective outcomes?

I think part of the trial design is to word a hypothesis and define which treatment effects matter according to the hypothesis and how they are measured. If you're saying a particular subjective outcome like pain relief is not the most or the only relevant treatment effect in a particular study that also objectively reported other relevant treatment (side) effects, then why use this example as an argument in favor of using solely subjective endpoints in trials were blinding is not possible?

(Edited for clarity.)

 

I have been wondering if the issue is one of whether or not the problem is temporary...
So - if you are measuring pain which is not chronic, as in childbirth, then a subjective measure is fine. But if you are measuring a subjective measure for a long-term illness/syndrome, then its not ok. You need an objective measure to be able to say that the subjective measure is actually worth measuring. Perhaps if you are looking just at pain, then a subjective measure is ok - but if you are looking at chronic pain, then your measures should include something objective about functioning as well as a subjective rating of the pain itself.
And - i do think that when we are talking about ‘therapies’ designed to alter the way we think, subjective measures, particularly when they are short-term only, are particularly problematic. I suppose the field of depression must be rife with problems of this nature...?

 

Absolutely. As I think @Jonathan Edwards has said in the past, subjective outcomes are fine even in open label trials provided objective outcomes are also employed to calibrate them against.

 

My attempt to better explain how I see the lack of blinding + subjective outcomes problem.

Lack of blinding makes subjective outcomes unreliable.

Unblinded clinical trials that attempt to determine if a treatment is effective with subjective outcomes are generally worthless.

There may be exceptions to this, if nonspecific factors are very carefully kept equal between treatment groups, but that is very difficult.

CBT/GET and studies of similar interventions do not keep nonspecific effects equal between groups. Bias that will affect subjective outcomes is built into the interventions because their goal is to modify the patient's perception of their illness and symptoms and induce an optimistic state of mind. The investigators seem to believe this is one of the active ingredients of the therapy. They do not consider the possibility that all they're doing is using known methods to introduce bias and confusing this with having found an effective treatment. If an intervention was more than just bias, then objective outcomes should improve. The objective outcomes however are consistent with there being no treatment effect. Proponents of CBT/GET and similar interventions simply ignore unflattering objective outcomes when they make claims of treatment efficacy.

The design of CBT/GET clinical trials and the many other problems with them (like outcome switching or absurd definitions of recovery) make it possible to obtain positive results for almost any intervention. This is not serious science. It is at best incompetence, at worst a deliberate attempt to mislead. This problem is not limited to ME/CFS, it is becoming a widespread problem affecting many interventions and health conditions. The ultimate outcome of this will be a proliferation of implausible and even absurd interventions (like for example the lightning process) that seem to work for a wide range of conditions that all have in common the absence of biomarker of disease severity.

Once researchers have obtained a positive result with such a flawed clinical trial, they tend to view it as confirmation that their explanatory models for the illness are correct. These models typically claim that the illness is somehow a product of the patient's mind (and this seems logical if an intervention that manipulates mental state appears to help).

Objective outcomes may also be somewhat unreliable, depending on what is measured and how. In the context of ME/CFS would not consider a brief improvement in objectively measure daily steps or a brief return to work as reliable evidence of treatment efficacy because the problem in ME/CFS is more one of being unable to sustain activities than doing them at all. If properly used, objective outcomes should be able to give much more accurate information on treatment efficacy.

 

From a psychologist's perspective - my understanding of GET and it's underlying rationale.

GET is aimed at increasing physical activity despite on-going symptoms and flare up of symptoms - because these on-going symptoms are understood within the GET model of ME/CFS to due deconditioning, misattribution of benign bodily sensations as malign, and patient focusing attention on these and getting distressed and avoidant of activity as a result. This cycle goes round and round maintaining the patients' symptoms.

The model asserts that flare ups of pain are normal when people start rehab after being inactive. Akin to the acute exacerbation of pain, stiffness and so forth post surgery or after an accident, for example, during acute physio rehabilitation. This is largely 'to be expected'. Any increase in pain or other symptoms is purely down to deconditioning, lack of fitness, stamina, lack of use, inflammation and so forth. Within the GET model of CFS model there is no physical reason why patients cannot increase activity safely and consistently - other than the patient blocks the process - due to fear of harm or worsening symptoms like pain and debility. The GET model assumes that the symptoms the patients experience are due to the patients misinterpreting and misattributing benign bodily sensations as malign. Once the patient starts to move and gets going, bit by bit they can and are encouraged actively to do more and more. The theory being that this process can be additive until the person is functioning well and largely as normal and has re-learned that their symptoms are benign. Lots of talk about two steps forward, one back - like in standard physio rehab of acute injury.

So, that should be straightforward to do in practice and to demonstrate objectively. Easy peasey. (If it were true).

It is in essence a behavioural intervention to try and overcome a fear / phobia of movement, activity and exercise. Phobias are straightforward to treat and overcome in many instances and circumstances. Again easy peasey (If it were true).

However, this completely misses the point: The patients main symptoms of post exertional malaise (PEM) and increased debility across a wide range of symptoms. The more activity (mental and physical) they do, the worse they feel and more debilitated they become. There is objective evidence for increased activity making pwME/CFS worse. When objective measures of activity are used and patients increase activity then go on to do less activity and report more pain and lowering of mood. That is the opposite of what would be expected by the GET model.

The patients voice is completely absent from the GET way of working. The underlying clinician beliefs and the GET model being used are not openly shared with the patients. When this is subsumed within the MUS model (TC et al see these things as the same, e.g. CF = CFS = ME = ME/CFS = MUS = SSD = BDD = FMS = IBS etc) sharing the underlying model is actively discouraged. It's opaque. This is, in my view, unethical. There is no way a patient can truly give their informed consent. It is the opposite of good medical care. It (GET) is 'done' to the patient who is not fully informed. I have no doubt that the clinicians who are 'doing' this are well intentioned - but that is not enough for professional, ethical practice. And no objective checks are made to see if the process is effective or has construct validity - that what is being 'done' in research or clinic resembles or is based on what the model says it is doing. It only appears to matter if the patient 'feels better'. Which they are going to because if they have failed to improve - it is by definition of this model - the patient who has failed. And no one likes that - so there is huge psychological and social pressure to conform, continue and smile - whether it is working or not. Especially if the clinician was nice, welcoming, supportive, caring and so forth. And the patient had been pre-primed and given messages throughput the process that GET was effect, safe and so forth. And that change was down to the patient to take forward. Not achieving a small, positive effect under such circumstances would be more startling.

From a theoretical perspective the GET model should easily show high levels of change if the model was correct and had good validity - including face & construct validity. I would expect large effect size changes which can be measures objectively, subjectively and can be independently verified. Assessors pre and post therapy can be independent of the treating clinicians. That could/should be done to reduce bias too. Small subjective changes should ring large alarms bells. It does to me. Absence or no change in objective measures or the active dismissal / minimisation of the usefulness of objective measures by researchers should be ringing massively large clanging bells of bias.

As the human is largely highly loss averse - approaching the idea that GET is not effective is psychologically a difficult process - if the researcher / clinician has truly and wholly believed in it. The belief that GET for ME/CFS works will be maintained pretty much at all costs - and the human will change the goal posts until the desired outcome for the belief is proven - i.e. persuade themselves, co-researchers, funding agencies, colleagues, peer reviews that switching outcomes and relying on subjective measures etc is ok - unless they are held to account independently (that should happen via peer review...) and by objective evidence. Otherwise it is all belief and wishful thinking - no matter how well intentioned or desired.

Joan Crawford
Counselling Psychologist

 

Excellent points w.r.t. the discussion on subjective & objective outcomes in an unblinded trial (bolding mine). @strategist sums up things very well. Hope you can read those posts @Hilda Bastian

 

Doing the above behaviours is what the human will do to preserve loss (face, professional identity) etc when they know they are wrong and lack the courage to face up to this fact. This kind of behaviour should alert others to the pretty obvious fact that there was a dud result that the researcher don't want to fess up too / lack the capacity to accept. When that's coupled with a vice like grip onto the 'model' (I'm right, I'm right, I'm right...... because I say so...) it won't be relinquished easily. And, if I'm being cynical, highly debilitated patients are the least likely group to kick up a fuss - so the researchers have carried on - in the face of objections. However, the researchers have completely misunderstood the human spirit: the desire to be understood and the drive for knowledge, health and decency. People will not rest until they are understood.

 

I'm still only able to occasionally read and post, so again apologies for missing most of the discussion when replying to a particular post.

Yes, having objective outcomes that are only self-reported is also problematic. I think there mostly exist some ways to reduce the risk that objective outcomes aren't adequately reported, though. Consumption of pain killers that need to be prescribed might be a feasible thing to double-check?

Of course, outcomes like taking pain killers or how many steps you walk each day are not utterly objective in all the senses of the term, e.g. if participants have a certain range of options to endure more pain or to endure the payback of any increased activity etc.

There are also a couple of other objective outcomes that aren't ideal to reliably measure an improvement. But then, a combination of objective outcomes could be used.

I think this example and all the discussions on other forum threads show that it's more complex than just subjective and objective outcomes; it's also about assessing the adequacy of outcomes in general, as well as the best way to measure and report both subjective and objective outcomes.

Many forum members are aware of this complexity and discuss implications for trial design, also together with investigators that are interested in getting our input.

However, that complexity to me still doesn't seem to challenge some basic facts about bias in trial design.

 

I missed that. What do you mean by a valid example, @Medfeb ? Do you mean that investigating pain as an outcome in an unblinded, not adequately controlled trial with self-reported pain relief as the only measure is valid?

I found these statements by @Hilda Bastian:

I am not sure what you mean here, @Hilda Bastian, but since I didn't read all posts I maybe missed it and you said somewhere that it also applies to unblinded trials that...

...and that this will provide reliable evidence?

I don't think you're suggesting only because people don't apply certain criteria in other studies this means it's OK to do studies that way?

Or that, only because it would be helpful to know something there always must be means available to know that?

Apologies for being trivial, but in case it could be helpful for the discussion:

Do you agree, that, if you have well-founded reasons to apply certain treatments but aren't sure about their superiority compared to other treatments, because for whatever reasons this can't be properly assessed, the best thing to do is to honestly say what you know for sure and what you don't know? That you should say what is your reasoning behind suggesting a certain treatment, but at the same time you should be very clear about the fact that there is not sufficient evidence to favor one treatment over another?

I think documenting and registering outcomes of a certain intervention in practice can be of great value. But it's something different than doing an RCT.

For ME research I think treatment documentations and observational studies could be very useful, but these approaches have to be clear about the biases involved, apply adequate protocols and shouldn't be used to exaggerate the evidence.

(Apologies in advance if I'm not able to get back to replies for a while.)

Edited for clarity.