Independent advisory group for the full update of the Cochrane review on exercise therapy and ME/CFS (2020), led by Hilda Bastian

[my bold]

How very strange. So because all disorders impact functionality, then functionality cannot be objective and cannot be measured ... ? Trying to fathom the logic is like trying to put your finger on a blob of mercury. So by the same logic, a broken leg stops you playing football, but subsequently being able to play twice a week would not be a reasonably objective measure of recovery? Along with other measures of course.

My mother suffered from deep depression throughout her life, which fluctuated over long periods. When she was more functional there were all sorts of predominantly objective measures - going to the shops, riding her bike, going out. And when not functional also a pretty objective measure - staying in bed day and night.

But no, the 'defence' here is that others know nothing of good scientific method.

 

It looks like that banner is above all their reviews, eg: https://www.cochrane.org/CD011776/NEUROMUSC_managing-symptoms-motor-neuron-disease

 

I agree.

 

We replied from the physios for ME account and also all replied individually as well! Karen even got blocked by them!

 

Indeed. Maybe Mark Vink has already done that?

 

Sure. The key point is that if participants (and any assessors) genuinely don't know which arm they are in, response biases and related distortions will be relatively similar across all arms.

Objective outcome measures of functioning in an unblinded trial don't eliminate bias, but they show that such improvements are genuine, rather than just a change in how someone filled in a questionnaire because they felt more optimistic or distracted from symptom focusing (or other response biases).
Obviously objective outcomes on their own are not enough, someone could do a lot more activity but report much more severe symptoms.
Secondly, outcome measures need to take into account baseline levels, rather than looking at mean group changes. The clinical benefit differs based on the baseline levels - many questionnaires fail when it comes to linearity - changes on different questions and from lower levels of functioning can have much greater real-world impact.

Many of us have given this a great deal of thought and we would love to develop our own composite outcome measures for use in clinical trials, if given the opportunity!

 

Yes, there is a big difference between reporting what could happen or what a participant thinks they could do, versus what they have actually done/has actually happened. Biases such as recall bias certainly apply, which is why an outcome such as urinary incontinence needs to be recorded daily. Likewise, this is why a participant reporting that they are in full time work with minimal symptoms is much more convincing than someone answering "Compared to one year ago, how would your rate your health in general now?".

For the SF-36 PF questions for example, there is a big difference between asking "Does your health limit these activities?", rather than "How often have you done these activities in the last week?"

Oh and while I'm at it, many of the studies that try to statistically determine "Minimal clinically important difference" are based on a fundamentally flawed circular argument - bias on the reference question is correlated with bias on the other questions, hence these statistical approaches are fundamentally flawed because they might not be measuring any fundamental health improvement at all. These approaches also fail to consider how biases on questionnaire answering behaviour in prospective studies can differ between studies.


Using a Delphi technique would be much more worthwhile, but the determinations of what is clinically relevant would need to be tested repeatedly in a wide variety of groups from different cultural and age backgrounds.

 

...which would count as a value from the trial (that it raised questions about other evidence for you, I mean). It provided data on adverse effects, and that's an unusually large sample size of women for that non-surgical treatment. (Since there is not only a single relevant trial, a single trial wouldn't be the appropriate sole basis for informing women.)

I didn't claim I was looking for an example that showed that: I explicitly said I wasn't, on grounds of time and value of such an exercise, and that I wouldn't vouch for the trial. I was addressing the question of what does that look like, and the thing that started all this: are such trials valueless. That you think it's some good stuff mushed up with stuff that's a bit whiffy is, as I've said several times in this thread, a key point (as per my post here): studies of all kinds are patchy like that, blinded or not. And that's why the systematic reviewing community has been moving towards rating the uncertainty around data per outcome, not throwing out babies with bathwater (or calling bathwater babies).

 

I wish your first point was true! There are many other potential sources of bias - what a particular practitioner (or more) did that deviated from the protocol, things going differently at one hospital versus another... And then all the things that can go wrong with the report (selective outcome reporting etc).

I truly understand the point with the particular types of trials you are speaking about, and I have criticized the PACE trial on the grounds of inadequate measurement tools. But you aren't going to convince people who have battled for exactly the opposite - for people's voices about their care to be taken seriously and given weight - to sign up to dismissing patients' reports so categorically. There are stronger arguments that do carry weight,* so it's not even essential. And other than blinding of allocation concealment, how good the evidence for blinding is from situation to situation is a topic that is still being studied and debated, because [insert here my usual "it's complicated" ].

* [including that it is demonstrably a problem in a specific case]

 

But surely @Snow Leopard's first point is true - referring as it does specifically to response bias and related distortions. There was no suggestion that other kinds of bias don't exist.

 

Patients with devastating illnesses tend to have difficulties remaining objective when evaluating a treatment because they so desperately want it to work. As such the efficacy of treatments is often greatly exaggerated in anecdotal reports or unblinded studies.

It is possible that some people do not understand this due to not having been in a position to observe this phenomenon, and confuse statements about self-report/subjective outcomes being unreliable with a poor attitude towards patients. Acknowledging this vulnerability of patients does not mean dismissing the patient voice. It is simply a fact that has to be taken into account if one wants to do competent research.

I have been involved long enough in the patient community to see that this difficulty staying objective is a common and major problem that leads to treatments being given credibility even when they later turn out to be ineffective. Patients are harmed by the side effects and expenses of undergoing the treatment that does not actually work.

The open-label and the placebo controlled studies of Rituximab for ME/CFS are a good example.

 

Absolutely. I was going to pick up on the same thing.

If a source of bias exists then it exists, and no amount of arguing other points will change that. Any source of bias must be addressed in its own right, else why run a trial. And if such bias significantly influences a trial's primary outcome, then the trial becomes worthless for any assessment of that primary outcome. Correcting for bias - any kind of bias - is not being dismissive of patients' voices.

The correction being suggested in this thread, for response bias in unblinded trials, is actually inclusive of patient reporting: ensure at least the primary outcome is an objective measure, and that any subjective measures can be validated/calibrated against one or more objective measures. If objective and subjective measures all match up, then wonderful. If not then you get the chance to know there is an issue and address it, else another far more insidious form of bias creeps it - the pan-trial illusion that the biased measures are unbiased, with nothing to disprove that ... p*ssing into the wind basically.

 

Yes! We have to be very careful to not conflate "self reported outcomes" with "subjective outcomes"; although closely related they are not the same thing.

Sources of bias can be hierarchical, layered. The very process of self reporting has the potential for bias: poor memory, difficulty ascribing a value to continuous variables, consciously lying for whatever reason, unconsciously believing something to be different from the reality, etc., etc. But a layer down from that, the variable being reported may itself be highly subjective and nebulous. Pain, fatigue, cognitive impairment, etc., etc.

However, there is also the possibility to self report objective measures, which is why I'm labouring the point of there being a hierarchical layering of bias here. An objective measure, independently acquired and validated, is pretty objective. So how objective is a self reported objective measure? I think the annoying answer is: it depends. The objective measure needs to be something that can be readily self reported. How many times did you go to the shops last week. What times did you get up and go to bed last week, including all times through each day, etc. If reported accurately then these measures could be very objective, but reporting accuracy could also be a major issue.

Reporting accuracy will likely be better if discrete raw values are asked for, not woolly infinitely variable values, or discrete values arbitrarily ascribed to woolly values. I imagine that within a trial some measures could be taken to improve reporting accuracy, or at least corroborate it.

So self reporting is its own source of potential bias. And the measures being reported will themselves have potential for their own degree of bias. Putting the two together means both must be heeded.

 

So much agree from me on this comment! Absolutely an individual outcome in a context has its own degree of bias - and it's higher if it's subjective/unreliable/without a solid research basis that it measures what it claims to measure/manipulable... And subjective outcome measures have a higher risk of bias - that just doesn't necessarily make them useless, and it doesn't doom the trial they're in to uselessness either, especially if other sources of bias are low.

There's a spectrum from very subjective to very objective, and in the middle there isn't universal agreement on some dividing line. Very easy to see where a very subjective outcome is so subjective, but awfully easy to over-estimate how objective "objective" outcomes are.

(Poor memory can affect patient-reported measures of any type - if a trial goes on a long time, and you end up filling in a week or two from memory, it's easier for some things than others, depending on how memorable it is, not how unambiguous it is - eating carrots or hours you slept, for example, versus days you went to a workplace.)

 

Yep, but how is information about 2 procedures reliable, if it says their outcomes are equal, when patients say one hurts more than the other? Yes, it's less reliable when you can't blind for the difference between the procedures, but it's a question of degree of reliability, not "reliably precisely right" vs "useless", with nothing in between. Getting a rough idea if bad pain is very common or quite uncommon can be quite valuable, especially when you don't necessarily trust the biases of the person telling you "most people only experience slight discomfort".

 

I seem to have missed this, but what was the update on 21 May?