Bias due to a lack of blinding: a discussion

This definitely seems to be true.

 

Indeed. And that is what the SF36 was designed for - for looking at overall population trends.

Part of the reason it has ended up as the go-to measure in these sorts of trials is expediency (laziness?) and comparability. Although whether its use truly makes all these trials comparable is moot. In some ways it gives way to an illusion of comparability, which is more dangerous.

It would be better if each trial designed their own scale that was relevant to the condition and patient group they are trying to treat. They also need to take more care over whether such a measure can be repeatedly used over time. But then they wouldn't be able to combine results in meta-analyses. There is an added illusion that because these scales have been used so much in trials over the years, that they were properly validated for the use they are being put to. In most cases, they probably (definitely) weren't.

But who is going to say anything about that now?

 

There's a third option, mentioned by the authors, namely a lack of precision. The previous studies of this kind did find an effect for blinding when subjective outcomes were used and in the current study, the confidence intervals are wide, suggesting that other samples might find a clear effect. It's possible that if you do this type of study a couple of times, only one would happen to find no effect of blinding (and that the Moustgaard study happened to be that one). I don't know how plausible this explanation is, but it's possible so best to keep it in mind.

I don't think this is about the validity of meta-analyses really. What they do in this type of studies is to look at trials which have a particular methodological characteristic (for example patients were blinded) and similar studies that don't have that characteristic (patients weren't blinded). Reviews and meta-analyses are used to find studies that differ in this aspect (blinding of patients) but are similar in all other regards (the same type of intervention, study population etc.). I think that's what they mean with "meta-epidemiological study".

As Jonathan suggested earlier this isn't how science usually measures effects. Suppose someone in Australia reports that ME/CFS patients fatigue got 30% better with treatment A. And a Canadian doctor reports that his ME/CFS patients' fatigue only got 15% better following treatment B. From this information we wouldn't normally conclude that the fatigue of 15% more ME/CFS patients got better with treatment A than with treatment B. We would require a proper randomized controlled blinded trials so that all possible confounders are controlled for.

Meta-epidemiological studies, however, use the methodology sketched above: they take trials that look alike except for the characteristic of interest (blinding) and then subtract the treatment effects to see whether the characteristic is associated with inflated treatment effects. One obvious problem could be that the trials being compared differ in all sorts of respects that have an influence on the treatment effect. That's why I hope to have a look at the individual trials that are being compared to better understand the results.

 

The sad truth appears to be that large portions of research done under a mental health banner, ostensibly to help patients, are junk. Patients are just being taken advantage of by professionals that seem to have agreed to collectively pretend that methodology that is with certainty known to be unreliable is actually reliable. They can't offer any coherent arguments to justify what they're doing or respond with distractions.

There is no illness that can be objectively measured that meaningfully improves with placebo. Not ensuring that treatments delivered are better than a placebo is very negligent.

Even with mental health conditions at least they could try to find the most reliable way to measure improvement. Instead they appear to be trying to find the therapy and methods that give best results on biasable outcomes, thereby in effect potentially gradually refining therapies to be more and more biased.

 

yup

 

Indeed. I am constantly reminded of Emerson's ' A foolish consistency...

 

I will have to look that up. I have no idea what that refers to. Is that Ralph Waldo?

 

Really, David, I thought you were a well read man of letters.

Ralph Waldo Emerson: A [certain? - sources vary] foolish consistency is the hobgoblin of little minds'. He attributed it to statesmen and religious folk but there is no lack of it in medicine and science.

 

Ah yes, ok. the hobgoblin of little minds part I recognize, although I wouldn't have been able to attribute it to Ralph.

 

Thanks! I read a lot of Emerson in my American Romantics course in college (Emerson, Thoreau, Hawthorne, Melville, etc). That was in, uh, 1976. So, you know. A while ago.

 

Using patient self report in a trial is always subjective but in ME the situation is even worse.

The SF36 gives numbers you can add up with a yes or no for each section so has a certain objectivity. Patients are asked if they can do things or not so the answers should be some reflection of reality.

However, the Chalder fatigue scale does not even do that. Marking a score of 1 - 10 for how fatigued a person feels might reflect reality approximately but trying to compare how you feel with "usual" gives a subjective answer to a subjective questionnaire.

The CFQ fails even the simplest definition of validated to mean the questions can be understood. If they really want to know how we usually feel then do they mean usual after 12 months of treatment? But that would give the same score as the beginning if the trial was a success so the usual feeling was now a lot better than at the beginning.

But we are told they mean usual to be what you felt when well - then why did they not say that? But that fails too. I can't compare how I felt 52 years ago with now!

I don't understand who validated this scale and thinks it is useful and someone who defends it is showing their ignorance and prejudices. They are accepting the word of authority without question over the concerns and well being of patients. I am sorry for their patients.

 

The trouble is the numbering only really gives an illusion of objectivity, a sort of scientists' comfort factor (certain scientists anyway). Presuming that someone's perception of changes to their perceptions follows a nice linear numerical scale, is a pretty fluffy presumption. And the way people's perceptions map onto reality is going to vary according to what sort of perceptions are being 'measured', so the mismatch between reality and numerical scale is also going to vary according to what is being 'measured'. So not only are all these numbers pretty meaningless, the notion of then adding them all together then gets even more fantastical. To me it all seems rather pretentious and not very scientific at all.

 

Well said. Its just another example of the "babble" family.

 

I agree that trials should never be based on things like the SF36, they are not objective enough. I may be remembering it wrong but I think that scale was designed to help doctors track how well cancer patients were doing, not as a research tool.

What I meant was that trials which rely on subjective scales should make some attempt at objectivity by asking questions like "How often do you manage to leave the house on the average week?" which the patient can count up or "Can you still do the sports you played before you were ill?" which can be answered yes or no.

Then there are scales where you have to estimate how much pain or fatigue you have on a scale of 1 to 10 which are little better than guesses so take the results even further from anything objective.

But the PACE trial and other BPS trials excel themselves by using things like the chalder fatigue scale where the patient has to guess what the question means before they guess at the answer. As far from objective as it is possible to go.

 

Yep

 

Hilda Bastian has blogged about this (although she mostly seems to summarize and quote other people):
https://blogs.plos.org/absolutely-m...linding-clinical-trials/#.XjdIM7_EPcs.twitter

 

Seems very fence sitting to me.
The metaBlind paper is based on too poor a methodology to be worth considering or repeating in my view and Bastian agrees that the person she quotes as saying that has a good point. But then she ums and ahs.