Bias due to a lack of blinding: a discussion

bleh

 

Might also be mixed with this line of thinking:

https://twitter.com/user/status/1222062872785801216

Code:

https://twitter.com/Chrissox/status/1222062872785801216

 

It is rather late to be adding to the Robert Howard view of the autism trial but I missed this yesterday. What I found astonishing was the BBC report on the Today programme. It was quite obvious that questions needed o be raised about the trial methodology and in particular blinding. They seemed more interested in when the drug might be available. Dreadful reporting.

Perhaps the whole science budget has been blown on getting reporters to Antarctica to contribute to global warming.

 

To start with, why do unblinded trials make it into some Cochrane reviews, but not others? Why do some outcome measure comparisons make it into Cochrane reviews but not others?

 

https://twitter.com/user/status/1222476712203575297


https://twitter.com/user/status/1222476874267254784


eta: the discussion seems to have split:

https://twitter.com/user/status/1222478098320515072

 

Most claims of validity doesn't even measure that. Cronbach's alpha for example simply shows that there is a common bias in the answering of a set of questions. That bias doesn't automatically imply that all the questions are measuring the same underlying 'construct', yet this is frequently assumed by researchers.

 

I suspect Howard is well-intentioned but misguided. Subjective outcomes have their place I'm sure, especially in psychology, albeit backed by objective outcomes where blinding is impossible, as will often be the case for psychological interventions.

But what is meant by validation? Research outcomes validated for psychological conditions cannot be automatically considered validated for research outcomes of non-psychological conditions, especially where the research mistakenly attributes a condition to be psychological when it is not.

When you validate something you have to validate it against some valid datum - if the datum is itself invalid then you have validated nothing.

 

You would have thought so, but not in this field.

For a professor in the field to not understand this or deliberately obfuscate seems to me less being misguided and more incompetent.

I have a terrible suspicion that part of the descent of medicine into this sort of post truth situation is to do with an obsession with being inclusive. We are not allowed to discriminate - not even against poor quality research and incompetent teaching. Everyone is to be treated the same.

 

"Subjective outcomes are bad and should be avoided"

"Unless I like the (obviously flawed) subjectibe outcomes, then objective outcomes are bad and should be avoided"

We all know that this field has been avoiding objective outcomes precisely because it would reveal the scam. There are many that can be used. There are always objective outcomes detached from interpretation that can be used. They systematically avoid them, or even drop them when they plan and receive funding for it.

Because the reality is that not much of the research in this field will hold up precisely because of this. So pointing out this excuse actually makes the opposite case. It is a field that almost universally makes bad research precisely because it focuses strictly on finding positive evidence of things they want to be true and always avoids trying to prove themselves wrong.

We can see right through this. Treating us like we're a bunch of drooling idiots doesn't change that.

 

A friendly reminder of the results of the PACE trial. Those claims are the complete opposite of the PR campaign and claims by the researchers. That has always been the problem, PACE invalidated the model but the opposite was reported. This is fraud, malpractice, research misconduct and malfeasance all rolled up into one.

Objective outcomes show plainly it provides no useful benefits:

And the "benefits" are nothing but distorted analysis of inaccurate and unreliable questionnaires of little usefuless:

This plainly means null results. It could not be any clearer. Yet those null results have somehow been promoted as meaning the opposite. That is the problem. Well, that and all the cheating and lying.

 

The BMC correspondence with Sharpe et al perhaps offers some insight into how their outcome measures might be validated:

If the data matches pre-trial expectations then presumably they consider the outcome measure to be valid. If not, as with the definition of recovery, they “prefer” to change it to make it match their preconceptions. Presumably, in their minds, this then validates the new way of measuring the outcome. It is valid in the sense that it corroborates their preconceptions, which are treated as axiomatic.

That appears to be why actigraphy was dropped from PACE. The minutes state that another trial had shown that it is was “not useful.” (See Whatever happened to Actigraphy?) In other words, they did not believe that actigraphy would corroborate the subjective outcome measures and their preconceptions. Instead of questioning the validity of their preconceptions and the subjective outcome measures they chose to reject actigraphy.

This is perhaps a generous interpretation of what happened, as it implies that they are stupid and/or incompetent and/or deluded. An alternative explanation would be that they knew exactly what they were doing and acted dishonestly.

 

That makes sense to me. But also, another way to view this possibly is that people confuse rationalisation with logic or being rational. I think sometimes people who are rationalising something they want to believe are not aware that is what they are doing seeing it only as logical thinking.

 

Yes, I think this is important. One of the positives that I hope will come from PACE is that it may eventually help to improve standards in psychology and therapist based research.

Drifting back on topic, a similar thought occurs to me with the Moustgaard meta-epidemiological study. There appear to be two possibilities: either blinding is not nearly as important as other research suggests or there is something very badly wrong with the methodology of the meta-analysis. To be fair to the authors, they suggest this themselves in the paper:

If a result of this paper was to further understanding of the limitations of meta-analyses then it could be very useful. Perhaps I’m over optimistic but I’m hoping there will be enough scientists out there who care enough about standards in clinical trials to properly scrutinise this study and expose any flaws, which may also be applicable to other meta-analyses.

 

I think it is more like "religious truth", the 'truth' being ... whatever you want it to be.

 

Yes, I suspect people naturally have a poor ability to be consistent over repeated assessments on subjective measures. Some people will be better than others. Any one person will be better at it some times than at other times. Moreover, it is a cognitive endeavour being required of people whose symptoms typically include cognitive impairment.

Dealing with the relative aspect is likely to result in an even greater scattering of self reported readings, because the baseline reference become ever vaguer.

Puts me in mind of trying to measure something that keeps changing in size using an elastic ruler.