When is lack of scientific integrity a reason for retracting a paper? A case study.(2020) Fiedorowicz et al. (about homeopathy for CFS)

When is lack of scientific integrity a reason for retracting a paper? A case study.
Abstract:

This editorial has just come out in the Journal of Psychosomatic Research. It is a discussion of issues that arose from this 2004 publication:

This study is a triple-blinded randomised controlled trial of homeopathy for CFS. It might not come as a surprise to you that the primary outcomes of the study were all negative.

The current discussion is related to one of the co-authors, who stood up in a talk recently and said that she had worked out a cunning plan for discovering whether participants were in the homeopathy or the control group.

The current editorial explains why the journal did NOT retract the paper (because the results were all negative anyway, amongst other reasons).

The interesting bit is the issues they chose to comment on (emphasis mine):

None of us here will miss the irony that these requirements for good studies are simply waived for psychotherapy trials, without anyone even caring about the risk of bias. Silly homeopathy folks - don't they know that all they need to do to get around this is to combine their medicine with a wee chat. Hey presto - blinding problems solved!

 

The correct answer is 'always'.

IMO.

 

But perhaps not if the lack of scientific integrity consisted of pretending to have lack of scientific integrity in order to have one's own paper retracted because the result caused a problem for one's practice income stream?

 

Thanks for flagging this @Woolie. This is a nice ironic illustration of the core problem the GRADE enthusiasts and Cochrane reviewers seem to find hard to understand. Perhaps I could have cited this and Hrobjartsson in my testimony to NICE.

The citation of Hrobjartsson is interesting, though, for the way it gives the impression that you can readily put numbers on the problem. The first thing that is puzzling is that Hrob. measures the problem in terms of odds ratios. Looking at the paper this is because they decided to consider results in terms of binary good or bad outcomes. Clearly that would not apply to a result based on a comparison using a graded scale. The is no odds ratio for PACE as far asI can work out (unless you use 'recovery'). So the figure of 36% does not tell us much, if anything.

I rather suspect this is statisticians extracting the wrong number. What we want to know is the chance that using unblinded assessors leads to a clinically significant difference in outcomes. And of course that cannot be assessed from sampling the very few trials available that used both blinded and unblinded assessors. These turn out to be trials of rather esoteric things like retinopathy. There would be no way to extrapolate them to ME/CFS or anything else much.

So maybe all Hrobjartsson's studies (there are several) can tell us is that yes there is a problem. Which is fine but we knew that and I think there is also a serious question about whether the studies might have a serious bias of their own.

Hrob's studies have the very unusual feature of using both blinded and unblinded assessors and making use of this to assess the importance of blinding. This means that the trials were deliberately designed in a frame of mind that took blinding seriously and wanted to analyse its effects. And arguably the whole problem about trials using unblinded assessors and subjective outcomes is that they are set up by people who are not in this frame of mind but rather prefer to sweep the problem of blinding under the carpet. Being an unblinded assessor in a trial assessing the importance of blinding is going to be very different from being an unblinded assessor in a trial that ignores it.

 

This is similar to how homeopathy is often delivered. Homeopaths believe that they can figure out the right homeopathic remedy by learning a lot of about the patient. They are an attentive listener who will take time for the patient. This is probably an important reason why homeopathy has survived.

It's not that different from psychosomatic medicine therapists searching for ways to connect the patient's medically unexplained symptoms to some psychosocial factors.

 

It may not be that different from the Hawthorne effect https://en.wikipedia.org/wiki/Hawthorne_effect indeed @Jonathan Edwards advice that he showed empathy to his patients and tried to support them --- OK Jonathan was developing treatments that worked rather than just being interested!

 

Actually the PACE trial's pre-registered primary outcomes were binary (whether fatigue was substantially reduced at trial end point, and also whether physical function substantially improved).

You're right that although ORs are used as estimates of effect sizes in studies using binary outcomes, they can't be directly compared to the kinds of effect size measures we use with continuous outcomes (like Cohen's d). I think the main point, though. about the Hrobartsson et al finding is that it suggests that outcomes can be very heavily inflated by bias resulting from the combination of non-blinding and subjective outcome measures - that is, not just by a bit, but by a lot.

 

No, I think they dug through literature for trials that ran in two phases - blinded and nonblinded.

 

That puzzles me because unblinded and blinded stages in drug development are widespread. If these are the very few that have unblinded and blinded phases within the same trial that still suggests to me that these people were taking blinding rather seriously and that people involved in the trial would be aware that they were in an unblinded or blinded phase. I can see that the original triallists may not have wanted to make a formal estimate of effect on odds ratio themselves but they must have taken blinding seriously to think it was worth repeating the same study with blinding after doing it unblinded (presumably to see if it had an effect)? That would be very different from the cases we are interested in where people firmly took the view that it was not important based on a textbook chapter that conveniently avoided spelling things out clearly! It still seems to me to be a very unrepresentative sample - but my concern about bias here is that Hrob is likely to have significantly underestimated the problem.

I agree that the value of the paper is in showing significant effects from unblinding. My scepticism is about trying to pull out numbers that then get quoted as indicators (largely because I suspect that the reality is much worse still).

 

I can only access the abstract--Elsevier and UC are in a dispute.

 

The statement in the editorial about caution regarding subjective outcomes considering that Per Fink, Rosmalen, Sharpe, White etc are on the journal's advisory board. They don't seem to agree with the journal's beliefs about the bias inherent in combining subjective outcomes and non-blindedness.

 

link:
https://www.sciencedirect.com/science/article/abs/pii/S002239992100057X
(not on sci-hub yet)

 

The link is broken because the "]or" has been inadvertently made part of the URL it is trying to go to. Works fine with that removed .

 

What a pity that it´s ony a case report!

 

Trial by Error by David Tuller: Psychosomatics Journal Linked to PACE Authors Highlights Bias from Subjective Outcome

 

@dave30th, its my impression that Jess Fiedorowicz is very much trying to clean up the dodgy science in this area and improve the quality of the research. I can't talk specifics in public, but he has been actively working to ensure BPS style papers are receiving rigorous reviews before publication. He's selecting reviewers from a range of perspectives (including those who've written against BPS-style approaches), and really listening to what they say.

 

That would be good. Perhaps he could clean up his advisory board at the same time!

 

Maybe that's not within his power?

 

I don't know. But I think I will send him a link to the blog post.