Outcome Reporting bias in Exercise Oncology trials (OREO): a cross-sectional study, 2021, Singh, Twomey et al

Jonathan Edwards said:
It compares in a way to another sort of analysis - looking up all the other papers written by the authors of a study. You never hear about this I peer review and I am sure it does not feature in the GRADE system but statistically it must be entirely valid. If author TC has, in addition to the paper under scrutiny, written a whole lot of really awful papers showing a complete lack of understanding of bias then is that not significant?
Click to expand...
Yes, regardless of the statistics, it's an approach that should be used a lot more when choosing which research proposals to allocate funds to.

I recall one proposal to trial a mixed supplement for ME/CFS. Some of the evaluators thought that it would be a nice project as it might produce a treatment - and it would be blinded, so surely there would be no bias.

The researchers had done a previous study on the same supplement mix for a different disease. There was a peer review that could be found online asking them to report that there was a null primary outcome in the abstract - they had neglected to do that in the pre-publication version. In the final version the null result was mentioned in passing, along with the widest array of excuses I had ever seen in a report; proposals for tweaking the mixture, suggestions that the length of study was not long enough. They even blamed their trial participants, suggesting that only the most treatment resistant patients turned up for trials and that substance abuse by the participants (remember this was a blinded, controlled study) had skewed results.

The proposed trial had the same supplement mix, and the same trial length, just a different disease, and a similar hand wavey 'inflammation' sort of explanation. It was clear that even a null result in the proposed trial would not be assumed to be evidence against a particular component of the mix, or a different dosage or recipe, and there was a significant risk that even a null result would be spun as a success.

It's easy to think of some pretty egregious research where the researchers just keep getting funded to produce more, often quite harmful and certainly wasteful, rubbish.

If people tasked with the job of allocating funds to research took their job seriously (and most often I think these roles are done voluntarily, squeezed in between other 'more important' things - perhaps they should be paid to devote time to it), then they would research track records. They would get opinions from knowledgeable patient organisations. And then they would not give more funds to researchers who switch outcomes without declaring it, or don't report results honestly. If that happened more, then researchers would either adapt, or go and do something else. There needs to be consequences.
 
I think one could do the same with p-values: normally those just below 0.05 shouldn't be that common as seen in most papers, even if there was an effect.

I doubt, however, that one can differentiate between data manipulation and reporting bias (or things like optional stopping).
 
The paper said:
When researchers do not provide such acknowledgements, one cannot rule out the possibility that the researchers’ preference towards a certain (often favourable) finding motivated the decision to omit outcomes and not declare discrepancies between the preregistration and the published article.
Click to expand...
And sometimes, just sometimes, the authors will literally plainly admit, using those very words, that they cheated to get the outcome they prefer and no one will do a damn thing about it. And frankly I think that's the biggest problem, that the self-correction has been effectively broken entirely where interests are opposed to reality. It's one thing to be able to push through something that shouldn't, but being caught and getting away with it is the main problem, the facts aren't even open to debate and yet they actually are subject to debate, a debate in which the main stakeholders are explicitly maligned and shut off.

A big problem I'm seeing, in medicine in general but especially for EBM, is that stakeholders are often not involved, only some and too often biased ones. That can't work, it has to be a rule, an immutable standard, that stakeholders be involved according to their, well, stake, and its relative importance. And of course for patients it is literally our lives, while for the people pushing their nonsense it's all about career advancement and what can only be described as a jobs program for people who should never have been granted a medical license, they have no real stake in the matter and yet control the entire process. They don't even care about the outcomes, and that makes them systematically reckless.
 
Michiel Tack said:
I think one could do the same with p-values: normally those just below 0.05 shouldn't be that common as seen in most papers, even if there was an effect.
Click to expand...

My understanding is that the inverted funnel plot more or less does this but actually is more powerful because it makes use of theory of variance.

Michiel Tack said:
I doubt, however, that one can differentiate between data manipulation and reporting bias (or things like optional stopping).
Click to expand...

The amusing thing about the telepathy studies was that there were fewer trials showing an obvious positive response than there should have been by chance. All the results came out exactly as only slightly positive as the study being replicated. I think you can probably show that this has to involve manipulation some sort.
 
You must log in or register to reply here.
Back
Top Bottom