Checklist to assess Trustworthiness in RAndomised Controlled Trials (TRACT checklist): concept proposal and pilot, 2023, Mol et al

It seems a farce.

A bit like suggested criteria for bad music (some applied to Mozart and Wagner)

'Too many notes'
'Fails to use sonata form'
'Too small an orchestra'
'Boring old stuff'
'Keeps playing the same tune'

It is a classic example of the floating expert phenomenon.
How do you generate the most expert advice? Ask in some experts. How do these experts know they are experts? Ask some more experts.

What is forgotten is that real experts cannot be bothered with this sort of drivel. 'Experts' are always by definition not the experts but people who would like to be.


And what the heck are 'stakeholders' doing in there? Surely the whole point is to avoid stakeholders and stick to rational argument.

 

This explains so much..

 

Well you *****y well shouldn't @Hutan!

1. If someone does not have a full grasp of problem issues with trials and how to apply them consistently they should not be an editor.

This isn't difficult. All clinicians get a training in all the things we discuss here. The big problem is that many of them are not intelligent enough to fully understand why these problems have the impact they do. An even bigger problem is that for 90% of clinicians vested interests or conveniences or back scratching patterns mean they conveniently forget not only everything they were taught but all the checklists you could possibly throw at them! We have seen this on the forum to an extent that even surprised me.

2. Even if you did think a checklist was useful it needs to bear some relation to reality. Where is the reference to subjective outcome bias, blinding and so on. I think it may be very instructive that these are yet again omitted.

Why should the number of authors matter? My initial study of rituximab in RA had two authors, only five patients, no controls, and was open label. It was turned down and even publicly ridiculed by the editors of the two major relevant journals. But I persuaded the second one to take it. In the meantime, the drug company scientists could see barn door that the data had to be significant because they used objective measures back up by pharmacodynamic profiles that could not conceivably have any other explanation. To see why that matters you just need common sense, not checklists, which in my case would have come out no, no, no ,no. And PACE sails through, not because the Lancet did not know perfectly well what the problems were, nor even the kindly Professor W who no doubt had a quick shoofly at it. They were stakeholders so who cares?

I think things may be a lot worse than you imagine. This proposal is eerily like the RoB tools from Jonathan Stern and co. It conveniently misses out the really bad crimes. It is relatively amateur in comparison to RoB but the errors in RoB were egregious enough. It comes from a group of gynaecologists who seem to be focused on an epidemiology and statistics unit in Amsterdam.

Why would it even occur to someone to suggest that a big study needed lots of authors? Easy. So that the staff of the Clinical Investigation Unit and all the statisticians got their names on the paper too. We saw this with RoB2. It is a deliberate exercise in brand whitewashing by professional trial paper churners-out I suspect. It even has the Cochrane stamp of approval!!

 

Yes this is exactly the sort of pseudoarithmetic nonsense we have seen with GRADE and RoB from statisticians. It has nothing whatever to do with reality.

 

The real cause of all dodgy trials is conflict of interest . Which never features in lists of factors affecting reliability of trials - the elephant in the room. Also, editors (who seem to make it their business not to understand science) have way too much power, and can never be challenged. Ridiculous.

 

Or if they are challenged, they don't have to respond

 

Indeed and if I am not mistaken these people are only too keen to release more flawed trials - except flawed in ways not on their list.

There is a word 'disingenuous'.
(As in 'as disingenuous as the Cochrane founders'.)

 

Most of this is up to individual judgment and open to interpretation. Which isn't necessarily bad in itself, but given the history of evidence-based medicine, where people who do trials usually grade trials similar to their own, this is the same old problem. None of this appears close to reduce bias, which is the dominant issue. Also I'm not sure how this applies to "RCTs" that aren't even properly controlled. Technically it shouldn't, but we all know that not being properly controlled is not an issue for most of evidence-evidence medicine, even though it's in the name, because it can conveniently simply mean 'clinical', without even downgrading its value.

Who decides what's plausible? Psychosomatic ideology isn't plausible in itself, and yet it's almost universally believed to a high degree, capable of fully explaining any and all symptoms of unclear origin. Who decides what's an abnormally large effect? In a field where abnormal effects are the most common, since they're not based on plausible mechanisms and don't work? Where trying the same thing again and again until they get results they like is an accepted way of doing things?

There is mention of pre-registration, but PACE was pre-registered, they just deviated from it in a specific way to turn a null into a fake abnormally large positive, and yet you point this out and are met with general indifference because reasons. Ah, well, it does mention changes in outcomes, but PACE is considered "gold standard", so clearly hardly anyone really cares about that. Just mumble a few words and ignore the issues raised and as long as it's popular it's all good. And there is nothing about bias, such as open interventions that explicitly seek to alter participants' response from people with a high degree of bias and conflict, even up to literally telling participants that the treatment they are 'trying' is safe and effective.

It also mentions short time between registration and publication, but nothing about abnormally long times. We've seen this recently with... one of the Crawley trials, I think, that took 8 years or so to publish their results because they were bad.

None of this still amounts to independent review either. And it's not accountable in any way. And still way too chummy, when we take examples like Cochrane where authors faced with a review having been found to be inadequate can simply not care and keep it published.

Obviously a good checklist would be useful here, but technically this already exists in the form of existing standards. The usual problem is that they're simply ignored whenever convenient, so this is just yet another standard where items can be conveniently ignored as long as the is a cultural demand for something, or political incentives, or literally any reason to exempt anything falling short of the standard.

 

I think it's too little, too late. It shouldn't be too much longer before AIs take over the task of checking studies for flaws, and doing a better job of it. The critical problem is making sure that the bad studies aren't being falsely claimed as good during the training period.

I propose that when an AI gets trained for this, it should be given access to S4ME and other "critical about bad studies" forums, to give it a wider perspective.

 

This cartoon is so good. I moved from Cochrane to what I thought was going to be a better life promoting the use of reporting guidelines to help improve the standard of reporting of medical research, trials etc. I can tell you that the enthusiasm for academics to develop and publish new and overlapping/contradictory reporting "standards" (aka guidelines) which nobody follows (journal editors publish papers whether they're well reported or not because $$$) is just as large as it is for academics to develop endless checklists to help them work out whether research, whether well reported or badly reported, is "trustworthy". Nobody does anything about trying to ensure the research is necessary and well conducted in the first place as I guess there's no $$$ in that. As the late Doug Altman said 30 years ago, we need less research, better research, and research done for the right reasons. [https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2539276/pdf/bmj00425-0005.pdf]