COMPare Trials, Ben Goldacre et al

Lucibee said:
It's really good that this is being looked at - so I commend this study by Goldacre - but there is an issue that always gets overlooked with regard to CONSORT, which is that these are "reporting" guidelines, not "conduct" guidelines.

Of course it is important that trials are reported correctly, but if the next step is then not taken, which is to use that report to assess the quality of the trial, then it is useless. It is assumed by some that simply reporting a trial correctly means that you can assume that it has been done correctly. You can't.

And it's not just outcome reporting. It's participant selection, randomisation, adequate controls and/or placebos, adequate blinding, intervention methods, numerous other things before you get anywhere near the analysis on the pre-specified measures.

fwiw, COMPare would not have picked up on PACE, because they didn't switch primary outcome measures with secondary measures, which is what they were looking for here.
Click to expand...

Exactly!! They would probably not have picked up SMILE either, as the swap was "explained" by Crawley as to do with the feasibility and the burden on participants. Ben and co. would have given that the all clear according to the FAQs https://www.compare-trials.org/faq. I don;t think this is made explicit in the protocol, rather ironically https://www.compare-trials.org/wp-content/uploads/2016/01/COMPare-Protocol-18.5.2016.pdf. I might be wrong about that, but if it's in the FAQs, it's probably not clear enough in the protocol.

I am thinking of proposing a study called "Beyond COMPare" (!!) to look at the same studies for other characteristics of the trials which will indicate whether the trials were done correctly in terms of the factors @Lucibee mentions, rather than if they were reported correctly. As we all know, and as @Lucibee says, you cannot assume reporting a trial correctly (according to CONSORT) means it is a well designed and well conducted trial. But people do that. In the same way, people assume that a Cochrane review is well-designed and well-conducted because Cochrane reviewers have to follow the Cochrane reporting process.
 
Caroline Struthers said:
I am thinking of proposing a study called "Beyond COMPare" (!!) to look at the same studies for other characteristics of the trials which will indicate whether the trials were done correctly in terms of the factors @Lucibee mentions, rather than if they were reported correctly. As we all know, and as @Lucibee says, you cannot assume reporting a trial correctly (according to CONSORT) means it is a well designed and well conducted trial. But people do that. In the same way, people assume that a Cochrane review is well-designed and well-conducted because Cochrane reviewers have to follow the Cochrane reporting process.
Click to expand...

That sounds like an excellent idea!
 
Adrian said:
He seems to be looking for the easy popular targets along with bashing business (big phama although I'm not sure what views he would have one small phama). He ignores the fact that it is universities and academics that now have the worst record of publishing outcomes.
Click to expand...


Agree. That one of course seems remediable. I asked a UK academic friend of mine regarding research methods, reliability etc at her place (given there isn't a quality assessment, certainly outside REF times of year) and she replied that it is a worry but her place did have a process of internal peer review with 2 people reading the work etc.

I think that we can all see the potential for flaws. Given fixed term contracts and way academic careers work simply making this external wouldn't fix it. Competition is something that works incredibly effectively in academia though (despite many saying they hate it that is what drives the place)

Switch to having them reviewed externally, blindly (as students' papers are) with a league table for methodological requirements. Have other depts cross-review different subjects to avoid individual areas getting 'caught in a circle'.

Prizes that are based on work that has good strong design (internal prizes as well as external ones as you have book prizes and other types). Given the 'voices from the shadows' videos would Wellcome be a good place to begin discussions? Maybe Microsoft/the Gates foundation (whichever arm is relevant)? Make good design and proper methodology ie research quality the 'next big thing'

A badge worth having (or prize), particularly if it targets mid-career and early-career academics could nudge quite a lot if the format and types of prize (ie include school-level where portfolios representative of overall quality be submitted etc)

e.g. Require a summary section next to the abstract succinctly diagramming and defining the design chosen and methodology based on the objectives. Require results tables of primary. ie layout that makes it obvious if something is 'of quality' or pre-that time'. Other bug-bears made disqualifying etc.
 
Last edited:
You must log in or register to reply here.
Back
Top Bottom