First analysis of ‘pre-registered’ studies shows sharp rise in null findings

[Andy]

Studies that fail to find a positive result are often filed away, never to see the light of day, which leads to a publication bias that compromises the credibility of scientific literature1.

An analysis now suggests that registering and peer-reviewing study protocols before research is conducted could improve this ‘file-drawer problem’, and help to correct the existing publication bias towards positive findings.

Researchers from Cardiff University, UK, report what they say is the first analysis of whether the practice is effective. They find that studies for which protocols were pre-registered are much more likely than the general scientific literature to report null findings2. The analysis was posted on 17 October to the PsyArXiv repository.

https://www.nature.com/articles/d41586-018-07118-1

 

[Snowdrop]

So just for memory challenged among us: PACE protocols were pre-registered but not peer-reviewed?

 

[Peter T]

So just for memory challenged among us: PACE protocols were pre-registered but not peer-reviewed?

Although PACE was pre-registered, they changed the design once the data was collected, so it was no longer as pre-specified, though the authors have claimed as they did not look at the data before the changes it still counts as a pre-specified study.

Being pre-registered does not guarantee there are not subsequent problems, but it does mean people know what studies were planned so that it can be identified how many studies are just shelved because of null results.

I seem to remember there was some element too of peer reviewing at the funding stage for PACE but may be getting confused. PACE was definitely peer reviewed by the Lancett prior to publication because the editor claimed that it was both fast tracked for publication because of its anticipated importance and simultaneously subjected to endless stages of peer reviewing, though it has not been revealed what that actually involved.

There are so many stages that supposedly ensure quality control, but they just seem to increase the opportunities to get it things wrong.

[edited for typos.]

 

[Woolie]

Its quite confusing.

Pre-registration means just that you made your protocol available publicly before you collected/analysed the data. You might have actually published the protocol in a journal, or you might have put it on your institutional website, or on one of those archive sites like PsycArXiv, or on ResearchGate or somewhere like that.

Registered reports are a deal some scientific journals now offer. It means you submit your paper, with a full introduction an methods section, but without the results. Then the journal accepts or declines it based on what you say you're going to do. Then when you have your results, you have a guaranteed outlet to publish your findings.

So PACE was pre-registered, but it wasn't submitted as a registered report.

 

[rvallee]

So just for memory challenged among us: PACE protocols were pre-registered but not peer-reviewed?

According to Sharpe (from a tweet) PACE was fast-tracked because it was pre-registered.

Nevermind they deviated from protocol and switched outcomes to such a degree that this makes it deliberate fraud but that would be an inconvenient fact to acknowledge.

 

[Woolie]

According to Sharpe (from a tweet) PACE was fast-tracked because it was pre-registered.

PACE was pre-registered - but in my view, pre-registration doesn't count for anything unless you stick to the protocol you pre-registered. Which they didn't.

And I don't think the fact of it had any effect on the Lancet. There's a lot of preregistered trials out there, why didn't they get fast-tracked at the Lancet? No. that's an error or a lie. Or both.

PACE was fast-tracked because everybody in Lancet was behind it and thought it was SO important it needed to be out soon. I would be willing to bet that not a single one of them had read the trial protocol, so they certainly weren't going by that.

 

[Esther12]

I'm sure those null results will start getting rare once people have realised they can get away with Crawley's trick of 'pre-registering' half way through your trial, like she did with the SMILE trial.

 

[rvallee]

PACE was pre-registered - but in my view, pre-registration doesn't count for anything unless you stick to the protocol you pre-registered. Which they didn't.

And I don't think the fact of it had any effect on the Lancet. There's a lot of preregistered trials out there, why didn't they get fast-tracked at the Lancet? No. that's an error or a lie. Or both.

PACE was fast-tracked because everybody in Lancet was behind it and thought it was SO important it needed to be out soon. I would be willing to bet that not a single one of them had read the trial protocol, so they certainly weren't going by that.

Can't remember who, I think it was an academic actually, but the reply to that tweet from Sharpe was precisely that they didn't follow the protocol anyway so how does pre-registration count here. No response, of course.

And now of course it's public record he will have a hard time to explain. If there's ever any accountability anyway.