Risk of bias assessments for selective reporting were inadequate in the majority of Cochrane reviews, 2019, Saric et al

Abstract

Objectives
The aim of the study was to analyze adequacy of risk of bias (RoB) judgments for selective reporting in Cochrane systematic reviews.

Study Design and Setting
We extracted RoB assessments, including judgment (low, high, or unclear risk) and supporting comment from Cochrane reviews of randomized controlled trials using computer parser. We analyzed sources of information mentioned in supporting comments. We compared judgments of Cochrane authors with guidance from the Cochrane Handbook for Systematic Reviews of Interventions (Cochrane Handbook) and categorized them into adequate or inadequate.

Results
At least 60% of judgments for risk of selective reporting bias of trials in analyzed Cochrane reviews were not in line with the Cochrane Handbook. Few Cochrane authors mentioned the trial protocol as a source of data for assessing selective reporting. Most of the inadequate judgments were made among trials that were judged with low risk of selective reporting bias; more than 90%. In 9% of analyzed RoB tables, Cochrane authors did not use this RoB domain at all.

Conclusion
Cochrane authors frequently make RoB judgments about selective reporting that are not in line with Cochrane Handbook and not mentioning trial protocol. Interventions aimed at helping Cochrane authors to make adequate RoB assessments in Cochrane reviews would be beneficial.

Paywall, https://www.jclinepi.com/article/S0895-4356(18)31135-1/fulltext

 

This is from three years ago...!

 

Who will police Cochrane the police Cochrane?

Since this was 3 years ago, it looks like it's their Neptunian division, which is only available on the 5th Tuesday of odd bissextile years during the month of Smarch. Also all mail is immediately forwarded to the core of the planet.

 

Only glanced at this but perhaps the the problem is that the Cochrane authors don't care enough to properly assess RoB - or maybe they have a conflict of interest. Either way it questions NICEs collaboration with Cochrane --- might check but I can't recall that the risk of bias was clearly identified in the business case, re NICE/Cochrane collaboration.

Also, if the Cochrane reviews can't be relied on then, as per rvallee's comment, NICE ---

So much the the "savings" identified in the NICE business case.

Then there's the "living guideline" goal, if NICE have to review the Cochrane review then will (reliable) reviews be available any sooner?