When beauty is but skin deep: dealing with problematic studies in systematic reviews 2021 - Cochrane

• Stephanie L Boughton
• Jack Wilkinson
• Lisa Bero

Authors' declarations of interest
Version published: 03 June 2021

https://doi.org/10.1002/14651858.ED000152

Systematic reviews provide high‐quality assessment of the evidence for healthcare interventions, but this assessment is predicated on the trustworthiness of the studies they include. The presence of untrustworthy, or ‘problematic’, studies in the literature poses challenges for systematic reviewers and is a threat to the reliability of systematic reviews. This is compounded by the absence of an agreed definition of what constitutes a problematic study and the lack of a validated method to identify them.

The most visible problematic studies are retracted studies. The number of retractions has risen rapidly,[1, 2, 3] suggesting a rise in problematic studies in the literature or an increase in awareness of them. Articles may be retracted for a number of different reasons, including error, naïve mistakes, or research misconduct,[4] and estimates vary as to the percentage of retractions that are due to misconduct versus honest error. One study found 67.4% of retractions were due to misconduct (comprised of fraud, suspected fraud, duplicate publication, and plagiarism).[5]

The MECIR standards (Methodological Expectations of Cochrane Intervention Reviews; community.cochrane.org/mecir-manual) are clear that retracted articles should not be included in systematic reviews, but retracted studies are likely to represent only the tip of the iceberg. A systematic review found that 2% of scientists admitted having fabricated, falsified, or modified data or results, while 14% reported having witnessed a colleague doing so,[6] suggesting the scale of problematic studies is far greater than the number of articles retracted. In an assessment by John Carlisle of 526 randomized controlled trials submitted to the journal Anaesthesia between February 2017 and March 2020, he believed 8% to be fatally flawed due to containing false data.[7] For those where he had access to individual patient data, this rose to 44% containing false data, and 26% being classified as critically flawed. It is unclear how many problematic studies go undetected during typical journal editorial processes, which do not usually include the sort of forensic investigation undertaken by Carlisle.

Even studies that are eventually retracted have downstream effects on the healthcare literature, including on systematic reviews, due to the time lag between their initial publication and subsequent retraction. A study of all retractions indexed in PubMed up to May 2012 found that the average time‐to‐retraction was 32.91 months, with a range of less than 1 month to 304 months.[8] Another study of retracted obstetrics and gynaecology articles up to June 2018 found a median time to retraction of two years, with one study being retracted 18 years after publication.[3] During this time there may be no indication to readers or those conducting a systematic review for which it is eligible for inclusion that there are any concerns about the study, though use of Expressions of Concern to alert readers to potential issues is increasing.[9]

Cochrane has developed a new policy to address potentially problematic studies in the context of systematic reviews. The Cochrane policy for managing potentially problematic studies and accompanying implementation guidance set out how Cochrane Review authors and Cochrane Review Groups (CRGs) should manage situations involving potentially untrustworthy, or ‘problematic’ studies in the context of Cochrane Reviews. It provides steps to follow when an included study is retracted or has a formal Expression of Concern published, and also how to deal with scenarios where there is no formal post‐publication amendment but review authors or the CRG have concerns about the trustworthiness of the data in an included study, or study eligible for inclusion in a review.

While this new policy provides guidance for review authors and editors where they have identified serious concerns about a study, there is an urgent need for a validated method to identify problematic studies reliably and fairly. Existing tools for assessing included studies, such as tools for assessing risk of bias, assume that the data are real. A variety of methods to identify problematic studies have been proposed. One example is the REAPPRAISED checklist for evaluation of publication integrity.[10] Suggested methods range from testing for statistical anomalies, searching for inconsistencies in how data are reported (for example different sample sizes for the same study reported in a publication, or across several publications), searching for awkward or repetitive phrasing, analyzing images, and scrutinizing all publications from authors who have previously published retracted or problematic studies. However, many of these methods involve subjective evaluation of data, and none have been fully evaluated. Use of unvalidated methods risks over‐ or under‐detection of problematic studies, and caution is needed as misclassification of a genuine study as problematic could result in erroneous review conclusions. Misclassification could also lead to reputational damage to authors, legal consequences, and ethical issues associated with participants having taken part in research, only for it to be discounted.

The scope of problematic studies encompasses a wide range of different issues, varying in scale and cause. They may involve individual articles, groups of articles by a particular author or group of authors, or systematic manipulation of the publication process where individuals or groups use fraudulent or dishonest practices to inappropriately influence the publication process,[11] for example so called ‘paper mills’, where authors allegedly buy services such as ghostwritten fraudulent or fabricated manuscripts produced by a third party.[12] Consensus is greatly needed on terminology and what constitutes a ‘problematic’ study. For the purposes of the Cochrane policy, we have defined a problematic study as “any published or unpublished study where there are serious questions about the trustworthiness of the data or findings, regardless of whether the study has been formally retracted”, but we know that such terms mean different things to different individuals, denoting a greater or lesser degrees of ‘seriousness’. As well as having different thresholds for what they consider concerning, individuals will also have different ‘red flags’ for what they believe is indicative of untrustworthiness.

The challenge ahead is to establish consensus around reliable, feasible approaches for identifying whether a study is problematic in the first place. As a starting point we need a standard, agreed definition of a ‘problematic’ study. Research is then needed to develop and validate methods to identify such problematic studies, in order to preserve the status of systematic reviews as a source of definitive health information.

see
Research Integrity

Research Integrity | Cochrane Community

 

Pretty amazing how just like their whole boasting about engaging with consumers, all of this is pure fantasy that they don't even bother applying to themselves. They seem to have created the ultimate paper tiger, and it's completely hollow.

We can even apply the same problems with an editorial written by Tovey that was posted here recently, which lists all the potential problems with systematic reviews, for which both the CBT and the GET reviews check all the marks, and he still approved the 2019 fake update that still checked all the marks.

I'm still baffled as to why this organization is considered reputable, let alone the best in the business. Then again, them being the best in the business only makes it all that clearer just how intellectually and ethically bankrupt the business of so-called evidence-based medicine is.

 

When I said that in a comment before, I am sure I was told that Tovey was the one who backed down from withdrawing Larun et al and allowed them to start on a redraft.

 

Thank you. I don’t always remember details correctly.

 

He was in charge around the time when Robert Courtney forced the issue, and initiated the update process, but that got cancelled by the CEO change. However given that it takes time to process things into an update being published, he must have approved its initiation. At least I doubt that review authors just spring out randomly with an update for review by Cochrane editors. It's also unlikely that he wouldn't have published it like Soares-Weisser did, so far these things only ever lean one way.

The odd person may do some nice talk, but they almost never get anything done, presumably because there is too much pushback. Basically almost everyone in this profession is a pure cynic who knows the system is gamed but plays along because it's impossible to fix. Sure, they mouth a few platitudes here and there and write articles like this, but they just slug along mending a broken machine outside of those brief bursts of virtue signaling, hopium or even pure hypocrisy.

Clearly there are people who see the issues but they are badly outnumbered and can't force the talking to become walking, even when in a position to do so. Tovey wrote this stuff after Cochrane published the ridiculous update, so even if he is a lone believer, he should clearly see how Cochrane, the standard in the industry, doesn't bother with any of this, and it doesn't bother him one bit. If it does but he doesn't say it, then it doesn't really bother him. It's really something to watch. It really sucks to suffer the outcomes, though.

And for sure Cochrane folks will keep going around talking about consumer engagement and rigorous evidence that is regularly updated and so on. People will applaud and laugh and so on, all knowing that almost everything they say there is bullshit, but playing along is part of the job. Absurd.

 

He did back down and allowed them to continue with a redraft, but he rejected the redrafted redraft in May 2019 days before he retired. Larun's boss Atle Fretheim aggressively pushed back against the rejection and David retired leaving Karla to handle it. A colleague of Fretheim (Andy Oxman, also a close colleague of Iain Chalmers) suggested they appoint an independent arbiter (Gordon Guyatt) to oversee the further changes needed to the text of the amended review to get it over the line. He said it was OK as long as they made it clear in the abstract and/or discussion that the effect size on fatigue was not clinically significant. They didn't do this obviously interpreting his clear instruction (made twice) as advisory only. Instead they removed any text indicating the size of the effect. Karla accepted that fudge and published the review in October.
https://acrobat.adobe.com/id/urn:aaid:sc:EU:e3be684a-95cd-439a-b046-38931dcc52a3