Reimagining the peer-review system for translational health science journals, 2021, Smith

Abstract
Retractions of coronavirus disease 2019 (COVID-19) papers in high impact journals, such as The Lancet and the New England Journal of Medicine, have been panned as major scientific fraud in public media. The initial reaction to this news was to seek out scapegoats and blame individual authors, peer-reviewers, editors, and journals for wrong doing. This paper suggests that scapegoating a few individuals for faulty science is a myopic approach to the more profound problem with peer-review. Peer-review in its current limited form cannot be expected to adequately address the scope and complexity of large interdisciplinary science research collaboration, which is central in translational research. In addition, empirical studies on the effectiveness of traditional peer-review reveal its very real potential for bias and groupthink; as such, expectations regarding the capacity and effectiveness of the current peer review process are unrealistic. This paper proposes a new vision of peer-review in translational science that, on the one hand, would allow for early release of a manuscript to ensure expediency, whereas also creating a forum or a collective of various experts to actively comment, scrutinize, and even build on the research under review. The aim would be to not only generate open discussion and oversight respecting the quality and limitations of the research, but also to assess the extent and the means for that knowledge to translate into social benefit.

Open access, https://ascpt.onlinelibrary.wiley.com/doi/10.1111/cts.13050

 

Would be a huge improvement from the current peer-review system.

At least you can avoid papers not being reviewed at all, as is what happened with the Music Therapy & CBT paper from Norway, where one reviewer "only read the abstract".

But also, to avoid groupthink and bias, which might prevent so much funding going to those biased groups, without their research being scrutinized in advance.

Sounds promising!

 

It's weird to frame prominent retractions as a failure when it's actually the system working and it's flawed or downright fraudulent papers not being retracted, or being published in the first place, that is the problem.

If anything, unless the system is improved to not allow so many junk papers being published (hell or junk research being funded at all, retractions should be so common people barely think about them twice because it's the system working itself out. Don't want papers retracted? Do your damn job, then. It's really not hard. Yes, it means way more disappointing null results but THAT'S LITERALLY HOW SCIENCE WORKS!

Because the problem isn't replication, it's validity, invalid papers, invalid research, being published and promoted and misrepresented in ways that are excessive. So either publishing gets a lot more scrutiny, or retractions become super common, a sort of immune system of science that must seek and destroy all the bad research polluting everything.

But regardless of how things could change, in the end it's about ending arbitrariness, exemptions from the rules that make the rules essentially a formality, or at best an honor system. The very same flaws that sink "research" like PACE are readily pointed out as fatal in other research, by the very same people who dismiss the flaws they arbitrarily are concerned about or not, depending entirely on their personal interests. Rules are just words that people choose to interpret in whatever ways fit their interests.

Especially with EBM, it's pretty clear that science is now working into problems that are too complex, too interpretable, without this simple binary thing that everyone must agree settles the matter, the smoking gun that silences all critics has stopped firing a long time ago. Evidence does not matter anymore, only perception and popularity. It's really gotten that bad, not because science got worse, but simply because the easy problems are over, what's left is too difficult for such a kludge of a system to work, especially one where politics and interests can completely overrule reality.