The Rise and Fall of Peer Review, 2022, Mastroianni

Absolutely, @Jonathan Edwards, I should have specified "among other things". I was only talking about the statistical aspect but in my mind I did not exclude other weaknesses... including the one you mentioned in the thread of this study. Moreover, I am in no way in a position to contradict the expert that you are in this field.

 

This. Risks leading to group think as well as to a culture of you rub my back I rub yours. Both seem highly prevalent in ME/FND/MUS BPS circles, circle being the operative word here.

I'd go a step further. Instead of [ETA: or as well as] peer review or even preprint review I'd like to see public preview of protocols and methods. Preferably pre any allocation of funding.

Obviously this wouldn't be workable for purely commercial research funded entirely by industry. But it should be a requirement for anything that uses any sort of public funding, however indirect. Research funded by patient donations would also benefit from undergoing a similar, more rigorous and public preview of protocols and methods.

 

I know a researcher, who works on Alzheimer's disease, and for much of his career (he's close to 60 now) he struggled to get funding.

Jonathan highlighted that his colleagues couldn't make progress in ME/CFS - they tried a few things and, when those didn't work, then they gave up. I think there are better (and improving) tools like GWAS - so there are now opportunities to understand diseases where there hasn't been any progress - like ME/CFS. We need Government/public funding for good research.

Reminds me of @Snow Leopard point ---they keep looking in the places they looked before and found nothing! They even seem to apply the same techniques to new diseases like long covid --- Jonathan's comment comes to mind "The vultures are circling - and babbling as they fly." [https://www.s4me.info/threads/news-...e-cfs-2020-to-2021.14074/page-151#post-314441]

 

Yea for some researchers it's about including objective outcome indicators like Actimetry (FitBit) since you cannot blind adequately. Failure to objectively measure (post intervention) outcomes seems to be de rigueur for some scientists and medical professionals --- makes sure they can prove their pet intervention/income stream works I guess!

 

Heh. As opposed to...?

In our own corner of the universe, things are even worse since the same thing is happening except it may as well literally be the same 5 papers published over and over again, with zero quality control, and clearly little discoverability as evidenced by recent "systematic reviews" that didn't find over 90% of the papers published on the same topic.

The criticism usually points to the best research out there as evidence that the system still works, but the reality is so much worse. In psychology, the poverty of peer review is essentially abused all the time, especially the massive HYPING of every small result from tiny studies that never replicate anything.

Right now we pretty much have the flaws of a giant self-serving bureaucratic mess, but without the benefits it can bring: no coordination or leadership, excessive redundancy and nothing is leveraged, no economies of scale, no sharing or pooling of resources while nothing builds on itself, everyone doing their small thing in their small corner, unable to bring the scale of work that other scientific disciplines like physics have succeeded at. It's eternal starting from scratch about a tradition that is nearly a religion, always promising, never delivering yet somehow seems to have always been there.

The author got the point right: this is about monoculture, about the lack of differing viewpoints. I can't say for other disciplines, but medical research is crippled by this, by groupthink and cultural enforcement of what's acceptable and what's forbidden. It's even worse than that when you consider the example of BPS ideologues who claim, and likely believe, that their excessively generic and coercive groupthink is the real courageous thinking, they literally boast about being fringe thinkers as they enforce their monoculture in secret behind closed doors.

The monoculture of medicine is probably its biggest flaw, it even prevents medicine from seeing its flaws, forcing the monoculture to remain the only acceptable way to think. Frankly it's borderline church-like behavior in terms of how conformist every idea is and how enforcement is cultural and political, never on merit, never on the actual evidence since medicine doesn't have math to settle debates, only a popularity contest where only the monoculture is allowed. Shockingly, one of the least curious group of people I've ever seen.

 

See also The Preprint Club - A cross-institutional, community-based approach to peer reviewing (2023)

 

From @MSEsperanza:

(Not a reply on Mastroianni but thought it could serve as a comment. Sorry it's pessimistic.)

James Heathers: The Right to Be Wrong Isn’t The Freedom From Consequences

And From Where Consequences Might Arise

https://jamesclaims.substack.com/p/the-right-to-be-wrong-isnt-the-freedom

Also from James Heathers, posted on Mastodon:

https://techhub.social/@jamesheathers/111133279337316953