Begone Good Faith:Editorial Review Isn’t Working; Fixing It Means Disposing With the Secrecy

Blog article: Primary Endpoints

Begone Good Faith: Editorial Review Isn’t Working; Fixing It Means Disposing With the Secrecy

Alexander Trevelyan

It must’ve been late 2013—my labmates and I, along with our advisor, were debating the upshot of an article in Science called Who's Afraid of Peer Review?

If you aren’t familiar with the article and don’t have the energy to read it, I’ll briefly sum it up for you: John Bohannon, the article’s author, had submitted an egregiously fake manuscript to 304 journals in an attempt to probe the state of peer review in open-access publishing.

What Bohannon found—surprise, surprise—was that most of the journals had no appreciable signs of peer review (which could actually be good or bad, depending on whether the editor simply rejected the manuscript outright) and that even among the journals that did undertake peer review, the paper was still accepted for publication an overwhelming portion of the time.

All accounted for, “only 36 of the 304 submissions generated review comments recognizing any of the paper's scientific problems. And 16 of those papers were accepted by the editors despite the damning reviews.” Bohannon later repeated a variation of the experiment targeting media outlets, coaxing them to report on his dubious study claiming that chocolate aids weight loss.

Full article: https://ajtrev.substack.com/p/begone-good-faith (Long !)

 

So far every critical piece of peer review and academia has avoided the worst offenders. The BPS approach to EBM, and frankly most of EBM anyway, takes all those concerns to such an extreme level, the entire discipline is essentially built on abusing those flaws. It has zero scientific validity.

And it's still not criticized. I guess because most people don't really see it as science anyway, which is in large part why it's so disastrous: all the power of a scientific process, none of the standards and zero responsibility.

It's a huge concern seeing this criticism dutifully avoid the very worst of what they are criticizing. That avoidance is purely political, has no reasonable explanation when it features everything they criticize, all the time. The very history of psychosomatics is basically this giant error in concentrated form, and pretty much where it's still causing the worst harm. But because it's political, not scientific, it even avoids being part of scientific criticism, despite having essentially having fully legal power over our lives as actual scientific evidence. Hell, a psychiatrist's opinion is valid expert testimony in a court of law, despite the standards of evidence in psychiatry being far lower than in judicial proceedings.

There is worst than doing bad science, and it's refusing to do science at all, instead going with narratives and tall tales. But it's everyone pretending it still counts, should still have power, that makes it so much worse.