Article: Scientists Aim To Pull Peer Review Out Of The 17th Century

[Indigophoton]

An article discussing the problems with peer review, focusing particularly on the biomedical field, the effect of the pressure to publish to gain career advancement, and a possible alternative approach.

The technology that drives science forward is forever accelerating, but the same can't be said for science communication. The basic process still holds many vestiges from its early days — that is the 17th century.

Some scientists are pressing to change that critical part of the scientific enterprise.

Here's what they're confronting: When researchers studying the biology of disease make a discovery, it typically takes nine months for them to get their results published in a journal.

One reason for that delay is it goes through a process of peer review that is both necessary and antiquated. The fate of that paper rests on just two or three scientists who have been asked to review it and decide whether it's worthy of being published.

Imagine how this would feel if the matter in question were a consumer product.

"If the only thing Amazon ever published were reviews of the first three people who bought a product, then we'd have a very ineffective system for knowing what was good and bad," says Michael Eisen, a Howard Hughes Medical Institute (HHMI) investigator at University of California, Berkeley.

https://www.npr.org/sections/health...m-to-pull-peer-review-out-of-the-17th-century

The new approach and the issues are described in detail here, APPRAISE (A Post-Publication Review and Assessment In Science Experiment)

 

[Peter T]

Well most people reading this will be aware of examples of where peer review has failed, and also where journals ignore concerns raised by reviewers.

 

[Alvin]

I very much agree the process is flawed but i don't think

"The fact that people are going to be reviewing and assessing the work ... should provide a very strong incentive back to authors to try to produce work that's actually reproducible and durable, rather than work that is just flashy and gets into the right journal," Eisen tells Shots.

is a panacea. It may be an incremental improvement at best.

Also this system is easy to game, "publish" something thats utter bullcrap but appeals to a fringe or reality denying group, their media or even mainstream gets a hold of it, hypes it up on the news and its a millennia of playing whack a mole even when its shot down and deleted. Two examples being vaccines causing autism (which got published then redacted) and homeopathy. Their harmful impacts despite being discredited garbage will probably never be completely undone

Replication may be the only final answer, but a costly and resource intensive solution.

 

[Jonathan Edwards]

Also this system is easy to game, "publish" something thats utter bullcrap but appeals to a fringe or reality denying group, their media or even mainstream gets a hold of it, hypes it up on the news and its a millennia of playing whack a mole even when its shot down and deleted. Two examples being vaccines causing autism (which got published then redacted) and homeopathy.

But that is the situation we have now. 90% of what gets published in biomedical research is already bullshit. The 'proof' that homeopathy worked through water memory was published in Nature and the anti-vax story and PACE in the Lancet. The point is that peer review simply does not work. What works is people pointing out that stuff is bad later - which is exactly what Eisen is suggesting we rely on. The only reason why these bad studies had any credibility for the time they did was because they had the stamp of peer review approval. Take that away and such studies have to stand on their own merit. As things are now someone would point out within 24 hrs that PACE is meaningless or that water memory is not possible when the tablets have no water in them!

It is interesting to look at the comments on this article. There are clearly still people who feel there has to be some sort of voice of authority. There is no need. You have to learn how to judge papers for yourself in science. There is certainly a need to get authors to write good quality text but at the moment the quality control is going down and down. If you just had to get your head of department's approval for publishing (which at the moment you do not) because each College published all its own people's stuff standard would rocket up. I would not have allowed my juniors to publish poorly written text if it the quality control was seen to be down to me.

 

[Trish]

If you just had to get your head of department's approval for publishing (which at the moment you do not) because each College published all its own people's stuff standard would rocket up.

I agree with most of what you say, but what if it's the heads of departments that are publishing crap, as seems to be the case with PACE, SMILE, FITNET etc. where the people doing the research are professors.

 

[Jonathan Edwards]

I agree with most of what you say, but what if it's the heads of departments that are publishing crap, as seems to be the case with PACE, SMILE, FITNET etc. where the people doing the research are professors.

I was only talking about the standard of writing there. A head of department would have to take responsibility for the fact that his juniors could not even construct a proper argument. If the head of department is no good at science itself, then that gets picked up in 24hrs by people like Tom Kindlon posting comments - just as you already can with PLOS One.

 

[Trish]

I was only talking about the standard of writing there.

Ah, OK. I guess those professors can write coherent English. It is their 'science' that is incoherent.

 

[Alvin]

But that is the situation we have now. 90% of what gets published in biomedical research is already bullshit. The 'proof' that homeopathy worked through water memory was published in Nature and the anti-vax story and PACE in the Lancet. The point is that peer review simply does not work.

No argument here

What works is people pointing out that stuff is bad later - which is exactly what Eisen is suggesting we rely on.

This is close enough what is happening now and is not working.

The only reason why these bad studies had any credibility for the time they did was because they had the stamp of peer review approval. Take that away and such studies have to stand on their own merit.

This works when readers and voters want to find the truth, in reality they believe what fits their ideology and reject what does not. Peer review is meant to weed out things but any scientist who thinks its an infallible stamp is an idiot who is not going to wisen up when given more power.

As things are now someone would point out within 24 hrs that PACE is meaningless or that water memory is not possible when the tablets have no water in them!

People point out lots of alternative facts but those who want to believe do. Giving more power to people to publish more lies means more failure. Its strange i have to tell you this, your the scientist, you should know this already.

It is interesting to look at the comments on this article. There are clearly still people who feel there has to be some sort of voice of authority. There is no need. You have to learn how to judge papers for yourself in science.

World events show people and even voters are not interested in the truth or know how to figure it out or have the ability to learn. Lets not forget even well meaning people believe taking from the poor and giving to the rich will benefit them even when they are the ones being taken from and that allowing companies to poison the land/water/air wont happen if you stop enforcing environmental laws or that climate change is a hoax...

There is certainly a need to get authors to write good quality text but at the moment the quality control is going down and down. If you just had to get your head of department's approval for publishing (which at the moment you do not) because each College published all its own people's stuff standard would rocket up. I would not have allowed my juniors to publish poorly written text if it the quality control was seen to be down to me.

We have had this argument before. This only works because your an ethical person, those who have no scruples will happily publish the most baldfaced lie and call it ice cream (or Italian restaurants).
Giving license to unethical people and believing they won't abuse it is ridiculous

Peer review is meant to add a few sets of eyes to catch flaws because unfettered publishing did not prevent lies from being published. Its very imperfect but the answer is not to undo the only failsafe available because it sucks, if it was the answer then attempted failsafe would never have been needed in the first place


BTW an author's reputation is already on the line, the names of the peer reviewers are not published

 

[Valentijn]

This is close enough what is happening now and is not working.

No, it's not what is happening now. The peer review process is still being treated as largely infallible, and post-publication commentary is being delegitimized on that basis. The primary pro-PACE argument is still appeal to authority - that the objections are from crazy/ignorant patients, and the papers were peer-reviewed and published in respected journals. The peer review process is helping authors avoid engaging with the issues being raised post-publication.

Peer review is meant to add a few sets of eyes to catch flaws because unfettered publishing did not prevent lies from being published. Its very imperfect but the answer is not to undo the only failsafe available because it sucks, if it was the answer then attempted failsafe would never have been needed in the first place

Peer review may have a valid role to play, but it should be recognized as a minor one. Making it a major component is introducing huge delays in the reporting of scientific discoveries - or the disproving of them, etc. And then it's delaying or completely preventing the retraction of errors, or even any scrutiny in the first place.

BTW an author's reputation is already on the line, the names of the peer reviewers are not published

Not really. The authors can (and do) claim that errors are not their fault (or cannot exist at all) because of the peer-review process having been involved. They get it past the peer reviewers, and suddenly they're no longer accountable. And peer reviewers are being more frequently named now.

 

[Alvin]

Peer review may have a valid role to play, but it should be recognized as a minor one. Making it a major component is introducing huge delays in the reporting of scientific discoveries - or the disproving of them, etc. And then it's delaying or completely preventing the retraction of errors, or even any scrutiny in the first place.

I agree its a minor one, i even argue its very fallible, frankly independent replication is the only true antidote
Peer review prevents retraction or scrutiny? I just can't agree, its an excuse

Not really. The authors can (and do) claim that errors are not their fault (or cannot exist at all) because of the peer-review process having been involved. They get it past the peer reviewers, and suddenly they're no longer accountable.

So authors being scrutinized use every excuse in the book. This is not news, this is standard operating procedure, you may eliminate one pathetic excuse but its ludicrous to believe free reign with even less accountability will cause improvement.

The primary pro-PACE argument is still appeal to authority - that the objections are from crazy/ignorant patients, and the papers were peer-reviewed and published in respected journals. The peer review process is helping authors avoid engaging with the issues being raised post-publication.

So removing one excuse will fix the problem? Excuses are a dime a dozen, its like playing whack a mole. When eliminating peer review doesn't work and they still claim we are making threats perhaps anonymizing authors will be the new fad, especially when even more shoddy research gets published with even less quality control? The excuse will become having to attach their names means they can't pursue the best research which is the real reason medical research is so slow
Is the grass is always greener on the other side?
This seems to be how con men gain followers, blame the imperfect checks and balances that do exist and keep repeating it till people believe it. If a system does not work pretending that eliminating it and going back to what failed will work is not the solution, what you really need to do is make improvements
Why is giving more power to those willing to abuse it the answer?

No, it's not what is happening now. The peer review process is still being treated as largely infallible, and post-publication commentary is being delegitimized on that basis.

So people who believe in homeopathy and vaccines causing autism would give up those beliefs if peer review is eliminated. No offense but thats laughable

Its funny that i'm the one defending peer review, i think its a bad idea. My point is that anarchy publishing won't work better, peer review was brought in as a response to unsuccessful self regulation
It still doesn't work but forgetting why unsuccessful self regulation is bad and pretending going back to it will self regulate is insane.



We need a better solution then self regulation or peer review.

 

[Valentijn]

My point is that anarchy publishing won't work better, peer review was brought in as a response to unsuccessful self regulation

No one is proposing anarchy, and I think it's extremely likely that processes and standards would remain even if peer review ceases to be the main safeguard. If there's an institutional publishing scheme at all of the Universities, for example, some would certainly be of poor quality. But there could also be ratings of the universities (or other entities) based on the quality of their publications, much like there is of journals now - except the focus could be on quality rather than popularity/sensationalism prioritized by profit-driven journals.

They're not proposing a wildly different setup, and reviewers would likely face some requirements:

But people who sign their name to their reviews, or are certified by a trusted body, would have an incentive not to simply shower praise on studies posted by friends and colleagues.

 

[Alvin]

No one is proposing anarchy, and I think it's extremely likely that processes and standards would remain even if peer review ceases to be the main safeguard. If there's an institutional publishing scheme at all of the Universities, for example, some would certainly be of poor quality. But there could also be ratings of the universities (or other entities) based on the quality of their publications, much like there is of journals now - except the focus could be on quality rather than popularity/sensationalism prioritized by profit-driven journals.

I don't think shifting the reputation from journal to university is going to make any difference, people already judge universities (especially when deciding which one to apply to) and that has not stopped bad science. How would you change the focus to quality?
Around here we have an expression, you don't put a fox in charge of the hen house

 

[Valentijn]

I don't think shifting the reputation from journal to university is going to make any difference, people already judge universities (especially when deciding which one to apply to) and that has not stopped bad science. How would you change the focus to quality?

The primary goal of a journal is to make a profit. Some universities also are profit-driven, but that varies between countries and institutions, and they are always driven to improve their reputation. A journal isn't going to be too bothered if it's reputation takes a hit so long as it's still making money in the short-term. A university will be, and it will also then be more directly accountable for the quality of work produced by its own researchers.

 

[Alvin]

The primary goal of a journal is to make a profit. Some universities also are profit-driven, but that varies between countries and institutions, and they are always driven to improve their reputation. A journal isn't going to be too bothered if it's reputation takes a hit so long as it's still making money in the short-term. A university will be, and it will also then be more directly accountable for the quality of work produced by its own researchers.

You realize studies published today typically identity the university involved so the bad quality is not sinking them. Identifying them as the publisher is not going to be a revelation or a revolutionary advancement when their name is already on the study.

 

[Valentijn]

You realize studies published today typically identity the university involved so the bad quality is not sinking them. Identifying them as the publisher is not going to be a revelation or a revolutionary advancement when their name is already on the study.

But the universities are generally not paying attention to the quality of studies produced by their employees. If the universities are responsible for publishing those papers, suddenly it's going to be a much bigger problem if there's a controversy about one of their papers.

 

[Alvin]

But the universities are generally not paying attention to the quality of studies produced by their employees. If the universities are responsible for publishing those papers, suddenly it's going to be a much bigger problem if there's a controversy about one of their papers.

Huh? If the university published the paper they will then care their name is on it but don't care now that their name is on bad research?
Will the extra bolding or having the name on the title page twice instead of once suddenly change how they are regarded?

 

[Arnie Pye]

One thing I wish would happen with scientific research papers is that authors would do their best to write in as clear a manner as possible using as much standard English as they can, and using statistics that explain and discuss their findings rather than obscure them.

If I take a look at serious medical research papers from, say, 1960 - 1970, I can understand many of them. There was clearly a desire to explain to and educate their readers rather than obfuscate.

I realise this is never going to happen though.