The New York Times: Peer Review: The Worst Way to Judge Research, Except for All the Others

[Kalliope]

Aaron E. Carroll: Peer Review: The Worst Way to Judge Research, Except for All the Others

One way to detect problems with research earlier would be to let researchers post manuscripts online before submission, for public judgment before formal peer review. This is already common in some sciences, such as physics. Medical journals would probably resist this, however, because it could reduce their ability to get press and attention once the research was fully published.

A significant improvement would require a change in attitude. Too often, we think that once a paper gets through peer review, it’s “truth.” We’d do better to accept that everything, even published research, needs to be reconsidered as new evidence comes to light, and subjected to more thorough post-publication review.

 

[obeat]

Perhaps psychological studies should only be published AFTER the longterm follow up data is available. It would be more representative of real life.

 

[rvallee]

Perhaps psychological studies should only be published AFTER the longterm follow up data is available. It would be more representative of real life.

It would be great also if psychological research, often unblinded with self-reported outcomes, was not allowed to arbitrarily overrule biomedical research. I don't understand how this shift is happening so quietly and without any concern when the entire field of psychology is undergoing a crisis of replicability.

This whole mind-body obsession is really corrupting rational thinking. It's fascinating from an academic perspective but it doesn't have any evidence yet and has no place in clinical guidelines.

 

[Sean]

A significant improvement would require a change in attitude. Too often, we think that once a paper gets through peer review, it’s “truth.” We’d do better to accept that everything, even published research, needs to be reconsidered as new evidence comes to light, and subjected to more thorough post-publication review.

Passing peer-review just admits a paper to the highest level of debate. It doesn't guarantee its veracity and validity. It is still very much a work in progress.

Perhaps psychological studies should only be published AFTER the longterm follow up data is available. It would be more representative of real life.

I would also make long-term follow-up the primary data/outcome. Especially for chronic conditions.

 

[Webdog]

UpToDate has a formal peer review process (I actually have a copy of this). Repeated peer reviews have slowly moved UpToDate from strongly recommending GET/CBT for all ME/CFS patients to a weaker stance. But peer review still accepts PACE as evidence.

The UpToDate peer review process in early Nov 2018 found that PACE, despite its limitations, is sufficently reliable to conclude "Counseling therapies (eg, cognitive behavioral therapy [CBT]) and graded exercise therapy (GET) may produce meaningful benefits in some patients."

 

[Sean]

may produce meaningful benefits in some patients."

I don't think that word means what they think it means.