Nature: "Science-integrity project will root out bad medical papers ‘and tell everyone’"

[forestglip]

Nature: "Science-integrity project will root out bad medical papers ‘and tell everyone’"

----

'The Center for Scientific Integrity — the non-profit organization that runs Retraction Watch — has launched a project aimed at rounding up flawed and fake medical-research papers and neutralizing their impact on health guidelines. The Medical Evidence Project will be run by James Heathers, a science-integrity consultant and self-described “data thug” based in Boston, Massachusetts.'

----

'The project, which has a US$900,000 grant from funder Open Philanthropy in San Francisco, California, to run for two years with a team of three to five people, aims specifically to root out flawed papers that have a serious impact on medical guidelines by skewing meta-analyses — reviews that combine the results of multiple similar studies to come to a statistically more powerful conclusion.'

----

'The Medical Evidence Project, which launched on 1 June, wants to provide a secure platform for whistle-blowers who are daunted by the idea of publishing their concerns about influential medical papers on public platforms such as PubPeer, or find that route ineffective. “I would like to be a resource for those people,” Heathers says. The project’s website will provide a mechanism to supply anonymous tips. People can also e-mail the project’s tip line.'

 

[Utsikt]

I think ME/CFS alone would give them plenty of work. But I doubt this will get the editors to listen.

 

[rvallee]

I think ME/CFS alone would give them plenty of work. But I doubt this will get the editors to listen.

The main difficulty is that although ME/CFS is an example of the worst of the bunch, the whole bunch of psychobehavioral ideology, and most of so-called evidence-based medicine, is just as flawed. That's almost every health care practice that isn't rooted in genuine understanding of biology. And pretty much all of clinical psychology.

It would be massively good if they had the spine to go there, but it would make so many people so angry at them for it, and probably prevent them from doing it where influencers are willing to accept it. As much as people want to imagine that truth matters, it just doesn't. What really matters is whether something can reliably make or save money. Everything else is up to opinion.

 

[Sasha]

'The Medical Evidence Project, which launched on 1 June, wants to provide a secure platform for whistle-blowers who are daunted by the idea of publishing their concerns about influential medical papers on public platforms such as PubPeer, or find that route ineffective. “I would like to be a resource for those people,” Heathers says. The project’s website will provide a mechanism to supply anonymous tips. People can also e-mail the project’s tip line.'

Seems like a perfect place to dob in the Larun Cochrane review, @Trish...

 

[Sasha]

And PACE, of course.

@dave30th

 

[Utsikt]

@Wyva maybe a place to send your letter?

 

[Trish]

I can't honestly see us getting anywhere with this sort of approach re PACE and the Cochrane Exercise review. I suspect they are just looking for outright fraudulent data, not things like inappropriate PROMs and subjective outcomes in unblinded trials.

 

[Sasha]

I can't honestly see us getting anywhere with this sort of approach re PACE and the Cochrane Exercise review. I suspect they are just looking for outright fraudulent data, not things like inappropriate PROMs and subjective outcomes in unblinded trials.

I get the exact opposite impression! My bolding:

a project aimed at rounding up flawed and fake medical-research papers... The project... aims specifically to root out flawed papers that have a serious impact on medical guidelines by skewing meta-analyses

If they were only interested in fake papers, surely they wouldn't have said 'flawed and fake'? And what is every single CBT/GET paper but a flawed paper that has a serious impact on medical guidelines?

We've seen Cochrane fail repeatedly to self-police. Time to bring in the actual data police, and this group have set themselves up to be just that. I'd love to see us rush in there and be the first test of this project.

 

[Jonathan Edwards]

Time to bring in the actual data police, and this group have set themselves up to be just that.

As did Cochrane.

I may be in a particularly cynical mood today but whenever we have seen these do-gooders set up their quality policing projects the end result often seems to be the opposite of what is expected. I seem to remember Dorothy Bishop being heavily into policing but only too happy to endorse PACE and complain about patient advocates. GRADE is supposed to ensure quality assessment but is pseudo arithmetic garbage. The 'improved' Risk of Bias tool allows more bias... And so on...

 

[Sasha]

As did Cochrane.

I may be in a particularly cynical mood today but whenever we have seen these do-gooders set up their quality policing projects the end result often seems to be the opposite of what is expected. I seem to remember Dorothy Bishop being heavily into policing but only too happy to endorse PACE and complain about patient advocates. GRADE is supposed to ensure quality assessment but is pseudo arithmetic garbage. The 'improved' Risk of Bias tool allows more bias... And so on...

This is Retraction Watch, though, just with more money, and IIRC, they've already been good on PACE.

 

[rvallee]

This is Retraction Watch, though, just with more money, and IIRC, they've already been good on PACE.

What I've seen of them suggests they may be up to the task. At least more than most other alternatives. It would be worth at least trying. I wouldn't put too much hope into where they would take it, but they do have a track record of not being awful, which is far better than average.

We've seen many people claiming some of this faceplant spectacularly, like the mental elf blog thing and the annoying health nerd dude. Most professional skeptics actually seem to feast on this pseudoscience more than the average physician.