Nature: Rein in the four horsemen of irreproducibility

Andy

Andy

Retired committee member
More than four decades into my scientific career, I find myself an outlier among academics of similar age and seniority: I strongly identify with the movement to make the practice of science more robust. It’s not that my contemporaries are unconcerned about doing science well; it’s just that many of them don’t seem to recognize that there are serious problems with current practices. By contrast, I think that, in two decades, we will look back on the past 60 years — particularly in biomedical science — and marvel at how much time and money has been wasted on flawed research.

How can that be? We know how to formulate and test hypotheses in controlled experiments. We can account for unwanted variation with statistical techniques. We appreciate the need to replicate observations.

Yet many researchers persist in working in a way almost guaranteed not to deliver meaningful results. They ride with what I refer to as the four horsemen of the reproducibility apocalypse: publication bias, low statistical power, P-value hacking and HARKing (hypothesizing after results are known). My generation and the one before us have done little to rein these in.
Click to expand...
https://www.nature.com/articles/d41586-019-01307-2
 
In 1975, psychologist Anthony Greenwald noted that science is prejudiced against null hypotheses; we even refer to sound work supporting such conclusions as ‘failed experiments’. This prejudice leads to publication bias: researchers are less likely to write up studies that show no effect, and journal editors are less likely to accept them. Consequently, no one can learn from them, and researchers waste time and resources on repeating experiments, redundantly.
Click to expand...
Definitely true. Something needs to change so the value of all useful contributions to the sum total of knowledge can be properly recognised. Discovering that something does not show significant effect is often extremely useful. It's so fundamental.
 
Trish said:
Strange that the author, as an experimental psychologist, says the worst offenders are biomedical scientists. What about the crisis in psychology research?
Click to expand...
The implication I took from it is that 'biomedical science' takes a lot more money and is what we feel holds the most promise for improvement of the human condition. So maybe it's not the worst offender, but its offenses carry the most weight.
 
Last edited:
Trish said:
Strange that the author, as an experimental psychologist, says the worst offenders are biomedical scientists. What about the crisis in psychology research?
Click to expand...
James Morris-Lent said:
The implication I took from it is that 'biomedical science' takes a lot more money and is what we feel holds the most promise for improvement of the human condition. So maybe it's not the worst offender, but its offenses carry the most weight.
Click to expand...
Yes, obviously we focus - rightly so - on the absurdities of much of BPS-driven ME/CFS pseudo science, but I would be amazed if biomedical research is not also prone to dodgy practices.
 
Barry said:
Yes, obviously we focus - rightly so - on the absurdities of much of BPS-driven ME/CFS pseudo science, but I would be amazed if biomedical research is not also prone to dodgy practices.
Click to expand...

I actually think all this stuff is a storm in a teacup. The real problem with science is something quite different.

Why are these studies so hard to reproduce? Because they were tests of not very clever ideas that turned out to be wrong. The real problem with science is that the vast majority of studies test hypotheses that aren't worth bothering with because there are good reasons for thinking they make no sense. Perhaps the commonest thing is testing a hypothesis that superficially looks as if it would explain something but with a bit of thought can be seen not to make those predictions at all.

The molecular mimicry hypothesis in immunology is an old chestnut. In ME I am sceptical about any hypotheses that try to explain symptoms on the basis of impaired energy metabolism. The symptoms don't fit with that because they occur after the exertion more than during it.

If you come across a really good idea, which if you are lucky you do once or twice in a career, then the results shout back at you that it is right. P values are not even needed.
 
Sly Saint said:
retweeted by Michael Sharpe (!?)


it takes one to know one
Click to expand...

I'm not sure if he's tweeting it in support of those. Seeing as it would essentially lead to retraction of most of his work, it's hard to imagine otherwise. Unless he just doesn't understand that, which is conceivable but... damn. He's certainly made it plenty clear he is a fool.
 
You must log in or register to reply here.
Back
Top Bottom