Replication games: how to make reproducibility research more systematic, 2023, Brodeur et al

Nature: comment article

Replication games: how to make reproducibility research more systematic

Abel Brodeur, Anna Dreber, Fernando Hoces de la Guardia & Edward Miguel

In some areas of social science, around half of studies can’t be replicated. A new test-fast, fail-fast initiative aims to show what research is hot — and what’s not.

In October last year, one of us (A.B.) decided to run an ad hoc workshop at a research centre in Oslo, to try to replicate papers from economics journals. Instead of the handful of locals who were expected to attend, 70 people from across Europe signed up. The message was clear: researchers want to replicate studies.

Replication is sorely needed. In areas of the social sciences, such as economics, philosophy and psychology, some studies suggest that between 35% and 70% of published results cannot be replicated when tested with new data1–4. Often, researchers cannot even reproduce results when using the same data and code as the original paper, because key information is missing.

Yet most journals will not publish a replication unless it refutes an impactful paper. In economics, less than 1% of papers published in the top 50 journals between 2010 and 2020 were some type of replication5. That suggests that many studies with errors are going undetected.

Open: https://www.nature.com/articles/d41586-023-02997-5

 

Yeah that's been one of the biggest factors why things are so messed up in evidence-based medicine. Sharpe actually replied once to someone criticizing the many flaws in PACE that they should just run their own multimillion dollar trial that takes 10 years to complete. What a ridiculous troll. You can't fix unprofessionalism like this with a few rules that people can just work around anyway.

There is no way to speed up this process unless the whole system changes. Of course it all needs to be streamlined, the whole thing is built on the most excessive, almost ostentatious, redundancy and inefficiency you can imagine, but this can't be done piecemeal, it's a whole paradigm change away from a model that has failed to deliver much, but I guess is a lot of fun because you can pretty much do whatever and get away with it.

Since it's so inefficient, it's very expensive, which means huge budgets, which sounds very impressive, allows people to boast about running multimillion dollar studies, even though it's actually shameful since they could have better answers for 10% of the cost and taken 10% of the time. But then they'd get too many results they don't like so almost no one is willing to go with that change, let alone be the very unpopular people pushing for it.

I don't see a human solution to this, the problem is human nature and culture, a set of interlocking gears that can't be fixed, only bypassed entirely. Only AIs will be able to end this failure loop. Fortunately, that will happen soon. The old guard will be very angry, but they'll be so outclassed that it won't matter much. They'll be like hand-shovelers trying to stop a giant excavation machine from scooping out a week's worth of their hard labor in a single swing. And it can't happen soon enough.