Nature human behaviour: Comparing meta-analyses and preregistered multiple-laboratory replication projects - 2019, by Amanda Kvarven et al

[Kalliope]

Nature: Comparing meta-analysis and preregistered multiple-laboratory replication projects - Amanda Kvarven, Eirik Strømland & Magnus Johannesson

Abstract:
Many researchers rely on meta-analysis to summarize research evidence. However, there is a concern that publication bias and selective reporting may lead to biased meta-analytic effect sizes. We compare the results of meta-analyses to large-scale preregistered replications in psychology carried out at multiple laboratories. The multiple-laboratory replications provide precisely estimated effect sizes that do not suffer from publication bias or selective reporting. We searched the literature and identified 15 meta-analyses on the same topics as multiple-laboratory replications. We find that meta-analytic effect sizes are significantly different from replication effect sizes for 12 out of the 15 meta-replication pairs. These differences are systematic and, on average, meta-analytic effect sizes are almost three times as large as replication effect sizes. We also implement three methods of correcting meta-analysis for bias, but these methods do not substantively improve the meta-analytic results.

The article is not on Sci-Hub yet, but can be downloaded as PDF here
Comparing Meta-Analyses and Pre-Registered Multiple Labs ...https://osf.io › brzwt › download

 

[rvallee]

Meta-analyses make sense, but not being bothered by garbage-in-garbage-out is just intellectually bankrupt. Like BPS, it's not the idea that is bad, it's an implementation that diverges from the pretend objectives. Instead of summarizing the best evidence, it is being abused to promote personal opinions and not even in a subtle way. In some cases it even summarizes the worst evidence and entirely out of self-interest.

But medicine is stuck in this weird state where it's acknowledged that obviously many things are wrong but not this specific thing, and not that specific thing, those are the literal word of god and one must assume mental incapacity for suggesting otherwise. Like Horton saying in his opinion that half of all research is wrong, just not his, just not anything in his journal, no, that's impossible, it's everyone else that is wrong and "activism" is good, unless it contradicts what Horton believes in, then it's obviously bad.

There's this huge house of cards that too many have taken residence in and that means the house must remain standing for the sake of keeping the house standing, not for any other purpose. No wonder patient engagement is systematically rejected, we see right through it and that makes us heretics, or something like that. Instead the card factory output has been increased to keep up with the demand from card creators and houses of card architects and designers.

 

[Snowdrop]

As a non-science trained person I'm having some difficulty following this.

For others like me there is a discussion of effect size; what it is/why it matters Here

I'm kinda surprised they found so many psych studies that had replications. There seems to be no surmising as to why there is such a large difference. Are they suggesting there is a problem with meta-analysis?

 

[James Morris-Lent]

I think the takeaway is that if you have to do a meta-analysis, it means there is no convincing definitive study.

If there is no convincing definitive study, you surely don't have anything that can be considered useful.

_______

There is another issue that is clearly particularly pertinent to psychology research, as we see with the BPS stuff:

An experiment may well be highly replicable, but the interpretation is off-base - the outcomes are not actually showing what the researchers say they are.

As we see with unblinded therapies with subjective outcomes, it is very easy to repeatedly manufacture significant differences in certain outcomes, but positive outcome can't be interpreted to mean anything.

Or in more basic experimental psychological research, you might elicit some behavior out of people in a contrived laboratory experiment, but that does not automatically mean you can reasonably or meaningfully extrapolate that behavior.