The Black Box of Meta-Analysis

[Sly Saint]

(was going to post this on another thread I set up, but the title to that thread has been changed so am posting this separately)
full title
The Black Box of Meta-Analysis: Personality Change
August 11, 2019

Psychologists treat meta-analyses as the gold standard to answer empirical questions. The idea is that meta-analyses combine all of the relevant information into a single number that reveals the answer to an empirical question. The problem with this naive interpretation of meta-analyses is that meta-analyses cannot provide more information than the original studies contained.

If original studies have major limitations, a meta-analytic integration does not make these limitations disappear. Meta-analyses can only reduce random sampling error, but they cannot fix problems of original studies. However, once a meta-analysis is published, the problems are often ignored and the preliminary conclusion is treated as an ultimate truth.

In psychology, the review process and publication in a top journal give the appeal that information is trustworthy and can be cited as solid evidence. However, a closer inspection of the original studies might reveal that the results of a meta-analysis rest on shaky foundations.

Updating theories in the face of new data is at the basis of science. Citing an outdated meta-analysis as if it provided a timeless answer to a question is not.

https://replicationindex.com/2019/08/11/the-black-box-of-meta-analysis-personality-change/

 

[rvallee]

The analyses of the Cochrane reviews by Mark Vink show how completely broken the process of meta-analysis can be, perfect example of garbage-in-garbage-out when biased reviewers are allowed exemptions from norms. Well, that and the initial peer review that was pretty brutal and still saw the GET review be slapped with a "low bias, high effect" label. All because people didn't give a damn about us and thought it was harmless since "there is no disease", nevermind that this is an opinion and a shitty one at that.

I have no idea how psychological research got granted so much affirmative action but that it happened this late and after so many harsh lessons about the value of the scientific method it has to rank as one of the top 10 blunders in the whole history of science, one that is filled to the brim with major disasters.

I'm sick of all this pseudoscience, it's turning out to be as bad for humanity as the major world wars. What a freaking mess.

 

[alex3619]

The Black Box of Meta-Analysis: Personality Change

I have long considered that meta-analysis can reinforce problems if there are systemic methodological biases in a field. CBT/GET comes to mind.

 

[alex3619]

The analyses of the Cochrane reviews by Mark Vink show how completely broken the process of meta-analysis can be, perfect example of garbage-in-garbage-out when biased reviewers are allowed exemptions from norms.

Precisely.