Blog: Social Psychology: Why I hate teaching the classics

[Indigophoton]

I’m approaching the end of my first semester teaching Intro to Social Psychology. As someone who came of age during the peak of the replication crisis (Bem, Stapel, Reproducibility Project), studies publication bias, and has had a hard time finding statistically significant results, I generally have a dim view of big chunks of the literature. I was worried that we would have very little to talk about given all the uncertainty, but we’ve made a good semester of it by talking about the general ideas, their strengths and weaknesses, and the opportunities for a young scientist to contribute by addressing these uncertainties.

But this semester’s teaching has taught me one thing: I hate teaching the classics.
What makes the classics, and why do I hate teaching them? The studies that my textbooks present as classics tend to have a few common attributes, some desirable and others undesirable.
The desirable:

1. They provide a useful summary of some broader theory.
2. They are catchy or sticky in a way that makes them easy to remember and fun to talk about.
3. The outcome is provocative and interesting.

The undesirable:

1. The sample size is tiny.
2. The p-values are either marginal or bizarrely good.
3. The outcome has little evidence of validity.
4. Data from the classic study tend to predate strong tests of the theory by several decades. The strongest evidence tends to come later (if at all) when people have cleaned up the methods and run more studies (often in response to criticism).

My concern is that these qualities of classics give students the wrong idea about what makes for good psychological science, leading them to embrace the desirable attributes of these classics without considering the undesirable attributes.

The author goes on to briefly outline some of the classic studies in social psychology and their flaws - quite amusing, and also alarming, if this is what (new) people in the field think might constitute good science.

http://crystalprisonzone.blogspot.co.uk/2018/04/why-i-hate-teaching-classics.html

There are also a couple of other articles on the site that may interest some, including one called "Curiously strong effects", which begins,

The reliability of scientific knowledge can be threatened by a number of bad behaviors. The problems of p-hacking and publication bias are now well understood, but there is a third problem that has received relatively little attention. This third problem currently cannot be detected through any statistical test, and its effects on theory may be stronger than that of p-hacking.

I call this problem curiously strong effects.

The Problem of Curiously Strong
Has this ever happened to you? You come across a paper with a preposterous-sounding hypothesis and a method that sounds like it would produce only the tiniest change, if any. You skim down to the results, expecting to see a bunch of barely-significant results. But instead of p = .04, d = 0.46 [0.01, 0.91], you see p < .001, d = 2.35 [1.90, 2.80]. This unlikely effect is apparently not only real, but it is four or five times stronger than most effects in psychology, and it has a p-value that borders on impregnable. It is curiously strong.

The result is so curiously strong that it is hard to believe that the effect is actually that big. In these cases, if you are feeling uncharitable, you may begin to wonder if there hasn't been some mistake in the data analysis. Worse, you might suspect that perhaps the data have been tampered with or falsified.

http://crystalprisonzone.blogspot.co.uk/2017/05/curiously-strong-effects.html

 

[alktipping]

building your field of study on sand inevitably leads to destruction. how did something based on the beliefs of a few people with ridiculous ideas become classed as a science in the first place. bid ideas do not translate often into provable facts or falsifiable theories. just my take on how this teacher follows dogma when he knows it will lead to the same poor experiments of the past.