Blog: Gelman, "The puzzle: Why do scientists typically respond to legitimate scientific criticism in an angry, defensive, closed, non-scientific way?"

Andy

Andy

Retired committee member
The puzzle: Why do scientists typically respond to legitimate scientific criticism in an angry, defensive, closed, non-scientific way? The answer: We’re trained to do this during the process of responding to peer review.

Here’s the “puzzle,” as we say in social science. Scientific research is all about discovery of the unexpected: to do research, you need to be open to new possibilities, to design experiments to force anomalies, and to learn from them. The sweet spot for any researcher is at Cantor’s corner. (See here for further explanation of the Cantor connection.)

Buuuut . . . researchers are also notorious for being stubborn. In particular, here’s a pattern we see a lot:
– Research team publishes surprising result A based on some “p less than .05” empirical results.
– This publication gets positive attention and the researchers and others in their subfield follow up with open-ended “conceptual replications”: related studies that also attain the “p less than .05” threshold.
– Given the surprising nature of result A, it’s unsurprising that other researchers are skeptical of A. The more theoretically-minded skeptics, or agnostics, demonstrate statistical reasons why these seemingly statistically-significant results can’t be trusted. The more empirically-minded skeptics, or agnostics, run preregistered replications studies, which fail to replicate the original claim.
– At this point, the original researchers do not apply the time-reversal heuristic and conclude that their original study was flawed (forking paths and all that). Instead they double down, insist their original findings are correct, and they come up with lots of little explanations for why the replications aren’t relevant to evaluating their original claims. And they typically just ignore or brush aside the statistical reasons why their original study was too noisy to ever show what they thought they were finding.

So, the puzzle is: researchers are taught to be open to new ideas, research is all about finding new things and being aware of flaws in existing paradigms—but researchers can be sooooo reluctant to abandon their own pet ideas.

OK, some of this we can explain by general “human nature” arguments. But I have another explanation for you, that’s specific to the scientific communication process.
Click to expand...
http://andrewgelman.com/2018/01/13/...sm-angry-defensive-closed-non-scientific-way/
 
I have been saying this for a while now. The scientific method is good. The institutions and people in control of it are very fallible.

If you say X is bad and you publish a paper on it, your ego is now at least somewhat invested in X being bad, and you are also now vulnerable to things like confirmation bias.

Now if that paper gets recognized and you gain some fame from it, you may get promoted. You may become an expert on X. Now, more people that confirm X get promoted. Fast forward a few years and you and people like you, with the same ideas on X are in powerful situations, possibly on boards that decide where grant money goes.

Now if some whippersnapper comes up with a grant request for a project that says X is good. Your ego is massively invested in this. Would be pretty embarrasing if you got social recognision / fame for a position on X is wrong. Your position in the scientific establishment is at stake. If X is wrong then you are no longer the expert on the matter, perhaps you get demoted. Your biases also prevent you from granting the proposal. This has been settled years ago, the grant money needs to be spent wisely, etc.

So the people trying to say X is good have to struggle to find grant money and have their names dragged through the mood via the scientific establishment. I have read several accounts of this.

This all means big change of establishment opinion takes way longer than it should.

Scientific method = Good.
Human = Bad. Very naughty human.
 
One of the biggest "wars" going on between groups of scientists these days is the war between those who think we should be eating low fat, high carb, and those who think we should be eating high fat, low carb.

There has been particularly fierce fighting in South Africa (the scientist being accused of heresy being Professor Tim Noakes) and Australia (with the casualty being Gary Fettke, who has been accused of "inappropriate reversal of Type 2 diabetes").

http://foodmed.net/the-noakes-trial/

https://drmalcolmkendrick.org/2016/...bohydrate-diet-are-silenced-around-the-world/

However, unlike many subjects in science, what people eat is not under the control of scientists, and so more and more people appear to have decided that low fat, high carb eating is bunkum, and are "doing it for themselves" by switching to high fat, low carb. Hooray!
 
As an engineer I am typically involved in development projects that can run for years. When you are deeply involved in some fascinating and important aspect of a project, it becomes "your baby"; something you have put lots of yourself into, including professional reputation - major personal investment. I can imagine that for researchers involved in clinical trials this must also be very true, probably even more so. It is therefore going to hurt like hell if things turn out less rosy than you thought. But the point is, if they cannot cope with that and do the right thing, they should never have got into it in the first place.
 
Last edited:
sb4 said:
I have read several accounts of this.
Click to expand...

I experienced it personally. I was an outsider who had discovered something new (being a woman in a male-dominated scientific world didn't help either). A common response was, to the effect - "But how could you have discovered something new? That means we missed it and we are the experts. You must be wrong."

I had reviewers of papers submitted for publication or grant applications recommend against, because it didn't fit in with what we know about xyz, despite not being able to find fault with my experimental data. Fortunately most Journal Editors were not swayed and I got enough funding to struggle on, but it was difficult.

So yes, scientists are just as flawed as other human beings.
 
ce9391143602fd8cd9ec0df530873ffc-g.jpg
 
@alicec Yes I can imagine what it's like. You may be interested in reading books by Dr Robert O'Becker, Dr Andrew Marino, and Gerald Pollack. Even if you do not agree with their conclusions (the former 2 believe that EMFs are harmful, the latter believes most of what we know about water is wrong) the lengths that scientific institutions will go to stop their research and discredit them is extraordinary.
 
I'm not surprised when scientists react badly to criticism. Sometimes you have to be quite single minded to push ideas through and in doing so ignore criticism. However, its good to listen and understand the criticisms as it can lead to strengthening the approach or the supportive arguments.
 
Paolo Macchiarini's position and defense of it by the Karolinska Institute would be a case in point where reputations and funding brought in by leading researchers outweigh the truth until a tipping point is reached that makes association with them toxic.
Journals also have difficulty correcting the record as this calls into question their reputation as gatekeepers and the quality of the peer review process that can make science clubbable http://retractionwatch.com/category/by-author/paolo-macchiarini/
(interesting aside that Paolo Macchiarini also published in The Lancet in 2011 with as much ballyhoo as the PACE trial )
 
You must log in or register to reply here.
Back
Top Bottom