Opinion piece in Nature: A toast to the error detectors

[Andy]

Last month, I got a private Twitter message from a postdoc bruised by the clash between science as it is and how it should be. He had published a commentary in which he pointed out errors in a famous researcher’s paper. The critique was accurate, important and measured — a service to his field. But it caused him problems: his adviser told him that publishing such criticism had crossed a line, and he should never do it again.

Scientists are very quick to say that science is self-correcting, but those who do the work behind this correction often get accused of damaging their field, or worse. My impression is that many error detectors are early-career researchers who stumble on mistakes made by eminent scientists, and naively think that they are helping by pointing out those problems — but, after doing so, are treated badly by the community.

https://www.nature.com/articles/d41586-019-03909-2

 

[James Morris-Lent]

I wonder if we should be actively trying to engage some substantial fraction of science graduates as professional 'error-detectors' and/or disinterested replicators.

I have to think that, for most people, it's unrealistic to ask them to churn out enough original research to stay afloat in academia and put in a spirited job of policing the quality of research in their general neighborhood. People only have so much stamina. Especially when you pile on the problems mentioned in the article that can come your way if you make a habit of speaking out against those more eminent.

 

[Sly Saint]

It’s time to be kinder to those doing the criticizing, and to demand more accountability and humility from those in power. Instead of punishing people who flag errors, we should scramble to hire them, give them prizes and award them grants so they can keep improving science. The least we can do is provide a space for fact-based criticism that is safe from intimidation and retaliation.

 

[chrisb]

Apart from research errors there seems to be a major problem with simple misstatements in books and articles. I have recently become rather incensed with them.

Looking on Amazon recently I saw the statement by Strauss in the Foreward to Chronic Fatigue Syndrome (1996) eds Demitrack and Abbey that

Recent epidemiological studies in England showed that the persistence of fatigue after an acute viral illness is strongly dependent on pre-existing and co-morbid psychiatric factors (Bruce -Jones et al 1994; Cope et al 1994)

I was surprised that I was unaware of this as one would have expected much to have been made of it. The simple reason for the lack of awareness is that the statement conveys a message opposite to the truth.

Here is the abstract

Two hundred and fifty patients attending primary care with glandular fever or an upper respiratory tract infection were studied prospectively up to 6 months after onset. Of these patients 228 were interviewed with the Life Events and Difficulties Schedule and the Schedule for Affective Disorders and Schizophrenia, giving Research Diagnostic Criteria for psychiatric disorders. The experience of severe social adversity (provoking agents) had a significant association with psychiatric disorder at 2 months (odds ratio = 5·3) and 6 months (odds ratio = 5·8) after onset of infection. This association was especially significant for depressive illness (odds ratio = 9·1 at 2 months and 11·9 at 6 months). In contrast, social adversity had little association with the development of the post-infectious fatigue syndrome, or delayed physical recovery. Social adversity may be an important maintaining factor for psychiatric disorders, especially depressive illness, following acute infections.


Bruce-Jones, W., White, P., Thomas, J., & Clare, A. (1994). The effect of social adversity on the fatigue syndrome, psychiatric disorders and physical recovery, following glandular fever. Psychological Medicine, 24(3), 651-659. doi:10.1017/S003329170002780X

How does Strauss not recognise the error, and how do the editors let it pass, especially as Abbey describes it correctly in her own chapter. Only those already fairly familiar with the literature will spot the error, and they are unlikely to be paying much heed to the book. Meanwhile the rest are wholly misled.


I was also appalled to see this in the BMJ from 1993 by PK Thomas

One such outbreak, which was to have an important influence on the course of events, took place at the Royal Free Hospital, London, in 1955. Little doubt exists that this was an epidemic of mass conversion hysteria,7 possibly triggered by a small nucleus of cases of postinfective encephalomyelitis. The term "benign myalgic encephalomyelitis," later abbreviated to myalgic encephalomyelitis (ME), was applied to this epidemic

BMJ. 1993 Jun 12; 306(6892): 1557–1558.
doi: 10.1136/bmj.306.6892.1557

The chronic fatigue syndrome: what do we know?
P K Thomas

Bear in mind that this is the man who was called in to advise the DWP and its junior minister a few months later. It seems that he, and the BMJ, were completely oblivious to the fact that Rachel Jenkins ,The Principal Medical Officer to the Department of Health, had in a lengthy introduction to Post Viral Fatigue Syndrome (1991) eds Jenkins and Mowbray argued against the mass conversion hysteria hypothesis. "Little doubt exists" and such doubt as does exist must not be referred to. And whoever thought there would be references to "mass conversion hysteria" in 1993?

These are the ways in which the history of our illness was subverted.

 

[Trish]

We see this being played out by Sharpe, Crawley et al. describing Tuller and others of harassment for picking up the errors (deliberate or otherwise) in their work. Perhaps the Vice Chancellors of Bristol and Oxford Universities should be awarding prizes to Tuller, Wilshire, Hughes, Kindlon and others who have pointed out the major flaws in their employee's work.

 

[Barry]

The crucial thing glossed over is that science often has a huge responsibility to the wider public. Abusing that responsibility is deeply unethical at the very least.

 

[dave30th]

We see this being played out by Sharpe, Crawley et al. describing Tuller and others of harassment for picking up the errors (deliberate or otherwise) in their work. Perhaps the Vice Chancellors of Bristol and Oxford Universities should be awarding prizes to Tuller, Wilshire, Hughes, Kindlon and others who have pointed out the major flaws in their employee's work.

I see she is at UC Davis. I will touch base with her and introduce myself.

 

[Kitty]

science often has a huge responsibility to the wider public

Indeed, and we're seeing the consequences in appallingly sharp relief. Decades on, so many people still think vaccines cause autism – how many children have died, or suffered life-changing damage from encephalitis, because they weren't vaccinated against measles?

 

[alktipping]

the phrase that says science is self correcting should also contain the words eventually after the retirement or deaths of the eminent scientist who provided the original work . but saying that all knowledge starts out flawed often because the wrong questions are being asked ,or the question being asked are motivated by political or financial institutions.

 

[InfiniteRubix]

Great thread

 

[Sly Saint]