Belief in the law of small numbers

[TiredSam]

For those who like this sort of thing.

People have erroneous intuitions about the laws of chance. In particular, they regard a sample randomly drawn from a population as highly representative, that is, similar to the population in all essential characteristics. The prevalence of the belief and its unfortunate consequences for psychological research are illustrated by the responses of professional psychologists to a questionnaire concerning research decisions.

Psychological Bulletin 1971. Couldn't find a cleaner copy than this:

http://stats.org.uk/statistical-inference/TverskyKahneman1971.pdf

 

[Woolie]

I like this sort of thing

Its a really famous paper, and still THE go-to source to cite when you're concerned about studies making claims based on small samples.

Though this is not the biggest problem afflicting psyc research into ME. The biggest problem is systematic bias. I reckon you could run the PACE trial a hundred times and although it wouldn't reach p < .05 every time, overall there'd be a significant outcome. Its just that outcome is not meaningful because of the bias inherent in the design.

 

[Alvin]

I like this sort of thing

Its a really famous paper, and still THE go-to source to cite when you're concerned about studies making claims based on small samples.

Though this is not the biggest problem afflicting psyc research into ME. The biggest problem is systematic bias. I reckon you could run the PACE trial a hundred times and although it wouldn't reach p < .05 every time, overall there'd be a significant outcome. Its just that outcome is not meaningful because of the bias inherent in the design.

Many if not most people seem to think statistically significant means correct. This is easily exploitable as you say, do enough trials you can cherry pick the one you like and voila you have "proved" lies.
There was a Dilbert episode where he is forced to prove Chronic Cubicle Syndrome exists and goes to his company's statistical distortion department, whose employee says no need to prove it, everyone knows it exists since its talked about on tv and in national discourse.
Perhaps the biggest thing modern science has to weed out is the people who are not interested in the truth but want to use science as a screen for harming others or legitimizing lies.

 

[Little Bluestem]

There was a Dilbert episode where he is forced to prove Chronic Cubicle Syndrome exists

I hope they were not making fun of Chronic Fatigue Syndrome.

 

[Alvin]

I hope they were not making fun of Chronic Fatigue Syndrome.

They were not, Dogbert invented a new condition to show how gullible the public is to media driven lies (eerily accurate for being written almost 20 years ago)

 

[Snow Leopard]

The most notable part of this is this bias is still common amongst scientists today.