Blog: Want to Know If Someone Is Manipulating Data? - Milton Packer describes how to distinguish science from magic

[Andy]

Why am I talking about magicians in a blog devoted to medicine?

Two weeks ago, I wrote a post about my experiences as a principal investigator in large-scale clinical trials. Several readers thought that my personal experiences did not represent the norm. Many thought that clinical trial data are commonly manipulated in order to put them in the best possible light. I had to acknowledge that their concerns were valid.

A respected friend suggested that I devote a post to describing how someone might manipulate data in order to make a negative trial look like a positive one. My challenge: how could I possibly describe it in a blog?

Soon the answer became obvious. Deception of the audience in presenting a clinical trial is based on the same strategy of misdirection that magicians use to make their performances work.

https://www.medpagetoday.com/blogs/revolutionandrevelation/78239

 

[large donner]

Surely we already know the WHOLE WORLD is a stage?

 

[Arnie Pye]

For anyone interested in how data from trials can be manipulated and distorted, this book is brilliant and well worth the effort of reading it :

Amazon product ASIN B00TCG3X4S

 

[Barry]

First and most important is the trick of missingness. The best way to make data look better is to take out data that you do not like or not bother to collect it at all. If the presentation does not account for missing data, all sorts of mischief are possible.
...
Second is the trick of not showing a planned analysis, or alternatively, showing an analysis that was not planned.
...
To be clear, these are not the only two tricks that people can play with the data from a clinical trial. But they cover a lot of ground.

How familiar that sounds.

 

[Hip]

When I saw the title of this thread, it brought to mind the fascinating Benford's law, which states that in any genuine set of data, if you count up all the numbers, you should find around 30% of those numbers being with a "1", and not around 11% as you might expect.

However, in a fake set of data, you will get around 11%. So that's one way to detect whether accountants or researchers have faked their figures.

 

[BruceInOz]

However, in a fake set of data, you will get around 11%. So that's one way to detect whether accountants or researchers have faked their figures.

Not always. From the wikipedia page

Benford's law tends to apply most accurately to data that span several orders of magnitude. As a rule of thumb, the more orders of magnitude that the data evenly covers, the more accurately Benford's law applies. For instance, one can expect that Benford's law would apply to a list of numbers representing the populations of UK settlements. But if a "settlement" is defined as a village with population between 300 and 999, then Benford's law will not apply.

So it depends on the nature of the data your dodgy accountants/researchers are presenting.

 

[Hip]

Not always.

Sure, but when the data falls into a normal distribution, which a lot of real-life data does, then Benford's law applies.

Of course if you are a dodgy accountant who knows about Benford's law, then you can actually generate fake data (ie, have 30% of numbers start with 1) which will pass the Benford test.

 

[BruceInOz]

Sure, but when the data falls into a normal distribution, which a lot of real-life data does, then Benford's law applies.

Wikipedia (and that is the limit of my knowledge about this) seems to be saying that it is not whether or not the distribution is normal but how wide it is. If it's width covers several orders of magnitude, Benford's law applies, but if it is narrow so only covers one order of magnitude then it doesn't. An example given of a distribution where it does not apply is human heights, which is a normal distribution.

For example, if human heights are measured in feet, heights that start with a "1" are likely to be rather rare, definitely not 30%.

 

[Hip]

Wikipedia (and that is the limit of my knowledge about this) seems to be saying that it is not whether or not the distribution is normal but how wide it is. If it's width covers several orders of magnitude, Benford's law applies, but if it is narrow so only covers one order of magnitude then it doesn't.

That makes more sense.

From memory, I thought Benford's law would always apply when you have a normal distribution, but maybe I got that wrong (it was decades ago that I read about Benford's law).