A general thread on the PACE trial!

In this case it meant that QMUL was able to completely evade the real concerns about access to PACE data, and present Monaghan's concerns as unfounded. I think that getting the details right at this stage is still going to be very important.

That was recordings of therapy sessions.

 

FWIW
https://twitter.com/user/status/1009576713607700480

 

QMUL are not claiming the data is lost, they're claiming a much more sophisticated version of dog-ate-my-data defence.

They're claiming that the statisticians have moved on and it is impossible for them to get another statistician to access the data. Yes, really.

 

Which really means it will be impossible for us to salvage our reputation if we give this to another statistician who doesn't cook the books like Peter White.

 

I may have found a new hole in the PACE trial. I have been looking into the theory of statistics, even though I cannot do math myself any more.

I don't know if this has already been looked at.


Preamble

We already know that there was no significant difference at long term follow-up.

We already know they used the wrong data set for SF36PF in calculating SD and normal.

We already know that SD is undefined for this data distribution, and that PDW knew it was a biased calculation. This is a deliberate use of a biased calculation.

P value is based on a calculation to assess risk that the result is due to chance. This does not take into account risk of confounds nor risk due to biased methodology.


Argument

Calculation of p value is changed if there are multiple outcome measures and you can choose between them. In my personal opinion its a simplistic version of p-hacking if you do not make adjustments.

My source for this is S. Nassir Ghaemi's "A Clinicians Guide to Statistics and Epidemiology in Mental Health".

This makes sense when you consider p value as a probability calculation.

The Bonferroni correction is an imprecise first-pass correction. If for example there are ten outcome measures then you can correct by dividing the p value for significance by 10. This would put significance at not 0.05 but 0.005. Simply divide the 0.05 significance value by the number of measures.

There are other corrections such as the Tukey test but I know nothing about them.

As the p value for significance changes, so too does the estimation that the result is due to chance. Again, this does not correct for confounds nor bias.

This is complicated by how to treat the dropping of activity data for patients.


Action

@Carolyn Wilshire @Graham @Dolphin @dave30th @Jonathan Edwards

I am hoping someone who can still do math can look into this if it has not already been examined. Alternatively it would be good to give this to some statisticians and ask what they think. I am content to be wrong, but this needs to be examined if it has not been examined already.

Their likely counter-argument is that the alternatives are secondary outcome measures.

PS The Bonferroni correction is imprecise because I think it presumes that all measures have equal relevance. This is part of the primary versus secondary outcome measure argument.

 

I don't think this is exactly right. They're claiming that even if they could get a statistician, which they obviously could, they shouldn't be required to go beyond what other departments would have to in order to comply, and most government units would presumably not have extra statisticians around. So to require the university to find someone else would be putting an extra burden on them under the law. That's the argument as I understood it, but I could be wrong.

 

Sorry Alex, you over-estimate my statistical skills. At A-level there are, as you probably know, three areas (just as in Science there is Physics, Chemistry and Biology). My preferences are Mechanics and Pure, with Statistics/Decision maths limping along behind.

But I do have a concern about spending time on further detailed analysis of their errors. Their decision to ignore objective assessments and focus only on the subjective assessments that they could manipulate is, to me, enough to blow their work out of the water (back to the SS PACE analogy!). Showing that they fiddled the targets and used utterly inappropriate methods to justify it simply shows that they are either utterly incompetent or totally devious. I don't think we need to grace their sophisticated analyses with any more effort: it could distract from the key message.

 

I have an MSc in Medical Statistics, if that helps. There are worse problems with the PACE trial than their significance testing. The main issue is that the although the result was statistically significant, it wasn't clinically significant. It's not about p values per se - differences in the effect estimates are more important.

But by far the biggest error is the use of self-reported subjective outcomes that can be easily biased by the intervention (re-education of how to complete a questionnaire, as @Graham has recently demonstrated). Significance is then the least of their problems.

 

Agreed. I only brought it up because I do not recall it being discussed, and its yet one more problem. The point here is that if this applies then it was not even statistically significant, even before you get to clinical signifance. There are a huge number of problems in this trial.

Recently I have been very concerned about participants reporting that their comments on being made worse were not even recorded. I heard the first one of those during the trial period. I am also keen on pursuing the deliberate misuse of SD. Most other things can be considered due to methodological problems that are common in psychiatry, or ineptitude, or some other argument. The misuse of SD shows deliberate use of biased methods. That deliberate use raises the question of scientific misconduct that I think needs to be pursued.

Significance is not as important as the methodological failings, as with confounds and bias the p value is distorted anyway. Yet the list keeps growing longer, and we have not looked hard at it from a political or social aspect, we have focused on the science in which we have stronger evidence. Yet in dealing with politicians and bureaucrats I think its worth looking harder at political issues than we have so far.

 

Yes, I think this is a good point. It's the reason why clinical trials are supposed to have a single primary outcome measure. So that you can't take bites out of multiple cherries and then pick the juiciest one. Technically, your primary outcome should be the "gate keeper", so you only proceed to test secondary outcomes if that's significant. If researchers approach it in this way, then there is no need to correct; by requiring a p value of less than .05 as a pre-requisite, you've effectively controlled type I error not only for the primary outcome, but you've also reduced it massively for the secondary outcomes.

Although in reality, researchers will often report any secondary measures that were significant, even when the primary one wasn't, and as you say, this should be corrected in some way to account for the increased opportunities you had to get a significant result.

Of course, all this assumes that you stick to your pre-registered primary outcome - you don't switch outcomes like the PACE researchers did.

There is now a wide consensus that the bonferroni is overly conservative. This became clear when we started doing fMRI where you might need to do hundreds of thousands of tests, one for each "voxel" (minimal volume unit in the brain). You can see that using the Bonferroni correction there would preclude ever finding anything significant without massive, huge power, because you'd need p to be less than something like .000005!

Tukey is still used in practice by some researchers, but the contemporary view in the psychological research methods community is to use False Discovery rate (FDR), or if you're doing a lot of comparisons, Permutation thresholding.

Some consider FDR to be overly lax, so there is a preference now in the neurosciences for permutation thresholding. Although I consider myself maths-minded, I admit to not knowing a lot about the stats behind either of these. I think I once read up on FDR, but have forgotten all I ever learned.

Another consideration is also the purpose of the research. Correcting for multiple testing is all about striking a balance between type I errors (falsely reported an effect when it isn't there) and type II errors (failing to report an effect that is genuinely there). If the purpose of the the study is to advance knowledge, you might be willing to tolerate a bit more type 1 error so as not to miss any real genuine effects that might yield new avenues of enquiry. But if your study is to provide evidence for the efficacy of an expensive and/or risky treatment, you might want to be extra sure you have a genuine effect. In this instance, using the Bonferroni correction, although stringent, might be wise, because then you can say "There's really virtually no chance this effect is spurious".

Yes, corrections for multiple comparisons are usually only employed when there are multiple comparisons done on the same dependent variable, but as you say, this is wrong, if you measure, say 20 variables, one of your comparisons is likely to come out significant by chance alone, where there is genuinely no effect. My feeling is that some sort of correction needs to be done in this situation.

In the situation where different outcome measures have different statuses (primary vs. secondary), if you want to report secondary outcome measures even when the primary outcome measure yielded no significant effect, then you should definitely correct in some way, but I'm not exactly sure off the top of my head just how.

 

For anyone who just glazes over when Type I and Type II errors are mentioned, a good way to remember the difference is the story of the boy who cried wolf, because the first error (reacting to a cry of "wolf" when there was actually no wolf) is a Type I error and the second error (ignoring the boy's cry when it was actually true) is a Type II. So first error is Type one, second error is Type II.

 

In my brain fogged and amateurish way I thought that the "lost" data bit referred to researchers wanting the objective measures data for reanalysis.
What happened to those results?

Please point me to the right place if I've missed this.

 

This gives an overview of the events that lead to the release of part of the data from PACE, http://me-pedia.org/wiki/PACE_trial#Release_of_Data - note that the complete trial data wasn't requested, and therefore only what was requested was released. If you read into the following section it gives links to the reanalysis itself.

 

I don't doubt that what they did was statistically significant, but if what they did was incorrect and biased, then it's how that result is interpreted that's important. No amount of reanalysis or fancy statistical adjustment is going to make any meaningful difference to that.

 

Brilliant.

When I'm not distracted by THE Westminster Hall debate (which I feel I can watch having read the thread on S4ME = brilliant too) I will certainly have a look at this.

Thank you for pointing me in the right direction.