The PACE trial video series by A Broken Battery

https://twitter.com/user/status/1156532610107097089

 

Ha! thanks @alktipping

 

Agreed. They are allowed to go unchallenged far too often.

 

Have posted a transcript at the beginning of the thread.

 

The line about recovery having multiple dimensions has a certain lack of logic to it. The point is that all four criteria have to be matched. If we just correct one of those criteria, the sf-36, and use a more appropriate target, the recovery figure drops dramatically down from 22% (I haven't done the analysis!).

My car's engine needs fuel, sparks, coolant and oil to function. If there are no sparks at the spark plugs, it is irrelevant that there are three more criteria in order for it to run.

So, correcting this one issue has a dramatic effect on the final result, without any need to raise any objection to the other criteria.

That fact demonstrates very clearly that the other criteria are relatively unimportant, or are similarly weakened.

 

Very impressive video, Andy. Succinct and precise.

 

I think you meant Adam...

 

It must depend surely on what importance each component has to the overall result, and how each contributes to it. The authors clearly presume there is a simple addition involved, else they could not be making such a statement. But given how important every component is, it feels to me that cannot be so, because if any one support falls then it all falls. Feels as if each component must have some multiplicative factor, possibly as well as an additive component. So if any one component drops to zero, then the whole (multiplicative aspect of the) result would drop to zero. Logically it's more like the contributions are in series, like a chain - any one link breaking and it is all broken.

In effect there must be a transfer function of some kind, where the individual components are the inputs to it, and outcome the output. It cannot, surely, be a simple additive transfer function, and the individual components not necessarily evenly weighted. Surely that transfer function should be pre-specified before a trial starts, or most certainly before awareness of outcomes is possible.

And any peer review should look at some "what if" test cases, to see if they make sense. The idea that someone could recover without any improvement in physical function (let alone if they get worse) should shout volumes that the transfer function has to be flawed.

I'm sure all you maths buffs will have a much better handle on this.

 

Psychology seems to assume that data is linear.

 

I always reckon I'm lucky to get the first letter right. Please accept my apologies Annie, and my congratulations to Alice.

(and no, it isn't the ME: I was this bad when I was teaching!)

All four criteria had to be satisfied. In set theory we would call this the intersection of the four groups: those members that satisfy all four criteria. As you correctly suggest, it is more akin to multiplication, but a better visualization is a sifting process. If all four sieves have large holes, a lot of stuff gets through, but it only needs one to be a fine sieve for very little to pass.

Another way of thinking about it is to think that each criteria subtracts from the pool of patients. So if we tighten up the first criteria and bring the percentage way down below 22%, further subtractions are pretty irrelevant.

It is obvious that originally they would have expected lots of people to pass each of the tests.

Perhaps I should go back to the data and see what effect this one basic change would have. I feel a video coming on. A short one. Does anyone have a link to one of the "rebuttals" that suggests that flaws with this one measure are not too important because there are three more criteria?

 

Thanks Graham, that's a great explanation.

As I was reading about sieves and patients my brain sort of cut out the middleman, and I found myself visualising it as patients falling through the sieves, with each sieve being sensitive to one of the criteria. Recovered patients have to make it through all four sieves into the "recovery bin" below. If patients are too 'big' (i.e. too unrecovered) on any criteria, they get stuck in that sieve. With PACE it's as if they found too many people were getting stuck in the Physical Function sieve, so they ... increased the size of the holes in that sieve, allowing them to drop through again.

Something seems to be full of holes.

 

Don't forget that they actually loosened all four sieves. It might be that they found that all four of them were too restrictive.

I think I would like to play around with the data and compare the severity of each sieve with each other, and with the original settings.

But the important argument is simply that changing the sf-36 target to 60 cannot be justified: it needs to be corrected. If so, the claim of 22% recovery is demolished. End of discussion. How strict the other sieves are is more of an intellectual but pointless challenge – it sounds right up my street.

I'm trying to think of a suitable analogy. How about advertising a clerical job? Applicants must have at least a grade C in Maths, at least a grade B in English, and to have had at least two years experience in a similar position? (Yes, I know the grading system has changed in the UK, but I'm old school?)

Dropping the requirement in Maths to a grade E or more makes that much easier to achieve, but the other two are still pretty restrictive, so the potential pool of those who pass all three won't increase that much.

If you want to increase the total pool of applicants, you have to make all three criteria much easier to attain. To my mind, that is what they did, asking for at least a grade E in Maths, and in English, and at least a month's experience in any office work. That is when changing the Maths requirement back to a C has such a big effect.

This might be a way of demonstrating that the other criteria were similarly weak.

I need a better analogy. Any thoughts?

 

Graham. I wish you had been my maths teacher. You have a wonderful talent for explaining things simply and coherently.

 

Table 1(b) of the recovery paper (Psychological Medicine (2013), 43, 2227–2235) shows that the transfer function used was the AND function - CFQ in range AND SF36 in range AND Oxford not met AND CGI 1 or 2. So a sieve, like Graham said.

 

Bearing in mind the grades were all self-reported anyway, and therefore liable to be pretty slack, it may be that as awareness began to dawn of how the outcomes were likely to pan out, maybe one in particular proved more 'stubborn'; outcomes perhaps more inclined toward the objective criterion it was purporting to represent - PF. (I don't know this! Just a thought). And if so then maybe spotted that messing around with the pass mark for this alone would make a big overall difference (except they had to mess with it to the point of absurdity). And then just fiddled a bit with the other criteria as well.

I refuse to believe they did not have strong awareness of how results were likely to pan out, and might therefore have undertaken a reverse engineering process to fix their issues.

 

Perfect! Any one dropping out drops the whole thing to Fail.

But wasn't something tweaked in this area, compared to the original? I seem to recall the way things were AND'ed introduced some slackness, compared to the original definition? It was subtle but there. @dave30th maybe reported it? Maybe in the reanalysis. Not got time to go hunting for it at the moment.

ETA: But of course the AND'ing is applied after the individual criterion pass/fail have been assessed, and effectively then become binary values in order to then AND them.

ETA2: But the AND'ing shows the invalidity of MS and Co. claiming that PF was only one of the criteria that had to be met, and therefore not a big deal to mess with. They all had to be met if the AND function is to be satisfied. But I still have a niggle there was something else ... ?

ETA3: (Sorry). When I say above "the way things were AND'ed", by that I mean that I think one of the AND clauses had a new OR introduced within it, with some weaker criteria on one side of the OR.

 

Thank you, but you do realize that after a year of being in my class your brain would have been warped forever.

 

Would it really have taken that long?

 

Does anyone have any suggestions of a totally different "medical" study where three or four criteria were used to determine recovery, or some similar concept?

 

Do you mean a real one that was previously done? Or a realistic hypothetical one?