The trial endpoint issue (135 weeks)

Hutan

Hutan

Moderator
Staff member
I thought it might be useful for us to get to grips with this issue.

The GET defenders have said that NICE chose the trial endpoints it looked at incorrectly. As I understand it, NICE chose the longest followup points where possible, and in the case of PACE, that was 135 weeks. The GET defenders have disagreed with that choice, saying the planned trial endpoint should have been used - at the one year mark.

In favour of the NICE decision for the longest possible time for the endpoint is that we might expect the CBT and GET treatments will be a one-off treatment if their underlying hypothesis is correct. That is, if they are fixing deconditioning and a fear related to activity, then, once they are fixed, those problems should not recur. Whereas, if all the treatments are doing are convincing people to fill out questionnaires more favourably, then that effect should wear off with time away from the therapist.

But, in favour of the BPS view that a shorter time period should have been used is that participants are lost to followup, and so statistical significance can be unfairly lost. Also, after a trial ends, participants will try other treatments. People in the APT arm might try GET; or people in the GET arm might try CBT. The PACE trial actively encouraged this. So, at followup, it can be quite hard to classify what treatments each participant have had.

So, do the GET defenders have a point? Did NICE make an error in its selection of time point data?


I've heard it said that it wouldn't have mattered if the 1 year endpoint had been used - GET and CBT still weren't significantly better than the SMC and APT treatment arms at that time. They still weren't cost-effective treatments. I can't remember if that is true myself, but I expect it is.

Also, I remember doing an analysis of the impact of total numbers of CBT and GET sessions at the PACE trial followup quite a few years ago now. There is data at followup to allow that to be done. So, ignoring when participants did the GET or CBT sessions (before or after the end of the trial, but before the followup point), if the treatments worked, we'd expect to see more improvement in those people who had done most of a full course of CBT or GET instruction, rather than zero or just a few sessions. I'd have to find the analysis to be sure what I looked at, but there was no pattern of improvement related to some threshold of number of sessions at followup.

Maybe people who know the detail of the NICE approach and reasoning, and the outcomes at the 1 year endpoint can add more?
 
My brain is tired after a too-busy day yesterday, so I might not be capable of logic.

But it struck me, reading your post, that the only reason for outcomes being worse at 135 weeks than at 52 (assuming that there are no confounding factors such as other illnesses or injuries in the meantime) would be that:

  • The treatments don't lead to long term improvements in ME.

  • The treatments (whether only one is used or a mix of them) are capable of making people worse than they were at the outset.
 
Reacting as a layperson: Don't they expect to have extra good results then if participants have done GET followed by CBT?

In the example Michiel Tack gave me this week it concerned the control group.

I thought: what a nice loophole, offering your control group the tried out therapy (CBT), and then saying you can't do long term follow ups because everyone has done it.


It takes skills to set up a good trial. Solving these claimed problems of long term follow-up are the trial-designers' to solve, not a reason to lower quality standards or to approve therapies for patients when the long term follow up results are either negative or unknown.
That's also an important point: this is about guidelines, wheter or not to advise treatment for a patient. In this case a treatment that not only has a lot of anecdotal evidence of causing harm but also more and more emerging physiological finds that explain why that treatment is causing harm/not working. Just saying a treatment should be upgraded to be considered acceptable and evidenced in patient care because you cannot supply reliable long-term effects doesn't cut it.

(I haven't read the guideline comments yet, but I know that at least the Royal College of Psychiatrists discussed this elaborately, and NICE answered them; NICE must have elaborately thought about this.)
 
Last edited:
This is the long term follow up (LTFU) paper for the PACE trial
https://www.thelancet.com/journals/lanpsy/article/PIIS2215-0366(15)00317-X/fulltext

Only about 75% of trial participants provided LTFU data.

Basically the small between group differences at 12 months had vanished at LTFU, with no longer any significant between group differences. The researchers tried to explain this away by saying that's because the ones in the APT and SMC groups had GET or CBT after the 12 months, allowing them to catch up. That is not borne out by the data, as they themselves admit. See below.

Within-group long-term outcomes are shown in figure 2, table 3, and online (appendix pp 3–4). The improvements in fatigue and physical functioning reported by participants allocated to CBT or GET at their 1 year trial outcome assessment were sustained. The improvements in impairment in daily activities and in perceived change in overall health seen at 1 year with these treatments were also sustained for those who received GET and CBT (appendix p 4). Participants originally allocated to APT reported further improvements in fatigue, physical functioning, and impairment in daily activities from the 1 year trial outcome assessment to long-term follow-up, as did those allocated to SMC alone (who also reported further improvements in perceived change in overall health; figure 2; table 3; appendix p 4). In an exploratory post-hoc subgroup analysis requested by a reviewer, the within-group improvements were observed whether participants had received additional CBT or GET or not; however, this finding must be interpreted with great caution, as we discuss later.
Click to expand...

If you look at the data and graphs in the paper, it's clear that all groups did about the same, barring some therapist effect in the first 12 months. The between group differences can, in my opinion, be easily explained by participants being persuaded to fill in questionnaires differently by being given different stories about what is wrong with them and how they should view their symptoms. Also by the CBT/GET ones being told they are the ones having the already proven successful treatments.

It's also important to note from the SF-36 results that the vast majority of patients still scored 60 or less, so were still sick enough to enter the trial.

But of course there were no between group differences on any objective measure even at 12 months, so the trial showed definitively that the treatments don't work.
 
Discussing these finer details is unnecessary because we already know with certainty that subjective outcomes in unblinded clinical trials are not reliable.

I always think of that one paper of albuterol and sham treatments for asthma. The subjective outcomes for all treatments were good but only the real medication actually improved the ability to breathe. The subejctive outcomes aren't measuring treatment effectiveness but other contectual things like expectations and satisfaction and how well dressed and nice the therapists are.

In PACE where the treatment wants to fix presumed false beliefs to be ill it becomes even more absurd to afford any credibility to the trial. Further analysis of trial problems is only of interest for showing how the authors continuously try to create the illusion of treatment effectiveness with various tricks.
 
Last edited:
The problem they are complaining about is that the people in the control groups have improved because they have now tried the effective treatments.

I seem to remember that for PACE a small graph showed that participants who tried GET after the trial ended had worse outcomes than ones who did not.

But that aside, their treatments are meant to be tools you continue to apply. That is the rational for short courses with no follow up, carry on doing it and you will be cured. With MS, patients have a yearly appointment to see how they are doing. If the BPS do not believe that patients will continue to improve they are very remiss in just abandoning patients.

So taking the longest term of a study should show a massive improvement in the figures which that also applies to controls who have now been treated. If the figures had shown that they could have argued statistics, but they do not.

Their complaints are just a misdirection from the failure of their treatments that shows up in trials.
 
I think the justification for using the longest followups possible from NICE was that that's what people are actually interested in. I agree with the points made that it's a bit of an unusual thing to do and has the issues which Hutan explained. If, say, they did do the analysis at 52 weeks and found good evidence for a large effect, that would probably justify recommending GET even if that effect didn't persist long term.

So, in principle I agree that they should have analysed the shorter term outcomes too. But, in reality, they would have known that those shorter term outcomes weren't going to have strong evidence or large effect sizes. If there was some moderate quality evidence (but there wouldn't have been, as they would have downgraded for indirectness and imprecision) for some small to medium effect, it still wouldn't have been recommended, because a short term small effect isn't what patients are interested in, isn't cost effective, and doesn't outweigh the wide reports and uncertainty around harm.
 
@strategist I agree with you. All this messing about with indirectness, which end point is used etc is simply a diversion from the fact that unblinded trials with subjective outcome measures are useless for showing effectiveness. The conclusion crying out to be drawn from PACE is that it actually shows clearly that the treatments DON'T work.
 
I think the basic issue here is that there is a compromise between making the most rational assessment you can and trying to protect the process from cherry picking. NICE have predetermined policies for the latter, which is good in many ways, although it will lead to blunted decision-making.

One policy is to take the last set of results in time. For PACE that may or may not be the most relevant. But at least it stops the sort of cherry picking that the criticism illustrates.

NICE could have decided to just take the primary outcome measure, as an indicator of a pre-set policy to avoid cherry-picking but they didn't. So to shift to the PACE primary outcome would have been cherry-picking.

If the argument is that the most rational assessment trumps rules then the most rational assessment is that PACE is not going to give us a reliable positive result anyway.
 
Kitty said:
My brain is tired after a too-busy day yesterday, so I might not be capable of logic.

But it struck me, reading your post, that the only reason for outcomes being worse at 135 weeks than at 52 (assuming that there are no confounding factors such as other illnesses or injuries in the meantime) would be that:

  • The treatments don't lead to long term improvements in ME.

  • The treatments (whether only one is used or a mix of them) are capable of making people worse than they were at the outset.
Click to expand...
And surely we still dont have enough studies on the progression of ME generally, do we even know if there are remission relapse phases and gradual deterioration later on as often happens in MS?
 
AknaMontes said:
And surely we still dont have enough studies on the progression of ME generally, do we even know if there are remission relapse phases and gradual deterioration later on as often happens in MS?
Click to expand...

No – as far as I know, it's all anecdotal and people's own experience.

For instance, I'm probably one of the lucky ones in having had good remissions as well as the relapses, but no evidence of gradual deterioration over the 45 years up to now (other than natural ageing). Other ME long haulers report very different outcomes.

(Probably also lucky in having got ill far too long ago to have been subjected to the BPS nonsense too!)
 
Ramsay was involved with the patients from the Royal Free epidemic and then as advisor to the ME Association so he followed quite a few people for decades. He decided that 25% stayed much the same or improved, 50% had a variable course and 25% gradually worsened to become severe.

It would be interesting for the views of Dr Speight who has been treating patients for a long time too.

Dr David Bell from the Lyndonville outbreak is a small town doctor (basically a GP, I suppose) so he has stayed in touch with the victims. He wrote a paper where he said some of them did not think themselves ill anymore but he found that their OI had improved and they had accepted the limitations of ME as normal.

You could probably try following up patients from the early days of the clinics as it has been a while now, or take very detailed histories from patients.
 
You must log in or register to reply here.
Back
Top Bottom