How can we create a patient-led group to influence individual trial designs before it's too late to change them?

[Liface]

The recent reactions to the poorly designed Charité LDA trial got me thinking about a common failure mode in trial critique in the ME community.

At the moment, we often first learn about a trial when it appears on ClinicalTrials.gov and predictably we're immediately outraged at yet another poorly-designed trial.

But by that point, funding may have been awarded, the protocol written and ethics approval in process, and the design might be hard or impossible to change.

I am a "just do things" type of guy, but I must admit my ignorance about the clinical trial process. So before taking action, a few questions / ideas:

- At what point does a trial design become locked? Is our best opportunity to intervene during the grant application, after funding but before ethics submission, or somewhere else?
- How do we stay informed early? Monitor signals like as grant awards, conference presentations, pilot studies and university announcements?
- What about creating a list of researchers and groups most likely to run ME trials and contacting them proactively, representing ourselves as a small patient working group offering rapid feedback on trial design while protocols are still being developed?

The approach would need to be constructive, for sure. I'm imagining the initial messaging as an offer of useful help and not outrage. But... there should also be some accountability if researchers decline or ignore early involvement and later produce trials with predictable problems.

Perhaps researchers could also agree to a few commitments, such as involving several patients before finalizing the protocol and explaining which major recommendations were accepted or rejected?

Anyway, the basic idea would be to create a standing group that identifies trials early and gives researchers access to serious patient input before poor design choices become expensive to change.

Does anything like this already exist or... should we just start it

 

[BrightCandle]

The hard pill to swallow is that a lot of these trial designs are bad on purpose. Its not that they don't know there is a better way, most of these people have had training on trail bias or have plenty of experience to pull on, they don't need us to point out the flaws. They have resisted attempts by patients to adjust trial design to something more likely to produce quality results for a long time. They will give all manner of excuses but I think its that the pressure on them is to produce a positive result or they wont get future funding so they are just playing the game that is front of them.

 

[Liface]

The hard pill to swallow is that a lot of these trial designs are bad on purpose. Its not that they don't know there is a better way, most of these people have had training on trail bias or have plenty of experience to pull on, they don't need us to point out the flaws. They have resisted attempts by patients to adjust trial design to something more likely to produce quality results for a long time. They will give all manner of excuses but I think its that the pressure on them is to produce a positive result or they wont get future funding so they are just playing the game that is front of them.

In conflict vs. mistake culture, my experience has me firmly on the side of mistake culture.

Everyone wants to help, and few differences are insurmountable. It's about putting in the work and approaching researchers in the right way.

 

[Yann04]

At what point does a trial design become locked? Is our best opportunity to intervene during the grant application, after funding but before ethics submission, or somewhere else?

There will be many answers to this. But as PACE showed, analysis protocol can technically be changed quite late (though it probably shouldn’t be).

 

[Liface]

There will be many answers to this. But as PACE showed, analysis protocol can technically be changed quite late (though it probably shouldn’t be).

What are the main factors that influence how late the protocol can be modified?

 

[fst]

The hard pill to swallow is that a lot of these trial designs are bad on purpose. Its not that they don't know there is a better way, most of these people have had training on trail bias or have plenty of experience to pull on, they don't need us to point out the flaws. They have resisted attempts by patients to adjust trial design to something more likely to produce quality results for a long time. They will give all manner of excuses but I think its that the pressure on them is to produce a positive result or they wont get future funding so they are just playing the game that is front of them.

I don't think that's true per se. Not blinding your trial? Maybe that's the pressure for positive results (or lack of funding).

But choosing bad endpoints such as the Chalder Fatigue Scale is more likely to give you a negative result. Why would you still choose it?

Edit:
Is this maybe pressure from reviewers who want to see a "tried and tested" endpoint?
It's always hard to deviate from the norm and get stuff published. Or is it maybe not knowing better?

 

[Utsikt]

I think that’s what’s currently happening with WE&ME and WWTF.

 

[fst]

Also, I remember seeing Scheibenbogens CD19 trial was initially rejected for funding. One reason was an "unjustified endpoint" (SF-36).

Do these bad endpoints already emerge in the grant application phase?

 

[NelliePledge]

I think that’s what’s currently happening with WE&ME and WWTF.

Hopefully a paper about patient participation will be part of their planning and that can then be used to promote better practice. Here’s what some other people do is going to be an easier message to get through to researchers than you’re doing it all wrong.

Decode folks did a paper about their PPI approach didn’t they, the more it goes on record as a model the better.

 

[Liface]

I think that’s what’s currently happening with WE&ME and WWTF.

WE&ME is an incredible start, and should be the template moving forward, at least for trial design.

They also have the advantage that they control the funding, and thus have all the power to set the requirements.

However, they are yet a small speck in our overall landscape: the initial outlay will fund only 7 projects.

What I'm imagining is something much wider-reaching, but unrestricted: appealing to the researcher's goodwill. Yes, many will ignore or say no, but if even a dozen stop and listen, that's a dozen or more studies that will come out better designed per year.

 

[Hoopoe]

Practically, we could demand that funding be preferentially given to clinical trials who involve a trusted patient representative.

Or something like that.

 

[Utsikt]

WE&ME is an incredible start, and should be the template moving forward, at least for trial design.

They also have the advantage that they control the funding, and thus have all the power to set the requirements.

However, they are yet a small speck in our overall landscape: the initial outlay will fund only 7 projects.

What I'm imagining is something much wider-reaching, but unrestricted: appealing to the researcher's goodwill. Yes, many will ignore or say no, but if even a dozen stop and listen, that's a dozen or more studies that will come out better designed per year.

I sincerely doubt that appealing to their goodwill will achieve anything.

They need incentives, either financial like WE&ME and WWTF, or a higher chance of being successful, which is what WE&ME, WWTF, DecodeME, SequenceME, the age of onset study, and others hopefully demonstrate.

Of course some researchers won’t care because they already think they know what they need to know, and can’t be bothered dealing with all of the negative patients that ask questions and don’t accept their answers.. If that’s the case, I don’t think they have what it takes to make a difference anyways.

 

[rvallee]

As long as it's done by humans with human biases and interests in mind (theirs, not ours), I don't think we can ever influence things to improve. Some of it might be somewhat subconscious, but the flaws are still all intentional. They're just "the way things are done", and combined with the extreme inefficiency of the entire process (years and millions of dollars despite very low quality, or less expensive but just as slow for even lower quality), everything is aligned against producing good outcomes. Getting ambiguous answers to questions taking years is just not good enough. No process can achieve this.

It's only when AIs run the loop that things can be improved, when they can at least 10x speed, cost and quality. I think it's just too early, the concept of clinical trials is not feasible on this limited scale. Like trying to do video streaming on dial-up internet, it's technically feasible, the underlying technology just needs to be much, much faster, and better (which is mostly a function of faster). Clinical trials are slow, they're massively expensive, mostly for no actual reasons, and the only way to keep the loop going is to keep the quality as low as possible. They are likely the least efficient expert process ever created to produce answers. It's pretty much the joke behind 42 as the answer to everything. A process that is too slow to produce useful answers is simply not useful.

Good, high quality trials will be asynchronous, they will have continuous recruitment on validated cohorts, instead of the constant churn of single-use cohorts. They will take weeks to publish, not years, at 1/100th the cost, and they will analyze 100-1000x more data. It's technically already feasible, but not as long as humans are in charge of the loop, because the people who run the loop will always maximize their influence on the loop over the quality and relevance of its output.

 

[rvallee]

Practically, we could demand that funding be preferentially given to clinical trials who involve a trusted patient representative.

Or something like that.

It's far too easy to select representatives for a specific outcome. It's pretty much common already. Same with trial participants. It's effectively been normalized, everyone just pretends it's not.

 

[Jonathan Edwards]

I agree with @Utsikt. The way forward is what we now have with WE&ME and also with PRIME. Those are probably the most important groups likely to be involved in the funding of something worthwhile anyway.

Beyond that the reality is what we have already seen. If you stick up for rigour and high standards you will be seen as awkward bastards by a good proportion of researchers, advocates and medics. If you take a laxer chummy approach there is no point. This is by definition about rigour.

Organisations like UK NIHR and various subsections of US NIH were deliberately set up to facilitate poor quality politically expedient projects - in parallel with the rigorous tracks. It will be the awkward bastards for them. The rigorous tracks have not been engaged in drug trials so far. But even for UK MRC the history is not encouraging, with PACE.

Up until recently we have seen that play out for S4ME but in the last month I have seen very positive developments. I sense that S4ME has earned respect through playing the long game. The research community is coming here, rather than S4ME having to go begging. I suggest keeping to the same strategy.

 

[Liface]

Appreciate all the different perspectives, this is really helping me sharpen my viewpoint.

I'm a huge fan of AI, and absolutely believe it will make the whole process far faster and better, but I do not think that means we should give up on improving trial design now.

Even preventing one bad dose choice, weak outcome measure, or avoidable exclusion criterion could materially improve a study. The goal is not to make researchers like us, or to provide a patient seal of approval. It is to get rigorous criticism in early enough that it can still change some protocols.

I also agree that patient representatives can be selectively chosen to produce the desired answer. That argues for multiple representatives, transparent selection, declared conflicts, and a record of what advice was given and whether it was followed.

S4ME’s strength has been its willingness to be fastidious, but so far it has been focused mostly on internal discussion, rather than leading with outreach to researchers. There are certainly downsides to that approach, the main being that all the great criticism that emerges here is rarely surfaced to the research community.

There is an opportunity to make that same scrutiny available wider and earlier, building on the ideas that came from the great partnership WE&ME, and

While I understand the criticism, as a startup founder before losing my life to severe ME, I'm not one to be shy about making big bets, even in the face of skepticism.

Interested to hear more discussion, and will likely start another thread in the advocacy planning forum about bringing this to fruition for those interested.

P.S. I am relatively new here, and understand that this forum has a particular culture, so I don't want to intrude on what you are already doing - the organization brought about by this will not run under the S4ME name, but likely a new name/organization.

 

[BrightCandle]

Is this maybe pressure from reviewers who want to see a "tried and tested" endpoint?
It's always hard to deviate from the norm and get stuff published. Or is it maybe not knowing better?

The same thing happened for the DecodeME project. They knew there was an issue with diagnosis in the UK but they still required it as an input despite then largely ignoring it and requiring CCC assessment anyway. They did it because of the expectation of UK doctors and trying to get credibility for the findings, in the end it bit them because they couldn't recruit the 25,000 they wanted to. It didn't hurt the patient criteria or result it just impacted on the total they could recruit, so a lower impact than usual.

No projects are immune to this influence, and it is an influence from the funding side, peer review and the medical system. The only research able to step outside that is patient funded and WE&ME being a pioneer of it. Alas most patient funded research is unfortunately very poorly funded because patients are really sick and don't have any money being in legislated poverty. The charity funded work ought to be better than it is but they also don't seem to listen to patient criticism to try and do better.

This is all to say even when they are involved in S4ME this influence still has more pull on them than S4ME review and patient feedback before the point of no return. Patients don't have a majority call on WE&ME either, we don't know how well that is going to go but patients could find themselves overruled there too to the detriment of the projects. This is just how the system works and its bad its really bad.

 

[Midnattsol]

The hard pill to swallow is that a lot of these trial designs are bad on purpose. Its not that they don't know there is a better way, most of these people have had training on trail bias or have plenty of experience to pull on, they don't need us to point out the flaws.

There is plenty of bad teaching material. Students are taught by the people who do bad research and learn that’s the way to do it.

It’s incredibly easy to point to the Chalder Fatigue Scale bring commonly used and thus it’s ok. Say it’s been validated for the patient group and you’re good to go.

Want to make your own scale? No problem! I have two in my phd, no one have batted an eyelid at me making arbitrary scales based on questionaire data. It’s just what we do in the field when you want to use what has been painstakingly collected (epidemiology).

There’s not really difficult to change a trial design either, although of course it would depend on how large the changes are and what type of study.

 

[Kitty]

But choosing bad endpoints such as the Chalder Fatigue Scale is more likely to give you a negative result. Why would you still choose it?

Possibly because it's seen as a 'validated' tool, and there aren't a lot to pick from. Possibly also because it makes a trial's endpoints comparable with others that used the same measure.

So perhaps one strategy is to question the use of the Chalder Fatigue Scale every time. Point out that ME/CFS involves a loss of function, not fatigue, so that's what should be measured. And while the FUNCAP scale isn't perfect, it measures function better than anything else that's been developed.

 

[Kitty]

What are the main factors that influence how late the protocol can be modified?

Realising you're not going to get the result you wanted without moving the goalposts? That seems to be what happened with PACE.