How can we create a patient-led group to influence individual trial designs before it's too late to change them?

[Felis Catus]

I agree with @Utsikt. The way forward is what we now have with WE&ME and also with PRIME.

@Liface , have you seen the PRIME ME/CFS Research Involvement Hub?

Launch of the PRIME ME/CFS Research Involvement Hub

We’re delighted to announce the launch of the PRIME ME/CFS Research Involvement Hub - a major new initiative placing people with lived...

www.actionforme.org.uk

 

[fst]

Possibly because it's seen as a 'validated' tool, and there aren't a lot to pick from. Possibly also because it makes a trial's endpoints comparable with others that used the same measure.

Agree. Although, comparing null to null is not so useful in the end. But that's probably not how the designers of these trials see it.

So perhaps one strategy is to question the use of the Chalder Fatigue Scale every time. Point out that ME/CFS involves a loss of function, not fatigue, so that's what should be measured. And while the FUNCAP scale isn't perfect, it measures function better than anything else that's been developed.

Scheibenbogen did that repeatedly during the Berlin conference (and I really appreciated that). Doesn't seem to have caused a second thought on the Abilify trial design.

 

[Creekside]

They have resisted attempts by patients to adjust trial design to something more likely to produce quality results for a long time.

There was an announcement of a new study recently. I considered sending them an email recommending that they use appropriately ill controls rather than healthy controls. I didn't bother, because deep down I was convinced that it wouldn't make a difference, so it wasn't worth the time it takes to load up my email software. There's no way they can't know that comparisons with healthy controls involves too many factors due to lifestyle differences, so their choice of healthy controls is for a reason.

Wouldn't it be interesting if all studies were published with an addition noting their obvious flaws? How difficult would it be to develop a list of flaws common to many studies?

 

[Jonathan Edwards]

S4ME’s strength has been its willingness to be fastidious, but so far it has been focused mostly on internal discussion, rather than leading with outreach to researchers. There are certainly downsides to that approach, the main being that all the great criticism that emerges here is rarely surfaced to the research community.

You may be unaware just how much outreach is made by members. Some of that is through open letters but a lot of it is personal emails, chatting with research colleagues at meetings and so on. The S4ME community has made a major impact on a lot of research projects. The same community had a significant influence on the way DecodeME came into being. Ideas flagged up here, including composite objective/subjective outcome measures, long term fit bit studies, etc. have been picked up and used by the research community. The researchers who want to get things right are very aware of what goes on here. There are probably empire builders who don't want to know, but they are going to be hard to reach however you try.

If another organisation is set up to canvas researchers who is going to decide what is good policy? The advantage of S4ME is that we argue things out in detail on a case by case basis. Even here there is no one person with a comprehensive grasp of all the aspects that need covering. Is there actually a consensus on what is supposed to be wrong with the Charité LDA trial?

 

[Utsikt]

S4ME’s strength has been its willingness to be fastidious, but so far it has been focused mostly on internal discussion, rather than leading with outreach to researchers. There are certainly downsides to that approach, the main being that all the great criticism that emerges here is rarely surfaced to the research community.

The people that want to hear it already seek it out, directly or by reading here.

While I understand the criticism, as a startup founder before losing my life to severe ME, I'm not one to be shy about making big bets, even in the face of skepticism.

My background is business management. So to speak that language: what’s the buy-in for the researchers? Why should they care about what anyone here says?

You’re assuming they all want to do the best research possible, and will act completely rational. It’s essentially the homo economicus assumption.

 

[PomegranateEmeralds]

So I used to not understand the value of trials for LDA, LDN, etc until I realized how egocentric I was being easily accessing these things, and that for patients in other contexts; they literally depend on their doctor skimming the abstract of such a trial, seeing a positive p value, or the trial ending up in some "evidence based care" module to be able to be prescribed meds and have them covered..


As such, I've now become really invested in how we can improve their design, in hopes of getting that significant p value (since most doctors do'nt know how to interpret research beyond that) and to me one of the most pressing issues

1) in addition to the issue of dose titration that trials keep bungling is

2) The issue of how crude and blunt instruments are for being able to detect a meaningful variance that would register as statistically significant and clinically meaningful.

Judging from my personal experience with severe, my function barely shifts; but the degree of my suffering does, extremely so! from the suffering that wishes euthanasia, to a level where now I can feel joy again..

To that end, and as others have been saying; I think the measures out there are woefully woefully inadequate. Like even the Depaul PEM questionnaire.

In my fantasy; if I'm better one day; I'd design, in community; a whole other multi dimensional PEM questionnaire that is both diagnostic and of research utility.

But until then; I have a scientific hunch that there might be 2-3 single items that end up having immense predictive validity, being very sensitive to outcomes, and very likely to yield statistically significant results that allow patients to be prescribed those meds.

e.g. asking patients to rate their general total feelings of sickness and malaise on a wide enough likert scale

for the subgroup that reports feeling poisoned in particular; same, to what degree do you feel poisoned

and generally; maybe a question about a sense of life/force general vitality vs the fatigue questionnaire; to me somehow they are different constructs, if co-correlated..and might be able to parse out variance.

Also, a 1-2 item measure of cognitive symptoms like concussion/swelling/fire

I feel it's not too late to ask researchers to literally add 3 item questions to their self report assessment section until we are able to design better scales and validate them, which takes years!

 

[Peter T]

In relation to researchers having been taught better, but still doing the same old terrible study designs, unfortunately I think a lot of people when they actually start to do things themselves end up not doing what they were taught but do what they have seen others doing. I am not sure what the best solution is, but the more good research that there is out there the more people will seek to emulate it.

In general I don’t think most people are wilfully seeking to harm, rather they are just taking the path of least resistance.

 

[PomegranateEmeralds]

The people that want to hear it already seek it out, directly or by reading here.

My background is business management. So to speak that language: what’s the buy-in for the researchers? Why should they care about what anyone here says?

You’re assuming they all want to do the best research possible, and will act completely rational. It’s essentially the homo economicus assumption.

their motive would be significant results vs. null findings; which despite scientists protesting otherwise; they're still only human, and they want shiny results that get them featured in news and highlighted by their university, invited to talks, etc

better design IMO is more likely to have sig results..I think a lot of the null findings out there are poor design/methodology issues rather than the interventions genuinely being completely impotent.

 

[BrightCandle]

Judging from my personal experience with severe, my function barely shifts; but the degree of my suffering does, extremely so! from the suffering that wishes euthanasia, to a level where now I can feel joy again..

This is something I have wondered about quite a bit. Functional capacity is a better thing to measure than fatigue, symptoms and severity is good too but how to do that well, but almost no one is really talking about the suffering aspect mentally and the pain. Pain scales aren't really part of ME/CFS at the moment but I sure find this condition painful and there is a lot of general suffering involved from that pain and how this feels. Its an aspect that has pretty much not made it into the field at all yet and it needs to. Even the comments we make on study designs are mostly missing this aspect, focussed more on functionality and general capability because its more measurable and economic but suffering (beyond just that caused by medical neglect and abuse) is something the field could do with too.

 

[Felis Catus]

So I used to not understand the value of trials for LDA, LDN, etc until I realized how egocentric I was being easily accessing these things, and that for patients in other contexts; they literally depend on their doctor skimming the abstract of such a trial, seeing a positive p value, or the trial ending up in some "evidence based care" module to be able to be prescribed meds and have them covered..

Accessibility is a good thing until an off-label or unlicensed medication harms a patient.

Without rigorous studies and published data, patients rely on their dr's words along the lines of "None of my patients have had any long-term side-effects" and "half the patients got much better".

 

[Andy]

The same thing happened for the DecodeME project. They knew there was an issue with diagnosis in the UK but they still required it as an input despite then largely ignoring it and requiring CCC assessment anyway. They did it because of the expectation of UK doctors and trying to get credibility for the findings, in the end it bit them because they couldn't recruit the 25,000 they wanted to. It didn't hurt the patient criteria or result it just impacted on the total they could recruit, so a lower impact than usual.

Some corrections and clarifications.

"they still required it as an input despite then largely ignoring it"
It was part of the process of defining our study cohort - I'd be curious to know what else you might have expected us to make of that specific bit of data?

"requiring CCC assessment"
Participants were assessed against the CCC and IOM/NAM criteria.

"They did it because of the expectation of UK doctors and trying to get credibility for the findings"
We did it to define our study cohort in the most reasonable way we could.

"in the end it bit them because they couldn't recruit the 25,000 they wanted to"
We had over 37k people who signed up to our mailing list and just short of 27k complete the screening questionnaire, of those just over 21.5k were eligible to donate saliva, and 18k of those returning samples to us. Yes, we had a headline number of 25k participants but always anticipated losing numbers at each stage and we actually ended up broadly where we needed due to higher than anticipated retention rates.

"It didn't hurt the patient criteria or result it just impacted on the total they could recruit, so a lower impact than usual."
I don't understand the point that you are making here. We recruited enough patients who donated sufficient saliva samples in order for us to be able to run a sufficiently powered GWAS.

 

[RainbowCloud]

Regarding the measurable scales:

I was under the care of the pain management dept for migraines years ago when I was mild with ME. I had to rate my migraine and headache pain when I had one, but as I got them virtually every day, I often marked down 4 and perhaps 6 at worst. What I later realised after they’d calmed down was that actually the 4’s were more like 6’s and the 6’s more like 8-9’s. I’d just become so used to the pain that I didn’t think it was as bad as it was.

I think a major flaw that’s difficult to control in ME studies is that we are so used to the extremes of everything that ME brings, and that may skew our rankings (a lot I would have thought!) So researchers somehow need to account for this and as was noted earlier, the FUNCAP does manage to mitigate some of this.

 

[Kitty]

Speaking of measures, I don't think I ever seen a 'flu scale' used.

Possibly because researchers have no idea that asking people to score how ill they feel would reflect the reality so much better than asking than how fatigued they feel?

It might also be a less common symptom than fatigue. For instance, the fatigue of active psoriatic arthritis is worse than that of ME/CFS, but you don't feel ill at all.

 

[V.R.T.]

Possibly because researchers have no idea that asking people to score how ill they feel would reflect the reality so much better than asking than how fatigued they feel?

This is a very good point.

 

[Utsikt]

As such, I've now become really invested in how we can improve their design, in hopes of getting that significant p value (since most doctors do'nt know how to interpret research beyond that) and to me one of the most pressing issues

This is backwards. The purpose of a trial is to test if the treatment actually works, not to get a p-value in favour of the treatment.

1) in addition to the issue of dose titration that trials keep bungling is

That could be a legitimate concern for these trials.

2) The issue of how crude and blunt instruments are for being able to detect a meaningful variance that would register as statistically significant and clinically meaningful.

I’d argue most are overly sensitive to change, because it’s very easy to get someone to change their answer on a questionnaire.

Judging from my personal experience with severe, my function barely shifts; but the degree of my suffering does, extremely so! from the suffering that wishes euthanasia, to a level where now I can feel joy again..

To that end, and as others have been saying; I think the measures out there are woefully woefully inadequate. Like even the Depaul PEM questionnaire.

I agree with the conclusion, but perhaps for different reasons.

their motive would be significant results vs. null findings; which despite scientists protesting otherwise; they're still only human, and they want shiny results that get them featured in news and highlighted by their university, invited to talks, etc

You don’t get more significant results by following advice from S4ME, you get far more null results.

better design IMO is more likely to have sig results..I think a lot of the null findings out there are poor design/methodology issues rather than the interventions genuinely being completely impotent.

This assumes the intervention actually has an effect, which is very rarely the case. We see the opposite in the decently designed trials of the hyped treatments for LC and ME/CFS: they get null results.

 

[bspielvogel94]

Speaking of measures, I don't think I ever seen a 'flu scale' used.

"The flu" almost perfectly describes my illness, but I'm aware that it can be insufficient for some (e.g., intolerance to light/sound would not be associated with flu-like symptoms, I would think).

Regarding the broader thread, I am very curious to hear if the S4ME community perceives different organizations / sources of funding to produce trials of different levels of quality? For example, does OMF typically produce/fund high quality studies whilst the NIH (U.S.) typically fund low quality studies?

I would think that studies coming from the non-profit research foundations would be of higher quality as 1) their funding comes from patient communities who might stop donating if the research quality is poor, and 2) they are often founded/operated by folks with seemingly genuine motivations, mainly family members who are extremely sick. But I have no idea if that's actually the case.

 

[duncan]

A patient-led group would have to earn the reputation of informed disinterest specific to ME/CFS, then evolve that to imprimatur status such that its stamp of approval was an incentive for any researcher to claim. Easier said than done.

I think we've achieved the former among a handful of researchers already. The trick is to getting them to promote our approval as an asset in their proposals.

Then we market by reaching out to other researchers in the community.

It would be a process.

 

[fst]

I’d argue most are overly sensitive to change, because it’s very easy to get someone to change their answer on a questionnaire.

Valid point. Although would argue that those fatigue questionnaires rarely get to the core of the issue and therefore, do not pick up the (right) signal.

In Scheibenbogens n=20 immunoabsorption trial, we've seen some decent improvements in SF-36, but needle on the fatigue severity scale barely moved.

Now this could be because SF-36 is overly sensitive to (placebo) responses, or because FSS is not responsive enough to pick up an actual improvement (whether it's placebo, fluctuation, or real).

I just wish they would have renamed this disease to SEID or something sensible without the word fatigue in its name.