Germany: IQWIG Report to government on ME/CFS - report out now May 2023

It's obviously not. But since PACE is the meat of this eminence-based evidence, they feel the need to defend it. Of course it's always an issue. It's even a common issue, this is why even the most fervent promoters of psychosomatics will dismiss any study featuring this flaw where they don't like the conclusions.

The discussion of how objective measures and subjective ratings disagree is plain silly. It basically amounts to rejecting science to so badly want to emphasize biased subjective ratings above objective measures. Again an issue that even the most fanatical BPS ideologues will dismiss any study featuring this that they don't like. If they didn't have double standards, they wouldn't have any.

The recommendations they make are shockingly bad, but there is still lots of lemonade to press here, especially where it intersects with the NICE debacle. Because what IQWIG did here basically validates most of we said. Somehow they still hold on to their predetermined conclusion, but they validate literally all the concerns and provide solid arguments.

1) The analysis dismissed 99%+ of the trials, showing how poor they are. They dismissed them for being too biased and for PEM indirectness. This mostly confirms the NICE analysis, even going further in simply not considering the small ones. Which shows that if you're going to be biased, just spend a lot of money on it, sunk cost creates an anchor around funders' necks.

2) This actually makes the PACE argument that they preferred the results in part because of their expectations based on prior trials, which IQWIG here considered as too biased to be used, even worse. This diminishes the validity of the PACE results, as meeting the expectations of trials too biased to be of use actually shows those "benefits" show that PACE was just as biased. That severely biased studies agree with biased studies is no surprise, especially when they are almost identical and some were made by the same people, with the same intent and formula.

3) The requirement of PEM is one of the major points, the argument of the lobby groups being that it invents a new disease. It obviously does not and IQWIG agrees that it is necessary, even if they lowered the threshold so that PACE could be included, which is seriously too meta. The NIH/IOM also agreed with this. So does the CDC.

One thing we can expect out of this is that future biased trials will pretend to take PEM into account, while never actually bothering. Just like PACE.

There is something to emphasize here in the general idea of: "German health authorities agrees that 99%+ of pragmatic trials in ME too biased and unreliable to be of use".

And the absurd argument of No true GET can be countered by the study that showed that it's actually PACE-like GET-trained therapists who do the most harm. I'm not sure if they'll care. It just makes no sense to rely on a biased (and so, so tiny) evidence base that strictly uses subjective ratings, self-reports, to be dismissive of self-reports. This is peak eating their cake and having it, too. This double standard where self-reports are only reliable if they show improvement is just the mother of all cheat codes.

 

I've had a not super thorough look at the google translated document (so I may be mistaken about some aspects). There's probably a lot to say about this report, however here is one of the more important parts. Here is the conclusion for GET vs SMC:

"When comparing GET versus SMC, the morbidity endpoints Fatigue, sleep quality, physical function, physical performance, social participation, and mental status, there were statistically significant differences between the groups in favor of GET for individual evaluation time points. Because the effect size was not considered relevant in these cases, no evidence of benefit or harm was derivable for these endpoints. For the endpoint general symptoms, however, an indication of a benefit of GET was derivable for the short-term treatment effects. With regard to the endpoint feeling of sickness after exertion, derived from the results of the PEM-survey, there is an indication for a benefit of GET in the medium term (see table 50). For all other endpoints-pain, activity level, cognitive function, post-exertional malaise, health-related quality of life (HRQoL), all-cause mortality, and serious adverse events - did not show statistically significant differences or there were no (meaningful) or no (usable) data were available." (p. 150)

So there was not a relevant difference between Standard Medical Care and Graded Exercise Therapy for most of the outcomes, including, notably, Fatigue and Physical function. The only outcomes that showed important differences were "general symptoms" - in this case measured with the "Clinical Global Impression Scale" (some vague scale that just asks how sick someone is) in the short term, and the PEM survey (asking someone if they feel sick after exertion, I think?) in the medium term.

But after "all outcomes were weighed across endpoints" (also p. 150) - I don't know what that means, and I don't know how they did that, I don't see any more detail about what that means besides that sentence - they still conclude that GET has benefit. I believe both the PEM survey and the CGI have no specific evidence for their validity in ME/CFS at all, and they're both pretty vague measures (they point this vagueness out with the PEM survey and also note the CGI wasn't applied as it was supposed to in the PACE trial)

So my issue with that is that the conclusion for GET is not really explained, nor does it appear to make sense. Besides that some other general issues I have are:

1. Generally no proper consideration of the validity of outcome measures (the evidence for validity of outcome measures in ME/CFS is generally lacking)
   
2. No consideration of ceiling effects
3. Assumption that they have any good measure of adverse effects, short, medium or long term (also related to points 1, 2 and 4)
4. Unclear how they incorporated high risk of bias (which includes issue with lack of blinding/subjective outcomes) into their final conclusion. They say they considered it (p. 156) but I cannot exactly tell how it affected their conclusion, especially for GET
   
5. They admit that including people without PEM may have caused the result to meet the clinically relevant threshold in CBT in PACE (p. 146), but dismiss this by saying the result would still have been statistically significant even excluding them. However clinical relevance is the important thing and therefore I think this may actually be grounds for downgrading the result for CBT further

I don't know German so I can't make a submission but I might be interested in contributing to a submission if someone makes one

 

If I understand correctly, they will publish a brief overview of this report on the website www.gesundheitsinformation.de. This will be information aimed at the general public and interested healthcare workers, similar to what the CDC offers on their website.

A draft of the information that will be published on this website is available at the very end of the document starting on page 275. I think this section should receive the most attention of submitted comments. For those who cannot read the whole document and technical section, I would recommend focusing on this part.

Overall, the information seems alright instead of the treatment section where they recommend GET and CBT. It reminds me a bit of the 2007 NICE guidelines as they also offered energy management/pacing as a treatment option.

The description of GET and CBT are toned down, possibly to avoid criticism. CBT, for example, is described asa therapy that helps to cope with the psychological stress caused by the disease, not the type of CBT for ME that tries to cure the disease by encouraging patients to no longer see themselves as fatigued. In other words, this isn' the extreme version GET and CBT as defined in the PACE trial.

It also seems that a section on severe ME/CFS is missing.

 

Yes I very much agree and I think this is worth repeating.

Even if we ignore the high risk of bias and the lack of control condition and only focus on the numbers, the IQWIG's own analysis showed no reliable effect for GET.

They looked at all of the following outcomes:

• fatigue
• physical function
• sleep
• pain
• activity level
• physical performance results
• cognitive function results
• social participation results (school and work resumption)
• mental status (anxiety and depression)

For all these outcomes they looked at short-term, medium-term and long-term results separately as recorded by the PACE and GETSET trials, the two biggest trials on GET. Yet for all these outcomes and for all three time-periods, the IQWIG concluded that "no hint of a benefit of GET compared to specialist medical care (SMC) could be derived."

In many ways this is a vindication of what patients advocates have been saying. Take for example the 6-minute walking test. The PACE authors highlighted that there was a statistically significant difference between GET and SMC. IQWIG now confirms what patient advocates wrote in letters to the Lancet, namely that the difference was too small to be clinically relevant.

As @petrichor mentioned, the only two outcomes where GET (temporarily) showed an effect compared to SMC were the global clinical improvement (GCI) scale and the outcome "feeling sick after exertion". The latter was only recorded in the PACE trial and was not one of the many primary or secondary outcomes listed in the protocol or statistical analysis plan for this study.

The GCI was only a secondary outcome and a problematic one because it arbitrarily reduces a 7-point scale to a percentage improvement score. Patients could indicate how they felt after treatment using 7 options: very much better, much better, a little better, no change, a little worse, much worse or very much worse. The data used was the percentage of patients reporting the first two options (very much better or much better) in the GET versus SMC group.

Even then, the data isn't convincing. Only at short-term assessment (12 weeks, when the PACE trial was only halfway) was there an apparent effect for GET versus SMC. For the medium and long-term assessments, again "No hint of a benefit or harm of GET compared to SMC could be derived".

So that is it. This is what the recommendation for GET is based on... very curious.

 

So in summary:

• There were only two randomised trials on GET used in the review and both had the highest risk of bias possible for all outcomes. They could not have a worse quality rating.
• They did not have a control condition: patients in the intervention group received GET + whatever patients in the "control" group received. The design was A versus A + B.
• For the primary outcomes, fatigue and physical function, no hint of a treatment effect could be found for short- medium- and long-term assessments.
• The same was true for almost all secondary outcomes: pain, sleep, mental status, physical activity, physical performance results, cognitive function results and social participation: no hint of an effect for or short- medium- and long-term assessments.
• One of these secondary outcomes, the global impression scale, did indicate an effect but only for the short-term assessment, no longer on medium and long-term assessments.

And still they recommend GET as a treatment option... They write:

"When all results are weighed up across outcomes, there is a hint of a benefit of GET compared to SMC for patients with mild to moderate ME/CFS severity in both the short and medium term."​

 

Some... ghostly shadow of a possible potential maybe-who-knows benefit. It was on a toast. Whole grain. No crust. When the Sun shines at exactly Noon on the Summer solstice. And you need the exact 3D glasses used during screenings of James Cameron's Avatar. Mint, in their original wrapping, and couriered by a virgin on no more than their 17th birthday and no less than their 13th.

Basically evidence-based medicine: "what if, and hear me out here, we did not have any standards whatsoever and just wrote whatever feels good?"

 

“we could argue over 95% with PEM versus 80% with PEM”

It’s like saying, we could argue over 95% with a broken leg versus 80% with a broken leg in a study about people with a broken leg. Those people who don’t have that should have been excluded from the study. If that wasn’t done, then the study is biased before it started and it should end up in the bin.

 

In regards to that Q-fever study, When we analysed them, we found this:
Our reanalysis found that the Qure study suffered from many serious methodological problems, which included relying on one subjective primary outcome in a study without a control group for the non-blinded CBT treatment group, using a post hoc definition of improvement, waiting 2 years before publishing their objective actometer results and ignoring the null effect of said results.

Moreover, only 10% of participants achieved a clinically meaningful subjective improvement in fatigue as a result of CBT according to the study's own figures.

Consequently, CBT has no subjective clinically meaningful effect in nine out of every ten patients that are treated with it. Additionally, the subjective improvement in fatigue was not matched by an improvement in disability, even though the disability was fatigue related according to the researchers. On top of this, CBT did not lead to an objective improvement in physical performance.

Therefore, it cannot be said that CBT is an effective treatment for Q-fever fatigue syndrome either. It seems therefore unlikely that CBT will reduce disability or lead to objective improvement in long COVID or in post-COVID-19 fatigue syndrome.

https://www.researchgate.net/public...m_the_Qure_Study_for_Q-Fever_Fatigue_Syndrome

 

On page 157 of the report (177 of the PDF) they seem to use the 2011 review of patient surveys by Tom Kindlon to argue that a lot of patients felt improved after GET and that the reports of harm were not that striking. @Tom Kindlon

The translation reads:

"Furthermore, other interpretations can be derived from the publication of the Irish ME/CFS Affected Persons Association [132] than those of Friedberg et al. as demonstrated by the following results. In the cited patient surveys on GET, between 28.1% and 82.0% of respondents reported that their condition worsened after GET, but also between 13.1% and 60.8% of respondents (own calculation) reported that their condition improved. In the patient surveys for CBT, between 7.1% and 38.0% of respondents indicated that their condition worsened after CBT and between 7.0% and 56.9% of respondents (own calculation) indicated that their condition improved."​

Here's the original text in German:

Zudem lassen sich aus der Publikation des irischen ME/CFS-Betroffenenverbands [132] auch andere Interpretationen ableiten, als die von Friedberg et al., wie die nachfolgenden Ergebnisse demonstrieren. In den zitierten Patientenbefragungen zur GET gaben zwischen 28,1 % und 82,0 % der Befragten an, dass sich ihr Zustand nach einer GET verschlechtert hat, aber auch zwischen 13,1 % und 60,8 % der Befragten (eigene Berechnung) an, dass sich ihr Zustand verbessert hat. In den Patientenbefragungen zur CBT gabenzwischen 7,1 % und 38,0 % der Befragten an, dass sich ihr Zustand nach der CBT verschlechtert hat und zwischen 7,0 % und 56,9 % der Befragten (eigene Berechnung) an, dass sich ihr Zustand verbessert hat.​

References 132 refers to:

132. Kindlon T. Reporting of Harms Associated with Graded Exercise Therapy and Cognitive Behavioural Therapy in Myalgic Encephalomyelitis / Chronic Fatigue Syndrome [online]​

 

Section 2.3.3.6 on page 179 (page 199 of the PDF) gives info on how they rate evidence. Here's the DeepL translation (my bolding):

A 2.3.3.6 Statements on evidence

For each endpoint, a statement is made about the evidence of the (higher) benefit or (higher) harm. There are 4 possible levels of certainty: either there is evidence (highest certainty), evidence (medium certainty), evidence (weakest certainty), or none of these 3 situations. The last case occurs when no data are available or the available data do not allow any of the other 3 statements. In this case, the statement "There is no evidence for a (higher) benefit or (higher) harm" is made.

The reliability of the results depends on the criteria shown in Table 54. The qualitative certainty of results depends on the design of the study. Results of randomized studies with a low bias potential have a high degree of certainty of qualitative results, while results of randomized studies with a high bias potential have a moderate degree of certainty of qualitative results. Results of non-randomized comparative studies have low qualitative certainty of results.

Finally, a cross-endpoint assessment of utility is provided. This cross-endpoint assessment also includes data completeness and the potential resulting bias due to publication bias, which may lead to the qualification of the conclusion.

 

So no matter how bad the trial is, if you randomize patients, it will still be counted as moderate-quality evidence?

 

Depends who you are and what your goals are, of course. Not everyone gets the exception, you must be that eminent.

Just like they ignore patient surveys when inconvenient, but emphasize them when they can be construed as having a hint of a shadow of maybe it's not so bad.

Cherry-pie evidence, carefully selected and baked to perfection.

 

Apologies for just popping in -- not able to catch up with the thread and the longer posts.

That's how understand it too.

Only after the "cross-endpoint assessment" when comparing endpoints at the final step, the quality of evidence can be downgrades to 'very low" / " a hint" (Anhaltspunkt).

See the example they give on their assessment of the PACE trial...

What I haven't understood yet is if 'cross-endpoint' means different endpoints in the same study or the same endpoints in different studies?

That what I was referring to when I wrote:

"In the evaluation of interventions, how did they address the incoherence of benefits between endpoints that should show some relation?"

For example:

p.149-150 (p. 169-170 in the machine translated PDF ) :

"When comparing GET versus SMC, there were statistically significant differences between the groups in favor of
GET for the morbidity endpoints of fatigue, sleep quality, physical function, physical fitness, social participation and
psychological status for individual analysis times. Since the effect size was assessed as irrelevant in these cases,
no hint of benefit or harm could be derived for these outcomes. For the outcome 'general symptoms', on the other
hand, an indication of a benefit of GET could be derived for the short-term therapy effects. With regard to the
outcome 'feeling sick after exertion', derived from the results of the PEM survey, there is a medium-term indication
of a benefit of GET (see Table 50)."

So if there was only a statistically significant difference but not of a relevant effect size so no 'hint of evidence' of a benefit (or harm) for all of the following (mostly self-reported) endpoints fatigue, sleep quality, physical function, physical fitness, social participation and psychological status , doesn't this cast doubt on an actual effect also on the endpoints 'general symptoms' and 'feeling sick after exertion'?

Especially, if 'feeling sick after exertion' wasn't clearly associated with duration, only with intensity?

See also table 50, I translated the endpoints in the displayed order and also added a machine translation of the footnotes (managed only to get a snippet)

Endpoints:

Fatigue - Sleep quality - Pain - Activity level - Physical function - Physical capability - Cognitive function- Social Participation -

-- Psychological status - General symptoms - PEM/ Feeling ill after exertion

 

If the results for the endpoints are as heterogeneous as they are here, doesn't that rather provide additional evidence that the results of the included RCTs aren't reliable?

Edit: wording

 

And yet, actual quote from the man-in-charge, because it keeps revolting me:

In response to someone asking him whether the PACE subjective "benefits" could be placebo. Literally the comparison for "no effect". I will be forever amazed at the mediocrity of this nonsense being actually believed by, it seems, 99% of physicians working today and 100% of medical institutions.

I've seen a lot of bad stuff on the Internet and not much disgusts me as much as this, coming from medical doctors makes this all so much worse.

 

I don't think that's exactly what they're saying here, although it sort of is. That's how it's categorized in table 54. So if it's a randomized trial and it has a high level of bias, then for a particular endpoint (fatigue, physical function, sleep quality etc.), you look at the line of the table along "mäßig" or "moderate" in order to determine if it's "beleg" (evidence) "hinweis" (indication) or "anhaltspunkt" (hint) according to the criteria (no. of studies, heterogeneity, statistical significance). If it had low bias you'd look along "hoch" or "high". And then once those individual endpoints are done (like on p. 148), the cross-endpoint analysis is done after that (like on p. 150).

They mean different endpoints in the same comparison, like Graded Exercise Therapy vs Standard Medical Care (and that comparison may include multiple studies). That's how they did it on page 150 for instance.

Perhaps they can argue that while most of the results were not clinically relevant, they still nethertheless had the same direction of effect.

 

That's helpful @petrichor -- even if I'm still in skimming-only mode.

That's perhaps what they mean in table 54, quoted by ME/CFS Skeptic?

"Unidirectional effects are present when, despite heterogeneity, a clear or moderate direction of effects is evident."

But does that make sense?

If there isn't a relevant effect of an individual endpoint, even if multiple statistically significant effects on different endpoints have the same direction, can you still add up those non-relevant effects from different endpoints and arrive at a statistically more significant effect?

Is this how their statistic calculations work? Is there actually a relevant increase in the probability that the non-relevant effects could be relevant?

These calculations to me all seem implausible.

Given that even by the IQWiG's calculation, in the GET/SMC comparison the cross-endpoints results both on fatigue and on physical function were assessed as heterogeneous even by the IQWiG classification, that seems even more odd (See table 50 quoted above. )

So if you have a number of (non-significant) effects with the same direction and a smaller number of (non-significant) effects with heterogeneous direction, how can you decide that that the minority of endpoints that have heterogeneous results ( not the same direction) haven't the same (low) quality of evidence as the homogeneous results?

If those are all just tiny differences in probability, how can you justify to classify this not only as a 'hint' of a benefit from an intervention, but even recommend that intervention?

For a recommendaton, I think you also have to weigh the individual endpoints for their relevance and reliability with regard to indicating a real improvement.

It appears from some explanatory notes that they aimed at doing that, but that they didn't take into account again that all morbidity endpoints were self-reported, among many other issues?


Edited/ clarity

 

Some points that I collected in a draft post but don't know whether I will be able to look up myself:

-- Which of the included RCTs included social participation as an endpoint? Did they use reliable measurements on these endpoints?

-- What time span is 'medium term' / how long did the alleged/ reported short-term and medium-term improvements actually last during and after the end of the intervention (the IQWiG seems have only considered the time after randomization)?

(Could defenders of the evidence for a benefit argue that with a longer duration of the interventions, the effects could have maintained?)

-- From memory I thought in the PACE trial, there was a deterioration in work status and more participants received social benefits. That doesn't seem to be reflected in IQWiG's assessment? (They equate work status with social participation).

-- How was social participation / work status assessed in the other two RCTs?

(IQWiG calculated a hint on a benefit from CBT both short-term and medium-term).


-- Which trial included 'activity level' as an endpoint and how?

(IQWiG calculated no hint on a benefit from CBT short-term, no results are provided for medium-term and long-term, also no results for GET)

I guess that, of the endpoints included in the IQWiG's analyses, the outcomes most relevant to pwME are:

- activity level
- PEM, both duration and intensity
- social participation

Do we have any sources about that guess?

Because if we could argue that those are likely the most relevant outcomes, the evidence base the IQWiG deemed as sufficiently reliable doesn't show even a hint of a benefit for the first two mentioned [*], and I think the benefits they calculated for social participation can be questioned, especially as even by their own calculation, they don't show any long term benefit.

[*] The IQWiG states that the PACE trial probably didn't measure 'PEM' when the participants were asked about 'feeling sick after exertion', as that is too unspecific to measure PEM. But was 'alleged PEM'/ 'Feeling ill after exertion' even included as an endpoint in the PACE trial? And at which time point(s) was it measured?