Germany: IQWIG Report to government on ME/CFS - report out now May 2023

In their 'general methods' paper on page 159 (pdf p. 174) they mention: "For the inclusion criterion with regard to the study population, it suffices if at least 80% of the patients included in the study fulfil this criterion."
So, at least it wasn't an exception in order to include those studies.

Btw, there is an english version of the general methods available directly from IQWiG: https://www.iqwig.de/methoden/general-methods_version-6-1.pdf

On page 9 of the same document (pdf p.24) they state the very obvious:
"The basis of a benefit assessment is the demonstration of causality. An indispensable precondition for such a demonstration is a comparative experiment, which has to be designed in such a way that a difference between intervention groups – an effect – can be ascribed to a single determining factor – the intervention tested."

I have no scientific background. Yet, as they recognise the high risk of bias of the included studies I don't understand how they can be sure enough about causality to conclude treatment benefits. Especially when we look at the extent of placebo responses in the rituximab phase II trial.

 

Thank you — IQWiG could have added a footnote with a reference to their methodological guideline in the report to clear up the origin of this rule.

I remain perplexed by this rule. It would, for instance, allow including trials for Covid where 20% of patients have the flu, which has a better prognosis; this would inflate the results of these trials.

 

ME/CFS preliminary report: comment period extended.

Comments can now be submitted until November 27, 2022.

On the preliminary report "Current scientific knowledge on myalgic encephalomyelitis / chronic fatigue syndrome (ME/CFS)" (project number N21-01): At the request of patients, IQWiG is extending the comment period by two weeks - now until November 27, 2022.

https://www.iqwig.de/presse/pressemitteilungen/pressemitteilungen-detailseite_80324.html

 

This is really baffling. I can't imagine as an information specialist agreeing to work with data that is only 80% correct. I wouldn't even agree to 99% correct if the data set is large enough, especially with an effect so trivial that chance can explain all of it. When you have invalid data mixed in with potentially valid data, it's all invalid data.

I don't get this profession. At all. It almost seems designed to fail in some places, while doing borderline miraculous work in others. Truly the best and the worst, thrown into an infinite soup, bubbling for centuries and definitely keeping some very old bits in the deep corners.

 

On page 163 (pdf 178) of the methods report they write:
"The classification as “high” of the risk of bias of the result for an outcome does not lead to exclusion from the benefit assessment. This classification rather serves the discussion of heterogeneous study results and affects the certainty of the conclusion."
No matter how bad the amount of bias is, they will still consider the evidence.

If I understand that correctly: Not only will they consider trials of high bias in the benefit assessment; if there are two highly biased trials showing concludent effects they will even be considered a "hint" and therefore appear in the then created "evidence-based health information for consumers".

Arguing that their general methodes are flawed would probably not be very promising.

I agree. It could be most relevant to focus on the brief summaries.
On page 129 onwards (pdf 144) they describe their process for creating those information.

Regarding harm they write on page 44 (pdf 59):
"In the event of particularly serious or even life-threatening diseases, for example, it is usually not sufficient only to demonstrate an improvement in quality of life by application of the intervention to be assessed, if at the same time it cannot be excluded with sufficient certainty that serious morbidity or even mortality are adversely affected to an extent no longer acceptable."
Could we argue that there was no sufficient certainty in their assessment of harm?

 

This is why they a) need cost effectiveness analysis, like NICE; and b) need a broad range of experts to interpret the data (which has to include patients and carers).

Exactly. The problem is that GRADE and similar processes are designed to find effects, not rule them out.

You need an expert committee to then interpret the results, or you get absurdities, like a national recommendation for 70,000+ people based on extremely low quality evidence from only two trials apiece that may or may not show any benefit (and where you admit you don't have real evidence about harms).

People assume GRADE is the end of the discussion, when it's merely the start of the discussion. But here, there's been no discussion at all. Just "computer says yes".

They also fall into the trap of assuming they can avoid any issues by saying GET must be delivered by an expert. But it's invariably the experts who cause the most harm -- the clinicians who think they know the most are often the ones who know the least but have the most inflexible views.

 

Folks, you may wish to note the following -
Blog article and preliminary comments (in English) on the IQWiG report on EMEC website:
https://europeanmecoalition.com/the-iqwig-report-on-me-cfs-in-germany/

Direct link to the comments:
https://europeanmecoalition.com/wp-content/uploads/2022/10/EMECs-comments-on-IQWiG-report.pdf

https://twitter.com/user/status/1583487790737940481

 

Here's the thing: Reputation, like Integrity, are things that are basically built on trust and cumulative actions. And on them always being good. If you do something not good you lose it, so harking back to past reputation sounds like a coded word really - is it a reputation for credible research? Or power

So what does pushing incredible research be made OK instead of something that changes your reputation for only producing credible research?

He's right to ask the question, but it should be of himself

eg: https://hbr.org/2007/02/reputation-and-its-risks

 

Agree -- but unfortunately the IQWiG's view on that is different, as they made clear in their reply to the comments on the report plan:

IQWiG:

"The scientific and professional independence of the Institute and the content it represents and publishes is enshrined by law in Section 139a of the German Social Code, Book V (SGB V), as well as in the statutes of the Foundation.

"In order to maintain its independence, IQWiG does not regularly involve any interest groups in the processes for the preparation of its reports beyond the regular involvement options according to the Institute's methodology. These are described in the report plan and consist of:

# an initial stakeholder meeting,


# a hearing on the report plan,

# a hearing on the preliminary report, i.e. on the entire report result including the draft health information,

# an analysis of qualitative research on patients' experiences and information needs in the context of the drafting of the health information report

# a user testing of the health information, especially by those affected by different degrees of severity [ I have to find out yet how that is done]


"The BMG has commissioned IQWiG with the review of the current scientific knowledge on the topic of ME/CFS, but not with the preparation of a guideline. Therefore, the current approach regarding the extent of involvement of affected persons fulfills the mandate."

Source: 'Appreciation of the public hearing', deepl translate of draft report, German version, p.212 (their page number) p. 232 of the PDF and of the google translated PDF.


They have taken into account some critique that pwME expressed but it seems to me they still don't see that they need our expertise as patients and carers, or/ and as citizen scientists, for a proper process, including the assessment of the evidence.

 

Edit: now redundant --We now have world files of the German versions. Thanks for the help. If anyone felt up to help with machine translations and other things, please check the members only thread .


I realize that people are already working with the google translations of the draft report, but in some parts they aren't really understandable, especially the documentation of the comments/ public hearing.

Also, if I copy quotes from the PDF into a text document or here on the forum in order to comment, I get only a muddle of the quote.

That seems to be due to the PDF layout that transforms soft breaks into forced breaks/ paragraphs and other hidden layout.

For the post above I needed to extract the quote from the German version and delete the forced line breaks, before deepl gave me both a useful translation and a text that I could easily copy and paste without producing a muddle.

Any idea on how transforming the entire PDFs -- including tables -- into a usable word document could work with not too much effort?

If anyone felt up to it, here are the 3 documents (German versions) of which word documents (both in the original language and machine translations) would be helpful:

Preliminary report: Current scientific knowledge on ME/CFS
DE, PDF, 2 MB, PDF

https://www.iqwig.de/download/n21-01_me-cfs-aktueller-kenntnisstand_vorbericht_v1-0.pdf


Health information draft (extract from the preliminary report):
DE, PDF, 810 kB, PDF

https://www.iqwig.de/download/seite...nisstand_vorbericht_v1-0_a9_wasserzeichen.pdf


Documentation of comments [public hearing, including oral discussion] on the report plan
DE, PDF, 2 MB, PDF

https://www.iqwig.de/download/n21-01_me-cfs-aktueller-kenntnisstand_da-berichtsplan_v1-0.pdf

Any help with that would be much appreciated.

 

Yea it occurred to me that you should design out that risk and that this is a common issue for example bias in selection panels --- the answer is to design out prejudice.

 

Does the report highlight the risk of harm from GET - I think Graham McPhee may have published on this?

 

Begs the question of why they did not do a reanalysis if apparently they knew which patients had PEM..

this seems a very likely situation - particularly when you read the phrase 'those with the mildest form of'

I'd also want the drop-out info to find out wat % of those had PEM

Putting aside whether the index is reliable - one big issue seems to be whether those who then write the conclusions use the specifics to hide what these terms really mean. A matrix of 'if low effect + moderate confidence = low', 'if moderate effect + low confidence = low' is needed to spell out what really comes off these things. Recent examples discussed here seem to have conveniently used 'the long form' and then dropped the low confidence bit.

There are big, big issues as well when you combine this with choosing not to include other studies. Or patient surveys

Bad evidence is worse than no evidence. For a start it will tend to influence where funding gets but and we know what a virtuous circle trap for the wrong things that causes.

Secondly, should there be a well-designed smaller piece of evidence coming through the horrible steamroller of EBM has meant that rather than common sense being applied in analysis of 'the whole literature' whilst there has been a common sense guideline put in as a holder, you end up with said studies having to 'prove wrong' these other flawed larger studies.

That 80% could end up being something carried forward too. I'm glad that they at least noted the PEM hasn't been well-defined, but taking the definition from the right place is another question - precedent and who gets to set it matters.

 

Thank you to all involved in the work on the EMEC comment.

To facilitate feedback, I will post their comment point by point.

The comment doesn't always follow the structure of the IQWiG draft report so I added both the section and the English headlines from the machine translation where they were missing.

Please bear with me until all 14 points will have been posted. (Done)

Attached to this post is a google translate of the draft report's TOC.


Note that the deadline for feedback on the IQWiG draft report has been extended to 27 November 2022.

 

EMEC Comment:
4.2.2.5 Epidemiological and supply-relevant aspects – Prevalence - Page 24

(Epidemiologische und versorgungsrelevante Aspekte – Häufigkeit, p. 24)


0.1% is a lower bound prevalence estimate

The IQWiG report states that there are approximately 70,000 adult ME/CFS patients in Germany. This is based on a 0.1% prevalence estimate found by Nacul and colleagues in the UK using the Canadian Consensus Criteria. [1]

This study, however, did not include a community-based screening of the entire population for cases of ME/CFS. Instead, Nacul and colleagues relied on general practitioners (GPs) for the detection of possible ME/CFS cases. The authors explained that this methodology might have underestimated the true prevalence rate of ME/CFS “as some people may not consult their GPs, or do not receive a diagnosis.” The researcher highlighted that their estimate represents a “lower range of the commonly assumed population prevalence for the UK.” For this reason, they referred to it as “a minimum prevalence in primary care.” [1]

This view is confirmed by comparing the prevalence for the Fukuda 1994 case definition found by Nacul et al. with similar estimates found by other researchers. While Nacul and colleagues found a prevalence of 0.2%, other studies found significantly higher rates. A recent meta-analysis calculated a prevalence of 0.89% for the Fukuda case definition of ME/CFS, or more than 4 times higher than the estimate found by Nacul et al. [2]

The IQWiG report uses a lower limit of ME/CFS prevalence estimates without acknowledging that this is likely an underestimation of the true prevalence rate. We recommend adding an explanation that clarifies that the prevalence of ME/CFS may be significantly higher than 70,000 adult patients in Germany and that this figure represents a lower bound estimate.

References

[1] Nacul LC, Lacerda EM, Pheby D et al. Prevalence of myalgic encephalomyelitis/chronic fatigue syndrome (ME/CFS) in three regions of England: a repeated cross-sectional study in primary care. BMC Med 2011; 9(1): 91.

[2] Lim EJ, Ahn YC, Jang ES, Lee SW, Lee SH, Son CG. Systematic review and meta-analysis of the prevalence of chronic fatigue syndrome/myalgic encephalomyelitis (CFS/ME). Journal of translational medicine. 2020 Dec;18(1):1-5.

 

Continued from 4.2.2.5 Epidemiological and supply-relevant aspects – Prevalence - Page 24


The pediatric prevalence found by Jason et al.

The prevalence of pediatric ME/CFS was estimated by Jason and colleagues in 2020. [1] The authors screened 10,119 youth, aged 5–17 in the greater Chicago area. Diagnoses were made after medical examination by an experienced team of physicians. The prevalence of pediatric ME/CFS was estimated to be 0,75%, more than 7 times the estimate the IQWiG report uses for adults. A British study of children (aged 11-16) using data from school-based clinics, found a similar but higher prevalence estimate of 1% using the 2007 case definition specified
by the National Institute for Health and Care Excellence (NICE). [2]

It is unclear why the estimate of 0.75% for young people is not mentioned anywhere in the IQWiG report. The report only states that the results by Jason et al. 2020 are not reliable (“nicht zuverlässig”) without providing further explanation.

The IQWiG report also states that extrapolation of the data by Jason et al. 2020 is not possible due to lack of information on age standardization. We believe that the figure of 0.75% should still be mentioned in the report. In addition, we recommend contacting the research team of Prof. Leonard Jason at the DePaul University of Chicago for more information on the age distribution of patients in his study.


References

[1] Jason LA, Katz BZ, Sunnquist M, Torres C, Cotler J, Bhatia S. The prevalence of pediatric myalgic encephalomyelitis/chronic fatigue syndrome in a community-based sample. InChild & youth care forum 2020 Aug (Vol. 49, No. 4, pp. 563-579). Springer US.

[2] Crawley EM, Emond AM, Sterne JA. Unidentified chronic fatigue syndrome/myalgic encephalomyelitis (CFS/ME) is a major cause of school absence: surveillance outcomes from school-based clinics. BMJ open. 2011 Jan 1;1(2):e000252.

 

EMEC Comment:
4.3.2 Results on therapy options – Page 30

(Ergebnisse zu Therapieoptionen)


Outdated case definitions

The IQWiG report included trials that selected patients with outdated case definitions such as the Oxford criteria if 80% of the trial participants experienced PEM. However, patients selected with these case definitions, even if they experienced PEM, may still differ substantially from patients selected with modern diagnostic criteria.

The Oxford definition for example requires that chronic fatigue is the “principal symptom” of the syndrome, which may not be the case for many ME/CFS patients. [1] Studies have shown that only approximately half of patients selected with the Fukuda-criteria met more recent case definitions such as the Canadian Consensus Criteria (CCC). [2]

The relevance of such studies to ME/CFS, as it is currently defined, is therefore questionable. Requiring 80% of patients to experience PEM does not fully address this issue as it is only one of several core features of ME/CFS.

We recommend using a stricter threshold where at least 80% of included participants meet modern diagnostic criteria for ME/CFS such as the CCC, the International Consensus Criteria, or the 2015 National Academy of Medicine criteria. This would ensure that results are applicable to ME/CFS as it is defined and diagnosed today.


References

[1] Sharpe MC. A report–chronic fatigue syndrome: guidelines for research. Journal of the Royal Society of Medicine. 1991 Feb;84(2):118-21.

[2] Nacul LC, Lacerda EM, Pheby D et al. Prevalence of myalgic encephalomyelitis/chronic fatigue syndrome (ME/CFS) in three regions of England: a repeated cross-sectional study in primary care. BMC Med 2011; 9(1): 91.

 

EMEC Comment:
5. Benefit assessments – Page 98

Nutzenbewertungen

Non-blinded studies with subjective outcomes

The evidence for GET and CBT was extracted from 3 trials: PACE, GETSET, and the Dutch study by Janse et al. All 3 trials used subjective measurements as their main outcomes while patients and therapists could not be blinded to treatment allocation. Consequently, these studies are at high risk of response bias and have limited validity. Patients who know they are receiving an active intervention rather than a passive control might be more optimistic about its effect on their health or report symptoms according to what they think will please the investigators. Patients in the GET or CBT group might thus have rated their health as better than patients in the control group even if the treatment did not actually improve their health.

The General Methods handbook by IQWiG version 6.1 offers several options to evaluate evidence of trials where blinding of patients and therapists was not possible. On page 171, section 9.1.4 it states that “If a blinded collection of outcomes is not possible, an objective endpoint should be chosen that can be influenced as little as possible in terms of its characteristics and the stringency of the recording by the person who collects the endpoint.”

[Original quote in German: “Falls eine verblindete Zielgrößenerhebung nicht möglich ist, sollte ein möglichst objektiver Endpunkt gewählt werden, der in seiner Ausprägung und in der Stringenz der Erfassung so wenig wie möglich durch diejenige Person, die den Endpunkt (unverblindet) erhebt, beeinflusst werden kann.”]

Objective outcomes such as the 6-minute walking test or actigraphy, however, did not indicate a clinical effect for GET or CBT. Other reviews have demonstrated that, in contrast to subjective measurements, objective outcomes of GET and CBT trials do not indicate improvements. [1-2]

The General Methods handbook by IQWiG also states in section 3.2.4 on Patient-reported outcomes (PROs), page 61: “Since information on PROs is subjective by its very nature, open-label, ie unblinded, studies in this area are of limited validity. The size of the observed effect is an important decision criterion for the question of whether an indication of a benefit of an intervention with regard to PROs can be derived from open studies.”

[Original quote in German: “Da Angaben zu PROs aufgrund ihrer Natur subjektiv sind, sind offene, d. h. nicht verblindete Studien in diesem Bereich nur von eingeschränkter Validität. Für die Frage, ob sich aus offenen Studien ein Hinweis auf einen Nutzen einer Intervention bezüglich PROs ableiten lässt, ist die Größe des beobachteten Effekts ein wichtiges Entscheidungskriterium.”]

A 2014 review by Hróbjartsson et al. on randomized trials that compared blinded and non-blinded groups found that the average difference in effect size for PROs was 0.56. [3] In other words, in groups where patients were not blinded, the reported effect sizes were inflated by approximately half a standard deviation. This is similar to the effect size found in trials on GET and CBT. [3]

It is therefore doubtful if the differences found in such trials constitute a true treatment effect. For fatigue, for example, the SMD reported by IQWiQ was −0,48 [−0,71; −0,25] for the comparison CBT versus SMC at the 52-week follow-up and −0,37 [−0,54; −0,19] for GET versus SMC at the 12-weeks assessment. For the other primary outcome, physical function, the SMD was 0,32 [0,09; 0,55] for the comparison CBT versus SMC at the 52-week follow-up and 0.19 [0.02; 0.37] for GET versus SMC at the 12-weeks assessment. These effects could have fully been the result of bias due to a lack of blinding.

There are several reasons to think that response bias was particularly strong in trials of GET and CBT. First, as part of the treatment, patients were encouraged to interpret their ME/CFS symptoms as the reversible result of deconditioning, stress, anxiety, or disrupted sleep patterns rather than an unknown disease that cannot yet be treated. Such instructions might have led them to view their symptoms as more benign than patients in the control group who were not primed in this way.

According to therapist manuals used in the PACE trial “participants are encouraged to see symptoms as temporary and reversible, as a result of their current physical weakness, and not as signs of progressive pathology.” [4] The CBT used in the trial by Janse et al. was based on similar principles as it focused on correcting patients' unhelpful beliefs about their illness. Therapists were encouraged to suggest that recovery is possible to raise patients’ expectations which in turn would lead to a change in the perception of symptoms as well as disability. [5]

Second, therapists were explicitly instructed to encourage optimism and instill the belief that patients could get better through GET or CBT if they committed themselves to the treatment. The therapist manual on GET for example stated: “it is important that you encourage optimism about the progress that they may make with this approach. You can explain the previous positive research findings of GET and show in the way you discuss goals and use language that you believe they can get better.” [4] Similarly, the PACE trial manual for participants in the CBT arm stated that: “Cognitive behaviour therapy (CBT) is a powerful and safe treatment which has been shown to be effective in a variety of illnesses” and that “many people have successfully overcome CFS/ME using cognitive behaviour therapy”. [6] Such statements might have persuaded patients that GET and CBT were effective, even before treatment started.

Third, because patients were made to think that they were in large part responsible for the treatment effects of GET and CBT, they might have been reluctant to report no improvement on symptom questionnaires. It might have indicated that they, rather than the treatments, failed.

As one PACE trial participant testified: “After repeatedly being asked how severe my symptoms were […] I started to feel like I had to put a positive spin on my answers. I could not be honest about just how bad it was, as that would tell the doctors I wasn’t trying and I wasn’t being positive enough. When I was completing questionnaires, I remember second guessing myself and thinking for every answer: ‘Is it really that bad? Am I just not looking at things positively enough?’” [7]

The therapist manual used in the GETSET trial stated: “It is essential that you demonstrate positive reinforcement when you work with patients. Often, patients may be more inclined to focus on what they have not achieved rather than what they have. It is therefore important that you emphasise and are very positive about what they have achieved. Every session you should positively reinforce all of their achievements.” [8] Such instructions can severely distort how patients rate their improvement or complete symptom questionnaires.

We recommend focusing on objective outcomes in trials where patients nor therapists could be blinded. Small to moderate effect sizes on subjective measurements such as symptom questionnaires are not reliable indicators of treatment effects in unblinded trials. The IQWiG report on ME/CFS, however, seems, to have based its recommendations for GET and CBT mostly on such questionable outcomes.


References

[1] Vink M, Vink-Niese A. Graded exercise therapy for myalgic encephalomyelitis/chronic fatigue syndrome is not effective and unsafe. Re-analysis of a Cochrane review. Health Psychol Open. 2018;5; 2055102918805187.

[2] Wiborg JF, Knoop H, Stulemeijer M, Prins JB, Bleijenberg G. How does cognitive behaviour therapy reduce fatigue in patients with chronic fatigue syndrome? The role of physical activity. Psychological medicine. 2010 Aug;40(8):1281-7.

[3] Tack M, Tuller DM, Struthers C. Bias caused by reliance on patient-reported outcome measures in non-blinded randomized trials: an in-depth look at exercise therapy for chronic fatigue syndrome. Fatigue: Biomedicine, Health & Behavior. 2020 Oct 1;8(4):181-92.

[4] Bavinton J, Darbishire L, White PD. PACE manual for therapists. Graded Exercise Therapy (GET) for CFS/ME. Final trial version: version 7 (MREC Version 2). PACE Trial Management Group; 2004. Available at: https://me-pedia.org/images/8/89/PACE-get-therapist-manual.pdf

[5] Knoop H, Bleijenberg G, Gielissen MF, van der Meer JW, White PD. Is a full recovery possible after cognitive behavioural therapy for chronic fatigue syndrome?. Psychotherapy and psychosomatics. 2007;76(3):171-6. Available at: https://www.researchgate.net/public...avioural_Therapy_for_Chronic_Fatigue_Syndrome

[6] Burgess M, Chalder T. PACE manual for participants. Cognitive Behavior Therapy for CFS/ME. MREC version 2 – November 2004 Available at: https://me-pedia.org/images/7/7a/PACE-cbt-participant-manual.pdf

[7] UK Parliament. PACE Trial: People with ME Volume 636: debated on Tuesday 20 February 2018. Transcript available at: https://hansard.parliament.uk/commo...-4566-940d-249f5026ff73/pacetrialpeoplewithme



[8] Clark LV, Tims E, White PD On behalf of the GETSET Trial Management Group. GETSET Study. Guided graded exercise self-help (GES) for CFS/ME. Therapist Manual. Available at: https://www.qmul.ac.uk/wiph/media/w.../GETSET-therapists-manual-with-appendices.pdf

 

Continued from 5 Benefit assessments (Nutzenvewertungen) – Page 98

No control intervention

An additional concern is that the data on GET and CBT did not include a control intervention.

In the Dutch trial by Janse and colleagues, for example, patients in the control group were simply put on a waiting list to receive web-based CBT, the intervention that patients in the other 2 groups did receive.

In the GETSET trial, specialist medical care (SMC) with guided graded exercise self-help compared with SMC alone. In other words, this trial design used an unfair comparison of ‘A versus A + B’ which is unsuited to measure the efficacy of an intervention.

Something similar was true for the data extracted from the PACE trial. Patients in the control group only received SMC which patients in the intervention group also received, in addition to GET or CBT.

The General Methods handbook by IQWiG version 6.1 argues that a control group is the first requirement for assessing the benefits of interventions. It states that: “An indispensable condition for proving causality is a comparative experiment that must be designed in such a way that a difference between intervention groups - an effect - can only be attributed to a single influencing variable - the tested intervention.”

[Original quote in German: “Unverzichtbare Bedingung für den Nachweis von Kausalität ist ein vergleichendes Experiment, das so angelegt sein muss, dass ein Unterschied zwischen Interventionsgruppen – ein Effekt – nur auf eine einzige Einflussgröße – die geprüfte Intervention – zurückgeführt werden kann. Dieses Ziel macht für klinische Studien erhebliche Anstrengungen nötig, weil es zahlreiche unerwünschte Einflüsse gibt, die einen Effekt vortäuschen oder auch verbergen können (Verzerrung).”]

This is not the case in these trials. Because the control group received no intervention, improved symptom scores might have resulted from patients’ expectations or increased attention and contact with their healthcare providers.