Germany: IQWIG Report to government on ME/CFS - report out now May 2023

EMEC Comment:
5.7 Summary evaluation of the results – comparison CBT versus SMC – Page 149

Zusammenfassende Bewertung der Ergebnisse - Vergleich CBT versus SMC– Page 149


The data on CBT does not indicate a treatment effect

The IQWiG report concludes on page 149: “When all results are weighed up across outcomes, there is a hint of a benefit of CBT compared to SMC for patients with mild to moderate ME/CFS severity, both in the short and medium term.”


[Original quote in German:

“Bei endpunktübergreifender Abwägung aller Ergebnisse ergibt sich für Patientinnen und Patienten mit leichtem bis moderatem ME/CFS-Schweregrad sowohl kurz- als auch mittelfristig ein Anhaltspunkt für einen Nutzen der CBT im Vergleich zur SMC.”]


In the appendix intended to appear on www.gesundheitsinformation.de, CBT is presented as a recommended treatment option. It is stated that studies have shown that CBT can help patients with mild to moderate ME/CFS to reduce their symptoms. The text states: “individual studies indicate that cognitive behavioral therapy and physical activation can help some people with mild to moderate ME/CFS to at least temporarily reduce certain symptoms.”


[Original quote in German: “Studien deuten aber darauf hin, dass die kognitive Verhaltenstherapie und die körperliche Aktivierung einigen Betroffenen mit leichter bis mittelschwerer ME/CFS helfen kann, bestimmte Beschwerden zumindest vorübergehend etwas zu mindern.”] Both statements are incorrect and should be deleted.


The literature review on CBT identified only 2 randomized trials: the PACE trial and the Dutch study conducted by Janse and colleagues. Despite the risk of bias in both studies, most of the primary and secondary outcomes failed to show a clinical benefit of CBT. At long-term assessments, not a single outcome measure indicated a benefit of having received CBT. The control group performed just as well as the CBT group.


We recommend that the IQWiG report explains that there is no scientific evidence that CBT improves symptoms of ME/CFS. In the long run, the control group that did not receive an intervention performed just as well as the patients who received CBT. While ME/CFS patients should be able to receive psychological support if requested, IQWiG should clarify that there is no scientific evidence for using CBT to treat ME/CFS.

 

EMEC Comment:

5.7 Summary evaluation of the results – comparison GET versus SMC – page 150

5.7 Zusammenfassende Bewertung der Ergebnisse – Vergleich GET versus SMC– S.150


GET data shows that it is not an effective treatment for ME/CFS


The IQWiG report concludes on page 150: “When all results are weighed up across outcomes, there is a hint of a benefit of GET compared to SMC for patients with mild to moderate ME/CFS severity in both the short and medium term”.

[Original quote in German:
“Bei endpunktübergreifender Abwägung aller Ergebnisse ergibt sich für Patientinnen und Patienten mit leichtem bis moderatem ME/CFS-Schweregrad sowohl kurz- als auch mittelfristig ein Anhaltspunkt für einen Nutzen der GET im Vergleich zur SMC.”]

This is an incorrect statement that cannot be justified by the evidence summarized in this review. In fact, the evidence review by IQWiG suggests the opposite, namely that GET is not an effective treatment for ME/CFS.

Only two randomized trials on GET were identified by the literature search: the PACE and GETSET trials, both conducted by the research team of Prof. Peter White in the UK. Despite the high risk of bias in both studies, the evidence review conducted by IQWiG found no hint of a benefit on the primary outcomes of the PACE and GETSET trials, namely fatigue and physical function. This was the case for the short-term, medium- and long-term assessments.

The same was true for almost all secondary outcomes namely pain, sleep, mental status, physical activity, physical performance results, cognitive function results, and social participation. The only exceptions where GET temporarily showed a hint of an effect compared to SMC were the global clinical improvement (GCI) scale and the outcome ‘feeling sick after exertion’.

The GCI scale was a secondary outcome in both the PACE and GETSET trial and is problematic as it arbitrarily reduces a 7-point scale to a percentage improvement score. The GCI scale allows participants to indicate how they felt after treatment using 7 options: very much better, much better, a little better, no change, a little worse, much worse or very much worse. The data used was the percentage of patients reporting the first two options (very much better or much better) in the GET versus SMC group. The option “a little better” was disregarded. Nonetheless, there was only an apparent effect of GET at the 12 weeks short-term assessment. For the medium and long-term assessments, no hint of a benefit could be derived.

The IQWiG report reviewed more than 10 outcome measures at 3 different assessment periods for the comparison of GET versus SMC, without applying a correction for multiple comparisons. At 12 weeks, the PACE trial was still ongoing, and patients only received half of their treatment sessions. It is therefore improbable that the higher percentage of improvers in the GET versus SMC group at 12 weeks reflects a true treatment effect.

The measurement ‘feeling sick after exertion’ was only used in the PACE trial. It was not assessed post-treatment or at long-term follow-up, only at the assessment 52 weeks after randomization. This measurement was not among the 14 primary or secondary outcomes measurements registered in the PACE trial protocol or statistical analysis plan. [1-2] It seems to have been added post-hoc by the PACE researchers and may thus have been the result of cherry-picking. The IQWiG report should not have included this outcome to assess the efficacy of GET.

The data summarized above fails to provide reliable evidence for the effectiveness of GET. Both the PACE and GETSET trial reported that the SMC group performed equally well as the GET group at the longest available follow-up. We recommend that the IQWiG report informs healthcare professionals in Germany that, despite two large trials conducted in the UK, there is no reliable evidence that GET improves ME/CFS symptoms or disability.


References

[1] White PD, Sharpe MC, Chalder T, DeCesare JC, Walwyn R. Protocol for the PACE trial: A randomised controlled trial of adaptive pacing, cognitive behaviour therapy, and graded exercise as supplements to standardised specialist medical care versus standardised specialist medical care alone for patients with the chronic fatigue syndrome/myalgic encephalomyelitis or encephalopathy. BMC neurology. 2007 Dec;7(1):1-20.

[2] Walwyn R, Potts L, McCrone P, Johnson AL, DeCesare JC, Baber H, Goldsmith K, Sharpe M, Chalder T, White PD. A randomised trial of adaptive pacing therapy, cognitive behaviour therapy, graded exercise, and specialist medical care for chronic fatigue syndrome (PACE): statistical analysis plan. Trials. 2013 Dec;14(1):1-23.

 

EMEC Comment:

A Details of the report/ A2 Method according to report plan/ A2.3 Benefit assessment/ A2.3.3 Information evaluation and synthesis.

A2.3.3.6 Statements on the evidence situation – Page 180

A2.3.3.6 Aussagen zur Beleglage – S.180

Biased assessment of results

The IQWiG report uses statistical significance to determine the strength of evidence. Table 54 on page 180, for example, shows that evidence is interpreted differently if results are statistically significant. We believe this has resulted in a bias where IQWiG undervalues trial results in which the data indicates, with sufficient precision, that there is no clinical difference between treatment arms.

An example may illustrate this point more clearly. The long-term follow-up results of GET versus SMC for the outcome of physical function resulted in an estimate of 2,0 [−4,0; 7,9] on the subscale SF-36 PF. The IQWiG report only notes that this was not a statistically significant result and therefore it is largely ignored in the rest of the document.

The confidence interval, however, excludes what is considered a ‘minimal clinically important difference (MCID)’ for this scale (usually around 8-10 points). [1] It thus indicates with sufficient precision that there is no clinical effect of GET compared to SMC at long-term follow-up. However, because this finding indicates no difference between groups rather than a treatment effect, it is disregarded in the rest of the report. There are other examples of this in the evidence review, depending on what one considers to be the MCID on the relevant outcome measure.

We recommend that IQWiG report no longer uses the arbitrary threshold of statistical significance to determine the strength of evidence. Evidence should be evaluated based on factors such as risk of bias or precision, not on the type of hypothesis it supports. Evidence that a treatment is not effective is also valuable information that should be shared with the public and healthcare professionals.


References

[1] Tack M, Tuller DM, Struthers C. Bias caused by reliance on patient-reported outcome measures in non-blinded randomized trials: an in-depth look at exercise therapy for chronic fatigue syndrome. Fatigue: Biomedicine, Health & Behavior. 2020 Oct 1;8(4):181-92.

 

EMEC Comment:

6 Discussion / classification of the work result/ Dealing with critical comments on ME/CFS studies -- Argument: "lack of consideration of patient surveys" – Page 157

6 Diskussion / Einordnung des Arbeitsergebnisses - Umgang mit kritischen Anmerkungen zu ME/CFS-Studien - Argument: „fehlende Berücksichtigung von Patientenbefragungen“ – Page 157


Reports of harms incorrectly dismissed

In multiple surveys, ME/CFS patients reported that GET worsened their health. The IQWiG report dismisses these reports using several questionable arguments.

First, the IQWiG report questions that the surveys indicate harm. It uses the 2011 review by Tom Kindlon to argue that an almost equally large proportion of respondents reported that GET improved their health. Between 28.1% and 82.0% of those questioned stated that their condition had deteriorated after GET while 13.1% to 60.8% stated that GET had improved their health. We find this comparison somewhat misleading as the percentage of participants reporting harm was quite consistently higher than the percentage saying GET was helpful if each survey is looked at separately.

The 2017 overview provided by Geraghty et al. shows that when respondents to all surveys on GET were added together, 57% reported deterioration after GET while only 26% found it helpful. [1] More recent studies have only strengthened this view. In a 2019 survey developed for the NICE Guideline Development Group and conducted by researchers from Oxford Brookes University, for example, 85% of respondents reported that GET worsened their symptoms. [2]

Second, the IQWiG report suggests that reports of harm were due to inappropriate implementation of GET and that exercise should therefore be supervised by an experienced physiotherapist. The data, however, strongly speaks against this hypothesis.

The 2019 survey by Action for ME, for example, found that “even when people are supported by an M.E. specialist, only one in 10 reported that GET helped manage symptoms, while nearly half reported a worsening effect.” [3]

Similarly, in a 2015 survey by the ME Association, “GET courses held by therapists stated to have an ME/CFS specialism made symptoms worse for 57% of respondents.” [4] In an earlier survey by Action for ME there was little difference in the reported rate of harms by GET whether the treatment was delivered by an NHS specialist (31%), the GP (45%) or others (29%). [5] To our knowledge, there is no data to support the claim that reports of harm are mainly due to improper implementation of GET. This argument seems to originate from two papers by Clark & White in which they misinterpreted the results of a small Action for ME survey from 2003. Kirke later clarified this misunderstanding, explaining that in the 2003 survey:“only 1 of 12 patients who did GET with ‘no professional’ reported a negative outcome, compared to 12 of 18 patients who did GET with the supervision of a physiotherapist.” [6]

Third, the IQWiG report claims that surveys are “are methodologically unsuitable for deriving reliable statements on the benefit or harm of a treatment”

[Original quote in German:

“Grundsätzlich sind Befragungen (Querschnittsstudien, retrospektive Vorher-Nachher Untersuchungen) in der Regel methodisch ungeeignet, zuverlässige Aussagen zum Nutzen oder Schaden einer Behandlung abzuleiten”].

While this may be true for measuring the benefits of treatments, the detection of adverse effects frequently relies on observational studies. [7] Harms are often poorly recorded in randomized treatment trials.

According to a 2014 report by the Agency for Healthcare Research and Quality (AHRQ) “observational studies are almost always necessary to assess harms adequately.” [8] The report clarified that observational studies include “a broad range of study designs, including case reports, uncontrolled series of patients receiving surgery or other interventions, and others. All can yield useful information as long as their specific limitations are understood.” [8]

Similarly, AMSTAR-II emphasizes that “the failure to include non-randomised studies in a review of adverse outcomes of treatment may be a critical flaw.” [9]

It is quite unusual that surveys have consistently pointed towards adverse effects of GET for a period spanning almost three decades and in multiple countries including the United Kingdom, Norway, The Netherlands, and Australia. There is no clear explanation for why patients would report harms from GET but not from other interventions. In a 2010 survey by the ME Association, for example, GET was reported to be more harmful than pharmacological treatments with known side effects such as hydrocortisone, thyroxine or modafinil. [11]

Similarly, in a 2014 survey by Action for ME, GET was rated as more harmful than pain- and sleep medication. [12] This suggests that reports of harm following GET cannot be fully explained by selection bias.

Randomized trials on GET have not reported an increase in adverse effects but the data they provided is quite limited. Adverse effects were only monitored in two trials, the PACE and GETSET trial. The GETSET trial did not assess the safety of GET but a self-help guide to exercise. Therefore, its findings may not be generalizable to full courses of GET in clinical practice. And as noted by Kindlon in 2017, data on harms in the PACE trial was not reported in accordance with its pre-specified protocol. [10] GETSET and PACE also did not provide objective evidence of adherence to GET.

We recommend that IQWiG takes the reports of harms from observational studies into consideration. Relying solely on randomized controlled trials to evaluate harms can be misleading, especially if there is a large and consistent literature that points to serious adverse effects.


References

[1] Geraghty K, Hann M, Kurtev S. Myalgic encephalomyelitis/chronic fatigue syndrome patients’ reports of symptom changes following cognitive behavioural therapy, graded exercise therapy and pacing treatments: Analysis of a primary survey compared with secondary surveys. Journal of health psychology. 2019 Sep;24(10):1318-33.

[2] Evaluation of a survey exploring the experiences of adults and children with ME/CFS who have participated in CBT and GET interventional programmes. Final Report. Submitted by Oxford Clinical Allied Technology and Trials Services Unit (OxCATTS), Oxford Brookes University, 27th February 2019. Available at: https://www.meassociation.org.uk/wp...d-GET-Oxford-Brookes-Full-Report-03.04.19.pdf

[3] GET and CBT for people with M.E. Action for M.E.'s Big Survey: two-page explainer. Available at: https://www.actionforme.org.uk/uploads/images/2020/02/Big-Survey-GET-and-GET-for-people-with-ME.pdf

[4] ME Association. ME/CFS Illness Management Survey Results “No decisions about me without me [Internet]. 2015. Available from: https://www.meassociation.org.uk/wp...No-decisions-about-me-without-me-30.05.15.pdf

[5] Action for ME and Association of Young People with ME (2008) M.E. 2008: What progress? Initial findings of a national survey of over 2,760 people with M.E. focusing on their health and welfare

[6] Kirke KD. PACE investigators’ response is misleading regarding patient survey results. Journal of health psychology. 2017 Aug;22(9):1168-76.

[7] Vandenbroucke JP, Psaty BM. Benefits and risks of drug treatments: how to combine the best evidence on benefits with the best data about adverse effects. Jama. 2008 Nov 26;300(20):2417-9.

[8] [1] Agency for Healthcare Research and Quality (AHRQ). Methods Guide for Effectiveness and Comparative Effectiveness Reviews. 2014. https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/cer-methods-guide_overview.pdf

[9] Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, Moher D, Tugwell P, Welch V, Kristjansson E, Henry DA. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. bmj. 2017 Sep 21;358.

[10] Kindlon T. Do graded activity therapies cause harm in chronic fatigue syndrome?.Journal of health psychology. 2017 Aug;22(9):1146-54.

[10] ME Association. Managing my M.E. What people with ME/CFS and their carers want from the UK’s health and social services [Internet]. 2010. Available from: https://www.meassociation.org.uk/wp-content/uploads/2010/09/2010-survey-report-lo-res10.pdf

[11] Action for ME. M.E. Time to deliver. [Internet]. 2014. Available from: https://www.actionforme.org.uk/uploads/pdfs/me-time-to-deliver-survey-report.pdf

 

EMEC Comment:

Appendix: ‘Health information’ -- Treatment of ME/CFS page 1 (page 287 of PDF)

Appendix: ‘Gesundheitsimformation’ -- Treatment of ME/CFS page 1 (page 287 of PDF)


Statements on revalidation [?] and return to work unjustified

The draft for www.gesundheitsinformation.de suggests that rehabilitation helps ME/CFS patients to maintain employability and that CBT can help patients return to school or work earlier. These are misleading statements that should be deleted.

Employment has been included in multiple randomized trials of rehabilitative interventions for ME/CFS. Almost all have reported that these treatments do not improve the ability to maintain employment or return to work. The most recent review of this subject by Vink & Vink-Niese concluded that “cognitive behavioural therapy and graded exercise therapy do not restore the ability to work.” [1]

Of the randomized trials included in the IQWiG report, the PACE trial reported data on employment and welfare benefits. These showed no difference between the intervention groups (CBT and GET) and the control group (SMC). [2]

Incorrect statements suggesting that rehabilitative interventions such as GET or CBT help patients to maintain or return to work can have a dramatic impact on their lives. It may not only result in false hope or wasteful investments but can also lead to healthcare professionals questioning patients’ recovery behavior and an unjust denial of disability benefits. We, therefore, recommend IQWiG to only make evidence-based statements on the topic of rehabilitation and ME/CFS.

References

[1] Vink M, Vink-Niese F. Work rehabilitation and medical retirement for myalgic encephalomyelitis/chronic fatigue syndrome patients. A review and appraisal of diagnostic strategies. Diagnostics. 2019 Sep 20;9(4):124.

[2] McCrone P, Sharpe M, Chalder T, Knapp M, Johnson AL, Goldsmith KA, White PD. Adaptive pacing, cognitive behaviour therapy, graded exercise, and specialist medical care for chronic fatigue syndrome: a cost-effectiveness analysis.

 

EMEC Comment:

6 Discussion / classification of the work result/ Dealing with critical comments on ME/CFS studies - P 155

Umgang mit kritischen Anmerkungen zu ME/CFS-Studien - P 155

The PACE trial, a questionable source of evidence

The evidence review by IQWiG heavily depends on the PACE trial; a controversial study that was criticized for having various methodological flaws. In evaluating the shortcoming of the PACE trial, IQWiG focuses on a paper by Friedberg and colleagues. This brief commentary, however, does not give an overview of all essential points of the debate.

It does not discuss, for example:

• that the PACE authors published a newsletter for participants that included positive testimonials from earlier participants about the benefits of the interventions.

• that the PACE trial used outcome thresholds for being within the normal range on the two primary measures of fatigue and physical function that demonstrated worse health than the criteria for entry.

• that the PACE authors failed to adhere to the Declaration of Helsinki.

To get a more comprehensive overview of the debate, we refer to a special issue of the Journal of Health Psychology on the PACE trial (Volume 22 Issue 9, August 2017).

 

EMEC Comment:

4.3.2.1.4 Lightning Process – P 48

4.3.2.1.4 Lightning Process – P 48


Problems with the SMILE trial

The SMILE trial included in the IQWiG also suffered from severe methodological shortcomings. As explained by Tuller D: “The investigators recruited more than half of the participants before trial registration, swapped primary and secondary outcomes after gathering data from the early recruits, and then failed to disclose these critical details in the paper.” [1] These issues were not mentioned in the evidence review conducted by IQWiG.


References

[1] Tuller D. 2019. BMJ should retract flawed research paper on chronic fatigue syndrome. Statnews.com.
https://www.statnews.com/2019/12/13...tigue-syndrome-research-paper/comment-page-2/

 

EMEC Comment:

4.2 Current state of knowledge/ 4.2.2 Results on the topics of the current state of knowledge/

4.2.2.2 Complaints / Symptoms – Degrees of severity of the disease P 9

Schweregrade der Erkrankung – P 9


No section on severe ME/CFS

The IQWiG report lacks a separate section devoted to the care of patients with very severe ME/CFS as was included in the NICE guidance. Patients with very severe ME/CFS need special care from healthcare providers. They require a low-stimulus environment and may suffer from extreme symptoms such as an inability to speak or swallow.

More information on this patient group was provided in a special issue "ME/CFS – the Severely and Very Severely Affected" of the Journal Healthcare (March 2021).

 

EMEC Comment:

4.2 Current state of knowledge/ 4.2.2 Results on the topics of the current state of knowledge/

4.2.2.4 Etiology / causes – P 21

4.2.2.4 Ätiologie / Ursachen – P 21



Longitudinal EBV-studies

The IQWiG report understates the evidence that an Epstein-Barr Virus (EBV) infection can act as a trigger of ME/CFS. It states, for example on page 21, that “herpes viruses, Epstein-Barr viruses or SARS-CoV-2 are suspected to be triggers of ME/CFS.” [Original quote in German: “Herpes-Viren, Epstein-Barr-Viren oder SARS-CoV-2 werden als Auslöser von ME/CFS vermutet.”]

Multiple longitudinal studies, however, have confirmed an increased prevalence of ME/CFS following EBV infection. [1-4] This is one of the few findings on ME/CFS that were replicated by multiple research teams. We think this could have received more attention in the IQWiG report.


References

[1] Buchwald DS, Rea TD, Katon WJ, Russo JE, Ashley RL. Acute infectious mononucleosis: characteristics of patients who report failure to recover. The American journal of medicine. 2000 Nov 1;109(7):531-7.

[2] Hickie I, Davenport T, Wakefield D, Vollmer-Conna U, Cameron B, Vernon SD, Reeves WC, Lloyd A. Post-infective and chronic fatigue syndromes precipitated by viral and non-viral pathogens: prospective cohort study. Bmj. 2006 Sep 14;333(7568):575.

[3] Katz BZ, Shiraishi Y, Mears CJ, Binns HJ, Taylor R. Chronic fatigue syndrome after infectious mononucleosis in adolescents. Pediatrics. 2009 Jul;124(1):189-93.

[4] Kristiansen MS, Stabursvik J, O'Leary EC, Pedersen M, Asprusten TT, Leegaard T,Osnes LT, Tjade T, Skovlund E, Godang K, Wyller VB. Clinical symptoms and markers of disease mechanisms in adolescent chronic fatigue following Epstein-Barr virus infection: an exploratory cross-sectional study. Brain, behavior, and immunity. 2019 Aug 1;80:551-63.

[5] White PD, Thomas JM, Amess J, Crawford DH, Grover SA, Kangro HO, Clare AW. Incidence, risk and prognosis of acute and chronic fatigue syndromes and psychiatric disorders after glandular fever. The British Journal of Psychiatry. 1998 Dec;173(6):475-81.

 

Perhaps it could be worth pointing out that, in addition to the NICE guideline, EUROMENE’s consensus statement on the treatment of ME also contains recommendations on severe and very severe ME? It shows that not only patients but also clinicians specializing in ME/CFS support such recommendations. https://www.mdpi.com/1648-9144/57/5/510/htm

 

Trial By Error: German Draft Report on ME/CFS Raises Alarms for Promoting CBT and GET

By David Tuller, DrPH

More at link:
https://www.virology.ws/2022/10/28/...-cfs-raises-alarms-for-promoting-cbt-and-get/

 

Some thoughts on PEM

Assessment of PEM was not adequate

In contrast to the quote above, on page 139 (pdf 159) of the draft IQWiG themselves recognize that the measure for PEM in PACE was too broad.

Spoiler: Excerpt from p. 139 (pdf 159)

There is good reason for caution.

PACE: Fukuda (1994) and Reeves (2003)

White (2011) state that they used CDC (Fukuda 1994) criteria with update from Reeves (2003).

Yet, the PEM item "post-exertional malaise lasting more than 24 hours" (Fukuda (1994)) was changed to "feeling ill after exertion" (PACE).
They diverged in three aspects from the Fukuda (1994) critera and the recommendations in Reeves (2003):

1. Retrospective assessment
   
2. Duration
   
3. Not specifying the PEM item

Spoiler: Excerpt from Reeves (2003)

"Indeed, post-exertional malaise lasting more than 24 hours is one of the accompanying symptoms that define CFS. Therefore, this requirement should be interpreted as referring to exhaustion unrelated to an excessively demanding schedule that would induce fatigue in an otherwise healthy adult."

PACE: 1. Retrospective assessment
By adapting the Fukuda (1994) criteria in PACE they didn't assess if the patients had PEM in general, but whether they "have had any of the following symptoms in the last week". Due to an FOI request, we know checking "more often than not" or more frequently resulted in a symptom being counted as present.

PACE: 2. Duration
The requirement of the Fukuda (1994) PEM criterion "lasting more than 24 hours" was dropped.

PACE: 3. Not specifying the PEM item
In Reeves (2003) they state regarding PEM that "this requirement should be interpreted as referring to exhaustion unrelated to an excessively demanding schedule that would induce fatigue in an otherwise healthy adult."

The last part is necessary to delimit normal exercise reactions, but wasn't included.

PACE: Consequences of PEM assessment

It is possible that some people with ME/CFS may have not checked the PEM item in PACE when having no/few PEM events in the last week. On the other hand, as ME/CFS is quite rare amongst people with fatigue, one could assume that there is a way higher risk of PEM over-reporting by using overly broad PEM criteria.

Experiencing a vague form of an exercise induced feeling of illness of unspecified duration in the last week "more often than not" being counted as PEM is not adequate. It appears contradictory that IQWiG recognized the over-reporting of PEM in PACE, yet accepted the PEM survey as reliable enough to allow study inclusion.

PACE: Reporting of the changes made

The PACE authors did not disclose the changes in the main publication or the published protocol. It is only known because the actual questionnaires in "PACE final protocol vers. 5" were uploaded to S4ME (CDC criteria on p.156).

The changes are especially odd as Prof. White is the lead author of PACE and co-author of Reeves (2003) and GETSET.

PACE: Self-assessment vs. external assessment

It seems likely that the Fukuda (1994) criteria were used to calculate the proportion of people with PEM in PACE. Only Fukuda criteria were cited in the PEM part of the outcome chapter in White (2011) and no further information was given in White (2007, study protocol).

The (adapted Fukuda-) CDC questionnaire appears to have been filled out by patients while the London criteria were queried by a research nurse. Given the apparent vague and unspecific view taken towards PEM in the PACE trial, we cannot be confident this was done adequately.

GETSET

Patients were included if they met NICE (2007) criteria for ME/CFS. The NICE (2007) criteria state regarding PEM (my bolding): "characterised by post-exertional malaise and/or fatigue (typically delayed, for example by at least 24 hours, with slow recovery over several days)". Yet, in Clark (2017) and Clark (2016, study protocol) the authors only mention: mandatorily "be characterized by postexertional fatigue or malaise".

The description in the brackets of the NICE (2007) criteria is most relevant to distinguish between PEM and e.g. feeling a bit fatigued for a couple of minutes. We don't know which question they asked. It is possible the time duration was once again dropped by the team around Prof. White, thus widening the criterion.

Janse 2018

Updated (Reeves 2003) Fukuda (1994) criteria were used.
Reeves 2003 does not define a specific question for assessing PEM. It's unknown which items were used in the questionnaires.

Conclusion

One example: A very inactive and severely deconditioned individual without ME/CFS regularly experiences 5 minutes of fatigue and light nausea after climbing the stairs. After 10 minutes the person feels well again.
The person would not fit the Fukuda (1994), NICE 2007, CCC, ICC, IoM 2015, or NICE 2021 PEM criterion.

Yet:

• The person would fit all PEM criteria used in PACE
  
• The person would possibly fit the PEM criterion in GETSET - depending if they assessed full NICE (2007) criteria or only "postexertional fatigue or malaise" without further explanation
• The person would possibly fit the PEM criterion in Janse (2018): If the PEM criterion was designed like in PACE the person would, if they the used original Fukuda (1994) criteria the person would not.

It is necessary to know the actual questions used to assess PEM in GETSET and Janse (2018) to reliably evaluate population indirectness. Without detailed information on how PEM was assessed no conclusion about potential treatment benefits for people with ME/CFS should be drawn.

In general, the lack of a time duration requirement in the assessment of PEM in PACE is in steep contrast to all modern PEM criteria (NICE 2007, CCC, ICC, IoM 2015, NICE 2021) and even Fukuda (1994). All those criteria draw a distinction to normal exercise reactions (see table below) - PACE-CDC and London criteria don't. The research case definitions should be narrower than clinical ones - here it's exactly the other way around. Even 5 minutes of feeling unwell could have been counted as PEM in PACE - and possibly in GETSET and Janse (2018).

To be considered "adequately" selective, PEM criteria should have specifications that exclude normal exercise reactions, most importantly contain a time duration requirement.

(all from a very amateur perspective)

Spoiler: Overview table: PEM criteria

 

Seeing stuff like that is hair-raising. I can't even imagine this thought process in any other profession. The idea of saying "this isn't well-defined but whatever we'll use it anyway" is just so many layers below the bare minimum it almost feels like a trap asking about it. Like admitting the studies are poor, unreliable and biased, but using them anyway, despite many other fatal flaws, to make real-life decisions about millions of people. Absurd.

"This machine may or may not work, according to standards no one is sure about, but some people say it works, at their labs, they can't show it, though, so we'll take their work for it and pretend it's good enough so we'll standardize its use starting now and will never look back at whether it actually works."

Seriously, medicine has the absolute lowest standards of any profession. And some of the highest. All mixed in together, with no differentiation anymore, all evidence is evidence, even bad evidence. Science? Pseudoscience? Sounds similar if you don't focus on it.

 

This all looks correct, great work. It looks like they relied on the PEM survey referred to in the iqwig preliminary report to determine how many people had PEM in PACE, which, as you've explained (and iqwig have explained too), is not at all sufficient for that purpose. It's a very big limitation to the report (unless there's something I'm missing)

If you would like a bit of feedback, this part:

I would tone down a bit to avoid making it sound like you're making personal comments on people, to make it seem more professional. Something like "Given the apparent vague and unspecific view taken towards PEM in the PACE trial, we cannot be confident this was done adequately" instead perhaps. You could also point out that even if the London Criteria was done adequately only about 50% met that criteria. It looks like they assessed the proportion with PEM using the CDC questionnaire based on the Fukuda criteria though since that's the criteria they cite after stating that secondary outcome in White 2011

For anyone wondering the relevant page number in this document is page 156, it'd be worth including that

 

The EMEC response is strong. @Pustekuchen's notes on PEM are great and worth submitting (perhaps with @petrichor's tweak for tone).

I hope our German friends can see this made right! Hopefully it's not too late. The whole methodology seems dodgy to me, though -- they chose the 80% PEM threshold after seeing the results of the NICE guideline, for example. They cherry picked the parts of our review they wanted to use and then made up their own rules for the rest of it.

 

Thank you, @petrichor! All good points, I'll edit the post in the next days.

@adambeyoncelowe Thanks!
The 80% are actually set in their general methods paper. I'm note sure if this makes the whole thing better of worse though?

 

We discussed the 80% threshold a few pages back — I believe it was @Pustekuchen who showed that IQWiG followed their methodological guideline and did not arbitrarily choose the 80% inclusion threshold.

 

https://twitter.com/user/status/1587810879046881286

 

Do you have a link I can check? It seems to me they would've known what our guideline said, and how we did it, since they used our evidence review as the basis for their own.

They must've also known an 80% threshold would favour the trials we downgraded?

Or do you mean they always intended to use the 80% figure before NICE had published anything?

The way I see it, if they wanted to use the NICE evidence to base their decision on, then they should've taken it as a whole, instead of picking the bits they liked.

The guidelines were the result of a whole process with lots of moving parts. Take the moving parts out, and it stops working as it's supposed to.

 

That seems to be the case -- see their "general methods paper";

English language version of the general methods available directly from IQWiG: https://www.iqwig.de/methoden/general-methods_version-6-1.pdf

The update on the general methods paper more or less coincided with the IQWiG's work on the ME/CFS draft, though; so to be completely sure that the latter didn't influence the general methods I've checked their previous version of the methods paper -- they used the same 80% figure also in the older version of the methods paper (p. 171 of the pdf / p. 155 their page number)

https://www.iqwig.de/methoden/general-methods_version-6-0.pdf

Edit / info on older methods paper added