Why the Cochrane review on exercise therapy for chronic fatigue syndrome is still misleading

[ME/CFS Skeptic]

I've written a blog post about the recent amendment to the Cochrane review and how it doesn't address the major flaws. I will post the full text below as this makes it easier to quote and discuss. https://mecfsskeptic.wordpress.com/...chronic-fatigue-syndrome-is-still-misleading/

I hope you don't mind that I've posted this in a separate thread. The one on the Cochrane review is by now more than 15 pages long and includes long discussions on complicated details so I suspect that some patients will have signed off to it.

In the blog post, I try to explain the problem as I would to an outsider. I've tried to keep it simple and focus on the main issues. Many problems with the review are not mentioned such as the reliance on the flawed Chalder Fatigue Scale or the lack of measures of compliance. Readers interested in that will have to dive into the long thread on the Cochrane review.

Feel free to point out mistakes regarding both content and spelling/grammar.


Why the Cochrane review on exercise therapy for chronic fatigue syndrome is still misleading
On Wednesday, October 2, Cochrane published a long-awaited amendment to its review of exercise therapy for chronic fatigue syndrome (CFS) [1] following a formal complaint to Cochrane’s Editor in Chief. Unfortunately, the published amendment does not address the main flaws of the review and continues to overestimate the evidence for exercise therapy in CFS.

In this blog post, I will argue that treatment effects found in the review are (1) small and lower than some estimates of the minimal clinically important difference (2) no longer statistically significant at follow-up and (3) contradicted by objective measurements. I will argue that there is currently no plausible mechanism for the effectiveness of exercise therapy in CFS and that the treatment effects found are better explained as bias due to a lack of blinding. Finally, I will explain how patient surveys suggest that some CFS patients deteriorate following exercise therapy.

The post-treatment differences are really small
The Cochrane review includes 8 randomized trials. I will focus my analysis on the main comparison of exercise therapy versus a passive control condition (treatment as usual or relaxation therapy).

For fatigue, the primary outcome of the review, the post-treatment effect size corresponds to a 3.4 point difference on the 33-point Chalder Fatigue Scale. This is close to the reported minimal clinically important difference (MCID) for this scale. [2-4] Ridsdale and colleagues, for example, used an MCID of 4 points, explaining that “our consensus view was that a difference of less than four, using a Likert scale, is not important.” [5]

The same is true for the post-treatment difference for physical function measured by the Short Form 36 Health Survey Questionnaire (score 0-100). Here the results are obscured by an extreme outlier that reported a remarkable 30-point difference. If this trial is excluded, the mean post-treatment difference between the exercise and passive control group is only 7.37 points, below most estimates of the MCID for this scale. [6-7]

The same principle applies to other outcomes for which lesser quality data is available. Despite the comparison with a passive control condition, exercise therapy only produces minor differences in reported health; differences so small that their clinical relevance could be reasonably questioned.

‘Follow-up’: no longer statistically significant
At follow-up differences between the exercise and control group become even cloudier. Whether one looks at fatigue, physical function, depression, anxiety or self-perceived changes in overall health, the difference between the exercise and control group is no longer statistically significant. The only exception is sleep, an outcome for which only three trials reported data.

It seems that twelve weeks of building up exercises under the guidance of a trained healthcare professional doesn’t make much difference in the long run. Patients, who received treatment as usual or listened to a relaxation tape, had a similar outcome. The term ‘follow-up’ might be somewhat confusing because the three largest trials providing most of the data in this review defined their primary outcome at this time point.

Objective outcomes: not reported
A major flaw of the Cochrane review is that it did not report on objective outcomes (the sole exception is service use). Given that none of the trials were blinded, one would expect that reviewers focus on objective outcomes as these are less influenced by the hopes and expectations of trial participants. The largest study to date on bias in randomized trials, the BRANDO project, gave the following recommendation:

“Our results suggest that, as far as possible, clinical and policy decisions should not be based on trials in which blinding is not feasible and outcome measures are subjectively assessed. Therefore, trials in which blinding is not feasible should focus as far as possible on objectively measured outcomes, and should aim to blind outcome assessors.” [8]​

Unfortunately, the authors of the Cochrane review did just the opposite: they focused on the subjective outcomes and ignored the objective outcomes. The 8 randomized trials had data on employment [9], disability benefits [10], activity levels [11] and fitness tests [12-13] that were not presented in the Cochrane review. These showed no significant difference between the exercise group and the passive control group. [14]

No plausible mechanism
The lack of improvement on objective measures of fitness is puzzling given that the rationale for exercise therapy was to recondition CFS patients. [15] Nonetheless, the results are clear and consistent: the four trials [12-13, 16-17] that conducted a mediation analysis all found that self-reported improvements in fatigue or physical function are not mediated by objective measures of fitness. In CFS, exercise therapy does not work by increasing physical fitness, which is contrary to the offered treatment rationale. [15] This means that exercise therapy currently lacks a plausible mechanism for improvement.

A lack of blinding
There is one mechanism that could easily explain small improvements on patient-reported outcomes that disappear at follow up and are not supported by objective measurements. It’s called bias due to a lack of blinding. If patients know that they are getting the intervention they might be more positive about their current health status (a placebo-effect) or they might report symptoms according to what they think will please the investigators (response bias). That’s why drug trials usually blind both patients and therapists so that their expectations do not influence the results. In the exercise trials, however, blinding was not possible and this could have distorted the outcomes. A 2014 review by Hrobjartsson et al. on trials that compared blinded and non-blinded groups, found that the average difference in effect size for patient-reported outcomes was -0.56. [18] That’s very similar to the effect sizes reported in the Cochrane review.

There are reasons to think that bias was particularly high in the exercise trials. For one, exercise therapy for CFS was already recommended by healthcare institutions when the largest trials were being conducted. [19-20] During the largest trial, the infamous PACE trial, patients were sent a newsletter that highlighted how the new NICE guideline had recommended exercise therapy. [21]

A second reason is that treatment manuals consisted of assertive encouragement to raise patients’ expectations. One patient booklet told trial participants: “You will experience a snowballing effect as increasing fitness leads to increasing confidence in your ability. You will have conquered CFS by your own effort and you will be back in control of your body again.” [22] An online description of the exercise therapy used in the FINE trial was even more assertive: “Focus on your achievements now. Symptoms and limitations are temporary” and “There is no disease. Go for 100% recovery.”[23] The manual for therapists in the PACE trial said about patients “[…] it is important that you encourage optimism about the progress that they may make with this approach. You can explain the previous positive research findings of GET and show in the way you discuss goals and use language that you believe they can get better.” [24] Perhaps one shouldn’t be surprised that, after receiving such instructions and pep talk, patients say they do a little better.

‘MitCure’: an analogy
A thought-experiment might help clarify things. Imagine an unblinded randomized trial for patients with CFS called ‘MitCure’. MitCure is just a sugar pill, but the researchers conducting the trial tell participants that it’s a drug that helps to restore mitochondrial function. Patients are given booklets that explain in detail how mitochondrial dysfunction could account for all their CFS symptoms and how MitCure could make them back in control of their own life. In the first sessions with their therapist, patients are told that MitCure has already shown promising results and that it will help them to become fitter and healthier. Patients are told that it’s important that they take the right amount of MitCure and that they should gradually build up their dose. Activity diaries are filled in to monitor progress. Over a period of 12-15 weeks, patients get weekly sessions with a healthcare professional who emphasizes their achievements and reminds them of how important it is to take Mitcure in order to keep making improvements. Patients in the control group are only given medical care, the same care that patients in the intervention are receiving in addition to MitCure. Many are on a waiting list to receive MitCure as well. Now, what would happen if you ask both groups to fill in symptom questionnaires directly after the treatment ended? Wouldn’t you suspect the intervention group to report a small improvement in fatigue compared to the controls, say … a 3.4 point difference on the Chalder Fatigue Scale?

The oxford criteria: an outdated case definition
A further problem with the Cochrane review is the case definition used to select patients. Approximately 85 percent of patients in the randomized trials were selected using the Oxford criteria for CFS. [25] These criteria date back to 1991, focus solely on the symptom of fatigue and are considered outdated. According to a 2016 document issued by the Agency for Health Research and Quality (AHRQ), the Oxford case definition is the least specific of the definitions and less generalizable to the broader population of patients with CFS. [26] The AHRQ document advised that “future research should retire the use of the Oxford (Sharpe, 1991) case definition.” [26] A CFS working group for the NIH wrote that “continuing to use the Oxford definition may impair progress and cause harm.” [27]

Post-exertional malaise
In the amendment, the authors of the Cochrane review tried to account for this problem by adding a brief statement that “patients diagnosed using other criteria may experience different effects.” In my view, this is a meaningless statement that could apply anywhere. What the authors should have said is that the diagnostic criteria used in their review do not require patients to experience ‘post-exertional malaise’ or a marked worsening of symptoms following exertion, while all other and more recent criteria do. This is particularly relevant in a review that tries to assess the efficacy and safety of exercise therapy.

Both the 2011 International Consensus Criteria (ICC) [28] and the 2015 case definition proposed by the National Academy of Medicine [29] consider a marked worsening of symptoms following exertion and not fatigue to be the hallmark symptom of the illness. Consequently, these criteria propose a name that no longer refers to fatigue such as systemic exertion intolerance disease (SEID) or myalgic encephalomyelitis (ME). Studies show that ME/CFS patients experience prolonged debility after physical exertion, usually a cardiopulmonary exercise test. [30] Researchers now use these exercise tests to temporary provoke post-exertional malaise in the hope of finding a clue into the underlying pathology of the illness. [31]

Patient surveys indicate harm
There is evidence that suggests some ME/CFS patients respond differently to exercise therapy than the patients included in the randomized trials. In multiple patient surveys, approximately 50% of respondents indicate to have deteriorated following exercise therapy. [32-33] As these surveys were not randomized controlled trials, it is difficult to determine whether deterioration was due to exercise therapy or some other factor such as the fluctuating course of the illness. There are however reasons to take the patient surveys seriously.

For one, there have been more than a dozen of these surveys covering thousands of patients from multiple countries and spanning a period of almost 30 years. [32-33] Many of these surveys did not solicit a negative assessment of exercise therapy. In a 2010 survey by the British ME Association, for example, members were simply asked how useful or harmful they found several CFS treatments. [34] These included hydrocortisone, sleep medication, antidepressants and allergy medication, treatments known to have significant side-effects. Yet, it was exercise therapy that received by far the most negative responses. This suggests that the negative responses towards exercise therapy cannot fully be explained by selection bias. And contrary to what is sometimes claimed, there is no evidence to suggest that the reported harms of exercise therapy are due to inappropriate delivery of the treatment. In a 2008 survey by Action for ME there was little difference in the reported rate of harms by exercise therapy whether the treatment was delivered by an NHS specialist (31%), the GP (45%) or others (29%). [35] In a 2015 survey of the ME Association, graded exercise therapy courses “held by therapists stated to have an ME/CFS specialism made symptoms worse for 57% of respondents.” [36]

Unfortunately, the Cochrane review does not mention the patient surveys indicating harm by exercise therapy. It only refers to previous reviews that support the conclusion that exercise therapy is a safe and effective treatment. The review does admit that the 8 randomized trials provide little data on possible harms of exercise therapy and that “we are uncertain about the risk of serious adverse reactions because the certainty of the evidence is very low.” [1]

An influential review
Despite its many shortcomings, the Cochrane review on exercise therapy for CFS has been quite influential. National guidelines in Denmark [37], Germany [38], Australia [39], and Belgium [40-41] referred to the Cochrane review in recommending exercise therapy for CFS patients. In the US, Mayo Clinic uses the Cochrane review to justify the following inappropriate advice to CFS patients:

“Gradually increasing the intensity of your exercise over time may help reduce your hypersensitivity to exercise, just like allergy shots gradually reduce a person’s hypersensitivity to a particular allergen.” [42]​

In the UK, the Cochrane review has been used to deflect criticism of the PACE trial by the Health Research Authority [43], the Medical Research Council [44] and the current Minister for Care [45], all claiming that Cochrane has independently validated the PACE trial’s findings. In Australia, there is an ongoing randomized trial, testing how to educate healthcare professionals about the “level 1 evidence” for graded exercise therapy for patients with CFS. The intervention consists of an educational program “based on a manual developed by the research group that drew on Cochrane reviews of […] graded exercise therapy (GET) interventions for CFS.” [46] Overall, it seems that the Cochrane logo is hindering a critical assessment of the evidence.

Corrections as soon as possible
In October 2018, almost exactly a year ago, 42 experts in the field of ME/CFS signed an open letter asking Cochrane to address the shortcomings in its review on exercise therapy for CFS. [47] The recent amendment leaves the main flaws unresolved. Given the influence of the review, the poor evidence base for its conclusions and the potential harm it could cause I would advise Cochrane to issue a full update as soon as possible.

 

[ME/CFS Skeptic]

References
[1] Larun L, Brurberg KG, Odgaard-Jensen J, Price JR. Exercise therapy for chronic fatigue syndrome. Cochrane Database Syst Rev. Version published: 02 October 2019. https://doi.org/10.1002/14651858.CD003200.pub8

[2] Pouchot J, Kherani RB, Brant R, Lacaille D, Lehman AJ, Ensworth S, et al. Determination of the minimal clinically important difference for seven fatigue measures in rheumatoid arthritis. J Clin Epidemiol. 2008 Jul;61(7):705-13.

[3] Goligher EC, Pouchot J, Brant R, Kherani RB, Aviña-Zubieta JA, Lacaille D, et al. Minimal clinically important difference for 7 measures of fatigue in patients with systemic lupus erythematosus. J Rheumatol. 2008 Apr;35(4):635-42. Epub 2008 Mar 1.

[4] Nøstdahl T, Bernklev T, Fredheim OM, Paddison JS, Raeder J. Defining the cut-off point of clinically significant postoperative fatigue in three common fatigue scales. Qual Life Res. 2019 Apr;28(4):991-1003.

[5] Ridsdale L, Godfrey E, Seed P. Chronic Fatigue in general practice: authors reply. Br J Gen Pract 2001, 51:317–318.

[6] Brigden A, Parslow RM, Gaunt D, Collin SM, Jones A, Crawley E. Defining the minimally clinically important difference of the SF-36 physical function subscale for paediatric CFS/ME: triangulation using three different methods. Health Qual Life Outcomes. 2018 Oct 19;16(1):202.

[7] Wyrwich KW, Tierney WM, Babu AN, Kroenke K, Wolinsky FD. A comparison of clinically important differences in health-related quality of life for patients with chronic lung disease, asthma, or heart disease. Health Serv Res. 2005 Apr;40(2):577-91.

[8] Savović J, Jones H, Altman D, Harris R, Jűni P, Pildal J, et al. Influence of reported study design characteristics on intervention effect estimates from randomised controlled trials: combined analysis of meta-epidemiological studies. Health Technol Assess. 2012 Sep;16(35):1-82.

[9] Jason LA, Torres-Harding S, Friedberg F, Corradi K, Njoku MG, Donalek J. Non-pharmacologic Interventions for CFS: A Randomized Trial. J Clin Psychol Med Settings. 2007, 14:275–296.

[10] McCrone P, Sharpe M, Chalder T, Knapp M, Johnson AL, Goldsmith KA, et al. Adaptive pacing, cognitive behaviour therapy, graded exercise, and specialist medical care for chronic fatigue syndrome: a cost-effectiveness analysis. PLoS One. 2012;7(8):e40808.

[11] Wallman KE, Morton AR, Goodman C, Grove R, Guilfoyle AM. Randomised controlled trial of graded exercise in chronic fatigue syndrome. Med J Aust. 2004 May 3;180(9):444-8.

[12] Wearden AJ, Emsley R. Mediators of the effects on fatigue of pragmatic rehabilitation for chronic fatigue syndrome. J Consult Clin Psychol. 2013 Oct;81(5):831-8.

[13] Chalder T, Goldsmith KA, White PD, Sharpe M, Pickles AR. Rehabilitative therapies for chronic fatigue syndrome: a secondary mediation analysis of the PACE trial. Lancet Psychiatry. 2015 Feb;2(2):141-52.

[14] Vink M, Vink-Niese A. Graded exercise therapy for myalgic encephalomyelitis/chronic fatigue syndrome is not effective and unsafe. Re-analysis of a Cochrane review. Health Psychol Open. 2018 Oct 8;5(2):2055102918805187.

[15] Clark LV, White PD. The role of deconditioning and therapeutic exercise in chronic fatigue syndrome (CFS). J Ment Health. June 2005;14(3):237 – 252.

[16] Fulcher KY, White PD. Randomised controlled trial of graded exercise in patients with the chronic fatigue syndrome. BMJ. 1997 Jun 7;314(7095):1647-52.

[17] Moss-Morris R, Sharon C, Tobin R, Baldi JC. A randomized controlled graded exercise trial for chronic fatigue syndrome: outcomes and mechanisms of change. J Health Psychol. 2005 Mar;10(2):245-59.

[18] Hróbjartsson A, Emanuelsson F, Skou Thomsen AS, Hilden J, Brorson S. Bias due to lack of patient blinding in clinical trials. A systematic review of trials randomizing patients to blind and nonblind sub-studies. Int J Epidemiol. 2014 Aug;43(4):1272-83.

[19] Department of Health (2002) A report of the CFS/ME working group: report to the Chief Medical Officer of an independent working group. London: Department of Health. Available at: https://www.meassociation.org.uk/wp-content/uploads/CMO-Report-2002.pdf

[20] Chronic fatigue syndrome Clinical practice guidelines – 2002. Produced by a Working Group convened under the auspices of the Royal Australasian College of Physicians. Available at: https://www.mja.com.au/system/files/issues/cfs2_2.pdf

[21] Tuller D. TRIAL BY ERROR: The Troubling Case of the PACE Chronic Fatigue Syndrome Study. Virology Blog. October 2015. http://www.virology.ws/2015/10/21/trial-by-error-i/

The relevant newsletter is available here: https://www.me-pedia.org/images/a/a0/Participants'_newsletter_Issue_3_December_2008_PACE_trial.pdf

[22] Powell, P (2005) FINE Trial. Patient booklet version. 9 (29/04/05) Chronic Fatigue Treatment Programme. Liverpool: Infectious Diseases Unit, Royal Liverpool University Hospital.

[23] Tuller D. Trial By Error, Continued: Why has the PACE Study’s “Sister Trial” been “Disappeared” and Forgotten? Virology Blog. November 2015. http://www.virology.ws/2015/11/09/t...-sister-trial-been-disappeared-and-forgotten/

[24] Bavinton et al. 2004. Manual for Therapists Graded Exercise Therapy (GET) for CFS/ME. Final trial version: version 7 (MREC Version 2). https://me-pedia.org/images/8/89/PACE-get-therapist-manual.pdf

[25] Sharpe MC, Archard LC, Banatvala JE, Borysiewicz LK, Clare AW, David A, et al. A report–chronic fatigue syndrome: guidelines for research. J R Soc Med. 1991 Feb;84(2):118-21.

[26] Smith MEB, Nelson HD, Haney E, Pappas M, Daeges M, Wasson N, et al. Diagnosis and treatment of myalgic encephalomyelitis/chronic fatigue syndrome. Evidence Reports/Technology Assessments, No. 219. Addendum July 2016. Rockville (MD): Agency for Healthcare Research and Quality (US). https://www.ncbi.nlm.nih.gov/books/NBK293931/pdf/Bookshelf_NBK293931.pdf

[27] Green CR, Cowan P, Elk R, O’Neil KM, Rasmussen AL. National Institutes of Health Pathways to Prevention Workshop: Advancing the Research on Myalgic Encephalomyelitis/Chronic Fatigue Syndrome. Ann Intern Med. 2015 Jun 16;162(12):860-5.

[28] Carruthers BM, van de Sande MI, De Meirleir KL, Klimas NG, Broderick G, Mitchell T, et al. Myalgic encephalomyelitis: International Consensus Criteria. J Intern Med. 2011 Oct;270(4):327-38.

[29] Beyond myalgic encephalomyelitis/chronic fatigue syndrome: redefining an illness. Washington, D.C: The National Academies Press; 2015. https://www.ncbi.nlm.nih.gov/books/NBK274235/pdf/Bookshelf_NBK274235.pdf

[30] VanNess JM, Stevens SR, Bateman L, Stiles TL, Snell CR. Postexertional malaise in women with chronic fatigue syndrome. J Womens Health (Larchmt). 2010 Feb;19(2):239-44.

[31] Analysis of Post-exertional Malaise Using a Two-day CPET in People With ME/CFS. ClinicalTrials.gov Identifier: NCT04026425. https://clinicaltrials.gov/ct2/show/NCT04026425

[32] Kindlon T. Reporting of Harms Associated with Graded Exercise Therapy and Cognitive Behavioural Therapy in Myalgic Encephalomyelitis/Chronic Fatigue Syndrome. Bulletin of the IACFS/ME. 2011;19(2): 59-111. http://iacfsme.org/PDFS/Reporting-of-Harms-Associated-with-GET-and-CBT-in.aspx

[33] Geraghty K, Hann M1, Kurtev S. Myalgic encephalomyelitis/chronic fatigue syndrome patients’ reports of symptom changes following cognitive behavioural therapy, graded exercise therapy and pacing treatments: Analysis of a primary survey compared with secondary surveys. J Health Psychol. 2019 Sep;24(10):1318-1333.

[34] Managing my M.E. – What people with ME/CFS and their carers want from the UK’s health and social services. Gawcott, England: ME Association; May 2010. Available at: http://www.meassociation.org.uk/wp-content/uploads/2010/09/2010-survey-report-lo-res10.pdf

[35] Action for ME and Association of Young People with ME. ME 2008: What progress? 2008 May. Available at: https://ssb4mesupport.weebly.com/uploads/8/0/5/0/8050248/action_for_me_survey_2008.pdf

[36] ME Association. ME/CFS illness management survey results: No decisions about me without me. Patient Survey. May 2015. Available at: https://www.meassociation.org.uk/wp...No-decisions-about-me-without-me-30.05.15.pdf

[37] Hansen R. Denmark has some recommendations for ME… and they’re pretty awful. ME Action. July 2018. https://www.meaction.net/2018/07/09...-recommendations-for-me-and-its-pretty-awful/

[38] Antwort der Bundesregierung auf die Kleine Anfrage der Abgeordneten Maria Klein-Schmeink, Dr. Kirsten Kappert-Gonther, Kordula Schulz-Asche, weiterer Abgeordneter und der Fraktion BÜNDNIS 90/DIE GRÜNEN – Drucksache 19/12204 – Myalgische Enzephalomyelitis/Chronic Fatigue Syndrome – Aktuelle Situation in Versorgung und Forschung. http://dip21.bundestag.de/dip21/btd/19/126/1912632.pdf

[39] Graded exercise therapy: chronic fatigue syndrome. The Royal Australian College of General Practitioners (RACGP). March 2015. https://www.racgp.org.au/clinical-r...ded-exercise-therapy-chronic-fatigue-syndrome

[40] CVS-patiënten verwerpen ‘klinische gids’ RIZIV (Update). ME-gids. July 2018. https://www.me-gids.net/module-ME_C...82.html?utm_source=dlvr.it&utm_medium=twitter

[41] Fysieke training voor de behandeling van het chronisch vermoeidheidssyndroom? Minerva. 2016. http://www.minerva-ebm.be/NL/Article/2071

[42] Tuller D. Trial By Error: Mayo Still Champions GET. Virology Blog. August 2018. http://www.virology.ws/2018/08/06/trial-by-error-mayo-still-champions-get/

[43] Re: The PACE trial and the Committee’s inquiry on Research Integrity. Health Research Authority. January 2019. https://www.parliament.uk/documents...Research-Authority-to-Chair-re-PACE-trial.pdf

[44] Criticism of the PACE trial. Medical Resourch Council. August 2018. https://mrc.ukri.org/news/browse/criticism-of-the-pace-trial/

[45] House of commons Hansard. Westminster Hall. PACE Trial: People with ME. Volume 636. February 2018. https://hansard.parliament.uk/commo...-4566-940D-249F5026FF73/PACETrialPeopleWithME

[46] Li SH, Sandler CX, Casson SM, Cassar J, Bogg T, Lloyd AR, et al. Randomised controlled trial of online continuing education for health professionals to improve the management of chronic fatigue syndrome: a study protocol. BMJ Open. 2017 May 10;7(5):e014133.

[47] A Statement in Support of Cochrane. Virology Blog. October 2018. http://www.virology.ws/2018/10/23/a-statement-in-support-of-cochrane/

 

[Esther12]

Thanks again Michiel. I've really appreciated all the work you've been doing on this. A few quick points, rushed out before I go to bed:

Just in terms of emphasis, I think that I'd have started with your section on 'a lack of blinding' and also made clearer that when you talk about outcomes like 'sleep' you're really just talking about questionnaire scores, not actual sleep. There is a danger of making the review/GET sound better than it is by raising concerns about some of these technical points while not first emphasising more central problems.

I think that their risk of bias assessments are misleading in and of themselves, regardless of how they affect the conclusions of the review.

re plausible mechanism: I get the impression that not much is needed to claim a plausible mechanism. eg some waffle about central sensitisation would probably be enough to let them get away with that in the eyes of many, so I'm not sure if this is that strong a point?

"Despite its many shortcomings, the Cochrane review on exercise therapy for CFS has been quite influential." - is this slightly misleading, as you're analysis is of the new Larun review, but here you're talking about the influence of the old review? Many of the problems you mention apply to both, but I just found this strange.

 

[Hutan]

Thanks very much Michiel, you have made a massive effort to investigate the issues and it's a very good summary.

If I was going to pass the blog on to people who don't know much about this Cochrane review (and I'd like to), I'd like to see some mention made of the Cochrane editor's statement about doing a new review soon, with the assistance of patient organisations. To me that statement was quite persuasive - Cochrane itself was admitting things needed to be better and that the usual people lining up to do these exercise reviews don't have all the needed answers.

One minor point:

On Wednesday, October 2, Cochrane published a long-awaited amendment to its review of exercise therapy for chronic fatigue syndrome (CFS). [1] Several changes have been made to the text following a formal complaint to Cochrane’s Editor in Chief. Unfortunately, the published amendment does not address the main flaws of the review and continues to overestimate the evidence for exercise therapy in CFS.

I found the second sentence confusing for a while. I was wondering 'Had there been some change to the amended review since it was published?' Then I realised that you were saying that this amended version differs in some respects from the original version. Maybe just delete that second sentence?

 

[MSEsperanza]

I reiterate the thanks.

One minor point:

Michiel Tack said: ↑
On Wednesday, October 2, Cochrane published a long-awaited amendment to its review of exercise therapy for chronic fatigue syndrome (CFS). [1] Several changes have been made to the text following a formal complaint to Cochrane’s Editor in Chief. Unfortunately, the published amendment does not address the main flaws of the review and continues to overestimate the evidence for exercise therapy in CFS.

I found the second sentence confusing for a while. I was wondering 'Had there been some change to the amended review since it was published?' Then I realised that you were saying that this amended version differs in some respects from the original version. Maybe just delete that second sentence?

Or delete a part of that second sentence and merge it with the first?

"On Wednesday, October 2, Cochrane published a long-awaited amendment to its review of exercise therapy for chronic fatigue syndrome (CFS). [1] Several changes have been made to the text following a formal complaint to Cochrane’s Editor in Chief."

Or start the sentence with "Following..." and put the date at the end?

 

[ME/CFS Skeptic]

Thanks for your interesting comments @Esther12. Think I disagree with you on most points, though. Will try to explain my thoughts below:

There is a danger of making the review/GET sound better than it is by raising concerns about some of these technical points while not first emphasising more central problems.

I see. But I think I still prefer the current format as the arguments that (1) subjective improvements are below some estimates of the minimal clinically important difference (2) disappear at follow up (3) are contradicted by objective outcomes and (4) there is no plausible mechanism, all build-up to the argument that what we're seeing is bias caused by a lack of blinding. I think if I would start out what the binding argument and cite the Hrobjartsson et al. study, readers would get the impression that I'm pushing my hypothesis rather than giving an analysis of the data. I think one has to the know all the four points mentioned above to really appreciate how important lack of blinding is in interpreting the results. And regarding the sleep outcome, I did say that the authors did not report on objective outcomes, with the sole exception of service use.

I think that their risk of bias assessments are misleading in and of themselves, regardless of how they affect the conclusions of the review.

I think the whole Cochrane RoB-tool and the GRADE assessment system are arbitrary and not really adequate to assess the evidence in trials like these, so I just ignored them in my blog post.

re plausible mechanism: I get the impression that not much is needed to claim a plausible mechanism. eg some waffle about central sensitisation would probably be enough to let them get away with that in the eyes of many, so I'm not sure if this is that strong a point?

I personally think this is an important argument. I think all patient booklets, therapist manuals and theories about GET put forward the deconditioning hypothesis. I think that most of the therapists all over the world who are are still prescribing GET to CFS patients believe in this rationale that they are making CFS patient fitter with exercise therapy. The fact that mediation analyses are pretty clear about this, that that this is not what happens, is a strong argument for the 'bias due to a lack of blinding'-argument.

Promoters of GET can shout central sensitization but that doesn't make any sense really. It is not clear what that term means, whether it is present in ME/CFS and wether exercise therapy would do anything about it. It's not like there's evidence that exercise treats Central sensitization in other conditions. In fact, Jo Nijs the Belgian promoter of this theory has written that conditions where the evidence for central sensitization seems strongest such as fibromyalgia, whiplash-related disorders etc. are also those where exercise therapy is most controversial/least accepted compared to other pain disorders. I've written a blog about this unfortunate use of the term central sensitization in relation to GET last year. See: https://www.s4me.info/threads/central-sensitization-a-matter-of-concern.5346/

I think nobody talked about central sensitization as a treatment rationale before the trials, it's only when the results were not as they want it, that this term was put forward. I'm also not aware of someone working out the hypothesis of how central sensitization would be a mechanism of exercise improving CFS symptoms without improving physical fitness, people just mention the term or say things like 'exercise will make their nervous system less sensitized'.

Sorry for the long explanation. In short: the fact that people shout the buzz-word central sensitization doesn't mean we have to accept it as a plausible mechanism. (I do think that the central sensitization theory can be an interesting hypothesis for explaining symptoms such as photophobia or widespread pain in ME/CFS I just disapprove of its unfounded use in relation to GET).

What promoters of GET currently use as a rationale in their papers is that exercise therapy is like a psychological treatment that treats patients' fears and makes them focus less on fatigue, makes them believe they are in control of their symptoms etc. Because of course, their mediation analysis shows correlations with symptom improvement and changes on psychological questionnaires on fear-avoidance, catastrophization, sense of control etc. But this doesn't make sense either because patients do not increase their fitness activity level, employment etc. No one has worked out this theory either. So I don't think it's a plausible mechanism. Overall, I think GET-promoters prefer to talk about a plausible mechanism as little as possible and let therapists believe that they are treating deconditioning in CFS patients.

"Despite its many shortcomings, the Cochrane review on exercise therapy for CFS has been quite influential." - is this slightly misleading, as you're analysis is of the new Larun review, but here you're talking about the influence of the old review? Many of the problems you mention apply to both, but I just found this strange.

Not sure what you mean. All the examples I mention are about the update by Larun et al., first published in 2015, not about the old version by Edmonds et al. 2004.

People have complained about the Larun et al. review but the amendment (small changes to the text) in response to those complaints leaves the main problems unresolved (the fact that Cochrane plans a full update suggest that they also think not all problems have been resolved). So I think I'm correct to point how influential the Larun et al. review has been in shaping policy with its misleading conclusion and how Cochrane has a responsibility to do something about this. If the Cochrane review was pretty much ignored by everyone than the arguments I made about it would be less important. The fact that the review is taken very seriously by governments and healthcare institutions makes the analysis I make more important and urgent. That's why I've added this section.

One caveat is that GET was already recommended by healthcare institutions before the 2015 Cochrane review by Larun et al., so not all examples I mentioned would be different if the Cochrane review was never published. There are plenty of other reviews that came to the same conclusion and that are also cited. Nonetheless, I think the prestige of Cochrane changed things and made it more difficult to have a critical assessment of the evidence. The HRA report in the UK and the Australian trial I mentioned are good examples of that.

 

[ME/CFS Skeptic]

I found the second sentence confusing for a while.

Or delete a part of that second sentence and merge it with the first?

Thanks for pointing this out. I will go with the version @MSEsperanza proposes.

It now reads:

On Wednesday, October 2, Cochrane published a long-awaited amendment to its review of exercise therapy for chronic fatigue syndrome (CFS) [1] following a formal complaint to Cochrane’s Editor in Chief. Unfortunately, the published amendment does not address the main flaws of the review and continues to overestimate the evidence for exercise therapy in CFS....

 

[Trish]

Excellent work, thank you, @Michiel Tack.
How about sending it to the Cochrane Editor?

Given the influence of the review, the poor evidence base for its conclusions and the potential harm it could cause I would advise Cochrane to issue a full update as soon as possible.

I would make a stronger statement than this, especially as the editor has already said there will be a new review on a timescale they probably consider 'as soon as possible', so you are not urging them to do any more than they have said they will do anyway.

If it were me, I would finish with 'withdraw the review immediately'.

 

[MSEsperanza]

Esther12 said: ↑
"Despite its many shortcomings, the Cochrane review on exercise therapy for CFS has been quite influential." - is this slightly misleading, as you're analysis is of the new Larun review, but here you're talking about the influence of the old review? Many of the problems you mention apply to both, but I just found this strange.

Not sure what you mean. All the examples I mention are about the update by Larun et al., first published in 2015, not about the old version by Edmonds et al. 2004.

People have complained about the Larun et al. review but the amendment (small changes to the text) in response to those complaints leaves the main problems unresolved (the fact that Cochrane plans a full update suggest that they also think not all problems have been resolved).

How about just adding to the sentence...
"Overall, it seems that the Cochrane logo is hindering a critical assessment of the evidence"

...something like:
"Since the amended review leaves the main problems unresolved, this misleading impact will likely continue." ?

 

[Medfeb]

Nice job @Michiel Tack

I agree with you that the point about implausible mechanism is a key point and reminds me of Rosen's article about scientifically implausible theories, discussed in this thread. The GET review not only fails to account for but just flat out ignores the substantial evidence of biopathology - including evidence of harm - that counters the multiple theories that the review postulate. From the Rosen article:

First, authorities in mental health research and all who read the scientific literature must move beyond randomised controlled trials alone and adopt broader science-based criteria that consider the plausibility—or lack thereof—of therapeutic rationales and proposed change mechanisms.
​

A scientifically implausible theory compounds the problems with reliance on subjective findings, post-hoc outcomes switching, and studying the wrong patient population (Oxford). .

 

[rvallee]

Excellent work, thank you, @Michiel Tack.
How about sending it to the Cochrane Editor?



I would make a stronger statement than this, especially as the editor has already said there will be a new review on a timescale they probably consider 'as soon as possible', so you are not urging them to do any more than they have said they will do anyway.

If it were me, I would finish with 'withdraw the review immediately'.

Agreed on that. Immediate withdrawal is the only reasonable choice to make. It is politically difficult to do and unlikely to happen but it is inevitable with time, as meanwhile Cochrane is recommending a treatment paradigm that is documented to be harmful in numerous independent surveys that consistently find the same outcome, exposing the organization to serious litigation (this last part can obviously be left out but it is inevitable nonetheless).

Outstanding work, Michiel. The only addition I would make for the final blow would be the initial peer review, which highlighted most of those problems, and the more recent correspondence which acknowledged those problems despite the latest update ignoring them all. But that's a lot of work in itself and no doubt it will brought back soon enough.

 

[rvallee]

Nice job @Michiel Tack

I agree with you that the point about implausible mechanism is a key point and reminds me of Rosen's article about scientifically implausible theories, discussed in this thread. The GET review not only fails to account for but just flat out ignores the substantial evidence of biopathology - including evidence of harm - that counters the multiple theories that the review postulate. From the Rosen article:

First, authorities in mental health research and all who read the scientific literature must move beyond randomised controlled trials alone and adopt broader science-based criteria that consider the plausibility—or lack thereof—of therapeutic rationales and proposed change mechanisms.
​

A scientifically implausible theory compounds the problems with reliance on subjective findings, post-hoc outcomes switching, and studying the wrong patient population (Oxford). .

There is probably a strong point to make about how this hypothesis conflicts with another concept promoted by the same researchers: the boom and bust. There is no such thing as fluctuating deconditioning anymore than there is such a thing as sudden deconditioning, two characteristics that are well-documented, including in their own documentation. Two truths that are mutually exclusive with one another yet both presented as valid.

The use of an evidence-based process does not grant license to ignore common sense and paradoxes. Deconditioning is a known concept and it does not allow for either variation or suddenness. It is perfectly valid to question a hypothesis that is not consistent with another hypothesis, one that is equally held to be true by its proponents, in the same model. It takes a serious suspension of disbelief to accept those mutually conflicting truths and such a thing has no place in science, even more so in medicine where actual human lives are at stake.

 

[ME/CFS Skeptic]

I would make for the final blow would be the initial peer review, which highlighted most of those problems

Thanks for your comments @rvallee . Althought I'm not sure what you mean by initial peer review. This was a problem in the Cochrane review that intended to use individual patient data but I'm not aware of this on the Cochrane review I've discussed.

The email correspondence discussed some of the problems but ignored the most important ones in my opinion, mainly not reporting on objective outcomes and bias due to a lack of blinding.

 

[Esther12]

Thanks for your interesting comments @Esther12. Think I disagree with you on most points, though. Will try to explain my thoughts below:

Welcome, and thanks for the explanations - I think that a lot of it is down to our own subjective judgements. A few things where we might have been misunderstanding one another?:

Not sure what you mean. All the examples I mention are about the update by Larun et al., first published in 2015, not about the old version by Edmonds et al. 2004.

People have complained about the Larun et al. review but the amendment (small changes to the text) in response to those complaints leaves the main problems unresolved (the fact that Cochrane plans a full update suggest that they also think not all problems have been resolved). So I think I'm correct to point how influential the Larun et al. review has been in shaping policy with its misleading conclusion and how Cochrane has a responsibility to do something about this. If the Cochrane review was pretty much ignored by everyone than the arguments I made about it would be less important. The fact that the review is taken very seriously by governments and healthcare institutions makes the analysis I make more important and urgent. That's why I've added this section.

One caveat is that GET was already recommended by healthcare institutions before the 2015 Cochrane review by Larun et al., so not all examples I mentioned would be different if the Cochrane review was never published. There are plenty of other reviews that came to the same conclusion and that are also cited. Nonetheless, I think the prestige of Cochrane changed things and made it more difficult to have a critical assessment of the evidence. The HRA report in the UK and the Australian trial I mentioned are good examples of that.

I see - so you're writing as if the 2019 publication and the 2015 publication are really the same review?

To me, it seems odd to talk about the problems with the 2019 version, and then go on to say "Despite its many shortcomings, the Cochrane review on exercise therapy for CFS has been quite influential", with all the examples being for the 2015 version (particularly as the 2015 version has a number of additional problems).

That's different to how I'd do it but I really don't know what the right way of writing about this is. It seems an unusual situation.

I personally think this is an important argument. I think all patient booklets, therapist manuals and theories about GET put forward the deconditioning hypothesis. I think that most of the therapists all over the world who are are still prescribing GET to CFS patients believe in this rationale that they are making CFS patient fitter with exercise therapy. The fact that mediation analyses are pretty clear about this, that that this is not what happens, is a strong argument for the 'bias due to a lack of blinding'-argument.

Promoters of GET can shout central sensitization but that doesn't make any sense really. It is not clear what that term means, whether it is present in ME/CFS and wether exercise therapy would do anything about it. It's not like there's evidence that exercise treats Central sensitization in other conditions. In fact, Jo Nijs the Belgian promoter of this theory has written that conditions where the evidence for central sensitization seems strongest such as fibromyalgia, whiplash-related disorders etc. are also those where exercise therapy is most controversial/least accepted compared to other pain disorders. I've written a blog about this unfortunate use of the term central sensitization in relation to GET last year. See: https://www.s4me.info/threads/central-sensitization-a-matter-of-concern.5346/

I think nobody talked about central sensitization as a treatment rationale before the trials, it's only when the results were not as they want it, that this term was put forward. I'm also not aware of someone working out the hypothesis of how central sensitization would be a mechanism of exercise improving CFS symptoms without improving physical fitness, people just mention the term or say things like 'exercise will make their nervous system less sensitized'.

Sorry for the long explanation. In short: the fact that people shout the buzz-word central sensitization doesn't mean we have to accept it as a plausible mechanism. (I do think that the central sensitization theory can be an interesting hypothesis for explaining symptoms such as photophobia or widespread pain in ME/CFS I just disapprove of its unfounded use in relation to GET).

What promoters of GET currently use as a rationale in their papers is that exercise therapy is like a psychological treatment that treats patients' fears and makes them focus less on fatigue, makes them believe they are in control of their symptoms etc. Because of course, their mediation analysis shows correlations with symptom improvement and changes on psychological questionnaires on fear-avoidance, catastrophization, sense of control etc. But this doesn't make sense either because patients do not increase their fitness activity level, employment etc. No one has worked out this theory either. So I don't think it's a plausible mechanism. Overall, I think GET-promoters prefer to talk about a plausible mechanism as little as possible and let therapists believe that they are treating deconditioning in CFS patients.

Part of our disagreement here may just be that you have higher standards for a 'plausible mechanism' than I. To me, it seems that 'plausible mechanism' routinely just means 'anything short of fantastical'. It could well be that my standards have been sent too low by reading so much work from dodgy researchers, but also, it does seem that we sometimes don't know why drugs work in the way they do, but when there's good evidence that they do work there must be some plausible mechanism.

I think that a lot of what you've just said is part of an important argument, but to me it isn't "No plausible mechanism", at least not by the standards I routinely see used. It felt like the point you're making is more that 'It seems increasingly likely that the unfounded claims made to patients as a part of their therapy about how exercise therapy worked were false, and that the theories which led to the development of GET for CFS reflect a misunderstanding of the condition'. I think that these are important points and the review should have made these problems clearer, but I see them as different to 'no plausible mechanism'.

I think the whole Cochrane RoB-tool and the GRADE assessment system are arbitrary and not really adequate to assess the evidence in trials like these, so I just ignored them in my blog post.

It didn't occur to me you would make such a bold manoeuvrer! Particularly after all your work going through those issues. I wonder if it could be worth noting that as I think that problems there are part of the reason "why the Cochrane review on exercise therapy for chronic fatigue syndrome is still misleading" and making no mention of them under that title could imply to some that you don't believe there were problems there? Or even just a brief note of 'there are many more problems with this review than I'm able to detail here'?

 

[Sid]

Great essay.

Re: plausible mechanism, my impression is that the BPS crowd does not believe in the deconditioning hypothesis and they only use it in manuals for political reasons because they think it's less controversial and less likely to trigger the patient than stating their actual view which is that there is no problem, just phobic avoidance of physical activity and therefore graded exercise therapy has nothing to do with reconditioning the body but everything to do with graded exposure & desensitisation therapy similar to what you would do in simple phobias or panic disorder w/ agoraphobia. These are the bread and butter of CBT at the end of the day. Hence their lack of interest in objective outcomes like fitness levels and how many metres a patient can walk. It's all irrelevant if you believe that nothing is wrong and the patient is simply choosing to continue avoidance of the fear-provoking stimulus.

 

[ME/CFS Skeptic]

It seems an unusual situation.

All the problems that I mention in the blog, the reasons why I think the review is misleading, apply to both the 2015 version and the recent amendment. I think these are the issues that are misleading to healthcare policymakers - if the review was corrected/updated for example by including objective outcomes, explaining the problem with lack of blinding, the oxford criteria etc. they might have taken a different decision. The fact that the 2015 version had some additional issues as well, changes little to that in my view. I didn't try to focus on the changes of the amendment (which are mostly about using a more neutral tone to describe the data) but on the data and meta-analysis itself.

it does seem that we sometimes don't know why drugs work in the way they do, but when there's good evidence that they do work there must be some plausible mechanism.

I suspect that in the cases where a drug works without us precisely knowing how researchers can still come up with more plausible mechanisms than the GET-proponents can with GET. I think mechanisms are important: it's the reason why it's wrong for that Cochrane review to call for more RCT's into homoeopathy for IBS.

I think in the case of the GET trials it's mostly about the interpretation what the small improvements post-treatment mean. Some people seem to be sure that these are not due to bias caused by a lack of blinding, but that these are real clinical effects. The fact that those improvements do not correlate with physical fitness and the lack of another plausible mechanism of improvement is, I think, a good argument to try to convince such people that those improvements seen on questionnaires are more likely due to bias.
That's probably why I focused on 'no plausible mechanism' rather than on the false promise of patients booklets as you suggest.

It's all irrelevant if you believe that nothing is wrong and the patient is simply choosing to continue avoidance of the fear-provoking stimulus.

I think if you take the fear-avoidance and kinesiophobia-theory seriously than one would also expect that patients do more physical activity after the treatment. If a patient has severe agoraphobia and doesn't go out much, you would expect a successful treatment to make the patient go out more, not just say he/she has less fear on a questionnaire. So even that hypothesis doesn't make much sense to me.

my impression is that the BPS crowd does not believe in the deconditioning hypothesis and they only use it in manuals for political reasons because they think it's less controversial

Yes, I suspect that if they would come forward that the rationale for GET has nothing to do with deconditioning and making the patient fitter, that it would be much harder for them to be taken seriously for example by journal editors, clinicians and funders of research.

 

[Sid]

Yes, I suspect that if they would come forward that the rationale for GET has nothing to do with deconditioning and making the patient fitter, that it would be much harder for them to be taken seriously for example by journal editors, clinicians and funders of research.

I don't think their funding and publication chances would be negatively affected. Psychiatric disorders, including anxiety disorders, receive vastly more NIH funding than ME/CFS and are considered legitimate career paths for clinicians and researchers unlike the ME/CFS field which is considered the graveyard of careers. I think they're worried that the patients would be offended and thus harder to recruit for GET trials. You need trial participants to "buy into" your drivel if you want them to take part in your experiments.

 

[Hoopoe]

Great essay.

Re: plausible mechanism, my impression is that the BPS crowd does not believe in the deconditioning hypothesis and they only use it in manuals for political reasons because they think it's less controversial and less likely to trigger the patient than stating their actual view which is that there is no problem, just phobic avoidance of physical activity and therefore graded exercise therapy has nothing to do with reconditioning the body but everything to do with graded exposure & desensitisation therapy similar to what you would do in simple phobias or panic disorder w/ agoraphobia. These are the bread and butter of CBT at the end of the day. Hence their lack of interest in objective outcomes like fitness levels and how many metres a patient can walk. It's all irrelevant if you believe that nothing is wrong and the patient is simply choosing to continue avoidance of the fear-provoking stimulus.

Yes it would make sense. I think patients are put off by this kind of psychiatry because they can sense that something isn't right. The CBT/GET people probably interpret this refusal as patients not wanting to admit that they have a "psychosomatic" problem or fearing the stigma of psychiatry in general. Their solution then is to be even more dishonest and manipulative, which may actually increase the alienation of patients.

 

[Cheshire]

graded exercise therapy has nothing to do with reconditioning the body but everything to do with graded exposure & desensitisation therapy similar to what you would do in simple phobias or panic disorder w/ agoraphobia. These are the bread and butter of CBT at the end of the day. Hence their lack of interest in objective outcomes like fitness levels and how many metres a patient can walk. It's all irrelevant if you believe that nothing is wrong and the patient is simply choosing to continue avoidance of the fear-provoking stimulus.

This may be the case. But even if we take the view that CFS is a phobia, there's still a big problem if your questionaire results are not in line with what happens IRL.

I can't imagine that a trial meant to treat spider phobia would be considered successful with only questionaires asking "Do you feel better?" "Are you less afraid of spiders" and no confrontation with spiders.
Or, because anorexia nervosa is a mental health problem, a trial that would just ask participants "do you feel better?" "Do you think you've put on weight?" without weighing them.

Even if you see CFS as a mental health problem, this trial is absurd and a methodological failure.

 

[rvallee]

Thanks for your comments @rvallee . Althought I'm not sure what you mean by initial peer review. This was a problem in the Cochrane review that intended to use individual patient data but I'm not aware of this on the Cochrane review I've discussed.

The email correspondence discussed some of the problems but ignored the most important ones in my opinion, mainly not reporting on objective outcomes and bias due to a lack of blinding.

I don't remember exactly and not sure if this is it, I thought what I remembered was peer reviewers' comments to a previous version of the review, but it seems to be the initial response to Robert's comments that basically acknowledged the problems. It certainly highlights that those are not nitpicking or that we are the only ones concerned with small details. And they certainly have not been addressed despite having been acknowledged to be significant.

http://www.virology.ws/wp-content/uploads/2019/03/Cochrane-Report-on-Courtney-Complaint-1.pdf