Science for ME PACE Briefing document

This briefing was prepared by a Science for ME working group of patients, most with a background in science and mathematics. The authors, including Tom Kindlon, Sean Kirby and Graham McPhee, have published over twenty papers and letters of expert commentary on the PACE trial and related matters in peer-reviewed journals such as The Lancet and the British Medical Journal.


[Updated 27/03/2018]

The updates consist of changing 'negative' to 'null' and giving the reference to the Wilshire et al. BMC Psychology paper rather than referring to it as 'in press'?

[Updated 17/06/2018]
Corrected typo in the image title

Link (and access for non members)
https://www.s4me.info/docs/PaceBriefing3.pdf

 

The PACE Trial Controversy: A Summary

This briefing was prepared by a Science for ME working group of patients, most with a background in science and mathematics. The authors, including Tom Kindlon, Sean Kirby and Graham McPhee, have published over twenty papers and letters of expert commentary on the PACE trial and related matters in peer-reviewed journals such as The Lancet and the British Medical Journal.

"White et al. conclude that they stand firmly by the findings of the PACE trial, presumably because of their inability to understand its basic flaws. As has been suggested by others, the flaws are so egregious that it would serve well in an undergraduate textbook as an object lesson in how not to design a trial."
Emeritus Professor of Medicine Jonathan Edwards [1]​

Background

In 2007, NICE recommended graded exercise therapy (GET) and cognitive behavioural therapy (CBT) for patients with chronic fatigue syndrome (CFS), also known as myalgic encephalomyelitis (ME).[2] But this guideline was based on weak evidence from small trials and so the much larger PACE trial was designed as a definitive test of these therapies.[3]

PACE had over 600 patients, cost £5 million,[4] and was taxpayer-funded, mostly by the Medical Research Council. Uniquely for a clinical trial, the Department of Work and Pensions also contributed to it.[3]

PACE used subjective outcomes as its primary measures, but also included objective outcomes such as aerobic fitness. In clinical trials, subjective measures – patients’ self-ratings of symptoms – are influenced not only by how they actually feel but by their expectations of the treatment and their wish to please the experimenter. This is why drug trials use identical looking pills to keep patients ‘blind to’ (ignorant of) whether they are receiving the new drug or the old one (or a placebo). In PACE, this ‘blinding’ was not possible because of the nature of the treatments, and so the objective measures were crucial.

Another important feature of the trial was that the researchers prespecified how they would analyse its data – an established method to avoid the later ‘cherrypicking’ of favourable results.

During the year-long trial, all of the patients received standard medical care. They were randomly split into four groups. Three of the groups received an additional therapy – GET, CBT or ‘adaptive pacing therapy’ – and the fourth, a comparison group, received none.

From 2011 onward, PACE’s findings appeared in The Lancet and other journals.[3] The researchers reported that in the CBT and GET groups, approximately 60% of patients ‘improved’ and 22% ‘recovered’ – more than for the comparison groups.[3],[5] The treatments were said to be moderately effective, and safe.[3]

These claims had two consequences. First, NICE left its recommendations for CBT and GET in place.[6] Second, widespread promotion of the trial’s success to the public, clinicians and academics[7],[8],[9] led to uncritical acceptance of the explicit assumption behind PACE’s CBT and GET: that CFS patients are just deconditioned and fearful of exertion, and can recover if they increase their activity and focus less on their symptoms.[3]

 

Why has PACE been criticised?

‘I’m shocked that the Lancet published it…The PACE study has so many flaws and there are so many questions you’d want to ask about it that I don’t understand how it got through any kind of peer review.’
Ronald W. Davis, Professor of Biochemistry and Genetics at Stanford University[10]
​

ME/CFS has been shown to be at least as disabling as multiple sclerosis,[11] with roughly a quarter of patients bedbound or housebound[12] – many for decades – and an estimated recovery rate in adults of only 5%.[13] Recent, major research reviews of the biomedical literature commissioned by the US government have emphasised that ME/CFS is not psychological, but is a severely debilitating, multi-system, physical disease.[14],[15]

PACE’s claims ran counter to patients’ knowledge and lived experience, leading some to examine the trial’s methods. Those who did found two main problems, both serious.

1. The objective results contradicted the subjective results

Objective results were poor.[16] The GET group outperformed the no-therapy comparison group on a six-minute walking test[3] but PACE’s authors did not make clear what a bad result this was: after a year of therapy, the GET group’s increase in walking speed was less than half that achieved in three weeks in a sample of Class II chronic heart failure patients receiving graded exercise.[17] In fact, the GET group’s walking speed was only about half that of a healthy age-matched sample.[18]

In PACE, aerobic fitness – and therefore probably also activity levels – did not improve in any group, but these results were not published until four years after the main paper.[19] Other objective outcomes, whose publication was also delayed, showed that the therapies did not return patients to work or reduce their uptake of sickness benefits.[20]

2. The planned analyses for improvement and recovery had been abandoned

After the trial had finished, the PACE authors discarded their originally planned definitions for improvement and recovery and replaced them with laxer ones that would inevitably inflate those figures.[3],[5] The new thresholds were so low that patients could get worse on a scale of fatigue or activity levels than when they entered the trial and yet be considered to have entered a ‘normal range’ and to have recovered.[21]

Queen Mary University London (QMUL), PACE’s data custodian, refused to provide critics with either the study’s pre-planned outcomes or the relevant raw data to calculate the figures themselves.[22] QMUL spent over £200,000[23] unsuccessfully fighting a Freedom of Information Act request to provide the data.[24]

Despite the limited data finally released by QMUL, independent researchers were able to carry out statistical analyses almost identical to those prespecified by PACE.[16] The percentage of patients who ‘recovered’ fell from 22% for CBT and GET to 7% or below in each group, and showed no statistically meaningful difference between groups: that is, CBT and GET did not help patients recover.




The ‘improvement’ results were also far less impressive. ‘Overall improvement’ rates (requiring specific levels of improvement on both fatigue and physical function) fell from roughly 60% for CBT and GET to roughly 20%, compared to 10% for the no-therapy group – again, not a statistically meaningful difference and so a 'null' result.16 For fatigue on its own, only the CBT group did better than the no-therapy group; and for physical function on its own, only the GET group did so. But the results only just scraped into being statistically meaningful in each case.



If these weak and null results had been published in 2011, it seems likely that NICE would have withdrawn its recommendation of the use of CBT and GET for ME/CFS.

Why was PACE’s failure not acknowledged?

PACE was a nonblinded trial with unimpressive or null results on its subjective measures, when analysed according to its prespecified methods. Its objective measures showed that CBT and GET did not improve patients’ fitness or ability to work. PACE’s long term follow-up results were also null.[16],[25]

The trial has, however, been defended vigorously by its researchers and their supporters.[26],[27] Crucially, they do not accept the key criticism that the nonblinded design of the trial favoured CBT and GET. They argue that patients’ pre-trial expectations of success did not particularly favour those two therapies.[3] However, any undue influence on self-ratings would have arisen during the many treatment sessions in the trial, in which CBT and GET patients alone were told that their treatments were ‘powerful’ and ‘effective’ and that there was nothing to stop them getting well.[28],[29]

The PACE authors have claimed that reviews by the Cochrane organisation of other CBT and GET studies confirm PACE’s results[30] – but the studies in those reviews are also nonblinded trials with subjective primary measures.[31],[32]

What patients need now

PACE’s serious flaws are increasingly widely recognised;[33],[34] it is even being used as a teaching example of bad scientific practice in a number of universities.[35],[36] In addition to a petition from more than 12,000 individuals – mostly patients[37] – over 90 scientists and clinicians and more than 50 patients’ groups worldwide have written to Psychological Medicine demanding retraction of the misleading ‘recovery’ results,[38] and The Lancet has been asked to correct the ‘normal range’ results.[39] The controversy has reached the British and international press.[40-44]

Patients want this recognition of PACE’s failure to turn into action, and their immediate concern is that GET is actively dangerous. PACE reported no more harm to patients in the CBT or GET groups than in the comparison groups,[3] but the lack of increase in fitness in any group in the trial suggests that most patients may not have increased their activity enough to trigger a reaction.[18] If patients in a drug trial do not achieve the required dose, the results do not help to evaluate drug safety: PACE’s safety data are similarly uninterpretable.

But ME/CFS’s cardinal symptom is that patients get worse with exercise.[14] Physiology researchers warn that graded exercise would be expected to harm ME/CFS patients, regardless of the skill of the therapist.[45] In surveys, more than 2,000 out of 4,000 patients, including children, reported being made worse by GET.[46] Some patients tell of being made bedbound.[47] On a national scale, the level of harm is likely to be considerable.

"Robert was 12 when he first started [NHS] graded exercise therapy. His mother [explained]: ‘Robert was moderately ill when the physio began but became severely ill and required a wheelchair after a few months... After just a few months, he lost the ability to walk... The physiotherapist wouldn’t accept they were causing the harm and blamed my son for not trying hard enough, saying he didn’t want to get better; they would not accept that there was anything physically wrong with him... He had a fit in the pool where they were doing the exercises, which the neurologist later said was caused by extreme pain. Eventually, we had to get a charity to intervene so that we could stop the graded exercise. My son is now 21 and is still severely ill and housebound."
Report in The Independent[48]​

Contrary to the claims of the PACE researchers and their supporters, PACE’s results show that CBT and GET do not work for this disease. In the US, the Centers for Disease Control recently withdrew its recommendation of these therapies for ME/CFS.[49] Patients urgently want NICE to do the same, before more harm is caused.[50]

This would leave patients without any recommended treatments, due to a long standing failure of government bodies in the UK and abroad to fund biomedical research at meaningful levels.[51] Patients need the British government to now fund a proactive, coordinated, urgent biomedical research strategy for ME/CFS in line with the scale and seriousness of the problem.

References

1. Edwards J. PACE team response shows a disregard for the principles of science. Journal of Health Psychology. 2017;22(9).

2. United Kingdom National Institute for Health and Care Excellence. Chronic fatigue syndrome/myalgic encephalomyelitis (or encephalopathy): diagnosis and management. 2007. https://www.nice.org.uk/guidance/cg53.

3. White PD, Goldsmith KA, Johnson AL et al. Comparison of adaptive pacing therapy, cognitive behaviour therapy, graded exercise therapy, and specialist medical care for chronic fatigue syndrome (PACE): a randomised trial. Lancet. 2011; 377:823-36.

4. RAE 2008 Research Assessment Exercise. http://www.rae.ac.uk/submissions/ra5a.aspx?id=176&type=hei&subid=3181.

5. White PD, Goldsmith K, Johnson AL et al. Recovery from chronic fatigue syndrome aftertreatments given in the PACE trial. Psychological Medicine. 2013;43:2227-2235.

6. United Kingdom National Institute for Health and Care Excellence, Centre for Clinical Practice. Review of Clinical Guideline (CG53) – Chronic fatigue syndrome/ myalgic encephalomyelitis. 2011. https://www.nice.org.uk/guidance/cg53/evidence/review-decision-2011-546258781.

7. Boseley S. Study finds therapy and exercise best for ME. The Guardian, 18 Feb 2011. https://www.theguardian.com/society/2011/feb/18/study-exercise-therapy-me-treatment.

8. Daily Mail reporters. Got ME? Fatigued patients who go out and exercise have best hope of recovery, finds study. Daily Mail, 18 Feb 2011. http://www.dailymail.co.uk/health/a...s-exercise-best-hope-recovery-findsstudy.html.

9. Beckford M. Exercise and therapy can help ME sufferers, study claims. The Telegraph, 18 Feb 2011. https://www.telegraph.co.uk/news/he...herapy-can-help-MEsufferers-study-claims.html.

10. Tuller D. Trial by error: the troubling case of the pace chronic fatigue syndrome study. http://www.virology.ws/2015/10/21/trial-byerror-i/.

11. Kingdon CC, Bowman EW, Curran H et al. Functional status and well-being in people with myalgic encephalomyelitis/chronic fatigue syndrome compared with people with multiple sclerosis and healthy controls. PharmacoEconomics Open. 2018;2:1-12.

12. Shepherd C, Chaudhuri A. ME/CFS/PVFS: An Exploration of the Key Clinical Issues (8th edition). 2016. The ME Association, UK.

13. Cairns R, Hotopf M. A systematic review describing the prognosis of chronic fatigue syndrome. Occupational Medicine. 2005;55:20-31.

14. Clayton E. Beyond Myalgic Encephalomyelitis/Chronic Fatigue Syndrome: Redefining an Illness. 2015. The National Academies Press.

15. Green CR, Cowan P, Elk R et al. National Institutes of Health Pathways to Prevention Workshop: Advancing the Research on Myalgic Encephalomyelitis/ Chronic Fatigue Syndrome. Annals of Internal Medicine. 2015;162:860-865.

16. Wilshire C, Kindlon T, Courtney R et al. Rethinking the treatment of chronic fatigue syndrome—A reanalysis and evaluation of findings from a recent major trial of graded exercise and CBT. BMC Psychology. 2018;6:6.

17. Meyer K, Schwaibolda M, Westbrook S et al. Effects of exercise training and activity restriction on 6-minute walking test perfomance in patients with chronic heart failure. American Heart Journal. 1997;133:447-453.

18. Kindlon T. Do graded activity therapies cause harm in chronic fatigue syndrome? Journal of Health Psychology. 2017;22:1146–1154.

19. Chalder T, Goldsmith KA, White PD et al. Rehabilitative therapies for chronic fatigue syndrome: a secondary mediation analysis of the PACE trial. Lancet Psychiatry. 2015;2:141-152.

20. McCrone P, Sharpe M, Chalder T et al. Adaptive pacing, cognitive behaviour therapy, graded exercise, and specialist medical care for chronic fatigue syndrome: A cost-effectiveness analysis. PLoS One. 2012;7:e40808.

21. Wilshire C, Kindlon T, Matthees A et al. Can patients with chronic fatigue syndrome really recover after graded exercise or cognitive behavioural therapy? A critical commentary and preliminary re-analysis of the PACE trial. Fatigue: Biomedicine, Health & Behavior. 2017;5:43-56.

22. Matthees A. Freedom of Information request: ‘Selected data on PACE Trial participants’. 2014. https://www.whatdotheyknow.com/request/selected_data_on_pace_trial_part#outgoing-343077.

23. Smallcombe, P. FOI Request 2016/F173. https://postimg.org/image/qru6r5jx3/.

24. First-Tier Tribunal judgement on Queen Mary University London v Information Commissioner and Alem Matthees (Appeal EA/2015/0269). http://informationrights.decisions.tribunals.gov.uk//DBFiles/Decision/i1854/Queen Mary University of London EA-2015-0269 (12-8-16).PDF.

25. Sharpe M, Goldsmith KA, Johnson AL et al. Rehabilitative treatments for chronic fatigue syndrome: long-term follow-up from the PACE trial. Lancet Psychiatry. 2015;2:1067-1074.

26. White PD, Chalder T, Sharpe M. Response to the editorial by Dr Geraghty. Journal of Health Psychology. 2017;22:1113-1117.

27. Wessely S. The PACE Trial for chronic fatigue syndrome: choppy seas but a prosperous voyage. 2015. https://www.nationalelfservice.net/...esyndrome-choppy-seas-but-a-prosperousvoyage/.

28. Burgess M, Chalder T. PACE Manual for participants: Cognitive behavioural therapy. 2004. http://www.wolfson.qmul.ac.uk/images/pdfs/4.cbt-participantmanual.pdf.

29. Bavinton J, Dyer N, White, PD. PACE Manual for participants: Graded exercise therapy. 2004. http://www.wolfson.qmul.ac.uk/images/pdfs/6.getparticipant-manual.pdf.

30. Sharpe M, Chalder T, Stone J. Correspondence: Don't reject evidence from CFS therapies. Nature. 2018;31.

31. Larun L, Brurberg KG, Odgaard-Jensen J et al. Exercise therapy for chronic fatigue syndrome. Cochrane Database of Systematic Reviews 2017, Issue 4. Art. No.: CD003200.

32. Price JR, Mitchell E, Tidy E, Hunot V. Cognitive behaviour therapy for chronic fatigue syndrome in adults. Cochrane Database Syst Rev. 2008;(3):CD001027.

33. Marks DM (Ed.). Special Issue on the PACE Trial. Journal of Health Psychology. 2017;22(9).

34. Sense About Science USA. Editorial: On PACE. http://senseaboutscienceusa.org/editorial-on-pace/.

35. Proal, A. Interview with neuroscientist Michael VanElzakker: Vagus Nerve, ME/CFS, latent infection and more. http://microbeminded.com/2017/12/07...-vagus-nerve-mecfs-latent-infection-and-more/

36. Tuller D. David Tuller #IIMEC12 Preconference Dinner Presentation London 2017.

https://www.youtube.com/watch?v=6zWnVB_2lUU

.

37. ME Action. Petition: Misleading PACE claims should be retracted. 2016. https://my.meaction.net/petitions/pace-trialneeds-review-now.

38. Ablashi DV, Baraniuk JN, Barcellos LF et al. 2017. An open letter to Psychological Medicine, again! http://www.virology.ws/2017/03/23/an-openletter-to-psychological-medicine-again/.

39. Davis RW, Edwards JCW, Jason LA et al. An open letter to The Lancet, again. 2016. http://www.virology.ws/2016/02/10/openletter-lancet-again/.

40. Whipple T. Scientists trade insults over myalgic encephalomyelitis (ME) study. The Times, 1 Aug 2017. https://www.thetimes.co.uk/article/...r-myalgic-encephalomyelitisme-study-slk0cv5lj.

41. Matthews S. 'Disgusting old fart, neoliberal hypocrite': Angry scientists throw insults at each other over the results of a £5 million taxpayer-funded study into chronic fatigue syndrome. Daily Mail, 1 Aug 2017. http://www.dailymail.co.uk/health/a...ultsregarding-flawed-study.html#ixzz59x64yen3.

42. Maxmen A. A reboot for chronic fatigue syndrome research. Nature. 3 Jan 2018. https://www.nature.com/articles/d41586-017-08965-0.

43. Rehmeyer J, Tuller D. Getting it wrong on chronic fatigue syndrome. New York Times, 18 Mar 2017. https://www.nytimes.com/2017/03/18/opinion/sunday/getting-it-wrong-on-chronic-fatiguesyndrome.html.

44. Duffy S. Will the UK establishment finally stop denying the reality of ME? Huffington Post, 2 Feb 2017. http://www.huffingtonpost.co.uk/dr-simonduffy/uk-establishment-me_b_18375968.html.

45. VanNess JM, Davenport TE, Snell CR et al. Letter to British Journal of Sports Medicine. Reproduced in Trial by Error: Letter to British Journal of Sports Medicine from CPET Experts. 2018. http://www.virology.ws/2018/02/05/t...-journal-of-sportsmedicine-from-cpet-experts/.

46. Kindlon T. Reporting of harms associated with graded exercise therapy and cognitive behavioural therapy in myalgic encephalomyelitis/chronic fatigue syndrome. Bulletin of the IACFS/ME. 2011;19:59–111.

47. ME Association. ME/CFS illness management survey results: No decisions about me without me. 2015. http://www.meassociation.org.uk/wpcontent/uploads/NO-DECISIONS-WITHOUTME-report.docx.

48. Wright N. Time for Unrest: Why patients with ME are demanding justice. The Independent, 7 Jan 2018. http://www.independent.co.uk/news/l...-chronic-fatigue-illnessdisease-a8133616.html.

49. ME Association. CDC removes CBT and GET as recommended treatments for ME/CFS. 2017. http://www.meassociation.org.uk/201...recommendedtreatments-for-mecfs-11-july-2017/.

50. Brown R. Petition: Stop harming ME/CFS patients – take CBT/GET out of NICE guidelines NOW. https://www.change.org/p/nice-stop-harmingme-cfs-patients-remove-cbt-get-out-of-nicenow.

51. Radford G, Chowdhury S. ME/CFS Research Funding: An overview of activity by major institutional funders included on the Dimensions database. http://www.meassociation.org.uk/wpcontent/uploads/mecfs-research-fundingreport-2016.pdf.

ETA 3rd Sept 18 - Update URL for ref 35.
ETA2 14th Mar 19 - Correct URL for ref 34.

 

This is a great summary -- a big, big thank you to all involved!

I understand it's work in progress. It will be extremely helpful for any kind of advocacy work.

[ETA: I am so slow - seems that posting it on the forum has been work in progress while I was writing my post ]

In particular, I like that it focuses strictly on the trial's flaws, and that is is convincing without the need to argue against the BPS approach per se, or providing examples of organic abnormalities being presented in studies of apparently low "levels of evidence" (not saying that I would be able to rate any medical study's evidence, I only formed this assumption after reading many commentaries on many studies posted in the forum).

Whereas I think it's indeed sensible to confine this "briefing" to the PACE trial, I would like to ask if further sequels are planned?

Or, do you think, all the GET-promoting studies that followed after PACE will sink together with the PACE-ship?

Perhaps, as long as the ship is still being defended by escorting boats, it would be of additional help to attach an overview of studies which are constructed on the PACE trial's methodology and findings? Actually, the paper makes it clear that with reproducing the PACE "results", other studies also reproduced the flaws. To more stubborn readers, a synopsis listing all related studies could perhaps be helpful?

A minor hint:

-> Now published and free online access! ;-)

Would be great if the JHP articles' URL / DOI could also be added.

Again, thank you very much!

(I read the document after having a nap, with a rather fresh mind, but got severely brainfogged while writing my comment. Anticipated apologies in case my post reads as silly as I feel now.)

 

A couple of links if people want to share it:

https://twitter.com/user/status/977197951528337408

 

Is it possible to do a spelling correction in the Figures? It says 'prelanned' instead of 'preplanned'.

Otherwise superb!

 

The authors would have to amend the original document. The tables above are just screenshots from the original.

 

Really excellent, clear and damning explanations, thanks so much to all involved, a very valuable resource.

 

Amended the post above with the new images.

 

Thank you to everyone involved in putting this together.

 

Great summary, thank you!

 

Thank you to all involved. Only able to process a part of this tonight, but will return.

 

Basically the PACE team were on a mission, not to really test their pet therapies, but to 'prove' they worked. Particularly to the DWP perhaps.

 

This is a really excellent reference, explaining not just the what but the why, which is important for comprehension, which in turn is important for advocacy. Many thanks to all.

 

Yes, the problem in the past has been trying to argue the case based on evidence of an alternative pathology. That places the burden of proof on the person making the claim, and it is something psychosocial researchers capitalise on to support their arguments. Their explanation stands because there are no other fully validated ones - they win the game by default (that's got to stop, btw!). It is better to point out the lack of evidence for their explanations and arguments than to present an alternative explanation. At least not until there is strong evidence supporting that explanation.

Looking at the responses from authors and BPS proponents to the Wilshire paper, we can start to anticipate the new line of defense coming out:

Defense: Never mind the PACE trial, there's all those other previous studies
Response: But they weren't properly controlled, isn't that why you did your study in the first place? To obtain higher quality evidence of the effectiveness of CBT and GET? Higher quality means better control of nuisance variables.

Defense: There's still modest effects, so its better than nothing
Response:
- Its not clear yet whether the modest changes on self-report measures are "real", or just reporting biases
- There is no such thing as "better than nothing". Treatments cost time and money: doctors, patients and health care providers need to be assured of the effectiveness of any treatment, independently of other options. Money may be better spent offering practical support and assistance.
- While there is a belief that an effective treatment exists, there may less less research interest in finding a better one.

Defense: You used a "preliminary" version of our protocol, we changed it later and it was approved by our steering committee
Response: They used the document you published with the title "trial protocol" and signed your names to. And as for your steering committee, they need a good lesson in the meaning of a trial protocol (hint: it means do what you carefully planned and declared you would do, unless there are extraordinary reasons to change - which there weren't). (another hint: if you still don't get it, read the Wilshire paper on why the justifications given were not adequate reasons to abandon the original outcome measures.)
Alternative Response: Nobody gives a f**ck what your steering committee says is ok. They do not get to make up the rules of science.

Defense: You reanalysed the data using more stringent criteria, so of course the effects are reduced.
Response: I don't need to tell anyone here the answer to this one (they did it the way you promised to do it in your f**ing trial protocol!).

 

Great summary. My only quibble though is that it might be more accurate to use the term "null results" rather than "negative results" - there's a clear but important distinction, statistically. I wouldn't want anyone to dismiss the whole text just on the basis of choice of wording.

 

Your observation sounds right to me.