Review Interventions for the management of long covid post-covid condition: living systematic review, 2024, Zeraatkar, Flottorp, Garner, Busse+

Except for two cognitive tests, all of their outcomes are subjective. I think this might become a problem if more rehabilitation trials are done where subjective outcomes show an improvement but objective outcomes (6minute walking test, CPET, employment etc) don't. It might cause the same issues we had with the Cochrane review on GET.

 

This is a living review which will be regularly updated (every 6 months or sooner if there is important new evidence) and the authors have funding to this for at leats 3 years.

Perhaps the patients involved could highlight some of these problems?

 

@ME/CFS Skeptic are you intending to submit a response to this article?

 

Archived copy of the article: https://archive.ph/2024.11.28-08255...ent-cure-long-covid-cbt-rehabilitation-study/

 

Long Covid Advocacy's response —

https://longcovidadvocacy.substack.com/p/stacking-the-deck-the-systemic-review

 

https://twitter.com/user/status/1862171351206805877



Hi @EricTopol - it was noteworthy that coverage somehow latched onto CBT and rehab sometimes being better than nothing. Surely the real take-home was that >4y on, we have nothing substantive to offer some 400 million people globally. We should be able to do better than this..

 

Imprecision
I think there is an issue with how they evaluated (im)precision and was hoping if anyone could double-check. In short, precision is determined by the variation in the measurement (e.g. standard deviation) and the amount of information collected (sample size).

In the protocol the authors say they used a ‘minimally contextualized approach’ and refer to this GRADE guideline:
https://pubmed.ncbi.nlm.nih.gov/35934265/

Here’s how I understand it. You are advised to test two methods:

• The confidence intervals (CI) approach checks if the 95% CI crosses the border of the MID. You ideally want the full CI to be above this MID threshold to be sure the effect is clinically meaningful. If It crosses it (the lower limit of the CI is smaller than the MID), you downgrade with one level because the data is consistent with no effect. If crosses the threshold in the other direction indicating harm (the lower CI shows the control group might have done more than one MID better), you downgrade with two levels.

• Then there is the OIS approach: sometimes the CI is above the MID but we still do not have confidence in the effect because the sample size is too low. In that cause reviewers should calculated the optimal information size (OIS) which is basically just the sample needed to detect a small effect with 80% power. For example, for continuous outcomes and a t-test this would require a sample size of approximately 800 (400 per group).

I made the overview below of some of the estimates of this review that are most likely to indicate an clinically significant effect. For the CBT and REGAIN trials the authors did not downgrade for imprecision while I think they should have. For example the REGAIN estimates for HADS-depression and PROPr quality of life clearly cross the MID. In other words the confidence interval is consistent with no effect at all. For REGAIN' recovery/improvement outcome and the CBT outcomes, the sample size is far too small to meet the OIS criterium, so it should have been downgraded as well.



The results for the hyperbaric oxygen and transcranial stimulation trial were downgraded with two levels for imprecision while it is unclear why. They look similar to the ones discussed above on behavioral interventions that were not downgraded for imprecision at all. This is especially strange for the quality of life estimate from the transcranial stimulation trial where the CI is above the MID. The imprecision evaluation should be similar to those of CBT (I would have downgraded them both with 1 level for not meeting the OIS).

As an explanation for the 2-level-downgrade, the authors write: ‘Likely much too few participants to achieve prognostic balance.’ This reasoning is similar to the one behind the OIS criterium, but it is unclear where the authors have set this threshold. At what point does the prognostic balance become ok? N = 70 for the transcranial stimulation trials led to 2 x downgrading, while the 114 for CBT needed no downgrade at all. In comparison, the GRADE guideline mentions that the OIS criterium is 800 participants, so far above both sample sizes.

Anyone who can make sense out of this?

 

Here's why I'm asking about precision: the outcomes above that were downgraded twice were non-behavioral interventions with low risk of bias. Those what were not downgraded were rehabilitative interventions and all high risk of bias.

So because of this weird approach to evaluating precision, the reviewed ended up recommending things from the high risk of bias outcomes and none from the low risk of bias outcomes. The effect sizes and CI are relatively similar so I do not see how they ended up with this conclusion.

 

I thought I'd have a go at writing a response, focusing on a couple of linked issues regarding the review's reliance on just one study for each main conclusion, and specific problems with the REGAIN cohort. @ME/CFS Skeptic's X thread helped me put this together, but he is doing a deeper dive into these issues and others.

Would appreciate any comments and suggestions on how to improve this response. Some of this stuff is quite technical, so I hope I've not made any glaring errors.


Heterogeneity in long COVID and inadequate assessment of post-exertional malaise contribute to issues of indirectness in trials with high risk of bias

Dear Editor

Zeraatkar et al. report that moderate certainty evidence suggests that CBT and physical and mental health rehabilitation probably improve symptoms of long covid [1]. However, despite being presented as a (living) systematic review, the present conclusions are based on results from only a single trial in each case (REGAIN for physical and mental health rehabilitation [2]; and a Dutch trial of CBT [3]). Both of these studies have a high risk of bias, being open-labelled (non-blinded) and reliant on patient-reported outcome measures (subjective questionnaires) [4].

Moreover, the REGAIN study only included patients who experienced severe COVID-19 infection, requiring hospitalization. This review “opted not to rate down the certainty of evidence for indirectness because there is no evidence that currently suggests the effects of the intervention may be different based on severity of the acute COVID-19 infection.” We disagree with this decision and the reasoning behind it. Only a quarter of long COVID patients required hospitalisation for their initial infection [5], and there are data to suggest that non-hospitalization is associated with a higher risk of developing post-COVID-19 ME/CFS [6]. This is likely why the REGAIN study reported no instances of post-exertional malaise (PEM), the cardinal feature of ME/CFS, during the trial or follow-up period, despite PEM being one of the most frequently reported symptoms in patients with long COVID. For these reasons, we suggest that the REGAIN cohort is not representative of the wider long COVID population, and that REGAIN’s conclusions should therefore not have been extrapolated to all long COVID patients, as is the case in this review.

The authors of this review will be aware that similar questions about the quality and certainty of trial evidence are central in the longstanding and ongoing debate around the efficacy and suitability of CBT and graded exercise therapy (GET) for ME/CFS. In fact the decision of the 2021 NICE ME/CFS guideline committee to downgrade the certainty of trial evidence was in part based on the same issue of indirectness concerning the representativeness of trial cohorts with regard to PEM [7].

The heterogeneity of long COVID and current lack of clinical biomarkers to characterise different phenotypes provide challenges in trial design, but adequate identification and reporting of PEM (which should include determining whether patients meet established ME/CFS diagnostic criteria) can help address potential questions about representativeness and patient concerns about the suitability of different interventions. Once again, these trial findings highlight the issue of reliance on patient-reported outcome measures in non-blinded randomized trials, and the importance of objective outcome measures [4].

[1] Zeraatkar D, Ling M, Kirsh S, et al. Interventions for the management of long covid (post-covid condition): living systematic review. bmj 2024;387. doi:https://doi.org/10.1136/bmj-2024-081318

[2] McGregor G, Sandhu H, Bruce J, et al, Clinical effectiveness of an online supervised group physical and mental health rehabilitation programme for adults with post-covid-19 condition (REGAIN study): multicentre randomised controlled trial. BMJ 2024;384:e076506. doi:10.1136/bmj-2023-076506

[3] Kuut TA, Müller F, Csorba I, et al, Efficacy of Cognitive-Behavioral Therapy Targeting Severe Fatigue Following Coronavirus Disease 2019: Results of a Randomized Controlled Trial. Clin Infect Dis 2023;77:687-95. doi:10.1093/cid/ciad257

[4] Tack M, Tuller DM, Struthers C. Bias caused by reliance on patient-reported outcome measures in non-blinded randomized trials: an in-depth look at exercise therapy for chronic fatigue syndrome. Fatigue Biomed Health Behav. 2020;8(4):181-192. doi:10.1080/21641846.2020.1848262.

[5] FAIR Health, Patients Diagnosed with Post-COVID Conditions: An Analysis of Private Healthcare Claims Using the Official ICD-10 Diagnostic Code (FAIR Health, 2022). Available at https://s3.amazonaws.com/media2.fairhealth.org/whitepaper/asset/Patients Diagnosed with Post-COVID Conditions - A FAIR Health White Paper.pdf

[6] Chen CW, Lee HH, Chang SH, Chen YL, Wang YH, Leong PY, Wei JC. Risk of chronic fatigue syndrome after COVID-19: A retrospective cohort study of 3227281 patients. Journal of Infection and Public Health. 2024 Nov 1;17(11):102559. doi:10.1016/j.jiph.2024.102559

[7] Barry PW, Kelley K, Tan T, Finlay I. NICE guideline on ME/CFS: robust advice based on a thorough review of the evidence. Journal of Neurology, Neurosurgery & Psychiatry. 2024 Jul 1;95(7):671-4. doi:10.1136/jnnp-2023-332731

 

Excellent. I think an omission in the following?

->

"REGAIN’s conclusions should therefore not have been extrapolated to all long COVID patients, as is the case in this review"

 

A minus sign I think. the - before the 13.11 got accidentally deleted when I made the table. The estimate is −8.4, 95% confidence interval (CI) −13.11 to −3.69. So it is above the MID of 3 points and does not cross 0.

I assume this is what you referred to? Apologies for the confusion (will change the graph above).

 

Thank you both. Edited to add 'therefore not'.

 

Two rapid responses to the article so far at https://www.bmj.com/content/387/bmj-2024-081318/rapid-responses, but at a quick glance both seem to be promoting the authors own interests rather than being interested in the paper itself.

 

Intention to treat
In the protocol the authors said they were going to use intention-to-treat (ITT): "reviewers will preferentially extract the results from intention-to-treat analyses without any imputations for missing data, when reported." But if you look at the data they extract in the Excel sheet this doesn't always seem to be case.

Take for example the improvement/recovery rate in the REGAIN trial. At 52 weeks, 98 out of 216 participants with available data, improved in the intervention group compared to 66 out of 226 in the control group. But with ITT you should take the number of participants randomized which were 298 in the intervention group and 287 in the control group. If you do that, you get (slightly) lower estimates. In my calculation the confidence intervals now cross the MID of 50 per 1000.

RR: 1.43 (95% CI: 1.1 to 1.87)
Absolute Risk: 99 more (95% CI: 27 more to 171 more) per 1000
​

I noticed that for adverse effects, they did use the number of participants randomised namely 298 and 287. Same issues with the improvement rates in the CBT trial: They used the available cases (54 and 54) instead of the number randomized (57 and 57). For adverse effects they do take the correct number namely 57.

So this seems like another (relatively) minor error.

 

I have submitted my letter. At least I think I have; the page took a long time to submit.