CBT combined with music therapy for chronic fatigue following Epstein-Barr virus infection in adolescents: a feasibility study, 2020, Wyller et al

Yes you're right. Nina E. Steinkopf checked and it turned out a word was missing. Her text is updated as of today with:

At the same time, [some] participants in the same group became more depressed, more fatigued, had more pain and a lower level of function.

She's grateful for the feedback

 

Yes, in either case it's clear that it is not an effective treatment. If it did anything good at all it may have improved coping with the illness, but even intrepreted this way, coping wouldn't have improved but diminished physical activity levels. Which would be a relevant outcome too, if it were investigated properly from the beginning.

Anyway, investigators need to define which illness they want to research -- prolonged fatigue after EBV infection or ME/CFS (or both, with clear criteria for subgroups.)

Edited to add: Of course, the fatigue scores as subjective outcomes should be interpreted as cautious as in other unblinded trials, even though what's interesting is, if I understood correctly and as others have pointed out, that in this trial, there weren't siginificant differences between the control group and the intervention group.

 

The problem basically is that their interpretation of their results ignores the reality of those results. None so blind as those who do not want to see.

 

And the authors' interpretation of their results is completely oblivious to both those possibilities. But I would have no problem with the possibility their intervention did nothing more than to inadvertently promote what the patient community have been telling pwME they need to do anyway. That would completely undermine the authors' own unhelpful illness beliefs.

 

Trial By Error: BMJ Retracts Music Therapy-CBT Study, But…

https://www.virology.ws/2020/10/27/trial-by-error-bmj-retracts-music-therapy-cbt-study-but/

 

BMJ isn't just missing the point here, they're blatantly dismissing it.

I have no idea what they think they are doing, though. Do they feel good about this? This is in line with a mission to promote good science? How? What purpose is being served by pushing through pseudoscience of obviously mediocre quality?

At this point just go straight to work for industrial polluters, tobacco or fossil fuel companies. Who seriously feels they're doing good hurting children by pushing gaslighting on them? You went into medicine and this is what you do with it? Seriously?! Grotesque.

 

It seems that the Intention to treat (ITT) analysis was changed.

Instead of "last observation carried forward (LOCF)" the authors now used an imputation method for missing values. I think that's why some of the ITT values are a little different, but the per-protocol analysis (which simply ignored missing values) is not. I suspect that one of the reviewers insisted on using the imputation method because LOCF has been criticized.

For most outcomes the differences are small. The most notable exception is the mean steps per day at follow-up. The new version reports a value of 5680 for the intervention group, much lower than the 7165 reported in the retracted version but the difference with the control group is still not statistically significant (possibly because the trial was underpowered).

 

I'm thinking about publishing the following comment on the republished version:

Changes are indicated in red.

This is a resubmitted version of a retracted paper and many issues remain unresolved

It is unfortunate that the BMJ Paediatrics Open website doesn’t make clear that this is an amended, republished version of a paper about which multiple methodological issues were raised [1] and that, eventually, was retracted [2]. The publication history, for example, does not mention the retracted version. Only at the very end of the article under “Provenance and peer review” is a link provided to the previous version. I suspect few readers will notice this. Hopefully, BMJ Paediatrics Open will place a more visible notification at the top of the page, for example the one it has put above the retraction notice [3].

Because this is a new publication that received its own DOI, the criticism of the retracted paper [1, 4] is no longer visible. This is unfortunate because, as I will try to clarify in this comment, many issues that were raised have not been addressed in this republished version.

In the retraction notice, BMJ Paediatrics Open explains that the retracted paper was misrepresented as a feasibility trial due to a mistake in the editorial process and that this was not due to error on behalf of the authors. As I will explain below, this is an implausible explanation given the information that is currently available. The review history shows that one of the reviewers was confused with how the authors originally presented their results. She stated: “I struggle to understand from the aims of the study and the way the study is described whether this was intended as a feasibility study – i.e. to look at feasibility (can this be done?), acceptability (how do participants experience it?) and to give some indication of potential effect sizes to power a future larger scale trial, or whether this was intended as a fully powered trial. Throughout, I think this needs to be clarified for the reader and interpretations/conclusions drawn in light of what the aim was.”

To this, the authors responded: “Thank you. We agree – this study should be regarded a feasibility study, and the manuscript has been rephrased accordingly.” So it seems that the authors explicitly stated that this study should be regarded as a feasibility study when this was not the case. This information conflicts with the editorial statement that the authors were not in error. I think more information should be shared to clarify this contradiction.

It is unfortunate that the peer review history of this republished version has not been made public. On its website, BMJ Paediatrics Open claims it is committed to open peer review and states “as part of this commitment we make the peer review history of every article we publish publicly available.” It is unclear why the peer review of this republished version became an exception to this commitment. In the peer review of the retracted version, one reviewer admitted that he hadn’t read beyond the abstract. Considering the problems surrounding this work, it seems particularly important that readers are able to read through the review process and understand the decisions taken by the journal and authors.

In the retraction notice, BMJ Paediatrics Open explains that the retracted paper was misrepresented as a feasibility trial due to a mistake in the editorial process and that this was not due to error on behalf of the authors. As I will explain below, this is an implausible explanation. In fact; the review history shows that authors wrote to one of the reviewers that “this study should be regarded a feasibility study” when this was clearly not the case.

On its website, BMJ Paediatrics Open claims it is committed to open peer review: “as part of this commitment we make the peer review history of every article we publish publicly available.” It is unclear why the peer review of this republished version hasn’t been made public yet. In the peer review of the retracted version, one reviewer admitted that he hadn’t read beyond the abstract. It is unclear if the peer review of this version of the paper was more thorough.

The paper doesn’t explain that the study was designed to test large effect sizes

A notable aspect of this study is that the number of participants the authors tried to recruit (n = 60) is much lower than the number the power analysis suggested was needed to detect a difference of moderate effect size (n = 120). In other words, the lack of power was not only due to recruitment problems or drop-outs; the study was designed as such.

The statistical analysis plan [5] shows that the study was powered to test a large effect size, namely 0.8 times the estimated standard deviation of the primary outcome measures (approximately 2000 steps per day). The authors justified this by saying that since "CBT alone is documented to have a moderate effect size in CFS/ME, only a substantial effect size is of direct clinical interest. Analogously, only a substantial treatment effect is of interest regarding markers of pathophysiology.” [5] In the protocol, they also argued that “the FITNET study suggests that larger treatment effects might be assumed in adolescent CFS/ME patients as compared to adults.” [6]

In this republished version of the manuscript, however, nothing suggests that the authors anticipated the intervention to produce a large effect size and that the intervention was designed to this hypothesis. The fact that the authors previously agreed to frame this study as a feasibility trial (when this was clearly not the case), suggests there was a willingness to disregard that the intervention failed to provide the anticipated effect sizes. Even though the republished version avoids the term ‘feasibility trial’, the issues remain the same.

There is little data to support the conclusion that the intervention is “feasible and acceptable”

As the intervention failed to provide the large effect sizes anticipated in the statistical analysis and protocol, the retracted paper focused on other aspects than the efficacy of the intervention, such as feasibility and acceptability. The same is true for this republished version which highlights that “combined CBT and music therapy is feasible and acceptable in adolescent postinfectious CF.”

The data do not support this conclusion. More than half of the eligible individuals (n=48) did not consent to participation. Of those who started the program 6 out of 21 or 28% dropped out, compared to only 4.5% in the control group. The claim that the intervention is acceptable seems to be based only on a high attendance of the 15 patients who did start and stayed in the intervention arm of the study.

The authors also point to a lack of statistically significant differences of adverse events between the intervention and control group. But given that the trial was “strongly underpowered” to test the efficacy of the intervention, the same reasoning would apply to adverse events as well.

It should also be noted that the trial registration [7] lists approximately 20 different outcome measures for this trial, and with the exception of adverse effects, none of these focused on the acceptability of the intervention. Therefore, the main conclusion of this republished version of this paper remains unwarranted.

The outcome measure for post-exertional malaise was added post-hoc

In the discussion section, the authors state: “we observed a concurrent tendency of improvement of many symptom scores, including fatigue and postexertional malaise, in the intervention group.” The authors fail to mention that approximately 20 outcomes measures were registered for this study but that the outcome ‘postexertional malaise’ was added post-hoc: it was not listed as an outcome in the protocol (1), statistical analysis plan (2), or trial registration (3).

The tendencies towards improvement were all quite small. The biggest difference found was for the primary outcome measure, where patients in the intervention group did worse than those in the control group. Patients in the intervention group had a mean of 6198 steps per day post-treatment, 2059 steps lower than in the control group. The text of the paper does not mention that this difference reached statistical significance in the per-protocol analysis.

The tendencies towards improvement were all quite small. The sole exception was the primary outcome measure for which patients in the intervention group had a mean of 6198 steps per day, 2059 steps lower than in the control group. The text of the paper does not mention that this difference reached statistical significance in the per-protocol analysis.

The reported recovery rates are misleading

Instead, the result section highlights a trend towards a higher recovery rate in the intervention group. The authors defined recovery as a score lower than 4 points on the Chalder Fatigue Scale using a dichotomous scoring method (range 0-11 points). A score of 4 or higher on the Chalder Fatigue Scale, however, was already used as an inclusion criterium. This means that participants could be classified as recovered as a result of reporting an improvement of just 1 point on the Chalder Fatigue Scale.

It should also be noted that the Chalder Fatigue Scale does not assess the intensity or impact of fatigue. Instead, it assesses whether participants experience fatigue-related symptoms such as having” problems starting things” or finding it “more difficult to find the right word” more than usual. Consequently, it seems inappropriate to use the term “recovery rate” for the percentage of participants who score lower than the threshold of 4 points on the Chalder Fatigue Scale. Post-treatment there was no difference in the percentage of patients meeting this 4-point threshold between the intervention and control group.

The intention-to-treat analysis of the Chalder Fatigue Scale ordinal scoring (range 0-33) was also reported, and this showed little difference between the two groups. A plausible explanation for what the authors describe as “a trend towards higher recovery rate in the intervention group” is the high drop-out rate. At follow-up, only 13 patients were in the intervention group. The analysis recovery rates highlighted by the authors do not take into account the 8 persons who were in the intervention group but were lost to follow-up. There are little reasons to suggest that more patients in the intervention group recovered than in the control group. It is unfortunate that the authors have used this term in their manuscript.

References

[1] Tuller D. Trial By Error: More on that Norwegian CBT/Music Therapy Study. Virology Blog. May 16, 2020. https://www.virology.ws/2020/05/16/trial-by-error-more-on-that-norwegian-cbt-music-therapy-study/

[2] Marcus A. BMJ journal retracts, replaces study on chronic fatigue in children. Retraction Watch. October 21, 2020. https://retractionwatch.com/2020/10...odBA9xf5vDnNPlIWayQ6mwr5JOVhslbloUjiRFmHwkL-k

[3] BMJ Paediatrics Open. Retraction: Cognitive–behavioural therapy combined with music therapy for chronic fatigue following Epstein-Barr virus infection in adolescents: a feasibility study. http://dx.doi.org/10.1136/bmjpo-2019-000620ret

[4] Tack M. Inaccuracy in reporting CEBA part II. BMJ Paediatrics Open. https://bmjpaedsopen.bmj.com/content/4/1/e000620.responses

[5] Statistical analysis plan – CEBA part 2. https://bmjpaedsopen.bmj.com/conten.../bmjpo-2020-000797supp002_data_supplement.pdf

[6] Akershus University Hospital. Research Protocol - processing. Mental training for chronic fatigue syndrome (CFS/ME) following EBV infection in adolescents: a randomised controlled trial. Available from: https://www.ahus.no/seksjon/forskning/Documents/Forskningsgrupper/Barne- og ungdomsklinikken/Paedia/Forskningsprotokoll - behandling.pdf

[7] ClinicalTrials.gov Identifier: NCT02499302. Available from: https://clinicaltrials.gov/ct2/show/NCT02499302

 

More impressive work from you, @Michiel Tack
This is marvellous! Well done!

I believe "is" might be missing here, otherwise have nothing to add

In this republished version of the manuscript, however, nothing suggests that the authors anticipated the intervention to produce a large effect size and that the intervention was designed to this hypothesis. The fact that the authors previously agreed to frame this study as a feasibility trial (when this ----- clearly not the case)

 

Thanks @Kalliope Changed it

 

Thanks Michiel for your work on this.

Now too tired to judge all of it, but at a first glance it looks very good to me.

Not sure whether I will be able to comment with a cleaerer mind soon, so just leave some quick questions here in case they make sense.

Did you see that they now inserted a 'review history' link?
https://bmjpaedsopen.bmj.com/content/bmjpo/4/1/e000797.reviewer-comments.pdf

Instead of displaying the reviewers' comments you will be told:

Sorry, there is no peer review to display for this article

For research papers BMJ Paediatrics Open has fully open peer review. [...]

In rare instances we determine after careful consideration that we should not make certain portions of the prepublication record publicly available. In all instances in which we have determined that elements of the prepublication record should not be made publicly available, we expect that authors will respect these decisions and also will not share this information.

 

In addition to that, I wonder why the authors had submitted their original paper already with an odd title ("exploratory randomized trial").

This seems not only a false characterization of the study, but also a very unusual term for a therapist-delivered intervention study --

see: https://www.s4me.info/threads/cbt-c...y-2020-wyller-et-al.14685/page-13#post-296195

It's laudable to report results from trials that failed in diverse respects to fulfill the goals set in the protocol. But not so much laudible if the study's investigators whitewash the failures instead -- and get help from the BMJ's editors when others point out these failures.

I'm sure ther is a more adequate term that investigators could use as a title if they wanted to report a 'failed' trial?

To just call it "randomized controlled trial" in the republished paper's title when it failed to be fully powered seems still misleading?

 

These two sentences to me seem a bit confusing, since you begin to talk about tendencies towards improvements, but the exception you mention and how you word it I think refers just to a difference compared to the control group at a certain time point but not a difference between time points; that doesn't necessarily reflect an improvement in either of the groups? But perhaps you meant to say the only significant improvement was in the control group, not in the treatment group?

Edited to add: Even more confused now and perhaps I misunderstood but -- sorry, not feeling up to check: The physical activity measures (= steps per day) deterioriated in the intervention group, right? Then I think this sould be worded accordingly.

(Edited again for clarity -- wrote "endpoints" but meant "time points".)

 

I'm sympathetic to @Esther12's position here. I worry that a word like "lie" carries implications around intent, and that we should avoid speculating on researchers' intent, and focus on the outcome. Think it is better to avoid that particularly word, with all it baggage, and say something closer to observable fact, like they "misrepresented" their study, and must have been aware they were doing so.

 

Well, they need to explain a lot of things, so it is not acceptable to not include the reviews. I'm glad Michiel is responding. We'll likely be sending in another letter to the editors and Dr Godlee.

 

and oxo...

No altering of the text - «recovery»:
In Malik et al. table 4 as «reports» the results of «recovery» we find discrepancies.

In the result after the intervention trial, at the primary endpoint at 3 months, there are two participants missing, one for each of the groups, respectively, mental therapy and general practitioner.

In the result of 15 months, reports Malik et al. different in the text cf. the numbers in the table. The text says "However, there was a trend towards higher recovery rate with a total of 8 (62%) recovered individuals in the treatment group, as compared with a total of 5 (39%) recovered individuals in the control group", while in the table states that seven (n = 7 and 37%) controls recovered.

 

Yes, thanks @Marit @memhj It seems like this was an error made in the very first very version submitted, and that remained the same in all the version that came afterwards.

I suspect that the text is wrong and the info in the table correct.

 

Another point, even if probably not relevant for your ( @Michiel's) comment:

If I remember correctly, according to the protocol (or trial registration or both?) the study aimed to investigate "CFS/ME", not unspecific chronic fatigue. Shouldn't the investigators mention in the paper that they amended the subject of their trial?

 

Agree, thanks for pointing this out.

Would this be better: