Just out from Hilda's Talk page - a submission from the IAG. It's welcome, although covering pretty much the same ground as the S4ME complaints that were waved away with barely the blink of an eye by Cochrane.
Project Talk Page
Independent Advisory Group: Submission to the Cochrane Editorial Board
By
Hilda Bastian on March 13, 2025
Today, the Independent Advisory Group (IAG) sent the submission below to the Cochrane Editorial Board, copied to the Governing Board.
This is the second of three recent actions by the IAG:
- January 2025: Open Letter to Cochrane;
- This submission; and
- March 2025: Comment on the Cochrane exercise review.
Hilda Bastian on behalf of the IAG
Submission to The Cochrane Library Editorial Board on Policies on Outdated and Potentially Misleading Reviews
By the Independent Advisory Group on the Cochrane review on exercise and ME/CFS
Dear Dr Soares-Weiser and members of the Editorial Board,
We are the members of the Independent Advisory Group (IAG), a stakeholder engagement initiative for the Cochrane review on exercise and ME/CFS, appointed by the Cochrane Collaboration. Our group includes advocates, researchers, and clinicians with an interest in ME/CFS, as well as systematic review experts whose senior Cochrane leadership roles span the Collaboration’s history, with methodologist, editorial policy, and Cochrane and PubMed publishing policy experience. [1]
We have recently expressed our dismay and concern to the Chair of the Cochrane Collaboration’s Governing Board at the decision to reject our formal advice to append an editorial note to the review stating that it is out of date and should not be used for clinical decision-making. [2] Instead, the review was re-issued as an apparently updated review, thus ostensibly affirming its contents as being of current relevance to people making healthcare decisions. We understand that this was on the advice of the Editorial Board.
This submission addresses three policy issues that relate to these actions:
- Retiring outdated and misleading reviews;
- Re-issuing an unchanged review as an apparently updated one; and
- Unaddressed criticisms.
We urge the Editorial Board to re-consider their recent policy direction and address the implications for the review’s editorial note. Our reasons are detailed below, and we would welcome discussion on the issues we raise. We look forward to your response.
Yours sincerely,
Hilda Bastian, Katharine Cheston, Lily Chu, Mike Clarke, Miranda Cumpston, George Faulkner, Peter Gladwell, Penelope McMillan (with Penelope Del Fante), Jaime Seltzer, and Samuli Tani.
Part 1: Retiring outdated and misleading reviews
In response to our recent communication with the Chair of the Cochrane Collaboration’s Governing Board, [2] Cochrane reported that:
“Cochrane’s editorial policies have evolved since 2019 to align with standard publishing practices for academic journals. Cochrane no longer uses post-publication notices to express a judgement about the status or currency of a review in terms of its question or findings.” [3]
No longer publishing notices about their status or currency would change a fundamental feature of Cochrane reviews. The
Cochrane Database of Systematic Reviews is not a standard academic journal. It is a database of reviews, marketed as “the leading database for systematic reviews in health care,” with “reliable findings to inform decision making.” [4] Being adequately up-to-date is critical for reliable findings on questions that have not yet been settled. Cochrane reviews are promoted as being regularly updated so that people “can base treatment decisions on the most up-to-date and reliable evidence.” [5] The
Cochrane Library, which contains the
Cochrane Database, carries a slogan reflecting this aim: “Trusted evidence. Informed decisions. Better health.” [6]
Current editorial policies provide for updating of reviews in the
Cochrane Database, or withdrawing them in exceptional circumstances. [7] Although developments since a review was published can compromise its suitability for clinical decision-making, withdrawal may not always be applicable to these reviews, and updates may not be planned, or may take considerable time to be completed. Users should be made aware that such reviews, as published, are no longer fit for purpose.
Editorial notes have been used to make users aware of reviews in these circumstances, as was done in 2021, for example, with the Cochrane review on cognitive behaviour therapy in chronic fatigue syndrome. [8] That editorial note includes: “…the review is no longer current. It should not be used for clinical decision-making.” [9] Editorial notes were used in this way into the second half of 2024. [10]
In 2024, when using post-publication notes this way was still current practice, we advised adding a similar editorial note to the exercise review, because it is outdated and contains misleading content. Here, we outline the policy basis for our advice, and why this particular review does not meet Cochrane’s standards for being a reliable source of evidence on a controversial topic.
The chapter on updating reviews in the
Cochrane Handbookspecifies that updating may be needed because of further studies for inclusion or updated methods. [9] The
Handbookacknowledges that maintaining the credibility of a review where findings are not of very high certainty may require incorporating information and evidence that “may shed light on more nuanced effects on the intervention,” whether or not it would substantially alter a review’s conclusions. This could include, for example, new information on additional outcomes, or information that makes estimates of effect more precise.
In 2019, the Editor in Chief stated that the exercise review is based on “a research question and set of methods from 2002,” requiring revision of its protocol. [11] The original protocol (published in 2001) predates fundamental changes in diagnostic criteria for this condition. [12] The 2019 statement remains attached to the 2024 editorial note, and stresses the need for an update to enable people “to make well-informed decisions.” This stands in contradiction to the apparent implication of the 2024 editorial note that the review remains of current relevance for decision-making. The 2019 statement indicates, rather, that this review falls into the category described in the
Handbook as one where it is “more appropriate to conduct a new review from scratch meeting current standards.” We agree with that assessment by the Editor in Chief, as published in 2019.
This review meets the
Handbook criteria for being outdated and requiring an update for its credibility because even on its current research question and methods, the review:
- Has findings without a very high level of certainty (from very low to at most moderate certainty on primary outcomes);
- Is based on a protocol with methods from more than 20 years ago;
- Did its most recent search for eligible studies more than 10 years ago;
- Is missing at least two recent trials that are eligible based on its current inclusion criteria; [13, 14] and
- Lacks information that would be crucial for many decision-makers, including substantive additional data on harms from new and already-included trials, [13, 15] with the evidence in the current version of the review for this critical primary outcome being of very low certainty.
These problems have contributed to the large quantity of criticisms of this review, many of which have remained unaddressed for years. This will require updating of the review if these valid concerns are to be addressed. We discuss this in more detail in part 3 below.
In addition to the underlying problems in the current review, some of its other content has become misleading since it was last amended in 2019. For example, the review states that exercise therapy for people with this condition is recommended in treatment guidelines, citing a 2007 NICE guideline for people with ME/CFS. [16] This is reflected in a statement in the Plain Language Summary that apparently encourages use of the intervention: “Exercise therapy is recommended by treatment guidelines and often used as treatment for people with chronic fatigue syndrome.”
This is misleading for modern-day users of the review because the NICE guideline was updated in 2021. [17] It no longer recommends exercise as a therapy, and the guideline stresses the possibility of harm. Recommendations in other major treatment guidelines are consistent with this, including national guidelines for the US and Germany. [18, 19]
Keeping a review reliably up-to-date may not be necessary when there is general consensus on a question, and high certainty of evidence to address it, making the review “stable.” However, this version of the review has been contentious for over 10 years and is still being cited in clinical guidance. [20] Discordant conclusions among more recent systematic reviews underscore the need for up-to-date, reliable, and credible evidence synthesis on this question. [17, 21, 22] In failing to retire this review, Cochrane is actively participating in misinforming people on a controversial topic, with a review shown to be outdated and misleading.
Part 2: Re-issuing an unchanged review as an apparently updated one
Issuing a new citation for an unchanged review is not usual practice for the
Cochrane Database. Editorial guidance provides for adding an editorial note to an unchanged review as a “What’s new” event which does not require a new citation, [23] as was done for this review when the 2019 editorial note was added.
When a current Cochrane review is re-issued with a new citation date, it is indicated in PubMed, for example, as an updated review. [24] This is in contrast to Cochrane editorial policies [7] on updates, which are unequivocal:
“An update of a Cochrane review must involve a search for new studies, with those studies incorporated into the Cochrane review accordingly. Any other change to a Cochrane review, and any change to a protocol for a Cochrane review, is classified as an amendment.”
Issuing this review with a 2024 date gives the impression that the review is up-to-date. Indeed, the Altmetrics page for this review [25] leads to a Wikipedia entry that recently described the review as “updated in 2024.” [26]
On 31 January 2025, the editorial note for this review was amended to include: “…which includes studies from searches up to 9 May 2014.” The May 2014 search date was already noted in the abstract. The specification of search dates does not clearly convey to readers who are not systematic review experts that this review may be out of date. The date of the most recent search alone is not a reliable indicator of currency. Reviews may already be out of date when they are published, or have findings that remain reliable for many years when there is a high level of certainty and/or the question is no longer studied.
Cochrane’s response to our letter of concern included the following rationale for re-issuing the review and assigning it a new publication date:
“The recent editorial note from December 2024 was published with a new citation version of the review to ensure it was discoverable and indexed by third-party services, such as PubMed.” [3]
If Cochrane wishes to publish notes that are discoverable and indexed, then they could be published with their own citations, using the publication type “comment.” [27] The note could still be included within the review, but the date of the review would not change.
As Cochrane’s editorial guidance makes clear, an editorial note can be added manually to a PubMed record. [23] Publishers have had direct access to editing citations in PubMed for many years. [28] Indeed, another review, which had an editorial note attached in the same week as the exercise review, was done in the usual way: The note was added in PubMed without changing the date of the review. [29] Further, the 31 January 2025 amendment to the editorial note on the exercise review was added to PubMed without needing a new citation.
Meanwhile, people searching for systematic reviews are likely to mistakenly believe that this is the most recent systematic review on this subject. Citations of the review will also give people the false impression that this review is recent. The misleading 2024 date increases the urgency of amending the editorial note to make it clear that this review is outdated and unsuitable for clinical decision-making.
Part 3: Unaddressed criticisms
One of the first projects started by the IAG was gathering, and analysing the content of, criticisms made of the exercise review up to August 2020. [30] Some of the criticisms had not been published, but most were publicly available. The collection included 12 comments published through the
Cochrane Database commenting system.
Our as yet unpublished content analysis identified over 100 unique points of criticism about the contents of the review, fewer than 10% of which have been made redundant by updates or amendments to the review, or by changed arrangements for it, such as moving responsibility for the review from the Common Mental Disorders editorial group. The implications of some other points of criticisms have been only partially addressed, for example, the change in diagnostic criteria for ME/CFS recognising post-exertional malaise (PEM) as the hallmark symptom of the disease. [12]
Since August 2020, a further nine comments have been published in the
Cochrane Library. In addition, we recently submitted another ourselves. [31] Further, complaints of serious error requiring investigation have been submitted directly to Cochrane. [32]
Our initial content analysis of critiques was emailed to Cochrane editors by the IAG lead (Hilda Bastian) in May 2021 for circulation to the authors, and again to the Cochrane editorial unit in 2023. Cochrane editors have replied to some comments on the
Cochrane Database indicating that the criticisms raised would be addressed in the planned update, for example in a response published in August 2020. [33]
The
Cochrane Handbook points to the Cochrane Collaboration principle of “being open and responsive to criticism” as part of ensuring the quality of Cochrane reviews. [34] This was not done for the criticisms of this review. The inability to adequately address the accumulated criticisms of this review without a full update increases the urgency of amending the editorial note to make it clear that this review is outdated and unsuitable for clinical decision-making.
References in
a post below
