Red for danger in systematic reviews?, 2021, Andrew Moore

This is an interesting editorial in the European Journal of Hospital Pharmacy:

"Red for danger in systematic reviews?" by Andrew Moore

The full article is available here: https://ejhp.bmj.com/content/ejhpharm/28/6/299.full.pdf

 

If I understand correctly, the author argues that systematic reviews are too lenient towards trials with high risk of bias (those with a red light in the ROB overview Cochrane makes)

He argues that the appropriate way to deal with such evidence is not to add it up, but to disregard it:

"The fact of this high risk of bias in individual studies is all too often ignored when making conclusions or recommendations, but ignoring the red signals is a dangerous business"​

"Ideally, a systematic review will highlight these problems and come to appropriate conclusions. For example, a comprehensive review of cannabinoids for pain included 37 randomised, double blind trials.2 None was without some risk of bias with green across the board, and 28 had at least one red light. The appropriate conclusion was that ‘Studies … have unclear or high risk of bias, and outcomes had GRADE rating of low-quality or very low-quality evidence. We have little confidence in the estimates of effect.’"

[...]

"Plucking the useful 3% from the many systematic review submissions is hard work, but rewarding. One of the top cited articles for the EJHP over the past 3years was an overview review on medication adherence.15 It used evidence from eight good systematic reviews; evidence from a further 24 was excluded because of limited evidence, poor review methods, or both, meaning that 75% of our literature on this topic had to be discarded. But it was cited because quality was valued."

"For readers of systematic reviews, just bear in mind that red means stop"

​

 

This is quite relevant to the evidence on GET/CBT where all trials are normally rated at high risk of bias because of lack of blinding.

Busse and colleagues argued that it was wrong of the NICE guideline committee to disregard this evidence even if it was of low quality. In their view, this should have still led to a (cautious instead of strong) recommendation for these interventions using the GRADE approach.

In my view, this made no sense because then all sorts of questionable treatments would be recommended using the GRADE approach, no matter how unreliable the evidence is. The editorial by Moore seems to support this view.

Here's a discussion with Jason Busse about this on Twitter:
https://twitter.com/user/status/1366827683385135113

 

https://twitter.com/user/status/1452567830436667392

 

If no therapy could be offered without moderate to high certainty evidence, clinical practice would largely grind to a halt. Low and very low certainty evidence is still evidence, and can be used to make weak recommendations - in favor or against - a therapeutic option.

This statement from Busse really is disingenuous.

GRADE quotes examples of recommendation on weak evidence but they bear no resemblance to the GET situation. An examples that it is recommended to give antibiotics for presumed severe bacterial infection within an hour rather than within six hours. The evidence that it makes a difference is weak but it would be bonkers not to recommend it.

 

To help me understand please: How does Grade discriminate between that weak-evidence recommendation for administering antibiotics, versus dismissing the the weak evidence for GET?

 

GRADE says that recommendations do not follow directly from evidence quality. The implication is that you have to use common sense to make the step. Which of course you do.

The only value of GRADE is as a recipe for trawling studies by people who do not understand the detailed context and as a check list of things to remember to worry about. As a way to get a sensible answer it is full of holes.

 

Also, if the evidence of benefits is of low or very low quality, presumably the evidence about possible harms is similarly hard to take seriously. So that would seem to balance out any idea of a weak recommendation.

 

Yes, very much what was in the back of my mind. If there is a slim/modest chance of something providing major health benefit, negligible chance of it doing harm, the ethics of it are sound, and the costs not prohibitive, then it seems a good bet to go. Common sense, as @Jonathan Edwards says. If those criteria are not all met, then common sense suggests it's a different call.

 

Busse's rapid response in the BMJ had a number of pretty significant basic errors in it (claiming there was no GRADE evidence summary of findings table in NICE, getting the suggested small effect in the Cochrane review wrong, and more) which strongly indicated he had read neither the NICE or Cochrane evidence reviews.

I don't think there's much point paying much attention to what he has to say.

He is right that low quality evidence can still lead to a recommendation, but a number of factors need to be weighed up for that, which is what NICE did, and it explained that in its evidence review.

 

Haven't read but perhaps interesting for the discussion?

Largent EA, Peterson A, Lynch HF. FDA Drug Approval and the Ethics of Desperation. JAMA Intern Med. Published online October 25, 2021. doi:10.1001/jamainternmed.2021.6045
https://jamanetwork.com/journals/jamainternalmedicine/article-abstract/2785557

(Paywalled)

Abstract
In justifying the accelerated approval of aducanumab (Aduhelm; Biogen), US Food and Drug Administration (FDA) officials emphasized that many patients with Alzheimer disease and their families “made it clear that they are willing to accept the trade-off of some uncertainty about clinical benefit in exchange for earlier access to a potentially effective drug.”1

This reasoning raises a critical question. When patients have a diagnosis of desperation that leaves them facing a life-limiting disease without good treatment options, what role should their willingness to try a drug of unproven benefit play in regulatory decision-making? In this Viewpoint, we argue that patient voices should be integrated into the drug approval process, but without such deference that the FDA abdicates its responsibility to ensure drugs are safe and effective.

Also tagging @Hilda Bastian in case the thread might be of interest to the IAG.