David Tuller: Trial By Error: Cochrane's Report on Courtney's Complaint

Cochrane's Report on Courtney's Complaint

I now have a copy of Cochrane’s report about Courtney’s complaint. It is clear from the analysis that Cochrane’s concerns involved the scientific accuracy and integrity of the exercise review. Contrary to the claims of the exercise review’s defenders, Cochrane was not responding to a hysterical mob of patients but to the cogent and well-reasoned argumentation of one patient in particular.

 

Do we know anything about the date on this document and by whom it was written?

 

The date is at the end: April 10, 2018.

The author is the person at Cochrane who was given responsibility to assess Bob's complaint.

 

What are Cochrane's reasons to reject and bury this report? It makes for a completely different review.

Where does it come from? Or is it sources and methods?

How does it respect Cochrane's policies to bury evidence of possible harm and deterioration and the true nature of bias and strength of evidence? This is basically complete distortion of the quality of evidence, the difference between high and low overall quality.

How likely is it we are going to find the same from the Lancet about PACE and PLOSone, especially about the release of the data on the latter?

Are there any steps we can take collectively to accelerate something, anything? There are cascading consequences that can come as a result of this move and the time for denial is over. Someone needs to answer all the questions and fast.

 

I'm not sure what "sources and methods" mean, but it is from a reliable source. Why Cochrane isn't open with this kind of stuff is a question, but I guess they would call it "internal work" or something like that. Cochrane doesn't like a PR scandal.

Someone did leak e-mails to Reuters last fall. I don't think Cochrane wasn't very happy with that. The troubling part is that only one side of the story got out. Seems like the Reuters journalist didn't read this complaint. That is a problem.

 

I've just added an update to the post. Norway has a website where agencies list their documents and correspondence and people--anyone--can ask to see them.

 

To me, it looks like Cochrane is trying to avoid embarrassing Larun and acknowledging that they got things wrong and that their processes failed. Courtney's complaint was so solid that they can't pretend he was wrong, but they certainly don't seem overly concerned that Larun was wrong in the first-place and then responded to Courtney pointing out her failings with dismissive "we must agree to disagree" nonsense. I don't think that they've 'buried' the report recognising Courtney was right (a report that was really unnecessary imo - Courtney clearly explained himself so that anyone could see for themselves that he was right) but they don't seem overly concerned by the fact that Larun has shown herself to be, at best, incompetent, or that her Cochrane review is continuing to cause patients problems.

That is a level of transparency that amazed me!

I just searched for 'lightning process'... 90 results: https://www.einnsyn.no/sok?offset=25&searchTerm=lightning process

 

yeah, I was surprised also.

 

Thank you, @dave30th, for bringing this report to light!
I wish Cochrane added this report at a prominent place to the review.

Have I understood properly that the person in charge of assessing Bob's criticism agreed with all of the key points Bob made?

P.S.: Typos in the added update: "That is also the month when Dr Tovey leaves his position as Cochrane’s as editor-in-chiefl it..."

Edit: I think I don't understand this paragraph:

Does this mean Bob's formal complaint included the omission of objective outcomes, but this point was not recognized in the Cochrane report as a key issue because Larun's justification had been accepted? Or was this issue not addressed in the report because all other comments, regardless how substantial they were, were not assessed with the same thoroughness as the formal complaint was?

 

So lets see Alem made himself much worse getting the PACE data, Bob dies from bad healthcare and perhaps his imprisonment and forced exercise.
Funny how both cases are about pushing beyond our envelope, what the PACErs claim should be curing us. Both these people should be alive and healthy by their energy expenditures
These two paid the price while the perpetrators claim they are the real victims

@dave30th A story idea?

 

Does anyone think it would be worth sending this new evidence with a request for a correction to Reuters and the Guardian?

 

Yes

 

I agree about requesting corrections, but I also know that any corrections made will be tucked away where no one much will see them except those who are looking hard.

 

"Authors must avoid giving direct recommendations, or referring to any intervention as ‘safe’
or ‘effective’. Specifically, the following statement must be removed;

“We think the evidence suggests that exercise therapy might be an effective and safe
intervention for patients able to attend clinics as outpatients.” "

If only this were implemented (not just on the Cochrane review).

 

I see we are going to have to keep an eye on @dave30th to ensure that he does not sneak any "literary" allusions past us.

 

I misread the title every time.

 

For comparison, this is what the Authors' conclusion of the abstract *still* says:

Have they provided any reason at all why this hasn't been changed?

 

So in summary, it seems that Cochrane agreed with Robert Courtney on using the FINE trial unpublished data and the PACE trial selective reporting bias. But it's not clear if this will significantly alter the results. It seems mostly an illustration of Larun et al.'s bias in conducting this review.

The most important critique - leaving out all objectives outcomes - is not mentioned. Vinks analysis showed that most of the GET-trials had objective outcomes. Some found improvements, but most of them didn't. So the conclusion of that would probably be that GET fails to show improvement on objective outcomes.

As I see it the main critique addressed in this document, is the fact that at follow-up, GET no longer showed a significant improvement on all outcomes except for sleep. So normally you would expect the review to write something like: "GET showed significant improvements post-treatment but no longer at follow-up". But instead of simply saying that, the Cochrane commentator suggests it is best to state that no reliable conclusion can be made of the follow-up data. So he/she wants the review to state:

So the outcomes at post-treatment are deemed reliable to mention, but the data at follow-up isn't. What the differences between the two?

• If we take fatigue as the main outcome, 453 patients in the intervention group and 387 patients in the control group gave data post-intervention. Data came from 7 studies.
• At follow-up there was data for 373 patients in the intervention group and 297 patients in the control group. Data came from 4 studies.

That doesn't seem like a large difference to me. It's confusing to only state the post-treatment outcomes and act like the follow-up doesn't provide reliable outcomes. There ain't much of a difference between the two. In both cases, the majority of patients came from three studies: Powell et al. 2001, the FINE trial (Wearden et al. 2010) and the PACE trial (White et al. 2011). The interesting thing is that those three trial reported the follow-up data as their main outcome! So I think it's misleading, as the Cochrane commentator suggests, to state the post-treatment data and consider the follow-up data too unreliable to mention. That's contrary to how these trials were conducted and reported. And for obvious reasons: the response bias is probably much larger shortly after treatment. That's why well-conducted trials measure their main outcome some weeks/months after the end of treatment.

 

It seems that the whole methodology of Cochrane fails to give an accurate view of what actually happened in these trials. Anyone looking at the data (not the propaganda of the authors) of the GET-trials neutrally would come to the following conclusion: "there seemed to be an improvement on subjective outcomes post-treatment but that disappears at follow-up and objective outcomes showed no improvement. The initial improvement was probably due to lack of blinding and response bias. So GET doesn't work." The Cochrane method of doing a review seems to avoid that conclusion by taking each part of the outcomes separately. The lack of blinding, for example, should be used to interpret the results. Instead, Cochrane simply addresses it by downgrading the certainty of their conclusions.

If they were to take each objective outcome (VO2 max, walking test, fitness test, employment, disability payments) separately, they would probably conclude that for each outcome there isn't enough reliable data to reach a conclusion. But that would miss the obvious conclusion that these trial do not show objective improvement if you look at all these objective outcomes together. I hope this will not be the case, if they decide to include objective outcomes.