PACE trial TSC and TMG minutes ordered released

[Adrian]

I really don't like the argument that publishing minutes stops debate. One research project that I ran we have quarterly 2 day meetings most of that time would be discussion and not minuted. But we also had a formal project meeting where decisions, finance, milestones, changes to the project plan etc were discussed along with formal reports from each partner and we had a project officer from the funding body attend that in a formal capacity. To my mind this formal project meeting would be the equivalent of these TSC and TMG with the discussion happening in research sessions and hence would not be prevented. But when it comes to formal decisions then the groups need to be accountable.

 

[Esther12]

I liked the decision much more on a second reading.
It is worth remembering this is a legal process and the ICO does not want to be appealed to the IT. They have to show they're being reasonable and have considered all the points made.

Yes, I think that I found the uncritical presentation of some of QMUL's misleading arguments frustrating, but it could be that they know to just try to avoid disputable section within a ruling. It's the overall judgement what really matters.

The ICO is well aware of the replication crisis. It was covered in the thousands of words and dozens of footnotes/references I provided.

Thanks for all your work on that. Maybe it helped affect the overall judgement, even if there were parts that still seemed strangely respectful of the processes around academia.

I took that more as a comment that it is not the ICO's job to judge the integrity of the researchers and that comments made in the judgement were not intended to do that.

Oh... I didn't take it like that, but I guess that is an alternative reading.

 

[Esther12]

In case anyone is interested, this judgement is now on the ICO website: https://ico.org.uk/media/action-weve-taken/decision-notices/2018/2258289/fs50687719.pdf

20 February 2018... I wonder when we're going to get those minutes, or hear that QMUL appealed?

Also, when Carol Monaghan asked Mark Walport about concerns over PACE, he used supposed TSC approval for protocol deviations occuring before data was available as a key part of his response (I've not seen the PACE trial researchers claim that they had approval from the TSC for their changes to their recovery criteria): http://parliamentlive.tv/event/index/6a03dbf8-ff09-4dca-b681-477d184c44c0?in=09:50:55

Maybe further highlighting the legitimate public interest in seeing exactly what was discussed? I think that it's getting harder and harder for QMUL to appeal.

 

[JohnTheJack]

20 February 2018... I wonder when we're going to get those minutes, or hear that QMUL appealed?

They have 35 calendar days from 20/02 to provide the information. Now, does 20/02 count? Does the 35th day count? Round about 28/03. 6:00 pm on the 27th?

 

[Jonathan Edwards]

Also, when Carol Monaghan asked Mark Walport about concerns over PACE, he used supposed TSC approval for protocol deviations occuring before data was available as a key part of his response (I've not seen the PACE trial researchers claim that they had approval from the TSC for their changes to their recovery criteria): http://parliamentlive.tv/event/index/6a03dbf8-ff09-4dca-b681-477d184c44c0?in=09:50:55

I think this is an important video snippet.

I know Mark Walport well from days in rheumatology and he is a highly intelligent individual. The implication of that is that he will absolutely see that the design of PACE makes it valueless. However, in this video he appears not to be aware of the problem. He says he has had some briefing. He then refers to some criticism of methodology but appears to think that this has been answered by the authors saying that they changed outcome measures before analysing the data. This implies that he thinks that this is the only substantive problem, when of course the overriding problem is built into the design from the outset.

One has to conclude that Mark Walport simply has no idea of the real problems with PACE. Presumably he has not read the trial publications. It would be immediately clear to an academic rheumatologist from the abstract of the main PACE paper that it was seriously flawed. Mark is an astute politician but the hallmark of a really astute politician is that they do not knowingly make themselves look stupid. I assume he simply does not know.

Having been head of medicine at the Hammersmith and moved on to posts in Wellcome and government admin Walport is perhaps the single most relevant person if one were to ask who represented the British establishment Medical Science viewpoint. And I do not mean establishment in a pejorative sense. Mark has a deep understanding of the complexities of medical research. He cannot possibly not see that PACE is absurd if he puts his mind to it.

I will email Carol Monaghan. Waldport says he will write about PACE. I think if he can be engaged seriously he will not be able to sit on any fences, as he seemed to do here.

 

[Adrian]

Also, when Carol Monaghan asked Mark Walport about concerns over PACE, he used supposed TSC approval for protocol deviations occuring before data was available as a key part of his response (I've not seen the PACE trial researchers claim that they had approval from the TSC for their changes to their recovery criteria)

I believe there was something about masked data being made available to the TSC (or a different committee) that was mentioned in the stats analysis plan and hence prior to approval.

 

[Sly Saint]

answered by the authors saying that they changed outcome measures before analysing the data

This gets repeated a lot. Have the authors ever said exactly what then prompted them to make the changes?

eta: it is baffling that all these great men of science, and academia don't require a definitive and congruous answer to this.

 

[Sasha]

This gets repeated a lot. Have the authors ever said exactly what then prompted them to make the changes?

Not in any way that makes sense.

 

[Amw66]

Being able to analyse FINE trial data and appreciate how to turn a null result into a positive one?

 

[Esther12]

I believe there was something about masked data being made available to the TSC (or a different committee) that was mentioned in the stats analysis plan and hence prior to approval.

I'd forgotten that. Would that have been to check for evidence of harms? It would certainly have also indicated that none of the treatments were dramatically more effective than SMC alone in the way thy had expected CBT/GET to be.

 

[Adrian]

I'd forgotten that. Would that have been to check for evidence of harms? It would certainly have also indicated that none of the treatments were dramatically more effective than SMC alone in the way thy had expected CBT/GET to be.

It was the data monitoring committee (not sure what that is!)

Blinding of the statistical analysis This document has been developed without reference to the PACE trial database. No analyses of outcomes relating to this strategy have been, or will be, conducted prior to final written approval of the analysis strategy by the TSC. Reports have been prepared with data presented descriptively by intervention (coded to maintain blinding) for the closed sessions of the Data Monitoring Committee. Consequently, both DMC and TSC were blind to intervention group, as were the trial statisticians. Data cleaning will be performed as blind to intervention allocation as possible. Decisions made during analysis concerning data or additional analyses will be documented.

Also in the stats plan

According to ICH E9, the statistical analysis plan should be pre-specified, completed after the protocol has been finalised but reviewed and possibly updated as a result of a blind review of the data carried out after the completion of data collection. It is suggested that details of the primary analysis should be clearly distinguished from those of supporting analyses and that the methods for handling missing data, outliers and multiplicity be described [12].

 

[Esther12]

I was just looking through reference 12 they referred to, and it had some stuff on blinding: http://www.ich.org/fileadmin/Public...Guidelines/Efficacy/E9/Step4/E9_Guideline.pdf

In single-blind or open-label trials every effort should be
made to minimise the various known sources of bias and primary variables should be
as objective as possible.

I wonder if 'blind review' of data is possible in an open label trial?

In the document the do still talk of blind review of data for an open-label trial though.

As everyone always thought, the statistical analysis plan is meant to provide more detail on analysis laid out in the protocol, rather than a dramatic change of every outcome, making it easier for the researchers to claim success.

5.1 Prespecification of the Analysis

When designing a clinical trial the principal features of the eventual statistical
analysis of the data should be described in the statistical section of the protocol. This
section should include all the principal features of the proposed confirmatory analysis
of the primary variable(s) and the way in which anticipated analysis problems will be
handled. In case of exploratory trials this section could describe more general
principles and directions.

The statistical analysis plan (see Glossary) may be written as a separate document to
be completed after finalising the protocol. In this document, a more technical and
detailed elaboration of the principal features stated in the protocol may be included
(see section 7.1). The plan may include detailed procedures for executing the
statistical analysis of the primary and secondary variables and other data. The plan
should be reviewed and possibly updated as a result of the blind review of the data
(see 7.1 for definition) and should be finalised before breaking the blind. Formal
records should be kept of when the statistical analysis plan was finalised as well as
when the blind was subsequently broken.

If the blind review suggests changes to the principal features stated in the protocol,
these should be documented in a protocol amendment. Otherwise, it will suffice to
update the statistical analysis plan with the considerations suggested from the blind
review. Only results from analyses envisaged in the protocol (including amendments)
can be regarded as confirmatory.

In the statistical section of the clinical study report the statistical methodology should
be clearly described including when in the clinical trial process methodology decisions
were made (see ICH E3).

 

[Jonathan Edwards]

I wonder if 'blind review' of data is possible in an open label trial?

This came to my mind. I think it is reasonable to assume that 'before the data were analysed' means nothing in the context of an open trial like this. Those involved will be constantly discussing with each other how people are getting on and if it is open label it will be obvious to everyone what the 'data' are going to show.

I wonder at what stage that newsletter saying how well people were doing was sent out? Was that by chance before the switch in criteria?

 

[Adrian]

From the quotes in @Esther12 message

If the blind review suggests changes to the principal features stated in the protocol, these should be documented in a protocol amendment. Otherwise, it will suffice to update the statistical analysis plan with the considerations suggested from the blind
review. Only results from analyses envisaged in the protocol (including amendments) can be regarded as confirmatory..

I'm assuming with PACE they replaced the protocol with the statistical analysis plan. I think they failed to document the protocol changes in the SAP.

I think a blind review of data could suggests different stats techniques should apply for example, due to the distributions of data.

 

[JohnM]

43. "When assessing the chilling effect the Commissioner has paid particular attention to its impact on the patient representative body involved in the trial. The involvement of such a body helps ensure that patient concerns are accommodated as far as possible and helps in the recruitment of patients to the trial. Their role is an important one. During meetings there may be occasions when a representative from such a body needs to give an initial view on the issue being discussed without having had the opportunity to consult their own colleagues on the matter. The Commissioner therefore considers the input of such a body may be more susceptible to the chilling effect. Therefore the Commissioner considers there are stronger grounds for withholding such information.", p12

Wouldn't that same representative of the patient representative body, acknowledge at the time, that they would need to consult with colleagues to confirm views expressed were indeed representative of their colleagues/members/the wider needs of PwME? If not, would it not be more likely, that they believed that their views represented the best interest of their colleagues/members/the wider needs of PwME?

...

 

[JohnM]

41. "... However that need for safe space only exists while those issues are being debated internally, and potentially for a short time after those issues have been decided. In this case the findings of the research were published back in 2011. Therefore by the time the request was made in 2017 the need for safe space had long passed.", p11

42. "The university’s main argument though is that disclosing these minutes would have a chilling effect on those involved in future research work and the candour of their discussions. The Commissioner accepts the possibility of there being such an effect. However the extent of that effect is dependent on the actual information in question.", p11-2

Why would this not also apply to the patient representative body?

...

 

[JohnM]

55. "A confidential annexe will be provided exclusively to the university setting out the types of information that may be withheld under section 36(2). This is limited to the anonymization of the minutes and the removal of information relating to the patient representative body.", p14

Other than the dismissal of the COI issue raised, this aspect of the ruling is what I find most troubling, and find "chilling". Essentially, representatives of a patient representative body would be allowed to express views privately, views which may significantly be removed from those they present publicly. As such, both patients and carers - be they members, supporters, or otherwise - and to a lesser degree perhaps, colleagues - would be unaware of which views have been expressed on their behalf.

This aspect of the ruling could well set a precedent, with far-reaching and 'unintended' consequences, particularly across the charitable sector?

Wishing everyone improved health and every happiness, John

 

[Esther12]

Essentially, representatives of a patient representative body would be allowed to express views privately, views which may significantly be removed from those they present publicly.

Maybe there's a view that private bodies should be able to present different views in private than they do in public? The UK's FOIA doesn't seem founded on that radical a commitment to transparency, and parts of it seem intended to allow quite some management of the public's understanding of important matters.

At this point, it is Action for ME that should be doing what it can to inform their members of exactly what they had been doing behind closed doors.

 

[Sly Saint]

inform their members