Bristol Randomised Trials Collaboration

I am thinking of sending him a personal letter along the lines of .... there are only a couple (?) of researchers and scientists in the UK prepared to stand up and say what a disgrace the PACE trial is: most of the ones signing the letter to The Lancet calling for an independent re-analysis are American – all but two from the UK are retired. If there are so many unable to risk the wrath of the old boy network, who prefer to pass on the other side and let patients suffer, what makes him think that this new group will be any different?

A longer version, of course. But would you prefer me to send my own version, or would you like to be involved in its creation and/or sign it?

 

That sounds like a great idea @Graham - I know for one that you will write a great letter so I’d be extremely pleased to put my name to yours!

 

OK, here is my first draft. It isn't in any way a scientific analysis, but is a personal rant.

As you know, I'm only too happy to receive highly critical analysis and rewrite the whole thing, or send my own version and help in creating a group one.

I have created a separate thread on it for anyone who would like to be involved (https://www.s4me.info/threads/bristol-new-network-prof-munafo.5802/).

Of course it won't do much good, but I'm not willing to let it pass.

 

The more I see of this the more it looks as if this is all part of a deliberate policy.

The blurb talks of 'high quality pragmatic randomised controlled trials' but there is no such thing. A pragmatic trial is something quite different from a standard RCT. What they mean by pragmatic here is 'rough and ready' - nothing more.

It is clear that this 'feasibility' gambit is a deliberate policy - in fact it may be what they think is their key party trick. Whether they learnt it from Dr Crawley or vice versa I don't know.

I think this is all part of the behaviour pattern of NIHR. NIHR was set up to do 'health service research' , which is supposed to focus on health care practicalities. But from the outset it was clearly a political tool for doing research that would bolster policies that somebody wanted to bolster.

At one time this sort of bad practive was shown up by Cochrane, but of course Cochrane has gone the same way.

 

Apologies for just popping in.

I am not able to read the manifesto linked on the University of Bristol website announcing a "Reproducibility Network that aims to improve the rigour and reliability of UK-led scientific research". Co-authors are Marcus Munafò and Dorothy Bishop, among others. At a first glance, it seems to make some reasonable points, but maybe also some less reasonable? Anyone read it?

https://www.nature.com/articles/s41562-016-0021

Dorothy Bishop already proved she won't act according to scientifically rigorous principles: http://www.virology.ws/2018/06/25/trial-by-error-my-exchange-with-professor-bishop/

[edited, thank you, @Jonathan Edwards ]

 

I could not see any link to a manifesto?
Do you mean the BRCT website or Munafo's network?
So far I id not think Munafo was linked to BRTC.

 

Apologies, I meant:
http://www.bristol.ac.uk/news/2018/september/reproducibility-network-.html

 

Maybe this:

https://wonkhe.com/blogs/a-manifesto-for-reproducible-science/

 

Seems to be a similiar manifesto. [Edit: It's an artcile by Munafò on the manifesto and the new network]

I referred to this one [the manifesto published on Nature (January 2017)]: https://www.nature.com/articles/s41562-016-0021. It's linked on the University of Bristol website: http://www.bristol.ac.uk/news/2018/september/reproducibility-network-.html

 

I meant Munafo's network. Confounded his network and the BRTC after reading (not properly) @Graham's post:
https://www.s4me.info/threads/bristol-new-network-prof-munafo.5802/#post-105367
and seeing Munafo's name also mentioned in this thread.

Now I understand that the only connection between the BRTC and Munafo's nettwork is their affiliation to the University of Bristol. (Perhaps a legitimate question to ask why Munafo, as a Professor of Biological Psychology, is not affiliated to the BRTC then?)

Apologies again for the confusion. Edited my previous posts.

 

The piece by Munafo and Bishop et al is all very well but I think it could be described as 'fluff-wiping'. (The practice of starting to do a job by picking something up, wiping the fluff off and then putting it down and forgetting to do the job.)

Pretty much everything in the manifesto is obvious and has been known about for generations. It makes nice lists but these are exactly the sorts of lists that give rise to the 'validated tools' of assessment that allow people like Cochrane to conveniently not notice the things that really matter. It seems a little bit like the mafia going to church on Sunday.

What I would suggest we actually need are things like exposure to public review. Any study published publicly (and every study that is started should be published publicly with open access) should appear on a webpage which has a mechanism for uncensored scientific feedback. The PACE paper should appear alongside any criticisms people can raise.

This might frighten some scientists but it is worth noting that this was in the past a standard method in some branches of medical science. At least until 1980 the Physiological Society in the UK published papers in its journal only if those papers had been presented to a meeting and following any comments anyone wanted to make in response were voted suitable for publication by consensus. I well remember a paper being blocked because the method for restraining an animal was considered unethical.

 

Thank you for reading and assessing the Munafó manifesto, @Jonathan Edwards .

 

I've just had a read over the manifesto. The full doc is here.

It seems to be authored by a lot of people who have genuinely interesting things to say about how to reform science. Bishop is on there, but probably one of the least heavyweight figures on the author list.

I find the document makes a genuine and useful contribution to science, by presenting in a simple condensed form a lot of the key issues in science today and the social, political and financial factors that shape them. It mentions the phenomena commonly identified as factors in poor reproducibility of science, including:

p-hacking. This is when you don't get the result you want, so you run another study in a slightly different way and only report the second, successful study. It also applies if your main outcome measure showed nothing significant, so you find another outcome measure that does, and report only that in the abstract.

Harking. This is when your result goes the opposite way you expected, but you fit your hypothesis to the findings, and present the paper as if you were predicting that outcome all along.

Publication bias. Only papers with significant results end up getting published.​

There are a few measures suggested, including lots of forms of "independent oversight", but I'm sceptical of that, because apparently the PACE trial had some form of that, and we all know what went down there. There can also be problems finding independent people who have sufficient expertise but at the same time, have no motive to either assist or obstruct the work. But some of the others are good, and nice to have collected in the one place:

Registered Reports. This means the journal publishes the details of the rationale and method of the study before the data is in. In true registered reports, acceptance of the paper is based on the rationale and methods presented there, and not on the outcome. The idea is that then publication will not depend on whether the study "worked out" or not.

Disclosure of COIs. Its good that they mention non-financial COIs. Like wanting to get published.

Funding replications. We don't have enough people out there bothering to find out whether they could replicate the results of some big study in their own lab. This is because its hard to get replication work published (journals and grant bodies are biased towards "exciting new" ideas).

Transparency and Openness Promotion guidelines. This is good. Providing guidelines as to best practice, including how to deal with issues like confidentiality.

Incentives. Encouraging employers to preferentially employ and promote/reward researchers that demonstrate a commitment to open science practices.

None of these ideas are novel - they've occurred to all of us interested in reform - but its useful to have them integrated into a single document.

 

https://twitter.com/user/status/1144180613664169984


Bristol Trials Centre

from #ICTMC2019

https://twitter.com/user/status/1181526829838082048

https://twitter.com/user/status/1181512334315008001

https://twitter.com/user/status/1180456357931212808

https://twitter.com/user/status/1181585751550676995

 

It's so worrying having Bristol take a lead on how clinical trials should be run.

 

It's baffling that they do not seem to apply what they are telling everyone else to what goes on in their own departments. Maybe the people presenting at these events are simply unaware of the detail of Crawleys research.
I could liken it to a certain ME charity.........