NICE and Cochrane sign collaborative agreement to deliver ‘living’ guideline recommendations

Has NIHR cut off Cochrane's funding, and if so, why?

 

There is going to be an official announcement in March. It is the infrastructure funding they are cutting off for UK groups and for the Cochrane Central Executive which has become big and unwieldy since Mark Wilson took over in 2012. There is a video here of Karla

https://www.youtube.com/watch?v=mfAlll5N71s

talking about the "challenges" - starts at around 2 mins - and encouraging people to watch three presentations by NIHR's Sally Davies (2013), then Tom Walley (2017), then Ken Stein (2020). The first two warning Cochrane to sort itself out and Ken Stein's effectively saying that Cochrane had blown it (

https://www.youtube.com/watch?v=ukr7B39pyio

) . The writing has been on the wall for eight years.

 

OK - I see

 

Copied post

I submitted a request for:
"The latest full business case for the NICE Cochrane collaborative agreement*. If not the same document, please also provide the business case that was presented at the highest-level meeting where the collaborative agreement* was approved.’
NICE replied - "A copy of the paper submitted to NICE’s executive team in June 2021, outlining the NICE Cochrane Collaboration Agreement, is attached to this email".

It's accessible at the web address in the next post.

Haven't looked at it

@Caroline Struthers

 

Here's the pdf document that @FMMM1 asked me to upload

 

Thanks Michiel.
@Caroline Struthers Michiel has posted the first response from NICE.

 

Moved post

I've had a brief look at the business case. Comments from others on this forum that the "logic" was saving money i.e. through avoiding "duplication" (NICE &Cochrane both doing reviews) came to mind. If you search for "dup" or some such then you'll see 4 hits.

Another thing was that NIHR have been funding Cochrane reviews [Note 1 - 3]. So have the current crap Cochrane reviews been paid for out of public money? @CRG has highlighted that NIHR seem to be funding some of the BPS research; so they have form. Perhaps the funding of poor quality (Cochrane) reviews, by NIHR, could be challenged via the APPGs &/or via MPs (parliamentary questions etc.). Ultimately the Department/Minister who funds a public body [Department of Health and Social Care funds NIHR] is responsible for the actions/behaviour of that body.

It looks like those interested/affected by NICE guidelines will have to scrutinise draft guidelines, reviews used in the development of those guidelines ---- NICE seems to be in danger of departing from its founding principles --- to produce evidence based guidelines.

Note 1
From the Risk Assessment Table:
"Risk" -
"Failure to align priority review questions between NICE and Cochrane.
"Mitigation and assurance"
"Cochrane are actively encouraged by NIHR to align reviewing activity with wider system needs. Proposed approaches to engagement include links with NIHR, Cochrane Executive Team and Cochrane UK to ensure priorities are aligned where possible."

Note 2
"NIHR-funded Cochrane reviews on NICE priority topics:
64. From 2018 to 2020, NIHR commissioned 14 Cochrane reviews on topics identified by the NICE Centre for Guidelines as being relevant to decisions to update guidelines, or to be used in the development of guidelines (eg Tobacco, epilepsy guidelines);
65. Feedback was positive from NICE, NIHR and Cochrane in that the commissioning process ensured alignment of review questions between NICE
and Cochrane, and important reviews were delivered to time and budget."

Note 3
"21.The NHS has invested heavily in Cochrane over the past 30 years, with funding currently channelled through the Evidence Synthesis Programme
(ESP) of the National Institute for Health Research (NIHR). This funding provides support for Cochrane infrastructure in the UK, but also provides
programme grants and one-off small grants to support specific reviews via 'incentive awards'. The ESP specifically encourages Cochrane to produce reviews of relevance to the NHS in general, and NICE in particular."

Few more extracts that caught my attention:
8. Collaborative working with Cochrane will maximise use of Cochrane reviews
and topic expertise and enhance efficiencies by reducing the number of
duplicated surveillance and evidence reviewing activity across the two
organisations.

• support funding requirements to deliver the introduction of novel
collaborative approaches between NICE and Cochrane to enable living
recommendations across NICE guideline suites.

13.In order to deliver living guideline recommendations, these need to be
underpinned by 'living systematic reviews' which has been defined as: “a
systematic review that is continually updated, incorporating relevant new
evidence as it becomes available”. Defining features of living systematic
reviews are:
a. continual, active monitoring of the evidence (e.g. monthly searches of
core databases);
[EDIT - Comment - someone on this forum called that search fetish - seems appropriate].


18.During guideline development, existing systematic reviews on similar review
questions are usually identified. Whilst these existing systematic reviews are
routinely considered, they are frequently excluded from the guideline due to
one of two reasons:
a. subtle but important differences in review questions that make the
review unusable by the NICE guideline committee. For example,
important interventions, comparators or outcomes are not considered
(e.g. those around health-related quality of life outcomes needed to
inform health economic analyses);
b. review currency and quality. For example, concerns around dates of
last searches and inclusiveness of searches, methods for assessing
risk of bias, analytic approach etc that would adversely impact on the
robustness of the evidence syntheses and reliability of the
interpretation and conclusions

19.Invariably, failure to use existing systematic reviews means that a new review
is undertaken by a NICE guideline development team. This duplication in
research effort comes at a substantial cost to NICE in terms of both time and
resource.


22.There are good reasons for working closely with Cochrane- effective
collaboration between NICE and Cochrane supports the underpinning
purpose, principles and strategic aims of both organisations, while also
promoting best use of UK public funding. Furthermore, collaboration also
supports the sharing of best methodological practice in the development of
high quality, high impact systematic reviews.

34.The GSD also seeks to recognise the value to NICE guideline development of
engaging with the topic and methodological expertise within Cochrane Review
Groups. For example, the mutual benefits of Cochrane review authors
contributing to relevant guideline committee meetings to support discussions is
highlighted. Opportunities for Cochrane to contribute to guideline development
through other routes such as registering as a NICE stakeholder or joining a
NICE guideline committee are also signposted.

synergistic

novel collaborative approaches

Appendix 1: Examples of collaborative working with
Cochrane Review Groups
During guideline development;
56.Tobacco: preventing uptake, promoting quitting and treating dependence
(update) - Publication expected Nov 2021

NIHR-funded Cochrane reviews on NICE priority topics:
64.From 2018 to 2020, NIHR commissioned 14 Cochrane reviews on topics
identified by the NICE Centre for Guidelines as being relevant to decisions to
update guidelines, or to be used in the development of guidelines (eg
Tobacco, epilepsy guidelines);

 

I'm unclear as to the full time scale, but the NIHR states: "Grants are awarded to support a substantial and coherent programme of work that includes up to 40 reviews - both new Cochrane reviews and updates to existing Cochrane reviews." and lists 22 that have been published: see Cochrane Programme Grants

https://www.journalslibrary.nihr.ac.uk/nihr-research/cochrane-programme-grants/

According to the 2017 funding review paper Evaluation of NIHR investment in Cochrane infrastructure and systematic reviews

https://www.journalslibrary.nihr.ac...t-in-cochrane/NIHR_Cochrane_Report_Feb_17.pdf

"SRs conducted by Cochrane are undertaken by 52 Cochrane Reviews Groups (CRGs)36 worldwide, of which 21 currently receive infrastructure costs funded by the NIHR,37-39 and 24 have an editorial base in the UK."

and

"NIHR have committed £16 million funding for CRGs over the five year contract period (2015-2020)"

Table 4: NIHR-funded Cochrane Review Groups
Cochrane Review Group Web address
1. Airways http://airways.cochrane.org/
2. Bone, Joint and Muscle Trauma http://bjmt.cochrane.org/
3. Cystic Fibrosis and Genetic Diseases http://cfgd.cochrane.org/
4. Dementia and Cognitive Improvement http://dementia.cochrane.org/
5. Common Mental Disorders http://cmd.cochrane.org/
6. Ear, Nose and Throat Disorders http://ent.cochrane.org/
7. Epilepsy http://epilepsy.cochrane.org/
8. Effective Practice and Organisation of Care (EPOC) http://epoc.cochrane.org/
9. Eyes and Vision http://eyes.cochrane.org/
10. Gynaecological Cancer http://gnoc.cochrane.org/
11. Heart http://heart.cochrane.org/
12. Incontinence http://incontinence.cochrane.org/
13. Injuries http://injuries.cochrane.org/
14. Neuromuscular Disease http://neuromuscular.cochrane.org/
15. Oral Health http://ohg.cochrane.org/about-us
16. Pain, Palliative and Supportive Care http://papas.cochrane.org/
17. Pregnancy and Childbirth http://pregnancy.cochrane.org/
18. Schizophrenia http://schizophrenia.cochrane.org/
19. Skin http://skin.cochrane.org/
20. Tobacco Addiction http://tobacco.cochrane.org/
21. Wounds http://wounds.cochrane.org

From that list it seems there has been only very limited opportunity for BPS influence to be paid for by the NIHR via grants to Cochrane.

The 2017 review has led to a reappraisal of NIHR funding of Cochrane and a final decision on what happens when the current commitments run out in 2023 is due in March 2022.

Re: the point about NIHR funding of BPS focused research, I don't have the figures but I'm pretty confident that since completion of PACE, NIHR has been the majority (perhaps the sole funder) of BPS research applicable to the NHS, i.e without the NIHR there would be no measurable BPS focus in UK health research.

I speculated in another post about how a closed system of NIHR > Cochrane > NICE could allow a paradigm (e.g BPS) held by NIHR to be foisted on NICE via a compliant or incestuous Cochrane CRG. I don't know how realistic that scenario is.

 

Too realistic---

Unfortunately, despite the fancy words in the business case, re aligning Cochrane reviews with the needs of the system (NICE's needs), I would not be surprised if the outcome was flawed Cochrane reviews leading to flawed NICE guidelines. E.g. @Caroline Struthers has pointed out that Cochrane doesn't currently remove all of its inappropriate systemic studies - I can't see a road to Damascus moment changing Cochrane to impose rigorous quality control.
The NICE committee, which revised the ME/CFS guideline seems (on balance) to have been both highly competent and lived up to the objective of producing an evidence based guideline. The 2007 NICE review panel did not deliver an evidence based guideline - so you can't necessarily rely on the NICE review panel.
Looks like those with an interest are going to need to monitor draft guidelines and respond to the NICE consultation + engage with APPGs, MPs etc. as necessary.

What's going to happen to the NICE technical teams i.e. who review the evidence and cause all of those problems by actually assessing it - bit like removing the checks in the system - that way you can get on with scamming people (us).

 

Hi there. Sorry - only just seen all this. Great you got the business case document. It will be useful as a follow up reference when I write to the new Chief Exec of NICE. They cannot justify using an unregulated charity to supply the scientific evidence underpinning clinical guidelines, surely...???

 

An unregulated charity which has a very poor record. I've copied an extract from my previous correspondence with NICE*. If NICE look at the quality of the base studies (which they claim*) and exclude studies which are not appropriate, then why link up with Cochrane? The claimed benefit is the avoidance of duplication yet, if the response from NICE* is accurate, then it's not clear there is any benefit - NICE assess the studies independently.

All of this relies on Cochrane objectively assessing the evidence - certainly contrary to the evidence re ME/CFS reviews. I noticed there was some cherry picking of the evidence for the business case - smoking studies were used, a rare example of Cochrane getting it right I guess.

Previous response from NICE to me:
*"In terms of quality assessment of the evidence, the NICE guideline committee exercised their clinical judgement to decide which components of GRADE were important for downgrading evidence quality, for example, risk of bias, population directness, and threshold(s) for minimal clinically important differences. The full details of the review protocol and data analysis relating to exercise therapy in NG206 can be found in Evidence review [G] for the non-pharmacological management of ME/CFS<https://www.nice.org.uk/guidance/ng...acological-management-of-mecfs-pdf-9265183028>. The clinical evidence summary tables for exercise interventions, including GRADE analysis, begin on p.147.

These differences between the methods and processes of Cochrane reviews and NICE guidelines can result in different quality ratings of the evidence in GRADE, as in this case with exercise therapy for ME/CFS.

Please also note that in Oct 2019 Cochrane announced<https://www.cochrane.org/news/cfs> that “a new approach to the publication of evidence in this area is needed; and, today we are committing to the production of a full update of this Cochrane Review”.

Regarding the announcement in Sept 2021 that NICE have signed a collaborative agreement with Cochrane. This is a high level agreement, and NICE is in the process of working with Cochrane in an attempt to align processes and methods in the future, and this will take time."

 

I know this sounds hopelessly idealistic (or just plain wrong) but a business case is supposed to inform and ask for (informed) consent. Since this cherry picked a few +ve examples (smoking) but did not accurately reflect the:

• breadth of Cochrane reviews; or
• bizarre e.g. the non-zero effect (of no clinical relevance); or
• failure to remove reviews, shown to be flawed, in a timely manner;

then it didn't inform the decision maker.

OK there are euphemisms like the need to align Cochrane reviews with the wider needs of the system (or some such); however, that hardly covers "informing" the audience (those asked to agree to the NICE/Cochrane collaboration).

@CRG highlighted concerns re NIH -- there's an old joke in policy --- there's evidence based policy development (official policy) and policy based evidence development --. The business case looks like the latter i.e. the evidence was selected to produce a positive image of Cochrane and thereby support the NICE/Cochrane agreement. The person(s) who drafted it was clever enough to insert a few euphemisms about aligning Cochrane reviews to the needs of the wider system, and the agreement being high level, and early stages --- covering up that Cochrane reviews are currently unfit for (NICE) purpose. If that's true then the question is, whose agenda is pushing this? Maybe that is why the NICE Chief Executive is getting off side - new culture.

Action?
I'm wondering if we could e.g. come up with a more realistic appraisal of Cochrane's systemic reviews? We could then highlight that to the APPGs (ME/CFS & Covid) and ask them to raise the issue - copy to NICE of course.

If nothing else it may alert the APPG MPs and highlight to NICE that concerns have been raised with interested MPs. There are probably a lot of folks in NICE who would welcome a challenge to the new culture (Cochrane) i.e. the folks who joined NICE to produce evidence based guidelines.

 

This is the nub of it!

How could anyone demonstrate Cochrane reviews as being fit for purpose?

The fact that Cochrane reviews cannot be withdrawn even when demonstrated to be out of date and/or erroneous, and also the major detail they are exempt fromFOI because they have charity status, should surely make it impossible for NICE to stand over this collaboration!

 

I could probably produce a report for the APPG on how many out of date Cochrane reviews are still on the Library with no update in progress. Maybe any older than five years? And also, which of these reviews are used in NICE guidelines. Cochrane is supposed to update its reviews every two years, or more frequently, unless the field is acknowledged to be static - ie. no uncertainty remains. I can check the policy as I think they watered it down as the reviews were virtually never updated this frequently when I worked for Cochrane. Cochrane also used to have a policy allowing withdrawal of out of date reviews, but this changed in 2019 (https://documentation.cochrane.org/pages/viewpage.action?pageId=123472593), just after I and others had met with Karla Soares Weiser to ask her (again) to withdraw the Exercise review, which was then published with amendments in October 2019.

 

Wow "I could probably produce a report for the APPG on how many out of date Cochrane reviews are still on the Library with no update in progress."
For an organisation (Cochrane) being recruited by NICE to produce "living guidelines" that sounds more like fossil than living.

Yes, I think that would be a good way to inform the APPGs (I'd try ME/CFS & Covid) of the reality and the risk to those they seek to represent (plus everyone else in the UK). It strikes me that the drivers, i.e. for the NICE/Cochrane collaboration, are political/ideological - I'm not persuaded that there are real world benefits (other than political/optics). EDIT - I'm surprised anyone is NICE is pushing this collaboration, for the reasons you indicate above, this looks like a recipe for years of trying to defend Cochrane from entirely reasonable challenges.

The evidence, you have set out above, should have been reflected in the business case and the actions, to address those issues, set out. So the cherry picking meant the issues were glossed over.

You're way ahead of me (much better informed) re this; I'm happy to assist in any way I can.

 

I think NICE are rescuing Cochrane (maybe Cochrane asked to be rescued?) because they are losing most of their funding from the NIHR. So I guess NICE will commission reviews based on what guidelines they are prioritizing, then and pay Cochrane for the data from reviews it commissions from its volunteer contributors (who may or may not be biased) to use to inform guidelines before the data is edited and published as a Cochrane review...or something.... I will read the business case between the lines. I doubt it was NICE's idea.

 

Interesting, but so far Cochrane systemic reviews seem to be characterised by the failure to assess the quality of the evidence [unblinded studies with subjective outcome criteria +++]. So this would require a pretty much 180 degree turnaround.
NICE having to validate the evidence looks like entirely reasonable duplication i.e. if Cochrane carried out the initial review; however, it at least reduces the saving (understatement it probably increase the cost).

Yes, if the new improvement is recruiting Cochrane, then the folks in NICE are probably dismayed by the fact that they have to spin/deal with all of the entirely reasonable crap coming their way. I can't see how anyone in NICE would be keen on this link up so maybe a new broom in NICE and/or push from somewhere else. Who are Cochrane chums with ---- Yes Minister episode, where the Minister/Sir Humphrey get some gong, comes to mind.

 

Yes, it would be channeling my inner archaeologist!

 

Thanks
Occurred to me that it might be a good idea to try to inform some of the charities, as well as the APPGs; i.e. since the charities probably lobby the APPGs (and indeed the Department/Minister for Health and Social Care).