News from Cochrane

Now if their product catalogue could shut down and be taken offline, that would be great. I imagine some people would still clutch to it, but it would be diminished. It's not as if they bother keeping up to date anyway, I don't even think they bothered doing anything about Long Covid. Not that it would have been any good.

Their business model is over anyway. Systematic reviews are already feasible using agentic LLMs. I have seen several software projects published in recent weeks that would take months of work for an experienced team, the quality is very high, and it will only keep getting better. Reviews like this are perfect for AIs, they're just very labor-intensive and the skill level of the top 20 AIs is already more than high enough. Much better review can already be produced in a few hours for a few hundred dollars, and the speed, quality and cost will keep improving, which is not something humanly possible, manually doing this is already fully maxed out, and is in fact regressing over time.

It just hasn't happened yet because both academia and health care are very slow at adoption, but even that can't stop the inevitable.
Imagine going from the byword for good evidence based reviews and practice and trusted worldwide to the situation they find themselves in now.
It was always a meme, an ad about an advertising product. It caught on but it was never more than a hallucination. Like the very concept of evidence-based medicine, it sounds good on paper but in reality it's just transparent varnish over a very messy process that is very corrupt and so prone to hallucinations it makes the first version of ChatGPT look serious.
 
Much better review can already be produced in a few hours for a few hundred dollars, and the speed, quality and cost will keep improving, which is not something humanly possible, manually doing this is already fully maxed out, and is in fact regressing over time.
Have you seen any that are able to properly assess risk of bias? That seem to be the point where most reviews posted here fail, either by being too lenient or by taking the author’s words for it (e.g. them claiming the study is blinded when only the data analyst was blinded).
 
Have you seen any that are able to properly assess risk of bias? That seem to be the point where most reviews posted here fail, either by being too lenient or by taking the author’s words for it (e.g. them claiming the study is blinded when only the data analyst was blinded).
That's probably the hardest part. The 'soft' things that can't be assessed objectively. Right now I think with the right instructions the top models would do better, but better is too subjective. Applying rigorous criteria would basically throw away 99% of non-pharmaceutical trials, and that would be considered a bad thing by many, even though it would be a dramatic, objective improvement.

I think they can do that right now. Like they could follow something like a version of GRADE, or better, and do a far better job at it, but the people who decide whether it does a good job would have to balance scientific integrity and the rising popularity of alternative pseudoscience. People and their interests are a bigger problem than what the models can do. There is so much that depends on it. Objectively, the amount of money wasted on CBT trials is very likely in the several billions, there are so many of them. And the industry won't be able to accept it, will whine, delay and obstruct.

So it will probably take a few rungs up the ladder of intelligence to push those concerns aside, but that'll probably happen while the holdovers are arguing about models several generations before that. We've seen a backlash from mathematicians to AI models being better at their work. Medicine will have this moment soon as well, and I doubt they'll handle it any better. This is transformative stuff.
 
There are different layers to this, though, and things that can be done right now. There is a huge bias starting in the literature search and selection of trials, and that would be more difficult to handle. The literature search is the more demanding part, and the selection of trials is very arbitrary. This is how they can get away with pretending that nothing new has happened since then.

I think this is one of the reasons why they hate the review from the NICE guideline. Usually psychosomatizers simply filter out 95%+ of trials because they're garbage, and doing a proper analysis of that would make it clear that there is no value in doing more because it's all been overdone many times over, and the average quality is extremely low. But that's different from what Cochrane does. They don't do systematic literature reviews, they only review treatments. Although systematic reviews tend to never be really systematic anyway. No one wants to get corrected the way NICE did to them.

However simply getting a top tier model, giving it the Cochrane handbook and telling it to apply its rules, to redo the review with the trials they selected, would for sure give a different assessment and conclusion. Very likely across the board, too. Most of their reviews would probably have most of the same flaws, just not as extreme.
 
Back
Top Bottom