'AAAS: Machine learning 'causing science crisis'', BBC article Feb. 2019

I can't read the article but as i have said in other threads machine learning is todays shiny new object
The data out is only as good as the data in and how good the algorithm is.
One can get lucky and sometimes its good enough to lead to something useful but i would never rely on it nor would i burn money on it that i don't have

 

A valid concern but scientists doing science the "old way" are not connected to some deeper level of reality, they use data with the same flaws and issues that comes with reducing reality into data, just on a much smaller scale because they have to reduce dataset size to be able to handle them. This problem has nothing to do with ML and everything to do with the fundamental problem of properly describing reality, one that has always existed.

This is a bizarre issue to raise in this context, it's like saying relying on electronic computers is potentially misleading because they make so many calculations that nobody can check them. True enough, but that's also true when it's a wholly human process. With a large enough dataset it becomes so much work that no one can afford the luxury of allocating the budget to double-check that calculations were done right initially.

Meanwhile some "scientists" are basically promoting the idea that you can treat serious chronic illness with clay modeling, interpretative dancing or jumping on a mat using statistical sleights of hand, cherry-picking and maximally inducing bias. There's always some good and some bad but the idea that scientists have a deeper intuitive understanding of reality that transcends data collection is just disconnected from reality. If anything, the history of science shows that intuitive understanding is almost always wrong, hence the need for a process that promotes objective conclusions as free from human bias as possible.

When humans were involved in PACE, they basically distorted the data at every single step. That's a wholly human problem and they could have done exactly the same thing decades before computers were invented.

 

I don't know the stats but I suspect that machine learning was only used in a tiny portion (if any) of the studies that led to the awareness of a replication crisis especially as the term was coined in 2012 and many of the studies failing replication predate that by many more years. Researchers have been using poor methodology, cherry picking, basing results and exaggerating conclusions long before AI ever came along.

 

I like that the evolution of machine learning is highlighting the huge real issues around medical research--that so much is wrong.

I'm worried here that a conversation with subject experts can wind up with the experts providing input skewed to a political agenda with the hope of getting data that is confirmatory but wrong. I don't know are there safeguards in place for this?

 

assumption:

- there was a global urn assigned to me
- containing the 10,000 most popular meds (by patient choice)
- and at every visit to the doctors office
- i had drawn 3 meds

im wondering, if this had been more successful than the shamans decision.

something like "hypergeometric prescription" ... perhaps they could test, if its more likely to have a "hit" with this.

 

I didn't say its completely useless but i do say its not a panacea.
And there is a huge difference between heuristics of reading the colour of traffic lights or determining if something is a dog or a bump in the road vs medical research where your trying to find patterns and figure out how biology and physics works.
I am also not saying don't use it, i am saying its at best a supplement and should not take priority over humans who have far superior heuristic abilities and ability to figure out how things work.
Since its a supplement fund it at that level. If i had to choose between a machine learning rig and its programming or a good human scientist i would pick the human. If i had the funding to do both then thats fine. But in ME research we are very money constrained, we need every expert cortex we can get.
Also Waymo/Tesla have huge amounts of money to spend on developing their dog/traffic light/traffic cone sensing algorithms. They also have lots of human capital and the ability to hire as many as they want.