Treating patients suffering from myalgic encephalopathy/chronic fatigue syndrome (ME/CFS) with sodium dichloroacetate, Comhaire 2018

I think its something to do with the maximum likelihood optimization over what are assumed to be probabilities using the cost function:
J(θ)=−∑i(y(i)log(hθ(x(i)))+(1−y(i))log(1−hθ(x(i))))
With y(i) being the target ith training sample and x(i) being the input for the ith training example and hθ being the function that is being optimized (with θ being the weights). Given its a class being fitted with y(i) being either 0 or 1 then the idea is this is the conditional probability of y|x. But I've not really understood yet. There is a tutorial from Andrew Ng at Stanford (regression then logistic regression) who does some excellent tutorials (and coursera courses) on machine learning.
http://cs229.stanford.edu/notes/cs229-notes1.pdf

It feels like there needs to be something around sample theory as well. Which is my objection to this work. If you take a small sample and use that to derive a model then basically you will pick up on random features within your training set. So you need to know that the data you train on is in someway representative of all the set of data otherwise features in your predictor variables that easily minimize error but just for that given data set can be picked up on. Normally a model would be tested against a separate data set or the training set is split into portions and a cross validation technique is used (averaging error on say 4/5 used for training and 1/5th used for testing over all the different combinations). But I don't think this has been done in this case (but I'm happy to be corrected and this is one of the methodological issues I have with this work).

I think when doing significance tests on a trained model against the data it was trained on you will just get something about how well the model fits the data it has been trained on which should be good (unless the function being learned is more complex than the model and the training data sufficiently captures that relationship).

[Edit]

It looks like given the optimization performed the outputs are probabilities given the training set. I still don't get the intuition but I may eventually.

 

So the impression I get from your paper (which could be wrong I only read it quickly) is that you gave a supplement to a number of patients with a disease that is varies over time and then sorted the patients into those who improved in a given time period and those who didn't. It seems hard to draw a conclusion about that given there is no control arm. It could be that this just represents the expected number of improvers with no treatment without a control we don't know.

But then you take a set of features that describe the patients such as answers to the fatigue scale and then you fit a model on all the data you have to claim that you can classify which will respond and which don't. This doesn't test a hypothesis further than saying you can fit a logistic regression model to a given data set. It says nothing about how the model generalities to unseen cases. To me (as someone who has done and is getting back to doing research in machine learning) this doesn't say anything about your data and it seems methodologically strange.

Then you seem to be making very positive claims about the supplements and the model. If you were saying that you think the results were positive and worth a blinded trial then I would agree.

That is why I'm concerned. But have I read the paper wrongly, did you have a control group? did you test your model on unseen data?

[added]
Currently it feels like hype without sufficient justification for the hype.

 

I wonder how much of that was people on this forum downloading it...