Machine Learning-assisted Research on ME/CFS

[mariovitali]

Thanks. If I understand correctly, though, your network analysis is based on mentions in abstracts and the text of a publication and not on the actual data of studies?

First of all, Thank you for the reply. Much appreciated.

What you say is correct. All of the work that was made was an attempt to connect medical concepts that have been appearing to the text of the abstracts. This was the first part of the work , the Network Analysis (which was later used by Wenzhong Xiao).

The second part of the work was taking the various symptoms of MECFS, retrieving abstracts related to each of these symptoms and then asking from machine learning to tell us which combination of topics could predict a symptom vs a non-symptom state.


I do not quite understand what you mean with "actual data" of studies so I would appreciate your time in explaining what this means. But for the moment let's look at results so far given the input I described.

The key question here is : Can the above appear by pure chance or not? What do we need to make a claim that the above methodology works indeed ? From what I understand a key part is to find the number of plausible topics for MECFS and then run an appropriate analysis. I find it extremely hard that this can reliably take place but I am open to any suggestions. I can provide a full list of identified concepts from my part.

If this is cherry picking then I see no harm done apart from wasting 15 years of my life in trying to convince others but at least I can now function and have a near-normal life. So harm for my time (and income) but no harm for the patient's health and wallets.

But if indeed is something taking place here then we are talking about negative bias consistently over the years. Can patients "afford" not to look at this methodology? @forestglip @Hutan also would appreciate your input.

 

[ME/CFS Science Blog]

I do not quite understand what you mean with "actual data" of studies so I would appreciate your time in explaining what this means.

The raw data of the experiments, for example, in CSV format.

A big problem I see is that almost all studies misrepresent their data in the abstract and text (to make it look like a bigger deal than it is or to promote the authors' favoured theory).

So I think machine learning/AI/network analysis, etc. will only be useful if we skip how authors present their findings and only train it on the actual data, with strong selection criteria so that only high-quality experiments, such as, e.g., DecodeME, are included.

 

[forestglip]

But if indeed is something taking place here then we are talking about negative bias consistently over the years. Can patients "afford" not to look at this methodology?

Looking at the list above, I do wonder about matching findings by chance. How many genes did your program identify, and how many genes did the latest PrecisionLife study identify? Didn't the latter report a few hundred or a few thousand genes? Seems to be an opportunity for a few genes to overlap by chance.

I think it would be important to clearly outline what exactly the methodology is. I only have a vague idea of what is being done.

I'm sorry, I don't otherwise have the energy to really try to understand this, and I'm not sure I'd be able to follow a detailed description of the entire pipeline anyway, but it might be good to make for demonstrating to someone that there's some promise to this tool. It's too hard for anyone new to these ideas to try to discern what is being argued from 10 pages of posts in this thread, and various assorted posts on other threads and websites.

 

[mariovitali]

The raw data of the experiments, for example, in CSV format.

A big problem I see is that almost all studies misrepresent their data in the abstract and text (to make it look like a bigger deal than it is or to promote the authors' favoured theory).

So I think machine learning/AI/network analysis, etc. will only be useful if we skip how authors present their findings and only train it on the actual data, with strong selection criteria so that only high-quality experiments, such as, e.g., DecodeME, are included.

I see so my understanding is that you reference Garbage-In, Garbage-Out (GIGO). If the input data is garbage then the output is Garbage.

The thing is that in my analysis I did not look at the results of the studies. The search space of the analysis was built by identifying which concepts appeared together in research papers so basically a co-occurrence analysis was at the basis of it. I wanted to understand the connections between symptoms and various medical concepts.

If this is so, does this change your GIGO belief?

@forestglip you said :

Looking at the list above, I do wonder about matching findings by chance

Wondering is one thing but we cannot dismiss a hypothesis -especially one with repeated confirmations- that easily, correct ? My message above was about how can we identify whether the computational techniques I used did in fact do better than pure chance. Which is the search space that I should use ? 19000 human genes ? How many pathways ?How many symptoms ?

In any case Thank you both for your time

 

[forestglip]

Wondering is one thing but we cannot dismiss a hypothesis -especially one with repeated confirmations- that easily, correct ? My message above was about how can we identify whether the computational techniques I used did in fact do better than pure chance. Which is the search space that I should use ? 19000 human genes ? How many pathways ?How many symptoms ?

One option is using a hypergeometric test to check how likely it is to have as large of an overlap as you got by chance. Described here with a built-in calculator: https://statisticsbyjim.com/probability/hypergeometric-distribution/ (Though the calculator doesn't seem to work when the sample size is too large, but this one does: https://sebhastian.com/hypergeometric-distribution-calculator/)

So the problem could be described as: there are ~20,000 genes, which would be the "population size" of genes (N). PrecisionLife identified 259 genes (or did you compare your results to the larger list of 2311 genes?) that we can consider the number of "true events" in the population (K).

Your pipeline came up with some number of genes (i.e. you selected a sample from the population of 20,000). Let's say your pipeline selected 300 genes from the full population of 20,000. This is the sample size (n). And let's say 4 of them were "true events" (they overlap with the 259 genes from PrecisionLife). We want to know how likely it is to get 4 or more success in your sample, if we assume your sample of genes was chosen randomly.

Using the numbers above in the calculator (just example numbers, since I don't know how many genes your pipeline produced or whether you used the larger or smaller PrecisionLife set): N=20000, K=259, n=300, k=4. If these numbers were correct, then you'd get P(X ≥ 4) = 0.54624, i.e. about a 50% chance of getting 4 or more PrecisionLife genes in your pipeline assuming your pipeline was just selecting 300 genes at random, and thus you would not reject the null hypothesis.

 

[mariovitali]

@forestglip this is very helpful. Thank you so much

 

[ME/CFS Science Blog]

The thing is that in my analysis I did not look at the results of the studies. The search space of the analysis was built by identifying which concepts appeared together in research papers so basically a co-occurrence analysis was at the basis of it. I wanted to understand the connections between symptoms and various medical concepts.

If this is so, does this change your GIGO belief?

Not really, because the co-occurrence of concepts is probably based on the text of the manuscript, not on the actual data.

Suppose that several studies found a slight increase in metabolite X, but do not mention it anywhere in their paper, it's only visible in the supplementary data: would the algorithm be able to pick this up?

Conversely, suppose several papers highlight immune cell Y in their abstract as being related to ME/CFS because this is a hot topic. But their underlying data do not support these statements; the results are described misleadingly to increase publication chances. Would the algorithm be able to see through this?

How does it deal, for example, with findings such as XMRV, which comes up in ME/CFS papers multiple times but turned out to be a false finding best to be ignored?

 

[mariovitali]

Not really, because the co-occurrence of concepts is probably based on the text of the manuscript, not on the actual data.

Suppose that several studies found a slight increase in metabolite X, but do not mention it anywhere in their paper, it's only visible in the supplementary data: would the algorithm be able to pick this up?

No this would not happen.

Conversely, suppose several papers highlight immune cell Y in their abstract as being related to ME/CFS because this is a hot topic. But their underlying data do not support these statements; the results are described misleadingly to increase publication chances. Would the algorithm be able to see through this?

Again, this would not happen

How does it deal, for example, with findings such as XMRV, which comes up in ME/CFS papers multiple times but turned out to be a false finding best to be ignored?

All of your concerns are valid and there are actually many other issues, for example Liver X Receptor may be found in the literature as LXR, NR1H3, NR1H2 so these discrepancies introduce further problems. You will have to see however this problem taking into account measures from Information Retrieval and more specifically Precision/Recall and the F-Score :

https://en.wikipedia.org/wiki/Evaluation_measures_(information_retrieval)

Now, the first example is related to having high precision. Remember, I wanted to build a map of associations between medical concepts. Supplementary data are irrelevant to such information. Your second example (XMRV) is spot on and it indeed introduces bias in the co-occurrence analysis. However :

1) Since there were millions of abstracts been analysed then it is logical to say that such bias is minimised, correct ? Well known associations will have a high co-occurrence while the ones you mentioned will have very low.

2) The abstracts I analysed were not only related to ME/CFS. They were related to symptoms of ME/CFS so any ME/CFS related concept bias is expected to be low.

For the record, someone else mentioned the following on how to validate the methodology I used :

As for your method, to validate it, you would need a bench test where you calculate the sensitivity and specificity of the predictions. You should also have random predictions to compare your method with.

You can use diseases with a consolidated list of genes (diabetes, schizophrenia, etc), and you could see if your method predicts from literature before - let's say - 2015, what has been discovered by standard GWAS performed after that year. If you only use predicted genes, rather than pathways, it should be much easier.

Once you have demonstrated that this method makes predictions on certain diseases that are significantly better than random predictions, you can apply it with confidence to other diseases.

Thanks again for your time.

 

[mariovitali]

I uploaded this video where I post a hypothesis given my 10-year research and latest studies. I will send it to researchers and patient organisations.

As always your views -especially the ones that identify errors and incorrect claims- are most welcome.

 

[mariovitali]

The slides of the video can be found here :

https://docs.google.com/presentation/d/1Pd4pSm6PM1teiaCWXFrM8Sfn6No6spKnhWr9rWUWTuo/edit?usp=sharing

 

[mariovitali]

Since this took place on Twitter , I can post here. Michal Tal (https://talresearchgroup.mit.edu/team) from MIT came across my work (dialogue at the end of this post) and wanted to connect.

I was not aware of her work. Apparently she is looking at CD47 + SIRPa , Mitochondrial stress and autophagy. CD47 is a critical gene for proper efferocytosis.

In ME/CFS, It appears that we have an interesting collection of several related genes to it identified such as LXR/RXR (Hanson / Armstrong), PTPN11 + GRB2 (Snyder et al). GRB2 is directly linked with Vitamin K (was identified by Network Analysis in 2017) and MERTK.

All of these genes (LXR, PTPN11, GRB2, MERTK) have been identified since 2017 onwards and there appears to be more convergence. I will keep you all posted.

 

[mariovitali]

This is how it is looking at the moment. As I mentioned in another thread, It is really great that all of our conversations are being recorded here for future reference. Entries in red are preprints

 

[hotblack]

Can you tell us more about what this image is showing and what it means @mariovitali Is it it showing studies which have replicated genes/processes you have previously identified? Is this just the most recent studies?

I look forward to hearing more about any discussion with Michal Tal too.

 

[mariovitali]

Sure, the image shows how machine learning, certain causal inference methods and network analysis have found earlier (by a median of 7 years) than conventional research several research targets. Yes it shows studies that have confirmed these earlier findings.


What is worrying is that there is no patient organization involvement despite my repeated requests to use this framework but most importantly having repeated confirmations over the last decade. In the video I uploaded I discuss a number of metabolic issues that may be at play. How many years will pass until they are looked at?

The way I see it, it is not only about funding. It is also about individual choices on which information gets exposure and which not, which research moves forward and which not.

Immunologists will put weight on the immune system being the cause for MECFS, brain experts will suggest the brain as the key player and so forth. No systematic effort to see the big picture is being made.

Hopefully, unbiased algorithms will take care of this in the near future and make these choices for us, based on facts.

 

[hotblack]

Have you got a complete list of the genes you are most interested in @mariovitali ? Apologies if you’ve already posted, thought it easier to ask the expert than search!

And to play devil’s advocate, this is a view of successes, do we have a view of failures or at least of not yet confirmed associations you have found? Something I’ve noticed with what I’ve been looking at is when there is loads of data I find it easy to draw conclusions or find associations for just about anything. It can often ‘feel’ right to me but that doesn’t mean I’ve really done more than cherry pick. It can be a nice way to explore ideas or come up with theories to be discussed, dissected and proved/disproved though.

And hope I have made it clear that no offence at your work is meant here. I’m also interested in how ML can help. Although wonder it may need to learn for itself rather than be grounded in human biases to really find something new.

 

[mariovitali]

Please find my responses inline, below

Have you got a complete list of the genes you are most interested in @mariovitali ? Apologies if you’ve already posted, thought it easier to ask the expert than search!

I will have to go through my notes however I made sure that there is a written track of everything via documents shared, e-mails, social media posts (mainly Twitter/X) and a blog which can be found here :

https://algogenomics.blogspot.com/

And to play devil’s advocate, this is a view of successes, do we have a view of failures or at least of not yet confirmed associations you have found? Something I’ve noticed with what I’ve been looking at is when there is loads of data I find it easy to draw conclusions or find associations for just about anything. It can often ‘feel’ right to me but that doesn’t mean I’ve really done more than cherry pick. It can be a nice way to explore ideas or come up with theories to be discussed, dissected and proved/disproved though.

I am glad you are asking this question which makes sense. This is very difficult to achieve and I would surely like some help here on building this pool of predicted concepts. If you look at the blog you see several concepts (now known) : LXR, ABCA1, GRB2, Choline deficiency, Bile acid metabolism but also others not currently been identified such as MAMs, CYP1A2 etc. This is pretty simple to do.

However, please look at the end of the post which shows a snapshot from a random forest model output which exists in the blog. My question is should all of these concepts appearing in the graph be included in the pool of predictions or not?

In the way I have been working, perhaps the top 3 concepts should be identified as predictive targets, not the rest of the list. But this is my interpretation.

And hope I have made it clear that no offence at your work is meant here. I’m also interested in how ML can help. Although wonder it may need to learn for itself rather than be grounded in human biases to really find something new.

Yes I do understand and I would like to Thank you for this attitude. You are having a balanced approach and not a negatively biased one.

 

[mariovitali]

I am posting this message mainly as a future reference. I posted many times previously about being happy that our thoughts are being recorded in this forum.

With full responsibility of what I am posting here, I hypothesise that the beginning of the end of ME/CFS is here, for the majority of patients. In my opinion and based on my 10-year research, we now have all the knowledge to improve a patient's QoL. More work is required for identifying how subtypes of problems affects the patients and the type and sequence of interventions. I did not say that it is over.

The elements of the solution are all here . The potential problems in ER-to-Golgi trafficking, ER Stress*, the phospholipids* disruption, the disruption of N-glycans*, glycosaminoglycans* and heparan sulfate* and how this leads to several immune, metabolic and vascular issues, the proteostasis problems, lipid transport (ABCA1*, LXR*, CYP7B1* etc). Efferocytosis may be the next major finding -and this is why we have delayed PEM (my hypothesis)-, hopefully some research group will look at it. Heparan sulfate (recall CA10 from the GWAS) could be a big one especially for vascular problems in patients. Everything related to N-Linked glycosylation is crucial also for the immune system.

( * = these concepts all previously identified via computational methods and confirmed by conventional research)

I have always been saying that when it comes to finding a solution for ME/CFS, it is not only about funding. What also is important is to be able to make the appropriate evaluation of research (A Big Thank you @ME/CFS Science Blog , @Hutan , @forestglip , @jnmaciuch , @Simon M @Trish by the way) but also make the right decisions on which research targets are actually being looked at (and essentially being funded).

During the beginning of my research efforts with computational techniques (2015), I found myself asking again and again whether I was "cherry picking" along the way. As more targets previously identified were appearing years after in published research, my confidence would increase that -most likely- there is no cherry picking involved. A friend of mine told me that "these confirmations are very unfortunate because otherwise you wouldn't have wasted so many years pursuing something you made no money from"

If I am wrong then this will mean that I am a very arrogant and wrongfully persistent person and that I basically wasted many years of my life trying to find a solution starting from a wrong basis.

Now it is time to hypothesise the possibility that all of this work was NOT cherry picking and despite repeated efforts no one listened. I have been trying to understand so hard on why no one has been convinced so far. Are the researchers, the patient organisations or the patients themselves to blame ? If despite repeated confirmations no action was taken who's mistake is it?

Your replies will say something. No replies will also say something. What I would kindly ask from you is to keep in mind that I gave a significant amount of my life trying to find a solution despite no funding.

 

[Utsikt]

A friend of mine told me that "these confirmations are very unfortunate because otherwise you wouldn't have wasted so many years pursuing something you made no money from"

If I am wrong then this will mean that I am a very arrogant and wrongfully persistent person and that I basically wasted many years of my life trying to find a solution starting from a wrong basis.

There is no way to know beforehand if research will pan out the way one hope it will. All exploration involves prioritisation, going down one path over all the others. The reasons for picking that path might be reasonable and logically sound, but still lead to a dead end. That does not imply the effort was wasted.

I have been trying to understand so hard on why no one has been convinced so far. Are the researchers, the patient organisations or the patients themselves to blame ? If despite repeated confirmations no action was taken who's mistake is it?

I am not your intended audience, so feel free to ignore my reflections.

As someone who’s only vaguely aware of this kind of AI, and try my best to follow the research on ME/CFS as a layperson, I have not been able to understand how you’ve gotten to the answers you say you have. And I’ve not been able to understand how certain concerns have been addressed, like how most publications and especially abstracts are polluted with wast amounts of noise and bias.

There might be a barrier of technical skills, but there might also be a barrier of communication. If people are not able to understand what you’re doing or what the data they are looking at means, how would they be able to justify prioritising spending time on it rather than something they already believe is worth their efforts? There is an alternative cost to working on something else, and people will need help understanding why they should pivot or divide their attention.

If someone perceive your models as primarily a generator of hypotheses, it will probably have little perceived value to them because they already have more than enough hypotheses and can easily make more on their own.

 

[LizWorthey]

Love this way of pulling out the concepts (and agree with all the discussion on potential biases and potential for issues in interpretation).

I feel that there needs to be consideration of some of the complexity on pathways/programs/components potentially being impacted in ME/CFS.

If there are two or more distinct (at some level; level being completely arbitrary) molecular mechanisms that are important for ME/CFS to occur (either both capable of leading to ME/CFS separately or both necessary for ME/CFS in a two hit model), then you could have sets of concepts not related to each other, but still of equal importance. Excluding some in favor of others would lead to loss of knowledge.

IMO this possibility should be built in to your model. Also, everything in cells is so interconnected that you will have a defect in a mechanism (with a set of associated concepts) that downstream initiates a cascade of subsequent, indirect, and damaging responses (secondary injury/effect) with their own sets of concepts.

If there are multiple paths to ME/CFS and there is convergence of mechanisms at some point in the pathobiology then the downstream signals would be stronger in the data that the causal component associated concepts. But knowing what is causal is incredible important (even though treatments often come from the downstream impact's signals).

There is now data on some of those relationships (based on scientific output from the last 40 years) so it should be possible using generative AI with curated datasets to try to arrange/identify which are primary cause associated concepts and which are secondary effect associated concepts using key pairs/directionality maps/ordered relationships etc.

Just some morning thoughts. I do like this angle.

 

[mariovitali]

In another coincidence (?) efferocytosis is very active during pregnancy.





using AI but fairly sure this is correct :



This is why PEM is delayed (my hypothesis). Unbeknownst to me, in 2017 I did not know about efferocytosis but I got very close (used Phagocytosis instead). Note that I should have used efferocytosis instead of phagocytosis but the paper which I took the figure from (https://pubmed.ncbi.nlm.nih.gov/25136342/) mentions efferocytosis. Back then I haven't identified ABCA1 yet :