Single-cell transcriptomics of the immune system in ME/CFS at baseline and following symptom provocation, 2022, Ahmed, Hanson et al.

[hotblack]

They seem to discus similar concepts in broad terms. But when it comes to actual evidence they are quite divergent.

Different changes in expression

The other paper

CXCL11, CCL2 and TNF were persistently elevated in individuals with LC compared to in NI donors up to month 9

This paper

Specifically, we observed upregulation of CCL4, CX3CR1, SELPLG and ITGALin patients, while CXCR4 expression was suppressed at baseline

Different characterisation of aberrant monocytes.

The other paper stimulated monocytes with an inactivated bacteria and observed

Stimulation resulted in a reduction in the numbers of CD14+ and CD16+ monocytes compared to unstimulated samples

They particularly base this on

stimulated LCAM CD14+ monocytes exhibited significant downregulation of the inflammatory response, IFNγ signaling, IL-10 signaling, cytokine signaling and IL-6–JAK–STAT3 signaling pathways

While this paper was an exercise challenge and reported changes based upon some software/analysis from another paper I don’t understand

we used Seurat v4 (Hao et al., 2021) to determine the number of significantly dysregulated genes per cell type

Certain cell types exhibit strong signals of transcriptome dysregulation in patients, with CD4+ T cells (naïve and effector/memory subsets; clusters 0 and 1, respectively), monocytes (clusters 2, 9 and 10) and cytotoxic NK cells (cluster 3) being the most prominent

This is just a sample. I don’t fully understand all the details nor have I been through thoroughly, but it seems the details are quite different in each paper.

Dysregulation, aberrant behaviour and inflammation can cover quote a lot. I’d be more interested if there was any commonality.

Not to say there may not be things here. But they seem different things.

 

[V.R.T.]

There might not be much to it, though:

Oh yeah, I remember now

 

[V.R.T.]

They seem to discus similar concepts in broad terms. But when it comes to actual evidence they are quite divergent.

Different changes in expression

The other paper


This paper


Different characterisation of aberrant monocytes.

The other paper stimulated monocytes with an inactivated bacteria and observed

They particularly base this on

While this paper was an exercise challenge and reported changes based upon some software/analysis from another paper I don’t understand



This is just a sample. I don’t fully understand all the details nor have I been through thoroughly, but it seems the details are quite different in each paper.

Dysregulation, aberrant behaviour and inflammation can cover quote a lot. I’d be more interested if there was any commonality.

Not to say there may not be things here. But they seem different things.

Ah yeah, not the same findings then. I'm most interested in the Hanson data. I wonder if the Hanson team has found the same thing in another cohort, or if they will try?

 

[Kilian_M]

One seemingly striking result in this paper is fig. 4C which shows the fraction of monocytes that are predicted to be diseased. I think this machine learning analysis is flawed. They did not publish their code and did not report on some details but I think they only split by cells into the training and test sets and they should have done by person. I have redone the analysis like that and the signal vanishes. I could not test how this affects the correlation of diseased monocytes and MFI since they did not release the MFI data.
You can check the code here (Anthropic Opus 5 generated)

 

[Utsikt]

You can check the code here (Anthropic Opus 5 generated)

Can you elaborate on what’s your own work and what’s AI?

 

[Kilian_M]

Can you elaborate on what’s your own work and what’s AI?

It was my idea that there was a poor split in the cells leading to this result. I prompted Claude to do the evaluation and present the results in a commented notebook. I checked the plots and the comments but only minor sections of the code. So the vast majority is AI generated. I don't think that makes it less true though.

 

[hotblack]

Ah yeah, not the same findings then. I'm most interested in the Hanson data. I wonder if the Hanson team has found the same thing in another cohort, or if they will try?

Yeah. Both say ‘something with monocytes’ but the mechanisms of what that is don’t seem to align.

I’m interested in their data too. Probably more than the findings; although I don’t think I’ve got the processing power (silicon or human) to do much with it.

Still not sure what to make of their results . I sort of wish they’d picked a thing and explained it well rather than thrown so much at us. Maybe they’ll follow up on that like you say.

 

[hotblack]

I think this machine learning analysis is flawed. They did not publish there code and did not report on some details but I think they only split by cells into the training and test sets and they should have done by person.

But we want to analyse at the person level don’t we? The machine learning algorithm is trying to learn the difference between healthy and control. If you look at things within a person at the cell level that’s trying to determine or learn something different isn’t it? And may be influenced or introduce other issues in the statistics. I’m not sure you can shift things like this and compare them in this way. I’m not an expert here! Just asking.

 

[Utsikt]

It was my idea that there was a poor split in the cells leading to this result. I prompted Claude to do the evaluation and present the results in a commented notebook. I checked the plots and the comments but only minor sections of the code. So the vast majority is AI generated. I don't think that makes it less true though.

No, but we have to be upfront about what’s what. You comment said the code was included and that it was AI-generated. Now you’re saying the entire notebook is AI-generated and that you haven’t even checked most of the code.

 

[Kilian_M]

But we want to analyse at the person level don’t we? The machine learning algorithm is trying to learn the difference between healthy and control. If you look at things within a person at the cell level that’s trying to determine or learn something different isn’t it? And may be influenced or introduce other issues in the statistics. I’m not sure you can shift things like this and compare them in this way. I’m not an expert here! Just asking.

They analyze on the cell level. You put one cell (i.e. the gene expression data from one monocyte) in the classifier and it tells you if it is healthy or diseased. They label cells from controls as healthy but do not label cells from cases since they are assumed to have a mixture of healthy and diseased monocytes. In the prediction step they compute the fraction of diseased cells for each person.
The problem is that if we train a model that tries to assign cells to persons instead of 'healthy' or 'diseased' it still performs quite well and therefore it also performs well with the 'healthy'/'disease' task. This memorization problem can be avoided by excluding all cells from a few people in the training phase and use them to test the model. They instead held out a random set of all monocytes (I assume).

 

[jnmaciuch]

One seemingly striking result in this paper is fig. 4C which shows the fraction of monocytes that are predicted to be diseased. I think this machine learning analysis is flawed. They did not publish their code and did not report on some details but I think they only split by cells into the training and test sets and they should have done by person. I have redone the analysis like that and the signal vanishes. I could not test how this affects the correlation of diseased monocytes and MFI since they did not release the MFI data.
You can check the code here (Anthropic Opus 5 generated)

Looking just at the text of the methods, it is possible there was train-test bleed like you describe. Although from your figures, the classifier trained with separation-by-cells had much much better separation between groups than the figure from the paper. The generated code also does all pre-processing from scratch which might add substantial discrepancies. For example, this code takes several huge leaps in how a monocyte is labeled--very different from the paper, which only used a specific cluster of classical monocytes that was defined by clustering all the cells.

This type of analysis requires dozens of deliberate analysis choices tailored to the dataset from the very first preprocessing steps. While you may be right about the underlying issue, I don't think the AI-generated code will be particularly trustworthy here