I’m sorry I don’t quite know what you mean. [Edit: I don’t think any chronic disease model is talking about a “single point of failure” except genetic disorders and that’s not what I’ve been discussing?] Maybe best to leave the conversation where it’s at to avoid derailing the thread.
I don’t want to speak over Daniel but from the methods it seems like imputation was only used for the specific step of pathway analysis using MetaboAnalyst—individual feature comparisons would not use imputation.
I think that part of the text refers to the joint pathway analysis--meaning that those 3 pathways had one lipid feature each, but the enrichment was also driven by features in other assays that are known to be part of the same pathway.
Here's Figure 1A. The caption and the text are quite clear that the things driving the selection of the three 'potentially dysregulated pathways' are each of the three features (ie one feature driving one pathway). The Steroid hormones biosynthesis datapoint is barely significant (I think that's before adjustment for multiple comparisons) and it rates approximately 0 on the x axis, which the caption suggests is a measure of pathway importance, which considers pathway enrichment.
Ah my bad! I thought that was the paragraph talking about Fig 2 not Fig 1 (since the bar plot showed some pathways having 1 lipid hit). I agree that having one upregulated metabolite doesn’t mean there’s evidence for enrichment of the whole pathway—I appreciate the text was upfront about there being only one metabolite at least.
I've just gotten back to the office so I haven't read the whole thread yet but just saw this comment (edit: I am editing in more responses as i see them). I'm not sure why it's raising a red flag for shoehorning so I'll explain what I did. I put the features into metaboanalyst and those three pathways came up as significantly affected. I reported what it determined just as it was described in MetaboAnalyst. I didn't manipulate or choose anything different.. it's an unbiased output of the software. The blue significance line wasn't drawn arbitrarily, it's the threshold that the software reported to me. I didn't linger on it and moved on to other results because I didn't think it was a clearly important result either (the polar pathway analysis that is, as we say it is based on individual metabolites which were not hugely different). I very intentionally chose not to belabour it further at that stage of the results section. I included everything that my analysis pipeline included for the sake of thoroughness and transparency even if particular results are or aren't convincing, and distributed my focus accordingly. I included a mountain of supplemental data for the sake of this completeness and transparency as well. I think my language was also pretty transparent in that it this little part of the analysis was an exercise included for thoroughness, just in case.
This is only for the metaboanalyst polar metabolite pathway analysis, the individual features I am showing in the paper and analysing elsewhere are all real data, no imputation. All of the scatter plots are real data. Most if not all of the lipids would have signal for every sample so there was no need to deal with missing values for those analyses. I tried very hard to present everything as transparently and readably as possible, hence the scatter plots with minimum stats graphics so as to let the points speak for themselves.
This is all why I intentionally used softer (but I think transparent) language to summarise the MetaboAnalyst output and in specific terms the rationale for its inclusion, and then move on to the more interesting results which received more focus and emphasis, especially at the Abstract and Conclusions which is where people are going to go looking for the important bits. As I say, it is there for completeness, transparency, and just in case it is a piece of the puzzle. The focus is firmly on the more clear results.
So, Figure 1B is just the actual values, and any imputed values aren't shown? How many missing values were imputed for the three polar metabolites in the Metaboanalyst metabolite pathway analysis?
I'm glad the suggestion of a dysregulated steroid hormone biosynthesis didn't make it into the discussion or the abstract, but I'm still a bit concerned that that sentence in the Results (edit - it's actually a title in the paper) overstates the situation. The paper says
It would be great if you could check the relationship between the actual values and age for LCL cholesterol sulphate levels.
The choice was made prior to obtaining any data, I decided to be thorough and include the whole pipeline so as not to keep any results from the community. That includes negative results.
Correct, and none.
It is based on the levels of that metabolite within context of levels of each metabolite in the whole metabolome, which is a different question to its levels in isolation.
See prior comment
What I'm saying in the prior post is that I have done this, specifically.
All good, please don't mistake my tone as terse, I'm just time poor atm so writing with brevity.
Yeah I'd seen what occurred with the Tate paper so I took this question to an accredited statistician we've worked with on other papers where PCA was used to test whether two features were able to produce clusters between clinical groups and they said that it's a valid application of PCA. In any case, the separation is clear from the scatter so I don't think anything is being misrepresented. Maybe scatter of the other lipid could have been better.. I'll take it on board for future analyses.
Yep, it's built in to the tool.
Were the adjusted p-values used in the paper? I see that there are FDR values in table S2 for the pathways, in which only vitamin B6 metabolism is below .05. But the text and figure 1A seem to be based on the raw p-values.
-Log10(p-value) is the standard choice for plotting things like enrichment of a bunch of pathways or a volcano plot, since FDR correction will leave you with a lot of redundant values. It's recommended to just use an additional visual indicator for which ones passed FDR or mention in the text/legend. I assume it wasn't mentioned here largely because it becomes a moot point anyways after explicitly stating that each finding was attributable only to one feature
Honestly, I'm a bit flabbergasted. After going to all that effort to highlight the problem with the use of a PCA in the Tate paper, one of my favourite ME/CFS scientists is aware of that and still commits even worse PCA abuse.... I don't understand why you would choose to do a PCA in this way.
Hopefully it's not too late to fix it in this paper? Yes, a scatter plot of one or more of the most differentiating lipids would be so much better.
That makes sense.
It looks as though there are 87 features that could have potentially contributed to a Steroid hormone biosynthesis and yet there is only one hit, only one feature that is suggestive of pathway dysregulation - and the pathway isn't significant after correction for multiple comparisons. It has an impact of zero.
I really don't want to be harsh and I agree a scatter plot would look similar. But the purposes of the two analyses are completely different and the axes labels make all the difference.
I'm not sure I agree, the context of PCA usage is really different here and in the Tate paper. For an epigenetic screen, the purpose of showing a PCA as the first step of your analysis is to determine whether your variable of interest represents a strong axis of variation in your data (i.e. one that emerges from an unbiased dimension reduction). So it is more misleading in that paper because what's being shown is at odds with the expected purpose of that figure.
I've done quite a few -omics analyses and I actually don't expect that the top two differential features would cleanly separate your groups of interest. Often what you find is that participants who are on the extreme end for feature #1 end up with middling values in features 2, 3 etc. So after you do univariate associations, it's common for analyses to do something like lasso regression to find the minimum number of features necessary to separate out your groups of interest (which I would expect to be somewhere in the range of 10-100 features from prior experience). Then you would take those top features, run a PCA, and do clustering on the PC values to confirm visually that your subset of top features separate your groups (PCA not only provides dimension reduction but also ends up reducing the effect of different ranges between features to allow for more representative clustering). When you have a small number of features you might just do those steps manually and see how many features it takes to get separation.
Yes, most plots like this will include that because you need a way to show where -log10(0.05) is. I agree that perhaps to someone briefly skimming the figures from the paper, this might have the unintentional effect of making it seem like those particular findings are stronger than they are. But like I mentioned above, the clear indication in the text, legend, and even superimposed onto the figure itself that each finding was driven by one metabolite only was more than enough to make me realize that this is essentially a presentation of null results.
Actually, I'm not really sure why the DG(36:2) feature was chosen for the PCA. The text says that it is the most elevated lipid. Looking at Table S5, DG(36:2) has a Q value (adjusted p) of 0.4 (not significant) and even its unadjusted p value is only 0.03. Fold change, which is a measure of difference relative to the control cell levels was 1.28. There are plenty of other lipids in that table with much bigger fold changes (e.g. 1.6). Perhaps there was a bigger difference in absolute levels? but that isn't quantified in the table. I think fold change is much better metric of relevance than a change in absolute levels; fold change is a scaled (comparable) measure.
I don't know if it's common practice or not - I'm pretty sure I've seen volcano plots where the significance line showed the cutoff of adjusted values (though I also found several that use .05 for the cutoff line just now, like you said).