Use of EEfRT in the NIH study: Deep phenotyping of PI-ME/CFS, 2024, Walitt et al

The IDs in the EEfRT data (eg HV A, PI-ME/CFS B) do not appear in the mapMECFS data at all. The IDs in the publicly available data have been made more user-friendly. The IDs in the mapMECFS data are different. I don't think it would be safe to assume that the order in which they appear in the publicly available data is the order in which they appear in the mapMECFS data. I've emailed Dr Walitt to see if he can provide a key so we'll see. Might be possible, might not.

 

I have received a great response from Ohmann. Since I have not asked for his permission for sharing this on a public forum (I simply asked for his opinion about a few questions I had via email) I don’t think it's fair for me to share this in public.

I think it’s fair enough for me to tell you my interpretation of the gist of his response, given that he’s an expert in the field and noting that this is my interpretation of an email that wasn’t intended to be shared publicly. As far as he can tell from afar the results appear to be valid even without calibration, but he notes that calibration would have been interesting. He agrees that the sample size is very small and random perturbations can always influence the results at these study sizes, but he also notes that this is unfortunately often the case when recruitment is difficult.

 

Others with access to the data will be able to graph things for you better than I ever could! I think the standard deviations and numbers in ranges 0-29, 30-59 and 60-100 give some of the info you would like. Here they are again:

 

I am not sure anyone said HC F had the best strategy, but it is clearly more effective. There are some ways to play that are just better if your goal is to win more money. For example, it is always better to fail easy tasks so long as you can complete and win two hard tasks. What you are saying is that people should not be allowed to play the game in the way that best achieves the reward (at least for healthy controls). If they only want to study X, don't introduce these other variables that are in conflict with X.

What specifically differentiates the strategies that you believe to be eligible to be examined from those that are not?

 

Am so grateful for all the work you've done to make these more accessible! I'm baffled by most things so far today (the odd thing gets in clearly). Not a hope of me finding hex references. I think you've done more than enough already and could save your energy for analysis. Thank you so much.

I agree that a more nuanced look at clicks and other measures could reveal something.

 

Just adding on, I think the authors reasoning for excluding HC F is that his split between easy and hard tasks don't match his true "effort preference". However, I think this is inappropriate because I don't think that the choice between easy and hard tasks correlates with a nonsense term like "effort preference". They say: in the case of HC F it doesn't match, but in all the others it does because we have no reason to think it doesn't (and even worse: it matches our preconceived notions about what we should see in these ME/CFS folks that clearly could be doing more but don't).

 

❤️ that is interesting and straight.

 

I don't think the argument would have to be this involved. I believe they are most likely excluding him because he deliberately fails to do certain tasks, but had previously shown that he can easily do them if he wants to (so they don't have to get into a complicated argument of choices but can focus on simpler argument of completion rates). I wouldn't be suprised if their argument will be backed by a 2SD from the mean type analysis, which it certainly is. That is also the feeling I got from reading Wallit's response to my email.

It's probably a bit harder to exclude someone on the basis of the ratio between hard and easy tasks, because that's more of an intrinsic property of all strategies and their whole argument revolves around these ratios so they might want to stay away from that.

 

One way to think about it is that it's almost as though HVF was playing by a different set of rules.

There's an implicit rule, I think, that would go something like "if you can complete a task you should complete a task".

There's no way to tell for sure that players are obeying this rule but I suspect that the validity of the exercise depends on participants playing within "the spirit of the game".

 

Agree wholeheartedly with these points.

Most days I can only skim so I will miss plenty of great content, unfortunately! Yes, the Ohmann paper has helpful bits like:

The Reddy paper - the first I've been able to look at - is nice in that they have laid out exactly how the EEfRT could have been modified to make it (potentially) valid in the NIH study.

Agree. I think the pwME with 100% hard task completion could still be separated from the HVs with 100% hard task completion on a task modified to challenge them further.

It clangs, doesn't it.

I think that's why the messaging from the authors in interviews has been a bit kerfubbled - too many things are being conflated into supposed effort preference. It's not voluntary but it is subjective - what? And patients' difficulty with repetitive grip testing and repetitive button pressing are hand-waved away. I don't understand that part.

 

Good news, Dr Walitt wrote back and directed me to a file for the EEfRT task in the mapMECFS datasets. It's under "Neurophysiology Data Files" amongst heart rate, tilt test, lumbar puncture etc and I had missed it. So you will have all the data you need. I'll look at it when I can too.

 

A preliminary scatter of hard task completion x physical function. Mistakes are possible so this should be double-checked by others with access to the mapMECFS data.

NB The number of dots does not equal the number of participants in the EEfRT task because repeats are displayed as one dot. For example, there are 7 participants with SF36PF of 100 who completed 100% of their hard tasks, but only one dot is displayed for those 7 participants. (If someone knows how to get Excel to reveal this detail, talk me through it and I'll post another graph later.)

 

That's because of the design of 2 random rewards. If they had made it cumulative, if they had made choosing hard tasks more rewarding, HV F would have chosen 100% hard tasks, and likely more would have, including patients. But they didn't, they designed it in a way that made this the optimal strategy, failing some tests to keep them out of the pool of possible rewards.

That's a design failure, that's on them. HV F played the game as designed, but they exclude the results precisely because he played the game as they designed it, rather than the way they wanted it to be played, which is to ignore their own design.

Anyway the whole test is absurdly invalid even before we get to that, but their reasons for excluding those results, which conveniently gives them the tiniest statistical significance they need to ague their absurd notion of effort preference, are complete bunk.

 

Thanks! Nothing pops out of this data immediately, at least when I look at it. Maybe plotting ME/CFS in red and HV in blue would be nicer visually, but the way this data appears here, makes it seem like it won't give us immediate insights.

One could probably also look at SF36 vs hard tasks chosen (with the cut-off at 35 rounds), but I don't expect much here either because ME/CFS H, who "destroys" the correlation between hard tasks chosen and not beeing able to do hard tasks, appears to have a high SF36 according to this plot, whilst the other "destroyer" ME/CFS D appears to have a low SF36.

 

I am surprised how much people scoring so low on SF36 seem able to do. Does this reflect issues with SF36 scale? I did score myself several years ago on the SF36 scale and though I can’t remember the exact score I was in the severe ME range. However I could not imaging being able go to participate at any level in such a research project.

Obviously there will be some correlation with the SF36 score and task completion, but why these outliers?

Does this argue the need to calibrate the task for each individual, or unfortunately seemingly support the nonsense idea of effort preference being the limiting factor rather than physical ability?

 

This is another potential area of confusion. SF36 is a general scale; it's not specific to upper body vs lower body function, but this can matter in ME.

It's also likely to include orthostatic intolerance as a factor that causes some of the participants' disability. Yet for people with enough capacity to take part in this study, orthostatic intolerance is unlikely to feature in a task that involved pushing buttons whilst sitting in a chair.

So you could have participants with high levels levels of OI-related disability—and therefore high SF36 scores—who are better at pressing buttons than a neighbour with a low SF36 score, because they happen to possess naturally good manual dexterity.

 

From what I can tell is really doesn't support anything, which may have to do with SF36 not being a good measure or may have to do with something completely different.

I don't see the SF36 supporting calibration, looking at ME/CFS H alone would be a strong indication of this alongside those people with a low SF36 who have no problem completing all of their tasks.

Maybe one shouldn't expect the SF36 to show much here in any case, after all it isn't supposed to measure "finger fitness" as part of a clinical trial. But it is worth the try to at least consider it.

 

I’ve had a closer look and I’m somewhat less convinced now. The problem here is that 90% itself is a rather arbitrary and for which I see little scientific justification.

I assume below mean "=<" and above means ">". In that case it simply means you’re cutting out only HV A & HV B on the healthy participants side, who added up together sit below the HV and ME/CFS mean rate of choosing hard, simply because it’s easy for HV’s to do hard. The fact that you manage to just hit HV B with this choice purely has to do with luck more than anything else. If you instead choose the value 90.00001% you only cut out HV A (HV B sits at precisely 90%).

On the ME/CFS side you obviously cut-out “high performers” (in the sense of selecting hard) like D, H and O but you also cut-out all the low performers. Or more precisely at above 90% you’re only left ME/CFS patients C,E,F, K who all sit quite above the ME/CFS mean as well as ME/CFS patients J and M who only sits marginally below the mean, .i.e. you're left with those that seem no different to HV's in the EEfRT.

As you mentioned there might be a dose-response relationship at a larger sample size, but at the current sample size I don’t find the argument too convincing (and the dose response relationship is partially driven by the high variation amongst ME/CFS patients, so even at a larger sample size things might be more complex). It’s of course a valid argument but it is rather weak to me from what I can currently see. Certainly something we might end up mentioning but I don’t currently see how it is a particularly strong argument (on the HV side you just manage to cut out one low performer, while the ME/CFS side suffers from such a massive variation that cut-offs aren’t a really stable operation and if you'd have an additional high performer your argument likely fails), maybe it's at least equally weak to their argument.

What do you think @Evergreen?

I assume the fact that there is not a relationship will be driven by the ME/CFS group of people that try the most but fail the most, i.e. the two participants ME/CFS H & ME/CFS D?