Use of EEfRT in the NIH study: Deep phenotyping of PI-ME/CFS, 2024, Walitt et al

The PIs write in the main paper that "The primary measure of the EEfRT task is Proportion of Hard Task Choices (effort preference)."

Do they give a justification for why this is the "primary measure"? Is there a validated precedent in the literature for using this particular ratio?

It's just that it strikes me that the button-pressing behaviour of ME patients is an awfully arbitrary thing to hang the conclusions of such an important trial on. I mean, of all the things they measured, why this? Is it meaningful in a way that I'm missing or did they engage in some exploratory "data dredging" and then work backwards once they found a number that was statistically significant?

 

I think that this is a really important point. This is a complete misrepresentation of EEfRT.

*Editing because it's more nuanced than my original response*
Technically the primary measure of the EEfRT task is Proportion of Hard Task Choices (PHTC). However, Wallit is using a slight of hand here, because he is implying that EEfRT was designed to measure OVERALL PHTC, which is how he defines effort preference. In reality, Treadway designed EEfRT to measure changes in PHTC under a particular set of manipulations and he never refers to the task as a measure of effort preference. In the absence of the manipulations, PHTC is meaningless to Treadway.

The accurate description would be The primary measure of the EEfRT task is Proportion of Hard Task Choices when reward value is high and probability of reward is uncertain. The creator of the task did not use it to measure "effort preference" and their primary hypothesis is not about overall Proportion of Hard Task Choices, it's way that this ratio changes given particular manipulations that Wallit ignores.

Yes, this. They tested all of the interactions that the task was designed to tease out (which have actual theoretical justifications), but they were all non-significant, so they invented a new hypothesis post-hoc based around the one significant result they dredged up.

 

They could only say that, that's exactly what you expect. Because if you have 2 means that are pretty close to each other it's very probable that roughly half of your healthy group sits below that mean and half of your ME/CFS group sits above or around that higher mean, especially when the higher mean in the healthy group is driven by more outliers (note: the healthy group has a higher variance from their mean of hard choices made, then the ME/CFS group has).

It makes little difference to the difference between the means of both groups in terms of hard tasks chosen. Ideally you have a result that is consistent with both approaches, that applies to whatever our results might be as well. The largest difference is if you do a division into half-times because then differences are driven by people who play more. If we want to write a critique of the paper, the most sensible thing for me would be to do what appears to be the most consistent choice in the literature.

 

So mysterious while it is probably just fatigue/illness that caused the difference.

If I understand correctly, Walitt and colleagues discount fatigue as an explanation because the choice for a hard task declined at the same rate in the patient and control group (figure 3A). But that means they only looked at fatigue caused by their 15 min clicking test, not by fatigue caused by a chronic illness that was already there at the start of the clicking test.

And it is at the start of the clicking test that they tested for a difference.

 

Has anyone found the R code that they used to analyse the results of the EEfRT? The paper says that the code is available here but I couldn't find the part where they analyse EEfRT data.
https://github.com/docwalitt/Nation...itis-Chronic-Fatigue-Syndrome-Code-Repository

 

I looked and couldn't find any code for the EEfRT analysis. It would be great if someone could find it.

It does say under "Statistical analysis of effort expenditure for rewards task" that "All GEE models were implemented in SAS 9.4" so maybe there isn't any R code for this section?

 

A good point. I think there may be a confusion between 'fatigue' in. the sense of fatiguing during a task series and the subjective symptom loosely known as fatigue. They are totally different things.

 

They can only discount in-game fatigue as the driving factor. They'll struggle discounting general fatigue because if you translate the games of HV forward in time by somewhere around 20 games, you'll see very similar chances of choosing hard tasks for HVs that ME/CFS had when the game started (to be fair that is exactly what you expect, because it's one of the more consistent findings in EEfRT studies) and it's very hard for them to rule out that the hardly significant difference they find isn't driven by some general levels of fatigue that were initially present in the one group but not the other.

One could suspect they might not want to talk too much about fatigue as their results are inconsistent with the EEfRT fatigue literature where a similar study proposed people with cancer fatigue choose to exert higher effort more often which in turn leads to chronic fatigue (I don't think that study is better, I just think most EEfRT results are inconsistent and replication isn't the norm).

 

As far as I can tell, despite claiming to have made all analysis code publicly available, they did not share any of the EEfRT analysis code. I've rerun the GEEs in R but I had to write it from scratch so it's definitely possible I did things differently than they did.

I'm also having a hard time even getting the R code that they did share to run. Have you been able to?

 

Hmmm ... I wonder why ... ( Sarcasm)

 

Can somebody help me out please because i have mentioned this a couple of times, as i think have others, i am very blurry mentally & cant seem to get my head around why it doesnt seem to getting any traction...

Theres a glaring difference between the controls and the PwME... they are comparing tasks with a reward, and saying the groups are equal & could do the tasks as well as each other, rewards the same etc etc blah blah....

but from my POV they are missing the main point which is that for the PI-ME/CFS group there is a punishment that comes later, completely wiping out the notion of any reward and being a ready explanation for the different choices.

If i'm the subject doing those tasks, i would actively, behaviourally, consciously choose
not to do the harder task.
Isn't it Skinner 101 - the brain gets trained to avoid or not want to do things that it 'knows' there will be punishment as a result of.
So the punishment in terms of increased symptoms that will surely come after the task is finished (in the form of PEM), obliterates any notion of 'matched controls' because the healthy controls will receive no PEM & therefore no 'punishment'. Its not a true comparison.

Why does this not matter? Why is it not the main thing we are discussing? How does this not make the whole thing moot? What am i missing?

surely the most junior psych researcher would know you cant do an experiment of choices with 'the same reward, & the same ability etc', when its known that 1 set of subjects are all going to be punched in the guts the next day if they make certain choices, and the other group is not.
I just cant get my head around why this doesnt just invalidate the results on it's own. Am i being thick?

Edited to add: !!! this was in NO way intended to take anything away from the amazing & essential work everyone has been doing on this thread, that is hugely beyond me. I only meant that i cant stop thinking about this one point & realising that it cant be that important since all of you much more clever people are not focussing on it & so I must be getting confused/not understanding something.

 

I think that this is as important and valid a critique as any of the methodological flaws we've found. I think this alone should be a letter to the editor, the task should never have been used in this way because it cannot isolate preference (this is also why the question is a useless question). For me, I just doubt that the authors are going to engage in a discussion of theory with any intellectual integrity when they've already demonstrated that they have no problems with such glaring contradictions, so my hope is that a methodological critique will be harder to ignore.

 

This does matter a lot. I consider it part of my #1 reason this test fails but you've made me realise it could be expressed more powerfully and directly.

 

I think this is likely. Another possibility for some patients - and another reason the test might not be valid - is some might be aware of the history and when they see this easy/hard game come along, choose hard as much as they can with one eye on how the data might be intepreted !!

 

I think they might even have muddled three different things: fatigue (subjective symptom), fatiguability (early failure), and incapacity (activity threshold lower than normal at the outset).

That would be a good question to put to them.

"How does your algorithm (that nobody can find) account for all the motivations* pwME have to game the game, which are unlikely to be present in HCs?"

* Insert list of the bleeding obvious.

 

Big picture:

This was a challenging paper to write - difficult topic and difficult circumstances. It needed an exceptional scientist to guide it. Nath was not that scientist. He let down the mission of science by permitting this to be published in its current form.

As the sample size receded they needed to rein in ambitions and make sure what got into the abstract was on an exceptionally strong footing. Instead they went grasping for marginal findings from experimental measures.

I can imagine the argument: Wallitt saying : this is basically our only significant finding between the two groups, it has to go in the abstract! And Nath saying ugh, fine because they spent a lot of money and kinda needed to find something.