Use of EEfRT in the NIH study: Deep phenotyping of PI-ME/CFS, 2024, Walitt et al

I was thinking about this yesterday. Looking at the entire sample of healthy volunteers and patients, people with success rates on hard tasks of above 90% chose hard tasks 44% of the time. People with success rates on hard tasks below 90% chose hard tasks 33% of the time. There is no difference when you go lower, ie there is no clear dose-response relationship where people getting dismal completion rates choose hard tasks even less, though this might emerge in a larger sample. And it's perfectly reasonable to suggest that just failing a bit is enough to affect your choices. Could those with more stats skills than I have see whether this alone could account for the group differences in hard vs easy task choice?

6/15 pts have success rates on hard tasks above 90%.
9/15 pts have success rates on hard tasks below 90%.

14/16 HVs have success rates on hard tasks above 90%.
2/16 HVs have success rates on hard tasks below 90%.

 

A counterargument to that would be that pwME choose hard more often in the second half of the game (I haven't looked at whether this difference is statistically significant). If there was an a priori choice of conserving energy for the study one wouldn't expect to see this.

Edit: My data on this is seems to be somewhat in conflict with what the authors presented in Figure 3a, but this plot would show even stronger that it wasn't an a priori choice that remained constant.

 

Agree. As Simon M said, they're "a bunch of triers":

 

I agree that the difference isn't big, but it exists and it's the basis for their whole argument (at least if I'm currently not misunderstanding things) and as long as it's statistically significant people will be happy with this argument even if it means nothing. So that alone doesn't convince me.

I believe what you're proposing is something similar to what I proposed here

i.e. showing that the data isn't robust and that other things offer a better seperation than their proposed model. I think this is very much in spirit of what @Evergreen proposes we should look at

I think this should be looked at. I will look at it once I get there. I don't think it's a direct counterargument to their argument rather than an alternative and possibly better interpretation of the data (if the argument indeed holds up to an analysis), which would just ultimately show how unrobust this method is.

 

Agree, we should definitely analyse the data under this aspect. I will do this once I get there. It's probably less of a direct counterargument rather than just shows that their argument meant little to begin with if other things offer a much better classification ( and one could also throw some random forest classifier in there because reviewers and the science community unfortunately loves that stuff).

However, I won't make conclusions yet. If this site has thought me anything we should first do the analysis/look at the data and then decide on what is plausible. But this seems a very sensible route that should be looked at next.

 

I would also like to mention that whilst the number of people opting for a hard trial increases in my plots from the first half to the second half (both for pwME and HV), this is in conflict with what the authors presented in Figure 3A. So I will have to look at my data again to see if I have made any mistakes. It would be nice if someone would have a look and see if they can spot an obvious mistake (better to have 2 eyes looking over this) or whether this has something to do with their fitting/smoothing/Fisher or even whether "half-time" meaning something different for people that play a different amount of games (since I'll also be using cut-offs at a certain amount of games for my next plots, we will quickly see whether this plays any role) could play a role in this.

 

You can. I work on a Mac, so my screenshots and some options might look slightly different to the way they do on Windows, but there's rarely a big difference in the actual approach.

Click on the fill colour picker and choose More Colours. A palette like this pops up:



Choose the middle one with the little squares. There'll be a bar showing the set of colours that you've currently selected (mine says Unnamed in the screenshot, but it sometimes shows Apple as the default). Next to it are three little dots – if you click that, you'll get the option to start a new colour set:



You'll then get a new set called Unnamed, and it usually adds whatever colour you last used as the first colour. You can add or delete colours easily by clicking the plus/minus box next to the colour name.



To name your new set, click on the three little dots at the side of the Unnamed set – one of the options is Rename. I've now renamed mine Graph Thingie and the first colour is Graph Beige (you can edit the colour name by double-clicking it).



To pick a new colour, go to the colour wheel option at the top of the pop-up. If you want dusty colours, place the selector closer to the centre of the wheel. If it's not dark enough, add more black by moving slider underneath.




Things like this always look really clunky when written out as a set of instructions, but it's quite intuitive once you've worked out that you can actually add new sets. It's odd that it's not highlighted anywhere—I only discovered it because I decided about 20 years ago that it must be possible to do it, and I was bloody well going to work out how.

 

To successfully complete this mission, any response will need to address their argument. It’s not that they didn’t notice what we’re noticing – that pwME successfully completed hard tasks less than HVs – they noticed it:

I’m not sure the bit after “but” makes sense – I think they mean that there was no between-group difference in the decline in button-press rate over time. In the blurb under fig 3, they write

So I think their argument is that pwME were less likely to be successful at completing hard tasks, but they did not fatigue more than HVs during the task, so it wasn’t fatigue that stopped them completing the hard tasks successfully, so we can ignore that wildly significant result. [Mind the gap.]

Regarding easy tasks they say in the blurb under fig 3:

This suggests that if they saw both a reduction in performance and a decline in button press rate over time, they would have interpreted that as fatigue.

What their argument does not allow for is pwME having less capacity for repetitive fast pinky presses from the beginning of the task. (And they really do, right from the practice rounds as @EndME ’s graphs illustrate so nicely.) That is what, we would argue, the lower rate of hard task completion shows. It doesn’t really matter what they would attribute this to (likely deconditioning). What matters is whether it affected the proportion of hard and easy tasks they chose.

That’s why I think it’s worth looking at the above 90% vs below 90% thing I suggested above, or @hutan’s similar idea.

 

Worth noting how "Matters Arising" works in Nature Communications.

You write your piece, then you submit it to the authors of the paper. You resolve what you can with them, and then, if you still think it necessary, you submit it to Nature Comms as "Matters Arising".

So it gets sent to the authors twice, once by you, informally, once by Nature Comms, formally.

Then it's either accepted or not. If it's not accepted, you write it as a comment on the website.

https://www.nature.com/ncomms/submit/matters-arising

Edit: added a sentence to clarify.

 

EndME quoting Walitt: "He mentions that this participant clearly had no problem completing certain tasks but rather chose not to (which we can all agree with), which to them indicates that this is either an invalid performance or an invalid task administration."

But the conclusion of the study seems to be that PWME had no reason to have a problem completing tasks but rather chose not to. You cannot invalidate in controls what you are calling out as causing the disease in patients.

 

I wanted to note that I'm not quoting him rather than paraphrasing him or writing down my interpretation of what he wrote to me.

The situation is a bit more complex I would argue. Person HV F has no problem completing trials but chooses not too, so that's the reason for excluding him. pwME have problems completing trials and don't choose to not try to complete them, so one doesn't have to exclude them. The conclusion of the study seems to be that pwME choose not to complete hard tasks, but that they choose not to choose hard tasks, that is a subtle but important difference.

What is not seen in their analysis is whether being able completing tasks creates a better seperation between HV and pwME, rather than just choosing to do hard tasks. That's something we still are working on.

 

A stupid question from me: if buttoning pressing does not decline more in ME cases but failure occurs more often is this due to pressing 'wrong' buttons?

I think the argument is definitely intended to be that no difference in decline of button pressing rate mean that the patients are not even 'centrally' fatigued so just opting to fail. But you don't opt to press the wrong buttons.

 

From what I've understood there's only one button. Failure is due to not pressing this one button often enough.

As far as I can tell the part about the button pressing rate is about in game fatigue not dominating the behaviour between the groups and if we look at success in trials as time progress this seems to be the case (but more analysis of the data still has to be done).

 

I have looked at the data and it appears to be correct.

I believe the difference between my plots and Figure3A is that the "half-time" plots are dominated by people that play more games. I will post plots of the first 17 (first half) and the next 17 games (second half) later today (i.e. everyone plays the same amount of games in each half).

 

But doesn't this come round in a circle a bit. Either they choose not to choose hard tasks because they fail them, which is a sign of a healthy strategy and so cannot be used to argue that they have ME symptoms because of 'effort preference'. Or in fact they did choose not to choose tasks they could have completed - effectively play-acting. It would be a different play-acting from the Las Vegas strategy of V, I agree, but if play-acting is considered a possibility then the whole experiment falls down for the same reason that open label trials with subjective end points do. The patients are not blinded to being a patient and they can bias their behaviour however they like. Trish would have brought in her bloody minded streak and refused to do the test but most people would 'help' the researchers by doing the test the way they thought the were expected to.

Or to put it simply if the confusion is that patients are conning themselves then it is foolish not to expect them to cn the researchers just as much. If at least one HV is cleverer than the people who designed the test then why not some of the patients?

 

It's not circular as far as I can tell. Their argument is: pwME choose hard tasks less often which is their "effort preference". Not being able to complete them or pacing is not part of this argument (as far as I can tell).

The pacing or limiting exertion as the game progresses refers to easy tasks, but not the hard ones (there was no pacing for the hard rounds according to their presentation).

If the authors would also be able to show that choosing not do hard tasks is somewhat independent of not being able to complete them, then they have a solid enough argument. But they didn't show this, at least not in the paper, but we also haven't disproven it yet. I currently believe we might be on the way to doing exactly that but we still have to analyse the data a bit more to really be able to show that and before we're able to make conclusions.

The problem of patients and controls not being blinded and everything being subject and the methodology not being robust and the sample size being miniscule is a whole different issue. But I don't think we even have to dive into this if we can do something like a PCA and actually show that their results don't show what believe they show and if we can show that the seperation simply occurs via people not being able to complete tasks rather than anything else.

 

But a I understand it being able to do the tasks and choosing not to choose them would be an abnormal 'effort preference' but not being able to do the tasks and choosing not to choose them would just be being sensible - or indeed pacing. So for the conclusion of abnormal effort preference to be valid we have to assume that the PWME could do the tasks as well as HVF.