Use of EEfRT in the NIH study: Deep phenotyping of PI-ME/CFS, 2024, Walitt et al

The term "effort preference" purely refers to pwME not choosing hard tasks as often (at least for the EEfRT task from what I can tell) and for the authors is independent of whether they can complete them or not. "Effort preference, the decision to avoid the harder task".

I have presented some plots that being less able to do hard tasks doesn't mean you're less likely to choose them as the game progresses (when looking at the entire groups as a whole rather than single participants and the game split into 2 halves, which given the variation among pwME are highly crucial caviats). This would indeed be in favour of their argument and doesn't support the "pacing/being sensible" argument.

However, these plots are dominated by people that play more rounds. I will later post the plots for a cleaned up version of this data (currently too exhausted) and then we'll see what's happening. I expect these plots to look quite different, but I don't want to jump to conclusions just yet.

The next step will then be to do something akin to a PCA to see why pwME choose hard tasks less often. If one shows that people independently of who they are choose hard tasks less often if they can't complete them and if we are able to show that this seperation implies or dominates the supposed separation the authors have found, then what we believe to be happening is actually happening. But until now we haven't done this analysis yet (as far as I can tell).

 

I named the category hard_task_completer, with 1 indicating success rate on hard tasks above 90% and 0 success rates on hard tasks below 90%. When you add it to the GEE, the effect of mecfs vs control group becomes non-significant while the effect of hard_task_completer is significant. I think this is pretty solid evidence that this is the more meaningful predictor of behavior.

Here are the results without hard_task_completer in the model.



Here are the results with hard_task_completer in the model.

The next question would be, is there a significant relationship between "percentage of hard tasks completed" and "percentage of hard tasks chosen", but unfortunately this isn't significant.

 

Awesome! That's already a very clinical argument and what I was looking forward too!

 

I might well be missing it but in windows there isn't any of that? SO this is what you see when eg conditional formatting and clicking through to choose colour for fill. Same thing when you click on fill.




Is there a way of saving colours because you get the usual (and it has been like this forever as far as I can remember) thing of a few scales and then your 'most recent' (about 10) at the bottom and can then click through to what isn't a wheel but is a similar idea just a different layout. You just when for example you go to 'fill' get a drop-down arrow you select on the right that takes you to the normal screen of pick from any of those colours or click more colours and then the screen above. I'm guessing that people have to just keep some sort of manual list of the hex numbers?

WHat I can't see is where I could eg save combos or specific colours so for example I was doing conditional formatting I could click on those 'custom' ones as easily as the ones that are inbuilt?

edited to thumbnail image

 

I haven't used Windows for 10 years, so I won't have seen the latest upgrade, but I worked it out on a Windows machine. I've never used conditional formatting, I simply click on the Fill Colour button. This brings up the palette:



I remember it looking slightly different on my work computer, but I still accessed via Fill Colour and as far as I can remember, the process was very similar. As I say, though, it's been 10 years, a lot of brain fog, and probably several major OS updates since then!


Edited to add: I need to go to my music session now, but if it helps the effort, I'll research how to do in Windows when I get back. I might be sh*te at maths, but am quite good at pretty colours.

 

Well this wasn't a validation study. It was a deep phenotyping study that made heavily emphasized assertions out of data that is only significant because a single tester played the game to effectively maximize the rewards. The authors made explicit claims about the fundamental mechanism of the disease out of it, featured them in the abstract as a wild hypothesis. As a completely independent minor study with a separate paper, sure, whatever. But this is absolutely not the case here. So that's a weird answer.

If anything, given the explicit requirement that it should not be an exhausting task, and the high rate of failure, as a validation study, the only valid conclusion is that it isn't a valid test, even more so with the lack of calibration and a design that allows an optimal strategy that can be excluded for being invalid. If the optimal strategy is invalid, then the whole test is.

 

Maths and stats are far from my strong point and I have probably lost the plot by now , but as it seems that no calibration per testee was done is there a way to tie the 1 day CPET and SF 35 results to the participants and these results which may hint at severity of disability impacting performance .

It seems that this may not have been considered at all ?
Or is this inconsequential?

 

You haven't lost the plot at all, and we'll definitely be looking into this. We just haven't gotten the SF36 data yet...

 

Thanks. The coffee must be working today

 

I agree, however I think if the point were more specifically worded there is something of an important note to be made that (perhaps because they were functionedly fixed into thinking ME-CFS is 'fatigue') ME-CFS can certainly mean physical disabilities will/should be present in a cohort that mean 'hard' as defined in this experiment isn't 'accessible'.

ANd the solution of course wouldn't be to just invalidate their data either by excluding them or by carrying on as they did and not thinking this completely affects how you can play the task. Because if you physically can't complete hard this changes your options significantly. If you 'might' be able to do some, but only say (by your own guess) maybe 8 if they were spread out a lot then you've again got a totally different factor in strategy but also in cognitive load on the choice-making.

One thing about this trial is that - whether we agree with the definitions and criteria or not, they did take a lot of time deciding on the cohort criteria and narrowed out a lot of people because of it on very specific reasons. So suggesting that just chucking out those who couldn't do it wouldn't be right - and also they are just the canaries in the coal mine certainly for all the ones in the middle of the ME-CFS, potentially all of them. SO it does need to be modified - and I don't see why other conditions like MS or Myesthenia Gravis etc wouldn't also be used in doing so for the calibration side of things to the physical task of hard being possible.

Of course I guess the point is that just like a condition could theoretically have physical and cognitive or psychological elements, an individual can also have more than one condition. So getting this calibration right is important - and should be doable - from both angles? Otherwise you'd have to eg when studying schizophrenia be excluding people with arthrititis - which may or may not be an issue depending on what cohort and issues you might be looking at (eg if you are looking at age-effects).

 

The IDs are different in the publicly available data and the mapMECFS data. So I don't think you will be able to do this, unless I'm missing something.

 

I think we've an underlying causal thing - the hard task being too difficult - which 'explains' behaviour in those with ME going into what looks like '3 groups'. But it isn't fatigue. That probably does layer in on top of it in additional ways. Fatiguability might be there as part of the handicap of people not being able to do more than 80 clicks on hard even at the start but that isn't the same as fatigue throughout the task?

If you look at the group who never had any chance of ever getting enough clicks to complete a hard (some are around 80 on the right hand side of the table) they've had to go into a game knowing this (from warm-up rounds) and come up with their own 'heuristic' that will necessarily be a pretty different decision-making task compared to HVs who have the game 'as intended'.

But we are no fools, and do remember what it was like not being disabled so I imagine many of us when given the instructions for that task can interpret what the experimenters are looking at/wanting to study and some of them might have thought the best they can do is 'decide as they would have' [if they weren't too disabled to actually complete the number of clicks] in order that they have that data. Seems a very reasonable assumption when you think you've signed up for that study so they can know more about how our minds and bodies work and they might have thought this is the best way

On the other hand you've a bunch of people (and who knows how far that extends into those on the left of the ME-CFS table) who completing the number of clicks for the hard tasks is 'near-miss' just a step too far, or 'maybe attainable if I were to pace myself and then really throw myself at it after a rest'. Well those people have a different choice in heuristics - do they do what the group above I've described have done, or do they play the game the best they can and try and maximise their abilities - adding another big element into the game and choice for them. AN element which overwhelms the subtleties of the other elements of the choice task.

Just because the two cancel themselves out. And just because it isn't 'fatigue' to me isn't an issue - because I think that the first problem is the lack of calibration to physical abilities. Not to fatigue. They could study fatigue if they like, but they'd need to have made sure that eg the hard isn't more than 85% of the maximum of what people were capable of doing on clicks (as another study did, and which I suspect is the situation on average for HVs) - then you might be able to actually study fatigue, like you've begun to spot with the HVs.

Or any other aspect. So it makes the data un-analysable? But one good thing comes from it, which is how important focusing on getting a format that is appropriate for the abilities of this cohort is/would be if you want to study anything in this cohort in future.

It could be a different 'game' or tool with a different issue: like a computer game that involved using a controller with button clicking and realising that potentially 50% of the group couldn't use one of the buttons (and it mightn't be a health reason it could have been a sticky controller), or a game that involved something more physical and a slippy floor meaning those who weren't wearing the right trainers were having to change their decisions based on not making any sharp turns. Neither would preclude there also being a fatigue issue on top of that.

And some might decide it was best in the circumstances (trying to get inside the mind of what the investigators would be looking at) to pretend the x button doesn't stick and play as if it worked normally, whereas (given it was a psych experiment after all) others might have 'worked around it'.

Treadway and OHmann have both tried to touch on the fatiguing issue and possible ways in HVs to avoid that influence (and of course that will be physical - whether you can do the buttons, but could also be cognitive - whether you want to just make your choice-making simpler by going 'only high probability')

WHich is another 'signal' layered on top of the choice stuff and what the test might be looking at too. But at least you'd expect to see it over time if you hadn't put people in rolling PEM before-hand of course - and if you had then the components of the task would also matter - because ME isn't as straight forward as people assume (we get wired-tired and sometimes it affects our pace sometimes it affects our accuracy whilst our pace stays high).

But my point is that these are still pure hypotheses because the floor/ceiling effect means that completing hard was inaccessible or 'above taxing' for most of one group. WHIch drowns out all of that data.

It really is like being thrown into a dance class with some sort of handicap and working out whether you do your best to get the steps right that you can and 'look gawky and don't finish' or whether you do your best to not stand out and look like you aren't struggling by gesturing towards what people are doing rather than trying the actual moves (and coming a cropper). You can't really assess those people on anything other than that 'situationally' - whatever the dance routine was intended to 'assess'?

 

Well, wow. Thank you for running that.
And, what is a GEE?

I think we can work with that.

 

It might seem so, but if effort preference is of any relevance to explaining ME it has to be something that occurs inappropriately in ME. Otherwise it is simply normal rational behaviour.

Let us suppose that all the healthy volunteers were put on an island where they all succumbed to an illness that makes you feel ill for months or years, as ME does. If they then ran these tasks then very likely they would show the same 'effort preference'.

I agree that effort preference is presented simply as a choice behaviour. But the healthy volunteers were not in a situation that we would expect to lead to this choice behaviour. So there is no reason to think it is a feature of ME rather than being ill.

If it is just a response to being ill then it tells us nothing. If, on the other hand it is some abnormal cognitive pattern that makes you avoid choosing tasks that you could perform - which is what HVF is accused of then you cannot discount HVF and not all the people in the ME group because they might be behaving in a similar way. You don't know that they are but you are assuming that they are if they are not just behaving rationally as His would having caught some other illness.

 

and by the way, I thought I had better clarify that I keep using the term 'handicap' in the context of golf and its usage - rather than being tactless about disability in general (although I think in this context it is a useful term to note that we can have a whole illness with lots of things going wrong but when faced with eg a certain test there will be certain 'sore spots' that it is pushing on)

I haven't yet through my brain around it but of course golf as a game tackles this issue head-on ironically a bit with the elements it combines, and by this type of handicap system (as well as of course equipment such as buggies and other adjustments)

 

Here are the plots I had promised earlier:

I have now done the plots for the first half of the game and the second half of the game for the first 34 rounds (the lowest number of completed trials is 35 by HV H, to get equally large halves I have taken the first 34 rounds).

(I will make the plots look nicer and get around giving the exact numbers as well once I have the energy to do so).






and the same plots excluding HV F



This plots are more similar to the data in Figure 3A presented in the paper.

The data fundamentally differs from the data I had previously presented (because HV V plays 35 games, whilst ME/CFS I plays 55 and the distribution of choices participants make is not uniformly distributed across time) and there’s multiple interesting things happening here and once again shows how non-robust some of the methods are.

It’s not really possible to say which approach is the better approach (taking half of the game for each person vs making sure everyone plays equally many games). The one approach accounts better for the range of different strategies, whilst the other one supposedly (according to other EEfRT studies) accounts better for fatigue and makes sure that people that play far more rounds don’t dominate the behaviour of the whole group. I think based on the already existing literature it makes more sense to stick with the second approach, i.e. take the first 34 rounds and cut them in half (it might even make sense to have a smaller cut-off, for example at 30 rounds and see if this behaviour is different, 30 is a number I’ve seen in some other studies that try to discount fatigue effects).

 

If 'physical disability' means 'you can't actually do it' - which Nath has specifically denied is the case I agree. But it is still legitimate to study motivation and effort in that context. What is not legitimate is to infer the wrong causal path - that an abnormal effort preference is involved in the not being able to do it - which is what they seem to be claiming.

My main point is that any suggestion that researchers are not allowed to study these things because ME is physical is the biggest booby trap for yourself in the book.

 

This seems a good way of putting it. You set up an artificial situation in which it is far from clear what strategy people are expected to take whether they are ill or well. No amount of clever measurements are going to resolve this as far as I can see.

The authors have made use of transcranial magnetic stimulation (TMS) and fMRI BOLD images to try to prove their point but I am not convinced by either. The TMS to the primary motor cortex showed that PWME could move their muscles OK. But that is expected. Whatever central signalling might stop them is likely to kick in before motor cortex, which is the last place before messages leave the brain. The BOLD studies suggest some differences in thinking about doing things but in the dance class scenario there is no way that anyone can work out what the signals mean because we know so little about the detail of how neurons signal to each other.

 

I'm conking out atm (not due to this, everything 'in general') a bit, so if you do have a way of looking at it then I will return to it and make those changes when I'm at a good moment if that's OK? Hopefully that won't be too long before I can return to fiddling with it, just need a change of task atm

In the absence of having another way of doing it in the microsoft excel I'm thinking I'm going to need a sheet with the Hex references on once I've got some good colourways so its worth me getting them right. And am open if you have any that you use regularly and have a 'hex' reference for it to you sending these across ?

 

and @Hutan when you look at the individual level strategies you can see that some of the people in the middle-capability zone of ME-CFS seem to be trying to 'actively manage' their capability-issue/limitations by spreading out their choice of hards OR it could just be luck of the draw that is how the 'high probability' (if anyone is selecting less than 33% hard we have to assume they are prioritising these are they are the ones that almost certainly do count) are spreading at different points in the game.

There are definitely a few ME-CFS where you see the pattern of 2 sequential hard tasks with a 'complete' then 'fail' vs then waiting eg 4 tasks and managing a 'complete' for a hard.

The cognitive load of the task means that these will all be approx heuristics people are using - where eg you either do 'all the high probability' (seeing a 33% hard choice) or 'all the high and medium probability' (where you see 66.6% hard choice) if you aren't bringing in other factors to bear. If your capability means that you can't successfully complete the numbers of hard tasks involved with 33% then we can assume that is a physical capability issue where there are non-completions. And later tasks will be impacted by the fact you either 'avoid hard tasks' or 'fail whatever you fail' (but pick as if you don't have an issue physically with completing them) or 'add the cognitive load of that additional choice factor'.

I think what @Hutan is trying to emphasise is that the way the game works, and then when you add in a finger-tapping issue, if you can see the end in sight and have a moment where you feel your little finger is 'having a good moment' you might revert to that approx aligned with probability BUT in essence the how many you can choose has been predetermined by something that isn't actually supposed to be a factor in the test. It is more than people are having to 'save up their strength' than 'fatigue' where they are in the 'near-miss' category (middle-zone on table, seem to be able to 'just about' manage 98 sometimes but generally miss by a few - so not in that zone which seems to be defined by the Treadway paper of at the start the hard task being 85% of max capability, more like 95-105% of max capability meaning they are working out how they can pull it out of the bag to just add the odd complete one to their basket).

It's a completely different game for all 3 groups: those who find the hard is well within their capability might eventually get fatigued and their 85% of capability means that they start failing some, those who were at it being 95-105% of their capability are being given a completely different task and for those that the 98 clicks was always 125% of their capability they must have been thinking all sorts (like is it a test of how they cope with adversity or something).

Of course it probably isn't even as clean as that because those who 'look good' in the ME group might be somewhere between the two (on a scale) and then the fatigue side of things doesn't always equate directly with finger-tapping ie severity in ME is a mixed bag where some might have more cognitive severtiy and less physical and vice-versa (but then at a more specific level with things affecting this task). No reason why those who managed earlier on, didn't feel the effects of having done those later in a way that those who had to pace from the start didn't etc.

And for the third group - if we didn't have the complications of the real task (as it was for HVs) being thrown in - well given they'd never have hit 98 then we'd be looking at click numbers and not on-off to see if their performance deteriorated. But you can't because of the choice factors. That whole group is sat in a floor/ceiling effect if you are just using complete/non-complete. So they must have been wondering whether the investigators were going to be looking at whether they 'still tried' and how many clicks they did, given what else would you think if you are given an unachieveable number from the start?