Use of EEfRT in the NIH study: Deep phenotyping of PI-ME/CFS, 2024, Walitt et al

That 33% if pretty key when you are thinking about those with disability.

I did another table since yesterday out of curiosity on this (and also looking at button press and choice time at individual level as was thinking similarly to @Karen Kirke ) because if you are struggling to complete as per the test is supposed to be designed then you have two choices: do less to complete more of them, or fail more.

I think we can assume anyone doing less than 33% it's probably about not being able to rather than 'choice' in the inferred sense. And if you know you can't do (if we assume eg 40 tasks per person 40/3 = 13) 13 hard tasks, except you don't know how many will be coming up - certainly not at the start until you might get more of a sense, then you have to be more choosy at that point of when you do select hard. And that choice might include whether your finger is 'well rested' as well as probability and reward magnitude.

So I was just nosing at the data I plonked in a table and ordered by % hard completed to see if there were correlations with increased time for hard choices where people were 'doing less to complete a higher %'

There is a group a few up from the bottom in the ME-CFS where they generally had less than 33% hard selected and in general the ave hard choice time is a bit higher indicating they've a bit more to think about, and you can see that - in the column number of hard - relative to the rows above and below them they have selected fewer hards. For some of them this has translated to them completing more, for others they still only complete 3 of them anyway.

You can also see on the two near the bottom in the HVs: HVN and HVB that they had a lower % hard selected and when you look at their ave choice time for 'hard' vs ave choice time overall there is a bigger difference between these two times for those two than the rest of the HVs.

It's an imperfect pattern though - showing how many interacting factors are probably at play due to all of these complications. ME-CFS I has very quick decision times and only selected 23% hard. I'm guessing that ME-CFS A has relatively swift choice times compared to the rest of the 'group' around them on this table is down to their % hard at 31% being an indicator that they just picked the heuristic of going with 'only high probability' (which would mean 33% hard), thereby reducing the cognitive load and as far as their physical capability just going with the flow and letting the chips fall where they fell?

My point is that the issues, I think, do affect how the game is being played - but not unusually would do so in a way that isn't consistent across everyone.

So the capability problem means the data can't really be analysed because it has forced people into completely different 'games'. And because the instructions and set-up necessarily aren't set-up to accommodate that, you are going to have different strategies across individuals to cope with those issues (and because they are having to second-guess what the real point might have been of being faced with a task they couldn't do and whether it was to plough through because they were looking for resilience in the face of an impossible task)/

EDIT: table edited for colour @Evergreen let me know if this works. will keep the hex numbers if so.

 

Yes, this invalidation of HV F seems to me a key point. Hopefully I've summarised correctly below without erroneous assumptions.

HV F is said to be a 21yo healthy male. He deploys an optimal (edit: near-optimal) strategy per the rules of the challenge. Motivation? Nothing more needed than "I like to win". (Possibly he had developed greater insight into the challenge than the researchers that were setting the rules. Similar sorts of theoretical tests are very common when interviewing for programming jobs in Silicon Valley.)

HV F was fully capable of completing the hard task at will, at every attempt. No fatiguing element.

By definition therefore, HV F was also fully capable of completing the easy tasks.

He chose not to complete easy tasks as part of the optimal strategy. His data was excluded due to failure to complete easy tasks.

Vs

How do we know that there weren't ME patients also attempting the same optimal strategy?

But with them the key difference is they couldn't complete the hard task at will at every attempt. If they had, their data would have been invalid also.

---
I don't think you can exclude HV F and not exclude any relevant (all?) ME.

 

I'm hoping that I'm not getting too obscure here.

But I think the ME-CFS break into 3 groups. We've the group that are less impacted by the hard task capability issue (but we don't know that it is 85% of their max, so who knows if it is) and maybe are therefore able to play as normal - but, given what we know about the rest of the ME-CFS I think there is enough canary in the coalmine that we can't assume they aren't slightly affected and having to deal with this using their choice behaviour.

The second group stood no chance. And if you were told to complete 98 and after 6 goes realised you weren't getting near, say about 80 clicks then you'd maybe assume they were testing your resilience to 'try anyway to do as many clicks' or all sorts of things would be going on in your mind. For these ones it would be interesting (if we'd had a much much larger number) to measure them by number of clicks rather than on-off of complete or not.

The middle group are close enough that they are considering adjusting their choices in order to increase their completion. If you are hitting 97 or 96 as near-misses then it is reasonable you mightn't think the whole task is something different, but that completion is still important. But to increase completion you might need to be selective.

Then the whole thing of these latter two groups loops back up to assuming the more capable ME-CFS are getting a task that is 85% of their capability. And that they aren't eg at 92% and having to adjust a bit vs the middle group who were at 95-105% who were having to adjust a lot to get any completions at all.

On this basis the whole game is utterly different for the ME-CFS group to intended. But also utterly different within the group. It might be somewhat interesting but because it is small numbers and mixed in with a totally different game and instructions that focus on the criteria for that , combined with the experimental conditions and as others mentioned the fact it is part of a trial so all sorts of things might have been assumed to be being tested (so it is completely different situationally than the norm where eg people are recruited onto only do this psychology experiment, or only on to do 5 tests that are about effort).

What it does flag is that they've at least managed to pick up that there is at least one capability issue that needs some very careful work done on it re: calibration but probably (when you look at the literature) a different approach and other adjustments if they want to go about validating such a test for use in people who have this condition, or anything similar. Even if it is just participants with comorbidities where the test is more used that will be useful (we know the diagnosis rate is an issue for the condition, and there are probably other conditions that would benefit from such work too) and why should they be excluded from a study eg into their anhedonia/schizophrenia or have their data misinterpreted just because they have ME-CFS.


On doing the test in the first place vs using it as they have in their analyses:

I don't see there was anything wrong with having tried to get this data on the basis of adding it to the literature if it had turned out there wasn't at least a calibration issue. To 'have it there' in case such tests got validated (and they didn't get to 'go back' and run it on participants later). Particularly given they'd only get one shot at what they wanted to colllect on these individuals when they were doing an 'in-depth', and that it was going to take 8yrs and who knows where the literature might go.

But it seems like it would have been better if they'd had the chance to run tests on other similar ME-CFS on these before to actually validate the test first to something that was adjusted to physical capability.

SO the data isn't applicable. Not on the basis of drawing conclusions. And certainly not to dot through the entire paper as a 'metric'. Because they aren't showing what they claim or think they are showing. I know we have a difference in opinion from Walitt who thinks maybe the disability stopping these people from clicking past 80 might be the brain stopping them too somehow but how's he rolled that into his model? At the same time as claiming the number of hard choices of those who clicked 97s was key.

To me, given everyone was basically playing a different game to the one he described this is null. But. This latter bit of the argument is perhaps where we just need to organise our common sense push back to make sure that we aren't caught off guard with anything silly there?

Just at the basic level it would be a very broad-brush way of designating the disability as 'everything' rather than wanting to explain it in the way Treadway's model seems something that is carefully interested in the different patterns after controlling for other issues.

 

Yes, in the paper's discussion they make it pretty clear that being unable to do tasks and having an effort preference not to choose to do them are two mutually exclusive alternatives. That is to say that their concept of effort preference does assume that the person, like HVF, can do the task.

I am still unclear how they explain ME patients' failure to complete the hard tasks. If it was 'deliberate' it is like HVF. If it was not then the problem isn't effort preference, it is not being able to do the task - at least on a binary hypothesis. It seems much more likely that inability to do at least repeated tasks and a strategy of avoiding hard tasks would be part and parcel of the same scenario but with the cause being the inability.

 

Yes I know when I looked through a number of papers (Ohmann's in particular) there is a lot of split-half analysis. Which tends to be possible because they used 'sets' of trials.

One thing we kind of need to know is the order of presentation and take a look at whether it is 'setted' and the second half is similar to the first. And if there is a natural 'set' within the number of trials. such as if 3 probabilities there could be 5 value levels and that set is 15 - whether the values are exactly the same in the next 'set' or just similar and in a similar order etc.

A quick flick through the probability of each trial by order shows there isn't a repeat pattern on these, but it is worth finding out perhaps directly because they could I guess have symmetrically inverted eg from 15 (assuming people would do 30 or something) or done some slightly more complicated sort

I think it is interesting if by memory you cut it in half around 24 17 because there is suddenly a rash of low probability trials EDIT: from 18-24. Which of course as well as affecting number of hard selected for those participants who were being logical, gave them all a 5-round break.

 

@bobbler, the colour palette has changed completely in Windows since I last used it. It has settings called themes, which are accessed from Page Layout > Colours.

The option to create your own set is at the bottom of the pop-out—it's called Customise Colours. You don't seem to get the colour wheel now, which seems a bit rubbish, but anyway that's where 'tis.

 

Agreed.

SO to focus on reduction in click rate but then only compare 'the rate' between HV and ME-CFS seems disingenuous. If we assume that rate has to be 5 or above to complete a task, such a reduction is more significant if your starting ability is exactly 5 vs it being either 6 or 4.

It becomes even more confusing as an argument when you look at the initial purpose the tool was designed for. Of course that would affect choices.

And one point I'd like to add in is the fact that it would be reasonable that the ME-CFS on the right side of the table - the ones who had the least capability on the clicks from the start - to only analyse their performance by clicks (rather than complete). Looking at their click-rate you can see that for that group where they have sequential or non-sequantial 'groups' of hard selected their click rate tends to deteriorate and then they do a number of easy and it goes back up.

It isn't a valid way to analyse them given it has screwed up the experiment, and everyone would have been thinking the game was something different to what it was and have different ideas of the purpose to each other and what was intended. But looking at their click-rate if it had been a different experiment (along the lines of what they might have assumed it was when thye realised they had no chance of getting near 98) does provide some interesting clues that might support our theories of what is going on/all these additional factors being weighed up.

I think it should be hard for them to argue that if your click rate (which was already not near) is going down when you selected too many hard together and it could be a virtue of the condition then that cannot be exclude from either impacting the choice rate or it being something potentially applicable to all of the ME-CFS (but they deal with it in different ways)

This is massively pertinent because: I suspect that whereas we can assume that the HVs on average have the number of clicks for hard set at around 85% of their max ability, and fatigue might therefore affect their choices or performance as the test goes on - but be within that range where they have 15% wiggle-room.

The same cannot be guaranteed for any of the ME-CFS. We know that the hard clicks required is more than 100% capability for about 5 of the ME-CFS before it even started.

There will be a group who are sitting with that 98 clicks being maybe 95-105% of their capability.

 

Oh hurrah! thank you. If I did know this existed then I'd probably forgotten it over the years. That seems far more sensible, I did have a sense of it seeming like there must have been an easier way I was missing!

 

I've just had another quick look and with the cut off/split at trial 17 and then finishing at 34 then:

- trial 18-24 is as follows: low, medium (high value), low, medium ($2.50), low, low - and most people wouldn't play hard for low probability, many not medium.

- trial 1-17 has 6 high probability trials, 18-34 only has 5 high probability

EDIT:

so far I've spotted that in the table organised by number of hard, then yes those HVs who did most hard did each do more hard in the second half than the first half.

Also - I hadn't spotted this before - but HV E has a big gap in the middle of the 'first round' where they pick easy. As does HV N, both from round 5 (but there are earlier ones too for HV E). HV E in particular seems to really get going in the second half.

and ME-CFS I seems to be similar, not doing a hard until round 10. But then is spacing them out by around every 5th round.

I can sort of see why for HV E and ME-CFS I as they both fail their warm-up hards. But Hv N doesn't.

Another interesting thing to note when looking at this on the table. Is that the extremes at both ends are going at it with the hards in a less patterned-out way than the others. You can see it where there are a batch of low probabilities etc across the table.

the 2-3 top HVs just bulldoze through on hard even where most of the rest have gaps. ANd the worst ME-CFS are almost doing similar, except they are nowhere near completing on the clicks. HOWEVER, the ME-CFS worst ones look like they conk out on choosing hard around trial 28.

I think it is worth remembering that the overall difference in number of hard per person is around 2 hard choices more on average by an HV vs an ME-CFS. With all of the data. for 15/16 people.

I don't know whether really there is point in just standing back and emphasising that point. Over around 40 trials. And that includes people who have significant completion-rate issues on one side and not on the other. And isn't across 1,000 people but a very small number.

I'm often slightly wary of the power of doing complex stats when sometimes - particularly with small numbers - the common sense stuff just rings out underneath it all. About whether that rings out as a 'neurological phenomena' based on an averaged total comparing 16 people with 15 people picking 2 more hards across 40 games. When 5 of that group had a completion rate less than 50% (38.5% and less).

It might be interesting, but it is just for curiosity to see whether completion rate in the first half had anything to do with % hard chosen in the second half.

But I slightly suspect we are going to be describing something which includes the effect where you have some of the ME-CFS who never got close to 98 clicks a bit within that.

 

I wonder whether there is a point to be added that whilst there is a whatever on exclusion of HV F in general, when they are talking about button-pressing and fatigue etc there is a bit of an issue with them having excluded.

Not from the analysis, but as a comparator. HV F is basically only doing 10 hard tasks. He might be flunking the easy ones deliberately so it could be argued he had less load there, but when we think about the different finger being used for hard tasks then is he not a useful comparator on the 'finger fatigue' front?

Both for those who only did eg 10 hard or thereabouts about whether 'additional hards' causes reduced pressing rate (and then there is the sequential and so on)

and for those who only did 10 or thereabouts who were ME-CFS but showed a reduction in press-rate vs HV F?

 

Agreed. My gut feeling is that, given this could be considered a psychometric measure somewhat then there is an argument to be made picking on that 85% of peak click rate that has been cited in other studies as a calibration.

The very fact that the authors have tried to use the argument of 'same rate of decline in button-pressing' in a game where for one group that means they went from being able to do 98 clicks to not, and in another group it meant they did their 98 clicks slightly more slowly (because they had that 15% contingency) says it all.

In fact it is worse than that because there are groups within that second group where they couldn't do 90 clicks to begin with. So for them a decline in button-pressing rate, is slightly less pertinent too than those who were around the 96, 97, 98 mark. Who I suspect effectively were in the position where the 98 clicks set for hard was eg 105% of their capability and they could only do it under certain conditions. Adding in an additional factor.

I also suspect that some of those ME-CFS who 'look normal' find that the 98 clicks is well above 85% of their max for them. And so if you are at 95% you don't have the same freedom as you also have to ration your choice of hard and fatigue will show.

Which makes analysing the data and comparing the groups as an off-on and even a scale on the basis of 'hard choices' moot. One group is doing the test as set and the other is a melee of people with varying access to being able to choose hard and complete it.

I haven't worked out how yet. But it feels arguable.

Certainly a first graph, given they've said 'button-press reduced at same rate', would be to do the same graph and put a big thick red line where the button-press rate to complete is (about 5) for both hard and easy? I don't know whether we have done this?

 

Apologies for replying to your post - I could have replied to anything.

Lets say you measure lactate in the brain with MRI - relatively simple [option 1] -

• lactate has a unique MRI signal (energy required AKA magnetic strength known) - you set the MRI to the appropriate intensity for lactate;
• you measure the lactate signal;
• you test it (MRI) e.g. in animal brains and do a standard/"true" (other) test for lactate.

OR
[option 2] you measure the MRI signal, in a wide rage of circumstances, and (by deduction) work out that this particular (MRI) signal relates to e.g. post infection state. Jonathan's point that you'd need a wide rage of controls - other diseases of known cause.

Evidently you can't do option 1 - you don't have a reference material/standard.
They haven't done option 2 - I doubt they could e.g. the thing they're trying to measure may not exist and even if it did it may not be measurable by MRI!

So basically this whole study falls since they haven't proved that the (MRI) signal relates to this underlying issue (ME/CFS).

Begs the question why was it published - oh they'll publish anything if you pay the fee!

 

❤️

 

I think that Treadway's response is certainly another point to discuss, I didn't know if there is some reading between the lines, @Murph might provide further context, but it is interesting. I'm aware that eg @andrewkq and @EndME have mentioned contacting him so slightly wary of how we might discuss this/what might be the best way etc?

 

Hmm. It is an important one. Definitely something for us to have a think on what to do/is possible.

Is it possible to get a sense of the data via a curve or anything in the mean time just to get a feel for the shape of it (I know the range is pretty large for ME-CFS vs HV, but don't know more than that and the mean?)

 

Agreed, and I have all sorts of scenarios I often think of to put HVs in a similar 'situation' symptom-wise and on a even keel.

My little tired brain is now wondering and intrigued by the rather simple idea of how a bunch of HVs would look on this tool if you simply had said the hard task required clicks to a different level. Eg at 120 clicks in the same time or 150 : would you end up seeing similar patterns and heuristics and strategies coming through.

 

agree

 

Oh and PS (distracted brain as I was writing) I managed to not square the circle on my point about calculating it being about '13' hards you'd select if you were going for 'only high probability'

If you look at this group, the only one in the middle: 'ME-CFS I' that doesn't have very long choice times at all, well hey presto they've chosen exactly 13 hards.

I haven't gone back and checked that this meant they just used the heuristic of selecting hard only for high probability, but if they did then that would have lightened the cognitive load somewhat?

If you were struggling cognitively as well as physically that sort of thing would make sense.

ME-CFS J is also another one with lower choice times for both hard and all. And their % hard selected is 34%.

Again I haven't gone through and checked this is that they've gone for 'only do high probability'. But it is pertinent because whilst they complete all of them, it indicates that potentially there might still be a calibration issue even for those who look good compared to the other ME-CFS on completion, and are having to use these heuristics due to both cognitive load on decision-making and physical limitations having to be part of their strategy.

 

Ooh on that point I know that Ohmann's modified EEfRT version uses an unlimited number of clicks approach in: Examining the reliability and validity of two versions of the Effort-Expenditure for Rewards Task (EEfRT) | PLOS ONE

I don't know if there are other trials that have done similar and if the data might be available to such a level as to see whether we can see where 100% is for HVs etc. And maybe whether this drops over time/as test goes on.

That might be useful for demonstrating that whereas for ME-CFS the number of clicks required/button press rate is at varying levels that are close to or under the number required for completion there is a lot more headroom in the HV population? ie that it is set at around 85%?

Perhaps the papers where calibration tests were done might have this info too?

 

I've been musing one way then another on this HV F issue.

To seem like I'm side-stepping it a bit (and I agree it was comparison with HVs not 'only HVs who don't exhibit x') it could be worth going at this one being a problem from two angles:
- having an HV F in your data wouldn't be an issue if you had sufficient statistical power ie numbers of participants in your trial. The existence of them having to do this points just as much to underlining how flimsy their numbers are as anything else

- if, like the original validation studies done by Treadway and others eg. Treadway et al (2009), they had been using 'ness' ie it was correlating with scales of traits/concepts/issues then HV F probably wouldn't be an issue either.

It's a problem, just as combining up all of the ME-CFS is a problem, because Walitt et al have chosen an approach of aggregating by group when there are significant within group differences.

EDIT: oh and my last point is that if we take this as 'an attempt at validation' then HV F requires a big part of the discussion. Instead of a footnote and removal.

As Ohmann et al (2022) discusses the issue with the original EEfRT is strategies for example you would expect that rather than seeking to just remove this, it would be a key finding as part of this being an exploration into what is needed to validate the tool.

And it is clear that instructions/adaptations might be learned from this cropping up.