Use of EEfRT in the NIH study: Deep phenotyping of PI-ME/CFS, 2024, Walitt et al

You're very kind. For me, making the white less white and the colours less bright would help. Pastels are good. Darker shades can also work eg navy instead of blue, burgundy instead of red, depending on what other colours are there. But I wouldn't waste much of your energy on it, certainly not just for my sake.

 

OK please do not be afraid to tell me if I've made it worse, paler but potentially brighter - I'm not sure myself if the pale blue seems even more bright (but the yellow seemed to just add more complication). I've changed the lines to navy. Let me know if it works for you or any tips on colourways/if there is any particular thing that is jumping out as annoying or I've managed to make worse!




Edited to thumbnail image

 

They could also be functioning correctly, and they're just missing the reason why they are. More likely, actually.

 

This is much better for me! (Except the two rightmost columns which are still problematic. Could they be a different colour entirely? Lilac or something like that.) The red is still a bit too intense - paler or darker would be better for me. There's an effect when blue and red are beside each other that makes the red pop, so a different colour might work even better - not sure if there are any inoffensive greens in Excel's palette. I always tone things down by reducing the brightness on my screen, but there are limits to what that can achieve.

Thanks so much for doing this.

 

@Evergreen OK please do not be afraid to tell me if I've made it worse, paler but potentially brighter - I'm not sure myself if the pale blue seems even more bright (but the yellow seemed to just add more complication). I've changed the lines to navy. Let me know if it works for you or any tips on colourways/if there is any particular thing that is jumping out as annoying or I've managed to make worse!



There are a few blues so I've gone for the navy set, can choose the other blues or look at lilacs if this 'isn't it'





Edited to thumbnail image and lost reply formatting so added a tag for context of stream of convo

 

So much better! If you can tone down the red I'll be able to feast my eyes on it!

 

I've tried rose, as lavender looks a bit bright, burgundy/maroons can't read through it (which is fine for the 'dotty bits' but then what I pair for the header parts is the question) and I didn't know whether the more orangey/brown red palette would be worse or better than going down the purplier route

You are teaching me a lot as it's easier for me to look at too!

All these years!



edited to thumbnail image

 

This is great! It's is gold for the critique. There is an established precedent in the literature for performing individual calibration of the hard task when the EEfRT is applied to a disabled population. Wallit et al failed to follow precedent, resulting in invalid data.

 

Just started to look at this* - yea he seems to be a figure head - is he a (NIH) Director? Guess that "seriously asking patients to believe this?" i.e. putting a +ve spin on uninspiring ($8 million) studies comes with the pay packet!

https://solvecfs.org/solve-responds...al-me-cfs-clinical-study-published-in-nature/

 

@bobbler, are you using MS Excel?

If you are, you don't have to use the automatic colour sets. You can set the RGB balance and black level to anything you like. It's the More Colours option, then click on the colour wheel.

(Sorry if you already know this.)

 

Aw listen, I know. It took me a while to figure out. Little things make so much difference, like black and ecru is much easier to look at than black and white. I find shades labelled "dusty" are brilliant, like dusty blue, dusty pink, dusty green - muted and calm.
In graphs I like the "More colours" option that @Kitty recommended.

 

I have received a very swift response from Brian Walitt to my questions. I believe they have been handling questions to their paper very well, since they surely must be getting an abundance of these. Since this is a public forum and since I haven't asked him whether I can share his answers in public I will briefly state summaries of my interpretations of his answers

• People being ambidextrous, shouldn't be a problem. They measured handedness for each participant.
• The rules were explained to the participants by an administering neuropsychologist and participants were able to ask questions. More precisely "Participants are introduced to the easy and hard trials of the task and informed of what each involves and how to choose each type of trial. Then, the varying rewards and probabilities are explained, and finally, the piece about receiving the winnings from only two winning trials is explained, and some examples are provided. Participants are strongly encouraged to ask questions before, during, and following the instructions, and are given every opportunity to do so. Finally, participants’ comprehension of the task instructions is assessed with four practice trials, and further clarification provided if necessary. It usually takes around 5 minutes to explain the task instructions. The practice trials are administered immediately after the instructions (but after the participant has had the opportunity to ask questions) and the remaining trials begin immediately after the practice trials are done (but after any additional clarification, if necessary)."
  • To me that sounds like these are all one on one sessions and participants can't discuss this game amongst each other. It would be nice if someone who participated in the trial could confirm this.
• The response to why exactly participant HV F was excluded has remained somewhat vague to me, or more precisely I haven't yet received precise information on the defined exclusion criteria apart from "Review of data for performance and task validity is part of the pre-existing way of handling the data; the evaluation of invalid performance and task validity takes place after the data is collected.". So it seems they indeed had prespecified criteria, rather than judging the data post-hoc. He mentions that this participant clearly had no problem completing certain tasks but rather chose not to (which we can all agree with), which to them indicates that this is either an invalid performance or an invalid task administration.
  • It would be nice to know what exactly the prespecified criteria are, I will ask him whether these can be provided, but I also don't think the authors would have to tell us these.
• Regarding something @Trish also mentioned, as one would expect I haven't received too much details on subsequent publications. However, he mentioned that they are looking at the relation between some of their findings (the EEfRT work, the fMRI measurements and celebralspinal fluid measurements) but that publication and timing depends on the findings itself.
  • Perhaps someone can get better information on some of these details. The response to me suggests that they will very likely be writing something, but that it hasn't been submitted anywhere yet. Was the plan for the subsequent release of publications cancelled or did I/we misunderstand this a bit from the beginning (@Dakota15?) ?

I have not received a response from Ohmann yet.

 

Below I will be presenting my first collection of plots. For the largest part this is what the brilliant plots @Murph, @bobbler and @Karen Kirke made have already shown. All of these plots and calculations plots were done in R, so the colours, formatting and presentation can easily be changed and these plots, code and calculations (mean, SD etc) are suitable for being published if the appropriate changes are made.

I will be presenting every plot once for all the participants and once excluding HV F (his data is pretty irrelevant for what will be presented here but I've included it anyways), the same applies to the calculations. All of these plots are centered around the completion rates of hard rounds, easy rounds and their completion rates have been ignored for now.

The first plot is similar to what @Karen Kirke first presented, but I chose a slightly different graphical presentation and also included mean values (red is for ME/CFS, blue is for HV and dotted lines are mean values for successful completion whilst the continuous line are the mean number of hard trials), that I believe already tell us a lot (@Simon M first brought up the idea of including these I believe, I have sticked to ranking them alphabetically for now but will still look at whether ranking them by things such as SF36 could reveal something interesting). The ratio of the different colours directly shows the percentages of hard rounds successfully completed, instead of this being a separate piece of information.


The same applies when HV F is excluded, see below.



These plots clearly indicate the massive difference between the percentage of pwME and HV that complete hard tasks. Note also have far away pwME stray from the mean, there is a lot of deviation. This had all been discussed at length already.

The second round of plots are now for the analogous plots from above, however only for first 35 rounds of trials (excluding the practice rounds). These plots are more "interesting" than the plots above as they ensure that all participants have played the same number of rounds and also limit in-game fatiguing effects. Focusing on a fixed amount of rounds has been done in several different EEfRT studies, the choice of 35 is somewhat arbitrary and varies from paper to paper (large enough to be significant and not too large for in-game fatigue to occur).



The same applies when HV F is excluded, see below.


What we see here is just as striking as what we had already seen before.

Now come the plots of the trial rounds:



The same applies when HV F is excluded, see below.



This plot clearly shows us the fundamental problem in the trial. In the trial rounds fatigue caused purely by the game and motivation/stragety will play less of a role. Yet pwME cannot complete the tasks. We cannot know whether the results of the study would have been different if trials rounds would have served as calibration rounds, however failing to use them as such means we cannot appropriately interpret the results of the EEfRT. Note here that pwME actually choose the hard task more often then the controls. This is the only time this is the case. Unfortuntely, due to the small sample size it's hard to conclude anything from that apart from the fact that not appropriately calibrating the EEfRT has led to inconclusive results. The median of HV playing hard rounds during the trial rounds appears to be proportional to that in the first 35 rounds.


Finally I have plotted the hard rounds as the game progresses divided into two halves, the first half of total rounds played per player and the second half (i.e. player HV A plays in total 48 rounds so the first half are the first 24 rounds and the second half the next 24 rounds and then we look at the hard rounds in those in those halves separately, if someone plays an uneven number of rounds then this extra round is part of the first half of the game). The behaviour of these plots might be dominated by people who play more rounds, I will later plot this using a cut-off so that everyones halves are equally big.





I believe this hasn't been plotted or discussed yet at all. It strongly suggests that the mix of learning effects+saturation effects+fatigue as the rounds progress actually dominate the completion of hard tasks in HV more then they do in pwME. Both are still opting to go for hard rounds just as much as in the beginning of the game, but even HV are completing them less often now. This has been seen in several other studies and is one of the reason a universal cut-off is typically used in combination with using trial rounds as calibration.

In terms of completing hard tasks: After having played the first half rounds HV are now closer to the same point pwME we at, at the beginning of the game. Would their behaviours eventually converge? We would need far more data to know more about that....


The same naturally applies if HV F is excluded (see below).




In a next collection of plots I'll consider the above data under the aspect of the SF36 data of the participants and also the different reward and probability levels of hard trials.

 

Isn't this an "effort preference" by HV F?

ETA:
Ie this test data is said to be invalid because it's deemed to be a pure preference on the part of the person - and not related to effort ("this participant clearly had no problem completing certain tasks").

But the ME patients' data is deemed valid, where the researchers are saying it's not a pure preference, it relates to effort.

So the discriminating feature between patients and controls is effort, and preference is simply a proxy measurement for the effort the patients experience that HCs do not.

 

@EndME - Nath said last week to me they are still planning subsequent papers to be published.

When I asked if we should expect to see these in Nature Comms he said “who knows.”

 

Thanks. Do you know whether that means they still have to write and come up with these papers or are they so to speak "ready to go" (for instance if they would submit them as pre-prints, which historically seems unlikely)?

 

Unfortunately I have no idea on that

 

As far as I've seen, I haven't seen the word "effort preference" show up anywhere, however terms such as "altered effort allocation" do show up in some of the studies and arguably those terminologies seem quite very similar, at least to me. I don't think the conclusion this study makes are too different to some of the other studies, them not calibrating the method to their cohort however seems to be the main problem.

Seems very sensible to me! I think it would make sense go through all of them at depth by assigning different categories to anybody who is ready to get involved in this (i.e. one person looks at Schizophrenia+Depression or something similar), once we've decided on the route we want to take in our response and once we've gained a bit more overview of everything. The most sensible thing in my eyes is to first write down a list of things we want to look for and comparisons we should be looking out for and then we assign them according to the categorisation I made on the thread Worth the ‘EEfRT’? The Effort Expenditure for Rewards Task as an Objective Measure of Motivation and Anhedonia, 2009, Treadway et al (one of the things one easily sees is that most other studies made sure people were able to succesfully complete the trials). The list and summaries of the papers I made there is of course far from complete, so that would be something to consider as well (if anybody finds a paper of relevance just add it to that thread and I'll insert it into the list I made there).

This thread has also gotten awfully long and most of us are probably stuggling to keep up with even half of the messages. As such we should probably ask ourselves: Where are we standing right now? What do we now know with certainty, what exactly don't know yet and what should we still have a closer look at?

 

Sorry I was asleep/resting for a while there. I'm going to have to find a way to save some of these colours. You'll probably tell me some of this is already available and I'm just missing the knowhow but..

Now I know all of this I'm a bit thrown why packages like excel don't have eg colourblind and muted versions on their standard options either, seems like one of those to do list projects in case (you never know) it genuinely is just for lack of someone putting it to them. I like the fact you can choose all colours and yes when charts get complicated you need to be able to play as it is what goes alongside each other, but having a 'muted colourwheel' to start with and a conditional formatting which has 'dusty colours' or puts in navy or burgundy lines as a starting point, well it just seems so replicable to so many. Or at least allow people to save their own combos.

I'm sure I've done all of this before in the past for something and then forgotten it all, and have memories of being in the office years ago playing with colours for people who had no health conditions at all so there is clearly demand.

Sorry on a complete sidetrack from the task at hand but.. the default tends to rule (and so you assume you need the contrast for people to spot things, when I know that isn't true from this work 'experience' in the past but it is so easily forgotten when faced with the 'norms') and if it's on the drop-down it is a good way of spreading awareness and getting change just because others will see it and realise it's less strain on the brain (I know you know all this already and I'm just being slow to catch on ). Do we have a section on this somewhere on this site yet? for top tips like this?

 

1. My memory of this research program is they brought each patient in sequentially. Each person was at nih for around a week, one after another. (Which is why it took so long). Somebody way upthread was pondering this too but there's no way for subjects to speak to each other irl.
2. Because this research is funded by the US federal government directly it should be possible to get our hands on information. Freedom of Information Act covers NIH and if they won't give info nicely someone can pursue them for it. An interesting request might be All correspondence and minutes of meetings relating to the use of EEfRT and susbequent analysis of said data - see if Nath and Wallitt had any disputes about how far to push it.
https://www.nih.gov/institutes-nih/...public-liaison/freedom-information-act-office. Probably a job for a US citizen to do the actual filing!