Use of EEfRT in the NIH study: Deep phenotyping of PI-ME/CFS, 2024, Walitt et al

I was only able to skim it briefly but it looks like a good blog that identified many of the same issues discussed here on the S4ME thread.

One issue that has the title 'False Recording of EEfRT Data' was new to me. Burmeister argues that there are errors in the raw data because the value 'Reward_Granted_Yes_is_1' is often 1 when 'Successful_Completion_Yes_is_1' is 0. So that would suggest that the reward is given even when the participant did not successfully complete the trial.

However, this seem to have happened more than 60 times which is quite a lot. Another possible explanation is that Reward_Granted_Yes_is_1 is simply the end result of the probability that the reward would be given (yes or no). A bit like you toss a coin and write up the end result: head or tails. If that is the case, then the rewards given are those that have a value of 1 for 'Reward_Granted_Yes_is_1' and for 'Successful_Completion_Yes_is_1' (instead of only 1 for 'Reward_Granted_Yes_is_1' which Burmeister calculated).

If I do that, I get slightly different results for the mean reward per group.

55.54 dollar for controls
48.42 dollar for patients​

So patients had a lower amount, but not because of hard task avoidance but because of lower successful completion rates.

 

Sorry if this is the wrong thread but I’m just dumping these Tweets here while I have the links as I’m running out of gas and they are indirectly relevant to the discussion above as they are by the same author:

I also noted this from part one of Burmeister’s blog series on EEfRT (https://thoughtsaboutme.com/2024/06/10/1/):

 

I don’t quite understand this. Apologies if I’m being dim but can you or anyone else make it clearer?

I think it’s important for any analysis to understand which interpretation is correct. If ME/CFS Skeptic’s suggested interpretation is correct, it does not invalidate other criticisms, but if an incorrect interpretation is used in any published analyses it could be used to deflect attention from and discredit valid criticisms.

 

In the experiment, participants had a chance to receive a (virtual) reward in each trial round if they were able to achieve the required number of button presses. This was a requirement but they did not get it each time they achieved the required number of button presses. There was also a probability that the reward was actually given: either 12%, 50% or 80%. So in orde to receive an reward, you have to be successful but also have a bit of luck.

Now in the raw data there is a variable called' 'Reward_Granted_Yes_is_1'. The most logical explanation of this is that indicates whether the reward was given or not. This is what Burmeister assumed. But then there is the issue that in 60 rows of the raw data this 'Reward_Granted_Yes_is_1' is set to 1 (yes) while the other variable 'Successful_Completion_Yes_is_1' is 0 (no). So that would suggest that the reward is given even when the participant did not successfully complete the trial which should not be possible. Burmeister therefore assumes that these are errors and they certainly could be.

But because it happened so often I was wondering if the variable 'Reward_Granted_Yes_is_1' might indicate something different. Perhaps it is simply the end result of running the probabilities (12%, 50% or 80%) without taking successful completion into account. A bit like tossing a coin that has a 50% of getting tails and you write down 1 if it was tails and 0 if it was head. If that is the case then the rewards that were given are only those that have a score of 1 for both 'Reward_Granted_Yes_is_1' and 'Successful_Completion_Yes_is_1'.

This is just off the top of my head. Don't have the energy at the moment to analyse Burmeister's blog or go back to the original paper. Hope others will have a more thorough look.

 

This makes the whole "stop criticizing us or we'll be sad and give up" look even worse. They can't possibly not understand that the criticism is fair and accurate, this was as egregious as what PACE did. They could have made this BS test some minor element with 1-2 mentions but they chose to make it the bulk of the study's interpretation. Even having made it a minor element the study would have been largely a waste because they made so many poor decisions along the way, and had the wrong people in charge of it. On purpose.

One thing I noticed is that the blog mentions that it isn't clear why the fringe control was dropped, but I think we discussed this here, that they considered that he didn't play the game as they wanted it, but that control actually played the optimal strategy, and throwing away his data is what tipped the tiny significance, which is obviously why they did even though there was a much stronger case to throw away the data from the patients who couldn't perform the hard tasks.

As is tradition, it's worse in context and the more context you add the worse it is. This paper has to be retracted. As badly as PACE. It probably won't because the goal is not to help us, but to push the same ideological quackery. Walitt has no business being involved in medical research and we were fully right to object to his involvement. He is as much an ideologue as Wessely.

This also makes the lengthy delays look even worse. This paper has nothing groundbreaking to it, it was published in a 2nd tier paper at best and basically all they've been doing ever since is damage control, saying somewhat the opposite of what is clear in the paper, which is about as coherent as clinics who argue that they must do GET, on the basis of GET studies, but that what they do isn't called GET, it's different, even though it isn't.

The main problem with politics is being unable for people to agree on what reality is, on what facts are. Science is supposed to be able to work based on facts, it's supposed to matter. And yet decade after decade we see people doing obviously wrong nonsense like this, in ways that fully betray that they know what they're doing, they're just confident that there's nothing we can do about it.

 

Did Jeannette state in her work (and blogs) if she was submitting this to anywhere (Office of Research Integrity with HHS, etc.)?

 

Now that I think about this, there is probably an easy way to test this. The mean of 'Reward_Granted_Yes_is_1' is 0.48 and if I take the mean for each of the probabilities of reward it looks like this.



So yes I think this variable does not take successful completion into account and simply reflects the probability of reward. It looks to be running the probability of the luck factor.

 

Thanks for doing this and for the explanation. Yes, I think you’re right. I will bring it to Jeannette’s attention on Xitter, as well the point Richard makes above about why control F was dropped. Does anyone know if she’s on S4ME?

 

I seem to remember from looking at the sheets of raw data that it is your interpretation ie probability outcome.

whereas I think number of clicks was what I used to see if people achieved the hard task


It’s a while back now so feel cagey in case I mislead but do remember having similar questions as I poked through working out the sheets and variables and task etc.

 

As @rvallee mentioned this above, I’m sharing @Murph’s excellent post from earlier in the thread, which shows that Healthy Volunteer F was playing an optimal strategy – and that EEfRT is fundamentally flawed:

@andrewkq, did you get a reply to your email enquiry about this? (Apologies if you shared it and I missed it.)

 

Jeannette said:

But the issue wasn't that they didn't find POTS in the patients. The issue is that the healthy controls also had POTS.



The discussion about orthostatic intolerance continues here
Orthostatic Intolerance in PwME (POTS?/NMH?)

 

And where one of the "healthy" controls later developed Long Covid after viral exposure (a first degree relative).

 

Jeannette is doing 2 more articles on this topic. I don't think she has (yet) mentioned about submitting.

 

I haven't yet. I'm trying to absorb Jeannette's blog posts.

 

The next blog post by Jeanette Burmeister

https://thoughtsaboutme.com/2024/06...e-study-lies-damn-lies-and-statistics-part-3/


"In this Part 3, I will discuss the EEfRT as a psychological measure, NIH’s frantic attempt of damage control in response to the firestorm reaction to the intramural paper, the agency’s decades-long obfuscating characterization of ME as merely fatiguing, its reframing of fatigue in ME as being purely subjective, the investigator’s fear of using a second-day exercise test, and NIH’s ongoing research of an allegedly dysfunctional Effort Preference in ME."

I need to start reading. It's a lengthy paper but not as complex as part 2. Interesting content.

 

Part 3 is taking me longer to read than I expected. I think this may be because it is based on a much deeper knowledge of the background to this study than I have, as well as the individuals referred to in the paper and the politics surrounding it all. Maybe our US members might be more familiar with all this.
I have some hospital appointments tomorrow and later in the week so I won't finish it but will post what I've read.



Jeanette began part 3 by arguing EEfRT is a psychological measure and was
developed specifically for use in psychiatric populations....and so should therefore not have been used on a physical illness.
"Claiming that the EEfRT is not a psychological measure is simply untrue."

The NIH has responded by arguing that it is not a psych concept, and bringing in senior staff ( Komaroff and Koroshetz) to emphasise that ME is real ( thus drawing attention to the fact that not everyone believes this) and to praise Walitt and Nath.

She outlines how the NIH supports the study and offers a critique.

"Koroshetz labored to persuade the audience that effort means something different to neurologists than it does to the lay person. He claimed that the study has shown that there is a computational problem in the brain of the ME patients that leads to them unconsciously overestimating effort and/or underestimating rewards due to no fault of their own. He was emphatic when he said that NIH is not claiming that patients do not want to make an effort, except that that is exactly how the term Effort Preference will be interpreted—the only reasonable interpretation. No intellectually honest person would argue otherwise unless completely clueless".....

Jeanette argues "Linguistically, the term preference strongly suggests a choice—in this case the choice of ME patients, due to some alleged miscalculation of how much they can exert, to invest less effort than the authors claim patients safely could. "

"The NIH disagrees and claims that their reference to "effort finding is based in neurobiology. Then why not name it something that sounds like neurobiology instead of laziness?.................it would have been easy to choose a respectful, scientific designation, such as TPJ Dysfunction, but that would have defeated the purpose of the EEfRT testing. Obviously, the term Effort Preference was chosen for maximum detrimental impact on the reputation of ME patients.".......


She introduces pacing as an activity of choice.

"The finding of a difference in effort preference is consistent with how participants describe pacing. One participant describes: ‘You have to make a conscious choice of how much energy [to use and] whether or not something is worth crashing for. It’s hard because no sane person would ever participant [sic] to suffer and … that’s what you’re doing [by choosing] an activity that will make you crash. You are going to suffer… You have to decide what gives you meaning and what is worth it to you.’” [emphasis added]"

It is a lengthy paper.

 

Final blog post in the "Lies, Damn Lies, and Statistics" series by Jeannette Burmeister:

The NIH Intramural ME Study: “Lies, Damn Lies, and Statistics” (Part 4)

Ends with a call to action:

 

Hadn't spotted that particular one, but apalling if that is the case surely?

As things were so knife-edge even with all of these coincidences added in