Use of EEfRT in the NIH study: Deep phenotyping of PI-ME/CFS, 2024, Walitt et al

Any update on the letter regarding effort preference and EEfRT? Has a summary of the issues identified here already been sent to the NIH researchers?

@andrewkq @EndME

 

It's still in the works. I crashed and had to take a break from working on it for a bit. The current plan is for me to have the next draft of revisions ready for the co-authors to review in mid to late June. So hopefully looking at sending it to the NIH researchers in July and submission a few weeks after that.

 

Great to hear it is ongoing. Thanks for working on this.

 

Posted on twitter by Jeanette Burmeister
https://twitter.com/user/status/1800316244845035671

Part 1 of my new 4-part article about NIH’s Effort Preference claim for ME. Read about how NIH has misrepresented their own data. Start w/ understanding that NIH built on Wessely’s work: ME as a disorder of the perception of effort.

Her first of four blog posts is here - https://thoughtsaboutme.com/2024/06/10/1/

Part 1 goes through a lot of historical background. She said she will share in part 2 her analysis of the EEfRT data and of NIH’s misrepresentation of the data that was required to make their Effort Preference claim.

In the meantime, not sure if this was already discussed here but in case not, her blog references comments about effort preference NIH made in their response to reviewer comments (page 12 line 244)

"The approach selected with GEE was necessary to determine the primary objective of our study, the existence of EffRT performance difference between the PI-ME/CFS and HV groups. The Cooper 2019 approach is not designed to determine group differences in performance. Rather, it is designed to dissect out how participants are making their decisions (i.e. which aspects of the task are being weighed in making decisions about hard/easy task selection). Use of the Cooper 2019 approach would help determine the contribution of individual aspects of the task to the performance outcome, such as how subjects integrate reward, effort, and probability to guide decision-making.

As our data did not show differences in reward sensitivity and probability sensitivity by group, this approach seems unlikely to provide information regarding the primary outcome. We have added this sentence to the Supplement, page 10, lines 419-420: As Models 2 and 3 did not show differences in reward sensitivity and probability sensitivity by group, further analysis was not performed. We have also justified the reporting of trial timeouts with the following sentence in the Supplement, page 10,lines 421-422: No difference in decision timeliness was observed as measured by task decision timeouts (0.3% versus 0.6%, p = 0.19)."​

 

Sharing this message I received today from an individual with HHS, if helpful to know for anyone still working on this here:

"Our office did refer this complaint to the Office of Research Integrity ( https://ori.hhs.gov/ ) which does deal with issues like research misconduct and misrepresentation of research results in publications. Because of this, you may be contacted by that office in the future."

 

Does it in some way avoid the ‘correlation issue’ of what is causing what given disability existed before the task —> correlation with completion can’t be ‘ in reverse’

and I’m not sure it makes logical sense to think someone would complete a smaller % because they chose the hard task less rather than vice versa.

that you can’t ignore those and just say disability level correlated to % hard tasks might help avoid obfuscation?

although whilst I try my hardest to predict the mind of…

 

Part 2 of Jeanette Burmeister's article on the NIH intramural study is posted on her blog. It is long and requires concentration to read but she shreds the study including Madian's response to an advance question.

https://thoughtsaboutme.com/2024/06...e-study-lies-damn-lies-and-statistics-part-2/


The blog is complex and I have done no more than skim it. Even this might be an exaggeration.

Jeanette's description of it is as follows.

"This post is the longest in the series and requires a fair amount of stick-to-it-iveness both in terms of length and complexity of the issues and details discussed. I realize that this will, unfortunately, be beyond the limits of many ME patients, but I decided not to divide it into smaller parts due to the connectedness of the issues and in order to allow for easy sharing with and reporting to the appropriate authorities and other interested parties of the main reasons for why this study should be urgently investigated and retracted."

 

While this is a very complex article covering the data, this quote sums it up nicely:

"It is easy to see why the authors chose not to generate a visual of what actually happened during the EEfRT and instead resorted to manipulating the data with statistical tools until they arrived at a figure that fit their desired outcome (Figure 3a and Supplemental Figure S5e). The latter allowed them to make it look as though patients chose significantly fewer hard tasks for every single trial throughout the EEfRT while the former shows clearly that their Effort Preference claim has no legs."

 

The comparison is astounding:

NIH chart:


Chart created from raw data (from blog):

And even if counting the little difference in the middle of the real chart as meaningful, the task was not to do as many hard tasks as possible, it was to win as much money as possible, which required strategically choosing between easy and hard tasks.

Oh hey, the pwME won more money with their strategy (though just barely):


chart from blog

 

Yes! I'm glad that she has pulled up the usage of the following charts. ANd how they seemed to be presented as if they showed the data, when they are instead statistical legerments.

It shouldn't take people more than a few seconds to eg look at the end point and see that it doesn't map to the number of rounds completed even, except most people won't have downloaded the raw data, and they would never think that someone would have done this so would naturally assume a picture of this kind would be showing the data?

 

I agree, when you roll in the lack of calibrating for disability (which is in Treadway's instructions, despite his reply), and the news articles and discussion noting that participants who had ME and not healthy volunteers were warned that they should be thinking about how these decisions affect their health. Using parametric tests when the data was obviously not 'normally distributed'. ..

Is there other stuff I've missed?

And then combine it with it being 4 individual choices across approx 40 trials numbers with not even even numbers of participants in each group (15 vs 17, and then there is the HV that got excluded).

I don't know what 'would seem to be manufactured finding' counts under regarding all the definitions of any sort of wrongdoing or being misleading etc?

But [to eventually end up with] 4 decisions (out of how many hundreds?) and it turns out that the ME group won more money, across only 15 participants, ALL of these events/errors/warnings/non-calibration etc had to take place, AND they had to fish around with what they inputted into their GEE (and was it here they used the wrong tests statistically, so they didn't apply to the data shape?), to even get some warped statistical output?


So to then present with diagrams (representing that statistical creation instead of the data itself) that suggest to most readers visually there is a consistent, or large difference, without clear labels flagging this is actually displaying some legerment that is particularly unusual...

That combination of things ceases to be in the realms of error or mistake, as it isn't even the same error carried through or anything?


What I cannot get past is the gall to then take a non-finding - as the paper confirms, a legerment - and try and use it to underpin some comparison with a handgrip and then treat this like some truism throughout the whole paper to re-frame and ergo 'pollute' other potential biological findings that were done at least with methods appropriate to the sample size and aren't such statistical artifacts...

 

Thank you to everyone working on dissecting the study.

In skimming-mode only and most is way beyond me anyway but stumbled across a quote from Jeannette's blog:

Too unwell to check myself -- so not clear to me: Do they mean the primary objective of the EffRT performance study as part of the whole intramural study besides other incorporated studied or do they mean the effort preference stuff was the primary objective of their whole study?

So only a primary outcome or the primary outcome of the whole project?

Anyway the "deep phenotyping" article seems to be the study report of the whole study.

I wonder how and when the effort preference stuff came up for the whole study? Was it in the protocol right from the beginning? If yes, what weight had the investigators given to that particular part of the study in relation to the other parts of the study -- all the biomedical investigations?

Anyone had a look at the protocol(s)?

And what did they say in the protocols about the required number of participants to have statistical significance?

If that has already been discussed -- could someone point me to the according posts?

 

This is Jeanette Burmeister’s summary of part 2 of her blog: https://threadreaderapp.com/thread/1800987624619782625.html

@dave30th Are you intending to blog about this too? (Apologies if I’ve missed it.)

 

I may be have been naïve, but to begin with I gave everyone the benefit of the doubt. But every time I go through these three stages now:

1. I can see that there are so many confounding factors and potential biases that results reported are not going to mean much anyway. I tend not to analyse further.
2. When others struggle through the data (or when I see a truncated Y axis when copying a graph to us at NICE) it becomes clear that there wasn't even a meaningful result.
3. Tracking the history it becomes clear that the authors must have known 1 and 2. In fact they had worked very hard to conceal both.

 

When I looked at these graphs I thought --- basically there's no difference between the two groups -- just the sort of "noise" you'd expect (measurement error). Then I looked again &, as per your comment, they're the wrong way around i.e. contradicts the claim!

We can't demand that he goes but I do think he's clearly making unfounded claims and that surely is in conflict with NIH's mission - perhaps I'm wrong, but the controversy around this will be noted by the higher echelons in NIH!

 

No wonder it took Walitt and Nath so long to write the paper. They had to find a way to present the data so it showed what they needed it to show, rather than what it did demonstrate.

Ugh. This was predictable and predicted once Walitt was given the role he was.


edit: duplicated word omitted

 

The sample size was so small that by hunting around in the data you could have found a way to support any number of hypothesis. Especially with the number of tests run, the authors could easily pick and chose what they wanted to highlight. From what I remember there were results in the appendix that were interesting, and may have been close to statistical significance, but weren't mentioned in the paper.

 

Just to underline the absurdity..
Pair this with the repeated grip strength testing, and imagine the title: “ME/CFS is associated with better risk/reward navigation.”