Use of EEfRT in the NIH study: Deep phenotyping of PI-ME/CFS, 2024, Walitt et al

I've been trying to calculate this by adding a new column: 'previous_trials_completion_average'. It initiates at value 0.5 (equal chance of completion or not) and then for row i takes the average of the completion rate of the previous (i-1 rows). It calculates these means from the 'Successful_Completion_Yes_is_1' column which has the completion rate of all trials rather than just hard ones. When it hits a new ID participant, it starts over.

Would this be a (rather poor perhaps) implementation of what you mean?

 

Awesome! Yes this would be a implementation of what I mean.

I think a more meaningful interpretation would use some sort of discounting on previous trials (it probably matters less whether you completed or didn't complete a hard trial 20 rounds ago, but one will remember a previous success/failure/effort more). The easiest thing one could do is say something like the previous hard trial counts with 50% and all other previous hard trials make up the remaining 50% (e.g. if your past history for hard trials, starting with your first hard trial and nding with your last hard trial, is failed-completed-failed-completed-failed-completed your rate completion rate would be 2/5*1/2+1/2=0.7 rather than 3/6=0.5), the choice of 50% and how many trials one counts towards being "in the previous hard trial" is rather arbitrary.

Alternatively one could try to do something more elaborate, which should also be easier to implement, is using something like using a discounting factor and then have the rate completion on hard trials (which I call r for now) be something like
(i.e. in the above example which for yields the thing you implemented. The smaller you choose , the larger you discount moves that lie further in the past (and for gamma=0 and the usual convention of 0^0=1 you get that only the previous hard trial contributes to your next decision). Of course for one then gets rates smaller than 1 which doesn't reflect the true completion rate for anyone (think of someone that is able to complete all trials but has a rate r smaller than 1), however since we're more interested in comparing people/groups than reflecting true values, I don't currently see why this would matter.

One step higher up would be to try to even use a Markovian model, but I think that might be far too complicated for now and I think starting off with something a bit easier might already tell us if it could be worth of any effort at all.

A priori I would think one should focus on only hard trials and whether those were completed or not, to not dilute rates of those people that do easy because they can't do hard (for example pwME do hard marginally less often so one would artificially dilute their values if easy tasks are included) and since everyone is able to do easy trials there's nothing interesting there.

 

Not sure if this goes here or in the other thread.

The paper noted:

But this thread describes the high rate of failure of pwME on the EEfRT tests. That suggests pwME are not accomplishing what they are intending. Has that discrepancy been discussed here and/or do the authors explain anywhere how both can be true?

Apologies if this was already discussed.

 

I agree that this is not expressed clearly in the paper. I think that when they say

they are referring to what pwME intend to do when effort preference kicks in, rather than what they intend to do at the beginning of the task.

If you view effort preference as involuntary (as Nath has said in an interview, contradicting the paper) or unconscious (as they write in the paper is an option, should this be subconscious?), then they're saying the pwME initially tries to do what they're asked, their brain stops them (likely coinciding with pain or weakness, though symptoms aren't reported in this paper) and they aim lower.

If you view effort preference as voluntary, ie pacing, then they're saying the pwME either aims lower from the beginning, or initially tries to do what they're asked, encounters a barrier (symptoms, they just can't do it), and then aims lower.

 

Apologies for being absent, my health has taken a turn for the worse so I've had to cut back on how much time I'm spending on this. I've been slowly working on a draft of a letter in the mean time but it probably won't be ready for another week or two. @ME/CFS Skeptic your letter is great and will definitely help speed things up for me. I'm still willing to coordinate getting a final group of co-authors together with @EndME to finalize and submit the letter but it's taking long than I'd initially hoped. If people are eager to get this out the door and want to move forward without my draft I'd understand, just let me know.

@ME/CFS Skeptic I'd feel uncomfortable submitting a letter that has sections written by you without at least acknowledging your contributions, would it be okay with you if we included an acknowledgement section that states something like "We wish to acknowledge intellectual contributions made to this letter by members of the online community Science4ME who wish to remain anonymous at this time."

 

The paper would've been less problematic if it said the subjects failed to exert because of symptoms. It instead surmised that pacing, and therefore the fear of symptoms, is responsible.

So, anticipation of the feelings of discomfort associated with continued exertion, not the discomfort itself, was holding them back, according to the paper. [edited for clarification]

 

Couldn't agree more.

And does it get more dismissive than "discomfort"?

That discomfort could theoretically be immediate or delayed, but given that we get told nothing about the symptoms patients were experiencing during the tasks, and they later say something similar after referring to the PEM after CPET paper by Stussman et al, I reckon you're right.

 

That's a very interesting reply @Robert 1973, because it allows to put the claims of the publication next to yet another "alternative" explanation, and because of its context.

Is that really an answer re. the neglect of PEM in the study? 'It was a criterion, now go look at the Q&A where we tell you that we left it out of our main paper discussing the "deep phenotyping" of ME/CFS.'

In 2016 patient advocates were promised by Walitt that they were "going to try to understand exactly what the biology of post-exertional malaise is." (NIH ME/CFS Advocacy Call | National Institutes of Health (NIH))

Instead the "deep phenotyping" paper spends a lot of energy on "understanding the biology" of Walitt's view of ME/CFS, and PEM is absent.

(At the time Walitt said this he was convinced that ME/CFS was a somatoform disorder characterised by a dysfunction of subjective perception, which he self-termed "interoceptive disorder", although in 2016 he was also trial ballooning "central sensitivity syndrome". He is now the director of his second unit for "interoceptive disorders" while he just launched a paper that - based on feeble substantiation but in line with his views- launches the term "effort preference" as an avoidant choice ME/CFS patients make, while his bio says that his deep phenotyping research focusses on "persons with disorders characterized by aversive symptoms that develop after exposures, such as infection." By which he means ME/CFS, Gulf War Illness and Long Covid.)

 

That's great to hear. I don't think there is a strict time limit here so please take your time.

An acknowledgement like the one you suggest would be great also for other S4ME members who contributed but may not be able to sign. Most of the arguments were already made by others forum members, I just tried to put them together.

EndMe made some good points about 'task completion' not being a good predictor of 'hard task choices' so that's an area that might need some further work.

 

I just want to say thank you to all who have contributed to this thread and particularly to those who are now working on a letter. S4ME at its best.

I agree with @ME/CFS Skeptic. I don’t think there is any rush to submit it, so please pace yourselves and take your time.​

 

I fully expect these to be dismissed, but I felt I had to at least try. I completed the following today:

- I filed a suspected ‘Research Misconduct’ with the ‘Agency Intramural Research Integrity Officer (AIRIO)’ that oversees NIH Intramural research (https://t.co/hRdxdhCtfx). They also have three individuals listed with AIRIO to share intramural-related study concerns at NIH with.

- I wrote today to my Regional Manager in the Office for Civil Rights, U.S. Department of Health and Human Services (HHS) to surface my concerns on the study, and what I feel warrants ‘Research Misconduct’ per HHS guidelines.

- I submitted a complaint to the HHS Office of Research Integrity & submitted an ‘Office for Human Research Protections (OHRP) Research Complaint Form’

 

From AIRIO: “We will assess this matter and get back to you within 30 days.”

 

Sharing from today. Healio: '‘There must be a persistent antigen’: NIH researcher discusses new PI-ME/CFS data’

https://www.healio.com/news/rheumat...gen-nih-researcher-discusses-new-pimecfs-data

 

Wow. I’ve just found this and am digesting it.

Thanks so much for sharing.

FWIW - I’m severe, but have masters in statistics and spent 21 years working with scientists and business teams. My background isn’t medical or research. And I’m rusty on stat details. Happy to collaborate or support as I’m able.

@EndME

 

wow. wowowowowow. Still digesting but so far it seems like this amounts to :

Walitt et al. recklessly* failed to calibrate equipment for EEfRT test, causing misleading results. This failure is tantamount to falsification of results via manipulation of process and equipment.

Y'all's work is causing eustress.

[*edit: At minimum negligence, but with Walitt’s history and other factors- may be able to show recklessness.]


Edited to add:
Has anyone looked into showing recklessness?
Or are you looking to ask for a correction/retraction versus misconduct?

Here's a quote and link, pointing out reckless is somewhere between intentional/knowing and negligent ("honest mistake"):

Unlike knowing and intentional conduct, however, reckless conduct cannot be easily defined with reference to an everyday standard, nor is “recklessly” defined under Part 93. Participants in research misconduct proceedings typically understand recklessness to fall somewhere between, on the one hand, intentional or knowing conduct (both of which constitute research misconduct) and, on the other hand, conduct that is simply negligent. Negligence, like honest error, is regarded as a state of mind devoid of any intention to deceive and is widely understood to differ from behavior meeting the definition of research misconduct (Committee on Science, Engineering, and Public Policy Citation2009). However, a broad range of conduct can fit between “intentional” acts to fabricate, falsify, or plagiarize, and “negligent” conduct that results in fabrication, falsification, or plagiarism. Institutions and their faculty research misconduct committees often struggle to articulate and apply a satisfactory and consistent recklessness standard to allegations of research misconduct.

https://www.tandfonline.com/doi/full/10.1080/08989621.2023.2256650#:~:text=Part 93 states that a,falsification, fabrication, or plagiarism constitutes

 

Do you mind sharing your complaint?

 

I’m reviewing some older comments and this still seems relevant. Given standard process for eefrt is specifically for motivation not fatigue & cautions that fatigue can confound results - falsification by design. & Speaks to intent.

Falsifying by data omission.

Just noting these may be relevant & I don’t recall seeing them in write-up above.

 

At this time it may be best to keep private, as HHS Office of Integrity has already informed me "The Office of Research Integrity (ORI) handles specific allegations of research misconduct involving data fabrication, falsification and plagiarism on US public health service (PHS) funded research according to 42 CFR Part 93 at §93.103. ORI is not able to assist you with your concerns on the overall study design, test used and conclusions that are subjected to differences in opinion."

AIRIO at NIH is doing the internal audit (I don't have high hopes) at this time

 

@ME/CFS Skeptic Apologies if this is stated elsewhere, but do you mind if I ask if there are plans to send to the authors or to Nature Comms (this great write-up that you have formulated)? Fighting through difficult cognitive dysfunction today and trying to decipher