Use of EEfRT in the NIH study: Deep phenotyping of PI-ME/CFS, 2024, Walitt et al

I noticed that 2 weeks ago Walitt et al. have added new files on EEfRT to the code Github repository for the study that were not there before.
https://github.com/docwalitt/Nation...igue-Syndrome-Code-Repository/tree/main/EEfRT

The Readme was also updated and now reads:

 

So they did exclude those with a choice time of 5 seconds. @Nitro802 @andrewkq

I reran the Python GEE analysis I did before and now got the exact same result as theirs: (OR) = 1.652 [1.029, 2.653], p = 0.038. So the minor difference (I previously got an OR of 1.60) was due to those with a choice time of 5 seconds.

 

Their Sas code looks like this:
https://github.com/docwalitt/Nation...ain/EEfRT/EEfRT Hard Task Choice Proc GEE.sas

 

My account request was denied. "Thank you for your interest in mapMECFS. Unfortunately, the scope of our tool is designed to allow researchers access to preliminary results from studies, with the understanding that these results may not yet be replicated elsewhere. As such, our site is not appropriate for your stated goal, and your registration request was not approved. However, you can see a list of datasets on site without logging in and access the original publication. If you have a specific study of interest, we can try to point you to the relevant publication."

The long covid study includes this test. Its the updated version as it lists the max you can earn as $8.60 now. I think they will have a higher sample size than the first test. So we are going to see this again.

 

I feel like they are reading this and directing this at us.

 

Just want to make sure everyone saw Betsy Keller's response in her recent 2 day CPET paper. She addresses ‘effort preference’ and the "not central or peripheral fatigue" claims from an exercise science perspective:

"A recent study of post-infectious ME/CFS (N = 17) and healthy controls (N = 21) assessed a comprehensive panel of physiological, physical, cognitive, biochemical, microbiological, and immunological variables [84]. Of these measures, only 8 ME/CFS and 9 controls completed a single CPET with an average VO2peak about 40% higher in the control group.

Based on this small sample size and inappropriately matched control group, authors suggested that impaired ANS function in ME/CFS, evidenced by diminished HRV, abnormal tilt-related symptoms, and other abnormal orthostatic responses, led to lower metabolic energy production and work output, and may be contributed to by a reduced ‘effort preference’. Effort preference was assessed in this study using the Effort-Expenditure for Rewards Task [85], which utilized a small motor task to assess for anhedonia typically associated with major depressive disorder.

The Effort-Expenditure for Rewards Task is not highly associated with measures of whole-body oxygen consumption or power output compared to conventional indices of effort (%peak HR, RER, RPE), none of which were reported in the study. Whereas the link between ANS dysfunction and impaired energy metabolism is not inconsistent with the systemic CPET data reported herein, their reasoning is misguided.

It has long been known that the magnitude of cardiovascular responses to exertion is predominantly influenced by the relative level of muscle activation (number and intensity of activated muscle fibers) via feedback loop from peripheral interoceptors (e.g., Golgi tendon organs, muscle spindles, etc.) to the motor cortex then to the brainstem [85]. Disruption of this feedback loop at any level, for example, due to infection of the vagus nerve postulated by VanElzakker [86] to emanate from the gut of ME/CFS, would negatively impact this tightly controlled process and downregulate central nervous system signaling of cardiovascular support peripherally for energy production. Consequently, during incremental exercise (i.e., CPET) accumulation of local muscle metabolites from insufficient blood flow coupled with dysregulated central signaling at the brainstem, will directly inhibit the relative level of muscle activation and thereby reduce effort.

Given that both ME/CFS and well-matched CTL in the present study achieved similarly high metrics at peak effort during CPET, we saw no evidence of reduced peak effort in ME/CFS."

https://translational-medicine.biomedcentral.com/articles/10.1186/s12967-024-05410-5

 

Thanks for pointing that out. Does anyone know if that's the first and/or only mention so far of these EEfRT results in a published paper?

 

Jeanette Burmeister has today posted a series of 4 tweets

1) NIH’s Agency Intramural Research Integrity Officer (AIRIO) has begun assessing research-conduct complaints made by ME advocates against certain NIH investigators involved in intramural ME study. This is the 1st in a 4-step process. 1/n

2)Due to the large number of issues raised with the intramural study and the fact that we are several weeks into this process already, the assessment stage will likely take more than the usual 30 days. 2/

3)
2nd step: inquiry
3rd step: investigation
4th step: determination by the Deciding Official whether NIH accepts investigation report’s findings and recommended actions
3/n

4) I feel optimistic that the requirements of the Public Health Service Policies on Research Misconduct (42 CFR Part 93) & NIH Policy will be satisfied for stages 1-3; that investigation report will, consequently, find research misconduct; and that Deciding Official will concur. 4/4

I have not been around for a little while so have not kept up with the hard work of those here who have been examining the study.

Hope so very much that Jeanette's optimism that the Deciding Official will find research misconduct comes to pass.

edit: I am not on twitter but occasionally receive tweets via email.?? I glimpsed a follow up conversation to these 4 points but now cannot access it. Someone on twitter might be able to.

 

@Binkie4 Curious (and hoping) if it will be a different conclusion for Jeannette (or hopefully this previous submission only helps..?)

Pasting from 5/1/24

 

Thanks for trying @Dakota.
I don't know if decisions in the US are as politicised as they have been in the UK.
Am sure we all hope that the large number of issues raised about the intramural will have an effect in the end.

 

Unless I hear about about the effort to send a letter to the journal and can contribute there, I am now going to start working on a powerpoint presentation in hopes of doing a webinar or even making a youtube video. I plan to use the charts and graphs in this thread and hope that's OK.

A wall of text is tough but I think a video / powerpoint is the perfect medium to get the concepts across.

 

@Nitro802 if you do want to do a YouTube and haven't seen previously, have a look at the videos linked in https://www.s4me.info/threads/me-analysis-the-3-pace-videos-factsheet.6106/.

 

Hi all--I'm getting around to writing a post on this after spending too much time recently thinking about FND. In resposne to @Nitro802's question, has a letter been sent at this point? And does anyone know what exactly promoted the NIH's review? Was it Jeannette's piece, or a direct appeal to NIH that forced them to at least make a show of investigating? Was there an NIH announcement, or do we just know this from Jeannette's tweets? Thanks!

 

1) To my knowledge, no letter has been sent to the journal. I would like to help get this over the hump and submitted and have some spoons for this
2) Jeannette had to send a complaint directly to AIRIO at NIH. Dakota already had his dismissed. I think there is a very high bar for the misconduct here. We only know because Jeannette annouced on Twitter, though.
3) Jeannette's call to action at the bottom of part 4 of her series has likely led to more AIRIO and other complaints. Unfortunately I think we have shown there are issues with points 3 and 5. Pasted below:

Call to Action
I urge readers to file complaints by sharing my four-part analysis with the following authorities:

1. your U.S. Senators and Representatives
2. the NIH Director, Dr. Monica Bertagnolli:
   • monica.bertagnolli@nih.gov
3. the Director of Research Integrity and the Agency Intramural Research Integrity Officer (AIRIO), Dr. Kathy Partin (https://oir.nih.gov/sourcebook/ethical-conduct/research-misconduct):
   • kathryn.partin@nih.gov
4. the HHS Office of the Inspector General (OIG). OIG complaints can be filed in a variety of ways as follows:
   • online: https://oig.hhs.gov/fraud/report-fraud/
   • by U.S. mail:
     • U.S. Department of Health and Human Services
       Office of Inspector General
       ATTN: OIG HOTLINE OPERATIONS
       P.O. Box 23489
       Washington, DC 20026
   • by fax: (800) 223-8164

with requests to:

1. investigate the study
   1. with respect to potential gross misconduct and potential research misconduct, including, but not limited to, falsification of data, i.e., manipulating research materials, equipment, or processes, or changing or omitting data or results such that the research is not accurately represented in the research record
   2. constituting a significant departure from accepted practices of the relevant research community and committed intentionally, knowingly, or recklessly,
2. discipline any individuals involved in the foregoing improprieties, if any, and
3. retract the NIH intramural study.

As we cannot know all the individuals who were involved with the EEfRT testing, the complaints should be made against:

1. Dr. Avindra Nath as principal investigator of the study and
2. any other individuals at NIH involved in any the EEfRT testing in any way (including, but not limited to, the design, administration, and interpretation of the EEfRT testing), including, but not limited to, Dr. Brian Walitt and Dr. Nicholas Madian.

Please feel free to use the following summary of some of the potential misconduct, the inclusion of which in your correspondence may expedite the authorities’ determination to investigate the matter. Please make it clear that this is a non-exhaustive list of potential misconduct:

1. The inclusion in the paper of a graph, Figure 3a, in order to support the Effort Preference claim, that was the result of statistical manipulation resulting in a completely misleading presentation of the data, which, when graphed, present a picture that contradicts Figure 3a
2. The claim that ME patients chose fewer hard tasks than controls “at the start of and through the [EEfRT]” with Figure 3a claiming that patients chose fewer hard tasks in every single trial
   1. During the May 2, 2024 NIH Symposium on the study, Dr. Madian stated, “We did again see a difference at baseline, which persisted throughout the task, indicating differences in effort discounting.”
   2. Out of the first four trials, ME patients and controls chose the exact same number of hard tasks per participant. For the very first trial, arguably “the start” of the EEfRT, patients chose twice as many hard trials as controls, even though the patient cohort consisted of one fewer individual than the control cohort.
   3. For 34% of the trials, ME patients chose hard tasks at a higher rate than controls. For another 2% of trials, both groups chose the same percentage of hard tasks. During an additional 14% of tasks, both groups’ hard-task choices were nearly identical, and the difference was, therefore, not statistically significant.
3. The inclusion of randomly assigned tasks (hard versus easy) in the analysis of hard-task choices despite no choice having been made in those cases, which occurred substantially more often in the case of patients than controls and with a substantially higher percentage of the randomly assigned tasks being easy tasks in the case of patients compared to controls
4. The omission of an analysis of the obvious impact of patients using a game optimization strategy and the conclusory claim (without discussion) that there was no resulting group difference in probability sensitivity despite the fact that there was a significant difference between groups for 12% and 50% probability trials but not for 88% probability trials, negating any basis for the Effort Preference claim
5. The use of an improper metric, number/ratio/probability of hard-task-trial choices, in support of the Effort Preference claim, as opposed to the correct metric, the average rewards earned by both groups, for which there was no significant difference between the two groups (less than 1%), negating any basis for the Effort Preference claim
6. The failure to address or even acknowledge significant confounding factors and to attempt to control for them or at least minimize their impact, contrary to other EEfRT studies, for example:
   1. the failure to exclude from the EEfRT patients taking benzodiazepines
   2. the failure to control for patients’ motoric or other physical impairment to complete hard tasks by calibrating the maximum required button-press rates to individual physical ability despite numerous prior EEfRT studies emphasizing the need to do so
   3. the failure, contrary to what prior EEfRT studies have done, to exclude five patients who were physically unable to complete hard tasks at a reasonable rate or at all (the combined hard-task completion for those five patients was less than 16%) leading to a significant group difference in the ability to complete hard tasks (96.43% for controls versus 67.07% for patients), invalidating the EEfRT data and analysis
      • During the May 2, 2024 NIH Symposium of the study, Dr. Madian stated, “What the [original EEfRT] paper describes is that the EEfRT was designed so that the sample of patients used within that original study could consistently complete the task. This does not mean that everyone who takes the task must be able to complete the task without issue for the administration or data to be valid or interpretable. It seems that the creators wanted to ensure that in general as many people as possible would be able to complete the task but without compromising the task’s ability to challenge participants. Furthermore, I think, it bears mentioning that although our ME participants did not complete the task at the same 96-100% rate as the participants in the original study or at the same rate as our healthy controls, they still completed the task a large majority of the time. To wrap things up, to answer the question, consistently completing the task is not a requirement for a valid EEfRT test administration, and by all accounts we believe our data is valid and is, thus, interpretable as a measure of impaired effort discounting.” This is a misrepresentation of the what the original EEfRT study found (required task completion by “all subjects”) and what subsequent EEfRT studies have stressed. Furthermore, it is untrue that patients “completed the task a large majority of the time.”
7. The inappropriate use of a test (the EEfRT), that was designed for and has been exclusively used for mental-health issues (or in healthy individuals), in order to support a novel and newly introduced term and concept, Effort Preference, in a physical disease
8. The failure to discuss the validity of the use of the EEfRT in an unprecented way, i.e., to measure alleged disrupted effort discounting as opposed to the established use of EEfRT results as an assessment of effort-based, reward-based motivation
9. The failure to identify any limitations of the EEfRT testing contrary to what other EEfRT studies have invariably done
10. The failure to exclude the data of four “spontaneously recovered” ME patients (about a quarter of the patient cohort), a recovery rate well above of what has been found by credible researchers, indicating that at least some of those patients were misdiagnosed
11. The over-generalization of the unsupported Effort Preference claim beyond the expending of effort for small gambling rewards, i.e., for any effort exertion by ME patients
12. The over-generalization of the unsupported Effort Preference claim to millions of ME patients worldwide based on the one-time EEfRT performance of 15 ME patients, some of whom seem to have been misdiagnosed
13. The inclusion of data from healthy controls with diseases that have substantial symptom overlap with ME (orthostatic issues in high numbers, Chronic Lyme Disease, and Psoriasis) as well as the inclusions of two blood relatives (siblings) of ME patients in the study despite the fact that there seems to be at least an infectious component to ME
14. The choice of a new and exceedingly prejudicial label for a patient community that has suffered grave harm from decades of misrepresentation of the disease nature and from sustained and relentless defamation, including by NIH
15. The use of three vastly different criteria (two of which are overly broad) for patient selection, including one set of criteria, the IOM definition, that is not a research definition, which likely resulted in including individuals in the patient group who were not ME patients
16. The claim to have established the ME phenotype based on an exploratory, hypothesis-generating study of a cohort of only 17 patients, with many tests run only on even smaller sub-subsets of patients
17. The misrepresentation of the nature of ME by reducing it to mere fatigue, exercise intolerance, malaise, and cognitive complaints, which is a non-specific description that does not capture ME, a multi-system disease with a variety of other disabling symptoms
18. The assigning of a researcher to design the study and run its day-to-day activities, Dr. Brian Walitt, who is on record with his unscientific views of ME, for example, that it is merely a normal way of experiencing life and not a medical entity
19. The hostility, derision, and unprofessional conduct by the principal investigator Dr. Avindra Nath, a high-level civil servant, toward the ME community:
    1. his persistent demands of unqualified praise from the ME community,
    2. his veiled threats as well as his overt intimidation with respect to future ME research in an attempt to silence criticism by advocates of the intramural study and NIH’s research and conduct, causing pain and suffering to ME patients,
    3. his presenting himself as a victim of ME advocates, gravely sick patients, causing reputational harm to the ME and ME advocacy community, and
    4. his relentless stressing that he and the other researchers in the study have allegedly been forced to donate their time and work without compensation for ME patients

 

Ha, multiple 3s and 5s in that list. Which points specifically are you identifying?

 

5. NIH used a column called "Reward is granted" It actually did not mean that, it just meant whether the run of probability was positive. Patients did not earn more "total rewards" than the controls. At no point were players told to maximize their total rewards or even maximize the number of hard trials so personally I don't see this as mattering.

3. The forced choices were not included in NIH's analysis

 

If a person chooses the hard task every time, or the easy task every time, their data is deemed invalid! So for the deliberate strategy you describe, you'd have to choose the hard task every time it makes any sense to, in terms of the size of the potential reward and the probability displayed on the screen, regardless of whether you think you can do it or not.

Any future studies using the EEfRT or other effort tasks will hopefully take account of participants' different abilities so that the hard task would be hard, but doable, for each individual.