Use of EEfRT in the NIH study: Deep phenotyping of PI-ME/CFS, 2024, Walitt et al

There's one tiny typo in an otherwise fantastic letter @ME/CFS Skeptic, "Walit" should be "Walitt".

 

This is fantastic, @ME/CFS Skeptic. I do hope this will find a way to be submitted

 

I have only skimmed the draft, so just a couple of notes on this particular bit:

For anyone who wants to quote any calculations based on data available from mapMECFS in a publication, the following must be cited (according to mapMECFS):

I have amended my original post to include this info as well.

If quoting the particular calculation I did above, then either someone with access to the data should be a co-author and should double-check it (either me or someone else) or include my original post in the references Deep phenotyping of post-infectious myalgic encephalomyelitis/chronic fatigue syndrome, 2024, Walitt et al.

If you/others do want to cite my original post in the references, then it may sound like a contradiction - "severely disabled" in the draft text but in my post the mean places the average patient just on the moderate side of the border of moderate and severe ME/CFS. Two different things - disability and ME/CFS - but it could still jump out to any reader who follows the link. You could get around that by using a word other than "severely".

 

That's a great starting point, thanks for getting the discussion going again! I've had to take a quick break, but am still very much looking into the data and analysing it and deciding what response we can write, @andrewkq is still very much involved in this.

 

My guess is that the authors will probably respond to such a letter with something along the lines of "inability to complete tasks does not predict lower choice of hard tasks for pwME within this trial, indeed those with the lowest completetion rate chose hard most often in pwME, as such we have shown that these results are due to "effort preference" and not due to lower completion rates".

I'm still analysis bits and pieces of the data to see what can be found...

 

Great news! I unfortunately (due to personal circumstances) will not be able to sign or co-sign a letter as I prefer to remain anonymous. So I hope that you, Andrew and others can take this further. Feel free to use or change anything from the draft above (I'm also happy to help out with further research and analysis behind the scenes if needed).

I think I made an error here because Andrew did not add successful completion rate to the model as I initially thought but a new binary category 'hard_task_completer' that refers to having a success rate of hard tasks above 90% (1) or not (0).

Yes good point. Perhaps it's worth a try to add the actual completion rates to the model, not just having a +90% completion rate. Maybe that results in a clearer correlation.

If not, then I think the explanation is that many patients kept trying on the hard task, despite repeated failure (so the opposite of their view that ME/CFS patients have lower effort preferences). The tasks were much harder for ME/CFS patients (resulting in a large difference in completion rate) but the difference in hard task choices was relatively small because patients compensated by being more motivated, more willing to try, more willing to spend effort etc. And that weakened the correlation between the two. That would be my guess.

 

Fantastic letter. I hope it gets published. Can a letter be published as coming from the forum itself? Without names?

 

Before diving deeper into the dynamics, i.e. rhythms and breaks of the game, I had one closer look at the values "highest amount of hard rounds played in a row that were won per player (for the first 35 rounds)" and looked at the relationship between this and "effort preference" i.e. the choice to go hard more often.

Since the differences were rather big between both groups (mean for ME: 2.3, mean for HV:3.6) I thought this would be worthy of at least another look. Since I hadn't posted the values before, the variance for "highest amount of hard rounds played in a row that were won per player (for the first 35 rounds)" is higher in HV (SD is 2.5) than it is in ME (SD is 1.5), which is what you expect since HV H represents a large anomaly (his value is more than double as high as anybody elses) in terms of "highest amount of hard rounds played in a row that were won per player (for the first 35 rounds)".

Given that I had previously discussed that "effort preference" is a bad marker for ME/CFS as it cannot seperate ME from HV (half of ME patients sitting above the HV "effort preference" mean and half of HV sitting below the ME/CFS "effort preference" mean) I looked at whether this was the case again for "highest amount of hard rounds played in a row that were won per player". It seems to provide a bit of a better seperation than "effort preference" does in the sense that now only 1/3 of ME patients are sitting close to the HV mean (4 ME are above the HV mean and one sits just below), however once again a lot of HV behave exactly as ME patients in the sense that 8 HV patients have a highest amount of hard rounds completed in a row number of 2, so half of HV patients are below 2.3 (which isn’t good to separate the 2 clusters), so once again the seperation isn't too strong and naturally one would expect the separation offered by "hard rounds completed in a row number" to be better than just "effort preference" because to some small degree "hard rounds completed in a row number" is a combination of "successful completion percentage on hard tasks" and "effort preference" and we already know that "successful completion percentage on hard tasks" has a large descrepancy between both groups.

Given those results I asked myself whether something like "highest amount of hard rounds won in a row" would be a good predicitor of "effort preference" or whether the opposite was the case, i.e." effort preference" being a good predicitor of "highest amount of hard rounds won in a row". Naturally one would expect a bit of correlation, because it is very reasonable that someone with a higher effort preference is more likely to have won more hard rounds in a row and vice versa.

Overall it seems that in HV “effort preference does somewhat imply max completion in a row number” especially at the highest and lowest numbers (i.e. low effort preference results in low completion number in row and high effort preference results in high completion number in a row). For ME the picture seems a lot more mixed especially around the ME effort preference mean where basically no conclusions can be drawn.

For the reverse direction there doesn’t seem to be a strong correlation between “max completion in a row number implying effort preference” in HV, if an implication does exist it might primarily be driven by someone like HV H, who is an outlier. In ME this max. completion number in a row does not correlate too strongly with effort preference, in particular for people with very low max. completion numbers little can be said about their effort preference which destroys any possible correlations.

Overall I would consider these results to rather be null results than anything else, but this would have to be confirmed by a rigorous statistical analysis.

Next I looked at "effort preference" being a good predicitor of "highest amount of hard rounds won in a row" and vice versa without any labels on participant status (i.e. without looking at the groups ME and HV seperate).

Due to specific averaging effects (for example very low max. succesfully completion numbers in a row didn't correlate well with low effort preference in ME, however in the general population the lowest numbers are all still due to ME patients, but the overall effort preference increases, making this correlation stronger in the overall population) it seems all previous correlations only become stronger when you leave away the labels of patients status.

Overall, I think it could make sense to have a rigorous look at this to see whether separations are created better without labels (HV vs ME/CFS) than with. In this first scenario it is once again an indication and argument that having ME doesn't drive the "effort preference" results, especially because one would expect "effort preference" to correlate with "number of amounts one successfully completes hard in a row".

(I won't be posting what and how I precisely looked at this because it will be far too long for this post and it wasn't a sufficiently rigorous enough analysis, but I think if someone were to make a scatterplot it might be convincing enough that we could have a closer look at this).

 

I wrote to Dr Bertagnolli and Dr Koroshetz expressing my concerns about the NIH study. For what it is worth, here is the reply.

[Moderators, please move or divide it appropriate.]

On 12 Mar 2024, at 17:35, NIH ME-CFS Working Group Questions <NIHME-CFSWorkingGroupQuestions@ninds.nih.gov> wrote:

Dear Mr. Saunders:


Your email to National Institutes of Health (NIH) Director Dr. Monica M. Bertagnolli and Dr. Walter J. Koroshetz, Director of the National Institute of Neurological Disorders and Stroke (NINDS), concerning myalgic encephalomyelitis/chronic fatigue syndrome (ME/CFS) has been forwarded to this office for reply.


We are sorry to learn that you have been affected by ME/CFS for more than 30 years. You expressed several concerns with the recently published paper, “Deep phenotyping of post-infectious myalgic encephalomyelitis/chronic fatigue syndrome.” Your desire that these concerns be addressed – and that research in this field receives the attention it deserves – is certainly understandable.


The following website was recently established to provide information about the NIH Intramural Study on ME/CFS: https://www.nih.gov/mecfs/nih-intramural-mecfs-study. This portion of the site provides helpful answers to frequently asked questions about the study:https://www.nih.gov/mecfs/nih-intramural-mecfs-study-qa.


You noted that the study had a low number of participants and that it appears to have neglected studying post-exertional malaise (PEM). PEM was a criterion for participation in the study. The Q&A link mentioned above provides information on PEM-related findings. The study was designed to examine a small group of well-defined patients that could be studied in depth. Patients were at NIH for a few weeks which limited the number of people that could be seen by researchers. When COVID-19 complicated the ability to bring in new patients, researchers determined that a sufficient number of patients had been studied and moved on to analyzing the data.


You also expressed concern that the use of the term “effort preference” would give the impression that people with ME/CFS need to “try harder” and that it would further stigmatize the disease. We understand your concerns and would like to explain this term and how it is used in the context of the Nature Communications article. In our day-to-day lives, the brain must decide how to expend its focus and energy on the tasks that are presented to it. The expenditure of energy is interpreted by the brain as effort. Not all tasks require the same amount of effort, with some tasks being easier and others harder. Tasks are also not equally valuable, with some tasks having more reward and others less. Effort preference is a measurement of the decisions the brain makes of how to utilize its energy based on difficulty and value of a task. We are often not aware that these processes are happening.


In this study, a series of tasks were given in which people with post-infectious (PI) ME/CFS and healthy volunteers had to choose between doing an easy or hard pushing task. The tasks were repeated many times, with different reward values assigned for successful completion. Persons with PI-ME/CFS were more likely to choose the easy task over the hard task compared to the healthy volunteers. This difference in task choice was not influenced by the number of tasks they performed or the value of the tasks. All the factors that did influence the choices are not known, and, as is the usual case, many are not conscious.


The full biological explanation for the observed difference in effort preference in PI-ME/CFS is not known. However, in the study, effort preference was related to differences in the brain, including decrease in catechol neurotransmitters and decreased activation of the right temporo-parietal junction, which plays a role in evaluating and initiating physical action. Therefore, the term does not describe a psychological difference in people with ME/CFS and people without ME/CFS; it describes an actual abnormal finding in how the body’s neural system is supposed to function. Further analysis of the relationship between effort preference and these biological observations is planned.


You also wrote that you hope NIH will invest in high-quality research to develop diagnostic tests and effective treatments. As you may know, NIH research is investigator-initiated. However, we would like to take this opportunity to provide information about the ways that NIH strives to increase quality research on this disease. In 2015, then-NIH Director Dr. Francis S. Collins announced renewed actions to advance ME/CFS research and asked NINDS and the National Institute of Allergy and Infectious Diseases to take the lead. Since then, 20 NIH Institutes, Centers, and Offices have coordinated ME/CFS research through the Trans-NIH ME/CFS Working Group. NIH and the Centers for Disease Control and Prevention also lead an ME/CFS Interagency Collaborative to foster interagency and stakeholder collaboration and communication. You can sign up for the Trans-NIH ME/CFS Working Group listserv by visiting the website (https://www.nih.gov/mecfs) and clicking on the “Join our listserv” icon on the right side of the page. The site also provides information about upcoming events and offers recordings of past events (https://www.nih.gov/mecfs/events).


NINDS also launched an effort to develop a Research Roadmap for ME/CFS, which will identify research priorities to move the field toward translational studies and clinical trials. The Research Roadmap site is at https://www.ninds.nih.gov/about-nin...l/nandsc-mecfs-research-roadmap-working-group. Eight Research Roadmap webinars were completed between August 2023 and January 2024. During this time, the Research Roadmap Working Group developed research priorities for each of the chosen webinar topic areas, as well as cross-cutting research priorities. These research priorities are available on IdeaScale for comment from the community:https://ninds.ideascalegov.com/c/campaigns/1286/about. The recommendations will be reported at the NINDS Advisory Council meeting on May 15 during the open session. The Trans-NIH ME/CFS Working Group is already discussing next steps and how to move the research forward after the report is approved by the Council. They are also discussing how NIH can partner with patient advocacy groups to support research in the future. Research priorities will also be the focus of the next ME/CFS conference in the United Kingdom (organized by Invest in ME) with the idea that we will work collaboratively with the European ME/CFS Research Network to address research priorities going forward. The Roadmap was recently discussed during the Trans-NIH ME/CFS Working Group Advocacy Call on March 4th. A recording and transcript will be available soon athttps://www.nih.gov/mecfs/events.


NIH supported three ME/CFS Collaborative Research Centers at Columbia, Cornell, and Jackson Laboratories as well as a Data Management Coordinating Center (DMCC) at RTI International for 5 years and issued RFAs to support Centers and a DMCC for another 5 years. Information and updates about the ongoing ME/CFS Centers can be found here: https://mecfs.rti.org/. Webinars from each of the Centers and the DMCC provide overviews of their research:https://www.youtube.com/playlist?list=PLJY2toEjWadPflbHSNYyozKcOlC4HzIcQ. In addition, the Myalgic Encephalomyelitis/Chronic Fatigue Syndrome Research Network (MECFSnet) has published two newsletters featuring research news and new and ongoing activities at NIH related to ME/CFS. The newsletters are available at https://mecfs.rti.org/newsletter/.


On December 11, NIH hosted the “Symposium For Promoting The Advancement Of Research Knowledge In ME/CFS” (SPARK ME,https://event.roseliassociates.com/me-cfs-symposium/), a workshop for young/early career investigators. This meeting had 100 individuals participating either in person or via Zoom, including high school students, undergraduate and graduate students, postdoctoral fellows, and early career faculty (assistant and research assistant professors). Participants were from the United States, United Kingdom, Norway, South Africa, Spain, Germany, and Australia, and NIH provided travel awards to 20 individuals. Participants shared oral presentations and posters throughout the workshop and the 2-day NIH conference that followed. There are plans for ongoing communication, collaboration, and networking of the participants and partnering with the Young European ME Research Group (EMERG) in the United Kingdom.


Finally, on December 12-13, NIH hosted a conference called “Advancing ME/CFS Research: Identifying Targets for Intervention and Learning from Long COVID.” More than 1,000 people registered to participate with around 150 in-person participants. Approximately 500 participants joined via the NIH webcast for each session. The presentations provided updates on the state of ME/CFS research with a look toward moving research to translation and clinical trials. Several people with lived experience spoke during the conference and provided very poignant talks about their experiences having ME/CFS and how it has impacted their lives. You can view the videocasts at https://videocast.nih.gov/watch=52631 and https://videocast.nih.gov/watch=52738.


We hope this information is helpful.


Office of Neuroscience Communications and Engagement

National Institute of Neurological Disorders and Stroke

on behalf of the Trans-NIH ME/CFS Working Group


​

 

Completion rates of hard tasks alone (without being in any model) very clearly offers a better separation in terms of ME vs HV than the effort preference means do (there's only 4 HVs that don't have a hard task completion rate of 100%, whilst 10 ME patients have a hard task completion rate of below 100% and these percentages do scale somewhat, so if you look at completion rate below 60% you get a "marker" with 100% specificity but only 40% sensitivity for ME), however that's primarily due to the fact that effort preference provides essentially no separation of the groups (half of ME patients have an effort preference around the HV mean), rather than hard tasks completion rates telling us much (after all there's sufficiently many pwME who have seemingly no problem completing hard tasks). However, it's rather hard to see clear correlations in the data, which is precisely why the authors decided to construct an arbitrary argument around one barely statistically significant measurement within a pool of hundreds of different measurements.

In the paper "effort preference" purely refers to percentage of hard tasks chosen (rather than choosing hard despite failure, even though that of course makes little sense overall and also doesn't agree with their press releases where Walitt has said “Rather than physical exhaustion or a lack of motivation, fatigue may arise from a mismatch between what someone thinks they can achieve and what their bodies perform”, because clearly there isn't a mismatch, as they can perform less and choose hard less, but I think we have to stick to what the paper says rather than what was said in press releases). Currently I think it might be hard to construct arguments around the other explanations you proposed because thus far we haven't seen much data to strengthen these ideas, because all the different differences between ME and HV seems to be driven by outliers rather than anything consistent, which is why the results are so "wishy-washy" to begin with. For example I find it very hard to conclude anything from this data about the motivation of participants and that is also what I would expect with data from a seemingly unrobust experiment where outliers plays a large role when the sample size is this small.

I fear any hypothetical (rather than evidence based) argument on why "effort preference" might be occuring will only strengthen their position and reinforce the position that it's significant and is happening, when in fact there is almost no evidence to suggest it plays any role to begin with, it offers the same separation a coin flip does and we might be able to come up with data that shows much stronger correlations than what their argument was based around or that their conclusions aren't consistent with the literature or if we point to other inconsistencies.

 

Some more data on the correlation between hard task choices and successful completion rates.


I found a value of r= 0.12, that is not significant with this sample size.



Patient D, H and A constitute the 3 dots at the bottom of the graph with low successful completion rates and a large proportion of hard trials. They are the patients that had the many high red bars and kept trying to complete hard tasks despite repeated failure.

 

Something that we haven't looked at yet (mainly because it's harder to do) is that thus far we've only looked at things such as percentage of total hard tasks completed vs percentage of hard tasks chosen, however that doesn't really reflect the nature of the game very well.

I believe a more meaningful analysis would have to look at what role a percentage of previously successfully completed hard trials plays in choosing a hard trial in the next round and so forth, i.e. we've only looked at the overall successful completion percentages on hard tasks once the game is over and how this relates to different things (proportion of hard tasks, maximum amount of successful trials in a row, etc.) however the game is played round by round, so it's more meaningful to see what a round by round or Markovian type assessment yields, rather than assuming that participants know their end-game successful completion rate on hard tasks, which is knowledge they don’t have. For such an analysis it might make sense to me to include the percentages from the practice rounds and using some discounting factor for previous rounds or some Markovian ideas as well as using a cut-off at round 35.

For someone with a 100% completion rate (which is more than half of the participants) or someone with a 0% completion rate (PI-ME/CFS H) this kind of analysis will make 0 difference, but it may make a difference for some inbetweeners.

However, given that the all results of the study are largely driven by the groups of people who have either 100% completion or extremely low completion rates, I expect the results of what had already been discussed on the forum to not change, but it might be worth the effort to have a look at this (perhaps it does push a barely significant value into insignificance or vice versa), or at least it might make for a more rigorous analysis.

 

I think you'll want to look at the successful completion rates on hard tasks, rather than all tasks, as the easy tasks anyways have a 100% completion rate for everybody (it obviously doesn't make any difference for the analysis above but it looks a bit cleaner to the eye) and possibly also use a cut-off at 35 rounds (w/o practice rounds), at least that's what I found most reasonable.

However, even then you will also find that the correlation is not significant (there's lots of data in this thread on that this is the case and why this is the case) but essentially ME/CFS patients D, H and HV A will destroy any more obvious correlations w.r.t. to this (and similarly there's always outliers w.r.t to other things we've looked at like SF36, max. number of hard trials in a row, etc).

 

Isn't this the key rebuttal? As in, if the effort preference of a good chunk of ME patients is no different to the healthies (and in many cases exceeds them), then it can't be a central, defining feature of ME.

And if Walitt wants to argue that low effort preference eventually leads to deconditioning, then why haven't the healthy volunteers with low effort preference become deconditioned and developed an ME-like syndrome?

I think there also has to be some question about how robust the test is to confounding factors. Does it really capture the rarefied and intangible concept of effort preference, or is it just a proxy for feeling lousy and under-slept, of having travelled long distances, of being overwhelmed by all the testing, of knowing that the future of ME/CFS research might rest on which button you press in a parlour game that you only half understand.

 

I think it would be one part of the rebuttal, but only one part of the many parts different brilliant users have already discussed here. One shouldn't forget that the authors don't claim any seperation or anything of that sort, they simply run the GEE analysis (which is a lot more involved than something like a t-test and makes a more intricate analysis possible).

They don't argue this (at least not in the paper), so there's no contradiction there. They more just put their spin on their findings, for Walitt that seems to mean that some BPS notion is causing "effort preference" whilst for Nath it seems to mean that some non-identifiable antigen causes some brain dysfunction that results in a different "effort preference".

 

Thank you to everybody looking in detail at the paper. The statistics and other mathematical details are beyond me. Only able to skim anyway so apologies for not being able to comment on that.

Others' comments look like a very impressive analysis so far -- just am not able to follow.

I still wonder how the investigators can both acknowledge that PEM is a cardinal symptom of ME/CFS and leave it completely aside in the investigations on which they report in what seems to be their main paper?

So NIH investigators published two papers on PEM elewehere and refer to these in their Nature Comms paper but only to reassure people that the inclusion criteria demanded PEM -- whereas the very paper that presents their main findings/ main hypothesis is about investigations that didn't take into account PEM in any way?


Edit to add and again edited to fix links to forum discussion on the two PEM papers:

Focus group paper here.

Paper on how they (tried to) measure PEM here.

 

In the paper they write: "Both the autonomic and central motor dysfunction result in a reduction in physical activity. With time, the reduction in physical activity leads to muscular and cardiovascular deconditioning, and functional disability."

It's the trailing "and functional disability" in the sentence that troubles me, as though there is no disability prior to the deconditioning, and that the disability is "functional" (whatever that means). Had they written "and further disability" it would have removed some of the ambiguity. But why include it in the first place? It must occur in other energy or mobility limiting condition like Parkinson's or MS, yet it surely wouldn't merit inclusion in deep phenotyping studies of those conditions?

 

That indeed seems like a very fair point to me and I stand corrected. It's hard for me to understand what they really mean with these sentences because it just seems to be about some made up TPJ story and drawing arbitrary connections which are not supported by any data. I'm also not sure what "with time" is supposed to mean. Are they not aware that many patients experience a disability that has an extremely sudden onset (in particular in the post-viral cohort they decided to study), rather than anything progressive?

I think the argument that they can make is that, that final abstract is anyways just hypothesis rather than being evidence based, so they feel ok with writing whatever they want.

I think the conclusions they try to draw between the different parts of the study and the EEfRT don't seem to not be based on the available evidence and are equally questionable to everything that has been discussed here, but we haven't really gotten to discussing that yet, since discussing the EEfRT alone already lead to this overly lengthy thread.

 

How can you claim to take PEM seriously, and then come out saying the Hallmark of the illness is that patients don't want to do things and propose some brain imbalance is causing this behavior.

If you believed PEM was real you would naturally just say this is a normal adaptation to the illness. Anyone supporting this view is a closeted BPSer.