A charter to improve ME/CFS research

[ME/CFS Skeptic]

I recently followed an online webinar about patient involvement in the Dutch ME/CFS program. It looks like the idea is to involve patients in as many aspects of the research project as possible, which is a great initiative. If I recall correctly, ME/CFS patients were already involved in ensuring quality of research in Norway. Patients can also have an influence on research through charities that provide research funding.

With this in mind, I thought it might be useful to write down a few general principles that might help to improve the quality of ME/CFS research without overburdening scientists too much. Here’s what I got thus far, hope others can complement.

1) Pre-registration: publish a protocol
Every scientific study involving ME/CFS participants should be well-thought in advance and publish a protocol that specifies hypotheses, experimental design, power calculation, outcome measures, etc. This prevents p-hacking and cherry-picking of results and makes a research project more transparent towards patients and other researchers. Without a pre-specific protocol, it is impossible to know how many analyses were conducted or whether highlighted findings were selected post-hoc. The Open Science Framework (OSF) makes it easy to pre-register a study protocol.

2) Involve ME/CFS patients in the design, and implementation of your study
ME/CFS researchers should involve patients in the design and implementation of their research. Patients can help point out practical difficulties in trial design such as problems with transportation or ambiguity in questionnaires. In addition, patients have a lived experience that offers a unique perspective on ME/CFS. In the past, research has resulted in dismissive, patronizing, and simplistic theories about ME/CFS that likely would not have taken this form if patients were actively involved in research. Lastly, engaging patients in research also helps to make sure that results are well-disseminated in the ME/CFS patient community.

3) Use recent ME/CFS diagnostic criteria correctly
It is recommended that research uses recent diagnostic criteria for ME/CFS such as the Canadian Consensus Criteria (CCC), the International Consensus Criteria (ICC), or the National Academy of Medicine case definition. When older definitions such as the 1994 Fukuda criteria are used for comparability, it is advised to measure symptoms and characteristics (such as orthostatic intolerance) that allow the percentage of patients meeting recent criteria, to be calculated.

When some aspects of ME/CFS case definitions are not fulfilled, it is recommended to clarify this in the publication title and description. For example, studies that did not include a clinical examination of patients are usually referred to as studying ME/CFS-like symptoms rather than ME/CFS.

4) Use the Common Data Elements for ME/CFS
It is advised that researchers use the Common Data Elements for ME/CFS research. These recommendations provide an overview of the most adequate tools and questionnaires to measure symptoms and quality of life in patients with ME/CFS. They help to make sure that research is harmonized and comparable.

5) Follow relevant reporting guidelines
ME/CFS research should be reported clearly and transparently. ME/CFS publications should follow relevant reporting guidelines to make sure studies can be replicated by other researchers and their findings are easily understood by doctors and patients. The Equator network provides a useful overview of reporting guidelines for many different study types.

6) Focus on effect size estimates and their uncertainty rather than statistical significance
The findings of ME/CFS research are ideally reported in terms of effect size and their uncertainty, rather than statistical significance. The latter is often an arbitrary threshold and findings that are statistically significant can be so small that they are clinically irrelevant. A focus on effect sizes and their precision also help the interpretation of replication studies.

7) Make your findings open-access
The findings of ME/CFS research should be available to patients, doctors, policymakers, and other researchers. If results are not published in an open-access journal or on a pre-print server, it is recommended that authors publish their submitted version on a personal website or academic server such as ResearchGate. Most scientific publishers allow the Author’s Original Manuscript (AOM), the version of a paper before submission and peer review, to be freely distributed.

8) Publish your data, not just the results
Researchers should make the raw data and statistical analyses publicly available in a way that ensures the anonymity and privacy of study participants. Publishing full datasets enables others to check publications for errors or conduct additional analyses. It also ensures a lasting impact of a study because the individual patient data can be used in future meta-analyses. Scientific studies are often expensive and require big efforts from ill patients. Withholding valuable information from these studies blocks scientific progress and should be viewed as lamentable, regardless of current norms on sharing data in the scientific community.

 

[ME/CFS Skeptic]

Another suggestion:

Use adequate controls
It is recommended to not only use healthy controls but patients with other chronic illnesses. Otherwise, it is difficult to determine if reported differences are due to being ill or due to having ME/CFS.

 

[ME/CFS Skeptic]

Use adequate controls
It is recommended to not only use healthy controls but patients with other chronic illnesses. Otherwise, it is difficult to determine if reported differences are due to being ill or due to having ME/CFS.

Some may think that I've missed the main issues in ME/CFS research but I've deliberately focused as little as possible on specifics because these depend on the particular study in question and it is ultimately the researchers who decide how to tackle a scientific problem.

Most of the suggestions listed above are about transparency and being up to date with modern practices. They help to make the most out of research funding without taking much time and effort of researchers. They help to prevent research waste and are therefore in the interest of the ME/CFS patient community in my view.

Most of the issues are about old norms in the research community that are not up to date with the age of the internet. They date back to a time where the space for printing findings was restricted and statistical calculation power was limited. I hope that patient advocates and organizations can help to raise these standards and make sure that ME/CFS research is at the forefront of this evolution.

 

[Peter T]

Some may think that I've missed the main issues in ME/CFS research but I've deliberately focused as little as possible on specifics because these depend on the particular study in question and it is ultimately the researchers who decide how to tackle a scientific problem.

This might be too specific for your intentions, but I would want to reiterate ‘wherever possible use objective outcome measures’, ‘in unblinded trials never rely solely on subjective outcomes’.

 

[Trish]

Is there a way of stating that both researchers and patient groups involved in ME research should not be ones that specifically promote a particular treatment. (I'm thinking of LP, where research is supported by groups that claim success with LP).

Also I think something on outcome measures must include objective measures that are meaningful to patients, and all outcome measures should relate to improvements in core symptoms and/or long term ability to function.

 

[Medfeb]

Good list. Thank you for doing this, @Michiel Tack

Some food for thought...

Consider adding that patients should be involved in the strategy/priority setting before specific projects are selected. That will help drive the overall research agenda to focus on what matters the most to patients. It's too late to drive that once projects start.

On the criteria, IMO, Fukuda should no longer be used at all because it's so non-specific that its hard to trust the research. And I'm skeptical that studies will report enough of the data to be able to make sense of it. If Fukuda is used, it should only be when cohorts are required to meet both Fukuda and one of more of CCC, IOM, or ME-ICC.
We are seeing Long COVID researchers using Fukuda to identify "ME/CFS" cases in their research. This could generate a great deal of uninterpretable garbage.

Maybe think about adding the need to agree to methods for how common data element instruments are applied. The CDE initiatives produced some useful guidance on instruments. But it was not scoped to say what criteria should be used (or even that PEM should be mandatory) or to define how the selected instruments should be applied. For example, what dimensions of SF-36 have to be impaired to satisfy the functional impairment criteria in the various definitions? The CDEs do not say. But one mid-2000s CDC study required impairment on any one dimension, including just emotional function. So patients could satisfy the definition's impairment criteria even without any physical impairment.

The other thought is to recommend the use of study design methods that include severely ill patients and other populations often marginalized in research and health care.

 

[Andy]

Like other points made, this may be too specific, but it might be worth including a note that any patients involved in any way will not be able to work sustainably at the pace of researchers. This was an adjustment that was made to the internal processes of DecodeME in order to facilitate the on going participation of our patient reps.

 

[Ravn]

2) Involve ME/CFS patients in the design, and implementation of your study

In principle I'm strongly in favour of involving patients in all stages of research. There's just one curly question: which patients?

Trish touched on the risk of groups with major COI infiltrating the process. The other risk is well-meaning but naive patients being used for whatever the equivalent term for green-washing is. And there seem to be a lot of patients who are so grateful to anyone doing any vaguely biomedical-looking research into ME they seem to lose their critical thinking ability.

To really make a positive difference you need patients with at least a basic understanding of the scientific method* and who are prepared to be critical grumps (of the cooperative sort). Don't know how to phrase that more neutrally.

*Can't remember the details but a while ago I read somewhere about increasing patient involvement in research (not ME related) and there was a suggestion of patients being given a short intro course into scientific research methods. That could be a way of selecting patients with at least some commitment to engaging properly with scientific questions. Still wouldn't screen out the COI types though.

 

[Hutan]

Thanks @Michiel Tack, this is very helpful. I'd like to see us draft something on the forum. I'd ideally like to see ME/CFS advocacy groups signing up to some international agreement, where they undertake not to support research and researchers not meeting the guidelines.

Some more points off the top of my head:

Trial design should take into account the following characteristics of ME/CFS:

• high rates of recovery and improvement during the first two years
• substantial fluctuations in the severity of symptoms during the day, weekly and longer term, making it difficult for patients to accurately assess minor subjective improvements
• exertion greatly affects symptom levels
• many people with ME/CFS wake later than healthy controls and have low activity levels

Taking these characteristics into account in trials may require:

• careful use of controls (people with ME/CFS; people with chronic health conditions placing similar restrictions on their lives)
• inclusion of objective outcomes in unblinded treatment trials
• assessment periods that involve months rather than weeks
• assessment of symptoms using repeated real time data collection
• standardisation of exertion levels for several days prior to, and during, a measurement event such as a repeated CPET

 

[Kitty]

Some really good ideas here.

I realise it's not always feasible, especially in early exploratory studies, but we need to work towards better-sized cohorts too. With such a heterogeneous condition, can you read anything much from a study of fewer than 20 patients? (Especially when it's a mixed-sex group too, as there may be metabolic differences that we don't yet understand.)

 

[Milo]

Is there a way of stating that both researchers and patient groups involved in ME research should not be ones that specifically promote a particular treatment. (I'm thinking of LP, where research is supported by groups that claim success with LP).

Do you mean conflict of interest and bias? I think research ethics boards usually take care of this kind of bias, like the recent Norwegian LP trial that has failed ethics review.

 

[Milo]

On the criteria, IMO, Fukuda should no longer be used at all because it's so non-specific that its hard to trust the research

Agreed. However tissue and body fluids that have been in freezers for a few decades may only come with a Fukuda criteria- dr Peterson’s patients come to mind, which lead to using this definition when that cohort was used for the Chronic Fatigue Initiative

 

[Milo]

Some really good ideas here.

I realise it's not always feasible, especially in early exploratory studies, but we need to work towards better-sized cohorts too. With such a heterogeneous condition, can you read anything much from a study of fewer than 20 patients? (Especially when it's a mixed-sex group too, as there may be metabolic differences that we don't yet understand.)

Definitely would be nice, however you need to understand that new researchers entering the field do need a ‘proof of concept’ or pilot study, usually with less than 20 patients to get funded for a larger study. The funding is a compounding problem, seed grants are usually barebones.

 

[Obermann]

Thank you, Michiel, for bringing the subject up and thank you for providing the list. I would like to rephrase point 1. All confirmative studies of ME/CFS should be pre-registered, but there must also be room for explorative studies and confirmative studies that are complemented with explorative observations. It must be clear from both the abstract and the conclusions section in a study which observations/conclusions are confirmative and which ones are explorative in nature.

 

[Ravn]

Thanks @Michiel Tack, this is very helpful. I'd like to see us draft something on the forum. I'd ideally like to see ME/CFS advocacy groups signing up to some international agreement, where they undertake not to support research and researchers not meeting the guidelines.

That sounds like a really good project. Patient involvement is starting to become a bit of a thing in medical research so it would be good to get in early on and steer the process in a productive direction so it doesn't become just a box ticking exercise.

 

[Hoopoe]

Most of these items seem to be in response to poor clinical trials of CBT/GET studies.

We need more than just stopping poor CBT/GET studies to move the science forward.

I think we need more emphasis on replication, adequate sample size, adequate controls and on designing studies that can tell us about possible subtypes of ME/CFS. The assumption that it's just one illness could be wrong.

Low funding seems to have a lot do with these problems so hopefully things will improve.

 

[Suffolkres]

I recently followed an online webinar about patient involvement in the Dutch ME/CFS program. It looks like the idea is to involve patients in as many aspects of the research project as possible, which is a great initiative. If I recall correctly, ME/CFS patients were already involved in ensuring quality of research in Norway. Patients can also have an influence on research through charities that provide research funding.

With this in mind, I thought it might be useful to write down a few general principles that might help to improve the quality of ME/CFS research without overburdening scientists too much. Here’s what I got thus far, hope others can complement.

1) Pre-registration: publish a protocol
Every scientific study involving ME/CFS participants should be well-thought in advance and publish a protocol that specifies hypotheses, experimental design, power calculation, outcome measures, etc. This prevents p-hacking and cherry-picking of results and makes a research project more transparent towards patients and other researchers. Without a pre-specific protocol, it is impossible to know how many analyses were conducted or whether highlighted findings were selected post-hoc. The Open Science Framework (OSF) makes it easy to pre-register a study protocol.

2) Involve ME/CFS patients in the design, and implementation of your study
ME/CFS researchers should involve patients in the design and implementation of their research. Patients can help point out practical difficulties in trial design such as problems with transportation or ambiguity in questionnaires. In addition, patients have a lived experience that offers a unique perspective on ME/CFS. In the past, research has resulted in dismissive, patronizing, and simplistic theories about ME/CFS that likely would not have taken this form if patients were actively involved in research. Lastly, engaging patients in research also helps to make sure that results are well-disseminated in the ME/CFS patient community.

3) Use recent ME/CFS diagnostic criteria correctly
It is recommended that research uses recent diagnostic criteria for ME/CFS such as the Canadian Consensus Criteria (CCC), the International Consensus Criteria (ICC), or the National Academy of Medicine case definition. When older definitions such as the 1994 Fukuda criteria are used for comparability, it is advised to measure symptoms and characteristics (such as orthostatic intolerance) that allow the percentage of patients meeting recent criteria, to be calculated.

When some aspects of ME/CFS case definitions are not fulfilled, it is recommended to clarify this in the publication title and description. For example, studies that did not include a clinical examination of patients are usually referred to as studying ME/CFS-like symptoms rather than ME/CFS.

4) Use the Common Data Elements for ME/CFS
It is advised that researchers use the Common Data Elements for ME/CFS research. These recommendations provide an overview of the most adequate tools and questionnaires to measure symptoms and quality of life in patients with ME/CFS. They help to make sure that research is harmonized and comparable.

5) Follow relevant reporting guidelines
ME/CFS research should be reported clearly and transparently. ME/CFS publications should follow relevant reporting guidelines to make sure studies can be replicated by other researchers and their findings are easily understood by doctors and patients. The Equator network provides a useful overview of reporting guidelines for many different study types.

6) Focus on effect size estimates and their uncertainty rather than statistical significance
The findings of ME/CFS research are ideally reported in terms of effect size and their uncertainty, rather than statistical significance. The latter is often an arbitrary threshold and findings that are statistically significant can be so small that they are clinically irrelevant. A focus on effect sizes and their precision also help the interpretation of replication studies.

7) Make your findings open-access
The findings of ME/CFS research should be available to patients, doctors, policymakers, and other researchers. If results are not published in an open-access journal or on a pre-print server, it is recommended that authors publish their submitted version on a personal website or academic server such as ResearchGate. Most scientific publishers allow the Author’s Original Manuscript (AOM), the version of a paper before submission and peer review, to be freely distributed.

8) Publish your data, not just the results
Researchers should make the raw data and statistical analyses publicly available in a way that ensures the anonymity and privacy of study participants. Publishing full datasets enables others to check publications for errors or conduct additional analyses. It also ensures a lasting impact of a study because the individual patient data can be used in future meta-analyses. Scientific studies are often expensive and require big efforts from ill patients. Withholding valuable information from these studies blocks scientific progress and should be viewed as lamentable, regardless of current norms on sharing data in the scientific community.

Have in place agreed Key Performance Indicators (KPIs) and Measure against these during annual required Quality Control Audits of Service Performance which should be made public and published.

 

[ME/CFS Skeptic]

Most of these items seem to be in response to poor clinical trials of CBT/GET studies.

Was not the intention. In that case, I would have focused more on things like objective outcomes and bias due to lack of blinding. It's mostly inspired by developments in the field of psychology following the "replication crisis".

In some way, one could argue that clinical trials are one step ahead of many other medical studies because the idea to pre-register, perform power-calculations are more common.

 

[Hutan]

Most of these items seem to be in response to poor clinical trials of CBT/GET studies.

I don't think they are. I had in mind exercise physiology, cytokine and cortisol studies and trials of supplements and medical devices when I thought of the items I suggested.

Lack of blinding and lack of objective outcomes are issues with a whole range of studies.

 

[Trish]

Do you mean conflict of interest and bias? I think research ethics boards usually take care of this kind of bias, like the recent Norwegian LP trial that has failed ethics review.

True, but the Crawley LP trial happened, and I assume had her tame fan club as her patient/carer group approving, and an ethics committee approving it. There needs to be a stronger system that disallows trials that include persuasion to ignore physical symptoms.