Bias due to a lack of blinding: a discussion

How would they know when researchers HAVE NEVER BOTHERED TO ASK PATIENTS WHICH OUTCOME MEASURES ARE MOST IMPORTANT?

We are the patients, we are the ones telling them their methodology is inadequate. They have NO RIGHT to presume what we think is important.

(caps because I'm feeling angry about this point)

 

Thank you all for your comments on the IQWiG reply (posts starting at #180 )

Apologies for the confusion. I've tried to add some context and order (update: and now also fixed the links.)

1) The section of the IQWiG's methods paper on which the quoted comments were submitted are here. (English translation)

2) The mentioned comments on that section and the reply of the IQWiG are here. (English translation)

3) The German original of the mentioned comments on that section and the reply of the IQWiG are here.


I haven't read through all comments yet but look forward to.

I think it is a very relevant discussion.

The methods paper was published recently, but is is reviewed every 2-3 years.

Perhaps more relevant: the IQWiG's draft report/ "preliminary"report on ME/CFS will be open to consultation later in the year.

What do others think, would it be worthwhile to send a comment on their reply (and cc it to the team working on the ME/CFS report)?

 

I find it very difficult to engage with documents of this sort, written by people who do not understand the material they are supposed to be expert in. If I had to respond I would probably have to say:

Several statements in the response to comments indicate a fundamental lack of understanding of the psychology of delivery of non-pharmacological treatments and how that impacts on patient reports and trial methodology. All I can offer to clarify my reasons is to refer people to my written expert witness testimony to the UK NICE guideline committee for ME/CFS.
https://doctorswith.me/wp-content/uploads/NICE-2021-Expert-Testimony.pdf


(I am not sure whether or not this testimony is still directly accessible through NICE, as it should be.)

 

That's good. But trying to Google it does not easily come up with the URL, despite putting in lots of relevant words.

 

Not quite true. They asked, it's just that the responses they get, symptom burden and disability, don't fit anywhere in their ideology, so they simply ignored it.

Every time I saw something like this, either directly asking or in more free form, it's almost always the same: recovery means no symptoms, the illness is defined by the presence of disabling symptoms, especially the pattern of increased disability following exertion. But since they reject the presence of actual symptoms, merely our perception of, and convinced themselves that PEM is actually the same as being sore and tired after exercise in deconditioned people, then it makes no sense for them since their ideology explicitly rejects what needs to be done.

Same principle behind describing what ME feels like. The most accurate word is sick, we feel sick, ill. That word is not accepted, somehow, so they ask for words that are less accurate, and they settled on one that sounds benign instead and made everything not just about fatigue, but somehow all derived from it, made fatigue a super-symptom, or something like that.

They did ask. They did hear the responses. They rejected reality and substituted their own.

 

Not really able to comment at the moment but I think it's worthwhile to discuss further on these two points in particular.

Another point I'd like to add to the discussion is more general:

In their reply, does the IQWiG just not consider the possibility that there could be a discrepancy between objective and subjective measures in unblinded trials?

Or would they rate the subjective outcomes to be superior to the objective outcomes?

With either reasoning, how can they at the same time justify that in drug trials blinding is needed?

 

This is not encouraging.

Is this only a reaction you experienced when ME was involved, or was it a general attitude towards patients?

if the former, do you also remember colleagues outside the field of ME who reacted differently?

 

Thanks, Jonathan.

Your expert testimony is already included in the submisson's references:

"The problem of bias due to non-blinding of study arms in non-drug intervention studies is mentioned here. Two ways in which these biases can be mitigated are mentioned. However, it does not explicitly call for a combination of the two possibilities. Thus, it is left open how to assess the evidence from studies that could not be blinded and only blinded data collection, but at the same time used exclusively subjective, self-reported, unexamined objective outcomes as primary endpoints.

"This does not adequately address a significant factor for bias in study results. [4], [1]"

[1] Edwards, J, The difficulties of conducting intervention trials for the treatment of myalgic encephalomyelitis/chronic fatigue syndrome: Expert testimony presented to the NICE guideline committee, 06.09.2019, https://www.nice.org.uk/guidance/GID-NG10091/documents/supporting-documentation-3

[4] Tack, M., Tuller, D, Struthers, C, Bias caused by reliance on patient-reported outcome measures in non-blinded randomized trials: an in-depth look at exercise therapy for chronic fatigue syndrome, Fatigue: Biomedicine, Health & Behavior, 2020, DOI: 10.1080/21641846.2020.1848262

And the commenter's proposed amendment:

I think in their reply, the IQWiG didn't really respond to this point of the comment. They did not include it in their 'appreciation', only the other two points (see their reply here ). So this is what perhaps could be criticized?

 

It relates to application of the BPS approach. That is predominantly an issue for unexplained chronic pain and ME but it can spread out into all rheumatological practice for many colleagues.

 

But nobody is likely to have read it.
My testimony is an attempt to explain the blinding/subjective issue to people who do not understand. I worried that it was redundant because everyone already understood, but in reality the problem is the many who do not understand never will. Fortunately, the committee included a number of people who did understand - so maybe it was redundant after all!

 

Another important point to discuss further I think.

 

What strikes me about this is a positive i.e. the availability of objective outcome criteria [EDIT - like data on school attendance] for unblinded studies - similarly, US Army studies (to support veterans) used geographical (movement) data from a phone --- another alterative objective outcome criteria.

 

I only ran across it as a comment in a published bit I was reading for other reasons. It was just nice to see an example of medical researchers pointing out how assumptions about this sort of thing need to be addressed (so that quite eminently treatable problems, with huge QoL impacts, do have an opportunity to be treated). I appreciate these counter examples, gives me hope.
I think this is it: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1774131/ published back in 2004.

 

I don't think that's fair. Some are demanding no methodology at all, we should just trust them. Many literally submitted that as a legitimate argument about the NICE guidelines, they don't need methodology, they know.

There's something like homeopathic thinking behind this, the less methodology the purer the placebo results must be. Or maybe they're just patient-hating charlatans, same difference.

 

Yes and it is strange that we antiscience patients are wanting research into mitochondrial function while the royal collages are defending the Lightning Process.

 

Don't think it was redundant.

Also, still hope I or anyone else will be able someday to a) find people who could (help) translate it into German, b) to get -- with the help of your testimony and the paper by Tack et al -- other patient communities / umbrella organizations interested in the issue and c) for the next update on the IQWiG methods handbook, suggest patient organizations coordinate their submissions and take that issue into account.

If I have been able to learn so many things from discussions on S4ME and still do, even if I am slow and it often takes me a while and many questions, I think others can, too.


Edit: I now remember someone made me aware a while ago that a German translation of your testimony already exists. I just thought it could do with some edits.

 

This paper posted on a thread of its own might be a useful addition to the discussion (not related to ME/CFS) ?