Bias due to a lack of blinding: a discussion

This statement is preposterous. How does learning to rate your symptoms as less provide any genuine relief?

Basically they are saying that if you can brainwash patients to say what you want them to say that is all that matters. This isn't even based on the idea that if the patient believes they are better they are. It is based on the idea that as long as they feel they should tell the therapist they are better they are.

The lack of understanding of simple psychology is deeply disturbing.

 

Are they confusing (perhaps deliberately) quality of life with the scores of quality of life questionnaires? Which patients told them that learning to 'to rate his or her own symptoms as less severe or threatening' is of utmost importance to them?

 

“If a person learns to rate his or her own symptoms as less severe or threatening, then this can be seen as a genuine relief”
Wow.

On the other hand, I found an article about anaemia in people with IBD in which they realised that the idea of symptomless anaemia from slow onset was actually when people adapted their lives so much and so gradually that they rated their much impaired quality of life as normal. …until they got treatment for the anaemia (just iron supplementation) when they actually improved not just their perception but their actual participation in activities of life. Objective outcomes matter.

ETA: whoops, looks like my reply missed that there were others who’d said the same thing.

 

But an interesting new example.

Should we move this part of the discussion to the thread on interventions that manipulate how patients report symptoms as a separate form of bias ?

On the other hand, the question why reporting an improvement of symptoms does not necessarily mean an actual improvement (even if treatment assessors are blinded) I think also belongs to the basics of trial methodology?

Edited to remove redundant part.

 

But surely this is not what the claim refers to.
If tinnitus is less intrusive it is less intrusive. There is no need to learn to rate it as such. It just is. Tinnitus is an error of acoustic nerve feedback. Distraction, and the best form is hearing aids, abolishes the tinnitus, it doesn't just make it less intrusive. I used to have tinnitus. With my aids I no longer have tinnitus whether I wear them or not.

If you learn to cope with a symptom you should still rate the symptom at the same level unless you are being asked questions about coping with it, surely.

The statement explicitly says 'learns to rate'. It is absolutely clear that the writer thinks the objective is to persuade the patient to say something untrue that makes it look as if they are better. That might seem incredible if it were not for the fact that of course this is absolutely what the intention of the BPS people is. They consider making people better to equate to conning them into learning to say they are better. The cynicism is quite overt when I talk to my BPS colleagues about this. It becomes covert whenever a patient is present. What has happened here is that the covert cynicism has been allowed to come through in a publicly available policy document. And unfortunately for the writer it has come through in a form that makes the hypocrisy explicit.

 

Could you quote the comment and reply by the IQWIG in German?

 

I thought I'll provide the context of the IQWiG's reply to the said comments on their methods paper:

The comments quoted in the other posts refer to the section 3.4 'Nicht medikamentöse therapeutische Interventionen'/ Non-drug interventions. The part the comments and proposed changes by the commenter refer to are in bold.


Form the IQWiG's draft of the Methods Paper:

3.4 'Nicht medikamentöse therapeutische Interventionen'/ Non-drug interventions:

[...]

"Compared with drug studies, studies in the non-drug field are often associated with special challenges and difficulties [502], but these are usually at least partially surmountable:

"Often, blinding of the personnel performing the intervention will be impossible and blinding of the patients will be difficult or also impossible. Nevertheless, a reduction in the power of a study can be mitigated by blinded endpoint ascertainment or by the use of objective endpoints.

# The skills and abilities of the users are usually essential for the benefit of a non-drug intervention that can be seen in a study:

Users can be both physicians and patients. Ideally, adequate selection and training will minimize the negative impact of learning curve effects on the outcome of the study.

# Many non-drug interventions change over the course of a study:

"Numerous small changes (often referred to as "step innovations") are common, especially for common with medical devices. Changes in the intervention during the course of a study usually do not require a new study, but can be evaluated by appropriate statistical methods. methods to investigate their influence on the study results with sufficient precision.

"In addition, non-drug interventions are sometimes more or less complex interventions [149]. In order to do justice to the aforementioned aspects, qualitatively particularly good studies are needed to achieve sufficient certainty of results. Paradoxically, the opposite has tended to be the case in the past, i.e., meaningful randomized trials are often lacking, especially in the non-pharmacologic field (e.g., in surgery [502]).

"In order to be able to make any statements at all about the significance of a particular non-drug therapeutic intervention, it may therefore be necessary to include non-randomized studies in the evaluation. However, quality requirements must also be placed on these, especially with regard to measures to ensure structural equality. As a rule, however, such studies will at best be able to provide only indications of (additional) benefit or harm due to their inherently lower certainty of results.

"The inclusion of studies of a lower level of evidence is consistent with the corresponding regulation in Chapter 2, Section 13, Paragraph 2 of the G-BA's Rules of Procedure [272], where, however, a special obligation to justify the inclusion is emphasized. There it is stated: "However, the recognition of the medical benefit of a method on the basis of documents of a lower level of evidence requires - also taking into account the respective medical necessity - all the more justification for the protection of patients, the further the deviation from evidence level 1. For this purpose, the potential benefit of a method must be weighed in particular against the risks of its use in patients, which are associated with evidence of efficacy of lower significance" [272].

"In other words, the mere unavailability of studies with the highest level of evidence cannot generally be considered sufficient justification for a benefit assessment based on studies with lower levels of evidence."

Source: Draft of the IQWIG's General Methods Paper,
https://www.iqwig.de/methoden/09-03-18_entwurf_methoden_kosten-nutzen-bewert.pdf

 

The German "bewerten" means both rate/ assess/ evaluate and 'conceive'.

But I think the gist of comment (3) still applies (see post above this post ).

 

That sounds interesting. Can you share a link?

This seems to fit with what I've observed in other people who are on the milder end of illness: they get used to feeling that way, build a life that is compatible with their limitations and then begin thinking that they're "almost normal", or at least not truly ill.

 

Good questions. And, as an addendum to my post #187, which might seem not quite to fit with what the BPS view is supposed to be, a further thought.

I didn't go in to psychiatry, not because the patients did not desperately need help, but because I got the impression that the health professionals were barmy - and more specifically that they were play-acting.

I think many people who go into 'psychological medicine' are play-actors and, more than that, they are people who see no valid distinction between reality and play-acting. One of the things that seems to me to have changed most in the last twenty years is that this denial of a valid distinction - the acceptance of alternative facts - has become the zeitgeist.

So, although there appear to be incompatible accounts of what BPS people believe or deliberately confuse, I think deeper down for them there is no fact of the matter, no reality; all that matters is playing a game. Medicine is getting patients to play your game, but not to let them have the ball quite as much as you. (Worth remembering that originally all medicine was this.)

Then there is a stratification between egoists and altruists. I think this confuses people a lot in politics. There are play-acting egoists but also play-acting altruists. There are realist egoists and realist altruists. The classic BPS pundit is a play-acting egoist. There are also many who join the BPS fold who are by nature play-acting altruists. Physio is probably mostly play-acting altruism.

But of course the play-acting normally involves giving the appearance of not play-acting - the whole point. What is unusual about the German document is that the mask slipped. But then we seem to live in a world in which clown's masks slip all the time and we are supposed not to notice.

 

To understand the background of the IQWiG's reply, I'll also post the second point of the comment which unfortunately might have facilitated confusion.

Will add original text in German in following posts.

From a submission to the IQWiG's draft of the Methods Paper:

Comment (point 2) on section 3.4 (Non-drug interventions)

Comment:

The actual testability of endpoints for objectivity should be made explicit. As a negative example, we refer to a large study on the benefit of alternative medical therapy in children and adolescents that used school attendance as an endpoint. However, the data on school attendance were self-reported by the study participants alone and were not verified, even in random samples.[22]

School attendance was eventually included as a secondary endpoint, rather than as a primary endpoint, as one of further deviations from the procedures specified in the study protocol. Only in a corrected version of the study article published in response to repeated criticism did it mention that data on school attendance were exclusively self-reported and not verified against objective data, again contrary to what was stated in the study protocol. [23] [24] [25]

Previously, experts invited by the Science Media Centre (London) to comment on this study had failed to recognize this issue and had erroneously referred to school attendance as objective endpoints in their comments.[26], [27]

Proposed change:

In addition, when using objective endpoints that measure an effect of symptoms, such as physical/cognitive activity, motor function, school attendance, ability to work, etc., care should be taken to ensure that these are actually collected objectively and not solely through self-reported data from the study participants.

Edit: Comment (point 1) and Comment (point 3) see this post.

 

Sorry @MSEsperanza, but I am now confused as to who is making these comments in response to what?
I have lost the plot.

 

Yes. Thanks @Hutan

The comments were submitted by a private person to the IQWiG's draft of their General Methods paper.

Each point of the quoted comment also includes a proposed amendment / change.

https://www.iqwig.de/en/participation/submitting-comments/

https://www.iqwig.de/en/about-us/methods/methods-paper/

The IQWiG now published the Methods Paper in January 2022:

https://www.iqwig.de/methoden/allgemeine-methoden-v6-1.pdf

And the submissions /comments were published a bit later together with the replies from IQWiG (documentation and 'validation' evaluation of the comments):

Documentation and evaluation of comments on the Draft of the General Methods 6.1

Will edit my posts to reduce confusion.

(Edited for clarity.)

 

But that is not what the German text says.
Moreover, my discussions with old colleagues about this when I presented the 'Horrors of PACE' some years back, and also chit-chat over the years, makes it clear to me that cynicism and hypocrisy spill over any half-coherent theorising. The shift of the head or overt wink is hard to suppress. A bit like people who only talk out of one side of the mouth.

The patients are accused of dissembling precisely because dissembling is such an ingrained practice for the professional.

 

This will take a while. Apologies.

I will ask my health insurance if they could provide a personal assistant for my forum activity.

Edit:

 

From the submission to the IQWiG's draft of their General Methods Paper (in German) (*)

(Each comment includes a proposed amendment.)

Comments on section 3.4 'Nicht medikamentöse therapeutische Interventionen'/ Non-drug interventions:

Comment (point 1)

Anmerkung:

Hier wird die Problematik der Verzerrungen durch Nichtverblindung der Studienarme in nichtmedikamentösen Interventions-Studien genannt. Es werden zwei Möglichkeiten genannt, wie diese Verzerrungen abgemildert können. Es wird jedoch nicht explizit eine Kombination beider Möglichkeiten gefordert. Damit wird offen gelassen, wie die Evidenz aus Studien bewertet werden kann, die nicht verblindet durchgeführt werden konnten und lediglich die Datenerhebung verblinden, zugleich jedoch als primäre Endpunkte ausschließlich subjektive respektive selbstberichtete, nicht überprüfte objektive Endpunkte verwenden.

Dadurch wird ein erheblicher Faktor für eine Verzerrung der Studienergebnisse nicht angemessen berücksichtigt. [4], [1]

Vorgeschlagene Änderung:

 Oft wird die Verblindung des die Intervention ausführenden Personals unmöglich und die der Patientinnen und Patienten schwierig oder ebenfalls unmöglich sein.

Um in diesem Fall dennoch aussagekräftige Ergebnisse zu erzielen, ist es in solchen Studien erforderlich, zusätzlich zu einer verblindeten Endpunkterfassung robuste objektive Endpunkte zu verwenden.

Generell ist bei der Bewertung des Designs randomisierter klinischer Studien sowie auch der Übersichtsarbeiten zu nichtmedikamantösen Interventionen zu beachten, dass aus der Nichtverblindung nicht nur eine Zurückstufung auf Grund der Verzerrung durch die Erwartungen sowohl der Behandler als auch der Patienten erfolgt, sondern auch geprüft wird, inwieweit diese Verzerrung durch weitere Methoden kontrolliert wird, insbesondere durch die Verwendung primärer objektiver Endpunkte. Werden bei einer nichtverblindeten Studie als primäre Endpunkte allein subjektive Messungen resp. selbstberichtete Daten verwendet, wird die Evidenz für den Nutzen der untersuchten Intervention als unzuverlässig bewertet.

Comment (point 2)

Anmerkung:

Die tatsächliche Überprüfbarkeit von Endpunkten auf ihre Objektivität ist explizit zu machen.[...]

Vorgeschlagene Änderung:

Bei der Verwendung objektiver Endpunkte, die eine Auswirkung von Symptomen messen, etwa physische/ kognitive Aktivität, Motorik, Schulbesuch, Arbeitsfähigkeit etc., ist zudem darauf zu achten, dass diese tatsächlich objektiv und nicht allein durch von den Studienteilnehmer:innen selbst berichtete Daten erhoben werden.



Comment (point 3)

Anmerkung:

Bestimmte psychotherapeutischen Verfahren zielen u.a. darauf, Symptome zu ignorieren oder als weniger schwer zu bewerten, auch bei Krankheiten, bei denen kein Nachweis dafür besteht, dass eine Überbewertung von Symptomen zum Krankheitsgeschehen beiträgt. Bei entsprechenden Verfahren gilt es, diese Ausrichtung als zusätzlichen Faktor für eine Verzerrung der von Studienteilnehmer:innen berichteten Messpunkte in Betracht zu ziehen.[28]


Vorgeschlagene Änderung:

Bestimmte psychotherapeutischen Verfahren zielen u.a. darauf, Symptome zu ignorieren oder als weniger schwer zu bewerten. Bei entsprechenden Verfahren gilt es, diese Ausrichtung als zusätzlichen Faktor für eine Verzerrung der von Studienteilnehmer:innen berichteten Messpunkte in Betracht zu ziehen.


IQWiG's reply to all three points ('Würdigiung'): (p. 9-10)

Weiter wird in einer der Stellungnahmen angeraten, beim Umgang mit Studienendpunkten
darauf „zu achten, dass diese tatsächlich objektiv und nicht allein durch von den Studien-
teilnehmer:innen selbst berichtete Daten erhoben werden“. Weiter wird in der Stellungnahme
davor gewarnt, dass eine Verzerrung von Studienergebnissen drohe, wenn eine der Studien-
interventionen darauf abziele, „Symptome zu ignorieren oder als weniger schwer zu bewerten“,
die Symptome selbst aber nicht beeinflusse. Dieses Problem komme beispielsweise in der
Bewertung psychotherapeutischer Verfahren vor.

Sicherlich ist es wünschenswert, wenn Endpunkte, die sich objektiv erfassen lassen, auch so
erfasst werden, weil dies in aller Regel die Reliabilität und Validität der Datenerfassung erhöht
(z. B. durch Verblindung). Umgekehrt jedoch sind patientenberichtete Endpunkte, wie z. B.
Schmerz oder Lebensqualität, von höchster Bedeutung für Patientinnen und Patienten und
damit auch für die Nutzenbewertung, obwohl sie sich naturgemäß nur subjektiv erheben lassen.

Dass viele Symptome nur subjektiv erfasst werden können, ist auch deshalb kein Nachteil, weil
letztlich allein die Patientin oder der Patient selbst den Erfolg der eigenen Behandlung bewerten
kann. Wenn eine Person es lernt, die eigenen Symptome als weniger schwer oder bedrohlich
zu bewerten, dann kann dies als eine echte Linderung gewertet werden, da auch hier allein die
subjektive Patientenperspektive zählt. Insgesamt wird daher in diesem Punkt kein Änderungs-
bedarf am Methodenpapier gesehen.


(*) Source: Documentation and evaluation of comments on the Draft of the General Methods 6.1 (in German), https://www.iqwig.de/methoden/allgemeine-methoden_dwa-entwurf-fuer-version-6-1_v1-0.pdf


Edit: fixed muddle from copying texts in the wrong order.

 

Post above translated into English by deepl :

From the IQWiG's draft of their General Methods paper:

Section 3.4 'Nicht medikamentöse therapeutische Interventionen'/ Non-drug interventions:

"[...] Compared with drug studies, studies in the non-drug field are often associated with special challenges and difficulties [502], but these are usually at least partially surmountable:

"Often, blinding of the personnel performing the intervention will be impossible and blinding of the patients will be difficult or also impossible. Nevertheless, a reduction in the power of a study can be mitigated by blinded endpoint ascertainment or by the use of objective endpoints.[...]"

From the submission to the IQWiG's draft of their General Methods Paper

(Each comment includes a proposed amendment.)

Comments on the above quoted point in section 3.4 'Nicht medikamentöse therapeutische Interventionen'/ Non-drug interventions:

Comment (point 1)

Annotation:

The problem of bias due to non-blinding of study arms in non-drug intervention studies is mentioned here. Two ways in which these biases can be mitigated are mentioned. However, it does not explicitly call for a combination of the two possibilities. Thus, it is left open how to assess the evidence from studies that could not be blinded and only blinded data collection, but at the same time used exclusively subjective, self-reported, unexamined objective outcomes as primary endpoints.

This does not adequately address a significant factor for bias in study results. [4], [1]

Proposed change:

Often blinding of the personnel performing the intervention will be impossible and blinding of the patients will be difficult or also impossible.

In this case, to still obtain meaningful results, it is necessary in such studies to use robust objective endpoints in addition to blinded endpoint assessment.

In general, when evaluating the design of randomized clinical trials as well as reviews of non-pharmacological interventions, it is important not only to take into account non-blinding and the resulting risk of bias due to both practitioners' and patients' expectations, but also to check the extent to which this bias is controlled by other methods, particularly the use of primary objective endpoints. If subjective measurements or self-reported data alone are used as primary endpoints in a non-blinded study, the evidence for the benefit of the intervention under investigation is judged to be unreliable.

Comment (point 2)

Annotation:

The actual verifiability of endpoints for objectivity should be made explicit.[...].

Proposed change:

In addition, when using objective endpoints that measure an effect of symptoms, such as physical/cognitive activity, motor function, school attendance, ability to work, etc., care should be taken to ensure that they are actually collected objectively and not solely through self-reported data from study participants:s.

Comment (point 3)

Annotation:

Certain psychotherapeutic procedures aim, among other things, to ignore symptoms or to rate [bewerten = rate/ asses /evaluate/ conceive] them as less severe, even in diseases where there is no evidence that an over-evaluation of symptoms contributes to the disease process. In appropriate procedures, it is important to consider this bias as an additional factor in biasing the measurement points reported by study participants:in.[28]

Proposed change:

Certain psychotherapeutic procedures aim, among other things, to ignore symptoms or to rate them as less severe. For such procedures, this orientation should be considered as an additional factor for bias in the measurement points reported by study participants.

References to the comments

IQWiG's reply to all three points ('Appreciation' /'Würdigiung'):

Further, one of the comments advises that when dealing with study endpoints, "care should be taken to ensure that they are indeed collected objectively and not solely from self-reported data by study participants." Further, the statement warns that there is a risk of bias in study results if one of the study interventions aims to "ignore symptoms or rate them as less severe , but does not affect the symptoms themselves. This problem occurs, for example, in the evaluation of psychotherapeutic procedures.

Certainly, it is desirable if endpoints that can be objectively recorded are recorded in this way, because this usually increases the reliability and validity of data collection (e.g. by blinding). Conversely, however, patient-reported outcomes, such as pain or quality of life, are of utmost importance for patients and thus also for the benefit assessment, although they can only be recorded subjectively.

The fact that many symptoms can only be recorded subjectively is also not a disadvantage because ultimately only the patient can evaluate the success of his or her own treatment. If a person learns to rate [assess /evaluate/ conceive] his or her own symptoms as less severe or threatening, then this can be seen as a genuine relief, since here too it is the subjective patient perspective alone that counts. On the whole, therefore, there is no need to change the need to change the methods paper in this respect.



(*) Source: Documentation and evaluation of comments on the Draft of the General Methods 6.1 (in German), https://www.iqwig.de/methoden/allgemeine-methoden_dwa-entwurf-fuer-version-6-1_v1-0.pdf

English translation of the relevant section of the methods paper see this post.

Edited for clarity.