Tolerating bad health research: the continuing scandal, 2022, Pirosca et al.

Abstract

Background
At the 2015 REWARD/EQUATOR conference on research waste, the late Doug Altman revealed that his only regret about his 1994 BMJ paper ‘The scandal of poor medical research’ was that he used the word ‘poor’ rather than ‘bad’. But how much research is bad? And what would improve things?

Main text
We focus on randomised trials and look at scale, participants and cost. We randomly selected up to two quantitative intervention reviews published by all clinical Cochrane Review Groups between May 2020 and April 2021. Data including the risk of bias, number of participants, intervention type and country were extracted for all trials included in selected reviews. High risk of bias trials was classed as bad. The cost of high risk of bias trials was estimated using published estimates of trial cost per participant.

We identified 96 reviews authored by 546 reviewers from 49 clinical Cochrane Review Groups that included 1659 trials done in 84 countries. Of the 1640 trials providing risk of bias information, 1013 (62%) were high risk of bias (bad), 494 (30%) unclear and 133 (8%) low risk of bias. Bad trials were spread across all clinical areas and all countries. Well over 220,000 participants (or 56% of all participants) were in bad trials. The low estimate of the cost of bad trials was £726 million; our high estimate was over £8 billion.

We have five recommendations: trials should be neither funded (1) nor given ethical approval (2) unless they have a statistician and methodologist; trialists should use a risk of bias tool at design (3); more statisticians and methodologists should be trained and supported (4); there should be more funding into applied methodology research and infrastructure (5).

Conclusions
Most randomised trials are bad and most trial participants will be in one. The research community has tolerated this for decades. This has to stop: we need to put rigour and methodology where it belongs — at the centre of our science.

Full text: https://trialsjournal.biomedcentral.com/articles/10.1186/s13063-022-06415-5

 

I tend to agree with the above abstract.

As an outsider, a patient, it has been disconcerting to see the poor state of health research. There are so many bad trials and studies, so much research waste.

I find it quite perplexing and do not quite understand how we ended up in this situation.

Patients deserve better. Standards and norms need to be raised.

It must also be quite frustrating for the minority of actual scientists who are doing valuable work, to have so much nonsense, hype and waste in their field.

 

Perhaps the authors should also have recommended that the statisticians and/or methodologists be independent from the team of trial investigators. CFS trials of exercise and CBT did include academic statisticians and methodologists, but this did not prevent them from being at high risk of bias.

One caveat of this paper is relying on Cochrane reviewers’ assessment of risk of bias. As we know, there is reason to believe that it is not always correct, especially for reviews where the authors are practitioners of the intervention in question. The true extent of bad health research is probably worse than this paper suggests.

 

There are a series of symposia organised on research ethics by scientists who care about this. They discuss outright fraud and problematic stuff like paper mills, but also QRP, questionable research practices. (In all kinds of disciplines, not just health.)

E.g. the end of may saw the 7th world conference on research ethics.

https://twitter.com/user/status/1531523883601387522


Next week in The Netherlands the 2nd National Symposium on Research Integrity will be held.

https://twitter.com/user/status/1531529901475495937


I understand that videos of the presentations will be available on Youtube after the symposium.



Table 2 Gopalakrishna et al




(Btw., a very important term to know in relation to research and particularly the bad BPS CBT trials: helicopter research. "‘Helicopter research’ occurs when researchers from high-income settings, or who are otherwise privileged, conduct studies in lower-income settings or with groups who are historically marginalized, with little or no involvement from those communities or local researchers in the conceptualization, design, conduct or publication of the research.")

 

I came across this paper, and am posting it here b/c of the topic. I think it is an interesting read for those who want to read/write about it. It also discusses bias, and modification to get to certain outcomes.

Also: an elaborate and relevant reference list for further exploration.


How Many Scientists Fabricate and Falsify Research? A Systematic Review and Meta-Analysis of Survey Data

Abstract:
The frequency with which scientists fabricate and falsify data, or commit other forms of scientific misconduct is a matter of
controversy. Many surveys have asked scientists directly whether they have committed or know of a colleague who
committed research misconduct, but their results appeared difficult to compare and synthesize. This is the first meta-analysis of these surveys.
To standardize outcomes, the number of respondents who recalled at least one incident of misconduct was calculated for each question,
and the analysis was limited to behaviours that distort scientific knowledge:
fabrication, falsification, ‘‘cooking’’ of data, etc… Survey questions on plagiarism and other forms of professional
misconduct were excluded. The final sample consisted of 21 surveys that were included in the systematic review, and 18 in
the meta-analysis. A pooled weighted average of 1.97% (N = 7, 95%CI: 0.86–4.45) of scientists admitted to have fabricated,
falsified or modified data or results at least once –a serious form of misconduct by any standard– and up to 33.7% admitted
other questionable research practices. In surveys asking about the behaviour of colleagues, admission rates were 14.12%
(N = 12, 95% CI: 9.91–19.72) for falsification, and up to 72% for other questionable research practices. Meta-regression
showed that self reports surveys, surveys using the words ‘‘falsification’’ or ‘‘fabrication’’, and mailed surveys yielded lower
percentages of misconduct. When these factors were controlled for, misconduct was reported more frequently by medical/
pharmacological researchers than others. Considering that these surveys ask sensitive questions and have other
limitations, it appears likely that this is a conservative estimate of the true prevalence of scientific misconduct

 

An important reason may be the toxic culture within academia. Highly hierarchic, where criticism can cost you your future. Lots, lots of egoes. Lots of pressure to publish in order to be given and maintain status. And money: publishing means status means income and a justification of you being there. Or income (grants) for the institute you work for. It's a potent mix of things all coöperating towards the same endpoint: crappy research b/c money/dependence/status/ego/a-holes are at the wheel and genuine interest to get to dependable knowledge through solid trials can be a passenger, but doesn't need to be in the car per se.

As an outsider I imagine part of the people in academics wanting change but not knowing where to begin, feeling powerless to steer it to a better situation when their surroundings, superiors and/or the whole institution they work for is like that and their own career and peer acceptation depends on being liked by those who accept these practices. And the other part being just fine with it, because cleaning up would mean a jeopardy to their careers and status. (I remember a piece Brian Hughes referred to on his blog. The author bascically argued that drawing a line and demanding psychological research to adhere to a set of standards that ensured its quality better was akin to making study authors the victim of a horrible injustice.)

Yes.

Although I think more accurately, patients are owed better. If a healthy researcher wants to "borrow" an illness to satisfy their curiosity and build part of their career researching it, then that is great for both parties, and out of that collaboration great things can develop. But they do not own the illness, they have no firsthand experience of how it is to live with it, it is not theirs. A good researcher (and there are plenty) is aware of that and builds their expertise by careful listening and observing, until through gaining secondhand knowledge, combined with their own finds that develop out of their patient-based research, they know their subject very well and can talk about an illness in a way that makes patients say stuff like "They really get it."

For a long time it has been the accepted custom to use patients as pawns for personal career development and status building. They have been talked about as guinea pigs with no agency, a thing through which the doctor or researcher could establish a claim of expertise. This exploitative attitude has led to mistreatment, as patients did not have a voice in the research of their own "property". I can't speak about other disciplines, but within psychiatry especially women and people of non-Western background have been a victim of this, as the appropriation of their illnesses went hand in hand with deep and harmful prejudice.

This attitude is changing, as it should.

I understand saying "patients deserve better", and I agree fully. But for me that word comes with a sort of passiveness, like patients have to be deserving this. They don't. Wheter it's diabetes, or ME/CFS, or AIDS, or hypothyroidism, or whatever, patients are the owners of their illness. It is theirs. And if you want to use it, great, but as a healthy person you are a borrower and should handle it with the care that comes with that. Which includes not doing shoddy research that can harm the people whose illness you used to produce a paper.


(Sorry for the longer piece. The topic of appropriation of illness by healthy people for their own ends and patient ownership is on my mind for a while now, and I have three quarters of an essay on it in my brain .)

 

Standards will be continue to be lowered until morale (seems to) improves (on BS questionnaires).