Article: Is my study useless? Why researchers need methodological review boards

[Andy]

"Should researchers have the freedom to perform research that is a waste of time? Currently, the answer is a resounding ‘yes’. Or at least, no one stops to ask whether there are obvious methodological and statistical flaws in a proposed study that will make it useless from the get-go: a sample size that’s simply too small to test a hypothesis, for example.

In my role as chair of the central ethical review board at Eindhoven University of Technology in the Netherlands, I’ve lost count of the number of times that a board member has remarked that, although we’re not supposed to comment on non-ethical issues, the way a study has been designed means it won’t yield any informative data. And yet we routinely wait until peer review — after the study has been done — to identify flaws that can’t then be corrected.

In my own department at Eindhoven, we’ve been trialling a different approach. Five years ago, we instituted a local review board that also evaluates proposed methods. Although some colleagues found this extra hurdle frustrating at first, the improvements in study quality have led them to accept it. It’s time to make dedicated methodological review boards a standard feature at universities and other research institutions, as institutional review boards are."

https://www.nature.com/articles/d41586-022-04504-8

 

[Jonathan Edwards]

This sounds a bit behind the times. When I was on the UCL ethics review panel in the 1980s we took seriously whether the study would produce useful results if it involved any risk to patients. That would be the case for any drug testing, invasive intervention or even anything that might cause distress.

The whole point of ethics boards is to look at risk benefit. That means you have to assess the likely benefits - which are nil if the design is poor.

 

[Peter T]

"Should researchers have the freedom to perform research that is a waste of time? Currently, the answer is a resounding ‘yes’. Or at least, no one stops to ask whether there are obvious methodological and statistical flaws in a proposed study that will make it useless from the get-go: a sample size that’s simply too small to test a hypothesis, for example.

I would argue that research involving human subjects that is never going to produce meaningful results is inherently unethical:

• Firstly it is wasting subjects time, which is particularly significant when subjects are also patients whose time and energy is a precious limited resource. A central purpose of an ethics committee is surely to safeguard potential human subjects.
• Secondly academic research by respected institutes is assumed by the public to have a useful purpose, so if it is the case that a research project is never going to produce meaningful results, unless subjects know this, they can not be said to have given informed consent.

[Cross posted with @Jonathan Edwards above]

 

[Sean]

Academic 'freedom' comes with a very serious and strict condition. The methodology, including the question being asked, must be of the highest possible technical and ethical standard.

Not up for negotiation.

 

[Ravn]

This sounds a bit behind the times. When I was on the UCL ethics review panel in the 1980s we took seriously whether the study would produce useful results if it involved any risk to patients. That would be the case for any drug testing, invasive intervention or even anything that might cause distress.

The whole point of ethics boards is to look at risk benefit. That means you have to assess the likely benefits - which are nil if the design is poor.

Going by the 'quality' of a great many of the studies we analyse here you must have been so far ahead of your time back then that few have managed to catch up yet.

I would argue that research involving human subjects that is never going to produce meaningful results is inherently unethical:

• Firstly it is wasting subjects time, which is particularly significant when subjects are also patients whose time and energy is a precious limited resource. A central purpose of an ethics committee is surely to safeguard potential human subjects.
• Secondly academic research by respected institutes is assumed by the public to have a useful purpose, so if it is the case that a research project is never going to produce meaningful results, unless subjects know this, they can not be said to have given informed consent.

Yes.

In addition it wastes resources that could have been spent on better research.

 

[Jonathan Edwards]

Going by the 'quality' of a great many of the studies we analyse here you must have been so far ahead of your time back then that few have managed to catch up yet.

I think it is the other way around. Standards of ethics committees have probably fallen with commercialisation of the research ethic.

 

[Hutan]

Thanks for posting this Andy.

In addition it wastes resources that could have been spent on better research.

Yes.
And, further in addition, it allows commercial and other interests to hype their products with 'evidence' of very dubious value (from tiny/unblinded studies with large numbers of (subjective) outcomes).

In New Zealand, the national ethical principles for health research include:

• "Tika refers to what is right and what is good for any particular situation. Importantly, in the context of ethics it relates to the design of a study, and whether the research achieves proposed outcomes, benefits participants and communities and brings about positive change.

• Beneficence for individuals and communities implies improving or benefiting people’s health or broader wellbeing. It is both the basic aim of good research and its fundamental justification. Health research should be designed, conducted and reported with the intention to improve outcomes. Beneficence also requires that projects have merit."

So, it is definitely expected that research studies will be well-designed. I expect that is the case with most countries' health research ethical principles. To what extent the ethics committees that sign off on proposed research actually check research against the required principles is another question.

Maybe we need more challenging of ethics committee decisions when research will not provide useful information. Or, after the research is done, we could at least bring deviations from ethical standards (which include requirements of research quality) to the approving ethics committee's notice so that research review improves.