How to assess if a research paper has used best-practice protocols?

Hi
A friend mentioned to me that they were considering using a new supplement* that others say has helped people with ME. She was confident that there was a good body of university-led research to support it. When I asked if she had verified that best practice was used for each of those research papers, she asked me how she could tell if it was good quality research. Well, I realised that I couldn't give her a list of things to look out for! - other than the assessments I have read of the PACE trial that have highlighted issues with that research.

I wondered what people here look for when reviewing new research papers?
Does anyone have a checklist that they use to assess the quality of the research process and reliability of the research results? Is there already a thread here on S4me for this?

*fyi supplement under her consideration is Protandium. And there is a thread on here that I have shared with her https://www.s4me.info/threads/protandim.11453/

A quick google and I found a few documents, and have stopped at WHO...

World Health Organization - Recommended format for Research Protocols
This document contains their guidelines for submissions of Research Proposals to WHO. So it is a research preparation list, not a research review list. And some is related to protecting the human participants, more than guaranteeing a quality research outcome. But perhaps will do?
https://www.who.int/ethics/review-committee/format-research-protocol/en/

Also found the WHO GCP (Good Clinical Research Practice) statement - which at 129 pages is not quite the short list I was after
https://www.who.int/medicines/areas/quality_safety/safety_efficacy/gcp1.pdf

Summary of Info from the WHO Research Protocol format page...

Clarity in what is written up would be important to.

Before I spent more time, I thought I would throw the question out here. What do you look out for when reviewing a research paper?

 

I am a bit sceptical about lists of methodological rules for judging study quality. In my view there are no general rules that always apply other than common sense. Small uncontrolled studies are sometimes fine. Large controlled studies can be useless. I think the criteria to look for in this situation may be simpler.

1. Are there studies listed on PubMed Central that appear to indicate that the treatment actually makes a group of people better in terms of quality or length of life? (In this case no - so end of search.)
2. Might the results of these studies be due to bias? To answer this you need to know quite a bit about how effects are measured in trials but much of the time the risk of bias is fairly obvious - as in open trials where results can be massaged in one way or another.

The simple answer to Protandim is that, as far as I can see, nobody has yet shown it does people any good. University based research on hypothetical benefits on 'oxidative stress' are irrelevant. Nobody knows if oxidative stress matters.

What I thought was interesting is that the only paper that comes near to testing whether Protandim does people good is one on athletic performance. It is negative. But why was the supplement tried on athletes? I have a sneaking suspicion that this trial was intended to be negative so that the supplement would not be banned in people doing competitive sports. So it can be sold to athletes as improving their health but not of course illegally tampering with their performance on track.

 

I usually start from the assumption that a study on a branded food supplement is part of the marketing strategy. It can therefore be ignored until it's been replicated in a reasonably well-defined group, with measurable outcomes...funnily enough, these tend to be few on the ground!

 

Just going off on a random thought now. Does anyone know if any researchers have applied to WHO for funding for ME/CFS research? Sounds like WHO would back research that is lacking?

 

I found some useful guidelines for journalists to assess 'scientific evidence' here...
https://www.sciencemediacentre.co.nz/coveringscience/

• https://www.sciencemediacentre.co.nz/coveringscience/types-of-scientific-evidence/
  
• https://www.sciencemediacentre.co.nz/coveringscience/peer-review-and-how-to-read-a-paper/
  
• https://www.sciencemediacentre.co.nz/coveringscience/spotting-bad-science-the-definitive-guide-for-journalists

​

 

Seems like it..

https://www.sciencemediacentre.co.nz/about/

https://www.sciencemediacentre.co.nz/about/international-science-media-centre-network/

 

Years ago the UK SMC had something like the materials the NZ SMC has. I wished that I would have downloaded the materials because they were very good at explaining to the lay person (i.e. journalists) the basics of how medical research works (and why the findings SMC were hyping were bad). Alas, several months ago when I went to look for it for the very same reasons as @RoseE, it was gone. Call me jaded, but I couldn't help but wonder if it was because SMC was much more interested in force feeding research findings to journos then helping them critical examine such research themselves. Though, to be fair, most journos don't have time to critically examine research. They just want the press release and a few quotes so they can bang something out on deadline--and SMC is, alas, always happy to oblige!

While the NZ SMC materials are a bit simplistic (as @Jonathan Edwards pointed out above, just because something is a randomized trial, doesn't mean it's any good), for someone who is totally unfamiliar with what a randomized trial is vs. an observational study vs. a clinical case study vs a retrospective study, it's not a bad start.

When I first became a patient nearly 20 years ago, I had no real sense of how to evaluate medical research. And like a lot of people, I just thought that if it's on PubMed, it must be true! It took a good 8 years or so before I began to understand the rudiments of how to critically examine research. And, of course, years of reading people here pick studies apart has been immensely helpful. But I wish we had a good place for new patients to go to (and that there would have been some place like that when I was first diagnosed--would have saved me not only money but real harm as well!).

 

I don't know if this is of any interest to anyone...

Title : How to read a paper

Subtitle : The basics of evidence based medicine

https://www.ebcp.com.br/simple/upfiles/livros/001HTRP.pdf

The above link gives the second edition of the book for free, but the latest edition (the sixth) is available for sale on Amazon (at an eye-watering price).

Code:

https://www.amazon.co.uk/How-Read-Paper-Evidence-based-Healthcare/dp/111948474X/ref=cm_cr_arp_d_product_top

 

Written by Trisha Greenhalgh, who is a great fan of BPS and is currently screwing up Long Covid the same way!

 

Yes, I know. But I hoped it was generic enough to apply to all sorts of papers.

 

I thought this article provided some content relevant to the topic of this thread.

We need to throw out a mindblowing amount of science and start again
Danyl McLauchlan reviews Stuart Ritchie’s book "Science Fictions: Exposing Fraud, Bias, Negligence and Hype in Science", which outlines the staggering systemic flaws in the funding and publication of scientific papers.

https://thespinoff.co.nz/books/19-0...indblowing-amount-of-science-and-start-again/