How to assess if a research paper has used best-practice protocols?

R

RoseE

Senior Member (Voting Rights)
Hi
A friend mentioned to me that they were considering using a new supplement* that others say has helped people with ME. She was confident that there was a good body of university-led research to support it. When I asked if she had verified that best practice was used for each of those research papers, she asked me how she could tell if it was good quality research. Well, I realised that I couldn't give her a list of things to look out for! - other than the assessments I have read of the PACE trial that have highlighted issues with that research.

I wondered what people here look for when reviewing new research papers?
Does anyone have a checklist that they use to assess the quality of the research process and reliability of the research results? Is there already a thread here on S4me for this?

*fyi supplement under her consideration is Protandium. And there is a thread on here that I have shared with her https://www.s4me.info/threads/protandim.11453/

A quick google and I found a few documents, and have stopped at WHO...

World Health Organization - Recommended format for Research Protocols
This document contains their guidelines for submissions of Research Proposals to WHO. So it is a research preparation list, not a research review list. And some is related to protecting the human participants, more than guaranteeing a quality research outcome. But perhaps will do?
https://www.who.int/ethics/review-committee/format-research-protocol/en/

Also found the WHO GCP (Good Clinical Research Practice) statement - which at 129 pages is not quite the short list I was after :)
https://www.who.int/medicines/areas/quality_safety/safety_efficacy/gcp1.pdf

Summary of Info from the WHO Research Protocol format page...
Rationale & Background info
  • a well-documented statement of the need/problem that is the basis of the project, the cause of this problem and its possible solutions. i.e. why? and what?
Study Goals & Objectives
  • Goals - Broad statements of what the research hopes to accomplish.
  • Objectives - simple, specific research questions stated in advance of the research being done.
Study Design
  • the type of study
  • the research population or sampling frame
  • who can take part (inclusion & exclusion criteria, withdrawl criteria, etc)
  • expected duration of the study
Methodology
  • The most important part
  • Detailed information on the...
    • interventions made - describe drug, device, vaccine being tested; or training or info given to individuals.
    • procedures used - biomedical, questionaire, observation
    • measurements taken,
    • observations made,
    • laboratory investigations done,
    • etc
  • If multiple sites are engaged, methodology should be standardized and clearly defined.
  • Details of randomization and blinding; stopping rules; etc
Data Management and Statistical Analysis
  • How the data was managed including data handling and coding for computer analysis, monitoring and verification.
  • Statistical methods were clearly outlined, including reason for sample size, power of the study, level of significance used, procedures for account for any missing or spurious data, etc
  • Any qualitative approaches require details of how the data was analysed.
Quality Assurance
  • Includes the GCP (Good Clinical Research Practice)
  • Clinical monitors in place
  • DSMB (Data and Safety Monitoring Board) - committee of clinical research experts
  • Data management
  • etc
Etc :)
Click to expand...

Clarity in what is written up would be important to.

Before I spent more time, I thought I would throw the question out here. What do you look out for when reviewing a research paper?
 
There must be a good list somewhere - off the top of my head and specifically in relation to assessing whether the findings of a trial for supplement can be relied on:
1. Conflict of interests - are the researchers independent? Do they stand to gain if they find one thing or another? Did the manufacturer pay for the study?

2. The researchers - do they have a track record doing a range of studies, or do they seem only to have made papers on a particular supplement/brand? Do they come from a credible institution?

3. The journal - is is credible or does it seem to exist only for this study/brand?

4. Quality of the writing - are the methods clear. Is there blathery trendy language that doesn't mean anything?

5. Hype - Does the study over-claim?

6. P-hacking/cherry picking - did they measure a whole lot of things but only report on a few select things (that might have given the right outcome just by chance)

7. Size - is the study sufficiently large? Appropriate size varies, but I'd probably want to see at least 20 people in each arm.

8. Are there controls, is the study blinded, are there objective outcomes? Check out the Mendus study of MitoQ that I wrote up on the Coenzyme Q10 thread for an example of what can go wrong if it's an open label study with subjective outcomes.

9. Selection of participants - did the participants have the same illness as you do? Did the participants have any reason to give a biased result? Were participants randomly selected, or selected to have certain characteristics that might limit the applicability of the trial to your situation?

10. The treatment - does it match the product you are considering? Or does it have extra ingredients or a higher dosage or something else different?

11. Reporting of harms - is reasonable care taken to identify and report harms?

12. Dropouts and missing data - is it clear how many people were assessed at each stage? Are reasons for dropouts given? Is it possible that different rates of dropouts in the various treatment arms might have biased results? Were there lots of missing data points? If so, how was that dealt with - was data imputed?

13. Data - are the figures in the tables right or do things not add up?

14. Significance - is any reported result sufficiently large to be clinically useful?

I guess there's lots of other things, but if all those were ok, that would be a good start.

Edit - Rose, if your friend wants to send me the papers, I can have a look through them. Or better, post them on the forum and a range of people can look at them.
 
I am a bit sceptical about lists of methodological rules for judging study quality. In my view there are no general rules that always apply other than common sense. Small uncontrolled studies are sometimes fine. Large controlled studies can be useless. I think the criteria to look for in this situation may be simpler.

1. Are there studies listed on PubMed Central that appear to indicate that the treatment actually makes a group of people better in terms of quality or length of life? (In this case no - so end of search.)
2. Might the results of these studies be due to bias? To answer this you need to know quite a bit about how effects are measured in trials but much of the time the risk of bias is fairly obvious - as in open trials where results can be massaged in one way or another.

The simple answer to Protandim is that, as far as I can see, nobody has yet shown it does people any good. University based research on hypothetical benefits on 'oxidative stress' are irrelevant. Nobody knows if oxidative stress matters.

What I thought was interesting is that the only paper that comes near to testing whether Protandim does people good is one on athletic performance. It is negative. But why was the supplement tried on athletes? I have a sneaking suspicion that this trial was intended to be negative so that the supplement would not be banned in people doing competitive sports. So it can be sold to athletes as improving their health but not of course illegally tampering with their performance on track.
 
I usually start from the assumption that a study on a branded food supplement is part of the marketing strategy. It can therefore be ignored until it's been replicated in a reasonably well-defined group, with measurable outcomes...funnily enough, these tend to be few on the ground!
 
Just going off on a random thought now. Does anyone know if any researchers have applied to WHO for funding for ME/CFS research? Sounds like WHO would back research that is lacking?
 
NelliePledge said:
Are the NZ SMC linked to the U.K. one?.
Click to expand...
Seems like it..

https://www.sciencemediacentre.co.nz/about/
The SMC was conceptualised on the success of the Science Media Centres in the UK and Australia, which have both have successfully promoted the voices, stories, and views of the scientific community to their national news media. Now there is an international network of science media centres.
Click to expand...

https://www.sciencemediacentre.co.nz/about/international-science-media-centre-network/
The New Zealand Science Media Centre is part of a global network of Science Media Centres (SMCs), all of which aim to inform public debate and discussion on the major issues of the day by injecting evidence-based science into headline news.
Click to expand...
 
The UK Science Media Centre has been a hotbed of BPS support. It has close ties with BPS proponents; I think Wessely has been on the board? It is regularly commented on in this forum. It often/usually seems to be a PR machine, rather than having any commitment to increasing understanding of science.

I don't think the NZ SMC has bothered much with ME/CFS (although it would be interesting to look into it further), and the material linked above isn't necessarily tainted by the UK SMC.
 
Years ago the UK SMC had something like the materials the NZ SMC has. I wished that I would have downloaded the materials because they were very good at explaining to the lay person (i.e. journalists) the basics of how medical research works (and why the findings SMC were hyping were bad). Alas, several months ago when I went to look for it for the very same reasons as @RoseE, it was gone. Call me jaded, but I couldn't help but wonder if it was because SMC was much more interested in force feeding research findings to journos then helping them critical examine such research themselves. Though, to be fair, most journos don't have time to critically examine research. They just want the press release and a few quotes so they can bang something out on deadline--and SMC is, alas, always happy to oblige!

While the NZ SMC materials are a bit simplistic (as @Jonathan Edwards pointed out above, just because something is a randomized trial, doesn't mean it's any good), for someone who is totally unfamiliar with what a randomized trial is vs. an observational study vs. a clinical case study vs a retrospective study, it's not a bad start.

When I first became a patient nearly 20 years ago, I had no real sense of how to evaluate medical research. And like a lot of people, I just thought that if it's on PubMed, it must be true! It took a good 8 years or so before I began to understand the rudiments of how to critically examine research. And, of course, years of reading people here pick studies apart has been immensely helpful. But I wish we had a good place for new patients to go to (and that there would have been some place like that when I was first diagnosed--would have saved me not only money but real harm as well!).
 
I don't know if this is of any interest to anyone...

Title : How to read a paper

Subtitle : The basics of evidence based medicine

https://www.ebcp.com.br/simple/upfiles/livros/001HTRP.pdf

The above link gives the second edition of the book for free, but the latest edition (the sixth) is available for sale on Amazon (at an eye-watering price).

Code: 
https://www.amazon.co.uk/How-Read-Paper-Evidence-based-Healthcare/dp/111948474X/ref=cm_cr_arp_d_product_top

Amazon product ASIN 111948474X
 
I thought this article provided some content relevant to the topic of this thread.

We need to throw out a mindblowing amount of science and start again
Danyl McLauchlan reviews Stuart Ritchie’s book "Science Fictions: Exposing Fraud, Bias, Negligence and Hype in Science", which outlines the staggering systemic flaws in the funding and publication of scientific papers.

https://thespinoff.co.nz/books/19-0...indblowing-amount-of-science-and-start-again/
 
You must log in or register to reply here.
Back
Top Bottom