1. Sign our petition calling on Cochrane to withdraw their review of Exercise Therapy for CFS here.
    Dismiss Notice
  2. Guest, the 'News in Brief' for the week beginning 18th March 2024 is here.
    Dismiss Notice
  3. Welcome! To read the Core Purpose and Values of our forum, click here.
    Dismiss Notice

Assessing Randomised Controlled Trials

Discussion in 'Psychosomatic research - ME/CFS and Long Covid' started by Sly Saint, Aug 19, 2018.

  1. Sly Saint

    Sly Saint Senior Member (Voting Rights)

    Messages:
    9,573
    Location:
    UK
    There are a number of tools available for journals to use to help assess RCTs.
    I wondered if anyone here had 'tried them out' for PACE and/or any of the other 'RCT's
    by Crawley or Chalder?

    I've seen CASP mentioned:
    https://casp-uk.net/wp-content/uploads/2018/01/CASP-Randomised-Controlled-Trial-Checklist.pdf

    and Consort:
    http://www.consort-statement.org/
    "
    The CONSORT Statement is endorsed by prominent general medical journals, many specialty medical journals, and leading editorial organizations. CONSORT is part of a broader effort, to improve the reporting of different types of health research, and indeed, to improve the quality of research used in decision-making in healthcare."

    this is one from the BMJ:
    https://bmjopen.bmj.com/content/sup...-2015-008807.DC1/bmjopen-2015-008807supp2.pdf

    and this is the one used by NICE:
    https://www.nice.org.uk/process/pmg...dology-checklist-randomised-controlled-trials

    (do most reviewers at journals use these tools? and do they submit them to the journals or just use them to help make their decisions? @Lucibee )

    @Tom Kindlon @Graham @dave30th
     
    MSEsperanza, MEMarge, Sean and 5 others like this.
  2. Graham

    Graham Senior Member (Voting Rights)

    Messages:
    3,324
    I'd love to have seen the assessment on PACE: no doubt full of the "although .... the committee decided that .... was acceptable and that bias was kept to a minimum ...."
     
  3. Tom Kindlon

    Tom Kindlon Senior Member (Voting Rights)

    Messages:
    2,199
    I agree such guidelines can be useful.
    I have cited CONSORT guidelines in some of my publications. For example:


     
    Last edited: Aug 19, 2018
  4. Jonathan Edwards

    Jonathan Edwards Senior Member (Voting Rights)

    Messages:
    13,267
    Location:
    London, UK
    I would hope that reviewers do not use these 'tools'. I certainly would not. It is a bit like having a tool to provide a jury instructions on whether or not to find someone guilty. Tools like this might possibly be useful for people who are not familiar with trial assessment to remind them of what they need to look out for. For people who are used to assessing trials (which should really include all properly trained physicians who have attended journal clubs throughout their careers) the important issues will be obvious. What may not be obvious are weaknesses relating to a specific trial context (like a trial that trains people to say they are better and then asks them if they are better) but standard tools are not going to pick them out.

    Part of the problem with Cochrane has been that they have a tool for assessing quality of evidence from trials. And it is completely useless because it misses the most basic problems - as far as we can see from PACE.

    The real problems I think are:
    1. A lot of people in biomedical science are actually quite dim, at least in terms of ability to evaluate results. I am not sure what one does about that. Unfortunately the editors in chief of Lancet and BMJ both seem to fall into this category.
    2. The establishment prefers to protect its own and cover up embarrassing failures rather than admit they got things wrong.
     
  5. Lucibee

    Lucibee Senior Member (Voting Rights)

    Messages:
    1,483
    Location:
    Mid-Wales
    I agree with @Tom Kindlon that CONSORT guidelines are useful, but they don't enable you to assess the quality of trials. They are simply a reporting checklist to make sure that info is reported - ie, have the said what randomisation method they used, what the primary outcome measures were, etc. It doesn't then check that they have used the appropriate ones of either.

    The Lancet does use CONSORT in this sense, but just as a box-ticking exercise.
     
  6. Sly Saint

    Sly Saint Senior Member (Voting Rights)

    Messages:
    9,573
    Location:
    UK
    This is probably a stupid question, but has anyone pointed this out?
    Is there another trial (not ME related) or hypothetical example, that could be used to demonstrate how these things are missed out using the existing tools?

    (Just had a quick search and see that the consort diagram was submitted for this 'Early intervention study', O'Dowd/Crawley https://www.s4me.info/threads/odowd-crawley-early-intervention-study.2931/ )

    And I see that the PACE authors said that they were compliant with Consort guidelines :
    https://www.s4me.info/threads/pace-trial-tsc-and-tmg-minutes-released.3150/page-22#post-88225

    Surely something as basic as no (adequate) control group/s in a Randomised Controlled Trial should get flagged up(?)
     
  7. Jonathan Edwards

    Jonathan Edwards Senior Member (Voting Rights)

    Messages:
    13,267
    Location:
    London, UK
    That is exactly what was not flagged up by Larun et al. If you have dim people applying 'tools' you don't necessarily get a classy piece of furniture, or even one that stays up when you sit on it.

    And when a president of a Royal College does not understand the basics you get to realise how big a problem dim people are.

    I pointed out to Cochrane that whatever evidence grading system they had it obviously was not working. There has been feedback suggesting that they realise that in the PACE case things were not applied properly but I am not sure it sounded as if they really understood the depth of the problem.
     
  8. Sean

    Sean Moderator Staff Member

    Messages:
    7,041
    Location:
    Australia
    Ticking the boxes for CONSORT is like having your car papers in order.

    It's just one of those minimal requirement things. There is still the question of how you drive it, and making sure you put the right fuel in it, and enough air in the tires, and securely locking it up at night so that your data doesn't get stolen.
     
    Barry, ladycatlover, Esther12 and 4 others like this.
  9. Sly Saint

    Sly Saint Senior Member (Voting Rights)

    Messages:
    9,573
    Location:
    UK
    I've watched this short series of videos of a talk given by Mike Clarke which I found on the Cochrane training website.
    "
    This resource is a four-part video recording of a talk given by Mike Clarke, the then Director of the UK Cochrane Center, as part of the Oxford MSc in Evidence-Based Healthcare Programme.
    The talk is about risks of doing multiple analyses, including looking at different subgroups, which can lead to false positive findings. The ways of reducing these risks are discussed."

    the first three are about 15 minutes long each and the last is 6 minutes.

    https://training.cochrane.org/resource/mulitiplicity-and-subgroup-analysis-beware

    "Mike has 25 years’ experience of the conduct and oversight of randomised trials, systematic reviews and other types of prospective research. Previously, Centre Director of the UK Cochrane Centre. He is currently the inaugural Director of the All Ireland Hub for Trials Methodology Research and the Co-ordinating editor of the Cochrane Methodology Review Group."

    as far as I can gather he is now at Queens University Belfast:
    https://pure.qub.ac.uk/portal/en/persons/mike-clarke(f6bee3f9-d4fd-4498-b28b-1e52b1dfaf91).html

    his contact details are on the last link.

    wonder if he is worth contacting?

    @dave30th

    (as an aside: interesting to see that Oxford do courses in RCTs and Systematic Reviews
    https://www.conted.ox.ac.uk/courses/randomized-controlled-trials
    https://www.conted.ox.ac.uk/courses/systematic-reviews)
     
  10. MEMarge

    MEMarge Senior Member (Voting Rights)

    Messages:
    2,733
    Location:
    UK
    From the Cochrane training site:

    "Mulitiplicity and subgroup analysis - Beware!

    This resource is a four-part video recording of a talk given by Mike Clarke, the then Director of the UK Cochrane Center, as part of the Oxford MSc in Evidence-Based Healthcare Programme.
    The talk is about risks of doing multiple analyses, including looking at different subgroups, which can lead to false positive findings. The ways of reducing these risks are discussed.

    After using this resource, you should be able to...
    • Understand and avoid the risks of using multiple analyses in clinical trials and systematic reviews"
     
    andypants, Barry and Trish like this.
  11. MEMarge

    MEMarge Senior Member (Voting Rights)

    Messages:
    2,733
    Location:
    UK
    I wonderif the Northern ireland groups know about him @Tom Kindlon @Keela Too

    Some of his words in the first video could be written directly for the PACE authors.
     
  12. Keela Too

    Keela Too Senior Member (Voting Rights)

  13. JoanNI

    JoanNI Established Member (Voting Rights)

    Messages:
    29
    andypants, MEMarge, Sly Saint and 2 others like this.
  14. MSEsperanza

    MSEsperanza Senior Member (Voting Rights)

    Messages:
    2,855
    Location:
    betwixt and between
    Might be relevant for the Cochrane review/ IAG thread?

    https://www.s4me.info/threads/indep...y-and-me-cfs-2020-led-by-hilda-bastian.13645/
     
    Sly Saint and Sean like this.

Share This Page