Tuller / Trial By Error: The Crawley Chronicles, Resumed

[Cheshire]

So let’s talk about Professor Esther Crawley’s SMILE trial, published in September by the journal Archives of Disease in Childhood, one of the BMJ Publishing Group’s titles. The study reported that a commercial intervention called the Lightning Process was an effective treatment for children with CFS/ME when offered along with what was called “specialist medical care.”

SMILE was an open-label trial relying on subjective responses, a study design notoriously vulnerable to bias. In this case, self-reported physical function was the primary outcome, just as it was one of two primary outcomes in PACE.

http://www.virology.ws/2017/12/13/trial-by-error-the-crawley-chronicles-resumed/

 

[Jonathan Edwards]

David knows the meaning of due diligence.
Impressive.

 

[chrisb]

One trusts that EC will be duly grateful for DT's work. Did she not say in one of the interviews that the outcome of the trial was not entirely expected?

Now she will know why.

 

[Allele]

How is she even still allowed to practice "science"?

 

[Andy]

As its a long post of his, David provides a short(er) version

This is another long and very complicated post. (Sorry!!) Here are the highlights:

*More than half the participants in the SMILE trial were apparently participants in an earlier feasibility trial. That means most if not all were recruited and provided data before the full-trial protocol was approved. Since SMILE lumped together these earlier data with those from participants recruited later, the full trial itself was not an independent investigation of the information generated by the feasibility trial.

*Based on the results of the feasibility trial, Professor Crawley swapped her primary and secondary outcome measures. The original primary outcome in the feasibility trial—school attendance at six months—was relegated to the status of a secondary outcome. The subjective measure of self-reported physical function, which was a secondary measure for the feasibility trial, became the primary outcome for the full trial. (In the full-trial protocol, self-reported fatigue was also listed as a primary outcome. For unexplained reasons, it was downgraded to a secondary outcome in the full-trial report.)

*Swapping the outcomes based on the feasibility study findings while simultaneously extending the feasibility study into the full study could easily have introduced significant bias in the final paper. How much bias cannot be ascertained at this point, since Professor Crawley has not provided a separate analysis of the feasibility study results for physical function and school attendance. That bias would have added to the bias already generated by the reliance in an open-label trial on a subjective outcome—self-reported physical function.

*Professor Crawley promised to seek verification of self-reported school attendance by requesting official school attendance records. Although she mentioned this in the protocols for both the feasibility trial and the full trial, these school records are not mentioned anywhere in the full-trial report. Nor did she discuss the feasibility of accessing these records in the logical place–the feasibility trial report. One possible and very logical conclusion is that she obtained these objective data but decided not to mention them because they did not provide optimal results.

*The trial registration, indicated that SMILE was a prospective trial. But the registration application date of June 7, 2012, coincided almost exactly with the end of the recruitment time frame for the feasibility trial, which provided more than half of those who ended up being included in the final sample. The full-trial paper did not mention that more than half the participants were from the feasibility study and that their data led to the decision to swap the outcomes. By definition, a prospective trial must not include data from previously assessed participants. If it does, it is obviously not a prospective trial.

*Based on the revised primary outcome of self-reported physical function, the full-trial paper reported that the Lightning Process combined with specialist medical care was effective in treating kids with CFS/ME. The full-trial paper also reported that school attendance at six months–the original primary outcome in the feasibility study—produced null results. Thus, the outcome-swapping that occurred after more than half the full-trial sample had already been recruited for the feasibility study allowed Professor Crawley to report more impressive results than had she retained the six-month school attendance measure as the primary outcome.

*Not surprisingly, media reports focused largely on the positive results for the self-reported physical function outcome and not the null results for the original primary outcome. Without the outcome-swapping that took place after more than half of the participants for the full-trial paper had provided data as part of the feasibility study, the final report would not have been able to present such an optimistic perspective.

*Given these major flaws and many additional problems cited by others, the inescapable conclusion is that the SMILE trial should never have been approved, much less published.

And nice to see that acknowledges that patients have raised the issues behind this trial.

Since it is not apparent from either the full-study protocol or the SMILE trial paper itself that more than half the subjects were retroactively enrolled as participants, how did this peculiarity come to light? Well, when the study was published, the Phoenix Rising squad of bulltwaddle-detectors sprang into action. Forum members quickly noticed something strange. The SMILE paper reported that the study began in September, 2010—the same month as the reported start of the feasibility trial. (Many of these bulltwaddle-detectors have since moved to another forum, Science for ME.)

 

[Barry]

It's why software has to be tested by independent test engineers, with their own test specifications. Even the most diligent and professional software developers will unconsciously test their software through execution paths they best favour as necessary, and may miss something a tester won't. EC's approach seems equivalent to a software developer deliberately avoiding certain tests ... "because there might be bugs in that area".

 

[Barry]

When Professor Levin says “NEVER,” he means NEVER. (He himself capitalized the word in his e-mail.) Yet that’s what Professor Crawley did, despite the likelihood of generating a self-fulfilling prophecy.

Yep, not just angelic, but "propheteering" as well.

 

[Barry]

One trusts that EC will be duly grateful for DT's work.

Quite so. He's doing some excellent peer reviews of her work.

 

[Luther Blissett]

It’s why software has to be tested by independent test engineers, with their own test specifications. Even the most diligent and professional software developers will unconsciously test their software through execution paths they best favour as necessary, and may miss something a tester won’t. EC’s approach seems equivalent to a software developer deliberately avoiding certain tests … “because there might be bugs in that area”.

And forum manual writers too

Another good piece by David Tuller.

Nice bit of promotion for the forum, so if he sees it, thanks. Skillfully done.

Why couldn’t the data for students who went through their GCSE’s and onto A Levels have just been separated into two measures for attendance? It would still be interesting if they could manage 2-3 hours per day, as many patients would like to reach that level. (For non-U.K members - after 16 years of age, at the end of the school year, students move from GCSE to A Levels. It’s like moving from general education to a more narrow set of subjects, hence the less time required to attend.)


Scientific Study Proposal:

I think we should have a study of how he, Jonathan Edwards and Bruce Levin maintain a good quantity of hair on their heads after researching all this stuff.

 

[Alvin]

Very well done
The long version is very long, i already fried my brain on the short one so in future i'll take a stab at the long version

Oh and something to point out, ethical scientists typically argue things based on the merits because we are right, but the other side will play dirty. I don't propose stooping to their level but i would play hardball, its unethical to use quackery on MS patients so how is it more ethical to force it upon ME/CFS patients?

 

[Dolphin]

Aside: I hadn't seen "goulash" used in this way before so I googled "goulash meaning". Didn't like the second entry I saw [urban dictionary]!

 

[large donner]

I'm not sure its true that kids only have to attend school 2-3 hours a day for A levels as opposed to 6 for GCSEs. It may depend on the Sixth form they attend though.

From the kids in my extended family who are doing A levels they attend for the whole school day. On top of that they seem to have alot of work to do outside of school hours for A levels.

 

[Arnie Pye]

That letter included the following: “Both the patient and their parent/s or guardian have provided us with written consent to participate in the study which is kept in their medical notes. As part of their consent, they have given us permission to check their school attendance record during their involvement with the study. This is important information as it is the principle outcome measure for the study.” (Of course, school attendance was no longer the “principle outcome measure,” but let’s put that inaccuracy aside.)

[My emphasis]

Shouldn't that be "principal" rather than "principle"? I do hope the mistake was E.C.'s rather than David Tuller's!

 

[BruceInOz]

“The reason for this is that many of the participants are transitioning from GCSEs to A levels in this study and therefore % of school attendance does not necessarily reflect illness severity. For example, a teenager may have decided to take 2 A levels and be attending school for 2-3 hours a day. This would be recorded as 100% school attendance but this does not equate to 6.5 hours a day of normal school attendance.”

This excuse only makes sense if schools only keep records for the percentage of expected hours attended and not the actual hours attended. Does anyone know if schools do not keep that data?

 

[Esther12]

Thanks once again to Tuller for all his work. That was epic.

When reading that I really felt like this study has the hallmarks of the sort of 'simple' junk-science that should attract wider attention from people who don't care about CFS?

Despite it's length, there were still bits that were so dense I wondered if people new to the topic would be able to follow it? What do others think? It shouldn't be a problem for people used to reading/writing about science though, and I suppose that's the key audience to try to influence. I love how Tuller works to include all the provisos and details that prevent PACE/Crawley etc claiming he's misrepresenting their work, but it does make it difficult to create an easy to read story. (Actually - PACE and Crawley still do claim he's misrepresenting their work... that's so unfair!)

Since it is not apparent from either the full-study protocol or the SMILE trial paper itself that more than half the subjects were retroactively enrolled as participants, how did this peculiarity come to light? Well, when the study was published, the Phoenix Rising squad of bulltwaddle-detectors sprang into action. Forum members quickly noticed something strange. The SMILE paper reported that the study began in September, 2010—the same month as the reported start of the feasibility trial. (Many of these bulltwaddle-detectors have since moved to another forum, Science for ME.)

Hey - it looks like it was me who mentioned that! My bulltwaddle-detection was going off like crazy over SMILE (apologies in advance for stealing credit from someone else, as I fear I am). Also seems @Valentijn was the one who pointed out that they failed to mention checking school records for assessing attendance levels. Concerns about bulltwaddle were what led to the move to another forum!

 

[Esther12]

One trusts that EC will be duly grateful for DT's work. Did she not say in one of the interviews that the outcome of the trial was not entirely expected?

Now she will know why.

Yup, she was really surprised that when she switched outcomes half-way through she could turn a null result into a positive one with an excited press release... she'll love that Tuller's helped explain this confusing state of affairs to everyone!

 

[Woolie]

Shouldn't that be "principal" rather than "principle"? I do hope the mistake was E.C.'s rather than David Tuller's!

Yes, I think so too!

 

[Luther Blissett]

This excuse only makes sense if schools only keep records for the percentage of expected hours attended and not the actual hours attended. Does anyone know if schools do not keep that data?

I would have thought that attendance would be registered at least twice a day (mandatory according to Government Rules), just like in the stone ages when we attended.

It's probably all done electronically now. At college for my A Levels and equivalent each class was registered as well. That would mean that each teacher would have a way of knowing who is 'missing' or late.

 

[Barry]

Shouldn't that be "principal" rather than "principle"? I do hope the mistake was E.C.'s rather than David Tuller's!

Here's the context ...

But that’s not all. Remember how Professor Crawley promised in both protocols to make an effort to check the self-report measure of school attendance against official school records? In fact, the application for extending the feasibility study while swapping the outcomes featured a draft of the letter to be sent to school officials.

That letter included the following: “Both the patient and their parent/s or guardian have provided us with written consent to participate in the study which is kept in their medical notes. As part of their consent, they have given us permission to check their school attendance record during their involvement with the study. This is important information as it is the principle outcome measure for the study.” (Of course, school attendance was no longer the “principle outcome measure,” but let’s put that inaccuracy aside.)

 

[Binkie4]

This is not very coherent. Am still not expressing myself clearly after anaesthesia but I wanted to comment below.

Have scanned David's article and I am even more concerned that MEA continues its link with Esther Crawley via CMRC. Have commented on MEA fb page on this if anyone cares to 'like' or comment.

Felt very vulnerable in hospital where exercise stress tests were being proposed, and GET still hangs round as treatment of choice for ME. I managed to get out without it but it's tricky.

@Barry - I think principle is ok.

Thanks all for your good wishes prior to admission.