SMILE patient cohorts

I wanted to make a thread separate from the letter's calling for retraction of SMILE so I don't bog them down with thinking aloud.

Hopefully some members can look at what I’m saying and say – no you’re wrong or this is normal in clinical trials or you’ve misread something. Or if something is already well known.


The topic of this thread is about cohort numbers presented in SMILE. An important reference will be figure 1, which is a patient cohort flow chart.



In figure 1, it is stated, 11 or 12 patients changed cohorts after randomization. As far as I can tell, this is not mentioned or explained in the text. 9 patients opted for SMC after being assigned SMC + LP. 2 patients opted for SMC + LP after being assigned SMC, 1 opted for only LP.


For this to be unexplained seems unusual as it does not represent a typical lost follow-up but a cohort switch. Were patients given the opportunity, then that represents a separate step that needs to be explained.

 

Also why would 4 who had undergone LP before seek to do it again ?
ETA excluded as done it previously, but not a good advert ?

 

Lots of possible reasons:

- The referring agents did not know/understand fully what the research involved, just passed on all their children with CFS whose parents/guardians agreed
- The referring agents did not know of the subjects previous experience of LP
- The referring agent believed further LP would be a good thing, for whatever reason
- Overlap with above, the referring agent desperate to try anything to help this ‘suffering’ child so will uncritically try anything on offer
- The subject/parents felt they should, for whatever reason follow the referring agents suggestions uncritically or unargued.
- The subject/parents did not know/understand fully what the research involved
- The subject/parents believed further LP would be a good thing, for whatever reason
- Overlap with above, subject/parents desperate to try anything to help this ‘suffering’ child so will uncritically try anything on offer
- Overlap with above, subject/parents wanted to get something very expensive for free

What I find more interesting is how they sought referrals, the study’s selection criteria and their large attrition rate from the point of initial referral. I will come back to this when I have reread the study.

 

The numbers of " not followed up" and withdrawn consents throughout the process are striking.

I with LP only - how does that work ?

Is this normal ?

ETA spelling

 

There are many possiblities I think. They may have recieved SMC and then after, SMC + LP. They may have been SMC + LP that changed cohorts, weren't included and then went back to SMC +LP. Or they may be the three from SMC that changed cohorts (note that 3 is the number that did so). I find the wording vague in both the origninal and new paper since it really doesn't tell us where they started.

Things will get weirder! I have a few posts/topics. But I have to go slowly as it's fatiguing to read through this and trying to track all the numbers and patients, as is with these 3 patients alone.

 

My comments above may need some amendment as, although the published information is [limited], it would seem that the [only] source of subjects was referral from the study’s lead author’s own clinic.

The final [amended version of the] study write up (see https://adc.bmj.com/content/103/2/155 “Clinical and cost-effectiveness of the Lightning Process in addition to specialist medical care for paediatric chronic fatigue syndrome: randomised controlled trial”) is unclear about subject recruitment and refers to the study protocol. The current amended version only says

However the preregistration protocol, published after the commencement of the study (see https://trialsjournal.biomedcentral.com/articles/10.1186/1745-6215-14-444 “ ) says little more

So the implication is that children were only referred by the Bath Service after they had been referred to that Service by ‘normal’ referral processes. It is always difficult to recruit participants in such studies and an open call for referrals can not guarantee random or unbiased patterns of referrals. One approach is to attempt to select all people diagnosed in a specific geographical area over a specific period of time, which the write ups might be taken to suggest.

Given recruitment took place over three years, when the Service would expect to see some 750 patients in that period. I am confused whether the initial appointment involving some 657 subjects, mentioned above by @dreampop, is the initial appointment with the Bath Service or the initial visit with the researchers. The reasons given for drop out however would seem to suggest the former, that the 657 was the number of initial appointments with the Bath Service rather than initial appointments with the research team.

Either way the reliance on referrals from just one clinic introduces a risk of bias resulting from that clinics reputation and referral practices, especially given the controversy surrounding the head of the clinic who is also the lead researcher. Further given the distinction the study makes between the research team and the clinic, it would be helpful to know how many of the headline 657 figure were seen initially by individuals both working in the clinic and part of the research team. It also has implications for the power dynamics between the research team and the subjects, [especially] given there are potential issues relating to concerns on the part of the children/families about child protection issues, which in turn has implications for self reported subjective outcome measures. [Given the high level of inappropriate involvement of child protection teams with children with ME/CFS and the not infrequent comments by parents on forums that they feel obliged to tell clinicians what they want to hear, this should be a consideration, making objective outcome measures even more essential.]

One way round this [potential selection bias] would have been to draw in subjects from other sources.

On the basis of the information given we can not rule in or rule out that no children we referred to the Bath Service in order to take advantage of a chance of free access to the Lightning Process.

That 136 of those referred to the Bath Service, just over 20% of the total sample, turned out not to have ‘CFS’ which is not surprising given other studies report potential for misdiagnose of up to 40%.

However some 125, just under 20% of the total sample did not return their contact forms. Unfortunately we can not say that this was a consequence of just getting lost in the complexities of everyday life with a seriously ill child, or indication that they did not wish to participate in the study. Though we don’t know if they changed their minds after their initial appointment or if they had previously been too polite or unconfident at that appointment to say no. Again this is a potential source of bias into the sample. And as with nearly all studies into ME/CFS there is potentially exclusion of the more severely impaired, [14 were straight away excluded for being ‘severe’ and] with the 75 excluded because they live too far, we have no idea whether the clinic/researchers defined too far or the subjects/families did, too far being a potential subjective [judgement] and potentially related to the the severity of the child’s condition [what is too far for one family will not be too far for another].

There was a further 33 drop outs, some 5%, before the start of the study, their reasons are not reported. Presumably the researchers have no information on this.

[This may be an unfair comment, but seems potentially less so given the study’s initially undisclosed elision of a feasibility study subjects into the full final study, but rather feels like there has been some at least post hoc tidying up of subject categories to keep the numbers up, which is not strictly consistent with random allocation. This might be acceptable in a feasibility study, but not in a full prospectively registered trial.]

[edited - corrections of grammar, and alterations shown in square brackets]

 

Can't they say that the participants were randomised to access to SMC + free LP vs SMC only? Even if some of those in SMC + free LP did not use the LP, and some in the SMC only group chose to pay for LP themselves, that's still a form of randomisation, even if it's far from ideal.

 

This got me curious about what they mean by SMC and here is the definition they used:

I don't see how the first sentence qualifies as specialist medical care, this description of a range of professionals being involved almost never happens outside of trials, there are no valid treatments anyway and there are no details about what it means to "focus on improving sleep".

The last part suggests this actually was APT of sorts, focused on gradually increasing activity. The link points to the NICE guidelines so SMC is basically whatever. Considering what we know of those services, that they have no monitoring, follow-up or even records of outcomes, this is not a valid comparison since those services aren't even standardized themselves, even within a single clinic.

Not sure if everyone else missed that but as part of SMC, CBT and GET were offered. Psychologists and occupational therapists are obviously not part of specialist medical care, they are not even medical care. So basically SMC was a free-for-all of various things with zero consistency between participants.

It even says that number and timing of whatever those consults were varied between individuals, so zero attempt at standardization.

I guess that because of the glaring flaws this went unnoticed but this is all a big load of nonsense. Absolutely nothing can be concluded from this trial and that's before we even get to the LP part.

 

I can't see how this qualifies as research at all.
They should be made to return the funding.

 

Cohort Participation

This is my best effort currently at explaining cohort participation. There is some missing information and things don’t line up perfectly.

The “results” text explanation in for cohort numbers reads

This is [semi]consistent with "figure 1" but requires some explanation. Here is my best guess at particpation in the study in any capacity from randomized cohorts.


For SMC+LP, we have the 51 patients that started. 9 chose SMC post randomization, 3 withdrew consent, 1 followed-up after 6 months, 1 was missing data. 3 repeated from somewhere. This is 40 patients (37 originally).


For SMC, we have 49 patients that started. 2 chose SMC + LP, 1 chose LP, 2 withdrew consent, 1 followed-up after 6 months. This is 43 patients.


So, 43 + 37 patients were used in primary analysis, and slightly more completed the course but weren’t included.

These totals are similar to the totals referenced by the authors. That the 11 cohort swappers are termed “not-followed up” makes the most sense as exactly 11 in the combination of 9 and 2 are “not followed up” at 6 months.

This would also mean no one else was “not-followed up” on at 6 months, which is strange considering the follow-up numbers at 3 and 12 months.


There is another possibility, that the cohort swappers were included in the study, but exactly 11 patients were not followed up at 6 months, which is even stranger.

-----

The bottom box in figure 1, “included in primary analysis” from “randomized to x” columns seems incorrect.

How are 44 patients from 51 included in primary analysis if 9 were excluded, 3 withdrew consent and 2 had data missing? Even if the 3 patients that did LP after were originally from LP, and counted twice, we get 40 patients.


There is the obvious: the numbers seem to be more on opposite sides.


The patients that chose cohorts seem to be found, labelled as “not followed-up”. The problem is they are in columns in which they do contribute towards the sum total, presumably the number of patients randomized to x included in primary analysis. This is profoundly bizarre because the “withdrew consent” numbers do contribute towards the sum total.

I have never seen anything like that before, and it’s one thing I hope members can check me on.

 

I believe this represents the house visit stage. Patients filled out a consent to find out more information, then a researcher would come to their house to talk about the LP, if they accepted they would consent to the study and call a number that would randomize them to a group.

I agree, the population (Crawley's reptutation) is biased and, the authors delibrately avoided severe M.E., probably a good thing, but makes the results less good.

 

Screen testing a cast for a research-o-mercial in a clinical setting for credibility.

 

Primary outcome tables and figures

Crawley states 39 patients completed the SMC+LP. Figure 1 states that 39 patients received full SMC + LP. 44-46 patients completed SMC. Figure 1 states that 46 patients received SMC.


Yet, all non-text material in the paper references a 6-month primary outcome SMC+LP=n=45 or sometimes 44 or 46. This includes figure 2, the most important figure in the paper, “Mean SF-36 physical function over time”. SMC=n=38.



This is also true in table 2 referencing primary outcome data and their respective groups. This is perhaps strangest because it references outcome data in line with respective patient totals.


And in figure 3, mean eq-5d score, another important outcome/result/conclusion.


It is unthinkable that figure 2 and figure 3 could have swapped participant totals since those are often what people immediately look at when consulting results of a trial. Table 2 is incorrect reference data for the primary outcome. Was data swapped or mislabeled as well? Hopefully not, but it certainly leaves you with a bad taste. Averages are referenced in tables, but not the text, so that may be something I have to figure out at another time.

Again, if this is wrong, please let me know. I want to try to be correct about the criticisms.

 

Sorry about the image size I hope that's not a problem.

 

Imputed Participation

Not knowing exact cohort participation creates problems w/ outcomes. One thing that stands out is cohort numbers for imputed data.

Table 2, primary outcomes data references 100 exact participation for imputed data. This is impossible since there must be 103 participation rate here, unlike the patient flow chart, because participation refers to participant data points and not absolute patient totals.

Here’s another example, and why this actually needs to be fixed, we don’t know how many patients the imputed outcomes refer. If it was 100, or 103, the imputed outcomes included the patients that chose their own cohort after randomization. The imputed cohorts in the “results” and “conclusions” would be unrandomized. This is not mentioned in the text.

 

Maybe this is what Goldin was referring to in her comment about the trial not being truly 'randomized'?

I'm too tired to think this through now but it did look like there was something odd with those figures and I hadn't seen people mention this before.