SMILE trial data to be released

[Trish]

Is it that the 'missing' bit is that the protocol said they would collect school attendance data from the schools, but that in the end they didn't and just asked the kids to remember how many days they were at school? So the data missing because it was never collected.

 

[Lucibee]

@Amw66 and @arewenearlythereyet - I agree that an average attendance collected over many weeks before baseline would have been a much better way of establishing a baseline figure for attendance, but unfortunately, that's not what they did.

Age would also have been useful, but I think it was deemed too much info and likely to lead to identification of participants (presumably because it will have been coded as DoB, and converting to something less identifiable would take work).

Other things that would have been useful to know would also be when the baseline, 3mo, 6mo and 12mo measures were made - I suspect some data (ie attendance) were missing simply because they occurred during the school holidays.

It also would be useful to know when each participant received therapy - particularly the LP course itself. There's an indication that this may have anywhere between baseline and 6 months into the study. The timepoints for data collection were also fairly 'flexi' (I think the paper has more details).

 

[Graham]

I'm still working back through the original documents, but originally they did intend to use school records, then ended up just asking the kids. Trish is more or less right about the measure: the possible responses were ½day, 1 day, 2, 3 , 4, and 5 days. What I haven't found yet is what constituted a "day". If they went in to school on three days for some specific lessons, did that count as 3 days or did they expect the kids to work out what fraction of a day was involved?

 

[arewenearlythereyet]

@Amw66 and @arewenearlythereyet - I agree that an average attendance collected over many weeks before baseline would have been a much better way of establishing a baseline figure for attendance, but unfortunately, that's not what they did.

Age would also have been useful, but I think it was deemed too much info and likely to lead to identification of participants (presumably because it will have been coded as DoB, and converting to something less identifiable would take work).

Other things that would have been useful to know would also be when the baseline, 3mo, 6mo and 12mo measures were made - I suspect some data (ie attendance) were missing simply because they occurred during the school holidays.

It also would be useful to know when each participant received therapy - particularly the LP course itself. There's an indication that this may have anywhere between baseline and 6 months into the study. The timepoints for data collection were also fairly 'flexi' (I think the paper has more details).

I get the feeling it’s a ‘where do you start’ when it comes to critiquing this. I suspect that this is part sloppy part deliberate. Thanks for your review so far. I still haven’t got round to re-reading the paper and associated docs but I think there will be plenty to go through from the looks of it.

 

[JohnTheJack]

@Amw66 and @arewenearlythereyet - I agree that an average attendance collected over many weeks before baseline would have been a much better way of establishing a baseline figure for attendance, but unfortunately, that's not what they did.

Age would also have been useful, but I think it was deemed too much info and likely to lead to identification of participants (presumably because it will have been coded as DoB, and converting to something less identifiable would take work).

Other things that would have been useful to know would also be when the baseline, 3mo, 6mo and 12mo measures were made - I suspect some data (ie attendance) were missing simply because they occurred during the school holidays.

It also would be useful to know when each participant received therapy - particularly the LP course itself. There's an indication that this may have anywhere between baseline and 6 months into the study. The timepoints for data collection were also fairly 'flexi' (I think the paper has more details).

Yes, age and gender were omitted as part of the anonymization.

 

[Esther12]

I'm still not clear that there is anything missing. I understood @Lucibee to be saying that all the data are there.

Perhaps you could say exactly what in terms of my request you think is missing.

I asked for:

You asked for:

"School attendance in the previous week, collected as a percentage
(10, 20, 40, 60, 80 and 100 %)."

You got a form of this, however in the SMILE projects initial protocol (http://www.bristol.ac.uk/media-library/sites/ccah/migrated/documents/smprotv6final.pdf) it said:

“The primary outcome measure for the interventions will be school attendance/home tuition at 6 months. Secondary outcome measures will be school attendance at 6 weeks, 3 months and 12 months; the SF36 (physical function) at 6 weeks, 3 months, 6 months and 12 months and pain visual analogue scale at 6 months.”

“Children and young people are asked about school attendance and home tuition in a two item inventory. We will ask for consent to check school attendance using school records and will do this at assessment, 3 months 6 months and 12 months.”

You don't seem to have the data from school records. They could argue that your request was a bit ambiguous, but I thought I remembered them arguing that the data should not be released because participants could be identified from school records, so it would be a bit of a stretch for them to treat your requests as if it included this useful data when they were fighting against release, then as if it did not once they'd lost.

Is it that the 'missing' bit is that the protocol said they would collect school attendance data from the schools, but that in the end they didn't and just asked the kids to remember how many days they were at school? So the data missing because it was never collected.

I'm still working back through the original documents, but originally they did intend to use school records, then ended up just asking the kids. Trish is more or less right about the measure: the possible responses were ½day, 1 day, 2, 3 , 4, and 5 days. What I haven't found yet is what constituted a "day". If they went in to school on three days for some specific lessons, did that count as 3 days or did they expect the kids to work out what fraction of a day was involved?

I've got to admit that I've forgotten a lot of SMILE details, but I don't remember reading that they chose to not collect this data. The Buzzfeed article on SMILE includes quotes from Crawley that seem to indicate that the data were collected:

"In response, Crawley said "all the outcomes were collected as planned, but children didn’t like our recommended primary outcome, school attendance, so we used disability."

Crawley told BuzzFeed News it was possible that there was some placebo effect involved, but that the questionnaires she used in the trial asked questions about how far you can walk and how much school you attended, rather than simply whether people felt better. She added that self-reported school attendance lined up very well with the schools’ records of attendance.

https://www.buzzfeed.com/tomchivers/inside-the-controversial-therapy-for-chronic-fatigue

 

[JohnTheJack]

You asked for:

"School attendance in the previous week, collected as a percentage
(10, 20, 40, 60, 80 and 100 %)."

You got a form of this, however in the SMILE projects initial protocol (http://www.bristol.ac.uk/media-library/sites/ccah/migrated/documents/smprotv6final.pdf) it said:

“The primary outcome measure for the interventions will be school attendance/home tuition at 6 months. Secondary outcome measures will be school attendance at 6 weeks, 3 months and 12 months; the SF36 (physical function) at 6 weeks, 3 months, 6 months and 12 months and pain visual analogue scale at 6 months.”

“Children and young people are asked about school attendance and home tuition in a two item inventory. We will ask for consent to check school attendance using school records and will do this at assessment, 3 months 6 months and 12 months.”

You don't seem to have the data from school records. They could argue that your request was a bit ambiguous, but I thought I remembered them arguing that the data should not be released because participants could be identified from school records, so it would be a bit of a stretch for them to treat your requests as if it included this useful data when they were fighting against release, then as if it did not once they'd lost.





I've got to admit that I've forgotten a lot of SMILE details, but I don't remember reading that they chose to not collect this data. The Buzzfeed article on SMILE includes quotes from Crawley that seem to indicate that the data were collected:





https://www.buzzfeed.com/tomchivers/inside-the-controversial-therapy-for-chronic-fatigue

That may just mean they checked a few against the records. I don't think it is clear that for participants they matched actual records against self-report. Indeed, there wouldn't be much point in having self-report if they were all checked against the actual records. They could just use the actual records.

It seems to me that was more ambiguity from Crawley.

The school attendance records as in Z, AA, AB, AC are all there is I believe.

 

[Lucibee]

She added that self-reported school attendance lined up very well with the schools’ records of attendance.

@Esther12 - It's possible they just did a cross-check on a few records during the feasibility study. Once they are sure it all 'lines up', they then don't feel the need to keep asking the school for records. I'd say, fair enough. Having to gather attendance records from the school and then marry it up to each timepoint for each participant is far, far more work for them than simply relying on self-report at the time of the questionnaire - so if they could avoid it, I can understand why they would do that.

 

[JohnTheJack]

@Esther12 - It's possible they just did a cross-check on a few records during the feasibility study. Once they are sure it all 'lines up', they then don't feel the need to keep asking the school for records. I'd say, fair enough. Having to gather attendance records from the school and then marry it up to each timepoint for each participant is far, far more work for them than simply relying on self-report at the time of the questionnaire - so if they could avoid it, I can understand why they would do that.

It may be also that they weren't allowed access to the school records and asked schools to check a few samples.

 

[Amw66]

It may be also that they weren't allowed access to the school records and asked schools to check a few samples.

In Scotland all hell broke loose over illegal data sharing between schools and third parties ( including health , social services) a few years ago. There has been a significant court judgement regarding this and some parents are pushing for and enquiry.

Politically it is too sensitive and i doubt this will happen.

It may be that some initial data pre GDPR 2018 becoming fully enforceable , was obtained in the feasibility trial, but the mechanism to obtain data for full trial may have become illegal.

Note - i don' t know if this is applied differently in England and Wales.

 

[Esther12]

That may just mean they checked a few against the records. I don't think it is clear that for participants they matched actual records against self-report. Indeed, there wouldn't be much point in having self-report if they were all checked against the actual records. They could just use the actual records.

It seems to me that was more ambiguity from Crawley.

The school attendance records as in Z, AA, AB, AC are all there is I believe.

@Esther12 - It's possible they just did a cross-check on a few records during the feasibility study. Once they are sure it all 'lines up', they then don't feel the need to keep asking the school for records. I'd say, fair enough. Having to gather attendance records from the school and then marry it up to each timepoint for each participant is far, far more work for them than simply relying on self-report at the time of the questionnaire - so if they could avoid it, I can understand why they would do that.

It may be also that they weren't allowed access to the school records and asked schools to check a few samples.

That could all be true, but there was no mention of it in any of the SMILE papers. They said that they would check school attendance using school records, and in the media Crawley has claimed that they did so and that this validate the self-report outcome they reported result for, but we have no data on this. That just seems weird to me, especially considering all of the other problems around this trial. It also wouldn't surprise me if LP did lead to an increase in social desirability bias on self-reported school attendance. If they have data from school records why have they not reported anything from it?

 

[Graham]

I've got to admit that I've forgotten a lot of SMILE details, but I don't remember reading that they chose to not collect this data. The Buzzfeed article on SMILE includes quotes from Crawley that seem to indicate that the data were collected:

So far I have read that they had problems with sixth-formers who were only required to attend appropriate lessons, which was why they proposed to change their primary measure from school attendance to sf-36. (page 10 of the feasibility study)

As an ex-secondary school teacher, I have no idea how they would collect accurate records of children's attendance if they were part-time, coming in for specific lessons. Registration takes place first thing in the morning and first thing in the afternoon, and a separate book would be kept in the office for pupils signing in and out (obviously, for fire checks, there has to be a check), but who would go through this to provide the figures, I have no idea, let alone whether it would be approved.

In the statistical analysis plan (page 7) they state that they "have obtained consent to check school attendance using school records at 3, 6 and 12 months" but do not say whose permission was obtained. I suspect it was from the parents, and then they found that they couldn't get it from the schools.

Can kids who have been put under pressure by the Lightning Process to deny their illness, be relied upon to give accurate answers to the questions on school attendance in the previous week? These are kids whose idea of a survey is often "How much TV do you watch? (a) not much (b) about average (c) lots.

 

[Amw66]

H

So far I have read that they had problems with sixth-formers who were only required to attend appropriate lessons, which was why they proposed to change their primary measure from school attendance to sf-36. (page 10 of the feasibility study)

As an ex-secondary school teacher, I have no idea how they would collect accurate records of children's attendance if they were part-time, coming in for specific lessons. Registration takes place first thing in the morning and first thing in the afternoon, and a separate book would be kept in the office for pupils signing in and out (obviously, for fire checks, there has to be a check), but who would go through this to provide the figures, I have no idea, let alone whether it would be approved.

In the statistical analysis plan (page 7) they state that they "have obtained consent to check school attendance using school records at 3, 6 and 12 months" but do not say whose permission was obtained. I suspect it was from the parents, and then they found that they couldn't get it from the schools.

Can kids who have been put under pressure by the Lightning Process to deny their illness, be relied upon to give accurate answers to the questions on school attendance in the previous week? These are kids whose idea of a survey is often "How much TV do you watch? (a) not much (b) about average (c) lots.

Here there is a coded system. with specific codes for authorised and unauthorised attendance - separate codes for medical appointments etc.
It can break down if you are on a reduced timetable - there is a signing in and out process at the front desk and this data is entered onto the system. Sixth year have their own attendance taken.
If kids have anxiety, or are prone to sensory overload and the front desk is busy, they will avoid signing in and out - so they can be in but marked off -another problem.
If you go home after an hour does that still count as half a day?

Note - here schools cannot now share data without parent's written consent. So here it could have been the other way around.

The elephant in the room that is never addressed is - are these kids capable of doing much when they are at school.
Attendance doesn't count for much if you struggle doing anything much once there. Many are simply being set up to fail.

 

[Graham]

If you go home after an hour does that still count as half a day?

If you go in for three half days, what does that count as?

 

[Barry]

If you go in for three half days, what does that count as?

A penny ha'penny.

 

[large donner]

@Esther12 - It's possible they just did a cross-check on a few records during the feasibility study. Once they are sure it all 'lines up', they then don't feel the need to keep asking the school for records. I'd say, fair enough. Having to gather attendance records from the school and then marry it up to each timepoint for each participant is far, far more work for them than simply relying on self-report at the time of the questionnaire - so if they could avoid it, I can understand why they would do that.

On the other hand unless you use objective information in studies its not acceptable to just say, "we couldn't get the objective info so we used subjective information".

Can you imagine if all parents had to do to insure their kids went to school whilst they were out at work was say, "did you go to school today".

In a situation wherby its not possible to design a study with objective info the outcomes are likely not able to stand by any conclusion.

The protocol promised school attendance records as a primary measure (regardless of the flaws in what that would show in terms of severity of disease, relapse etc), the records are about as objective as any study could wish for in terms of factual information, they should have delivered accordingly.

If attendance was to be used as a measure then the actual school records should have been used not just asking kids if they went to school or not.

 

[Wonko]

Can you imagine if all parents had to do to insure their kids went to school whilst they were out at work was say, "did you go to school today".

Things may have changed but when I was 'at school' I wasn't, a lot of the time, and i was never asked about it unless I got in some form of trouble that drew attention to my not being at school. Do parents really ask their kids if they went to school today?

 

[large donner]

Do parents really ask their kids if they went to school today?

No, Esther Crawley does that.

 

[Sean]

"In response, Crawley said "all the outcomes were collected as planned, but children didn’t like our recommended primary outcome, school attendance, so we used disability."

Kinda like PACE's (initial public) justification for ditching actometers being that they were too much of a burden to patients?

To be honest, I don't give a fig whether the kids liked it or not. The important questions are 1) is it an unduly intrusive or onerous measure, and 2) is it an appropriate, reliable, and useful measure?

No, and yes, would be my answers.

I don't like the only primary outcomes in PACE being unblinded subjective self-report, and all objective measures being dumped, downgraded, or just ignored. I really really don't like that. But I don't remember that preference being taken into account, ever.

 

[Hutan]

2) is it an appropriate, reliable, and useful measure?

As an ex-secondary school teacher, I have no idea how they would collect accurate records of children's attendance if they were part-time, coming in for specific lessons.

I'm with Graham; school's records of attendance are probably a whole lot worse than you might first imagine. One of my son's schools had an automated system where the parent has to approve an absence by text if an absence is registered - more than half of the time, my son's absence wasn't recorded. The more absences he had in a term, the lower the chance of them being recorded as teachers gave up bothering to report it.

At the end of one term, when my son was only getting in for about a quarter of the time, was struggling to walk up stairs and certainly hadn't attended any physical education classes, the section in the report from the PE teacher noted that my son was showing pleasing progress but was quite quiet.

Of course self-reported attendance is more hopeless, particularly if you've been given the message that absences are a sign of not trying hard enough. My experience with questionnaires given to boys is that they have little interest in filling them out, so I'd bet there were a lot parents doing the estimation of school attendance. And if you are the kind of parent who thinks putting your child through the lightning process might be a good idea, then there's a good chance that you might want to convince yourself that it did work.

Also note that 40% of the data are missing (in both groups) by the time we get to 12 months.

That alone, I would have thought, makes the result useless. What was the percentage of data missing at 6 months?