New Zealand: Dr Matthew Phillips, neurologist

[rvallee]

This is wild to me. There is so much gray area in medical research, so many unknowns. Why in the name of all that is holy does trial design remain so muddled? Surely that is something that can be pinned down. Seems like such a massive drag on progress.

The purpose of a system is what it does, there is no point in claiming that the purpose of a system is to do what it constantly fails to do.

There just isn't any interest in rigorous trials that give clear "no" answers to almost every question. It might be the stated purpose of those systems, but it's not what they actually do.

 

[bobbler]

I have tried to look up the current education programmes for medical doctors in Norway. I have not been able to found any direct mentions of trial methodology. I also hear from people that have gone trough the training or worked/studied with those people that they really don’t learn anything about it.

It might be mentioned in some lectures, but I’m fairly certain it’s not emphasised based on those anecdotes, at least not to the extent that it should.

It also aligns with my experience from economics. Our methodology class for our master’s thesis essentially told us you have to write that something might be this way instead of that it is this way, but qualitative research was presented as a perfectly valid method and not as one with substantial limitations.

Agreed. there is spoon-feeding to exams in most subjects and sadly for example when learning statistics they are taught to be more concerned about ticking the box on the p-number than on why that p is even used in the first place. Or who needs to be in that sample for there even to be any point about calculating it.

I don't think a single behaviourist for example would be able to explain what external validity of their sample (vs a population) is.

I do think that such things should be known and it should be not just a source of shame/embarrassment when they get it wrong or make a mistake that wasn't an honest I feel so mortified typo error done every so often rather than trying it on, and should have that written all over the literature.

But there was no shame when their therapist-delivered literature was analysed as low and very low quality by Nice, just them playing the victim and focusing on targeting patients - again - with slander instead of any intention to be any better themselves at what they claim to be capable of doing.

 

[rvallee]

And a big part of this ongoing production of poor quality trials must be that these designs produce positive outcomes, even for ineffective treatments. If you already believe the treatment works and just want a trial result to promote it

It's become so normalized that published protocols can simply state it and no one cares, they'll still pretend there is any "equipoise". From Efficacy of blended digital and face-to-face psychotherapy compared to enhanced psychotherapy for patients with [SSD] (iSOMA+): 2026 Jutzi et al:

Nonetheless, the superiority of blended psychotherapy is assumed, given the multimedial and interactive design as well as further persuasive, that is, engagement-enhancing, features of iSOMA (i.e., automatic reminders, progress bar, unlocking achievements), compared to passive bibliotherapy

It's become a game of persuasion, it's not even subtle anymore. It's become normalized to try to get people to change how they think, then asking them what they think, which might not even be what they actually think, but it looks good enough on a biased questionnaire, as long as no one thinks about it.

Like I say often, this is far bigger than just us, or even neglected/discriminated illnesses. It's become the purpose of the system to fake positives. It "gives people hope", or whatever.

In almost every other industry this would be considered fraudulent practices. In health care it's just become "the way things are done". It's all part of the same game of persuasion, and now is more persuaded of this than the people who control, influence and work in those systems.

 

[bobbler]

Agreed. there is spoon-feeding to exams in most subjects and sadly for example when learning statistics they are taught to be more concerned about ticking the box on the p-number than on why that p is even used in the first place. Or who needs to be in that sample for there even to be any point about calculating it.

I don't think a single behaviourist for example would be able to explain what external validity of their sample (vs a population) is.

I do think that such things should be known and it should be not just a source of shame/embarrassment when they get it wrong or make a mistake that wasn't an honest I feel so mortified typo error done every so often rather than trying it on, and should have that written all over the literature.

But there was no shame when their therapist-delivered literature was analysed as low and very low quality by Nice, just them playing the victim and focusing on targeting patients - again - with slander instead of any intention to be any better themselves at what they claim to be capable of doing.

Then there is the fact that even basics which would get you a fail on an undergrad project class or stats exam of not checking whether data is parametric or non-parametric when choosing which test to use - and those aren't final year tests or knowledge either - ended up being used by one big project didn't it without anyone pulling it up or checking

I wonder whether those reading it were understanding enough of what it meant to just pass it off as 'ahh stats, I don't like it, a test is a test, they are close enough' or the issue was so basic that noone checked? and because of this assumption the working is not required to be spelled out so it is easily checked without a reader downloading the data and having to do all the working themselves to see?

 

[rvallee]

It was not something you got in lectures or text books. It was something you absorbed from day to day involvement with evidence analysis. Maybe now that has been replaced by 'pathways'.

It seems to place the change there, then. It's learning through culture, rather than formal, and the culture has moved to want to lower standards. Which seems kind of expected, frankly. Something this critical being informally taught is bound to cause issues like this.

Now, promoting a small % of positive anecdotes, presented with 'hope' and "mind over matter" magic as "evidence-based" gets applause at conferences and awards. The culture has spoken in favor of this, and with nothing formal to stop, or even slow, the decline, well that explains the race to the bottom. There's something natural about people describing the emperor's new clothes yet again that to make their descriptions interesting, they will only get more flowery over time, otherwise there just isn't much to say.

It's an influencer economy now, where controversy is currency. Pseudoscience like this used to get dismissed without giving it a first thought. Now we get absurd nonsense about sniffing chocolate while lifting weights and firmly-held beliefs that we must, MUST, somehow expect bad things to happen, even if we don't, because that's what the models say, or whatever.

 

[Hutan]

I guess we have someone here to help us answer some of our questions. @Matthew Phillips, I'm genuinely very grateful that you are here, and I think you can give us some insight.

I see your job title:
Clinical and Research Neurologist, Director of Neurology at Waikato Hospital
I see that you are an Honorary Senior Lecturer at the Auckland Medical School.
so, the issue must surely not be lack of sufficient intelligence or lack of exposure to information on good clinical trial design.

Do you understand that the flaws in your Alzheimers trial mean that a null result on the objective outcome and small improvements on some subjective outcomes is evidence that your ketogenic diet intervention didn't work? If not, how do you come to a different conclusion? Do you still think that the trial was well designed and the outcomes accurately interpreted?

How certain were you when you planned your trial that it would show a benefit? How certain are you now that the diet is helpful?

Who did you consult when you were planning your trial? Did anyone at all say to you 'your trial participants and partners are really going to want to report a benefit, aren't you worried that their enthusiasm and hope will bias your results? Perhaps you need an active control'? Did you think 'I have an objective outcome, so things will be fine', and then, when you got the results and the objective outcome showed no benefit, did you think 'oh well, it just must be that the trial wasn't long enough to move the dial on that outcome, so it can be pretty much ignored'?

Where does your certainty that a ketogenic diet is helpful for Alzheimers come from? I've watched a presentation you made at the Low Carb Down Under conference on a case study of a man with ALS with the ketogenic diet, and I can see how the personal contact with a grateful patient who seems to do better for a while might create a sort of evangelism, a belief that you can help people with awful diseases (with that case potentially outweighing the cases of all the patients who also tried the dietary intervention and who did not see a benefit). Now, I don't know if a ketogenic diet is useful for people with ALS, I haven't looked at the evidence, but perhaps the belief you have developed in low carb diets means that you didn't look at your Alzheimers trial results with equipoise?

What do you think happened with your Alzheimers trial?

 

[Concentric_Spiral]

Naive question but is there anything that can be done to systematically discredit bad science or influence the raising of research / publication / treatment standards? (Cue the joke about the new intern blithely asking, 'have you considered raising revenue?') It seems like this forum could put together a damning meta-analysis. After all, the governments that fund and implement this stuff are getting fleeced.

(If it wasn't unethical / impractical / probably unhelpful / etc. I half-like the idea of a sensationalist approach: run a trial using the same design to get a false positive for a fake intervention.)

 

[Concentric_Spiral]

Yes, it is incredible to me that this is allowed too.

I think part of the problem is that Ethics Committees seem reluctant to scrutinise the scientific merit of a research proposal, failing to recognise that a poorly designed trial can do much much more harm to the public than any direct harm to the trial participants.

I think part of the problem is this odd but prevailing belief that if it is hard to blind a treatment, then it is fine to make a trial that is unblinded and does not have objective outcomes. It's just 'what everybody does' when it comes to non-drug treatments. I think there is a lack of understanding about the harms that non-drug treatments can do.

I guess there is the issue that there are people who aren't trained in research methods and think they have some special insight about a treatment and have the means and authority to conduct trials without getting good advice. The enthusiasm for the treatment seems to override any empathy that would allow them to put themselves in the place of the people participating in the trial, to understand the biases participants will have when making a subjective outcome report.

And a big part of this ongoing production of poor quality trials must be that these designs produce positive outcomes, even for ineffective treatments. If you already believe the treatment works and just want a trial result to promote it, if you will benefit in terms of reputation or financially from a positive outcome, and if everyone around you is fine with a reliance on subjective outcomes for unblinded treatments, why would you make your trial design rigorous?

Basically, they do it because nothing is stopping them, not their knowledge of research design, not their desire to actually find the truth, not their concern that they might be judged incompetent and certainly not the presence of people and structures that prevent poor quality misleading research from happening.

In this particular case, the Goodfellow Unit is promoting the outcome of the Ketogenic diet trial. The Goodfellow Unit (there to educate GPs) is not going to be the source of quality control either, because it is run by Bruce Arroll, the proponent of the Lightning Process and CBT for CFS. He has a big stake in the ongoing use of subjective outcomes in unblinded trials to prop up ineffective treatments.

The way you describe this is chilling. Seems like a massive regulatory failure.

 

[Jonathan Edwards]

Something, maybe multiple things, are happening to nullify basic intelligence.

It is called human nature.
Except that there are some people who call it psychology. And they are the people who are worst at understanding how it affects trials (and I wouldn't be surprised how it affects everything else).

 

[Theresa]

So now you’re saying that memory and attention is predictive?

Yes, of course I had that from spending days at high altitude.

But you’re missing the point: that if fatigue is predictive, and there suddenly was a rapid change in what my muscles were able to do due to increased oxygen delivery when I went down, then you’d expect me to feel fatigued because the brain got signals it didn’t expect. I felt the opposite.

And this is the exact same situation as when people you claim have FND get fatigue:
You claim there is nothing wrong outside the brain, but they expect something to be strenuous and therefore they get symptoms because the expectations do not match what the body actually experiences.

So why didn’t I get fatigue in that same scenario?

No, you’re actually saying that it explains everything that goes on in the brain because you said that predictive processing is how cognition works.

You can’t just say that predictive processing is the normal model for how things work, and then say that the normal model explains abnormal results (symptoms).

This is a very good point. And if it isn’t predictive processing in the early stages, why can’t it just be whatever it is in the early stages all the time?

Sometimes the functional people say that the predictive processing can be unconscious and based on prior experiences but I don't know if it is plausible to think you can consciously expect one thing while your brain expects something else? (I doubt it is). Or if it did how you could consciously alter it by changing expectations?

Regarding predictive processing being the normal model FND people usually say the topdown signalling has become abnormal and is overwhelming the normal sensory input but I don't know if this makes any sense scientifically?

 

[Jonathan Edwards]

I don't know if this makes any sense scientifically?

Nope. It is just a handwaving get out.

 

[rvallee]

Sometimes the functional people say that the predictive processing can be unconscious and based on prior experiences but I don't know if it is plausible to think you can consciously expect one thing while your brain expects something else? (I doubt it is). Or if it did how you could consciously alter it by changing expectations?

Regarding predictive processing being the normal model FND people usually say the topdown signalling has become abnormal and is overwhelming the normal sensory input but I don't know if this makes any sense scientifically?

Psychosomatic ideology has never bothered with plausibility. Throughout the decades, going back to the earliest days, when the same models that are around today were the exact same set of ideas framed slightly differently, it simply never mattered. This is why the modern messaging is entirely around creating narratives. It's all they have. Narratives about themselves. The narratives have nothing to do with us, it's just what they imagine is a plausible explanation for things. Not much different than a creation myth about a giant cow licking a salty cup, or whatever.

Spirits, demons and other esoteric stuff used to get the blame. In the last century everything's been put on invisible ghosts in the machine, shadows of imaginary fears and traumas, even where they never actually happened, because ultimately nothing matters. It's been decided that this is the way things are done, and everything's been done to disable all the alarms and safeguards against failure. And for a few decades they truly have all been turned off. There is not a single working alarm system left, the system has completely disabled everything that can stop bad ideas.

I watch a lot of engineering videos for fun. Not my thing, I have no expertise in it, but it's really interesting. And in engineering it's constantly emphasized how learning from mistakes is critical to everything they do. They will make mistakes, it's normal, they accept it as part of what they do, and they make up for it by constantly learning from those. And they measure everything, whereas health care is shifting more and about towards beliefs and asking the equivalent of primary school crush "do you like me yes[ ] no[ ]?" questionnaires.

Health care has chosen the opposite direction: they never acknowledge mistakes if they can avoid it, which is usually the case, and so never learn from them, never grow. Hence the stagnation outside of cutting edge science, from the "biomedical model", one of the most successful scientific models in human history. Mostly because engineering mistakes are expensive, and so the people who pay for it want to avoid them. It's never about the lives that are lost in disasters, it's always about the costs, the losses. There is no such mechanism in health care. In fact it works exactly to amplify all the mistakes and all the harms. This is how we get to a place where the /r/CFS subreddit decided to ban suicide notes, which ends up covering up the horrible negligence from the health care industry. All the mistakes in health care get washed away, hidden, covered up, very often gagged under threat of penalties. It's all so damn nihilistic.

 

[Theresa]

It is hard to understand how ordinary neurologists not involved in the treatment of functional disorders stay silrnt rather than saying the functional explanation makes no sense.

 

[Utsikt]

Maybe things have changed in the last twenty years. I would not be surprised. But when I was involved weekly journal club sessions where bd trial methodology was pulled apart were routine. We all knew that trials were randomised, controled and double-blind in general and we all got to understand why. That process sets you up for understanding why bad rials are bad - if you have common sense.

It was not something you got in lectures or text books. It was something you absorbed from day to day involvement with evidence analysis. Maybe now that has been replaced by 'pathways'.

I don’t know what goes on outside the official curriculum.

Naive question but is there anything that can be done to systematically discredit bad science or influence the raising of research / publication / treatment standards? (Cue the joke about the new intern blithely asking, 'have you considered raising revenue?') It seems like this forum could put together a damning meta-analysis. After all, the governments that fund and implement this stuff are getting fleeced.

(If it wasn't unethical / impractical / probably unhelpful / etc. I half-like the idea of a sensationalist approach: run a trial using the same design to get a false positive for a fake intervention. Edit: ChatGPT proposed: "a simulation study. For example: 'We generated thousands of simulated trials with no true treatment effect but with the design characteristics common in this field. How often did they produce statistically significant positive findings?'" I have no idea what that means or if it makes any sense.)

I don’t think there are any failsafe ways to do that because it will always involved the judgement of humans.

The issue, at least to my mind, is that too many of the people are incompetent and too many do not care enough because the entire system rewards quantity and not making a fuss about anything.

It is hard to understand how ordinary neurologists not involved in the treatment of functional disorders stay silrnt rather than saying the functional explanation makes no sense.

They either don’t understand why it doesn’t make any sense, don’t care that it doesn’t make any sense (it’s a convenient way to deal with the bothersome patients), or don’t want to ruffle any feathers.

 

[Eleanor]

It is hard to understand how ordinary neurologists not involved in the treatment of functional disorders stay silrnt rather than saying the functional explanation makes no sense.

Some of them believe that it improves outcomes for patients to be given a comfortingly confident-sounding explanation of their illness rather than an honest answer that includes the words "we don't yet know". The idea is that uncertainty in itself causes distress and distress gets somatised into symptoms, therefore certainty is therapeutic even if it's false.

(Others just think that the patients are consciously or unconsciously faking, so it doesn't matter if the explanation isn't real because the illness isn't real either.)