I came across this tweet. Where someone was doing research that they thought could be controversial (gender bias in hiring) and so asked others to be a red team to look at the methodology. The red team terminology comes, I think, from cyber security where you basically hire a team to attack your system to see how far they can get with an attack. It does feel a good idea for research methodology - particularly when deviating from the norm. Although if I was following a cyber type methodology I think there are simpler, less human intensive things that could be done - following an equivalent of a threat analysis where you set out your system in detail, look for potential threats (often using standard threats) and show how you mitigate these threats. If the PACE authors had taken this approach they would never had followed the protocol they did. They would of course had to change things to deal with all the flaws discovered. But then if you look at early comments you could argue patients (particularly @Tom Kindlon ) told them early on about the flaws and they ignored this. https://twitter.com/user/status/1723936087414812923
Even the level of risk assessment I had to do when taking fully functioning adults to a public building would have consigned it to the bin! But it'd be a great idea to come up with a risk checklist. The same criticisms about diagnostic criteria, control-matching, cohort sizes, the way results are presented, and statistical stuff that's beyond my understanding come up in discussions here all the time.
Red teaming is common in technology fields. It's a great process, it's very effective at poking holes in models, and making fools out of people who make assumptions. It's a big reason why technology fields are so successful: criticism is all over the place, accepted and used constructively. I'm not sure it'd work out in healthcare because there is too much groupthink and conformity involved, they all think too much alike. The only way this could work would be having patients, or at least people who aren't healthcare professionals, play the hostile. And that'd be pretty controversial. And very effective. It would massively improve outcomes IMO, but it goes against one of the main principles of healthcare: doctors know best. A principle that is often wrong, largely because there is too much groupthink and conformity. And it shouldn't be a one-and-done thing, it needs to be a continuous process, one that allows patient complaints to be properly recorded, all of them. And that would also be very controversial, as we know it's a common attitude in medicine to view patient complaints as being invalid and unworthy of attention. They're viewed as insulting, as childish as it may be. I keep seeing any bit of criticism of research being framed as personal attacks, it's a serious problem out there. Imagine how a red team process would be viewed. There has to be a shift in attitudes to get there. Maybe red-teaming in research methodology could lead there, but without the counterpart where complaints are taken seriously after-the-fact in the real world, all the flaws and limits of research remain. It's largely because complaints about psychosomatic medicine / the biopsychosocial model aren't allowed that it continues to dominate and fail. I don't think you can fix that in research alone, because far too often the public version is misleading on purpose, and happens very differently when the product is out there in the hands of clinicians. It's fundamentally what psychosomatic ideology is built out of, so moving towards that would spell its end. Deep down I think a lot of HCPs understand that, and those whose career depend on its continuation would fight like hell to protect their domain, as would many who just believe in it. We know that all too well.
On paper, a red team seems to be a great idea, but in practice it could run into the same issues as peer review: the red team could be made up of like-minded scientists (not unlike choosing a journal to publish in that is favorable to the content of the article) and there remains the problem of an imbalance of power such that criticizing someone’s work (especially when it is as a long and involved study) may be detrimental to one’s career. This especially applies to data sleuths who, from what I have seen, tend to be young and less academically advanced than eminent scientists whose work they may come to review. Therefore I am not sure that a red team would have prevented the problems with the PACE trial. It surely wouldn’t have included citizen scientists who were vocally critical about the trial’s methodology like @Tom Kindlon, but more likely members of an academic “clinical trials unit” department from an institution affiliated with the authors (King’s College, University of Aberdeen, etc) that might have refrained from raising (too many) red flags.
But there's not reason patients shouldn't flag issues that come up again and again in trials, but because there's no feedback mechanism to the researchers, they just keep happening. One of the worst problems is the diagnostic criteria. I understand that researchers want to be able to compare their results with previous trials, but those using criteria not requiring PEM are probably pretty useless anyway—they could well include people who don't even have ME. A line has to be drawn under that history and stay firmly in place until we have a clearer, or additional, diagnostic indicator.
I think one of the things about red teams is that they are paid and funded to find flaws which is very different from peer review which is best effort comments from someone who should know the area. Those in red teams do have a very different attitude to those who build systems though. I think if you are getting to the stage of paying for a red team you should take it seriously - however, I see with patient co-production that some medical professionals/researchers will go through the motions but not really take input. The point about the medical profession culturally not taking criticism well is important. I suspect this is central to many problems but this is also with IT products things have changed over my career to processes where we get others to review, apply analysis tools, red teams and bug bounties etc to try to get better products - its expected now (and demanded by government (or people like NIST - US standards body)). So I think change can happen but it is slow.
