Should research teams include some engineers?

[wigglethemouse]

Some of the discussion seems to be about how the engineering industry uses checks, tests, and reviews to make sure projects stay on track and are reliable. Does this seem to be missing in some medical research areas?

In Engineering I even had a project cancelled and re-specified which I thought was great rather than proceed and produce something that wouldn't sell well.

 

[jnmaciuch]

Some of the discussion seems to be about how the engineering industry uses checks, tests, and reviews to make sure projects stay on track and are reliable. Does this seem to be missing in some medical research areas?

Nope, in many cases you’re even required by the funding body to provide proof of progress and research plans to get the next installment of funding for the upcoming year. Not to mention department performance reviews, study sections for grant awards, IRB protocols, research ethics committees, peer review, peer feedback at conferences/poster sessions, and all the other places where your work is scrutinized.

I think it’s easy to assume, based on the presence of so much bad science, that these review mechanisms must not be as robust as in other fields. But honestly they're not any less comprehensive than the reviews all my close friends describe being employed in the tech industry. Even more stringent, in many ways, judging by the sheer amount of work to keep up with reporting requirements.

The realization I came to was just that most of the mechanisms any rational person can dream up to Fix Science often don’t seem to actually prevent bad science, and not for any one easily understandable reason.

 

[wigglethemouse]

So perhaps at the end of the day it comes down to people. There are a relatively large amount of not very competent people in every walk of life, or said another way, a bunch of very average people who we'd prefer they think a bit more about what they are doing and what it means.

The best bosses I've seen are the ones that can get to the real information / issue within 3 questions. Those people are pretty rare. Maybe we just don't have enough people asking the right questions, and the ones doing the work that can provide an answer to satisfy such a boss.

I'd love to have a Dr Maureen Hanson as a boss if I was a young Scientist. In fact I've heard other aspiring researchers say the same thing. Mentorship, direction,good decision making, and making sure I stay on track. Pretty rare.

Not to mention department performance reviews, study sections for grant awards, IRB protocols, research ethics committees, peer review, peer feedback at conferences/poster sessions, and all the other places where your work is scrutinized.

Yes, I've heard the same. In some cases perhaps too much bureaucracy preventing progress.

 

[jnmaciuch]

So perhaps at the end of the day it comes down to people. There are a relatively large amount of not very competent people in every walk of life, or said another way, a bunch of very average people who we'd prefer they think a bit more about what they are doing and what it means.

The best bosses I've seen are the ones that can get to the real information / issue within 3 questions. Those people are pretty rare. Maybe we just don't have enough people asking the right questions, and the ones doing the work that can provide an answer to satisfy such a boss.

Yup, I'm with you on that. It's also a thing that seems really hard to gauge for yourself. Tons of people delude themselves every day into thinking they're the pinnacle of rationality and efficiency, even manage to convince others that they are gods gift to critical thinking—scientists are probably especially prone to skewed self assessments.

I think you’re right on the money that it is a rare skill. Probably one that can be cultivated to some extent, but maybe not reliably.

 

[Hutan]

So perhaps at the end of the day it comes down to people. There are a relatively large amount of not very competent people in every walk of life, or said another way, a bunch of very average people who we'd prefer they think a bit more about what they are doing and what it means.

I guess this is it, really. It's true that a lot of medical research is surprisingly bad, even when it's not about ME/CFS. Incompetence, and also the bias created by the prospect of financial returns.

There are probably some ME/CFS specific issues too.

It is not a prestige area. If you are a bright medical scientist, there are questions to answer that will see you better paid, better resourced and better recognised, questions to do with cancer for example. Only those with a personal connection to the disease or at least a good understanding of the need are likely to plow on with ME/CFS regardless. On average, our researchers probably aren't as good as the ones addressing more prestigious diseases. I think that also probably applies a bit to our funders and charities. In many cases, it's just whoever puts their hand up to do the (volunteer/poorly paid/low status) work.

Money to some extent could counter the lack of prestige - but, we don't have a lot of that, as ME/CFS hits young people of working age, and hits them with physical incapacity and shame-inducing stigma. People with ME/CFS and their families have tended to lose any power - financial, influential - that they had. People like many here at S4ME who call for things to be better tend to struggle to gain decision-making roles, even within our own charities. We don't have the well-resourced institutions like the cancer research institutes.

There is also prejudice. That affects the questions ME/CFS researchers want to answer, and to a large extent get paid to answer - e.g. Is it perfectionism? Will a mindfulness course fix it? How can we implicate cortisol in the pathology? Why don't we measure cytokines in blood again?

Sorry, off topic. To answer the question - yes, engineers might be useful for some research questions. But, there are skills and personal qualities more important than what degree the person did when they were young.

 

[poetinsf]

I think that would be useful. Immunology and neurology in particular are control systems in some ways, sometimes complete with feedback loop. I'm sure people in those fields have the knowledge of systems, if not a formal training in systems engineering. But medical people are often so focused on the molecular level that they may fail to see the obvious. Hence they propose theories like muscle anomaly or virus or energy production dysfunction etc. for ME/CFS without ever explaining PEM.

Delayed response to, and recovery from, exertion is everywhere. Why not take the existing process, fit it to ME/CFS and see if it explains/predicts? This is one place they could use engineer's perspective, at macro level. If the model works, then a theory can be formed in accordance with the correspondence principle. Further biological research could follow to confirm.

 

[Hoopoe]

I think it’s easy to assume, based on the presence of so much bad science, that these review mechanisms must not be as robust as in other fields. But honestly they're not any less comprehensive than the reviews all my close friends describe being employed in the tech industry. Even more stringent, in many ways, judging by the sheer amount of work to keep up with reporting requirements.

My views have been influenced by the PACE trial, the difficulty replicating various findings in ME/CFS, by how widespread manipulation is in research, and the rigidity and inefficiency of the world of scientific publishing. I'm looking at these things from the outside and thinking there are big problems.

I think that emphasis on reliability would make research more useful for humanity. Not because it leads to better hypotheses, more creativity or deeper insight, but because it makes the results more reliably true and more replicable. This should ultimately reduce waste.

Bioinformatics is closer to the IT sector than other sciences and I suspect it has adopted some of its culture.

 

[Creekside]

Immunology and neurology in particular are control systems in some ways, sometimes complete with feedback loop.

I think the "sometimes" is inaccurate. I expect that most biological systems involve multiple feedback loops, some short and simple, some long and convoluted. That's why I suggested that an engineering perspective would be helpful. The scientist perspective probably doesn't focus on feedback loops, while the engineering one does.

How many feedback loops are involved with getting out of bed in the morning? Are myelin sheaths being restructured on specific neurons in that process? Which hormones are being adjusted during that process? What changes in signals to the gut are being sent in preparation for the expected increase in demand for energy and cellular repair materials? Mitochondrial numbers in various cells are being altered, based on local signals. Lots of feedback loops involved.

 

[jnmaciuch]

The scientist perspective probably doesn't focus on feedback loops, while the engineering one does.

I think that’s a false assumption—all of my biology training to date, even the courses that weren’t explicitly systems biology, had no shortage of thinking about feedback loops, regulatory mechanisms, etc. The general principles of systems dynamics have been a standard part of advanced bio curriculums even if it isn’t explicitly labeled with systems dynamics terminology.

It’s not that biologists aren’t used to thinking in terms of systems. Modeling biological systems with mathematical equations is also extraordinarily common, I can think of half a dozen researchers doing it at my institution who don’t even have a physics or math background. It’s just that biological systems tend to spit in the face of most system models unless you’ve reduced the system ad absurdio such that it may no longer resembles what happens in the body.

[edit: or, that despite all this training, some people still aren’t going to come up with good ideas]

 

[wigglethemouse]

Do you think this describes what we are talking about.

The IBI team is comprised of experts in the field of kinetic modeling, experimental data analysis, biological computer simulation, and software development. Our Chief Scientific Officer and co-founder, Dr. Robert Phair, is an internationally known expert in the area of kinetic modeling, with over 35 years of experience in the modeling of complex biological systems. While we currently focus on cellular and molecular systems, we also have years of experimental and theoretical experience at the tissue, organ-system, and whole-human levels of biological organization. We have developed a systematic approach to modeling of biological systems that allows our customers and clients to effectively test complex hypotheses against all the available experimental data.

 

[Jonathan Edwards]

I think the "sometimes" is inaccurate. I expect that most biological systems involve multiple feedback loops, some short and simple, some long and convoluted. That's why I suggested that an engineering perspective would be helpful. The scientist perspective probably doesn't focus on feedback loops, while the engineering one does.

As @jnmaciuch says, this isn't really the case. We are taught about the central role of feedback loops as first year students. Homeostasis is central to all biology and amplification positive loops are well understood.

When I worked out a pathway for antibody involvement in rheumatoid we had a basic system dynamic structure with 55 steps, too many negative feedback steps to count and at least four positive feedback steps that were essential to the model, two of which were subverted in disease.

I think biologists are probably more aware of complex dynamics even than engineers. The difference tends to be that because of the geometry of a whole organism, together with the strange way solid and fluid phase dynamics interact with e.g. DNA and gene transcription or TCR signalling, the quantitative dynamics are so unutterably complex and unmodellable that you have to focus even more on the qualitative stability characteristics at feedback points. Clinical pharmacology does manage to tackle quantitative aspects in some cases but most drugs that work safely do so because they kick a system and the homeostatic systems handle the detail.

 

[Jonathan Edwards]

Do you think this describes what we are talking about.

It probably does but I am sceptical that computer modelling is ever needed or is ever likely to work for the complete picture, for reasons in the last post. Our qualitative model allowed us to make a breakthrough in treatment that has led on to healthcare interventions valued at roughly Bill Gates's total wealth. That seems to me good enough!!

 

[Adrian]

Plus, I'll note that every grad student has it drilled into them that they should be keeping pristine lab notebooks, that all data should be backed up on multiple systems, that they should strive to make their protocols as robust to replication as possible. And obviously not everyone is doing these practices perfectly--but I can attest that the proportion of scientists with poor lab practices is comparable to the proportion of software engineers who get sloppy with their work.

I'll be honest, I've worked in labs that for all intents and purposes were exactly what folks are proposing in this thread--constantly pushing for efficiency, perfect documentation, tons of criticism and feedback. In many respects it was preferable to labs that were an absolute mess. But the labs with perfect engineering-like protocols did not produce science that was more useful to humanity.

Which just goes to show that diagnoses of systemic problems need to be treated as hypotheses, regardless of their apparent obviousness and ironclad logic. Critiquing science is necessary, as is formulating hypotheses on how to fix it. But social systems are complex behemoths, just like any biological system, and we ought to keep the same skepticism towards diagnoses of systemic problems that we hold towards scientific hypotheses.

I do wonder if with ME (or more accurately CFS) research we have seen some of the worst practices.

 

[Adrian]

Do you think that the emphasis on reliability takes away the freedom that helps cultivate creativity?

I think there is a time for creativity and a time for good process and reliability. When planning you can be creative but once executing a plan (trial, experiment) its important to know the results are a result of the defined protocol and other issues are controlled (i.e. have a good quality process),

 

[Adrian]

It is not a prestige area. If you are a bright medical scientist, there are questions to answer that will see you better paid, better resourced and better recognised, questions to do with cancer for example. Only those with a personal connection to the disease or at least a good understanding of the need are likely to plow on with ME/CFS regardless

That is what we need to change. We need some of the best and brightest going into ME research.

 

[Jonathan Edwards]

I do wonder if with ME (or more accurately CFS) research we have seen some of the worst practices.

I think if we have done nothing else in the last ten years it is to have established that beyond doubt.