2026: Science for ME reviewer nomination project, in conjunction with WWTF and WE&ME

[Benjamin Missbach]

I don't know if this comment will be useful or not, I'm not involved in this initiative at all so I may have the wrong idea of what is wanted.

Even if I have, I think there is value in explaining that something that I had to get my head around in my patient representation roles is that it is not my role to be an expert on the science, but to offer my opinion based on my own experiences as a patient.

Now those experiences can certainly include and be informed by all that I have read and discussed on the forum, and I believe that the same applies with this initiative.

So I'd encourage all those involved to have confidence in your own opinion, which is probably better informed than you think it is, and to give your opinion, as a patient and a forum member, because, ultimately, that is what you have been asked to do.

Anyway, I hope that helps at least one person in some way.

@Andy: thanks for sharing this. This forum has and especially the reviewers who signed up for the proposals do have in-depth knowledge of the scientific discourse, so I'm very confident that the science part is very well covered, in addition to the expert by experience part. In fact, many ME/CFS academics I have met the last years do have a strong connection to the disease spectrum itself, so they can also contribute the expert by experience part, at least a bit. At the end, we have a very thorough quality assessment where at least 4 academics + at least 2 S4ME members will provide reviews and the panel (n = 6; Chris Ponting (Chair) | University of Edinburgh, UK; Alba Azola | Johns Hopskins Hospital, US; Resia Pretorius |Stellenbosch University, SA; Elisa Oltra | Catholic University of Valencia, ES; David Putrino | Icahn School of Medicine - Mount Sinai, US; Keith Geraghty | University of Manchester, UK) will then use this information for the discussion to make a final judgement and recommendation to our WWTF Boards.

 

[Benjamin Missbach]

Yes, I am not suggesting group efforts. It is normal when reviewing to get confidential input from others on things like points of factual detail and I think people should feel they can do that. But I agree that group discussion would raise problems.

Confidential consultation is ok. At the end every individual reviewer is responsible for their review.

 

[Andy]

Thanks for confirmation and feedback Benjamin. Could I make the suggestion that in addition to the reviews that you might ask the patient reviewers for feedback on the recruitment process itself?

I know that when I am considering a new role, due to my limited resources, I want as much clarity as possible. I want to know aspects like what is involved, what is the end goal, how much time would be needed, and where in the wider project structure the role would sit, in more detail than is likely the case with healthy potential recruits, because I need to try to predict the likely impact on my capacity.

From what I have seen in the discussion here I get the sense that some of this kind of information was not clear to potential patient reviewers, so improving on this for potential future recruitment could well be useful.

 

[Benjamin Missbach]

yes, will do!

 

[Jonathan Edwards]

Thanks for the clarification, Ben. I have so far limited my interaction to:

"Have a got this right, Jo?'
"Looks right to me".

I think you are right to hope for in-depth knowledge from patient members. I continue to be blown away by how much people with no special training in the science are picking up on crucial details. And patient members are bound to be reticent about their ability to contribute initially.

 

[Jaybee00]

At the end, we have a very thorough quality assessment where at least 4 academics + at least 2 S4ME members will provide reviews and the panel (n = 6; Chris Ponting (Chair) | University of Edinburgh, UK; Alba Azola | Johns Hopskins Hospital, US; Resia Pretorius |Stellenbosch University, SA; Elisa Oltra | Catholic University of Valencia, ES; David Putrino | Icahn School of Medicine - Mount Sinai, US; Keith Geraghty | University of Manchester, UK) will then use this information for the discussion to make a final judgement and recommendation to our WWTF Boards.

Mind if I ask how you chose Pretorius and Putrino for this panel? I’m underwhelmed by these individuals, as I believe others on this forum are.

 

[Benjamin Missbach]

@Jaybee00: thanks for asking. WWTF decides on the configuration of our panels for each call separately. Our calls are always tailor-made, that's why we have a different set-up for each individual call. When deciding who to bring on our panels, we have a set of criteria (seniority, experience on panels, expertise level, generalist + specific focus, etc). Prior to each call, and as we are a local funder, we know our potential applicant community quite well, we have to roughly estimate what topics will be addressed and based on a jury matrix we decide on who to invite to our panel. If we stumble upon missing expertise along the way, we can add panel members + for most calls, we have external reviewers to provide potentially missing expertise (as in this consolidation call).

I hope you understand, that I will not discuss individual members of our panels in this forum, but I have been aware of the discussion around different researchers on different forums.

Mind if I ask how you chose Pretorius and Putrino for this panel? I’m underwhelmed by these individuals, as I believe others on this forum are.

 

[hotblack]

I’m not a fan of the work or approach of everyone on the list. But IMHO it’s no bad thing to have a range of views, including some people may disagree with.

Variety of perspectives is useful in these decisions as in many areas of life. There’s also a lot of different opinions out there and different people wanting different ideas represented or investigated. Even just on this forum there is far from unanimity, look at some of the complaints recently about there being potential for a genetic bias for an example of this.

Having a range of views also stops accusations of a closed shop or other arguments that could dismiss what is being done. If everyone on a panel was from a school of thought you or I disagreed with it would be easy to dismiss the whole project. That there is a mix hopefully limits that. What will matter is the quality of the projects funded and tbh that things are being funded at all.

 

[Benjamin Missbach]

I’m not a fan of the work or approach of everyone on the list. But IMHO it’s no bad thing to have a range of views, including some people may disagree with.

Variety of perspectives is useful in these decisions as in many areas of life. There’s also a lot of different opinions out there and different people wanting different ideas represented or investigated. Even just on this forum there is far from unanimity, look at some of the complaints recently about there being potential for a genetic bias for an example of this.

Having a range of views also stops accusations of a closed shop or other arguments that could dismiss what is being done. If everyone on a panel was from a school of thought you or I disagreed with it would be easy to dismiss the whole project. That there is a mix hopefully limits that. What will matter is the quality of the projects funded and tbh that things are being funded at all.

@hotblack: this is very well said and right on the spot

 

[Utsikt]

I’m not a fan of the work or approach of everyone on the list. But IMHO it’s no bad thing to have a range of views, including some people may disagree with.

Variety of perspectives is useful in these decisions as in many areas of life. There’s also a lot of different opinions out there and different people wanting different ideas represented or investigated. Even just on this forum there is far from unanimity, look at some of the complaints recently about there being potential for a genetic bias for an example of this.

Having a range of views also stops accusations of a closed shop or other arguments that could dismiss what is being done. If everyone on a panel was from a school of thought you or I disagreed with it would be easy to dismiss the whole project. That there is a mix hopefully limits that. What will matter is the quality of the projects funded and tbh that things are being funded at all.

I wouldn’t mind having people with different ideas about how to approach the problem. But that’s different from having people that have repeatedly shown that their bar for what passes as «good research» simply isn’t high enough.

ETA: if you asked all of the staff of S4ME about which project they think should be prioritised, you’d get all different answers. But everyone would still agree that the science has to be done right.

It should be in the interest of WWTF to select panel members that are able to ensure that funding isn’t wasted on what are dead ends by most reasonable standards, or on projects that can’t generate data of high enough quality to be able to guide future projects.

 

[Kitty]

But that’s different from having people that have repeatedly shown that their bar for what passes as «good research» simply isn’t high enough.

Researchers whose work is considered good by S4ME standards are in depressingly short supply, though. And even those who've done below-par work, or fallen short on things like objective measures or putting a spin on the results without publishing the underlying data, aren't necessarily incapable of doing better.

I guess part of the role of the patient reviewers is to help them improve by picking up on these issues in their project outlines. Maybe in one or two cases, by spotting that a project's unsatisfactory because it can't be done properly on the amounts available under the scheme.

I suspect a fair bit of research money gets wasted through people trying to pursue questions on insufficient funding. I know from a different field how tempting this can be, especially if there's not much else available at the time. But you need to know when it's better not to do the work at all than be forced into economies that'll undermine the result.


ETA: just re-read this, and wondered if it might come across as suggesting the funding made available for these projects isn't enough. I didn't intend that at all—only that all budgets are finite, and there'll always be ideas you can't shoehorn into them.

 

[EndME]

Researchers whose work is considered good by S4ME standards are in depressingly short supply, though. And even those who've done below-par work, or fallen short on things like objective measures or putting a spin on the results without publishing the underlying data, aren't necessarily incapable of doing better.

Yes, I think it should also be clear that researchers can at the same time easily spot and point out errors, mistakes or poor quality research in the work of others, whilst at the same time ignoring similar issues in their own work. Duality of (wo)man.

 

[Jaybee00]

And even those who've done below-par work, or fallen short on things like objective measures or putting a spin on the results without publishing the underlying data, aren't necessarily incapable of doing better.

Not sure if I agree with this. I think some researchers are just in over their heads, and they just have to continue to phone it in.

Is there an example of an ME/CFS researcher who started out poorly and then made a big step up to do high quality research?

 

[Utsikt]

Researchers whose work is considered good by S4ME standards are in depressingly short supply, though. And even those who've done below-par work, or fallen short on things like objective measures or putting a spin on the results without publishing the underlying data, aren't necessarily incapable of doing better.

I guess part of the role of the patient reviewers is to help them improve by picking up on these issues in their project outlines. Maybe in one or two cases, by spotting that a project's unsatisfactory because it can't be done properly on the amounts available under the scheme.

I suspect a fair bit of research money gets wasted through people trying to pursue questions on insufficient funding. I know from a different field how tempting this can be, especially if there's not much else available at the time. But you need to know when it's better not to do the work at all than be forced into economies that'll undermine the result.

They might be in short supply, but they are also short on funds so I don’t think WWTF will have any issue spending their money with very strict criteria.

 

[butter.]

They might be in short supply, but they are also short on funds so I don’t think WWTF will have any issue spending their money with very strict criteria.

In a small field like ME/CFS, assembling a (strong) jury inevitably involves trade-offs. It's difficult to find highly qualified reviewers, jury members etc. it's time-consuming and not particularly well compensated, and ideally you want the best researchers to be able to submit proposals rather than be excluded because they're serving on the jury.

 

[Utsikt]

In a small field like ME/CFS, assembling a (strong) jury inevitably involves trade-offs. It's difficult to find highly qualified reviewers, jury members etc. is time-consuming and not particularly well compensated, and ideally you want the best researchers to be able to submit proposals rather than be excluded because they're serving on the jury.

This particular call has a restricted applicant pool already (it’s a follow up for Vienna-based researchers), so I don’t think the argument applies here.

 

[butter.]

This particular call has a restricted applicant pool already (it’s a follow up for Vienna-based researchers), so I don’t think the argument applies here.

But it applied to the original call, and it's the same jury (which has/would have many advantages) for the follow-up, I believe.

 

[Utsikt]

But it applied to the original call, and it's the same jury (which has/would have many advantages) for the follow-up, I believe.

The initial call was also for researchers in Vienna - that’s the whole point for WWTF.

Aside from that, the reviewer would only have to excuse themselves from being involved in assessing the proposals from people they have worked closely with lately. So it would probably only be an issue for one proposal per reviewer at most. That is completely within the norm.

 

[butter.]

The initial call was also for researchers in Vienna - that’s the whole point for WWTF.

Aside from that, the reviewer would only have to excuse themselves from being involved in assessing the proposals from people they have worked closely with lately. So it would probably only be an issue for one proposal per reviewer at most. That is completely within the norm.

This only addresses the leave the best researchers out issue, and not the find good researchers that will do it issue. That said, this might be a good opportunity for members of S4ME to come up with a good list for next time. I think some people will be surprised to find that researchers are not always super eager to work in a jury for such a call.

 

[Utsikt]

This only addresses the leave the best researchers out issue, and not the find good researchers that will do it issue. That said, this might be a good opportunity for members of S4ME to come up with a good list for next time. I think some people will be surprised to find that researchers are not always super eager to work in a jury for such a call.

I have never argued it’s easy to find people to do it. Just that there should be a minimum standard. You can’t expect anyone to judge the scientific excellence of someone if they do poor trials themselves. The vetters must be vetted themself, otherwise the poor methodology will continue to flourish. The next step could be to involve S4ME or others with similar expertise when creating the jury.

I might be overly optimistic, but I hope some of the better researchers would find some appeal in helping out a funder that really want’s to get things right, and where there is a real possibility of achieving it.