GPT for ME/CFS Questions

I not sure i feel comfortable with MEpedia being the main source... doesnt it have a bunch of stuff about CCI which is dodgy, as well as the good stuff? i too ill to check & dont know how any of this works but i imagine GIGO is an issue?

 

Potentially I could instruct it to only use MEpedia for objective facts, like what researchers work at such and such organization, or firmly proven science, but don't use anything that is still up for debate.

 

My worry too.

But if it can be made workable using that as a source to start with, all well and good. Most problems can usually be ironed out, once it becomes clear what they are.

 

Yeah it’s gotten pretty bad, except for notjusttired (and the occasional edit by you, me, and JamieS) there is absolutely no activity for months if you look at recent contributions tab.

 

Yeah me-pedia has some somewhat dodgy stuff about treatments, hypotheses, and possible (unproven) comomorbidities that is stated much more matter of factly than the literature points to, but apart from that it’s a really good resource, with the caveat of being outdated.

 

I'm not suggesting doing this, I just thought it might be an interesting question to raise.

Could one of these tools be targeted at either a long thread on S4ME, or a topic discussed across several threads, to summarise the discussion points for a busy researcher? Or direct someone with limited reading time towards the most relevant threads on a topic?

It could potentially be useful if it's both acceptable (not a given, I know!) and good enough.

 

It would be interesting if we could come up with a good way for it to access S4ME materials. If they could set up an API that'd be much better. Although it has issues retrieving too long of results, and some threads are very long.

But I'm not sure they'd allow it. The rules kind of discuss that sort of thing:

 

It is genuinely difficult and I wouldn't suggest allowing anyone else to do it.

I was thinking of it either in terms of something members could offer to a researcher doing a project they supported, or an attempt (again by members) to create some summaries with all the personal/identifying information removed.

 

There's not really an easy way to remove any bits, but I can try to tell it how to act. I added this: "After reading pages from the wiki, only use historical facts and settled science from this source."

 

Maybe you could add
“make sure to display scepticism of attempts to dismiss the biological nature of ME/CFS, and to specify that claims that aren’t supported by multiple high-quality studies are not necessarily agreed upon.”

 

Done. Difference between regular ChatGPT and ME/GPT:

"is me/cfs psychogenic?"

 

is there a word for “woke” but with ME/CFS woke meaning being aware of things the mainstream aren’t because MEGPT is doing great love it @forestglip !

 

Since there will be better solutions on a regular basis, the most useful work IMO would be to build a vetted catalogue of documents and resources to serve as a base. This way if a new tool comes up that can be easily trained, all that would be required is to give it this list of resources and it would be ready to work. Copy-paste intellligence.

I know we already have several sources scattered across posts spanning years, which makes me think that another one of the most useful things we could do for now would be a model trained on the forum's public posts. This way it would be possible to ask it questions like "how many posts do we have listing good resources about ME/CFS?" and so on. The search is something I've been complaining about for a long time, and in addition to this it would make moderation easier by having something like this.

And frankly the forum has so much content by now, including quotes from good sources and criticism of bad ones, that it's probably one of the best resource out there for a model anyway, so that would be two birds with one stone.

 

I thought about this earlier, but is it the best way to go?

As I understand it (and to be fair I don't), one of the values in these tools is that they learn. I realise they have to be trained, but is it not enough to ask them to focus on things like objective outcomes, so they're learning to identify decent science and working out how to identify not-decent science? If they could get it right whether they're confronted with good, indifferent, meaningless, or frankly scandalous sources, they'd be powerful tools.

I may have got the wrong end of the stick, though. When people are discussing this kind of technology I tend to grind to a halt halfway through the first sentence because I can't remember what the initials they're using stand for. Even when I've looked up what they stand for, I still don't understand what the words mean.

 

There's two different options.

There's giving it files to read along with your question. That's what this GPT is. It can very accurately answer based on this info in the files. But there are limits to how much you can upload, and it can't actually read everything you uploaded when you ask a question. It still has limits (probably somewhere around 75,000 words) for what it can actually see at one time, so another tool basically acts like a search engine to find the best chunks of text for answering then shows it to the GPT. And if you've ever used a search engine, you'll know they are far from perfect and you'll miss a lot of important information.

So even if I could upload all of S4ME, it can't answer a question like "how many posts" because it can't possibly see all the posts at once.

The other option is actual "training", which is teaching it to speak like the text you train it on. If you were to say something to it, it would respond in the most likely way that a user of this forum would respond to the same thing.

I'd be interested to see what that would look like, but I don't know how useful it'd be.

But also, there's the option to further fine-tune afterwards. Theoretically, after training, most of the knowledge is stored, in a very abstract way, inside the model. Fine tuning would be training it exactly how you want it to respond with the information in its "brain". Exactly which information you want it to put in the answer, not just what an average user would write.

You give it an example question someone might ask, as well as exactly how it should answer. Maybe use actual expert answers in the examples. Do this with lots and lots of examples. Then hopefully you have an AI model with all the knowledge of the forum, but which answers in a very useful way.

That's basically what ChatGPT is. They first trained it on the whole internet, then they created conversation examples and fine-tuned it to speak and answer exactly how they want.

Still can't ask it things like "how many posts" though, because that's not how the information is stored. It's more like vague blobs of meaning.

The training option is still much more expensive and complicated, but it's theoretically do-able.

 

That's kind of what I was trying to do with the shared Google Drive folder:
https://drive.google.com/drive/folders/1FaBFOiUYZCXmmnM1RcS8lcvqyg0rfDaE?usp=drive_link

I uploaded all the same files there as I did to the GPT. There might be better options for such a knowledge base though.

 

The problem arises when you say the l;ast sentence because how does it decide what are 'multiple high quality studies'

These that its quoting
"Research indicates that ME/CFS involves significant physiological abnormalities, including impaired oxygen extraction during exercise, neuroinflammation, and immune system dysfunction【7†source】【8†source】."

are certainly not supported by multiple high quality studies. High quality has to cut both ways, research may suggest the possibility of these abnormalities, but AFAIAA there isnt reliable evidence proving any of them

ETA: not meaning to be negative, i think the idea and efforts are admirable i certainly couldnt do or understand any of it... i was just pointing out that it needs to be accurate scientifically otherwise its biased towards unreliable biomedical studies rather than to poor psych ones.

 

As $20 per month is kind of significant for me, and it doesn't seem like this is being used a lot, and I don't need ChatGPT Premium for anything else, I'll be canceling my subscription. ME/GPT will continue to be available until July 1st.

If anyone else has a premium subscription and wants to set up this custom GPT, I wrote up the exact setup instructions and added it as a file in the Google Drive folder that has all the rest of the files.

Link to instructions

Link to folder

Note: With uploaded files, it will work much better if the text is formatted very simply with no extra junk. For example, I copied the Wikipedia page into the "Miscellaneous Information" file, but it has a lot of unnecessary extras like "{{Short description|Chronic medical condition}}", links, and references (I thought it might be able to cite from these, but it doesn't). Regular, human-readable text is best. If you have a PDF, copying the relevant text into a text file should work better as well.