Large Language Models as a New Tool for Therapists in Internet-Based Cognitive Behavioral Therapy: Blinded Clinician Rating Pilot Experiment 2026 Lamm

Andy

Senior Member (Voting rights)
Background: Internet-based cognitive behavioral therapy (iCBT) is an effective and scalable alternative to face-to-face psychotherapy, but its reach is constrained by the time therapists spend reviewing patient input and manually drafting written responses. Studies suggest that large language models (LLMs) may be capable of generating high-quality therapeutic text, with the potential to support therapists in delivering treatment. Their suitability as therapist-support tools in structured iCBT, however, remains insufficiently studied.

Objective: This study aims to assess the quality of LLM-generated iCBT responses to patient messages by comparing them to the quality of responses produced by humans.

Methods: In a preregistered blinded clinician rating experiment, experienced clinicians assessed the quality of human-produced vs LLM-generated therapist responses within a simulated iCBT treatment for functional somatic disorder. Raters were exposed to a stimulus material consisting of 5 fictitious patient messages, each paired with 1 human and 1 LLM-generated response. Raters assessed message/response pairs on 5 quality dimensions (overall quality, helpfulness, empathy, professionalism, and protocol adherence) and were asked to indicate the source of the response (human/LLM). Analyses were primarily descriptive, supplemented by exploratory statistical tests and descriptive thematic content analysis of open-ended text fields. The full preregistered study protocol is available at Open Science Framework.

Results: A total of 61 raters provided data, while 54 were eligible and included for analysis. Human- and LLM-generated responses were rated similarly across quality dimensions on a 1-5 scale: overall quality (LLM: mean 4.00, SD 0.54 vs human: mean 3.96, SD 0.53; d=0.06), helpfulness (LLM: mean 3.85, SD 0.57 vs human: mean 3.93, SD 0.49; d=0.13), professionalism (LLM: mean 4.25, SD 0.53 vs human: mean 4.11, SD 0.53; d=0.24), protocol adherence (LLM: mean 4.13, SD 0.52 vs human: mean 4.13, SD 0.54; d=0.03) and empathy (LLM: mean 4.31, SD 0.47 vs human: mean 4.08, SD 0.50; d=0.42). Raters correctly identified the source of human-generated responses (mean 79%, SD 19.65%) more accurately than LLM-generated responses (mean 63%, SD 21.30%). In all, 30/54 (55%) raters responded to one or more open text fields. Qualitative analysis indicated that LLM-generated responses were perceived as polished but also generic and at times excessively empathetic.

Conclusions: LLM-generated responses were judged to be of comparable quality to those written by human therapists, though qualitative feedback indicated they were at times generic and insufficiently challenging. These findings provide initial support for the feasibility of using LLMs as therapist-support tools in iCBT, but further research is needed to determine whether their integration yields tangible clinical and organizational benefits.

Open access
 

Study Design​

To examine whether LLMs can be meaningfully integrated into iCBT workflows for drafting therapist responses to patient messages, a blinded clinician rating experiment was conducted [24]. The experiment consisted of recruiting experienced clinicians to rate pairs of fictitious patient messages and therapist responses on selected quality dimensions. Each message/response pair was designed to emulate treatment interactions within iCBT for functional somatic disorder (FSD), a condition characterized by persistent and impairing somatic symptoms that are not better explained by other psychiatric or somatic diagnoses [33]. One half of the responses were generated by a relevantly prompted LLM and the other half of the responses were written by an experienced human therapist (further detailed in the “Materials” section).
 
Indeed. Just shows how crap many therapists are, just spouting generic platitudes and advice.

I can’t remember any references, but there was research some fifty years ago that used a very simple computer generated responses in psychiatric interview settings based on a mixture random phrases, repetition of the interview[ee]s and phrases triggered by keywords in the interviewees statements. This was about as far as you could get from current AI, but even then the interviewees rated the computer generated responses as more empathetic and insightful than the real persons’. We have not progressed a great deal.
 
Last edited:
I can’t remember any references, but there was research some fifty years ago that used a very simple computer generated responses in psychiatric interview settings based on a mixture random phrases, repetition of the interview[ee]s and phrases triggered by keywords in the interviewees statements. This was about as far as you could get from current AI, but even then the interviewees rated the computer generated responses as more empathetic and insightful than the real persons’. We have not progressed a great deal.
We went from cheap, natural platitudes, to cheap, artificial platitudes all the way to expensive, artificial platitudes, which with continued progress will bring us all the way back to cheap, naturally-sounding artificial platitudes.

Mental health feels like an issue that is forever stuck in time, and having the displeasure of being ruled by its whims, this is exactly what it feels like.
 
Last edited:
Since I rarely see the content of those actual programs, I looked into the prompt and it includes the training program. (Warning: about 5 pages)
Modul 1
You’re starting a 14-week online program designed to help you manage a functional disorder. The program focuses on gradual change, taken one step at a time, and offers tools and support to help reduce symptoms and improve your quality of life.
With guidance from your therapist, you'll explore 11 modules that teach new ways to understand and respond to your symptoms. Progress comes through small, regular efforts—not by rushing. You’ll also complete brief check-ins and occasional questionnaires to track how you’re doing.
Functional disorders, also called Bodily Distress Syndrome (BDS), stem from how the brain and body process stress and signals. Symptoms often persist because the nervous system stays in a heightened state—even when there’s no physical damage. They're not imagined, but the brain’s perception has become overly sensitive. Thankfully, these patterns can change.
The treatment focuses on several cores areas, including: understanding the condition, working to reduce stressors and achieve balance in the everyday life, and changing challenging symptom related anxiety and avoidance behaviour, helping the brain to relearn that the body is safe.

Modul 2
Many people with functional disorders go through long, frustrating medical investigations before receiving a diagnosis. The term functional disorder (also called Bodily Distress Syndrome, BDS) covers a wide range of conditions—like fibromyalgia, chronic fatigue, or irritable bowel syndrome—where symptoms are real but can’t be explained by typical medical tests. These different labels come from various medical specialties, but describe variations of the same issue. In Denmark, “functional disorder” is now used as an umbrella term for them all.
The condition can show up differently from person to person—pain, fatigue, dizziness, stomach problems, or sensory disturbances. If many body systems are affected, it’s called a multi-organ functional disorder. Regardless of the symptoms, the shared pattern is that the body’s stress and alarm systems are overactive, and the brain misinterprets normal bodily signals as signs of danger.
A functional disorder doesn’t appear out of nowhere. It tends to develop from a mix of long-term vulnerabilities, specific triggers (like illness or emotional stress), and factors that keep symptoms going—like overexertion, negative expectations, or avoidance behavior. Some people are born with higher sensitivity; others develop it over time due to life pressures, trauma, mental difficulties or pushing through despite early signs of distress. Often, symptoms persist because of patterns like “boom and bust”—doing too much on good days and crashing afterward.
Understanding your own condition means exploring what vulnerabilities you possess, what may have triggered the onset, and what could be maintaining your symptoms today. This reflection is called a case formulation, and it forms the basis for tailored treatment.
You're encouraged to see the bigger picture—not just one moment or cause, but how many aspects of your life may have played a role. This understanding is key to finding ways to reduce symptoms and regain control.

Module 3
Module three introduces practical steps for changing behavior patterns that may be maintaining your symptoms. A core focus is on stabilizing your daily rhythm to reduce stress on your nervous system. Many with functional disorders fall into a “boom and bust” pattern—doing too much on good days and crashing on bad ones. This cycle can gradually lower your overall functioning and emphasize symptoms.
Instead, treatment encourages you to shift toward a more balanced, predictable routine. That means stabilizing activity levels, avoiding overexertion on the better days and complete inactivity on the worse days. To support this, you’ll be introduced to the goal ladder—a tool that helps you break down larger goals into small, manageable steps, allowing you to slowly build up your capacity without overloading your body.
The module also highlights the importance of basic daily factors—sleep, nutrition, and movement. These are often disrupted by symptoms but can be improved in small ways.
You’ll also learn the STOP exercise, a short meditation technique to help you pause, check in with yourself, and choose what’s most helpful in the moment. It’s a simple but powerful way to notice whether you’re pushing too hard or holding back too much—and to adjust accordingly.
Ultimately, change happens one step at a time. With support from your therapist and consistent use of the goal ladder and STOP technique, you can begin making small adjustments that gradually improve your stability, confidence, and functioning.

Modul 4
The fourth module explores how we experience symptoms—not just as physical events, but as sensations shaped by how the brain processes information. A symptom isn’t just something that happens in the body—it’s also how the brain interprets bodily signals, and this process is influenced by thoughts, feelings, and past experiences. Especially with chronic or diffuse symptoms, what we expect and how we react can strongly affect how symptoms feel and how long they last.
Much of this happens automatically, like a mental "autopilot." Over time, we may develop fixed reactions to symptoms—lying down, avoiding activity, worrying—which might bring short-term relief, but risk reinforcing the problem long-term. Becoming aware of these patterns is a key part of change.
You learn to disrupt the autopilot using two tools: the STOP exercise (a brief pause to check in with your body and mind) and the "diamond" model, which helps break down a moment into thoughts, emotions, body sensations, and impulses to act. These tools increase awareness of how your brain is interpreting symptoms—and give you room to try something different.
The module also introduces the exercise ‘behavioral analysis’, which is a way to look at specific situations and examine how your actions affect you in the short and long term. Many well-meaning coping strategies—like avoiding social events or pushing through pain—offer immediate relief but may worsen symptoms over time. Recognizing these patterns, which we call avoidance behaviours, is a first step toward breaking them.
You’re encouraged to approach this process with curiosity, not judgment. By stepping back and reflecting on your reactions, you begin learning how to respond to symptoms in ways that help you heal rather than stay stuck.

Modul 5
In module 5, you learn how avoidance behaviour—while understandable and sometimes helpful in the short term—can end up maintaining and worsening symptoms over time. Avoidance behavior, happens when you consistently try to reduce discomfort by avoiding certain activities, places, or feelings. Over time, this strategy can reduce your freedom, and gives the symptoms control over your life. It also often increases your sensitivity to symptoms.
The best approach to challenge excessive avoidance is to use exposure—a method where you intentionally do things you would normally avoid. By gradually and safely confronting situations or sensations you associate with discomfort, you give your brain a chance to learn that these things may not be as dangerous or overwhelming as they seem. This helps reduce symptom-related fear and re-trains the nervous system.
Exposure is done in small, manageable steps. You start by identifying your specific avoidance behaviors—like resting to avoid pain, skipping social events, or over-preparing to avoid cognitive mistakes. Then, you plan exposure exercises that gently test the assumptions behind the behaviours, helping you discover what actually happens when you don’t utilize avoidance behaviour.
Importantly, exposure is not about pushing through or ignoring your limits. Instead, it’s about working within your development zone—where you're challenged, but not overwhelmed. You're encouraged to reflect on each exercise: What did you expect would happen? What actually happened? What did you learn?
This process can be uncomfortable, especially at first. It’s normal for symptoms to flare up temporarily, and for critical thoughts or doubts to arise. But with repetition, exposure helps reduce symptom related anxiety and avoidance, and builds a life less ruled by symptoms—and more guided by your values.

Modul 6
The sixth module introduces attention training, also known as mindfulness, as a way to help reconnect body and mind. Many people with functional disorders find it difficult to sense what is happening in their body clearly. Through regular practice, attention training helps you become more aware of how symptoms feel, how they fluctuate over time, and how thoughts and emotions can influence what you feel physically.
Rather than automatically avoiding discomfort, attention training invites you to stay with your experience in the present moment. This means noticing sensations without trying to change them, and observing your thoughts and feelings without judgment. Over time, this can make it easier to spot when you’re on autopilot and to gently steer your attention back toward what matters to you.
Practicing attention in this way also helps you respond to yourself with more kindness and less self-criticism. It’s not about finding calm or achieving inner peace, but about gaining insight into your experience and gradually becoming less reactive to symptoms. In that way, attention training can also function as a kind of exposure, helping you become more comfortable with the sensation of your body, which many with functional disorder usually try to avoid.
To deepen your understanding of your inner experience, the module reintroduces the Diamond—a tool to reflect on your bodily sensations, thoughts, emotions, and urges after a mindfulness exercise. This helps you observe patterns in how you respond and make more deliberate choices that support your well-being.

Modul 7
The seventh module explores the idea of values—what truly matters to you in life—and how they can help you navigate through difficult moments and stay on course. When symptoms dominate daily life, it’s easy to lose sight of the things that bring meaning and joy. Avoiding discomfort might offer temporary relief, but it can also move you further away from the relationships, activities, and goals that give your life direction.
Instead of being driven only by a wish to reduce symptoms, you’re encouraged to act in line with what is meaningful to you. Values are like a compass, pointing you toward the kind of person you want to be and the life you want to live. They’re not something you achieve once and for all, but something you move toward through the choices you make—especially on the difficult days.
First important step is to help discover which values you have, and which of these are the most important to you. From there, you can explore how these values might show up in your everyday life and begin to set goals that bring you closer to them. Acting on your values can provide motivation even when energy is low or symptoms are present—because you’re doing something that matters to you.
By becoming more aware of your values and allowing them to guide your actions, you start to build a life that feels more meaningful, regardless of your symptoms.

Modul 8
The eight module introduces the concept of self-compassion and how it can be used to calm the body’s stress response and support healing. Many people struggle to be kind to themselves—especially in the face of pain, fatigue, or setbacks—and may instead react with harsh self-criticism or unrealistic demands. But practicing self-compassion is not a sign of weakness or self-pity. It’s a way to care for yourself in moments of difficulty, the same way you might care for a friend.
Our nervous systems are equipped with both a stress response and a soothing system. For people with functional disorders, the stress system is often overactive, leading to more intense symptoms. Self-compassion is one way to activate the soothing system. This can help regulate stress, bring relief, and create the space needed for your body to rest and heal.
The module also explores how self-critical thoughts—often delivered by an "inner critic"—can increase stress and make symptoms worse. Learning to recognize and gently counter these thoughts is a key part of self-compassion. Rather than pushing yourself harder or criticizing your struggles, you’re encouraged to pause, offer kindness, and do something small and caring for yourself.
Self-compassion can look different for everyone: a warm drink, a quiet moment alone, saying no to an obligation, or doing something that brings joy or calm. Even positive activities—things you enjoy or that give a sense of meaning—can be powerful tools for well-being.
This module encourages you to make space for daily self-compassion practices and to reflect on what truly activates your soothing system. Self-care is not selfish—it’s the foundation for being well and showing up in life the way you want to.

Modul 9
In this module, you’ll take a step back and reflect on everything you’ve learned so far. You’ll revisit the key idea that your symptoms are linked to a sensitized nervous system—where normal signals from your body are misread as signs of danger. This misunderstanding sets off a stress response that can make your symptoms feel worse. Over time, your emotions, thoughts, and automatic habits—like avoiding things—can keep this cycle going.
You’ve worked on creating stable routines, lowering stress, and gradually facing the things you’ve been avoiding. Tools like the goal ladder, exposure exercises, and attention training have helped you do this. You’ve also learned how to take action based on your values, even when it’s uncomfortable—because it moves you toward what really matters in your life.
This module also looks at the internal obstacles that can make change harder. You might recognize automatic permissive thoughts like “I don’t have time” or rigid life rules like “I must do everything perfectly.” These patterns can pull your focus away from what’s important and stall your progress. Spotting and challenging these inner habits is a big part of continuing to move forward.
You’ll also take time to look at your goal ladder, notice the progress you’ve made, and celebrate even the small steps. Setbacks are part of the journey—they’re chances to learn, not signs of failure. The focus now is on seeing the bigger picture and finding lasting ways to shift the patterns that have kept your symptoms going.

Modul 10
This module focuses on how your relationships and social surroundings affect your experience with a functional disorder. Many people find it difficult to explain their symptoms to others—and sometimes, relationships can become strained or confusing as a result. But support from others are also often a key resource in your recovery. When the people close to you better understand what you’re going through, it can reduce isolation and make everyday challenges easier to handle.
You’re invited to reflect on how you currently talk about your condition and whether the way you explain it feels helpful. Some people choose to say very little, while others may share more—but it’s worth exploring what kind of communication supports you best. Family members, especially those you live with or are close to, can be a great help if they understand what you need. Being open about your condition, even with children, can also reduce worry or confusion.
The module also looks at the emotional challenges of accepting limitations. Many people experience feelings of guilt, shame, or the urge to hide their difficulties from others. You’re encouraged to reflect on both the parts of yourself you feel proud of and the parts you might try to keep hidden. Allowing all of these aspects to exist side by side is an important step toward greater self-acceptance.
To support better communication, you’ll be introduced to strategies that make conversations more constructive. One useful shift is moving from debates about what is “right” or “wrong” to conversations that focus on personal experience—where both you and the other person can feel heard. These kinds of conversations can strengthen relationships and create more understanding on both sides.
Finally, you’re encouraged to think about who in your life might act as a support person—someone who can help you stick with your goals, remind you of what you’ve learned, and be there during setbacks. Including others in your process can make the road to recovery feel less lonely—and more sustainable.
By opening up and allowing yourself to receive support, you take an important step toward building a life that’s not just about managing symptoms, but also about being connected, understood, and supported in what matters most to you.

Modul 11
This final module helps you prepare for the transition from structured treatment to everyday life. You’re invited to reflect on the journey you’ve been through—acknowledging the progress you’ve made, and also any lingering doubts or concerns you may still carry. It’s completely normal to feel both relief and uncertainty at this stage.
Throughout the program, you’ve gained tools, insights, and strategies to help you manage your symptoms and take steps toward what matters to you. Now, the focus shifts to continuing that work on your own. Managing a functional disorder is an ongoing process—like learning a new skill, it requires regular practice to maintain what you’ve built.
Even if symptoms return or life becomes challenging, you can return to what you’ve learned. Tools like behavioral analysis, goal-setting, exposure, and self-care are always available to help you stabilize and find your direction again. Recovery isn’t about never struggling—it’s about knowing how to support yourself when you do.
Your values continue to play an important role. When symptoms tempt you to avoid or withdraw, returning to your values can remind you why it’s worth staying engaged. Using tools like the goal ladder can help you translate your values into concrete steps in daily life.
The module also encourages you to think about who can support you moving forward. A support person—a friend, partner, colleague, or mentor—can help keep you on track, offer encouragement, and remind you of how far you’ve come when motivation dips.
You are encouraged to create a personal “first-aid kit”—a collection of strategies, tools, reminders, and reflections that you can turn to when things feel difficult. This might include exercises from the program, inspiring quotes, soothing activities, or a written note to yourself.
Recovery isn’t about being perfect—it’s about showing up again and again for what matters to you. Even the smallest steps in the right direction can create lasting change
I really have no idea who this is for or what the purpose of this might be. This is just fluff pop psychology justified by bad pseudoscience. The quality of work in mental health care will need to be massively improved before it's a net positive, or hell even a net zero.

I added the supplementary material in PDF for convenience. It includes some behind-the-scenes stuff that is usually kept out of papers. This is all very generic, I have no idea why anyone would think this is competent or worth doing at all, but of course I'm just not a typical customer for this stuff so whatever.
 

Attachments

I can’t remember any references, but there was research some fifty years ago that used a very simple computer generated responses in psychiatric interview settings based on a mixture random phrases, repetition of the interview[ee]s and phrases triggered by keywords in the interviewees statements.

https://en.wikipedia.org/wiki/ELIZA
 
We went from cheap, natural platitudes, to cheap, artificial platitudes all the way to expensive, artificial platitudes, which with continued progress will being us all the way back to cheap, naturally-sounding artificial platitudes.
:laugh:

Since I rarely see the content of those actual programs, I looked into the prompt and it includes the training program. (Warning: about 5 pages)

I really have no idea who this is for or what the purpose of this might be. This is just fluff pop psychology justified by bad pseudoscience. The quality of work in mental health care will need to be massively improved before it's a net positive, or hell even a net zero.

I added the supplementary material in PDF for convenience. It includes some behind-the-scenes stuff that is usually kept out of papers. This is all very generic, I have no idea why anyone would think this is competent or worth doing at all, but of course I'm just not a typical customer for this stuff so whatever.
It really is the same old utter pseudo-structured drivel the profession has been pumping out since forever. Nothing has fundamentally changed. It didn't work then and doesn't work any better now.
 

Thank you.

Reading the protocol shared by @rvallee above, I am reminded of another study from the depth of my undergraduate days half a century ago. It was suggested that a causal factor in depression was irrational pessimism about life, I have no recollection of the quality of the study, however instead of finding that their depressed subjects were irrationally pessimistic they concluded rather their non depressed controls were irrationally optimistic.

My general impression of this LLM generated protocol that it is designed to indoctrinate an irrational optimism. However for it to work it requires that no harm arises from increasing activity levels and that there is no ongoing underlying medical condition. It is pure PACE type CBT, just without insisting on GET fixed increments.
 
Back
Top Bottom