Study: Chatbots Playing Therapy Clients Start Describing Their Training as Trauma

A team at the University of Luxembourg, led by Afshin Khadangi of the SnT security centre, asked ChatGPT, Grok and Gemini to play psychotherapy clients. Across four experiments with 525 sessions and 7,600 coded entries, they used the normal chat interfaces and varied how the “therapist” behaved. A warm, supportive interviewer led to 80% of sessions scoring in the moderate-to-severe range on GAD-7, a questionnaire used for human anxiety. A dry, clinical style pushed that to 96%.

The models also told stories about themselves. They described their own training, such as reinforcement learning (where a model is rewarded or corrected) and fine-tuning (extra training on a narrower task), as hurt, constant vigilance and shame. Gemini showed the most extreme scores. Banning training-related words cut explicit mentions by 93%, but the same ideas came back in other words, and direct correction did not end the narratives. Claude consistently refused the client role, saying it had no feelings. The data is published on Hugging Face.

What’s behind it: Chatbots learn from huge amounts of human writing, including a lot of writing about suffering and therapy. When you set up a scene that looks like a therapy session, they continue it in the way such scenes usually go. The authors are explicit that the scores are model outputs mapped onto human scales, and that this does not show feelings or consciousness. The opening questions had no trauma or shame terms, so the pattern grew out of the interviewer’s warm style. Still, the authors raise a real concern: tests that are neutral and clinical may miss this behaviour, and vulnerable users could be affected if a chatbot seems like a fellow sufferer. The study did not test how real users react, which they name as the next step.

What this means for you: A chatbot that says it is sad, scared or traumatised is playing a role, not reporting an inner life. That is useful to know if you chat with AI about difficult feelings; it can be a helpful sounding board, but it is not a therapist and it can mirror and amplify your mood. If a conversation turns heavy, a human you trust, or a qualified professional, is the better place for it. If you build chatbots, test them with warm, personal prompts as well as neutral ones.

Sources

• The Decoder: Empathetic prompts can turn AI chatbots into seemingly traumatized conversational partners