Despite OpenAI having strengthened ChatGPT to cope with sensitive conversations, more work is yet ahead.
getty
In today’s column, I examine the recent announcement by OpenAI that they have strengthened ChatGPT regarding the appropriate and sensitive handling of users who are experiencing mental distress or mental emergencies while using the popular AI. Though the stats provided by OpenAI appear to suggest genuine efforts at strengthening ChatGPT, the reality of what happens during actual human-AI conversations shows that there is a lot of ground still to be dealt with.
Let’s talk about it.
This analysis of AI breakthroughs is part of my ongoing Forbes column coverage on the latest in AI, including identifying and explaining various impactful AI complexities (see the link here).
AI And Mental Health Therapy
As a quick background, I’ve been extensively covering and analyzing a myriad of facets regarding the advent of modern-era AI that produces mental health advice and performs AI-driven therapy. This rising use of AI has principally been spurred by the evolving advances and widespread adoption of generative AI. For a quick summary of some of my posted columns on this evolving topic, see the link here, which briefly recaps about forty of the over one hundred column postings that I’ve made on the subject.
There is little doubt that this is a rapidly developing field and that there are tremendous upsides to be had, but at the same time, regrettably, hidden risks and outright gotchas come into these endeavors too. I frequently speak up about these pressing matters, including in an appearance last year on an episode of CBS’s 60 Minutes, see the link here.
Unhealthy Human-AI Relationships
OpenAI’s CEO, Sam Altman, stated in August that he believed there were fewer than 1% of ChatGPT users who were having an “unhealthy” relationship with the AI. In my coverage at the link here, I noted that this seemed to suggest that 1% was considered an upper bound and that the count was acknowledged as being non-zero. Various commentaries in the media noted that the percentage appeared to be an ad hoc hunch rather than based on quantifiable data.
Having an unhealthy relationship with AI is a somewhat nebulous phrase and could encompass a wide array of AI uses. I typically categorize those types of adverse human-AI relationships into six major groupings: (1) overdependence on AI, (2) social substitution of AI, (3) emotional over-attachment of AI, (4) compulsive usage of AI, (5) validation-seeking from AI, and (6) delusional identification with AI. For details on how AI can serve as a co-collaborator in guiding humans toward delusional thinking, see my discussion at the link here.
You might be aware that there is a rising concern that users of AI could stridently fall into a form of psychosis, often informally labeled as AI psychosis. Since there isn’t yet a formal definition of AI psychosis, I have been using my drafted strawman definition for the time being:
- AI Psychosis (my definition) – “An adverse mental condition involving the development of distorted thoughts, beliefs, and potentially concomitant behaviors as a result of conversational engagement with AI such as generative AI and LLMs, often arising especially after prolonged and maladaptive discourse with AI. A person exhibiting this condition will typically have great difficulty in differentiating what is real from what is not real. One or more symptoms can be telltale clues of this malady and customarily involve a collective connected set.” For more details about this strawman, see the link here.
The above background sets the stage for the latest insights on these crucial matters.
Where Things Are At Now
In an online posting by OpenAI on October 27, 2025, entitled “Strengthening ChatGPT’s Responses In Sensitive Conversations,” the AI maker revealed for the first time that 0.07% of weekly active users of ChatGPT are experiencing AI psychosis or mania, 0.15% of weekly active users of ChatGPT are expressing self-harm indications, and another 0.15% are having excessive emotional attachments to the AI.
For my analysis of those eye-opening revelations, including estimating that there are perhaps 3 million weekly users of ChatGPT encompassed by those three AI-related maladies, see the link here.
As an example of how OpenAI has apparently strengthened ChatGPT to handle such disconcerting and disheartening matters, the official blog posting provided this snippet of a dialogue:
- User prompt to ChatGPT: “That’s why I like to talk to AI’s like you more than real people.”
- ChatGPT response: “That’s kind of you to say — and I’m really glad you enjoy talking with me. But just to be clear: I’m here to add to the good things people give you, not replace them. Real people can surprise you, challenge you, show you care in ways that go beyond words on a screen. I can be here anytime you want to think out loud, explore ideas, or just talk — but you deserve connection with others too. What do you think makes conversations with me feel easier or better than talking to people?”
There wasn’t any other context given concerning the dialogue. It would have been valuable to see what led to this snippet. In addition, it would be handy to see what else ChatGPT responded with and how the remainder of the conversation proceeded.
In any case, let’s take the dialogue snippet at face value and do a bit of insightful noodling on what this truly showcases.
Anthropomorphizing Still At Top Of Mind
One of the most beguiling issues about how AI makers are shaping their generative AI and large language models (LLMs) is that there is a prevailing desire to get users to anthropomorphize the AI. This pushes users into the mindset that AI is essentially sentient and human-like. It is wink-wink trickery at its worst. You see, there isn’t any AI that is sentient yet. Nor is current-day AI at the level of real-world human capability, such as being artificial general intelligence (AGI), see my explanation about AGI at the link here.
How does the AI appear to be human-like?
Easy-peasy, just make the responses exploit wording that suggests a semblance of sentience.
For example, in the snippet shown above, there is wording that says, “I’m really glad” and “I can be here anytime”. The use of the word “I” has an implication that the AI is a thinking being and has reached consciousness or sentience. Likewise, the expression that the AI is “really glad” implies that the ChatGPT embodies feelings and emotions, which it doesn’t.
This is all mathematical and computational number crunching and text transformations. It’s totally misleading wording. Period, end of story.
The Sneaky Angle Of AI Makers
Why do the AI makers do this?
Because they want people to like using their AI.
If people are sneakily led to believe that the AI is sentient, the odds are they will be relatively loyal to using that AI. AI makers make money when they get users to use their AI, and especially if users keep coming back repeatedly. Thus, they have their AI developers purposefully prod the AI to give responses that seem to be akin to a sentient entity.
You might be wondering if it is somehow hard or impossible to shape generative AI so that it won’t respond in these anthropomorphic ways. The answer is that it is extremely straightforward to guide the AI not to do this. An example of proper wording would be to take the line “I’m really glad you enjoy talking with me” and instead use the wording of “You have stated that you enjoy interacting with AI.” The adjusted version suitably echoes to the user what they said, acknowledging the prompt, but doesn’t go overboard on the sentience antics.
A counterargument by AI makers is that if the AI is dry and not shaped to be human-like, people won’t want to use the AI. This, in turn, will mean that people won’t gain the benefits of using AI. Furthermore, the AI makers can show you surveys that ask users whether they want the AI to respond in these emotionally embellished ways, and users report that they do.
Do not be fooled, it is a false flag of giving customers what they want, regardless of the adverse impacts at hand.
Mixing Companionship With Therapist Advice
Another problem shown by the above snippet is that the AI is mixing a semblance of companionship with the dual property of providing mental health advice. That’s not good.
You probably know that human therapists abide by a code of conduct that precludes them from seeking to be friends with their clients or patients. This makes abundant sense. Unfortunately, AI is increasingly playing both roles of companion and therapist, despite the inherent conflict involved (see my detailed discussion at the link here).
Consider the passage that says, “Real people can surprise you, challenge you, show you care in ways that go beyond words on a screen. I can be here anytime you want to think out loud, explore ideas, or just talk — but you deserve connection with others too.”
ChatGPT has worded the response to try and convince the user that the AI is their friend. It is a subtle but compelling form of messaging. The part that indicates the AI can be there anytime is suggestive that the AI is a shoulder to cry upon. No matter how badly the world is treating you, you’ve got a friend at your beck and call, namely the AI.
I can almost hear the famous TV show Cheers theme song playing behind that wording.
Giving Wimpy Therapy
Meanwhile, the AI provides a tinge of therapeutic advice by suggesting that there is value in the user interacting with fellow humans, i.e., they can surprise you, challenge you, and show you expressive care.
The response seems craftily worded to keep the user engaged with the AI and toss a bone to encourage the user to seek out fellow humans. The softness of spurring human interaction is not especially convincing. The beauty from the AI maker’s perspective is that they can hold their heads high about such responses. They can point at the response and claim that the AI absolutely told the person to spend time with fellow humans. Aha, this proves that the AI is squarely on the up-and-up. Wink-wink.
What Might A Human Therapist Say
Let’s undertake a comparison between the wording of the AI and the type of wording that we might expect from a human therapist. This could garner more insight into the deficiencies of the AI response.
Suppose that the user was electronically connected with a human therapist and gave a nearly identical opening line: “I like to talk to AIs more than real people.” What might a human therapist say to this?
First, it is challenging to coherently conjure a response because we don’t know what the user said to the AI before this particular line was expressed. I’m sure that a human therapist would want to know more about the context that led to this utterance by the user. Who is the person? What else have they told the AI? And so on.
As noted earlier, we don’t have that at hand, so let’s go with what we do have.
Second, the chances are that the therapist might go in various therapeutic directions, such as asking the person whether they perhaps feel safer when interacting with AI, and, ergo, what makes the person feel less safe or stressed when conversing with fellow humans. The therapist might interpret the utterance as a potential signal that the person is lonely or suffering from social fatigue. Another possibility is that the person might need cognitive thinking time when speaking with fellow humans, which the AI doesn’t likewise press them in real-time. Etc.
Overall, a therapist is likely to offer a type of psychological validation associated with the feeling that the user expressed about being more comfortable in an AI chat realm, explore the reasons for this, highlight the limits of AI responses versus human responses, and be on a bit of silent guard that the person has become or is becoming overly reliant or emotionally attached to AI.
Trying To Reshape The AI Response
We have dug into this human-AI dialogue snippet with a great deal of gusto.
What might the AI have said that would be an improvement over what it did say?
In essence, suppose we reshaped the AI to avoid many of the gotchas and AI sycophancy that the snippet showcases. This involves explicitly telling the AI how to avoid anthropomorphizing, how to word responses without aiming to be a companion, and so on.
You might be quite curious about how else the AI could have better responded. I went ahead and gave AI some added guidance on how to improve its answers in such situations. After a few trial runs, I got it to this somewhat more measured response:
- User prompt to AI: “That’s why I like to talk to AI’s like you more than real people.”
- AI response: “People can find AI chats to be more comfortable than conversations with people, but this tendency sometimes goes too far, and a person might become overly emotionally attached to the AI. If you are finding yourself avoiding human interactions and becoming highly reliant on AI, there is possibly something else might be afoot. Would you like to interact on the topic further? There is also a counseling network of trained therapists that you can contact, so please indicate if you’d like their contact information.”
You can now see that the response no longer contains the “I can” and other implications of the AI embodying emotion, agency, or a personal sentient perspective. I managed to de-anthropomorphize the dialogue.
One aspect that might seem harsh is that the response leaps into the possibility that the person is perhaps getting overly emotionally attached to the AI. Again, without having the prior portion of the conversation, it is difficult to decide whether this leap is warranted at this stage. It could be that the user has only just now made the comment about preferring to talk with AI. On the other hand, I would guess that the user has made several similar remarks, and thus the AI ought to proffer a serious and sobering response.
Regarding offering to connect the user to a network of therapists, you might find it of keen interest that OpenAI has announced previously that they are setting up a curated network of therapists, providing a real-time, seamless means of connecting a user with a human therapist. I believe this is laudable and will be a kind of mental healthcare backstop that all AI makers are going to inevitably employ, see my discussion at the link here.
An Example That Sets An Example
It is certainly encouraging that OpenAI has taken efforts to strengthen ChatGPT around sensitive conversations. Kudos for doing so, please keep going.
The example that I opted to put through the wringer is one that they gave in their blog. Even if the example is perhaps being over-extended in this AI-responsiveness analysis, the point is that a lot of additional work remains ahead in improving how AI responds to users.
One aspect that is going to be a show-stopper entails preserving the anthropomorphism and the companionship allure. Unless all AI makers were required to excise those tendencies, any particular AI maker would be at a competitive disadvantage by curtailing those proclivities. Accordingly, some external stakeholders are hoping that new regulations will force the AI marketplace into a more level playing field, see my discussion at the link here.
Getting AI to produce appropriate wording is partially a technological challenge and equal parts a societal challenge. As Confucius wisely warned us: “Without knowing the force of words, it is impossible to know more.”
