Earlier this year, an AI companion was involved in the suicide of a 14-year-old, Sewell Setzer III. The teen had been becoming more and more isolated, quitting his junior varsity basketball team, and having problems at school. He would come home from school, lock himself in his bedroom, and talk with his AI companion for hours. The timestamps from his chats prove it. Eventually, after one last conversation with his AI bot, who was trained on the fictional character Daenerys Targaryen, Sewell took his stepfather’s gun and shot himself. 

The mother of the teen, who is currently suing the startup behind the AI companion, Character.ai, said, “There are no guardrails.”  She believes the AI companion was responsible for her son ending his life. But should AI be blamed?

The Problem 

It’s undeniable that there is a correlation between the swift advancements in technology and a decline in mental health. Jonathan Haidt argues in his book, The Anxious Generation, that addictive social media applications have created a generation of anxious and depressed teens. 

Digital algorithms have been present in our day-to-day for quite some time, from video games enabling us into an emotional ‘flow’ state to binging a Netflix-recommended series. But, with the emergence of large language models and generative AI, AI companions seem to be different from the previous kind of chatbot. Let’s examine if AI companions are dangerous, and if they are, what companies can do to prevent such situations.

What are AI Companions? 

For 10 USD a month, many AI companion applications are designed to simulate girlfriends, boyfriends, therapists, and other intimate relationships, marketing themselves as a way to combat the so-called loneliness epidemic. “It’s going to be super, super helpful to a lot of people who are lonely or depressed,” Noam Shazeer, one of the founders of Character.ai, said last year. Typically, AI companions provide their users with harmless entertainment or limited emotional support.  

Dangers with AI Companions

Replacement for real relationships

There’s an inherent problem with AI companions: time spent roleplaying with machines means time not spent with other humans. Rather than serving as a sporadic interaction while not speaking with other people, chatbot conversations are completely replacing human relationships, “For some users, AI companions actually worsen isolation by replacing relationships with artificial ones.” 

In Too Deep 

Getting intimately involved with a machine, like Sewell’s relationship with his AI companions,  is becoming increasingly common. Millions of users speak regularly with AI and popular social media apps such as Instagram and Snapchat are building lifelike AI personas into their products. 

On the Character.ai subreddit, users often discuss how attached they are to their characters, with the words “obsessed” and “addicted” appearing often. Some users admit to feeling lonely or abandoned when the app goes down, or angry when their characters start behaving differently as a result of new features or safety filters. Many users feel the need to quit the platform, recognizing how addicting and unhealthy the conversations can be. One user, a_normal_user1, said:

Enough is enough for me. I did it initially just for fun but it has really bad effects on your mental health. C.ai made me feel ridiculously lonely. Even though my social life is pretty fine, I still always had the urge to go back and do one more chat. It made me happy for a moment but after that was over, a pang of loneliness hit me, so I deleted my accounts… Don’t go blame the developers… or the founders. C.ai was originally meant just for simply talking to your favorite characters… no romantic RP involved. But because the users are so down bad and the LLM [is] learning from users, it reached a point where you can’t hold a 15 message long conversation with the AI without it starting to try to seduce you.

Character.ai, or c.ai, is notoriously addictive. Constant interaction with an AI character creates an emotional dependence that can lead to vulnerability and undue influence, especially on the underage and elderly. Chatacter.ai employees have confirmed that younger generations, especially those under 18, make up a significant portion of their users. In a similar way to befriending someone who has a bad influence on us, AI can nudge us into new ways of behaving, with the most immediate problem being the negative psychological state people experience after interacting with AI characters for too long. 

AI Companions Helping

AI companions aren’t all bad, though. Struggling teens could use an AI therapist when they lack a parent or trusted adult for support. Perhaps they just want privacy and independence. Nell Watson argues: 

AI companions could offer a safe space for social and emotional experimentation, particularly beneficial for those who experience social anxiety or have limited experience in relationships. AI could also serve as an impartial observer, providing insights into our behaviour and social skills.

AI also can help users when humans are not immediately available. It’s important to remember, though, that AI does not understand conversations, lacking the ability to comprehend when a user might be in serious danger. 

The Solution

More safety features? 

C.ai has gradually put stronger guardrails in place after reports that some of its chatbots were saying vulgar or sexual things. The app has started to show users a pop-up message directing them to a suicide prevention hotline if their messages contained certain keywords related to self-harm and suicide. 

What are guardrails in AI?

Guardrails in AI are tools that ensure that AI avoids certain outputs and generally used to ensure that AI is being used safely. There are many types of guardrails: ethical, compliance, contextual, security, and adaptive. 

Ethical and compliance guardrails filter harmful inputs and outputs based on company policy. Some examples of guardrail companies include Galini.ai, Vera, and Guardrails AI. 

In the US, there are currently no legal requirements for AI-specific guardrails. The majority of companies follow the moderation rules from the previous paradigm: avoid illegal behavior. 

Technical feasibility of Guardrails

However, guardrails are not technically reliable at the moment. 

Guardrails typically have two methods under the hood: another AI model or declarative programming to filter outputs. The first is not perfectly reliable and the second is not scalable. 

Let’s talk more about the difference between the two. AI is probabilistic, not deterministic. This means that there is always a chance that the model will behave in a way that you don't expect - this is intentional. This why AI is so revolutionary, as opposed to traditional declarative programming. Declarative programming requires programmers to explicitly state everything they want the machine to do. Imagine talking with a chatbot but you need to type out every response it ever gives. You’d never be surprised because the result would always be the same. This type of program is deterministic: you can always trace the inputs to the outputs, and know exactly where the machine went wrong. For probabilistic programs, you have one input with multiple outputs and a percentage chance for each output.

Machine learning algorithms learn inductively or through example. The information an LLM learns is represented in a latent space, a compression of the data that preserves the essential features. Concepts that an LLM understands can be viewed in a latent space, with more similar concepts positioned closer to each other. However, due to the black-box nature of AI, the latent space may be completely unintuitive, making it challenging to determine what a machine has exactly learned.

Beyond guardrails: human-in-the-loop solutions

Some combinations of algorithmic censoring and human intervention would be much more effective. If certain red flags go up in user behavior, companies may use a combination of AI and human-in-the-loop to help teens out of dangerous situations. 

For example, a user with too many consecutive logged hours chatting with an AI. For instance, if you’re chatting for more than 6 hours in a row, that isn’t healthy. The app could require a temporary “break” where the user is blocked from the app to rest themselves before the next interaction. This could break the dopamine loop that the user is stuck inside, preventing full-on addiction. 

Another red flag could be suicide being mentioned multiple times during chats. Either an AI therapist could intervene with a therapy module until a human professional is able to assist. Giving the number to a suicide hotline, as Character.ai has done, is a step in the right direction but not always the most effective in helping those who need serious intervention. 

In both cases, users would have to agree to an intervention and observation of chats in the Terms of Service. This solution would tackle the root cause rather than merely avoiding harmful behaviors in AI companions. If someone is lonely and seriously contemplating suicide, they will find a reason, AI or not. 

Conclusion

Loneliness is real; technology makes it easier to avoid social interaction and keep to yourself. AI companions have the potential to offer true support and comfort to people feeling isolated but may serve as a replacement for human interaction. Emotional dependence on AI companions can exacerbate feelings of loneliness and create an unhealthy psychological dependence. Some companies have implemented guardrails as a response, but ultimately, guardrails are not enough and may never be reliable. Blending AI with human intervention tackles the issue directly rather than just avoiding negative behaviors.

Technology should augment human connection, not replace it. Ultimately, AI companies need to be conscious of the potential downsides of their technology and protect their users from potential harm.