Deaths linked to chatbots show we must urgently revisit what counts as ‘high-risk’ AI

Last week, the tragic news broke that US teenager Sewell Seltzer III took his own life after forming a deep emotional attachment to an artificial intelligence (AI) chatbot on the Character.AI website. As his relationship with the companion AI became increasingly intense, the 14-year-old began withdrawing from family and friends, and was getting in trouble at school.

In a lawsuit filed against Character.AI by the boy’s mother, chat transcripts show intimate and often highly sexual conversations between Sewell and the chatbot Dany, modelled on the Game of Thrones character Danaerys Targaryen. They discussed crime and suicide, and the chatbot used phrases such as “that’s not a reason not to go through with it”.



This is not the first known instance of a vulnerable person dying by suicide after interacting with a chatbot persona. A Belgian man took his life last year in a similar episode involving Character.AI’s main competitor, Chai AI.

When this happened, the company told the media they were “working our hardest to minimise harm”. In a statement to CNN , Character.AI has stated they “take the safety of our users very seriously” and have introduced “numerous new safety measures over the past six months”.

In a separate statement on the company’s website, they outline additional safety measures for users under the age of 18. (In their current terms of service, the age restriction is 16 for European Union citizens and 13 elsewhere in the world.) However, these tragedies starkly illustrate the dangers of rapidly developing and widely available AI systems anyone can converse and interact with.

We urgently need regulation to protect people from potentially dangerous, irresponsibly designed AI systems. How can we regulate AI? The Australian government is in the process of developing mandatory guardrails for high-risk AI systems. A trendy term in the world of AI governance, “guardrails” refer to processes in the design, development and deployment of AI systems.

These include measures such as data governance, risk management, testing, documentation and human oversight. One of the decisions the Australian government must make is how to define which systems are “high-risk”, and therefore captured by the guardrails. The government is also considering whether guardrails should apply to all “general purpose models”.

General purpose models are the engine under the hood of AI chatbots like Dany: AI algorithms that can generate text, images, videos and music from user prompts, and can be adapted for use in a variety of contexts. In the European Union’s groundbreaking AI Act , high-risk systems are defined using a list , which regulators are empowered to regularly update. An alternative is a principles-based approach, where a high-risk designation happens on a case-by-case basis.

It would depend on multiple factors such as the risks of adverse impacts on rights, risks to physical or mental health, risks of legal impacts, and the severity and extent of those risks. Chatbots should be ‘high-risk’ AI In Europe, companion AI systems like Character.AI and Chai are not designated as high-risk.

Essentially, their providers only need to let users know they are interacting with an AI system. It has become clear, though, that companion chatbots are not low risk. Many users of these applications are children and teens .

Some of the systems have even been marketed to people who are lonely or have a mental illness . Chatbots are capable of generating unpredictable, inappropriate and manipulative content. They mimic toxic relationships all too easily.

Transparency – labelling the output as AI-generated – is not enough to manage these risks. Even when we are aware that we are talking to chatbots, human beings are psychologically primed to attribute human traits to something we converse with. The suicide deaths reported in the media could be just the tip of the iceberg.

We have no way of knowing how many vulnerable people are in addictive, toxic or even dangerous relationships with chatbots. Guardrails and an ‘off switch’ When Australia finally introduces mandatory guardrails for high-risk AI systems, which may happen as early as next year, the guardrails should apply to both companion chatbots and the general purpose models the chatbots are built upon. Guardrails – risk management, testing, monitoring – will be most effective if they get to the human heart of AI hazards.

Risks from chatbots are not just technical risks with technical solutions. Apart from the words a chatbot might use, the context of the product matters, too. In the case of Character.

AI, the marketing promises to “empower” people , the interface mimics an ordinary text message exchange with a person, and the platform allows users to select from a range of pre-made characters, which include some problematic personas. Truly effective AI guardrails should mandate more than just responsible processes, like risk management and testing. They also must demand thoughtful, humane design of interfaces, interactions and relationships between AI systems and their human users.

Even then, guardrails may not be enough. Just like companion chatbots, systems that at first appear to be low risk may cause unanticipated harms. Regulators should have the power to remove AI systems from the market if they cause harm or pose unacceptable risks.

In other words, we don’t just need guardrails for high risk AI. We also need an off switch. If this article has raised issues for you, or if you’re concerned about someone you know, call Lifeline on 13 11 14.

.