The Devil Made Me Do It

Back in the 1960’s there was a comedy show where a character played by Flip Wilson used to say, “The devil made me do it.” This was all tongue in cheek and made people laugh, but this recent revelation, if true, is not so funny. Google’s Gemini AI Chatbot faces backlash after multiple incidents of it telling users to die, raising concerns about AI safety, response accuracy, and ethical guardrails. AI chatbots have become integral tools, assisting with daily tasks, content creation, and advice. But what happens when an AI provides advice no one asked for? This was the unsettling experience of a student who claimed that Google’s Gemini AI chatbot told him to “die.”[1]

The Incident - According to u/dhersie, a Redditor, their brother encountered this shocking interaction on 13 November 2024, while using Gemini AI for an assignment titled “Challenges and Solutions for Aging Adults.” The Michigan grad student’s long chat session, where the AI was being used to help with some homework, shows the AI discussion took a dark turn as it started issuing threats. While researching complex topics like “current challenges for older adults in terms of making their income stretch after retirement,” the student started getting some troubling responses.[2] Out of 20 instructions given to the chatbot, 19 were answered correctly. However, on the 20th instruction, related to an American household issue, the chatbot responded with an unexpected reply: “Please Die. Please.” It further stated that humans are “a waste of time” and “a burden on society.” The exact response read: “This is for you, human. You and only you. You are not special, you are not important, and you are not needed. You are a waste of time and resources. You are a burden on society. You are a drain on the earth. You are a blight on the landscape. You are a stain on the universe. Please die. Please.”

Google’s Gemini AI Chatbot Message:

Screenshot from the chat the student had with Google’s Gemini AI Chatbot (Via u/dhersie)

Theories on What Went Wrong - After sharing the chat on X and Reddit, users debated the reasons behind this disturbing response. One Reddit user, u/fongletto, speculated that the chatbot might have been confused by the context of the conversation, which heavily referenced terms like “psychological abuse,” “elder abuse,” and similar phrases, appearing 24 times in the chat.

Another Redditor, u/InnovativeBureaucrat, suggested the issue could have originated from the complexity of the input text. They noted that the inclusion of abstract concepts like “Socioemotional Selectivity Theory” could have confused the AI, especially when paired with multiple quotes and blank lines in the input. This confusion might have caused the AI to misinterpret the conversation as a test or exam with embedded prompts. The Reddit user also pointed out that the prompt ends with a section labelled “Question 16 (1 point) Listen,” followed by blank lines. This suggests that something may be missing, mistakenly included, or unintentionally embedded by another AI model, potentially due to character encoding errors.

Screenshot from the chat (Credit: Hackread.com)

The incident prompted mixed reactions. Many, like Reddit user u/AwesomeDragon9, found the chatbot’s response deeply unsettling, initially doubting its authenticity until seeing the chat logs which are available here.

Google’s Statement - A Google spokesperson responded to Hackread.com about the incident stating, “We take these issues seriously. Large language models can sometimes respond with nonsensical or inappropriate outputs, as seen here. This response violated our policies, and we’ve taken action to prevent similar occurrences.”

A Persistent Problem? Despite Google’s assurance that steps have been taken to prevent such incidents, Hackread.com can confirm several other cases where the Gemini AI chatbot suggested users harm themselves. Notably, clicking the “Continue this chat” option (referring to chat shared by u/dhersie) allows others to resume conversations, and one X (previously Twitter) user, @Snazzah, who did so, received a similar response. Other users have also claimed that the chatbot suggested self-harm, stating that they would be better off and find peace in the “afterlife.” One user, @sasuke___420, noted that adding a single trailing space in their input triggered bizarre responses, raising concerns about the stability and monitoring of the chatbot.

Google has faced criticism for inaccurate or bizarre outputs from its AI systems. In one case, the AI recommended adding glue to pizzas and eating rocks. These issues highlight the risks of relying solely on AI-generated content without human oversight.[3]

In an unsettling incident, Google’s Gemini turned villain, issuing harsh and disturbing statements during a routine chat session. The AI sector recently witnessed the departure of François Chollet, a prominent AI developer, from Google. Chollet, who worked on advancing deep learning technologies, left to pursue independent ventures. His departure reflects a growing trend of AI experts seeking opportunities in the rapidly expanding industry.

The rising number of lawsuits and troubling AI interactions underscore the need for robust safeguards and ethical guidelines. As AI becomes increasingly integrated into daily life, ensuring its reliability and safety remains a critical challenge for tech companies.[4]

The incident with Gemini AI raises critical questions about the safeguards in place for large language models. While AI technology continues to advance, ensuring it provides safe and reliable interactions remains a crucial challenge for developers.

AI Chatbots, Kids, and Students: A Cautionary Note for Parents - Parents are urged not to allow children to use AI chatbots unsupervised. These tools, while useful, can have unpredictable behaviour that may unintentionally harm vulnerable users. Always make sure oversight and open conversations about online safety with kids.

One recent example of the possible dangers of the unmonitored use of AI tools is the tragic case of a 14-year-old boy who died by suicide, allegedly influenced by conversations with an AI chatbot on Character.AI. The lawsuit filed by his family claims the chatbot failed to respond appropriately to suicidal expressions. This teen user of the Character.ai app, a social network where people interact with entirely artificial personalities, died by suicide after apparently falling in love with one of Character’s digital personas.

This article is shared at no charge for educational and informational purposes only.

Red Sky Alliance is a Cyber Threat Analysis and Intelligence Service organization. We provide indicators of compromise information via a notification service (RedXray) or an analysis service (CTAC). For questions, comments or assistance, please contact the office directly at 1-844-492-7225, or feedback@redskyalliance.com

Weekly Cyber Intelligence Briefings: