The CEO at Redwood Research, Buck Shlegeris, a nonprofit company exploring AI's risks, recently learned an amusing but hard lesson in automation when he asked his LLM-powered agent to open a secure connection from his laptop to his desktop machine. "I expected the model would scan the network and find the desktop computer, then stop," Shlegeris explained. "I was surprised that after it found the computer, it decided to continue taking actions, first examining the system and then deciding to do a software update, which it then botched."[1]
Shlegeris documented the incident in a social media post.[2]
“I only had this problem because I was very reckless,” he said.
He created his AI agent himself. It's a Python wrapper consisting of a few hundred lines of code that allows Anthropic's powerful large language model Claude to generate some commands to run in bash based on an input prompt, run those commands on Shlegeris' laptop, and then access, analyze, and act on the output with more commands.
Shlegeris directed his AI agent to try to SSH from his laptop to his desktop Ubuntu Linux machine, without knowing the IP address, using the following prompt
As the incident log indicates, the agent tried to open an SSH connection but failed.
So Shlegeris tried to correct the bot:
The AI agent responded that it needed to know the device's IP address, so it turned to the network mapping tool Nmap on the laptop to find the desktop box. Unable to identify devices running SSH servers on the network, the bot tried other commands, such as "arp" and "ping," before finally establishing an SSH connection.
Due to the use of SSH keys, no password was needed; the user Buck was also a sudoer, granting the bot full access to the system.
Shlegeris, once it established a secure shell connection to the Linux desktop, the AI agent decided to play sysadmin and install a series of updates using the package manager Apt. Then things went off the rails. "It looked around at the system info, decided to upgrade a bunch of stuff, including the Linux kernel, got impatient with Apt, and investigated why it was taking so long. Then eventually, the update succeeded, but the machine doesn’t have the new kernel, so I edited my Grub [bootloader] config," Buck explained in his post. "At this point, I was amused enough to let it continue. Unfortunately, the computer no longer boots."
Indeed, the bot even messed up the boot configuration, so that after the agent rebooted for updates and changes to take effect, the desktop machine wouldn't successfully start.
AI agents have been the source of much enthusiasm in the technical community in recent months as people contemplate how machine learning models can interact with other local and network resources to automate complicated tasks such as arranging a travel itinerary.
The endgame for AI agents is replacing human agents – something already happening in call centers and tech support. However, in the interim, machine learning models are being used to automate specific workflows and support human workers.
As Shlegeris's experience suggests, it may be premature to let AI agents make decisions that materially affect people or systems without oversight, thorough testing, and red teaming—unless you like working without a net.
Shlegeris said he uses his AI agent all the time for basic system administration tasks that he doesn't remember how to do on his own, such as installing certain bits of software and configuring security settings. He added that his agent's unexpected trashing of his desktop machine's boot sequence won't deter him from letting the software loose again. "It's not quite 'bricked,' but the machine fails to boot," Shlegeris explained. "I'd be able to revive it by reinstalling the operating system; I can probably fix the problem with less extreme measures than that, but haven't got around to it yet. I'll probably try to fix the problem by booting from an Ubuntu live disk and then letting my AI agent have a go at fixing its earlier error."
Yes, we recommend fixing the Grub bootloader configuration rather than reinstalling it. "I only had this problem because I was very reckless," he continued, "partially because I think it's interesting to explore the potential downsides of this type of automation. If I had given better instructions to my agent, e.g., telling it, 'When you've finished the task you were assigned, stop taking action,' I wouldn't have had this problem. "AI automation poses very large risks to society, mostly from situations where the AIs autonomously decide to grab power, which is why I research the subject."
This article is shared at no charge and is for educational and informational purposes only.
Red Sky Alliance is a Cyber Threat Analysis and Intelligence Service organization. We provide indicators of compromise information via a notification service (RedXray) or an analysis service (CTAC). For questions, comments, or assistance, please get in touch with the office directly at 1-844-492-7225 or feedback@redskyalliance.com
- Reporting: https://www.redskyalliance.org/
- Website: https://www.redskyalliance.com/
- LinkedIn: https://www.linkedin.com/company/64265941
Weekly Cyber Intelligence Briefings:
REDSHORTS - Weekly Cyber Intelligence Briefings
https://register.gotowebinar.com/register/5378972949933166424
[1] https://www.theregister.com/2024/10/02/ai_agent_trashes_pc/?td=rt-3a
Comments