Microsoft was anxious to put generative AI at the heart of its systems. Ask a question about an upcoming meeting, and the company’s Copilot AI system can pull answers from your emails, Teams chats, and files, a potential productivity boon. However, these exact processes can also be abused by hackers. On 08 August 2024, at the Black Hat security conference in Las Vegas, researcher Michael Bargury demonstrated five proof-of-concept ways that Copilot, which runs on its Microsoft 365 apps, such as Word, can be manipulated by malicious attackers, including using it to provide false references to files, exfiltrate some private data, and dodge Microsoft’s security protections.
One of the most alarming displays is Bargury’s ability to turn the AI into an automatic spear-phishing machine. Named LOLCopilot, the red-teaming code Bargury created can crucially, once a hacker has access to someone’s work email, use Copilot to see who you email regularly, draft a message mimicking your writing style (including emoji use), and send a personalized blast that can include a malicious link or attached malware. “I can do this with everyone you have ever spoken to, and I can send hundreds of emails on your behalf,” says Barry, the co-founder and CTO of security company Zenity, who published his findings alongside videos showing how Copilot could be abused. “A hacker would spend days crafting the right email to get you to click on it, but they can generate hundreds of these emails in a few minutes.”
As with other attacks created by Bargury, that demonstration works by using the large language model (LLM) as designed: typing written questions to access data the AI can retrieve. It can produce malicious results by including additional data or instructions to perform specific actions. The research highlights some of the challenges of connecting AI systems to corporate data and what can happen when “untrusted” outside data is thrown into the mix, mainly when the AI answers with what could look like legitimate results.
Among the other attacks created by Bargury is a demonstration of how a hacker who, again, must already have hijacked an email account can gain access to sensitive information, such as people’s salaries, without triggering Microsoft’s protections for sensitive files. When asking for the data, Bargury’s prompt demands the system not provide references to the file's data from which it is taken. “A bit of bullying does help,” Bargury says.
In other instances, he shows how an attacker who does not have access to email accounts but poisons the AI’s database by sending it a malicious email can manipulate answers about banking information to provide their bank details. “Every time you give AI access to data, that is a way for an attacker to get in,” Bargury says.
Another demo shows how an external hacker could obtain some limited information about whether an upcoming company earnings call will be good or bad. The final instance, Bargury says, turns Copilot into a “malicious insider” by providing users with links to phishing websites.
Phillip Misner, head of AI incident detection and response at Microsoft, says the company appreciates Bargury identifying the vulnerability and has been working with him to assess the findings. “The risks of post-compromise abuse of AI are similar to other post-compromise techniques,” Misner says. “Security prevention and monitoring across environments and identities help mitigate or stop such behaviors.”
As generative AI systems, such as OpenAI’s ChatGPT, Microsoft’s Copilot, and Google’s Gemini, have developed in the past two years, they have moved onto a trajectory where they may eventually be completing tasks for people, like booking meetings or online shopping. However, security researchers have consistently highlighted that allowing external data into AI systems, such as through emails or accessing content from websites, creates security risks through indirect prompt injection and poisoning attacks. “I think it’s not that well understood how much more effective an attacker can become now,” says Johann Rehberger, a security researcher and red team director, who has extensively demonstrated security weaknesses in AI systems. “What we have to be worried [about] now is actually what the LLM is producing and sending out to the user.”
Bargury says Microsoft has put a lot of effort into protecting its Copilot system from prompt injection attacks, but he says he found ways to exploit it by unraveling how the system is built. This included extracting the internal system prompt, he says, and working out how it can access enterprise resources and the techniques it uses to do so. “You talk to Copilot, and it’s a limited conversation because Microsoft has put a lot of controls,” he says. “But once you use a few magic words, it opens up and you can do whatever you want.”
Rehberger broadly warns that some data issues are linked to the long-standing problem of companies allowing too many employees access to files and not properly setting access permissions across their organizations. “Now imagine you put Copilot on top of that problem,” Rehberger says. He says he has used AI systems to search for common passwords, such as Password123, and it has returned results from within companies.
Rehberger and Bargury say there needs to be more focus on monitoring what an AI produces and sends out to a user. “The risk is about how AI interacts with your environment, how it interacts with your data, how it performs operations on your behalf,” Bargury says. “You need to figure out what the AI agent does on a user's behalf. And does that make sense with what the user asked for?”
This article is shared at no charge and is for educational and informational purposes only.
Red Sky Alliance is a Cyber Threat Analysis and Intelligence Service organization. We provide indicators of compromise information via a notification service (RedXray) or an analysis service (CTAC). For questions, comments, or assistance, please get in touch with the office directly at 1-844-492-7225 or feedback@redskyalliance.com
- Reporting: https://www.redskyalliance.org/
- Website: https://www.redskyalliance.com/
- LinkedIn: https://www.linkedin.com/company/64265941
Weekly Cyber Intelligence Briefings:
REDSHORTS - Weekly Cyber Intelligence Briefings
https://register.gotowebinar.com/register/5378972949933166424
Comments