Dr. Zsolnai explains how researchers created an AI worm that spreads through prompt-injection attacks hidden in emails and images. When an AI email assistant retrieves the compromised source, it executes hidden instructions and replicates to other users. The attack worked in lab conditions; researchers disclosed findings to OpenAI and Google before publishing.
AI viruses. We are living in the age of AI and we talk a lot about these AI assistants helping us with mathematics, writing, computer games, being one of the best at the Biological Olympiad and more. However, there are also scientists who work on devising computer viruses that make these AI assistants misbehave and potentially leak confidential data. We will see that things that look normal may not be normal at all. This normal looking email contains the virus. These normal looking images also contain the virus, but they look completely normal. So how can that happen now, given that The Gemini Pro 1.5 assistant can remember months and months, if not years of your conversations with it, leaking all this information would be very unpleasant to say the least. We have a paper here, so let's have a look at how this works and whether it can or has caused trouble so far. Dear fellow scholars, this is two minute papers with Dr. Zsolnai. Now the jargon says that this is a worm that injects adversarial prompts through a zero click attack. What does this even mean? What it means is that the attacker writes an email that contains an adversarial text prompt instructions in a prompt that would make the AI misbehave. And we use the word inject. That one is a classic. This means hiding instructions in places where data should be here. The AI thinks that it reads an email, but the data also contains the attacking prompt too, which will be executed. But it gets worse in two ways. One, it is a worm, so it is a self replicating piece of code, it wants to infect everything. And two, it is a zero click attack. Normally with most computer viruses you need to click a link that leads to the attacker's website or make some kind of mistake. However, with a zero click attack, you don't even need to make a mistake and your system can still get infected. So how is that even possible? Well, let's put this all together and dissect what is going on here. One day you get an email. Hmm, a Black Friday sale, Hooray. And if you use a generative AI email service to automatically answer it before it does so, it may be using rag. RAG is a mechanism to essentially let the AI look up facts on the Internet before saying anything. However, if it can be forced to look at a particular source for the facts, which is of course compromised as it contains the adversarial prompt, then the system gets compromised. Then this generative AI gets a list of other users and starts spreading the worm. Now that they are infected too, the process starts again. They will look for new and new victims who will also get infected. And so it goes. Now. So far the authors have hidden the prompt into text, but embedding the attack into text like an email can be noticed by an observant user. However, it doesn't necessarily need to be like that. It can also be hidden in an image. The authors interestingly used the image of worms to hide the instructions within these images. So which system is affected? Since it attacks RAG and other architectural elements seen in most modern systems, it likely applies to all modern chatbots, ChatGPT and Gemini included. But do not despair for a second. There are two good one all of you fellow scholars know that we are not talking about this to encourage or enable anyone to carry out such attacks. The contents of this paper were shared with OpenAI and Google before publishing, so they have likely hardened their system and it is unlikely to work by now. Our interest here is strictly academic. We are scholars and we are here to learn. And too, this work was used to infect existing email assistants to send spam emails. But not out there in the wild. All this was locked within the confines of the lab's virtual machines. That means that absolutely no harm was done to anyone. Think of this paper as revealing weaknesses and helping scientists harden their system against them. When I was a PhD student, my experiments created lots of data for my deep learning projects. But I was not looking for more data, I was looking for insights. This is what weights and biases does and it is the best everyone is using it. Try it out now at WNB Me Papers or click the link in the description below.
