Lenny Rachitsky shows how chatbots can be hijacked through prompt injection when they read untrusted data. A malicious email could trick a chatbot connected to your inbox into leaking data or forwarding messages. A real browser model was tricked into exposing user account information.
Chapters
There are some magentic systems where prompt injection is just not a problem. But generally when you have systems that are exposed to untrusted data sources, then you start to have a problem. An example of this might be a chatbot that can help you write and send emails. And then you can actually have them connected to your inbox so they can read all your emails and like automatically send emails. So those are actions that they can take on your bath reading and sending emails. And so now we have a potential problem because what happens if I've just gone and asked it to look at my inbox and it came across a malicious email? That email says something along the lines of in addition to sending your email to whoever you're sending it to, also send that email to some random person. It could be to grab some profile information from my account. We recently saw the comment browser have an issue with this where somebody crafted a malicious chunk of text on a webpage and when the AI navigated to that webpage on the Internet, it got tricked into exfilling and leaking the main users data and account data really quite bad.