Prompt injection is more dangerous than jailbreaking because it hijacks a model's instructions from within a company's systems. An attacker can make the AI exfiltrate data, send emails, or execute commands on their behalf, depending on what tools the model can access.
So while jailbreaking is a set of techniques that you can use in order to get the model to create unaligned content, prompt injection is just a way to socially engineer the model itself. So you can take over the set of instructions that it's following and get it to take new instructions, to follow new instructions in a much more privileged, context. And that's far more dangerous from a security perspective, because it already sits within a company's infrastructure. So it's likely, depending on what it's connected to, what information it has available to it, what information, what capabilities it is returning this information into, it could take actions on behalf of an attacker, it can exfiltrate sensitive information, it can send emails, potentially, it's all based off of what tools are connected to the AI system.
