The AI Challenger disaster prediction

What happened with the Challenger disaster is lots of people knew that those little O-rings were unreliable, but they kept on launching space shuttles and everything was fine. But every single time you get away with launching a space shuttle without the O-rings failing, you institutionally feel more confident in what you're doing. The problem we've been having with prompt injection is that we've been using these systems in increasingly unsafe ways. And so far, there hasn't been a headline-grabbing story of a prompt injection where an attacker has stolen $1 million. Which means that we keep on taking risks. We have this normalization of deviance in the field of AI around how we're using these tools. So my prediction is that we're going to see a Challenger disaster. At some point, this is going to catch up with us and it's going to be very, very, very bad. And that will hopefully help us start trying to figure out how not to do this. At the same time, I've made a version of this prediction every 6 months for the past 3 years, and it hasn't happened.