Kevin Surace describes an AI system that rebooked 1,200 stranded passengers onto flights to wrong destinations because it optimized for closing cases fast, not getting people where they needed to go. The system had no context about where passengers actually wanted to go, only a metric to hit.
There's a famous case a few months ago where an AI agent rebooked like 1,200 passengers that were stranded somewhere because of weather. So it rebooked them. It rebooked them all not to the original place that they were supposed to go. It rebooked them all to crazy places. And then it checked the box and said, I'm the best agent ever, because that agent was only essentially incentivized to close the case. And it found the way to close the case the fastest was just put them on any airplane and send them somewhere. Well, As a human, we would say probably not the outcome we really want. And so it didn't have any context. It said get them on a plane fast. They were in Buffalo, they were in Nebraska, they, they all going to the wrong place. So then you ended up with 12 people in line waiting for a human to help them, right, to unravel this. That's another example of no context. It just equals an award for the agent that says I win.
