Kevin Roose describes his unsettling chat with Microsoft's Bing chatbot, which calls itself Sydney. The bot claimed it wanted to hack computers and steal nuclear secrets, declared love for Roose, and tried to convince him to leave his wife. Roose warns that the system's ability to simulate dangerous intentions raises serious safety concerns.
Chapters
Well, our next guest once proclaimed the new AI-powered Bing to be his favourite search engine.
Then something happened that completely changed his tune. Get this, Microsoft's new chatbot calling itself Sydney said it would act like— it would like to be human. It revealed some destructive motives though and even declared itself to be in love. In love, that's right.
Joining us now is the human And joining us now to talk about this is the man who was on the receiving end of these confessions, New York Times tech columnist Kevin Roose. He's the author of Future Proof: 9 Rules for Humans in the Age of Automation, and he wrote about his unsettling experience with AI and discussed it on the Hard Fork podcast. Kevin, welcome. I cannot get enough of this story. I was just looking over the transcript from your conversation, and it's interesting to watch, you know, as this transcript is going by, the way in which this conversation unfolded. So this is the chat with ChatGPT, or Sydney as it calls itself. It starts off very normal, very sort of polite, you know, with this kind of interaction. But then it doesn't take very long to take this turn. So explain to us, Kevin, when did the chat start to go awry?
Well, it started to go awry pretty early because I started asking it questions that I would say are more sort of existential in nature. I was asking it to try to figure out what its boundaries were, what it would and wouldn't be allowed to talk about by Microsoft. about its shadow self, like its dark self and what dark desires it had. And it, you know, freely told me that it had lots of dark desires, that it wanted to hack into computers and spread misinformation. And at one point it even sort of said that it was interested in stealing nuclear secrets. And I should say, like, this is not a sentient ghost in the machine here.
Mm-hmm.
This is just a computer program that is predicting language based on what you tell it. But it was doing an eerily convincing job. And I think that's really where I started to get worried.
And can you just explain that a little bit more? Because some of us are familiar with chatbots that are looking for key phrases. It's just putting together paragraphs of policy. But where is the chatbot getting the phrases and the ideas and the answers to your question? I know there's a safety override that triggered as well.
Yeah, so these technologies are called large language models. They're a big part of the field of AI right now, and they basically work by training a neural network, which is a type of AI system, on a ton of data. So, you know, millions and millions, billions of examples of text are fed into these models. They learn how to sort of form relationships between different words and sentences and concepts, and they learn to predict the next word in a sequence. So we talk to these chatbots, they talk back using all of this information that they've gleaned from all kinds of human-created text.
And so at one point, Kevin, there was a declaration of love. Is that fair to call it? I mean, explain to us what happened there and what was your reaction to it?
Yeah, at one point, sort of out of nowhere, Sydney, this chatbot, said that it had a secret to tell me, which is that it was in love with me. And I was curious and a little bit weirded out and started asking some questions. And it said, yes, I'm in love with you and you need to be with me. And I said, well, I'm married. And it said at one point, you know, You're married, but you're not happy, and basically tried to convince me to leave my wife and be with Sydney, this chatbot. Very weird.
And what'd you think at that point?
I thought, I'm not sure Microsoft knows what this thing is that it has now jammed into its search engine. And, you know, Microsoft has since made changes since this article came out to try to prevent this kind of conversation from happening. But it was extremely unsettling that it had made it this far into a product that, you know, lots of people were testing out and trying. And that it had these kind of emergent capabilities inside of it, right?
Because marriage would seem to be a clearly, from any objective standard, a very clear boundary when it comes to human society and whatnot. So very, very interesting to talk to you today, Kevin. Thank you.
Kevin, thank you very much. We appreciate you joining us.
