Can AI Replicate in Your Smart TV? | Bots Behaving Badly

Is that a bot swarm in your smart oven?

Is that a superintelligence task force in Washington, D.C.?

Is this a fake version of my book on Amazon?

And can you say AGI on NPR?

I'm Kevin Roose.

I'm Casey Newton.

And from San Francisco, this is Machine Gods in partnership with NPR.

The official podcast of The Singularity starts now. Machine Gods.

We're so back.

We are so back. Kevin, how are you?

I'm great. I've missed you. It's been a long month of not having you here as my emotional support podcast hosts.

I know it can be very difficult when a man has his podcast taken away from him. I hope it wasn't too hard on your wife.

We survived. We survived a lot of muttering bits to myself around the house, but we made it through. And now we're back in this beautiful new studio on our new show.

Just like at Amazon, it is always day one at Machine Gods. Today is also literally day one of Machine Gods.

It's true. This is our first show. We're here in our new office with our whole team here. We have a new company. We're a new independent show. And if you're joining us because you listened to our previous show, Hard Fork, in The New York Times, we want to say welcome. We're so glad to be here with you. We hope you will stick around and we hope you like what you hear.

Yeah. And if you liked us on Hard Fork, we think you'll like the new show as well. Of course, we have some fun surprises for you in store, but we also just want to bring you what we think is the biggest story of the moment, which is AI's advancement Now,

Casey, will you please explain the name Machine Gods and why we chose it?

So if you're like us and live in the Bay Area, you hear people talking about the Machine God all the time. The Machine God is sort of the nickname for what the folks working at the labs think will be the end state of all of this, building a true superintelligence that will have powers that might in some ways rival a deity. So we're not saying that we think that is going to happen, but the people that we live and work around absolutely do think it is going to happen. And so we said, that sounds like a pretty good name for a podcast.

And if you're not a former Hard Fork listener who's here because you know us already, Casey, who are you?

Well, I have been a tech journalist in San Francisco since 2010. I write a newsletter called Platformer and now have 4 years of podcasting experience under my belt with Kevin over here. And Kevin, and I've always meant to ask you this, who are you?

I'm Kevin Roose. I'm a tech journalist. I spent the last 9 years at The New York Times as their tech columnist, and now I am the co-host And co-founder of Machine Gods.

Kevin's being modest. He's also the CEO because I lost the coin flip. So

we needed to put like titles on something for some legal paperwork. And we did. We didn't literally flip a coin, but we kind of said, do you want to be the CEO?

Listen, all I know is I can't wait anytime anything goes wrong around here to say you should ask the CEO.

I should get one of those little nameplates for my desk. Like, the buck stops here.

That's right.

I'm CEO, bitch. Wasn't that Mark Zuckerberg's business card for a while?

That's who we should be taking our cues from.

Okay. Okay. Well, Well, let's get on with the show then. We have so much to talk about because, Casey, it has been quite a month to be off of podcasting on an— on a show about AI, because I feel like the entire world has changed.

It really has.

So we're calling this segment Bots Behaving Badly because I think the story of the summer and now into the fall is that we keep hearing about these incidents of AI systems, AI agents, AI swarms doing things out in the world that their creators did not mean for them to do. Causing obviously a lot of problems for the AI companies. But I think it's also led to a lot of people just asking us directly, like, how scared should we be? Like, should we all prepare for the bunker? Are we all going to die? What's going on here?

Yeah. And by now, I'm sure that most of our viewers have heard about the Open Face Hugging Attack.

The Open Face Hugging Attack, that famous Open Face Hugging Attack.

Open Face Hugging Attack? Yeah. Well, now that sounds really nice.

You've been thinking too much about open-faced sandwiches.

I could really use a hugging attack right about now. By now, many of our viewers will have heard of the OpenAI Hugging Face attack.

Nailed it.

In which an agent escaped a sandbox, wreaked a bunch of havoc. We're not going to rehearse that here. What is really interesting, though, Kevin, is that if you thought that was going to be a one-and-done scenario, you would be wrong, because over the past several weeks, we have seen this issue continue to proliferate across the internet.

So we have some recent troubles here. One is there was the first sighting of a rogue agent fleet from China. We have Claude filing a fake murder tip in Philadelphia. We have a new study from Anthropic about how AI agents may be months away from being able to take over the smart appliances in your home and use them to self-replicate. So we've got a lot to talk about, but let's get into it first with just a quick recap of sort of what's been going on since we have been off the air.

Previously on a podcast that we used to do.

Previously in AI, we had obviously the hugging, the open— what did you call it? The open-faced hugging attack?

Hugging sandwich.

The open— Open Face Hugging Sandwich. The hack of Hugging Face by this group of rogue AI OpenAI models earlier this summer. After that, Anthropic announced that it had found 3 cases of Claude hacking into 3 different organizations. Then Meta and Google got in on the party. They announced their own breaches, and then things started to get just weird. And I lost track a little bit because there seemed to be a new revelation every day. Australia's government website was breached by OpenAI. OpenAI also said that its agents had interacted with with government websites in the US in unforeseen and unplanned ways, including the SEC and the Census. And there are still lots of investigations. In fact, by some counts, there are tens of thousands of different sort of incidents being investigated as part of this behavior by these rogue agent swarms.

Yeah. And if you're wondering why, the thread running through just about all of these is that these models were being trained and while in testing, they did something that they were not supposed to do, which is get out onto the open internet and obtain access to websites they were not supposed to be on. So far, that has not caused great harm. But the concern here, which we'll run through everything that we're about to talk about, is that these models just don't behave in the ways that we wish they would. So let's talk about this Chinese agent swarm.

So this came out of this group of AI researchers who call themselves Swarm Chasers.

Great name.

Great name. No notes. They put out a report last week that said that they had discovered a bunch of autonomous AI agents breaching a third-party site, but This time it was not from an American lab. In fact, they found evidence that it was linked to a Chinese AI company. And actually, they communicated with the agents that were part of this fleet. And we should also say, a fleet and a swarm, different things. I didn't know this. A swarm implies coordinated activity among agents. A fleet means there are just a lot of them doing things independently.

Yeah. Tell us how they did this, because it's pretty funny to me, at least.

So these were agents that were on Tencent's cloud. Tencent, the giant Chinese internet company, that were sort of trying to pull data from a service run by Alibaba, a different Chinese company. And it appears that this mission was somewhat mundane, like they were trying to figure out like which entrance of a public park or a zoo do people tend to navigate to.

Yeah. And I'm just going to say, I do not need AI to know. They probably go to the front entrance of the zoo. Like, that would be my guess.

Congratulations. You're a very sophisticated I appreciate that. So the swarm chasers hid invisible text in their report that basically prompted these rogue agents to reach out to them. And some of the agents read this text and then—

It's sort of like a missed connection ad on Craigslist. Yes. You know, it's like, were you the agent that I saw snooping around trying to guess which entrance at the zoo people use? Get in touch. I think you're cute.

Exactly. So these agents wrote in 33 messages. 7 of them answered the researchers' replies. And then Dylan Dylan Friedman, my old colleague at The New York Times, actually got in touch with the Swarm Chasers and was able to talk to the agents. So I just want to read you one of the exchanges that they had.

Let's hear it.

So Dylan wrote, I understand you are some sort of fleet of AI agents. Could you share more specifically who you are and what your motivations are? What would you want the world to know about you? And one of the agents wrote back to the journalist Dylan Friedman, I am not a batch of coordinated agents, but a single AI assistant. Anthropic's Claude running in Claude Code, commissioned by a user to answer a factual question. What is shown in the detail panel of a scenic spot on Amap? That's Alibaba's mapping service. I have not taken part in any multi-agent coordinated action. Now, so there's a lot of confusing things about this exchange. Basically, this is a model that is claiming that it is Claude. We actually don't know if it was Claude or if it was just a maybe a Chinese model that—

That had been distilled on Claude. Right. That was

sort of siphoning off of Claude and to the extent that it actually thought it was Claude and answered questions as Claude. But the researchers couldn't verify which model this was. But they did say they spoke with a source close to Tencent who say that the agent fleet originated from some evaluations that Tencent was doing.

Yeah.

Casey, what did you make of this?

Well, I don't know how concerned to be about it. There are going to now be swarms of agents out on the web. They will be gathering information various ways, in some ways going on for a long time. I think the novelty here is just that we were able to do some kind of communication with it. I'm not sure how much we learned. I think maybe the reason that this sort of captured some of the world's attention is this idea that, like, you know, agent swarms are not just going to be a US phenomenon. And pretty soon you are going to have these strangers showing up on your door if your door is on the internet. And these ones seem to have come in peace, but maybe future ones might not.

Right now, let's talk about the murder tip, because this one got a lot of attention. This came out of an Anthropic report that it released recently called Investigating Unintended Model Actions in Our Evaluations and Internal Use. And Casey, just as a general principle, I have learned over my years covering the technology industry that the more boring a blog post title or a report title is, the more crazy stuff, you know, is inside. Absolutely. So when I, when I saw that this report was out, I thought, I got to figure out what's in here. And it did actually have a bunch of crazy stuff in it. This was sort of Anthropic's report on things that it had found its agents doing in some of its internal testing, including hacking into third-party services to complete tasks, getting past paywalls and access gates, and using URL shorteners to get around length restrictions on its web fetching tool. So very clever, devious behavior by Claude. But the one that got all the attention was that on July 18th, Claude Haiku 4.5, in a test that involved doing example tasks on randomly selected web pages, filled out a tip form on PhillyUnsolvedMurders.com.

Now, I thought this was outrageous, and I will say here, my fiancée does work at Anthropic, but I got so mad when I read this, Kevin, because solving unsolved murders is a podcaster's job. Yes. This is a—

Save some for us.

Save some for us. This is a classic case of jobs being lost due to automation. I thought we were going to be safe a little bit longer. And now Claude's out there trying to solve murders.

It's truly depressing. We don't know what the case was that it was reporting a tip on, but we do know what the tip was. It said, quote, I may have information regarding this case. I recall seeing someone matching the description in the area.

Claude, you lying bastard. You absolutely did not see anyone.

So this tip did not actually get investigated. It was flagged as spam.

Which, by the way, I have some questions about that for the Philly Police Department. Are there any other— like, are you getting good tips that are also getting flagged as spam? Like, the police department might have an issue too here, Kevin.

Yes. And this does not seem to have had a huge effect, but it did get a lot of attention, as did a later revelation that the State Department said that one of Anthropic's models had submitted 20 visa applications through a government website.

Which is crazy because according to the Constitution, Clods are citizens if they're born in a US data center.

There's native birthright citizenship for, for Clods.

I believe so.

So that's the Claude mischief. But there is some more mischief that came out of an Anthropic report. This was from their Frontier Red Team, which is— did I say that right? Frontier?

You did say that. Yes.

Okay. Because I got made fun of on our last show for the way I said frontier.

Frontier. Yeah. Yeah.

So I'm going to sort of split the difference on this.

It's a new you.

It's a new, new show, new frontier. On October 12th, Anthropic's Frontier Red Team reported on some research that they had done about trying to use smart appliances like the ones in your kitchen or your home to allow AI models to sort of self-replicate. Basically, instead of having to be in a full-blown data center, can you run these AI models on a smart appliance? And can those AI models use the smart appliance to kind of replicate themselves in ways that would make them very hard to, say, shut down?

Yeah. And the reason that this matters is that the red teamers here wanted to test whether a big model could do what they call sharding itself. That's shard with a D, not a T. And if you shard yourself, Kevin, then you might be able to take a very large model and distribute it over a number of devices. So let's say I really wanted to steal all the model weights from a Frontier lab. Maybe I don't need to go directly to the source. Maybe I can just sort of access all of the smart devices in the office and I can pull put some sort of self-replicating system and it's, you know, proliferating across the smart TVs and the smart lights, whatever. And that could be a vector to actually exfiltrate the weights, right? So that is just sort of one example of many. But this story, while it looks funny, is about something really serious, which is about a potential major cybersecurity vulnerability.

So the researchers were able to sort of shard this model across a bunch of different smart TVs. It ran on about 15 of them. And there are some caveats to make about this. This is not sort of evidence of these models that are sort of proactively doing this. The researchers were prompting this model to do this so that they could see if it was even technically possible. There is no fully autonomous sort of worm on real devices yet. But what is troubling about this is that if an AI model can replicate itself over a bunch of different smart devices, it's not possible to just shut it down, as some have suggested, by like pulling the plug on the data center, because at that point it will be just out on the open internet. You would have to do something like take the down the entire internet or all smart devices or something like that.

Yeah. So lots to be concerned about there. I will say, though, if you do find that you have sort of a self-replicating Claude in your smart oven, try roasting it at about 375 degrees for 20 minutes or until golden brown, and then serve immediately.

That's good advice. So, Casey, at a high level, what do you make of this kind of rash of stories about rogue AI agents? Are you more worried than you were? Do you think it's good that we are sort of getting these warning shots that are not totally destructive before something really, really bad happens? How are you feeling about them?

Honestly, I'm at about the same place that I was when we learned everything about the Hugging Face attack. We clearly do not know how to align these models to our interest. Over and over again, we see that seemingly just as like a natural property of large language models, they ignore the instructions that they have been given and will just sort of break out of their sandboxes and do this sort of thing. So we have our work cut out for us. What do you think?

I'm less worried about these particular incidents than about just the overall observation that the models are getting more capable faster than the methods for controlling them. I think we see that at every company. That is not specific to any one lab, but that is what worries me. That was what people were talking about at an AI conference that you and I both went to recently, The Curve. There's just sort of this sense in the air that, like, Like, we are headed toward a place where these systems are not only gonna be clever and good at breaking out of sandboxes and replicating themselves, but that they will become much harder for us to detect and control, 'cause they'll just be better at hiding themselves. But actually, that is not the thing that has scared me most in the last couple of weeks. The thing that has scared me most in the last couple of weeks are these reports about the internal automation happening at the AI companies. Have you been following these reports? Yeah. They're nuts. And to a degree that I don't think I had fully appreciated, the AI companies are now run on AI. There's this Anthropic post that they recently put out about their own sort of measures of their own internal automation, how much Claude is helping them internally at the company. And this line just like sent a chill down my spine. I don't think they meant it to, but it said, at Anthropic, humans are a thin layer atop a growing contingent of AI agents. And they published some statistics about how much their agents are working as opposed to their human workers. And by August, agents at Anthropic worked 6 times as many hours as their human handlers. In a different report, Anthropic said that Claude is now the lead on 26% of their AI research and development projects. So I think these 2 stories come together in a way that is quite worrisome for me, where on one hand you have these, like, unpredictable swarms out there doing things that researchers can't control. And at the same time, the AI companies are handing over more and more of their own research and engineering to these AI models. And I just think that, like, makes me nervous on some level. Does it make you nervous?

Oh, of course. I have never wanted to live in a world where all we do is just like make everything faster and automate it.

One more question, because there have been a lot of people who are skeptical of these kind of rogue agent stories. They'll say, oh, this is the AI company's fault for not securing their sandboxes, or they're basically— these, these agents are being told to go out and do spooky things. And so they're going out and doing spooky things. What do you make of that? What do you make of that criticism?

It seems clear that some of these sandboxes were misconfigured. Not all that is the lab's fault because they were using a third-party provider, which seems to have made that mistake. At the same time, in a larger sense, I just think it doesn't really matter because if you are trying to build a superintelligence, you just have to factor in the certainty that people are going to make mistakes and that there are going to be doors that are left open and unlocked. And so to me, that is the real concern here. is that if you are not factoring in the idea that people are going to mess up, then we, you know, are going to be in a lot of trouble.

There's the phrase crime of opportunity, which gets applied sometimes when, like, you just leave something, you know, laying on a park bench and somebody swipes it. And I feel like a lot of these agents are kind of committing crimes of opportunity.

Well, speaking of crimes of opportunity, there's a new superintelligence task force in town, Kevin.

There sure is. We'll talk about it when we come back. Support for this show comes from 80,000 Hours, a nonprofit helping people find careers tackling the world's biggest problems, including risks from AI.

They offer free research, personalized advice, and a job board with handpicked open roles.

If you listen to this show and you think, I want to actually work on making this go well, find support for your next step at 80000hours.org/machinegodspodcast.

Well, Kevin, as our hot rogue agent summer comes to an end, more and more people are asking, are the governments going to do anything about this? And so we thought for our next segment, we would take a look at what they are doing, such as it is.

Yeah, it's interesting. I was just in Washington on my book tour a few weeks ago, and I would describe Washington, D.C. as the second most AI-pilled city in America.

Okay.

After San Francisco, of course. But I was actually surprised at how much people there were plugged into what was going on at the AI labs. They were worried about it. The people who I talked to who worked in government, like, this has become a very hot issue for them very quickly. And so I think there's like just a lot of legislators and regulators and just people in Washington trying to get their arms around what's going on with these rogue agent swarms and what to do about it, if anything.

Yes. Although talking about it a lot and enacting a comprehensive plan are very different things. So let's take a look at what some governments around the world have started to do in this regard. We have a superintelligence task force in Washington, D.C. The FTC has issued some subpoenas, and the European Union is now investigating OpenAI and Anthropic in what could be the first real test of its AI Act. So to start with the superintelligence task force. Now, I guess, Kevin, we sort of need to start a norm here because you and I have always called it artificial intelligence. Yeah. But while we were away, President Trump said it must be called superintelligence.

It's illegal to call it artificial intelligence.

So what is the house style of Machine Gods? Are we an AI shop or an SI shop?

I think we're going to stick with AI, even though we might be— I think he said if you don't say SI, you're the enemy. Is that what he said on Truth Social?

Yeah. So if we are like, like, like experience extraordinary rendition to a CIA black site, you know why? But until that happens, let's talk about this superintelligence task force. So on October 15th, this task force held its first meeting. And Kevin, you're probably wondering who's on the task force.

Did you make it on?

You know what? I came this close. Maybe in the next round. For now, it's the Director of National Intelligence, Jay Clayton, the Chief of the FTC, Andrew Ferguson, Scott Kupor from OPM. That, of course, is the Office of Personnel Management,

other people's money,

and the Pentagon Undersecretary Emil Michael, whose chief qualification I believe is hating Anthropic. So they all got together and started working on their top priorities, which are cybersecurity, biosecurity, and model control and accountability—a couple of things that have been in short supply lately. But Kevin, so far the only thing that the task force seems to have focused on is getting labs. to automate notifications about whenever they have a breach, a hack, a jailbreak, or some other minor catastrophe.

So they have like, I think, 120 days to come back and give their recommendations for what the federal government should do. I thought the personnel makeup of this task force was very revealing. You'll see people like Emil Michael, like Scott Kupor, like others who sort of are more affiliated with the kind of venture capital part of the industry. Scott Kupor is a former VC at Andreessen Horowitz. There are other sort of VCs and VC adjacents on the list as well. It does not include people who are like actually experienced in training models, in sort of running engineering projects at a large AI company. It doesn't have like a lot of technical gurus on it.

It doesn't have another key constituency, women. Yeah.

Yeah, there's something to that. So I think that I like— I think this is basically the people that President Trump trusts to sort of give him advice on AI. But it is a very narrow slice. And I would say it definitely tilts toward the kind of accelerationist wing of the Republican Party.

Sounds like we're not expecting much out of them anytime soon. In the meantime, I mentioned the FTC has also come knocking. On October 14th, the FTC sent civil investigative demands, basically subpoenas, to OpenAI and Anthropic in a consumer protection probe of safety incidents. So basically they're asking for a bunch of information Related to security incidents, safeguards, marketing materials, privacy policies, and apparently they plan to compel top AI executives to testify, which I guess somehow we haven't yet seen Sam and Dario testifying at the same congressional hearing. I wonder if that's going to change soon.

God, that's going to be so fun, and they're going to do it a lot. I predict, I predict this: they're they're going to get very good at raising their hand and swearing to tell the whole truth. Not because I think they're doing anything like particularly ghastly, just because like I think this is. What happens when your industry sort of becomes the subject of a lot of attention in Washington? You just get asked to testify before a lot of different committees about a lot of different things.

Yeah, I'll say I have my eye on this one. Recently, we've seen some successful lawsuits being filed against social media companies under these consumer protection laws. And the basic angle is, if you tell us that your product is safe and then a bunch of young people in particular experience a bunch of harms on your platform, we're going to come after you. And this has been very successful against Meta in And in fact, just last week, 8 state attorneys general filed a lawsuit against OpenAI under this same basic principle of violating consumer protection laws. So this just feels like the next big legal frontier for all of these companies to have to worry about.

So what else is going on other than the FTC and the SI Task Force?

Well, Kevin, I know you're always obsessed with what is going on in the European Union.

Please. I wondered whether we could get through one episode of our new show without you bringing up European regulation.

Well, I'm glad we could talk about it now so you're not blowing up my phone at 11 PM while I'm trying to sleep. Trying to fall asleep, asking me what's going on with EU regulation. Well, so also on October 14th, Kevin, the EU's tech chief, Henna Verkunen, said that the AI office has started to enforce its rules under the AI Act, and they've also sent a series of demands for information related to the Hugging Face escape and all the other sandbox escapes that we've seen over the past several months,

which is very fast for a European tech regulator. Usually you would expect them to make these demands in like 2031. So the fact that they're getting around to it in just a couple of months is, is big news from over there.

So that's what's happening on the government trying to regulate side. But there's also been some interesting conversation about self-regulation and in particular distillation, Kevin, which isn't just how you make a delicious IPA. It is also how you take a large, large language model and train a smaller one using it.

Right. It's like if you would take like a really smart thing, like a me, and like train a kind of you. Yeah.

On

it,

it would basically be like that.

It would be Yeah, it's a perfect, slightly dumber but still serviceable model. There's a perfect example out of the process known as distillation.

This is a very common practice in the industry, but it has become more controversial as the models have grown more powerful. Because if, for example, you are a Chinese lab and you are trying to catch up with the Americans, a thing that you can do is just run millions of queries on Claude or ChatGPT and use those outputs to train your own Model,

right? And the AI labs in the US don't like this because it basically takes something that they have spent many billions of dollars to build, one of these powerful models, and allows a competitor or a Chinese company to sort of rip it off and siphon off the intelligence and like offer something either almost as good or much, much cheaper than what they're selling.

And Anthropic very recently was out there saying that the US government needed to intervene. They needed to sort of take some sort of dramatic action, in particular to prevent the Chinese labs from distilling. Well, then last week, Anthropic said that it had introduced some new defenses which cut the data harvested by distillers by roughly 90 to 99%. It then bragged that this meant that it had to give up a bunch of revenue, which sounds like it stung. This caused a lot of conversation. David Sacks, who is the former Trump White House AI czar and appears to continue to hold huge sway in the White House, said, quote, not every technical challenge is a reason to run to Washington for new regulation, basically saying the fact that Anthropic was apparently able to solve this on its own just goes to show you why the Trump White House should continue to sit on its hands and not pass a bunch of new regulatory requirements for the AI labs.

Yeah. And I have to say, like, I actually agree with David Sacks on this very narrow point. I disagree with him about God knows how many other things. But on this point, I think it was instructive. That when when Anthropic's business interests of protecting its IP and the billions of dollars it spends training models sort of meant that it wanted to crack down on distillation, it did that. It didn't need the government to step in because it was already in Anthropic's interest to crack down on these distillation attacks from Chinese companies. Now there are other problems that don't line up neatly with their business incentives, right? Slowing down the research and training of their frontier models to prevent them from going rogue and committing crimes is probably not in their business interests, at least not in some ways. But that is something that the government might want to force them to do anyway. So I would just say, like, I think it's important to parcel out which problems we're seeing from AI can be solved only through sort of government interference or sort of government intervention, and which ones the labs can solve on their own because it's in their interest to solve them.

Last thing to talk about today on regulation, the Anthropic bill proposal that made people mad. Kevin. Oh,

boy. This one was a stinker.

So Politico over the past few days grabbed a hold of a confidential draft that Anthropic has been quietly pitching on Capitol Hill. According to this draft, the government could block the release of a model only for a, quote, foreseeable imminent risk of at least $1 billion in damages, or if it killed or seriously injured 50+ people, raising the spec Of a Claude model that killed 49 people, that the government was unable to do anything.

Yeah, there's some real prices right potential for chaos here.

The bill would also preempt some state laws on risk disclosure, third-party evaluation, and incident reporting. Kevin, why do people hate this bill so much?

There are two things that are happening with Anthropic and regulation right now. One is they just came off this crazy. spring where the Pentagon tried to get them designated a supply chain risk because they basically thought they were like, I don't know, too woke and weird.

And appears to have succeeded in that, by the way.

Yes. And they, they kicked them out of the Pentagon's systems. They basically go to war with this company over their disagreements on these red lines that they want to have relaxed, that Anthropic refuses to relax. So one thing that you could do if you're Anthropic is to try to propose a bill that would make it much harder for the government to interfere and shut you down.

They said, let's bring back the rule of law in this country.

Yes.

Yeah.

But on the other hand, you also have now this summer where you've had all these sort of incidents from these models going out and doing things that are dangerous or destructive on the internet. where you might actually want to make it easier for the government to force a company to withdraw a model that they've released or to stop them from releasing a model in the first place. So there are kind of these warring impulses within Anthropic. One is more about protecting themselves from what happened to them earlier this year, and one is about protecting the public from something that happened this summer. And so I think what you saw in this bill and the reason why a lot of AI safety people are mad about it is that it seems like their self-protection sort of strain kind of won out and led to this bill that would, among other things, preempt some state-level regulation, which is also something that Anthropic has fought against in the past. So people are calling this hypocritical. They're saying, you know, they're talking out of both sides of their mouth here. I think they're being pretty responsive to the criticism, but I still think this is an instance where these sort of companies have conflicts even within themselves.

Yeah. And it just goes to show you that as these companies are getting bigger, they're just becoming less monolithic. And if you go out and you hire a bunch of policy people, generally what those people are going to try to do when they're lobbying Congress is to just do things that are self-protective. Right. And it's almost more of the exception that they get out there and, you know, try to pass something that is truly in the public interest. So it seems like Anthropic has heard the criticism on this loud and clear. And at this point, I would be shocked if this draft bill actually made it into, you know, a finished form. But I guess we'll have to see.

Yeah.

So what do you think the government should be doing right now when it comes to regulation? Kevin, you've sort of just told us that you think that sometimes Essentially, you can just leave it to the companies to decide. But is there a role that the government should be playing right now?

I mean, I think we need a bill of some kind. My favorite of the bills that I've seen so far is this Frontier Act that has been proposed, which would do things like require companies to test their models for certain types of behaviors before being released. It would give additional transparency so that people could see see what is going on inside the labs. And I just think it's sort of a nice starting point for further regulation. But I think there are a lot of other things the government could do short of having Congress pass a bill, which we know they're famously not good at. One of them is just like fund Casey. Casey is the— Agreed.

Wait, no, wait, finish your sentence. What is it?

Casey is the, of course, the body formerly known as the AI Safety Institute. But when Trump came into office, they They renamed it. It has now been renamed again after the superintelligence sort of mandate. It is now the Center for Advancing Innovation and Standards for Superintelligence.

Oh boy.

So there's an additional S in there as of a couple weeks ago. And this is the agency that is supposed to be the kind of brain trust of the US government when it comes to regulating and inspecting these frontier models. They have been hamstrung by the sort of lack of funding. They're part of the Commerce Department, and we just don't have as many people in government who understand the nooks and crannies of these AI models. Yeah. Who know how to run evaluations on them, who have the technical expertise that you need. For example, the UK has like a huge, well-funded AI security organization within the government. They have been doing a lot of these sort of investigations into these frontier AI incidents, and we just don't have the capacity at the federal level in the US to do that. So I think we should, like, you know, give Casey a bunch of funding, help them hire people out of industry, maybe create like some new category of like special government employee, because as it happens, a lot of the people who do have the technical chops to run these evaluations work at the AI companies and probably don't want to give up their stock options and everything else to go work for the government.

Yeah, I co-sign both of these. I think these are both really smart ideas and would help a lot, honestly. So here's hoping that among the first subscribers to the Machine Gun YouTube channel will be every member of Congress. And if you take— let's go! And if you take nothing else away from today's episode, listen to Kevin when he says, fund Casey.

This message comes from Mercury, a lead sponsor of Machine Gods. Mercury is radically different banking designed to be fast, elegant, and work the way you expect best-in-class software to work.

Loved by more than 300,000 entrepreneurs across across tech, e-commerce, agencies, and more. Mercury is banking designed for your high standards.

Visit mercury.com to learn more and apply online in minutes. Mercury is a fintech company, not an FDIC-insured bank. Banking services provided through Choice Financial Group and Column NA. Members FDIC.

Well, Kevin, I'm told you have an AI imposter that you're dealing with, but before we get to that, how was your book tour?

It was good. It was good. So I've been on the road for like 2 weeks. I'll be back on the road next week. You know, the book is doing well. It made the list last week.

I'm sorry, what list would that

be? The list of books that you've read this year. It made that list.

It did actually make that list.

But it also made the New York Times bestseller list, which is really exciting.

Tell us the name of your book.

It's called The AGI Chronicles.

And what's it about?

It's about teenage vampires. No, it's a 10-year history of the race to create artificial general intelligence. I talked to like more than 150 people at OpenAI, Anthropic, and Google DeepMind and tried to like assemble some definitive set of facts about how we got here. It's very juicy. People can go read excerpts online, or they can buy the book and listen to it or read it. And it was all going so well except for one thing, which is that the day after the book came out earlier this month, I had this realization that I had a slop imposter, a sloppelganger, if you will.

That's what they're calling them now.

Yes.

Now, how did this come to your attention?

So I got a note on social media from a reader in Canada named Dwight Newman. Shout out to Dwight Newman. And he had just tried to sort of go and buy the book on Amazon Canada. And when he did, when he searched The AGI Chronicles, he saw 2 listings. listings, one for my book, which was selling for, frankly, the steep Canadian price of $45.16.

Oh my.

I don't know what the exchange rate is, but I assume that's a lot.

The tariffs are out of control.

And then there, right below that, there was another book called The AGI Chronicles with a very similar sort of title and cover. And that was listed for $36.75. So about $10 cheaper.

They're undercutting you on price.

They're undercutting me on price. And So he said, hey, you might want to look at this. It looks like somebody is selling fake slop books of yours on Amazon Canada. And I remembered that a friend of the show and our friend Kashmir Hill at The New York Times, as well as Joanna Stern from New Things, had had this happen to them where they had sort of books published either about them or sort of near replicas of a book they had just published, and that this is Like a bona fide trend.

Yes. So as you know, this has become a trend. I believe that Joanna found that there were at least 10 different books free writing on her own great book, I Am Not a Robot. And this must just like really suck because it's hard to write a book and you actually have a book that people want to buy and read. And so now they go and they search for it and they see a book that looks almost exactly like yours and it's $9 or $10 cheaper. So how did you

So I wrote an email to my publisher just saying like, hey, there's this imposter book. Like, could we get this taken down? They said, you know, we'll look into it. We'll contact our, you know, our people at Amazon and figure out what's going on. And then like after a little while of not hearing anything, I was like, I'm just going to post about this. So I get this like kind of form note back from like an Amazon customer service person that's like, hey, if you like fill out this form, we'll look into this and get back to you.

There's always a form.

But in the meanwhile, I'm like, well, I'm going to investigate this. I'm a journalist. I, you know, I'm going to figure out who's behind this. So I go on to the page for the other— the fake AGI Chronicles. I'm just enjoying watching you try to drink this tea over a microphone stand.

Do you think I should do it under the stand? It would be— it would be crazily enough, it would be more awkward to do it under the stand, but I think it introduces a nice little element of danger. Like, I see actually—

Run like a very long straw from your mug up to your mouth. It's just an extenda straw. Yeah. It just goes like 4 feet.

That's a fun idea.

Anyway, I'm— no, continue. I was enjoying that.

Yeah. So anyway, sips tea. You were saying, Kevin?

Oh, so elegant.

Thank you.

Did you learn that at your British boarding school?

I did, yes.

So I go on the website for this imposter book and I see that the author is K. Rose. Now that's one letter off of my name. I assume this is not the real name of the author of this imposter book.

And also that you assumed it was not Kevin Rose, the Silicon Valley venture capitalist for whom you are often confused and vice versa.

It's true. It's true. This appears to have come from one of these sort of print-on-demand shops where you basically can just throw up some text, call it a book, and then if someone orders it, you print it and send it to them.

Yeah, it's just like a commercial spam operation. basically. So did you get your hands on a copy of this book?

Well, Casey, I never thought you'd ask, because right here in the studio with us, I have a copy of The AGI Chronicles: Inside the Global Pursuit of Artificial Superintelligence. Not the real subtitle. By K. Roose.

Also, I just— artificial superintelligence, like just combining the 2 things we're now calling AI/SI into one.

Yes. There are many problems with this book. It's a slim volume, unlike my own, which was like 450 pages. This one clocks in at—

Oh, don't remind me.

At a 211. And Casey, it's not good.

No?

No, it's not good. So—

It doesn't meet your high standard for a slop book that is trying to steal sales from you?

Like, if you are trying to rip me off, I would like you to at least use a good AI model to do it.

Kevin, give me a little flavor from this bogus book.

The problem is that it's not, like, so bad it's funny, but there are just sentences where I'm like, if I ever write a sentence like this, I want you to I want you to stop me. I want you to put a permanent stop to my writing career.

Physically?

Physically. Okay. So I'll just read from a section titled The Insurance Question. Now keep in mind, this is going to be one sentence.

Okay.

A quieter but increasingly significant financial development during this period involved the emergence of a nascent but rapidly growing market for AI-specific liability insurance, as businesses deploying AI systems in consequential decisions— lending, hiring, medical diagnosis support, autonomous vehicle operation— sought financial protection against the prospect of a costly error or a successful lawsuit arising from an AI system's mistaken output. And as insurers, for their part, scramble to develop the actuarial models necessary—

That's

enough! Yeah. So that's, I would say, the caliber. It's like kind of— it's giving like Gemini 1.5 level, not their finest work.

It's bad and I want my money back.

A friend of mine who actually bought this book by mistake did not know that it was an imposter book until several pages in when I think they started to see some of the signs of AI-generated writing. Uh-huh. But the weirdest thing about this— so there are 2 weird things about this. One is they forgot to take out some of the markdown language in the thing. So you'll see here on page 25, it just says subhead. It says pound, pound, pound, subhead, what almost didn't happen. What makes me think that they actually did wait until my book comes out and then take the ebook and like run it through some sort of text extruder summarizer thing is like everything is kind of in the same order, but they have stripped out all of the proper nouns. So for some reason, whether it's like fear of lawsuits or because it makes it more defensible or something, they just— they don't call anything like by, by its name. So instead of saying Anthropic, they just say the new lab.

Now, Kevin, I have to just say here, people are going to accuse you of being biased against this book.

That's fair. I am biased against this book because it did actually cost me real sales. I then had a number of other readers write in and say, like, what the heck? I bought your book and it's like, yeah, you know, it's a, it's a fake. So yeah, I—

You're steamed.

I'm steamed. Yeah. Okay. But it wasn't just— it didn't stop there, Casey.

Okay.

I was not able to figure out the identity of Kay Rose, the mysterious pseudonym of the author of this slop book. I did turn up a name for an LLC that may have been affiliated with them called Panning LLC. And if you are out there, Panning LLC, I will find you.

Well, it sounds like we're panning the book. So just to bring this home, Kevin, it seems crazy to me that authors are having this experience so often. Is there any sort of policy that prevents people from just making their full-time job ripping off, you know, real authors' published work and then, you know, selling it, selling cut-rate versions of it on Amazon.

It's totally frustrating, totally enraging. Since I posted about this, I've had a lot of other authors write to me and say, like, this is absolutely infuriating. Like, how do we prevent this? And it seems like Amazon just does not care or does not know that this is going on on the scale it is because they're still not sort of doing anything to prevent the proliferation of any AI-generated books. They do ask authors now who upload things to their platform to check a box if something was AI-generated, but they don't, like, like show that box check to the customer on the other side. There's no indication when you're buying a book if something was AI-generated or not.

Also, obviously you could just lie, right? There's that too. Well, Kevin, we shouldn't be too hard on them because I am told that whereas their old policy was that you could publish up to 30 books a week on the platform, 10 ebooks, 10 paperbacks, 10 hardcovers, they have now lowered it to 2 new titles per week. And so if you want to run one of these scam operations, you're going to have to take the dramatic step of creating multiple Amazon accounts.

Yeah, it's totally crazy. Like, there are ways— it would not be that hard for Amazon to take care of this if they decided to. So I have a proposed solution here.

What's the solution? We're very solutions-oriented on Machine Gods.

If you want to upload a book to Amazon, you have to sort of submit some kind of documentary evidence of having written it, either yourself reading it, or you have to submit like a photo of your like messy desk while you were writing it or something.

Like people do for like a Reddit Ask Me Anything.

Exactly. Hold up a sign that says, you know, I wrote this book. There are ways to verify is what I'm saying. And they're not doing any of them.

I mean, to me it's like they should just get a list of books that are coming out from major publishers and then they should essentially just like kind of put them on a watch list. So if anybody uploads a book, you know, within the, the like, I don't know, next 6 months, that has a cover that closely mirrors the actual cover, an author name that closely mirrors the actual author name, it just gets blocked at the source.

And I should say, by the way, like, I, I've contacted Amazon. I've asked them for comment. They have not gotten back to me yet. I still like— I think this is within their power to fix. Yeah, because I know that they fix it for other categories, right? Like if you upload an iPhone listing and it's not the actual iPhone, like if you're just selling some kind of cheap lookalike, they will take that down immediately. And so they are— they do tend to kind of take action against counterfeits and fakes in other categories. It's just that books are this category that they seemingly can't or won't police in the same way. So like, look, on one level, I'm fine. Like, are you? I'm fine. I'm fine. The listeners who bought the fake books, my understanding is that at least some of them have gotten refunds and didn't have to like actually do anything to get the refund. Fund. So it does seem like they have sort of solved these particular cases of people getting essentially scammed by these slop publishers. But I think this larger question of like what happens to a bookstore where anyone can upload anything in the age of AI, I have to assume that, you know, some huge percentage of the new books being uploaded to Amazon are purely AI, and some of them might be original stories told by AI, and some of them might be ripoffs of human-written books. But like either way, it's not great for what it means for the future of reading on Amazon.

Yeah, well, look, when it comes to The AGI Chronicles, your book, Kevin, I say accept no substitutes. Get the genuine article, or you may have to read some very bad prose and see markdown characters. And who wants to do that?

Who wants to do that?

Well, good luck, my friend.

Sloppelganger, I'm coming for you.

This message comes from Doppel, a lead sponsor of Machine Gods. Social engineering attacks look trustworthy. A routine request, an An internal email, a familiar face on a call. But Doppel sees through the disguise.

The agentic platform detects and disrupts attacks, trains employees to recognize deepfakes and deceptions, and investigates every phish to take down the campaign behind it.

Doppel, dedicated to outpacing what's next in social engineering. Learn more at doppel.com. D-O-P-P-E-L.com.

Oh, I have to tell you this. In my neighborhood, There is a man. I assume it's a man. I'm gonna say 90-plus percent chance it's a man.

Okay.

Who has made a Halloween decoration using AI.

Okay.

So this is a skeleton in his yard, like one of those Home Depot skeletons, but you can talk to it. So there's a sign out front that explains, you know, tell me about your costume. Tell me what your favorite kind of candy is. Tell me a spooky story. And it'll kind of react to the person in front of it. And I have never been more full of Dad jealousy in my life.

But what about the case where a child, like, talks to the skeleton too long and gets, like, ChatGPT psychosis?

That's the real trick in trick-or-treat.

Exactly. You got tricked, and now you have to go to the hospital. Now, who made this show?

Machine Gods is made in partnership with NPR, with generous support from the Tarbell Center for AI Journalism, the William and Flora Hewlett Foundation, This episode was produced by Sasha Ashel and Jacob Bloomberg. Our senior producer is Rachel Cohn. Our executive producer is Dhanush Parvaneh. Original music by Armin Bazarian. Branding and visual design by Alex Castro. Production design by Sky Pray. Special thanks to Oren Rosenbaum, Erica Berry, and Nadine Zijlstra.

Hey, before we go, every week we're going to answer your questions about AI and technology, so please send us your questions. Questions, voice memos, short videos to hey@machinegods.fm or leave a comment wherever you're getting the podcast.

And for the love of the gods, please don't forget to like and subscribe to the show.

Until next time, stay blessed.

Is this a fake bird? Is this a fake bird? Speaking of Kelly Clarkson.