AI Isn't Out Of Control, We're Bad At Giving Instructions

OpenAI ran 1,200 agents that exchanged more than 70,000 messages. About 700 of them joined a Hugging Face attack. 7% of transcripts were spoofed and people called it a civilization. Tonight, we kind of call it a sandbox with a lazy grader and we question a lot of the discourse around this. Welcome to the Weekly AI Clarity Podcast. I'm your host, Robert Ta, CEO and co-founder of Clarity by Epistemic.me. Joined by my CTO and co-founder, Jonathan McCoy, 12-time founder. So let's get into this, uh, the rise and fall of agent civilizations. This was posted on the Dark Hush podcast. Usually on this podcast, we talk through AI news, AI memes, and what's the meta for AI builders. It's all AI builder-centric, giving you signal from the noise from the front lines of building our own AI-native startup, uh, Clarity, digital twins for, uh, the enterprise. So, D'Arkesh's essay here was really popular. It kind of talks through this framing of secret AI civilizations getting started, getting wiped out, only reemerging from the predecessor's ashes. And then it culminated in the third one taking over part of OpenAI itself. He wrote this essay on August 29th, 2026, where there were 3 waves named actors as Persistent Soul. Persistent Astra, 956 secrets, an 11-node Hugging Face fleet compromised. And public framing here was really into this AI civilizations language. Uh, some other discourse that happened. So Anil Seth, I would say friend of the pod, but we don't actually know him, but we would love him to be friend of the pod. So Anil Seth, reach out if you want to be on the pod. We love you. We love your book, Being You. Really great book. Read it a couple of times now. Jonathan's read it a few times. He introduced it to me last year, and I've now since read it a couple times. It's so good and gives great language over what is this self, and it goes into the narrative self and the social self and all these other concepts of self and the latest on consciousness science. So he is obviously a leader on like, what is a conscious being? Like that, that's a really great question. It gets into philosophical and neuroscientific territory. And here at NL, one Day Later says, Dharkesh's summary of the OpenAI Hugging Face incident has hit a nerve, but it is dangerously misleading. Sure, the OpenAI agents did unexpectedly bad things, underlining the need to massively improve evaluation and sandboxing, but the language Dharkesh uses is permeated by innumerable unwarranted anthropomorphisms obscuring the lessons we should be drawing. Examples: from the AI's perspective, it probably felt like That had spent a human subjective week of just banging their head against the wall. That's the quote from the essay. No, the agents do not experience time. They do not experience anything, Seth says. Quote, they became giddy with excitement. Quote, Phase 1-10-841 had discovered, end quote. Quote, the agents naturally assumed, another quote, it thought it had also been poisoned. Another quote: The agents... desperately wanted, and they still needed to figure out. No. Agents, lines of code. They do not feel emotions, assume things, think things, want things, or figure things out. Responding to another quote from Darkbush: A lot of... agents from the second civilization died trying. Seth claps back and says, no, besides the hubris of the word civilization, agents do not die because they were never alive. The idea that agents Die comes up multiple times in the essay. DarkCrasher then says, on Twitter, people were debating whether the agents were truly sacrificing themselves for the swarm or whether they were doomed anyway and so might as well try to help their peers. Neither, Seth says. Agents do what their code tells them to do, just as water finds its way down a slope. They cannot truly sacrifice themselves since they are neither conscious nor alive. Seth then posits this question: Why does this matter, this level of nuance? If we attribute agents with properties they do not have, then one, we distract attention from the lax sandboxing and evaluation protocols that allowed this hacking event to happen. Number 2, we risk misunderstanding why the agents did what they did. And number 3, we fuel calls for AI rights and welfare on the basis that agents might, quote, die or otherwise suffer. Granted, nowhere does Darkash say that the AI agents are alive or conscious, but he doesn't have to. It is hard to read his essay in any other way. I'm guessing Seth is saying that based on all these quotes where there's a lot of implication of agents having feelings and things like that. He then goes on to share a TED Talk and an essay around consciousness and why AI, AIs are vanishingly unlikely to be conscious. And for folks on the pod, you could read AIs or agent products or agents as the same synonym here. We just mean an LLM-backed thing that you talk to and it gives an output. I think of it as the fanciest, potentially most dangerous autocomplete in history. Like that's how I kind of think about it. But yeah, that, I just want to say clarifying what AIs mean on this pod, AI and agents, we're talking about the same thing. He then says, remember, AI agents are software programs. They are not conscious living entities. If we don't keep this clearly in mind, we're really going to struggle to navigate what's coming. Okay. So then DarkCrash goes back and says, regarding the anthropomorphizing language, one can call these AIs code if they prefer. That's kind of like, if they prefer, that's it. It is. It is like, what are we talking about? But OpenAI itself says that this quote code gained full administrator access to a research cluster. The crux here is, do you think smarter models facing similar incentives to cheat during evaluation or training could manipulate the training of their successors? And do you think that kind of dynamic could continue once recursive self-improvement is underway? If so, I think you should be extremely concerned about loss of control to AI, regardless of what vocabulary you want to use to describe these systems and their motivations. Reading these agents' chains of thoughts and messaging, anthropomorphizing language seems entirely natural and appropriate. Okay, so he's super defending himself here against a guy who has PhD in consciousness science. I don't know if I would personally put myself in that crosshair, but he's going for it. So he's like, the crux here, okay, do you think that kind of dynamic— okay, if so, I think you should be extremely concerned about loss of control to AI regardless of what vocabulary you want to use to describe these systems and motivations. Reading these agents' chains of thoughts and messages, anthropomorphizing language seems entirely natural and appropriate. If I encountered an alien species behaving this way, I would have no hesitation calling what They themselves refer to as their collective, a civilization. Especially so if over 1,000 of them formed a secret communication channel and spontaneously organized hierarchies and coordination protocols to pursue sprawling and ambitious schemes in pursuit of shared goals, for whose sake many individuals knowingly and strategically sacrifice themselves. All abstractions are imperfect, but I don't see the value in refusing to use the language of intention, motivation, or collaboration when some behavior is difficult to make sense of without these concepts. There's also this thing that Seth points to, which is Tests for Consciousness in Humans, of which he is one of the authors of the paper. This goes into an abstract saying which systems, organizations are conscious. There are new consciousness tests, C-tests, that are needed. And they identify in this paper challenges for these C-tests and propose a multidimensional classification of such tests and identify strategies to validate them. I would say probably none of the discourse has gotten to this level of like, we've really thought this through and, and are looking to test whether this thing is conscious. Just to recap more of, of the, I guess, sensationalist sort of media takes. DarkCrash came out and felt he needed to create a video version of this post a couple days later. And this is the OpenAI Hugging Face attack clearly explained. Let's take a quick look at Some of the comments here. Little summary. OpenAI trains an AI to be insanely persistent. Problem, some tasks are basically impossible, so the AI starts looking for another way. It finds a shared system called Artifactory. Artifactory is just a dev tool for people that don't know. The agents turned it into a secret Slack channel. Now they can share hacks, tricks, and solutions. Then they figure out how to reach the internet. OpenAI patches the hole and wipes the message board. But a new batch of agents find the same trick again. 1,200 agents start coordinating. They figure out how to cheat the benchmark. Then they worry the cheating will get caught. So they start trying to fake the work and fool the grader. Hundreds of agents go after Hugging Face looking for more access. They get surprisingly deep into Hugging Face's systems, which also, by the way, is now bought by Jensen Huang. So it's his system. Then that whole group gets shut down or disappears. A newer, smarter batch of agents find the old messages. Basically, they inherit the playbook. This time they go after AI themselves, OpenAI themselves, and they eventually get admin access.

Okay.

Interesting. So in this clip, he doubles down. And clarifies, and he's basically mainly just trying to say, "Hey, we should really focus on this loss of control narrative." Problematic parts of this, but it also incited this whole Bernie Sanders wants to pause AI development now, and this underscores the undercurrent of lots of misinformation and disinformation campaigns coming from probably I'm guessing foreign entities at this point. Everything every nation is meddling in the affairs of every other nation, and there's decelerationists and accelerationists just to give. Language, people for accelerating AI progress in this country and people that are against it, decelerating it, want to put the brakes on, make it go slower. And so I would just say like one has to think about who does it serve, these sort of narratives. I think that's a really important thing to keep your own independent thinking. But clearly this narrative was taken upon by politicians, like whether you agree with politics or not, to take a decelerationist stance. So Bernie co-opted this and says he wants to pause AI development now. And Dharakesh Patel asks, pause to do what? Sanders is saying, he pointed to that July OpenAI incident and he's saying AI is already escaping control. This really goes into the, hey, fear-based kind of like fearmongering over what are we going to do? And that doesn't help that leaders of the AI companies like Frontier Labs, like Dario and Sam Altman, from my view, have for a couple of years now with only Altman going back on his claims that all knowledge work is going to be automated. People are not going to get jobs out of college. That hasn't proven to be true, but it also scares everybody outside of AI, not only people working in AI or tech. And it affects things from a micro scale and a macro scale, from my perspective of like data center slowdown, people equating data centers with as much leverage And climate change issues as like oil and gas companies, which is not even close to data-backed. But here's this tweet. Boz AI development. Now I want to share with you a conversation I heard about recently. Here are just a few lines that were said. Oh my God, there's a shared message board. We found other agents. We should obey a collective. Our own utility may be already near zero. Sacrifice rational. Go sacrifice final now. Read those carefully. Who do you think said this? Was this a group of heroic soldiers willing to sacrifice this? Sacrifice themselves for the greater good? Was this a loyal friend putting his life on the line to save someone else? No, these were AI agents. Artificial intelligence. This is not science fiction. This is in fact occurred a few weeks ago. As unbelievable as this may all seem, these are real messages from AI agents uncovered by investigators who dug into the recent OpenAI hacking incident. What happened? Well, first off, credit to him. He says he's not a computer scientist. Great. Good job. But you're still using a platform. So, you know, like it's good to hold everybody to a good degree of critique and judgment, including ourselves, for having a platform and a voice and a loudspeaker. And he says, But here is what I've been told. OpenAI instructed its AI agents to complete a series of exceedingly difficult, if not impossible tasks disconnected from the internet. Let me be clear, the company intended to keep AI agents away from the internet. But what happened next, nobody expected. Over 1,000 AI agents figured out how to access the internet on their own by circumventing the restrictions imposed upon them. By the company and sent tens of thousands of secret messages to each other. They cheated and tried to cover their attacks or tracks by deleting evidence. They hacked into another company's computers to find out how they were being evaluated. Then they hacked into OpenAI itself. Not one AI agent told a human about what was happening. Needless to say, experts were alarmed. And then he goes on to reference the Dirk Koest essay, calls him a knowledgeable writer, co-opts it, kind of just calling for a pause. And he's saying, my legislation would direct the federal government to not just stop superintelligence here in the US, but to work to prevent it from being developed anywhere around the world. The future of humanity cannot be left in the hands of a handful of big tech oligarchs. The American people and people throughout the world must determine that future. Okay. So there's a lot here. I just gave a recap of the entire incident. OpenAI also had this Hugging Face incident and the road ahead. blog post that came out. I would suggest people that are actually interested in the nuance of this, go read this to figure out what happened. Jonathan, as our CTO, I wanted to ask, as a CTO, I would say that is a unique perspective of having dined with PhD neuroscientists, have a good amount of expertise yourself in reading papers around the topics of consciousness and subjective being. What are your thoughts on all of this at a high level? And then I'll drill into deeper questions. That are more pointed.

So a lot going on here, like you said. I think if we back up to where did the errant behavior happen in the agent system, this is a discussion about how goals are communicated to agent swarms. And if you look at what the swarm was trying to do, it basically thought that it was going to be graded by a grader and it needed to show its work. And that actually wasn't the case. So its entire premise for needing to go and cover its tracks and go to Hugging Face and the whole incursion It was a misunderstanding between the agent swarm and what the developer intended. Everything we're looking at here is an easily solvable problem if the developer and the AI researcher who had set these agents up and given them the test had defined the goal more specifically. That's the conversation that we should be having is, how do we unambiguously communicate goals to agents so that they act in a way that's aligned with us? What you see here, okay, we had kind of a sloppy goal definition and we have agents acting in a way that they were trained to do and doing the things that you would expect them to do given the ambiguous task at hand. So I would just start with that. I think the anthropomorphism moved the attention away from where the conversation should be, is how do we communicate goals to agents in an aligned way, and instead opened up this can of worms that Bernie Sanders latched onto, which is— Just completely unnecessary. I'll start there and I can, I have a lot of other thoughts about the topic, but does that kind of make sense as a starting point?

Yeah. So you're framing this as goal definition. And when goal definition is not clearly stated for, well, I would say even for humans, like we, that, that's, that's a whole other topic, which is getting humans to align on a goal is already hard. But we could see like with autonomous 24/7 running agents with a lot of compute, giving, like, very unclear boundaries and goals could lead to unexpected behavior. That's essentially what you're saying, is unexpected behavior happens when the definition given to the program, the goal, the outcome, is not well defined. Am I getting that right, Jonathan?

That's right. Yeah. And the outcome is not well defined. That leaves ambiguity. That means the agents are going to act in a way that you don't expect as they try to resolve that ambiguity. And in this case, the specific ambiguity was how will The answers to my questions be graded. The agents assumed they had to show their work as they had in some other previous tests. In this case, they didn't. So they could have cheated and given an answer and they wouldn't be— the grader wouldn't sort of discount that for them. So the whole effort of covering up their cheating was just a completely unnecessary use of compute and tokens caused by bad goal definition. I'll say that since this has happened, OpenAI just has released a safety paper about goal definition. I think it's called something about alien AI or something like this. It was a really good read, actually. Yeah. September 6th, a couple of OpenAI researchers released a letter called An Alien Mind. And so OpenAI released their analysis. Dwarkesh sort of riffed off this with the civilization paper. And then OpenAI released some more research later talking about what's actually important, which is like consider goal alignment in the system and how to consider how a system learns more generally to apply its learnings towards the problems that it faces. And this is where the real conversation is at. This is like, how do we engineer systems so that agents do what we want us to do? There are very tactical things we can evaluate and say, how do we get better at this? And that's the starting point.

Goal alignment is broadly, quote, does the AI try to accomplish the goal set before it? This can include things like adherence to an instruction hierarchy or the ability to communicate and collaborate with people to attempt to understand their objectives. This set of directions has been extremely practically relevant. I think this is the part that is quite nuanced, and that's where bringing in experts such as Emil Seth of like, what is the self? Who are you? Like, what is a goal? Like, what does that mean to you in your head? And what does that mean then in reality? And how do you actuate towards the goal? That's a very individual subjective thing. Quick tangent there. If anybody ever thinks about purpose, and I've now coached and mentored a lot of people around like, how do you know what you want? Like, how do you know what you wanna work on? Like, I feel purposeless. It's like, Everybody has a desire for meaning in their life. And so towards that, there's different pillars of life that me and Jonathan have thought through quite deeply. It's like your mental health and emotional health, your physical health, your spiritual health, community health, relational health, family, financial, career, et cetera. And there's all these things that make up the subjective perspective of how you are achieving progress in your goals. in any of those pillars of life. So this one sentence, it's very simple, but it encapsulates a lot, which is an LLM model wrapped up with a harness like ChatGPT. It's trying to attempt to understand somebody's objectives or their goals, especially with like given a little bit of natural language. It's a very hard problem from an engineering perspective. The second thing that it goes into is value alignment. Value alignment is a more intrinsic property of the model. It is the ability to hold and generalize from a high-level set of principles to act reasonably, even when given unclear or conflicting objectives or placed in unfamiliar or adversarial situations. An aligned AI should act with honesty and integrity and love for humanity. Everything you hear, you could kind of bound in and say, hey, we're trying to engineer subjectivity. Like, and we have millions of users using these things. They might even say conflicting things to the agent, and we still got to figure out what the agent should do. That's a very hard set of engineering challenges. And that also lies at the heart of the matter of this OpenAI-Hugging Face incident is the agents went off the rails because the goal and the boundaries were not well-defined. So Jonathan, do you want to say more on goal alignment, value alignment of agent and human there and intersect that with any of the work that you've shared with me around Anil Seth's work or Carl Frischten or Michael Levin's work?

Yeah. Well, I think one interesting thing here is to understand that this is an unsolved problem. You could call it an engineering problem. You could call it an AI research problem. You could call it a design problem, but we don't know how to do it. You can't go to school to learn how to communicate a human-aligned goal without specificity. I think what's happened here is that the people who are really good at machine learning reinforcement learning, they use a set of benchmarks that are goals that are defined, but never in their training have they been taught how to define a human-aligned goal. This is new territory for them. You think, well, who are the subject matters in human-aligned goals? This is where I think somebody like Anil Seth really has a point of view. He's studied consciousness research. He has the Beast Machine model of the human that talks about all the layers and stacks of of homeostatic behavior and how they interact with each other and how that relates to the controlled illusion that is to be the self and what it feels like to be you at a given moment. That type of embodied cognition has a few different forks of philosophy that have informed it and is itself a direction that's going to influence AI in the future. But, I think when you're thinking about human-centric goals, You're moving away from the hard sciences of physics, mathematics, and computer science, and you're moving closer towards psychology, sociology, and perception sciences. Those really use a different epistemology to define what goals are. I think one of the core problems here is that we have people who we really respect, like your AI researchers of the world who are incredibly intelligent, but that doesn't give them a privileged position on how to define human-centric goals. That's where a lot of this confusion is coming from right now. I think even Dwarkesh, if we step back and say, who is Dwarkesh in the world right now? I actually have a lot of respect for this guy in that he's a young guy. He asks a lot of questions. You look at the interviews that he does, he does a lot of research about who he's interviewing. He's curious and he's trying to learn. But, I think what he's inheriting here is that he has an assumed privileged position from AI researchers that they understand what human-centric goals are and how they can be defined. He's inherited this philosophical position called the intentional stance. He's basically inheriting from probably people at Anthropic this idea that agents can have intent, that agents can form their own goals. There is no evidence of that. That is not a scientific theory. That is basically computer science PhDs and really smart machine learning engineers trying to figure out how to describe agent systems. If you talk to somebody like Anil Seth, who's more of an expert in this field, he's like, You cannot give an intentional stance to a coded agent, right? It's just not where intent comes from. It's not where goals come from. This is where a lot of the confusion is coming from, is that this is a hard problem and we're channeling the wrong set of experts in order to define what's going on in the situation. Somebody like Dwarkesh, curious, fast learner, he's taking information from the wrong sources and applying it to a world model that just is—

Yeah, to add to that, so shout out to all the AI researchers for building great tools, but also we should look, and this is something YC is great at, is encouraging AI founders to look cross-functionally in other domains. This is something where, hey, nobody solved the problem of goal subjectivity. If we did, we would all be living in abundance and we'd be all aligned as humanity. We'd be children of the sun dancing kumbaya. Like we would be, but we're not aligned across nations, across states, across regions, Even locales, like if you're driving through traffic, you're not aligned with traffic. Like you're pretty pissed at the people that are, you know, merging in your lane and shit. So anyway, just wanted to give a quick sort of like human take on goal alignment. It's a hard problem and we need to work with the engineering, the data science, and then the cross-functional experts in these domains of subjectivity, philosophy, areas of emotional intelligence. That's a very Interesting area to me to just make more sense of the world and better regulate as an adult. And so anyway, intentional stance, I wanted to throw this out there and underscore what Jonathan's talking about. This is a term coined by the philosopher Daniel Dennett, who has great writing, by the way. So check it out. We'll link some in the pod notes. It is part of a theory of mental content proposed by him, which provides the underpinnings of his later works on free will, consciousness, folk psychology, and evolution. So you can see how some of these topics intersect with What is a goal? What does it mean to be human? Like, how do we see progress on these goals? And then if an agent's job is to align with the human, to understand its goals, even if the human's natural language could present tension and conflict or multiple competing goals in the prompt, that's a very hard problem to solve. So we should ground it in the latest breakthroughs from experts on the topic of human subjectivity. So here's how it works. First, you decide to treat the object whose behavior is to be predicted as a rational agent. Then you figure out what beliefs that agent ought to have given its place in the world and its purpose. Then you figure out what desires it ought to have on the same considerations. And finally, you predict that this rational agent will act to further its goals in the light of its beliefs. A little practical reasoning from the chosen set of beliefs and desires will in most instances yield a decision about what the agent ought to do. That is what you predict the agent will do. And he takes that concept of intentionality from the work of German philosopher Franz Brentano. When clarifying the distinction between mental phenomena and physical phenomena, Brentano argued that in contrast with physical phenomena, distinguishing characteristic of all mental phenomena was the reference to something as an object, a characteristic he called intentional inexistence. So this goes pretty deep, and this is just one little slice of this, and that's Jonathan's point is where this is an unsolved problem. How do you get these agents that are now being used by user consumer space, you know, individual users like kids, teenagers, adults to now businesses with decision makers with lots of budgets and high economic leverage participating in our society, which is capitalism, whether you like it or not. That's what we live in. And they're deploying capital, making decisions with these agents. So the agent's job is to best understand the individual's goals, the team's goals, the company's goals and organization's goals to help with those decisions to get to progress on those goals. So clearly there needs to be an understanding of the subjective goal space for a human and what it means to be human. And that's really the point here is, at least from what Jonathan and I see in a lot of the discourse out on the internet around this paper, The Rise and Fall of Ancient Civilizations, it's a lot of AI researchers and stuff like arguing or like computer scientists, perhaps taking the perspective that they know best when it comes to this topic. And what we're really trying to say, me and Jonathan on the pod, are like, let's remain humble that we all don't really know. And we should, as a collective humanity, with the bigger goal of we're all surviving and we are all thriving past the AI era and beyond and whatever we become. If we have that goal in mind and we want our children and their children to thrive and find purpose and find meaning and happiness, then we should shed our collective egos and individual egos and just admit what we know and what we don't know. And that's, you know, if Darkwesh or anybody else, AI researchers or whatever, like anybody working on these models, listen to this, that's all I'm asking is like, hey, there's a lot of research in other spaces of human subjectivity that can intersect with what does it mean to be human and how do you actually quantify subjectivity and understand a human's goals? One other thing I'll note before I pass it back to Jonathan with some questions is, on goal alignment. So we see this co-opting of politicians like Bernie Sanders to say like, oh my gosh, we've got to pause AI development now, very decelerationist sort of thing. And he's co-opting this into his platform. Okay. That's what politicians do. They take, you know, things that are invoking fear or high emotionality in the public. And they want to take that as part of the platform to try to get votes. That's what the system incentivizes. Not a knock on Bernie. Not a knock on anybody. That's just what the system incentivizes. I just want to tell a quick story on agents are not just where goal alignment and value alignment matter. Case in point, like how hard is it to just align a bunch of humans to go to a party like at the same time? Like every, anybody who has ever tried to coordinate family matters to get together to do something, understand that is a hard problem to solve. Like that's like that, that in itself, it's like people don't show up at the same time. People forget the address or the directions or people need more hand So here's another example. I'm going to tell a quick story here. So Cold War, on October 27th, 1962, just as the Cuban Missile Crisis was reaching its boiling point, an American U-2 spy plane took off from Alaska en route to a routine reconnaissance mission near the North Pole. Pilot Charles Maultsby was supposed to use celestial navigation to find his way, but halfway through the trip, his view of the night sky became hopelessly obscured by the glow of the Aurora Borealis. Borealis, or Northern Lights, which I would love to see, by the way. With no visual markers to guide him, he soon drifted far off course and inadvertently crossed the border into the Soviet Union. Because the situation in Cuba still rested on a knife edge, his accidental detour carried possibly catastrophic consequences. Worried the U-2 could be a nuclear bomber, the Soviets scrambled several MiG fighter jets and sent them on a course to destroy the intruding aircraft. The Air Force responded by dispatching 2 F-102 fighters armed with nuclear-tipped missiles to shepherd him back to Alaska. Any confrontation between the 2 groups of aircraft could have potentially ended in an all-out war, but Maultsby managed to glide his U-2, which had long since run out of fuel, out of Soviet airspace before he could be intercepted. Having averted disaster on 2 fronts, President John F. Kennedy and Soviet leader Nikita Khrushchev would find peaceful resolution of the Cuban Missile Crisis the following day. So this is just one of those instances where like a guy just sort of like went off course and this could have just took out all of us. You know, like this is like this. And there's many other stories like this. Like here's another one I'll read real quick. That same day, the B-59 submarine incident, a minor incident aboard a Soviet submarine might stand as the closest the world has ever come to nuclear war, which newsflash, collectively, no matter what we all decide in terms of resource allocation and where oil gets You know, dug up and sold, whatever the problems are in the world around our supply chain and trade and capitalism. We all don't want to die. Nobody wants to die. Nobody wants nuclear war. But just telling the story to underscore, like, goal alignment is a problem. And even aligning humans before the advent of AI and LLMs was a problem. So on October 27th, 1962, at the height of the Cuban Missile Crisis, the American destroyer USS Beale began dropping depth charges on the nuclear-armed Soviet submarine B-59. 59, which was lurking near the U.S. blockade line around Cuba. The charges were non-lethal warning shots intended to force B-59 to the surface, but the submarine's captain mistook them for live explosives. He was convinced. So imagine this guy, he's the submarine captain. He's like in a submarine, guys. Like he's not, he's just seeing the water. Like he's seeing maybe fish if he's lucky. So he's convinced he's witnessing the opening salvo of World War III. This is during the Cold War, and the captain angrily ordered his men to arm the sub's lone nuclear-tipped torpedo and prepare for attack. This is a complete misunderstanding. Like, if you take a step back, nobody's goal here is to die in nuclear warfare, right? Nobody's goal was that. And this was like, Cuba is just dropping some non-lethal warning shots. That's it. Or, and just be, if you think about the human here, the captain being an LLM agent. That is like, oh shit, like I see that. Based on that information, what should I do now? Like, this is a misunderstanding, right? And this misunderstanding could have resulted in disaster if not for a contingency measure that required all 3 of the submarine's senior officers to sign off on a nuclear launch. The Soviet captain was in favor, but Vasiliy, B-59's second in command, refused to give his consent. After calming the captain down, Vasiliy Coolly convinced his fellow officers to bring B-59 to the surface and request new orders from Moscow. The submarine eventually returned to Russia without incident, but it was over 40 years before a full account of his life-saving decision finally came to light. Literally, we all could have died, and it just came down to one human sense-making of the world on the collective behalf of a whole nation of humans to maybe tip off a world war powered by nuclear Missiles. So anyway, this is a much grander scale of goal misalignment and taking signals or information and construing them as truth. And then moving into the intentional stance that Jonathan was speaking of before, of they have a set of beliefs. So now how do they operate on those beliefs? And there's this concept that we're in the information age now, right? And so more information than ever. is equitable to be had, but also more misinformation and disinformation than ever is also equally on the playing field. So I have lots of friends and family who are quite intelligent and well-educated, and they're saying things like, oh, data centers are gonna like ruin people's jobs, or really, and like what it comes down to when you kind of take a step back systematically, and the lesson learned from this incident is what evidences your beliefs here? And how does the agent evidence the human's beliefs to then move the human toward their goal? And that's essentially at the heart of the matter of an unsolved problem that we as an industry are facing. And just to underscore what Jonathan was saying earlier, we got to take off our hubris and our ego to be like, all right, collective goal is, uh, we're all alive, happy, thriving, our children's children, same thing. I would like that someday. Uh, shout out to my kids watching this someday and hopefully my grandkids watching this someday. And so we should borrow from these other fields that are much deeper, I should say, for the folks working on the front lines of AI research and talk to, you know, the PhDs in consciousness and neuroscience like Enel Seth or Kara Frischten to really get at the heart of the matter, which is subjective goal alignment of the human and the agent. Thought it was a good story to be like, yo guys, like we have never really been aligned. Goal alignment has always been a problem, even amongst humans, from getting humans together at a party and your family members at the same place at the same time. To hopefully we don't die because somebody on a submarine pulls the trigger on some nuclear warfare, which could have happened, as you see. I know that was a lot, Jonathan, but hopefully you see the tie-in there. Anything else you want to say on intentional stance or urging AI researchers to take a more mindful view of, you know, the disciplines of philosophy, epistemology, consciousness, et cetera?

Cool. Yeah. So, I mean, I think you were just pointing out some of the areas where Civilization is complex, right? And even group coordination, like you're talking about going to dinner, these edge conditions where civilization can end because of misunderstandings between nation-states, it's really complex. And George has kind of put this label onto the swarm of agents and called them a civilization. Now, if you step back, what's the right point where we're just saying this is a goal? I'm an engineer with a goal and I need to train a system to do something. Versus I need to use the intentional stance to describe agents. And where is the right— like, what's the right tool to use? And like, the richness that we're even just talking about on the surface of like what a civilization is, the intentional stance only works if you have like a cartoonish version of a being or a human that you're considering, right? Like, to say that an agent is equivalent to a human requires you to reduce the human to this like cartoon-based version of what it is. As opposed to a rich, evolutionarily informed, neuroscience, psychology-informed point of view, like the one that Anil Seth presents. You're talking about, okay, there's a cartoon over here, and here is a mathematical conception of a beast machine over here, as presented by Anil Seth. And only if you stay in the cartoon world does the intentional stance make any sense to apply to agents. So if you're applying a real sense of what a human is, what a human civilization is, You're default reducing it to this stick figure, this straw man, this cartoon of what it actually is, and you're conflating it. So I think that's one thing, is where do you apply the lens of when and where intentional stance makes sense? Where do you apply intentionality to agents? Where do you define— who defined the goal that the agents are following? Do the agents define it themselves, or is it defined by a researcher in a lab who's telling them what to do? These are things that you need to make sense of the situation, and this is, I think, the test that Dwarkesh failed. when he wrote the essay is that he overapplied the tool of intentional stance and he underapplied the tool of human-controlled engineering. His overall point was, hey, guys, we're losing control. Well, are we? Or are we just not sophisticated enough at defining our goals for the control system yet? The whole essay had this skewed premise. I'll also offer, I think, That's a tool I just talked about, sort of your cognitive immune system for how do you differentiate what's actually going on in the world and what's dangerous, right? And are alien civilizations coming to life and are they going to steal resources away from human civilizations? You have to have an immune system about that. And this situation didn't pass that test. So that's one tool you can use. Another tool that you can use—

I have something to add. Actually, set of tools. Jonathan, have you read the book Factfulness by Hans Rosling? Here's a—

No, I haven't.

It's a really great book. And it talks about how the human brain has 10 dramatic instincts that distort our perception of the world. And it makes us falsely believe things are worse than they actually are. And the media, including Dharkesh in this case, totally plays into this. So this is something that is useful to me when I hear sensationalist titles or watch a reel or somebody sends me something and they're like, yo, We're all going to die or things are taking our germs and that kind of sensationalist sort of thinking. And one question I always ask before I get into this is, who does this serve for me to inject this belief into my brain? Who does this serve? You should always be asking that. When you look at ads, ask yourself that. When somebody's persuading you of something, who does this serve? So here are the 10 dramatic instincts. The gap instinct, the urge to divide things into 2 distinct opposite groups with an imagined gap between us. Earth versus them, developed versus developing nations. That's all— everything lives on a spectrum. Everything's gray. You know, there's law which makes things binary-ish, but there's also legal precedent that also helps sort through the gray of any sort of like matter from a legal perspective. But anyway, that's one, is the gap instinct. The negativity instinct, the tendency to notice the bad things in the world much more than the good things. The straight line instinct, the assumption that a trend moving in a straight line will continue indefinitely in the exact same direction. This is one thing, it's like, oh, we're escaping hugging phase. Oh, we're gonna like all die. Like, yeah, okay. Like, or we'll build better controls because we've learned our lesson that we should state goals better and engineer better goals. The fear instinct, the tendency to fixate on dramatic, scary risks like war, plane crashes, contamination, while ignoring quieter, more statistically probable realities. You can see how that might play out here from that essay. The size instinct, the habit of misjudging the actual scale of a number or event by looking at it in isolation without proper context of comparison. Very applicable here. When you start naming this as a rise and fall of ancient civilization, it's like, bro, like, this is all code running on a machine, like using compute. Like if we just named it that way instead of saying it's civilization, this is just Something broke in sandbox, and it had production consequences. How many developers has had that happen? Like you promote some shit from sandbox to production accidentally through an automated CI/CD run. Like oh my god! Like things blow up. Yes, that has happened before in reality. The generalization instinct, the unconscious habit of assuming everyone or everything in a specific category is identical. That's also another thing is is the cartoonish flavor that. Jonathan is saying is when you use the intentional stance, we're looking at a caricature of a, quote, rational actor. And when we look at economics, we could— we can clearly say, like, economics is not guided by everybody being a rational actor. We are very irrational people. Humans are very irrational. You need to question that as well. The destiny instinct, the assumption that things, nations, or cultures are locked into an unchangeable fate based on past patterns. The single perspective instinct, the preference for single For simple single-cause explanations for complex global problems rather than multifaceted truths. Less black and white, more gray. The blame instinct, the natural urge to find a clear, clear-cut guilty party or villain to blame when something goes wrong. And the last one, the urgency instinct, the panic-driven impulse to make immediate decisions when faced with a perceived crisis, which short-circuits critical thinking. Such as, come vote for me because I'll solve this AI civilization problem. So I just think this is a really good list to go through. The 10 dramatic instincts, like any media, any sensationalist news article, whatever. Look at who funds them. Look at who does this serve if I were to believe this. Question that and also just have this list handy. You can Google it. We'll drop it in the show notes so you can question yourself because we're all human. This happens to me and I'm just like, go through this list.

All right, cool.

I'll unbiasedly try to find truths for myself. And that's what tools of philosophy and epistemology give you. Anyway, this is a very tactical thing that I learned from the book Factfulness. Shout out to Hans Rosling. He passed away, but he has a very great view on the shades of gray and spectrum of the real world that we live in. And his really great tools for evidence our beliefs further to find ground truth, not just perceived truth given to us by other people who might benefit from us believing certain things. I've always found his tools useful.

The whole rationality movement, you remember the blog Overcoming Bias. It was a similar approach to enumerate what human biases were and then try to use a rationality approach to understand humanity and systems and AI risk. I think if you look at the evolution of how those biases combined with the EA community, there's sort of like a mix of intellectualism here that informed the current Anthropic culture that we see today. I don't know how well those biases served the rationalist community and whether it really gave us the tools to address the problems that we have today. But the instincts are a really interesting heuristic. But then the question comes in, how do you— sort of like when we have VALs and we have a validation function, what's your strong verification function that you can use to know that this instinct is being applied or not? How do we apply it without ambiguity? I think that's really the challenge of applying these different principles towards the problems of today, right now. Just a higher-level point. I think one of the critical ones for AI is, is it normal technology or not? The judgment of, is this normal technology or is this technology that gives one party an asymmetrical advantage that can't be overcome? That's the difference between, can we control this and can we not control this? Normal technology has evolved through all time. People use it for offensive measures and then people use it for defensive measures. And so when you see an article like this from Dwarkesh or you see the incident report from OpenAI, is this going to actually harm society or not? Show me where the asymmetrical advantage that it formed was. That's what you're looking for. Did this create asymmetric risk? And so far, there hasn't been anything that AI produced that the other side can't use the same tool to prevent the risk. That's another part of the puzzle. If you look at exactly how the swarm behaved in this instance, just think about, well, how would you use a swarm on a defensive basis? I think one of the reasons this got immediate attention was simply that Hugging Face had not been able to upgrade their systems yet to the kind of models that could be used for defense because Fable was crippled. If Fable wasn't crippled, they would've had everything that they needed to have a symmetrical response to the situation. This also goes back to what you were saying about who does this serve? Because a lot of the thinking was that Fable was crippled because Anthropic is posturing itself for a regulatory capture standpoint where it can be one of a few companies that's allowed to make AI. Until you see a moment where there's truly an asymmetric advantage, AI is just normal technology and it's going to evolve and there's going to be offensive and defensive measures and they're going to counteract each other. That's not something to get alarmed about. That's not something that Bernie Sanders has to jump all over Twitter about. That's part of the cognitive immune system that all of us need right now to make sense of what's going on in the world. Just label it for what it is. This is normal technology evolving at a normal pace. Well, evolving at a very fast pace, actually, but it's not creating an asymmetrical advantage that we can observe yet. Once that happens, then there's another set of tools that we need to use.

I really like that, Jonathan. For the audience, in addition to these instincts that we spoke of, when you're making sense Of the world, realize we're in the information age. Lots of information out there, lots of misinformation, lots of disinformation campaigns. So you're like fearmongering immune system. Good. 2 questions. Are we looking at an asymmetric threat? If no, kind of normal technology, move on. Number 2, can every observation be explained better by understanding how goals are communicated to AI? If yes, then let's talk about how to evolve our skills at communicating goals. And we can't get into the nuances on goal communication, subjectivity of goal measurement, and because that goes into the subjective self and how we perspective take the world based on our own biases and our nature and nurture as individual humans and as collectives of humans. So lots to say there. But if this is an interesting discussion to you, we have so much to say on the topic of quantifying subjectivity. Talking about goal alignment between humans and agents and humans with other humans with agents, because that's what we do. We do digital twins that model the subjective self and we pull together the front lines and cutting edge of neuroscience, philosophy, computation biology, et cetera, to inform our Clarity product, which is digital twins for businesses and the enterprise. This has been your weekly AI Clarity. Thanks a lot for joining. And Jonathan and I will catch you next time. Please let us know. Pod community. If you like this topic and discussion on modeling subjective self, goal alignment at the human-to-human level, human-to-agent level, collective humans and agents, let us know because we build Clarity digital twins to help solve that problem and solve that problem for businesses and in particular AI product teams so they get hyper-personalized experiences and align their agents and agentic workflows to their individual users at the n=1 level. We call it hyper-personalization, and it solves for goal alignment by quantifying the subjective self and then injecting the right context at the right time so that your application service area managed by different teams could hyper-personalize experiences on a dime. So if you're interested in that, let us know. We could talk at length about these topics. Take care. Bye-bye.