In this conversation, Robert Ta and Jonathan examine the misuse of the intentional stance when describing AI systems as a "civilization." Jonathan argues that applying a cartoonish, intentional‑stance lens to complex, human‑like agents reduces the richness of neuroscience and psychology, citing Anil Seth’s evolution‑informed view of consciousness. He critiques Dwarkesh’s essay for over‑applying intentional stance and under‑using human‑controlled engineering, warning that such oversimplification can obscure who defines AI goals and how control is maintained. Robert then introduces Hans Rosling’s Factfulness as a practical toolkit, outlining its ten dramatic instincts (gap, negativity, straight‑line, fear, size, generalisation, destiny, single‑perspective, blame, urgency) that distort perception and fuel sensationalist AI fearmongering. He proposes using these instincts as a “cognitive immune system” to evaluate media, political rhetoric, and hype around AI, urging a more mindful, epistemologically grounded approach to AI risk discourse.
Anything else you want to say on intentional stance or urging AI researchers to take a more mindful view of the disciplines of philosophy, epistemology, consciousness, et cetera?
Civilization is complex. Even like group coordination, like these edge conditions where civilization can end because of misunderstandings between nation states. It's really complex. And George has kind of like put this label onto the swarm of agents and called them a civilization. If you step back, what's the right point where we're just Saying this is a goal. I'm an engineer with a goal and I need to train the system to do something versus I need to use the intentional stance to describe agents and what's the right tool to use. The richness that we're even just talking about on the surface of what a civilization is, the intentional stance only works if you have a cartoonish version of a being or a human that you're considering, right? To say that an agent is equivalent to a human requires you to reduce the human to this cartoon-based version of what it is. As opposed to like a rich, evolutionarily informed neuroscience, psychology-informed point of view, like the one that Anil Seth presents. You're talking about like, okay, there's a cartoon over here, and here is like a mathematical conception of a beast machine over here, as presented by Anil Seth. And only if you stay in the cartoon world does the intentional stance make any sense to apply to agents. If you're applying like a real sense of what a human is, what a human civilization is, you're default reducing it to like this stick figure, this straw man, this cartoon. of what it actually is and you're conflating it. So I think that's one thing. You're trying to sense-make this world, right? Like, things are moving really fast. Where do you apply the lens of when and where intentional stance makes sense? Where do you apply intentionality to agents? Who defined the goal that the agents are following? Do the agents define it themselves, or is it defined by a researcher in a lab who's like telling them what to do? These are things that you need to like make sense of the situation. And this is, I think, the test that Dwarkesh failed when he wrote the essay, is that he overapplied the tool of intentional stance and he underapplied the tool of human-controlled engineering. His overall point was, hey guys, we're losing control. Or are we just not sophisticated enough at defining our goals for the control system yet? The whole essay kind of had this, like, the skewed premise. I'll also offer, like, I think that's a tool I just talked about and, like, sort of your cognitive immune system for how do you differentiate what's, like, actually going on in the world and what's dangerous, right? And are civilization— agent civilization coming to life and are they going to steal resources away from human civilizations? You have to have an immune system about that. And this situation didn't pass that test.
That's like one tool you can use. I have something to add. had actually set of tools. Jonathan, have you read the book Factfulness by Hans Rosling? Here's a—
Uh, no, I haven't.
It's a really great book and it talks about how the human brain has 10 dramatic instincts that distort our perception of the world and makes us falsely believe things are worse than they actually are. And the media totally plays into this. This is something that is useful to me when I hear sensationalist titles or watch a reel or somebody sends me something and they're like, yo, we're all gonna die or things are taking our gerbs and You know, that kind of like sensationalist sort of thinking. And one question I always ask before I get into this is, who does it serve for me to inject this belief into my brain? Who does this serve? Like, you should always be asking that. So when you look at ads, ask yourself that. When somebody's persuading you of something, who does it serve? So here are the 10 dramatic instincts. The gap instinct, the urge to divide things into 2 distinct opposite groups with an imagined gap between us. Us versus them, developed versus developing nations, That's all. Everything lives on a spectrum. Everything's gray. You know, there's law which makes things binary-ish, but there's also legal precedent that also helps sort through the gray of any sort of like matter from a legal perspective. But anyway, that's one is the gap instinct. The negativity instinct, the tendency to notice the bad things in the world much more than the good things. The straight line instinct, the assumption that a trend moving in a straight line will continue indefinitely in the exact same direction. This is one thing. It's like, oh, we're escaping hugging phase. Oh, we're going to like all die. Like, yeah, okay. Like, or we'll build better controls because we've learned our lesson that we should state goals better and engineer better goals. The fear instinct, the tendency to fixate on dramatic, scary risks like war, plane crashes, contamination, while ignoring quieter, more statistically probable realities. The size instinct, the habit of misjudging the actual scale of a number or event by looking at it in isolation without proper context or comparison. Very applicable here. When you start naming this as a Rise and fall of ancient civilizations. It's like, bro, this is all code running on a machine, like using compute. Like if we just named it that way instead of saying it's civilization, this is just something broke in sandbox and it had production consequences. How many developers has had that happen? You promote some shit from sandbox to production accidentally through an automated CI/CD run. Like, oh my God, like things blow up. Yes, that has happened before in reality. The generalization instinct The unconscious habit of assuming everyone or everything in a specific category is identical. That's also another thing is, is the cartoonish flavor that Jonathan is saying is when you use the intentional stance, we're looking at a caricature of a, quote, rational actor. And when we look at economics, we can clearly say like economics is not guided by everybody being a rational actor. We are very irrational people. Humans are very irrational. You need to question that as well. The destiny instinct, the assumption that things, nations, or cultures are locked into an unchangeable fate based on past patterns. The single perspective instinct, the preference for simple single cause explanations for complex global problems rather than multifaceted truths. Yeah, again, like less black and white, more gray. The blame instinct, the natural urge to find a clear-cut guilty party or villain to blame when something goes wrong. And the last one, the urgency instinct, the panic-driven impulse to make immediate decisions when faced with a perceived crisis, which short circuits critical thinking, such as Come vote for me because I'll solve this AI civilization problem. I just think this is a really good list to go through, the 10 dramatic instincts. Like any media, any sensationalist news article, whatever, look at who funds them. Look at who does this serve if I were to believe this. Question that and also just have this list handy so you can question yourself because we're all human. This happens to me and I'm just like, go through this list. All right, cool. I'll unbiasedly try to find truth for myself. And that's what tools of philosophy and epistemology give you. Anyway, this is a very tactical thing that I learned from the book Factfulness. Shout out to Hans Rosling. He passed away, but he has a very great view on the shades of gray and spectrum of the real world that we live in.
