In this dialogue, Robert Ta prompts a discussion on the challenge of goal and value alignment between AI agents and humans. The speaker argues that defining human‑aligned goals is an unsolved, interdisciplinary problem that current machine‑learning practitioners are ill‑prepared to tackle because their training focuses on abstract benchmarks rather than the nuanced, embodied cognition insights from fields like consciousness research, psychology, and sociology. References are made to Anil Seth’s layered model of homeostatic behavior, as well as the work of Carl Frischten and Michael Levin, suggesting that these perspectives could inform future AI alignment. The conversation critiques the prevailing “intentional stance” – the notion that AI agents possess intent or self‑generated goals – labeling it as a philosophical shortcut lacking empirical support. It also highlights a misallocation of authority: AI researchers, despite their technical brilliance, are not privileged to define human‑centric goals, a gap that leads figures like the young researcher Dwarkesh to adopt flawed models inherited from AI circles. Overall, the excerpt underscores the need to involve experts from consciousness and embodied cognition to properly address human‑aligned AI.
Chapters
Jonathan, do you want to say more on goal alignment, value alignment of agent and human there and intersect that with any of the work that you've shared with me around Anil Seth's work or Carl Frischten or Michael Levin's work?
Well, I think one interesting thing here is that to understand that this is an unsolved problem. You could call it an engineering problem. You could call it an AI research problem. You could call it a design problem, but we don't know how to do it. Like you can't go to school to learn how to communicate a human-aligned goal and without specificity. What's happened here is that the people who are really good at machine learning, reinforcement learning, they use a set of benchmarks that are goals that are defined, but never in their training have they been taught how to define a human-aligned goal. So this is new territory for them. And you think, well, who are the subject matters in human-aligned goals? And this is where I think somebody like Anil Seth really has a point of view. He's studied consciousness research. He has the beast machine model of the human that talks about all the layers and stacks of homeostatic behavior. And how they interact with each other and how that relates to the controlled illusion that is to be the self and what it feels like to be you in a given moment. And that type of embodied cognition has a few different forks of philosophy that have informed it and is itself a direction that's going to influence AI in the future. But I think when you're thinking about human-centric goals, you're moving away from like the hard sciences of physics and mathematics and computer science, and you're moving closer towards psychology and sociology and perception sciences. And those really use a different epistemology to define what goals are. Well, I think one of the core problems here is that we have people who we really respect, your AI researchers of the world who are incredibly intelligent, but that doesn't give them a privileged position on how to define human-centric goals. And that's where a lot of this confusion is coming from right now. And I think even Dwarkesh, if we step back and say, who is Dwarkesh in the world right now? I have a lot of respect for this guy in that he is a young guy. He asks a lot of questions. You look at the interviews that he does, he does a lot of research about and who he's interviewing. He's curious and he's trying to learn. But I think what he's inheriting here is that he has an assumed privileged position from AI researchers that they understand what human-centric goals are and how they can be defined. And he's inherited this philosophical position called the intentional stance. He's basically inheriting from probably people at Anthropic this idea that agents can have intent, that agents can form their own goals. And there's no evidence of that. That is not a scientific theory. That is basically computer science PhDs and really smart machine learning engineers trying to figure out how to describe agent systems. And if you talk to somebody like Anil Seth, who's more of an expert in this field, it's like you cannot give an intentional stance to a coded agent. It's just not where intent comes from. It's not where goals come from. So this is where a lot of the confusion is coming from, is that this is a hard problem and we're channeling the wrong set of experts in order to define what's going on in the situation. And somebody like Dwarkesh curious, fast learner, he's taking information from the wrong sources and applying it to a world model that just is completely unfit for the situation.
