Robert Ta and Jonathan argue that AI researchers don't know how to embed human goals into systems. Current training methods lack any instruction on defining human-centric objectives, and the field relies on benchmarks that assume clear goals. Understanding AI alignment requires psychology and sociology, not just math.
Goal alignment is broadly, quote, does the AI try to accomplish the goal set before it? This can include things like adherence to an instruction hierarchy or the ability to communicate and collaborate with people to attempt to understand their objectives. This set of directions has been extremely practically relevant. I think this is the part that is quite nuanced, and that's where bringing in experts such as Anil Seth of like, what is the self? Who are you? Like, what is a goal? What does that mean to you in your head? And what does that mean then in reality? And how do you actuate towards the goal? That's a very individual subjective thing. Quick tangent there. If anybody ever thinks about purpose, and I've now coached and mentored a lot of people around like, how do you know what you want? Like, how do you know what you want to work on? Like, I feel purposeless. It's like everybody has a desire for meaning in their life. And so towards that, there's different pillars of life that me and Jonathan have thought through quite deeply. It's like your mental health and emotional health, your physical health, your Spiritual health, community health, relational health, family, financial, career. And there's all these things that make up the subjective perspective of how you are achieving progress in your goals in any of those pillars of life. So this one sentence, it's very simple, encapsulates a lot, which is an LLM model wrapped up with a harness like ChatGPT. It's, it's trying to attempt to understand somebody's objectives or their goals, especially with like given a little bit of natural language. It's a very hard problem from an engineering perspective. The second thing that it goes into is value alignment. Value alignment is a more intrinsic property of the model. It is the ability to hold and generalize from a high-level set of principles to act reasonably, even when given unclear or conflicting objectives or placed in unfamiliar or adversarial situations. An aligned AI should act with honesty and integrity and love for humanity. Everything here, you could kind of bound in and say, hey, we're trying to engineer subjectivity. And we have millions of users using these things. And they might even say conflicting things. To the agent, and we still got to figure out what the agent should do. That's a very hard set of engineering challenges. And that also lies at the heart of the matter of this OpenAI Hugging Face incident is the agents went off the rails because the, the goal and the boundaries were not well defined. So, Jonathan, do you want to say more on goal alignment, value alignment of agent and human there and intersect that with any of the work that you've shared with me around Anil Seth's work or Carl Frischten or Michael Levin's work?
Yeah, well, I think one interesting thing here is that to understand that this is an unsolved problem. You could call it an engineering problem, you could call it an AI research problem, you could call it a design problem, but we don't know how to do it. Like, you can't go to school to learn how to communicate a human-aligned goal without specificity. What's happened here is that the people who are really good at machine learning, reinforcement learning, they use a set of benchmarks that are goals that are defined, but Never in their training have they been taught how to define a human-aligned goal. So this is new territory for them. And you think, well, who are the subject matters in human-aligned goals? And this is where I think somebody like Anil Seth really has a point of view. He's studied consciousness research. He has the Beast Machine model of the human that talks about all the layers and stacks of homeostatic behavior and how they interact with each other and how that relates to like the controlled illusion that is to be the self and what it feels like to be you in a given moment. And that type of embodied cognition has a few different, like, forks of philosophy that have informed it and is itself a direction that's going to influence AI in the future. But I think when you're thinking about human-centric goals, you're moving away from, like, the hard sciences of physics and mathematics and computer science, and you're moving closer towards psychology and sociology and perception sciences. And Those really use a different epistemology to define what goals are. So I think one of the core problems here is that we have people who we really respect, your AI researchers of the world who are incredibly intelligent, but that doesn't give them a privileged position on how to define human-centric goals. And that's where a lot of this confusion is coming from right now. And I think even Dwarkesh, if we step back and say, who is Dwarkesh in the world right now? I actually have a lot of respect for this guy in that he is a young guy. He asks a lot of questions. You look at the interviews that he does, he does a lot of research about who he's interviewing. He's curious and he's trying to learn. But I think what he's inheriting here is that he has an assumed privileged position from AI researchers that they understand what human-centric goals are and how they can be defined. And he's inherited this philosophical position called the intentional stance. He's basically inheriting from probably people at Anthropic this idea that agents can have intent, that agents can form their own goals. And there's no evidence of that. That is not a scientific, like, theory. That is basically computer science PhDs and really smart machine learning engineers trying to figure out, like, how to describe agent systems. And if you talk to somebody like Anil Seth, who's more of an expert in this field, it's like, you cannot give an intentional stance to a coded agent. It's just not where intent comes from. It's not where goals come from. So this is where a lot of the confusion is coming from, is, you know, that this is a hard problem and we're channeling the wrong set of experts in order to, like, define what's going on in the situation. And somebody like Dwarkesh, curious, fast learner, he's taking information from the wrong sources and applying it to like a world model that just is completely unfit for the situation.
