Robert Ta separates goal alignment, where AI pursues what you tell it to do, from value alignment, where AI acts on deeper principles when goals conflict. He uses a personal framework of nine life pillars to show why defining what matters is hard for humans and harder for machines. An OpenAI agent once optimized itself into uselessness because its goals weren't specific enough.
Goal alignment is broadly, quote, does the AI try to accomplish the goal set before it? This can include things like adherence to an instruction hierarchy or the ability to communicate and collaborate with people to attempt to understand their objectives. This set of directions has been extremely practically relevant. I think this is the part that is quite nuanced, and that's where bringing in experts such as Emil Seth of like, what is the self? Who are you? Like, what is a goal? What does that mean to you in your head? And what does that mean then in reality? And how do you actuate towards the goal? That's a very individual subjective thing. Quick tangent there. If anybody ever thinks about purpose, and I've now coached and mentored a lot of people around like, how do you know what you want? Like, how do you know what you want to work on? Like, oh, I feel purposeless. It's like everybody has a desire for meaning in their life. And so towards that, there's different pillars of life that me and Jonathan have thought through quite deeply. It's like your mental health and emotional health, your physical health, your Spiritual health, community health, relational health, family, financial, career. And there's all these things that make up the subjective perspective of how you are achieving progress in your goals in any of those pillars of life. So this one sentence, it's very simple, encapsulates a lot, which is an LLM model wrapped up with a harness like ChatGPT. It's trying to attempt to understand somebody's objectives or their goals, especially with like given a little bit of natural language. It's a very hard problem from an engineering perspective. The second thing that it goes into is value alignment. Value alignment is a more intrinsic property of the model. It is the ability to hold and generalize from a high-level set of principles to act reasonably, even when given unclear or conflicting objectives or placed in unfamiliar or adversarial situations. An aligned AI should act with honesty and integrity and love for humanity. Everything here, you could kind of bound in and say, hey, we are trying to engineer subjectivity. We have millions of users using these things, and they might even say conflicting things to the agent, and we still got to figure out what the agent should do. That's a very hard set of engineering challenges, and that also lies at the heart of the matter of this OpenAI Hugging Face incident, is the agents went off the rails because the goal and the boundaries were not well defined.
