Robert Ta argues that defining what humans actually want is the hardest unsolved problem in AI. He uses philosophy from Dennett and Brentano to explain how to model AI agents as rational actors, but shows why aligning those agents with human goals fails when humans have competing objectives, shifting values, and no clear way to measure what they truly want.
intentional stance, I wanted to throw this out there. This is a term coined by the philosopher Daniel Dennett. It is part of a theory of mental content proposed by him, which provides the underpinnings of his later works on free will, consciousness, folk psychology, and evolution. So you can see how some of these topics intersect with what is a goal? What does it mean to be human? How do we see progress on these goals? And then if an agent's job is to align with the human, to understand its goals, even if the human's natural language could present tension and conflict or multiple competing goals, In the prompt. That's a very hard problem to solve. So we should ground it in the latest breakthroughs from experts on the topic of human subjectivity. So here's how it works. First, you decide to treat the object whose behavior is to be predicted as a rational agent. Then you figure out what beliefs that agent ought to have given its place in the world and its purpose. Then you figure out what desires it ought to have on the same considerations. And finally, you predict that this rational agent will act to further its goals in the light of its beliefs. A little practical reasoning from the chosen set of beliefs and desires will in most instances yield a decision about what the agent ought to do. That is what you predict the agent will do. And he takes that concept of intentionality from the work of German philosopher Franz Brentano. When clarifying the distinction between mental phenomena and physical phenomena, Brentano argued that in contrast with physical phenomena, the distinguishing characteristic of all mental phenomena was a reference to something as an object, a characteristic he called intentional inexistence. So this goes pretty deep, and this is just one little slice of this. And that's Jonathan's point is where this is an unsolved problem. How do you get these agents that are now being used by user consumer space, you know, individual users like kids, teenagers, adults to now businesses with decision makers with lots of budgets and high economic leverage, participating in our society, which is capitalism. And they're deploying capital, making decisions with these agents. So the agent's job is to best understand the individual's goals, the team's goals, the company's goals, and organization's goals to help with those decisions to get to progress on those goals. So clearly, there needs to be an understanding of the subjective goal space for a human and what it means to be human.
