Jonathan argues that self-improving AI systems like those at Anthropic lack true agency because they optimize against external benchmarks rather than setting their own goals. He draws on Michael Levin's work on embodied cognition to explain how real agency develops, and proposes that true recursive self-improvement will come from manipulating biological evolution itself, not just language models.
Jonathan had a really good tweet. We're headed for a dual singularity and recursive self-improving embodied cognition is going to be much more important than recursive self-improving LLMs. Jonathan, say more on recursive self-improving embodied cognition.
I guess probably most people have heard of RSI, recursive self-improvement. We're on the precipice of that. And Anthropic even released data saying 25% of the research is done by recursively self-improving agents now. That could be 100% in the next year, but What Levin's really talking about is embodied cognition. The problem with the RSI from LLMs is that you still have benchmarks. So you just have this RSI that's like benchmark matching and it can't define its own goals and doesn't have its own agency. What embodied cognition tells us is how agency derives. And that's what Levin's paper really gives a foundation for. It uses this concept called the Platonic space to map out what is a cognitive light cone for embedded cognition. And basically Levin's giving us the way, this is how you define the goals. This is how agents define goals. This is how agency defines goals. And once we understand that, we'll be able to better inform benchmarks that are subjectively informed. You can like actually take LLM-based RSI and make it, you know, sort of aligned with human beings. So that's the premise. But what I meant by that tweet is there's going to be a point in time where recursive self-improving humans, we're taking evolution and we're taking adaptation hypotheses that we can test during a single phenotype cycle during a lifetime. We're no longer waiting for populations to like test hypotheses through gradual natural selection. I think to me, that's real singularity. Nobody's talking about it yet, or at least not in enough detail. That's where Michael Levin's like really pointed at is this point. He says it himself. He's like, look guys, get ready for all kinds of intelligent life forms that are going to be our, your environment. You know, we can change our embryonic process and we can change our nervous system. We can integrate with technology. There's all kinds of spectrums of like cyborgs that we can become. There's all types of like animal life and new social life that can evolve. That to me is like the real recursive self-improvement. It's recursive self-improvement of evolution of life.
