Computational Life: How Self-Replicators Arise from Randomness, with Google’s Ettore Randazzo and Luca Versari

Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week we'll explore their revolutionary ideas, and together we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Thornberg. Hello and welcome back to The Cognitive Revolution. Today, my guests are Ettore Randazzo and Luca Versari from Google's Paradigms of Intelligence team. Ettore and Luca are co-authors on a fascinating recent paper called Computational Life: How Well-Formed Self-Replicating Programs Emerge from Simple Interaction. This paper explores how self-replicating programs can spontaneously emerge in an extremely simple computational environment. And while the paper itself is focused on a narrow set of technical experiments, I found it to be deeply thought-provoking about the nature of life and the potential future of artificial intelligence. By demonstrating how complex lifelike behaviors can arise from very simple setups and basic rules, it helps develop our intuitions for the origins of life on Earth and the possibility of artificial life emerging in our increasingly complex digital ecosystems. In this conversation, we go deep into the esoteric details of their experimental setup, which uses a minimalist programming language to process random interactions between random short strings of code, and yet surprisingly quickly gives rise to complex phenomena including multiple classes of self-replicators, competition, evolution, and even extinction— all without any selection mechanism or optimization function whatsoever. It really is profound to behold just how much complexity emerges from randomness in such a tiny digital toy universe. Toward the end, we zoom out to consider the implications of this work for AI development and safety. While Ettore and Luca are cautious about drawing too many conclusions, I can't help but understand this work as yet another reason to expect the unexpected. If such simple digital life can form from pure randomness, I have to expect that the optimization power of the economy as a whole will give rise to meaningful forms of self-replicating AI systems. And yet, even with determined effort, I find it extremely hard to imagine what those new forms might be like. This episode exemplifies why I love making this show. We start with a seemingly niche technical paper and end up grappling with some of the biggest questions facing humanity and all the other life forms with which we share planet Earth as we collectively hurtle toward an AI-altered future. As always, I appreciate it when listeners share the show online. Feel free to tag me. And of course, we welcome your feedback via our website, cognitiverevolution.ai, or you can DM me anywhere you like. Now, I hope you enjoy this horizon-expanding conversation with Ettore Randazzo and Luca Versari from Google's Paradigms of Intelligence team on the emergence of computational life.

Ettore Randazzo and Luca Versari from Google's Paradigms of Intelligence team, welcome to The Cognitive Revolution.

Glad to be here.

Nice to be here.

Yeah, I'm really excited about this conversation. It's prompted by a recent paper that you guys put out that I have found really super interesting and quite thought-provoking. It's called Computational Life: How Well-Formed Self-Replicating Programs Emerge from Simple Interaction. And that's a pretty good summary of everything that we're going to be unpacking over the next hour and change. I would love to start with some real kind of basic stuff, and we'll get into a lot more detail. I'm coming to this from a very AI-obsessed perspective. I assume you guys think a lot about how your work connects to the future of potentially many AIs and potentially AIs becoming a new sort of life form as well. I'm always trying to figure out where is this going in the big picture? Is it crazy to think that AIs could be at some point properly viewed as a new life form, or is that in fact realistic? And what criteria would it need to satisfy? So I thought this paper was really enlightening and again, really thought-provoking in a number of different ways. But one kind of simple early argument that you make in the paper is that the capacity for self-replication is really fundamental to the notion of life. So I'd love to just give you a chance, first of all, to start with that high-level observation and unpack that argument for us.

Okay. So from my point of view, of course, getting into the business of defining what life is, is pretty complicated. But one thing that I think people can agree on is that life is like something that has some inherent order. Let's say it's not Like a random collection of molecules or whatever is probably not life. So from that point of view, one can think of things like the second law of thermodynamics that would say that like it's hard to have order effectively. And in that way, then basically a self-replicating system is one of the simplest systems in which you have this force that acts against the second law of thermodynamics because copying oneself introduces some structure, some, some shape, some coherence into what otherwise would be just completely disorder. So that's why I personally think that self-replication is one of the first steps towards life.

So in the field of origin of life and also biology and artificial life, I think self-replication appears always at the very beginning because it looks like it's the best way that nature at the very least has figured out to preserve and accumulate information over a long time. You can essentially see that we try to explain in the paper as well and show that it happens there. Some things can happen even before there are self-replicators, but until there's not a self-replicator, it's just a dynamical system. And when there's the first self-replicator arising, then you start accumulating information that gets repeated over time. And then you can think about biology and nature as an evolution, essentially, as a fight against many different self-replicators that try to compete for limited resources. And what you observe in the current world is essentially the most efficient self-replicators for any given niche that you observe in the planet Earth.

Maybe I am overextending my claims here, but one could even claim that intelligence evolved as a way to self-replicate better.

It doesn't mean it's the only way that you can get to intelligence, but at the very least, we know that nature did it this way. So it would be, I would say, unwise to not explore this direction.

Cool.

So I also came away from the paper with maybe one other sense of why it matters, which is that it seems that prior to the appearance of self-replicators, the systems as a whole are in some ways just noisy, and the most kind of interesting complex dynamics only seem to emerge in the presence of multiple self-replicators. Maybe I'm overreading that aspect of the paper, but that seemed to be pretty important. And it almost seemed like there was almost like a value judgment implicit in that. And I don't know how you guys feel feel about that, but it seemed like there is something that we value this sort of higher level of complexity on some level, as opposed to the more noisy reality before self-replicators.

I think that one way that you can think about what happens before is that it's almost like noise, essentially. Like, there are things that happen, there are complex behaviors that are going on, but if they cannot be preserved and continued, they are just transient. And when you have some way to preserve information and also to, let's say, reproduce, so to have more of you that do the same thing, then you start observing it with more and more frequency. And then, of course, there's the second part of it, which is once you've got self-replication and once you are competing for space, evolution kicks in. And then you can really start to have more complexification of behaviors because competition and collaboration really helps with that.

Yeah, gotcha. Okay, so the study that we're going to go deep on here is, if I were to describe the motivation to someone, I would say these guys set out to find the very smallest, simplest environment that they could possibly come up with that could still give rise to self-replicators. Is that a fair characterisation? Is that the actual motivation that you had coming into this?

So on one hand, for sure, simplicity is a motivating factor. In fact, what I consider one of the selling points of this paper is that we have an environment that doesn't explicitly favor the existence of self-replicators. It's just self-replicators appear, but we're not looking for that. So it's really hard to talk about what the simplicity of an environment is. There are many ways to define that. What I think is the— what actually got us into this was when one of our co-authors did the first experiment and saw this phenomenon of just things suddenly getting into— very suddenly without asking for it, without anything. And we found that fascinating. And that's what I think. is the most important part where, you know, how in— there are systems in which we don't ask for a replicator. It just, there is some spontaneous evolution and that creates structures that are complex enough to copy themselves. And that's, that's the part that is interesting.

One reason that we, we don't want to focus too much on the simplicity, although simplicity really matters, of course, because you, you don't want to study too complex things. But you can think about a simple system where you've got a copy operation and then everything can be thought as a self-replicator, but that's uninteresting. Or at least it's not interesting to think about how did we get from nothing to self-replicators. I think the most important point on what we are trying to show here is how we can pass from a period that, which has, let's say, a random soup of programs that do nothing and that are randomly initialized, that they have a transition that goes into a period where there's self-replicators. How this happens and why. is part of what you're trying to explore here.

Yes. And this work doesn't go into the why yet. We are doing follow-up work that tries to figure out exactly what is going on. But for now, we just thought it was so interesting that we couldn't not show with the work that we have this kind of phenomenon.

And I think this was quite an interesting, almost philosophical note in the paper, where on the one hand, you're looking for some measure, right, to characterise these systems. And the extremes are both uninteresting. Noise, pure noise is hard to compress, but that doesn't make it fundamentally interesting. And then on the other extreme, pure order, a pure crystal lattice or whatever that's totally regular is high order, but also not super interesting. And so you end up looking for something in the middle that has the Goldilocks properties of interesting complexity. Maybe you can give more kind of motivation or theory or philosophy behind this, but I didn't have a great sense for kind of exactly how principled that idea is. Or maybe you can first of all just define the metric that you ended up using and then give a little more colour on could reasonable people disagree on whether that's a good metric, how should we think about this metric of complexity?

Okay, so first let me just note that I have a bit of a background in information theory and compression, which has colored some of the ideas behind this specific definition of complexity. And of course, another important note is that I think you could have discussions for years about what complexity actually means. So what I wanted to capture when using this definition of complexity was basically trying to have a somewhat mathematical definition for structure, like amount of structure that is present into— in a string or something else. Of course, as you mentioned, this notion of structure has some issues because too much structure is also not too interesting. And I think that the metric we ended up defining has a reasonable trade-off between overemphasizing structure and ignoring structure entirely. Anyway, so the metric we ended up going with is, at least theoretically, and I'm not actually giving an exact definition because there are a lot of details that one needs to get right, but intuitively, the idea is to consider the difference between the Shannon entropy and the Kolmogorov complexity of the data that we have. Now, why would someone do this? Intuitively, one can think of Shannon entropy as how hard would it be to describe, let's say, this string of characters if I were completely ignoring any kind of structure that it may possibly have. For example, a sequence of half A's and half B's has the same Shannon entropy regardless of whether it first has all the A's and then it has all the B's, or whether they are just mixed together completely randomly. Shannon entropy is a measure that completely ignores structure. Kolmogorov complexity is basically sitting at the opposite end. It's saying, what's the shortest description you could give of this string? You can do whatever you want. Any Turing machine, what's the shortest Turing machine that will produce this string as output? So a Kolmogorov complexity absolutely can tell the difference between first all the a's and then all the b's, or alternating a's and b's. So then the idea of taking the difference is that one ignores the structure, the other one lets you take any amount of structure in consideration. So what is left is basically only the structure. So that's the main motivation for the definition. I think also that may be a good idea to make an analogy with the real world. that maybe will help to clarify the definition a bit. So imagine you have a building. A building contains a lot of molecules, a lot of different materials, a lot of different things. And you can imagine the equivalent of Shannon entropy would be something that tries to define the position of all the molecules independently for the context of a building. The equivalent of Kolmogorov complexity would basically be, here's the blueprint blueprint of the building, and given the blueprint, you know how to build this building. So that tells you what the building is. It's a description of the building. Now imagine that you take the same building and you completely destroy it, and it becomes just a pile of rubble. For a pile of rubble, it has the same molecules, so it has the same entropy, but it doesn't have a simple description like a blueprint. If you wanted to get the same pile of rubble, you would need to know where all these random small pieces So because of that, in this analogy, the pile of rubble would have a significantly lower complexity or structure. And that's more or less the intuitive reason why I think that defining complexity in this way makes sense for if one wants to emphasize the amount of structure that there is in, in a—

Hey, we'll continue our interview in a moment after a word from our sponsors. Is there a philosophical extension to that or implication of that?

When I read this paper, I can just look at the graphs.

There are these graphs where it shows complexity rising through time as the dynamics of the system evolve. And we'll get to the actual details of the nature of the system in a second. But I really was left with this feeling that there is some implicit value judgment over these states. And I don't know if that's something you would want to sign on to or not, but it brings to mind these various thought experiments for me where, if I understand correctly, a tiled universe of happy people, if you were a utilitarian, you might say, oh, geez, if I had this one very happy person and I made a zillion copies of them and filled the whole universe with this same thing over and over again, That would be great because all these people would all be happy and we would count up their utility and have very high utility. So fantastic. And then somebody else might say, doesn't that sound kind of boring? Shouldn't there be some sort of discount for the fact that this same thing is taking up all the space? Don't we want more diversity for its own sake? It seems like this measurement that you've defined implicitly suggests that second frame. Is that fair or would you not go that far?

I have to say we did not assign any value judgments, philosophical implications of this definition of complexity. We did not consider that. In fact, I would be somewhat surprised if this definition of complexity becomes useful outside of this paper. Pleasantly surprised, but still surprised. So one thing I want to say is that This is a measure of structure, right? It's not a measure of value, or it doesn't take into account diversity. It doesn't— it's not a measure of utility, right? So it's just a measure of structure. Because of that—

Wait, just quick interjection. There is some way in which it counts diversity, right?

To some extent, yes. So it's a bit hard to explain exactly how repetition gets taken into account by this measure. This middle ground is the best point, as in if you have no repetition at all, then very likely this complexity will be low. On the other hand, if you have a fully repeating— n copies of the same character, for example, then the measure of complexity will also be very low. So you do want some amount of repetition, and ideally repetition that itself has some structure. But not too much. But that doesn't imply that you can still have repetition of one relatively complex structure that's just the same but repeated a lot of times, and that will have high complexity. So it's a bit of a trade-off. If you have too much repetition, complexity is low. If you have too little repetition, complexity is high. In the experiments we have been doing since programs at 64 bytes, that turned out to be a reasonable trade-off, and high complexity effectively correlates extremely well with the soup that's full of self-replicators.

And to be clear, this is also purely a measure, right? This measure of complexity is not being fed back into the system in any way, shape, or form.

Not at all.

Yeah.

Okay. Also, just, I think I have to make the disclaimer. Kolmogorov complexity is not something you can actually compute. So we're not actually using Kolmogorov complexity. It's just mathematically easier to reason about Kolmogorov complexity than it is to reason about whatever other proxies for Kolmogorov complexity you may end up using.

Gotcha.

Okay.

About the complexity metrics that we used, we have also used something else once, which is counting the number of unique tokens arising into the soup. And this would mean that if you got a self-propagator that takes over, we don't count the reproduction of this self-propagator to another string as creating new tokens. And this shows that, like you see when self-propagators arise, very stark decrease in the number of unique tokens that there are into the soup. And then we checked, and this is very well correlated with this other complexity metric. So you can think about us using this higher order entropy metric for, as a proxy for this unique token count, because of course it's much easier to compute.

Gotcha. Yeah, that's a very good point. One of you guys can take just setting up what is the little micro self-contained universe that you've created, and I'll ask any clarifying questions that I have.

So the very simplest experiment that we performed is as follows. Imagine that you have a lot of, a soup of random strings. Those random strings represent code, but you can, they can also represent data. Those are essentially what you would pass to a compiler to execute some code to a computer. And what we say is, you randomly take a pair of strings, concatenate them, and execute the code starting from the very bottom left. And you execute the code for a long time, and then you just split those strings again and put them back into the soup. Now, repeat this many times and have a lot of strings do that in parallel. So this is the only thing that happens in the simplest setup. You just take 2 random strings, concatenate them, execute them, and then split them again. So there's no objective function whatsoever. There's nothing else that happens from our side, at least in one of the possible setups. And the only thing that happens is that those strings modify themselves by reading each other's code and, and modifying the code.

Let me show you an example. which is also the example that captivated us into digging deeper into this. Let's see. Okay, so here you can see a few programs from the soup. Each program is 64 bytes. This is one program. There are 2 columns, not for any particular reason, like it's just that it makes better use of space. And you can see that most of the bytes are random, and there are a few bytes, you can see the ones here that are highlighted, that actually represent instructions, and everything else is just nothing. There's also a special character that represents the byte 0, which is special, and you can probably see that here. I'm not sure how well it's visible across the video, but it's a null byte. Okay, so if we let these programs randomly execute, you can see that they start modifying themselves, and sometimes some strings that appear, like copies of the same character, or maybe the ends of the string gain some characters, lose some characters. There's also a few mutations that just erase things from here and there. This evolution continues, There's a lot of interesting behaviors. There are like things that extend, things that— for now, there's nothing too interesting that is happening. By the way, this is the same example that is explained in the paper in figure 2, I believe, where there is like a story of the replicators that appear here. So somewhere around now, you notice that there was a lot of activity and that activity left a lot of null bytes. Now you can see here we have a count of how frequent is each byte, and you can see that the null byte now is way more common than it was before. And if you just go back a couple of seconds, okay, if you go back a couple of seconds, there are basically no null bytes, are not special. But in the time of just a very short number of executions, there are a lot of null bytes. After that, there is another little bit of nothing too interesting happening. And then somewhere around now we get actual replicators and this explodes in a lot of activity. And at the end, like now it's paused, but at the end you can see that there is the same structure that repeats itself basically everywhere in the soup with some minor modifications. So seeing this for the first time is what gave us the intuition that there was something interesting happening here. This kind of weird evolution over time, different behaviors and all of this.

So what happens before separate repertoires arise is actually not completely random. There's a distribution of characters that changes from the very beginning. You can think about it as just a complex dynamical system where there are operations that are growing inside, but in this transient way that is not stable until there's self-replicators arising. And this is, if you were to talk about it in the origins of life terminology, you could think about this as being the pre-life period, a period where there's dynamic interactions. Usually in that case, it would have been with some catalysts that catalyze each other or create some complex dynamics until to the point where self-replicators arise. And then this moves into what sometimes is called the life period where self-replicators arise. And the important part there is that things happen also in the pre-life period, but whenever there starts a replicator, this change is very stark. There's a very big transition in dynamics, and that's why it's useful to think about it in these terms. So our paper is mostly focused on the, how do we get to this transition? How do we get from a random soup to something as complex as a Frigates.

So, for the uninitiated, which included myself up until very recently, let's take just a little bit more time on what we were just looking at there. And you can maybe even pull it back up again, just to have the thing on the screen. So, you start off with just this totally random set of strings, each of which are 64 characters. So that's worth just pausing on for a second to say that is a remarkably small number of characters for anything really interesting to happen. But that's the whole thing, right? You've got 64 characters. And if I understand correctly, you're literally generating these out of ASCII characters at total random?

Not ASCII.

So literally just, just bytes, but yes.

Okay. Random, independently uniform. Like every character is selected independently from everything else.

Gotcha.

And then the way that this is going to be interpreted, I'd never actually heard of this before, is through this extremely minimalistic programming language with the charming name of Brainfuck.

Yes.

And this is a prior art, right? But this is something that somebody made to try to say, what is the smallest possible computing paradigm that we could imagine? And stop me at any point here if I'm wrong, but what actually happens is you have a 2-head computer that both start in the first position, if I understand correctly. When you take 2 strings, you put them together. Now, why do you take 2 strings and put them together? That's essentially simulating the randomness of the universe, right? We've got molecules bouncing around, Brownian motion in solution, whatever, in a gas, whatever they're doing. They're bouncing around and they're bumping into each other at essentially random, right? Or at least we're modeling it as being random in this case. So take 2 of these strings out and just simulate an interaction and just arbitrarily, tell me if there's more to it, but it seemed arbitrary that we just concatenate them and then we'll treat this as if it were Brainfuck language. There are only, looks like 10 different commands in the entire language. And all the other characters out of the 200 and— it's 256, right? Possible bytes. All the other things are just considered to be neutral data.

And then I'm—

so the commands that can happen are like, okay, you've got these 2 computer heads, they both start at 0. One thing to do is move ahead from place to place.

Mm-hmm.

And you literally just move them one square over, right? So it's move one to the left, move one to the right. There is increment and decrement of value. So you have a value, I guess that assumes that there's like an order to the bytes as well, or do you just increase the bit value by 1? So the heads can only move 1 position, 1 step at a time, and the bytes can only move 1 value at a time.

Yes.

And then you can also copy from 1 position to the other.

Yes. And there is the more important instruction, the one that actually makes this language Turing complete. And just for full disclosure, this is not exactly Brainfuck. This is very close to Brainfuck, but the standard Brainfuck language only has one head and then the input and output, like the copy operations, read and write things to the terminal for humans to interact with. But in this setup, there are no humans, so it makes sense to have 2 heads that modify things themselves. So let me show you, this is a representation of an execution of a program. There are actually 3 cursors, not 2. The 2 heads have to do with the position that you read and write from, and the 3rd head is the position of the next command that you execute. Which is fundamental. In particular, the next command that you execute, head, is the only one that can move by more than one position per step. And those are the square brackets. What the square brackets do is they look, when you execute that instruction, if the character is like on the, I think the blue head, I'm not sure if I remember correctly. Is 0, and that's why 0 has a special representation, then you skip that execution, like you skip those brackets if you are going in one direction, or just go through. If it's not 0, you jump to the other bracket. So that's how you can implement things like conditions and for loops and all of those things, which is what allows for this language to be Turing complete.

Just one note. So one note, Brainfuck is not necessarily the, the, the even smallest language that would be Turing complete. It was created, as far as I understand, to make it very hard for humans to code in it.

Yes.

You can have even simpler instruction sets that are still Turing complete.

Hey, we'll continue our interview in a moment after a word from our sponsors. I just want to make sure that people have a very maybe not full understanding, but at least an intuitive sense for just the sort of universe that we're operating in here. So maybe just try to say it back to you one more time. There are 3 computer heads on this that operate over this ultimate space. When 2 different strings are attached together, they go from 64 to now combined, they're 128. So you've got 128 bytes, 3 positions, 2 can read, write, and be moved left or right and can increment or decrement. And the other one is the one that handles the commands. That one is the one that can bounce around when it hits these open and close parentheses type operators. And I guess there's maybe 2 more things that matter here. One is the definition of a self-replicator. That is, When the first 64 bytes manage to copy themselves over to the second 64 bytes exactly.

Mostly. It's slightly more complicated than that. Because they need to copy themselves to the other 64. The other thing they could do is have the other 64 bytes be the copy of the original program, but destroy the original program while doing so. And that would not be a self-replicator. Something moves.

Also, to make things more complex, a self-replicator doesn't have to be only with 64 bytes, only if it's entirety of 64 bytes. You could copy a smaller part of the, of this substring, let's say of the 64 bytes many times, and this will still count as self-replicator. So it's actually quite complex to tell whether you have created a copy of yourself as a substring, because it's not just saying 64 bytes. on the left need to be equal to 64 bytes on the right. It could be that it's like 13 bytes on the left that replicate on a very random position on the other side, or some of them that get replicated many times in this 128-byte concatenated string. And this makes it even more important to figure out a complexity metric to understand what's going on, because it's very hard to detect explicitly for a perfect self-replicator otherwise.

Yeah. The way I like to think about it is that saying that it's hard to say whether one program is self-replicator, but it's a lot easier to say whether the soup is full of self-replicators.

And it's even more complex than this because in some experiments we have one single long tape where there's no concept of individual strings. So to determine whether there's a self-replicator there, you cannot avoid it. You really need to think about, there are sides of which can be anything for a substring, which is a self-replicator. And how do you tell whether a self-replicator is replicating in any given offset?

And that's basically just an experimental variant where instead of having these discrete units and merging them randomly, you just literally run the program indefinitely over the one super long string of characters.

I believe the way that setup works is that periodically you stop the program and you start it again from a random point.

Yes.

Gotcha.

But I think that thinking about this example is the simplest. And there are also extensions which are putting one dimension or 2 dimensions into the equation. So it's no more uniform distribution, but it's a uniform distribution among your neighbors, but it doesn't change the concept. Thinking about just having uniform distribution of random strings concatenating should work.

Gotcha.

Okay.

And then the one other thing that I wanted to note is that there is the possibility for errors in these programs, right? Since the data was all generated randomly in the first place, when you try to run this random string of characters as a program, it can error out. And specifically, it seems like it errors out if the sort of control logic breaks. If it hits one of the, a particular special character, which is the zero. character, and it's meant to look for the next or the last parentheses to handle control logic, and there isn't a corresponding proper place for it to go to, then that's basically just the end of that run. Is that right? And everything that happened up until that time, all changes that are made, those are just left in place and we move on to the next step in the simulation.

We tried a few different setups and the overall result doesn't change that much anyway. But yes, I believe in this setup, if you encounter a jump instruction that doesn't have anything to jump to, it just stops execution. And there are a few other things, like if you end up trying to execute past the end of the string, it also stops execution because what would it execute anyway? But in general, one important thing is, and also one of the reasons why we are using Brainfuck and not C++, Is that we want the interpretation of programs to be as permissive as we reasonably can. If you don't do that, then, you know, then basically you never actually get anything interesting because you just error out every single time.

Okay. So I think that's a pretty good overview of the setup. It's both remarkably simple in some ways and obviously has some nuance. Let's talk about what happens as you run these things. I think you've alluded to it a little bit already, but essentially you have this pre-life, or at least pre-self-replicator period in which everything is still random. And these things are being concatenated, they're being run, data's changing. You can maybe tell us a little bit more about what evolution of the overall system you can see during that phase. And then there's a very sharp phase change. And I think this notion of phase change is a really important one that happens when the first self-replicator pops up. And again, probably can't stress enough, there's no incentive for this self-replicator to pop up, right? No loss function here like we're used to thinking about in AI systems. Literally just blind execution of the program stumbles on through random interaction between 2 strings, something that is a self-replicator. And now when that happens, very quickly, it starts to replicate a lot. And now we're into like a different regime in the overall story of this little universe.

So since we initialize everything with random bytes, in the beginning, everything is random, right? So you can see, okay, so this is already after a few executions. It's not exactly quite uniformly distributed, but here we have the least common character and here we have the most common character. And you can see they are pretty close to each other. No character appears more than half a percent of the time. So the first thing that happens in the first few thousand executions is that this distribution shifts. And you can see here, after a few thousands of times that programs interact with each other, we start having a few characters that appear like 2% of the time.

Hmm.

We are still investigating exactly what is happening here. There's clearly something that is important. Here we can see that this character, this is one of the characters that moves the heads, for example, is getting particularly popular. And that some characters also become particularly unpopular more than others. It's not like there's just one character that becomes significantly more common. It's also that other characters become less common. And then we have some of these state transitions, as we were saying, where the behavior changes significantly. Here you see there is a lot of activity. for a really short time. And that leaves you with a completely different distribution. And some characters almost completely disappear.

And there's one interesting thing that one of our co-authors observed, that you can think about the initial distribution where you got some operations, which are code that you can execute, and other operations which are just data. So if you try to execute there, nothing happens. So even at the very beginning, the only thing that can generate more code is code itself. And since the very beginning in this kind of system, the number of instructions that get executed and that are present in the soup starts increasing immediately. So even before having self-replicators actually appearing, we still have some, one thing that could be very similar to the concept of autocatalytic sets, where those codes, strings of codes or pairs of codes generate other pairs of codes that then can have a chance of generating themselves again.

Yeah, that's really interesting. I'm noticing just looking at the top of this that it seems that both the highest frequency characters, although maybe not that equal sign.

No, the equal sign is not.

If I'm reading this right, I'm seeing the open arrow. The open arrow is the 4.94%, right?

Yes. So here we have the 16 most common characters and the 16 least common characters. It is quite interesting indeed that plus and minus tend to be not that interesting, but I think perhaps makes sense because if you execute a plus, that, that will likely make an open parenthesis into a completely different character. So that's likely to ruin meaningful code. And the pieces of code that are useful for to copy, like the instructions that are useful to copy things over, like the, the loop instructions, the, the instructions that move the heads. And the copy instruction are super common. That's, at least we believe that's because if you have a lot of copy instructions, then it's easy to produce more copy instructions because you just will read something and that something is probably a copy instruction itself. So you will copy that over. And this is a self-reinforcing effect. Like having many copy instructions just gets you even more copy instructions. And probably there is something similar happening with loops, loop instructions. Yeah.

Cool. That's fascinating unto itself. The instructions that enable copying tend to get copied and the ones that scramble data tend to scramble themselves out of existence. And, and this is before we've even got to self-replicators. Now, I guess one could ask, is there a sharp line between this notion of Even you raised the concept of an autocatalytic set. Is there a bright line between that and a self-replicator? It seems like there is, because there's a phase transition. But when I just look at a pair of characters could be in, in a sense enough, right, for it to be copying itself over. I feel like I'm missing some critical distinction there.

Yes, I think I can help here. So there are also people that would very easily call autocatalytic sets self-replicators. It's a matter of how efficient they are, because the biggest difference between the pre-life and the life period is the speed of, let's say, information to be replicated and be copied into their neighborhood. When you have got the pre-life period, you've got some autocatalytic sets that happen, but they're very slow in copying each other, also because they need to do many more steps before they become multiple of what they were before. And it's also by chance. But when you got a function as a replicator, in biology, the best example is like something that people suspect being the RNA and with the polymerase and the ribosome, then this just sped up very quickly. And then it, in much shorter timelines, it takes over the entire soup.

So looking at the paper, there is a Figure 1. that shows us the number of unique tokens or even the complexity. And you can see that the number of unique tokens slowly decreases in the initial phases because—

Quick clarification. A token here is a 64-length string, no?

A token is a marker that basically identifies— is what Ettore was saying before with the tracking of copies. So number of unique tokens is the number of bytes that have not been copied.

Yeah. So in a string, there will be at the beginning 64 unique tokens because it's 64 bytes.

And then if you copy half of the string over itself, then you have 32 unique tokens because half of those bytes come from the other half of the string. So it's just keeping track of where the bytes originally came from. So it's—

And you're actually keeping a full log of every copy event.

We did it once. Let's put it that way. But yes, this is, this number of tokens takes into account the history of every byte and looks at where it came from. And if the token came from the same byte, then it's the same token.

So there's one caveat, which here becomes quite important. So far we talked about separate self-modification being the driving factor of self-replicators to arise. And this can be proven even by just having experiments where there's absolutely zero background noise. So the only way to modify some characters is to execute the code. But we usually also add background noise. So some, every now and then, some characters get randomly modified. This helps to speed up the process. And so you need to find self-replicators which are more robust. But it's not necessary. We still observe everything happening even without background noise. What you're seeing here in this graph shows that unique tokens decrease. And that's also because, sorry, they decrease, but there's the impact also of having, adding unique tokens every now and then that counterbalances it.

Yes. So if you look here, once we do have a self-replicator, the amount of unique tokens increases. That's obviously not possible if you don't add new tokens every now and then. So this increase here is the effect of mutation. So what is interesting, tying back to the conversation we were having before, is that here the number of unique tokens, even before we have anything like a full-formed self-replicator, is steadily decreasing.

Mm-hmm.

So that's basically saying that there is a Fair amount of copying that is happening even before we have actual self-replicators.

So that first line, when the first self-replicator pops up, is obviously a big event in the history of this system. You've got a major drop in unique tokens. You've got a huge rise in top 32 tokens. And that is, should I understand that as the sourced back to an original Initial token? Like this is the—

So I believe that—

A character that has been copied the most?

Yeah, basically top 32 tokens is basically saying how many copies exist of the 32 most common bytes. So as you can see here, there is a sharp increase because at that point we have a self-replicator that copies itself a lot. And by byte, I don't mean like byte value, I mean original byte.

So then the Complexity also spikes because we go from a noisy environment to a more ordered environment, and then it drops back down. And should I understand that as being a period of tiling? Is this— is that where we've overshot and gone to the other extreme in complexity, where the self-replicator is everywhere for a while?

Actually, no. So let's see if I can find the moment corresponding to that. Point here in the video. I believe it was here. So that's, that's basically an extinction event.

I'm, I'm still on the, um, on the graph.

Yeah. Okay. So in a few seconds you will see there is going to be a lot of movement and then nothing will happen after 1 second of a lot of movement. And that's the spike. See here, there was a lot of changes. And now everything is stopped. Yes, there is still some modifications, but it looks like it looked in the beginning, right? Like, it doesn't look like the self-replicator regime here. When you do have successful self-replicators, there is a lot of movement around. And you can see that, but only for a really short time. And that's the time that corresponds to this spike. Because there we have, you have replicators take over, but they are this initial kind of replicator that we have here as an example. And the interesting property of these replicators is that if the bytes that they are trying to write themselves over are 0, they just don't do anything. If they try to overwrite a set of bytes of any kind, it works. You can see here that it's just, you know, slowly copying itself in reverse. But if the bytes you had there are actual zeros, then it doesn't work out. So it just doesn't do anything. And so what this does, and you can see this in the soup here, what this does is basically very quickly you have every single byte at the end of the strings be a null byte. You see here. And now the replicators cannot replicate themselves anymore. And so basically the replicators go extinct, nothing happens. And then afterwards we find a different kind of replicator that doesn't have this problem, like manages to copy itself over null bytes without any problem. And because of that You have the second state transition here. The soup becomes entirely self-replicators and the complexity just jumps up. So this is basically just a short extinction event. And it doesn't happen always for that matter. Like in most of our experiments, we just see a sharp jump in self-replicators and that's it. It's just that we were lucky enough that the standard setup ended up having this particularly interesting example. So, which is the one that we ended up showing here, but. It doesn't always happen. What's more typical even is just that as soon as you have one replicator, you, the soup takes over and there's no extinction event in the middle.

Not one replicator in the sense that we have actually also tested the hypothesis of if you had the one self-replicator initialized in the soup, what's the chance that it takes over? And it's actually quite low. It's what, 23%?

22, 23.

Because there are many things. Yeah. Things that can happen that destroy a self-replicator. If there's background noise while they self-replicate, they may just be destroyed by the noise. And if there is no background noise, you might be in the wrong order when you try to replicate. So if another string is the one on the left and it starts from this random code, it might destroy you and therefore you will not self-replicate. So there's actually low chance for one self-replicator to take over.

In particular. We don't— we'd not test this because, as we said, it's hard to see if one program is a self-replicator. But what we expect is that there actually have appeared other self-replicators here that just died. What is remarkable about this specific evolution is that the self-replicators took over a fair bit and then still died.

Gotcha.

So can you Give a little bit of, I don't know if a taxonomy is too much to ask of the different kinds of self-replicators that arise. I might be overinterpreting the graph, but I came away with the sense that there are some primitive ones that pop up first, but then they have these problems with zero, and so they sometimes fizzle out, and then more robust ones come later, but maybe they take longer to emerge. But maybe I'm telling more of a story there than I should. Maybe it's more random actually than all that narrative that I just laid out.

So, we certainly have observed different kinds of self-replicators. So here you can see the first replicator to appear in that setup. And it has this structure where it starts a loop and then it first copies the characters, then moves the 2 heads and then loops back. So you can see here the execution, right? So when it's execution— executing this character, it copies this space to here. Then it moves the green head back. Sorry, the blue head by 1 position. It moves the red head forward by 1 position. And it says, this is not 0, let's loop back. Sorry, this is not 0. That's the byte that gets checked. And so here it repeats. And so now it just copies itself over. And as I was saying before, this has the problem that if it tries to copy itself over, when it does this check of, okay, should I go back here? It says this is 0. So no, I will just go forward and then nothing interesting happens. The second replicator that appears is somewhat more resilient and it's fundamentally still the same thing, right? It still does a loop. It still moves the 2 heads, but the difference is, I don't know if you noticed this, but the order of the heads is swapped. So here you write to the blue head, and here you write to the red head. So what this means is that having zeros in the string you're trying to overwrite doesn't give you problems. You do get problems if you have zeros in the string you're trying to copy. If this were a zero, Then you fail at copying yourself because you just copied the first part of the program. You don't copy the entire program. And note that the program is copying itself in reverse order. When execution starts from the next program, it starts from here. So it's important to copy these first few bytes. The image doesn't go to the end of the execution, just shows the first few steps of execution because it's— Takes a while. Yeah. The other replicators will eventually manage to copy the, this part over to this part. And so you get an actual copy of the full replicator. But with this one, there is, as I said, this problem that if you have a zero byte in between the 2 halves of the self-replicator, then it doesn't actually copy itself. Now that's not as much of a problem as it is for the other replicator. Because there is a difference between I need to have a null byte myself and the program I copy myself to has a null byte. If you have even just one self-replicator that doesn't have a null byte, then it will succeed at copying itself over. And so it will survive way more easily. But the problem is if the, let's call it food or rest of the soup is full of zeros, then this self-replicator cannot copy itself at all. That sometimes appears in, after a longer evolution is, or this one specifically is handcrafted, but there are other replicators that sometimes appear that are resistant to zeros, both in the destination and in the source. Usually they contain like nested loops. So it's a bit harder to explain here what is going on. So this is still the same replicator that we had in the previous example, like the more successful replicator, but around it there is an additional loop. And what this loop does is just— suppose I exit from the first loop because I happen to have found a zero byte, then let's just take that zero byte, decrement it or increment it, whatever. Now it's not a zero byte anymore. So let's jump back and get back into the previous loop. So this is one of the reasons why I was saying that it's hard to define what a self-replicator is, because at the end of the execution of this program, neither of the halves of the self-replicator is the same as the original program, because this null byte is not a null byte anymore in the original program, and in the new program is also not a null byte. But still, you would say that this program copied itself, right?

Because the important parts And this program is also very robust to noise, by the way. So, in noisy environments, this is particularly good because even if you can copy yourself perfectly, you can have a random mutation that could destroy other programs. But for a random mutation to destroy you, it has to be in the few operations that you have in this case on the left.

Yeah.

We also observed that in other languages that there's this typical behavior of set propagators that that if there's nothing else as a pressure, they're trying to become shorter and shorter. In this specific one, it's tricky because actually it's a specular self-replicator for the nuances that this Brainfuck variant has. But in other languages, that would be relevant. And we've also observed that different kinds of self-replicators that use different operations take over in other languages that, for example, are CPU instruction-based. But I want to stress out that the main focus of this paper was to focus from the how you get out of the pre-life period. So we didn't explore nearly as much as what happens after the first replicator arrives. This is something that we very much care about, but for future work.

Yes, we just have some observations of like more resilient self-replicators appearing over time.

But so there were at least 3 classes.

Tell me if I missed any. There's an initial one that— maybe I shouldn't order them, but there's working from least to most resilient. There is one that can copy itself, but if it hits a zero where it's trying to copy 2, it fails.

Mm-hmm.

There's one that is highly susceptible to mutations in its own code. And can fail as a result of that. And then there's the robust kind that can both write over the zeros and is much more tolerant of mutations in its own space as well.

Yes.

And we know that all these things can arise. How often do multiple of these things arise? I guess you just said there's more future work to do to play out from the emergence of these things to the long-term dynamics, but How often do we see multiple different things battling it out versus is this like one replicator's lonely struggle for survival or are they doing battle with each other frequently enough?

It's a bit complicated to evaluate, especially in the multidimensional setups. Interactions are a lot more sparse and since interactions are a lot more local, there is space for niches to appear and going into different directions in different places. And then they start fighting with each other at the border and trying to take over other self-replicators. But even in this like very dynamic setup where everything can interact with everything else, you can see that the structure is similar, but not that many self-replicators are identical because the effect of mutations gives you this, right? Where now everything is equal, but then one self-replicator mutates a little bit, and then if that one starts getting copied a little more, then you have replicators that fight with each other. Even if one self-replicator became a lot more efficient than the other one, then it probably would just win. But in this setup, if you have multiple self-replicators that are similarly efficient— for example, if you look at these 2, they are identical except like some of these random unused bytes here are different. So their efficiency is the same, but you will see a bunch of both kind— of both kinds. Yeah.

Interesting. That, and that's, if I understand correctly there, it's like you can have the same mechanism, but you can have different like filler data. This is almost like junk DNA. I don't know that there is such a thing as junk DNA, but very similar. If you believe in the notion of junk DNA, this is the junk DNA of your Yes. So it's amazing how many concepts like that we are seeing in just this very small thing.

Right.

We've got different mechanisms for self-replication. We've got mutations that can be deleterious or that can be survivable depending on the mechanism. We've got some notion of pollution with the zeros popping up that can become so common that they make it difficult for the self-replicators to succeed. We have the junk DNA, we have competition for niches and fighting at the border. It's really an amazing— this is what kind of just captivated my imagination looking through these results, because it's like, that is such a small universe. in which to find so many of these advanced concepts. Is there any other kind of recognizable concept from the study of macroevolution that you have observed that we haven't touched on yet?

Actually, yes, at least one. So we have some interesting cases of something similar to symbiosis. This is in a different language, and I don't think I should go into too many details because It would take a while. But the idea is that we have like a different language, which is a variation of Fort, in which there is always one byte self-replicator that just copies that, just that one byte. So if a program starts with that byte, then that byte copies itself over. Maybe it's not some way, this is not the correct term. I don't know, I'm not a biologist. But what happens is that there is evolution that happens that starts to exploit that 1-byte self-replicator until you get self-replicator that is able to copy the entire string that uses that 1-byte self-replicator. And because of this, because of the existence of this 1-byte self-replicator and using this 1-byte self-replicator, evolution is a lot faster than in a lot of other fault variants where you don't have this. So it's like enabling further evolution just by being there. And it's not a parasitic kind of relationship because the self-replicator still survives, but it's a sort of cooperative.

But some of our colleagues have also observed some early examples of parasitic behaviors in 2D environments. And this is very consistent with other research done in artificial life, where if you got 2D environments, you can have these waves of parasites that try to destroy some self-replicators that replicate in the other direction.

Yeah, it's really remarkable how basically a lot of things we see from the world and like biology appear in this completely different setup. In fact, for me, it would be really interesting to manage to prove that this kind of evolution that we observe in our world is not something that is special to our world, but just happens naturally. It's, you know, it's a very natural thing to occur. And it's not that the world is special and that had a lot of coincidences that caused evolution to go on the way it did. It's just even in simple setups or whatever, it would be really nice to prove that we managed to have this like emergence of life, self-replication, whatever way you want to call it.

That is maybe a good transition to some big picture philosophical questions that I have for you. One thing is just, it seems like most things go extinct. Should that be my understanding? You said even when you put in a self-replicator purposefully, there's still only like a 23% chance, I think you said, of it taking over. Do I understand correctly that the other 77% of the time— It's going extinct.

Yes.

And if I think of this through, like, a Robin Hanson Great Filter sort of lens, this sort of work would make me update my worldview away from the idea that the origin of life was the super hard step, or that the formation of self-replicators, maybe to be a little bit more generic in my, or abstract in my terminology. And it makes me more think that a lot of things are probably popping up in the universe and fizzling out before they get to the point where they are putting out the sort of radio signals that we're putting out, or certainly like starting to colonize space. Does that seem like a justified interpretation?

I mean, my way to think about that is yes, once you have a self-replicator, you still have only 1 chance in 5 of actually doing something. But the chances of having a self-replicator, in my opinion, are like even lower. It's just all steps are hard. And maybe if what we think of what's the simplest step is, okay, there, there is a self-replicator, like now it just needs to take over. That's already hard. Then maybe just on the other hand, it's just that the, all the other steps were even harder.

But I, I want to be more of an optimist right now. Even in our experiments, even if it's only 23%, the chance that one single self-replicator takes over, we observe a much higher than that chance that in 16,000 epochs, self-propagators arise. So I think it's about 40% in the first 16,000 epochs, and you can go for much longer. So even if the individual one might not be the trigger for this state transition, it's still quite likely to happen.

Yes.

The real question that I think matters more is how close is this to what you observe in the real world if you're interested in the biological implications of this. And there are many things that this thing doesn't have yet. For example, a clear way to accumulate complexity, because as you pointed out, we only have 64 bytes per string, and it's very difficult to go beyond that number because there's no easy way for strings to connect to each other in a stable way. And biology has that all the time. You can accumulate more and more complex strings. So we would want to explore directions which are similar to that as well. If you want to try to assess how likely it is for life in the biological world to happen. But there's still so many similarities that I think it's an interesting starting point.

Yeah, you can see here there is representation of how often you actually get a self-replicator. And within the first 16,000 steps, it happens like 40 to 60% of the time, which is quite remarkable because it's still order appearing from chaos, right? So you would think that's less likely. This again, this is not the probability of self-replicator appearing. It's the probability of self-replicator taking over. This is even if the first one doesn't succeed, then someone else will afterwards, the reasoning that you can have.

So NA here means that it didn't happen.

It didn't happen. And you can see that at, at any point it didn't happen like 40, 60, 65% of the time. In this case, there's just so much noise that nothing could possibly happen. But in this setup, this is the setup in which you observe a self-replicator dying in 80, like 77% of cases. But even if it does die, then, you know, something else takes over anyway afterwards.

That bit about the frequency of mutation is really interesting too. When you get up to that 1% mutation rate, then it becomes very hard for anything to persist. I was thinking that when I look at so much of The, at least the popular science discourse around life out somewhere in space sort of says, oh, this planet might have water and therefore it might be conducive to life. And unsaid there is, as we know it.

Mm-hmm.

And this work also makes me think that there's probably just a lot more space for life that, who knows, right? It could be extremely Weird, but I no longer feel like, oh, if you don't have water, you couldn't have life on that planet. Now I'm thinking you could have a gas giant that could have life in it, right? Who knows, right? It just seems like there's— if you can see this much complexity in this simple of a system, then I don't really have any reason to be prejudiced against what's going on at some depth in Jupiter. Maybe I would have some reason to think that, like, planets need an atmosphere or they need some sort of shield because they can't be just totally bombarded with crazy radiation that's going to cause too much noise in the system. But I feel like even at the level of substrate, I have to be pretty open-minded now in view of these results. You're nodding. Sounds like you— it looks like you interpret that similarly.

Yeah. I mean, even with very high levels of radiation, you can still have life happening. It's just that it's going to be probably smaller. And even that would be interesting to observe. One The extra thing that this shows, at least I think is an interesting direction, is sure, you can think about what can happen in our universe, but we are not restricted to our universe when we do simulations. So what we are showing here is a completely different computational substrate, which has similarities to what biology can show, but we can do many different things. And we can really be creative here for trying to figure out how to create interesting and complex life in those directions.

Yeah.

And ideally, ultimately intelligence as well.

Perfect segue to one of my big questions, which is what should we take away from this for the purposes of understanding what the future of life and artificial intelligence turned to life might look like on our planet? My gut says, like, again, I should just be very humble in terms of what my expectations are. If you can create something that looks this much like life in this tiny little space, then I think we're headed for a world in which some form of AI begins to self-replicate. We obviously have a very large digital ecosystem, computational environment for possible things to self-replicate through. And here, obviously, there's not intelligence involved. But if you add on intelligence and these sort of interesting primitives and put them in a big open space like we seem to be doing right now with open source models, this significantly increases my sense that whether it's Llama 3 or 3.1 or Llama 4 or whatever, that some version of this sort of thing pops up that has the recipe for self-replication, possibly a mix of smart and dumb. But it just seems like the barrier to getting there is not nearly as high as people might be inclined to intuitively think.

So I think one thing that is important to keep in mind is that here we are getting this spontaneous emergence of self-replication, but that's through a lot of interactions. The timescale at the end of the execution when we have the first self-replicators, we have seen 50 billion instructions executed. We run, I believe, 3 million paths of programs. And yes, you can see that running in 1 minute. But imagine if open source AI models gain like a chance of the capability to self-replicate. And even if that were to happen, it would probably be quite unlikely for it to happen at the frequency. And at a speed where we have timescales that are compatible with what we are observing here. Which of course doesn't mean that it's not a risk we shouldn't take into account. In particular, what I would be more concerned about is the risk of subsequent modifications, like imperfect self-replication, just taking things farther away from its original intended limitations or whatever it is that That were properties. But I don't think that the insights we see here are particularly applicable to larger scale models that would execute at much slower speeds and with restrictions. And we still also, the space that we let them execute in is significantly more guarded in some sense. Here we have a world in which you can just copy yourself over on any other thing. And I do not know this, but I would expect that for even for open source models, it would be quite hard for them to be able to just take over any random computer for the foreseeable future. If that were to happen, I would not be— I would be concerned by other things first and then by the subsequent ability of further evolution happening. Like, I think we would have other problems.

Can you put a little more detail on those other problems too?

I think, you know, that would imply that all of our computer infrastructure is utterly broken and security infrastructure and like any malicious actor.

Oh, that's true though, right?

Yeah. That's why I'm not like that confident that it will not happen.

It's not all broken, but it's like a lot broken.

Yes. That's fair.

So yeah, you guys at Google live in a world of the best Computer security infrastructure in the world by perhaps a significant margin, and the rest of us out here are barely updating our passwords. So I would say if the problem hinges on or if the upshot would be that that would mean our security is bad, look no further than just how many different branches of the U.S. government have been hacked at the highest levels. Like the White House has been hacked, Congress has been hacked. It's all been pretty. My working assumption is that we're all pretty well and thoroughly hacked.

That could.

I think the main biggest takeaway is not to underestimate the effect of subsequent modification, like over time. Like, I don't think that open source models copying themselves over are even just the biggest risk of open source models themselves. But my area of work is not AI risks, and I don't think I'm qualified to make an actual statement on what I should be working with.

There's a lot of different vectors of attack and a lot of different ways that things could get pretty crazy. This just to me feels like one that people haven't considered nearly enough. And I'm not sure I couldn't even begin to put a probability estimate on it, but whatever my implicit, non-explicitly calculated sense previously was, this does pull it in for me because I'm just like, man, There's just a lot going on here. The possibility of a sort of ecology rising in such a limited space. Then I just look at the complexity and the vastness of the digital ecosystem that we have today, and I'm like, there's a lot of greenfield out there. Even if Google's systems are super secure, plenty are not. And it seems like there is at least some non-trivial possibility that you could have a digital ecology start to come online, or an AI ecology start to come online. And maybe that's even good. One of the things I always think about with life in general is just the fact that it is homeostatic and that there's these sort of many buffers that exist within any given system. Those things may be really good for us long term. We might be in a better situation if we have a more ecological model of open source AI versus a sterile planet that suddenly has something super powerful pop onto the scene and have a much larger chance of going to dominance. I have no idea, really. It's very speculative, but this work has at least got me reconsidering with a more open mind, which I think I might be totally going in the wrong direction, but it still feels like we're in this phase of time where These sorts of thought provokers, for those of us that do sit around trying to think about what's going to happen with AI, they're valuable regardless because we need to be more open-minded. We need to entertain the weird, I think, a lot more than we're naturally given to. So for me, this is a great spur in that direction.

I mean, this goes back to the point that Luca was trying to say. Yes, we shouldn't underestimate the cascading effect of interactions. So if you have a replicator with a certain amount of mutation and then it mutates many times, the results can be unpredictable. But at the same time, we need to highlight that first of all, if you're just copying something which is equal to something else, so if you have a perfect copy, nothing interesting would come up. And only when you've got mutations and variants, then something interesting can happen. And this is very much an open problem that the field of artificial life in particular, but also deep learning and evolutionary strategies has been studying for a long time. We still don't know how to create unbounded complexification of behaviors. Right now, one very open question, even in this very small experiment that we have done, is would this have a limit in the amount of complexity that it can show? Or how could we make it not have a limit in the amount of complexity? Most experiments that I'm aware of did eventually plateau. at a certain level where they couldn't escape. So before ringing alarm bells and saying this might go all the way to a high level of intelligence, or like copying an LLM many times with variation would go to a high level of intelligence, we would need to have a much stronger belief and evidence that we can accumulate complexity in this way.

Yes. Although we are making something that is extremely complicated and capable in a lab and putting it out there in a way that it can potentially interact with other things. Right. The way I'm thinking about the open source AIs now, and I don't mean this to be in a prejudicial way of good or bad, because I'm increasingly more open-minded to this notion that an AI ecology, should such a thing exist, might be protective or might be good in its own way. But this seems like we're certainly giving some future AI life forms, like, a major shortcut by building their brains for them and then releasing them into the wild. It seems like we've done one of the really hard parts of creating this future in a way that was not random, right? Obviously, we've engineered these systems and figured out how to train these things. And so my kind of mental model, I wonder if you guys could attempt some simulations like this. I don't have nearly enough competence in this domain to know exactly what this would look like, but I wonder if there would be a way to combine something that has intelligence with something that's really simple or primitive in the self-replication sense. Is there a way to look at— people are doing these sorts of things with, can an LLM, can GPT-4, can Gemini or whatever find day zero exploits in software? And I'm trying to close in on something that's, man, you have these pretty simple self-replicators that seem to work here. And then separately, we have these language models that are able to code and able to assess code and debug code to varying degrees. Of course, is there a sort of symbiosis type model there that could make sense? Is there something in that mix that could be modelled in a closed environment? that could shed light on what the future of open source, just as it evolves through time, might look like? I don't have the answer to that question, but I'd love it if you did.

No.

Well, we don't have an answer, of course, but I think that the closer we get to high level of intelligence, the more relevant trying to have a setup for understanding the risk becomes and how we can control it. In our simple experiments, there's little to no risk that they can take over, especially because they cannot do more than modify 64 bytes. But if you get to more and more levels of complexity, such as an LLM, then you really need to think about what you're doing. And I think that this should be taken quite seriously.

Well, I have maybe an interesting point of view here. So first of all, if defining life is hard, defining intelligence is harder. I will not claim to have a definition of intelligence. When you say these things about can you find zero-day exploits with LLMs and things like that, that makes me think, I have a bit of a background in Computer Science Olympiads. And for that matter, there are actually sponsored experiments you've probably heard of. There is this AI-MO. I don't know if that's something you're familiar with.

I'm not immediately sure.

Okay. So this is, I think, extremely fascinating experiment. I am aware I'm digressing significantly from the main topic of the paper at this point. There is an extremely fascinating experiment where there is this International Mathematical Olympiads, and there are these efforts that try to get AI to just solve half of the problems in the Math Olympiad. We are not there yet. Like, I don't know if you want to classify ability of doing mathematical Olympiads as intelligence. My opinion is it's correlated.

Certainly some form of intelligence.

Exactly. You can be intelligent if you, even if you cannot do Math Olympiads, I would claim. But if you think of this kind of questions, then I'm not even sure if we can actually say that the system is intelligent now. So when you say, okay, start mixing in self-replicators and intelligence, I don't think I even know how something like that would look like, if that makes sense.

Yeah, that's fair enough. I tend to use a pretty functional definition of intelligence. One that I've been using in very different types of conversations where I'm talking to business owners about using language models to automate tasks is that intelligence is the ability to perform useful work without fully precise instructions. So your number recognition is a great toy example that I tend to start with, the MNIST dataset of you can recognise these numbers. Nobody knows how to write fully explicit code that can recognise numbers, but we can definitely train an AI to do it. That sort of ability to succeed when we don't have the full algorithm, Is, I think, an interesting functional definition. I'm not sure how much the definition really matters. I think what would matter maybe most in the context of the work that you guys are doing is, is there a way that you could mix in a language model or any sort of model? I think most about language models into a simulation like this in a way that makes the self-replicator much more likely to take over.

In practice, no. For a very simple reason that you are not going to be able to run a million iterations of a language model in any reasonable amount of time.

Yeah, you need real compute for that.

Probably you could do that experiment if you start using half of the computers on the planet, but arguably there are better things to do with half of the computers on the planet than just, you know, seeing if you get self-replicators slightly faster.

Yeah, although it depends on how much you think the AIs can help succeed, right? If they're really effective, then you might not need that much compute. We imagine ourselves taking over sterile planets, not through random brute force search of the ways to do it, but by actually being smart enough to show up and get it right the first time. And I don't think Llama 3.1 is probably there today, but you could imagine a level of effectiveness that might greatly reduce the compute requirements, right?

Yeah, I think it, it makes sense to think that a variant of our experiments with self-propagation can be used to find better ways to find, let's say, evolved agents. This is part of what we want to explore, but of course, like, we are starting from literal strings of code and getting to something as complex as an LLM will take quite a while. And I think that we should take account for the risks at every level and Increasingly so the more complex our systems become. But Luca's point is very well taken that the bigger the model, the more costly this would be. So to have something like this work, we would need to have much more efficient models to start with.

Okay, cool. This is a fascinating discussion. I love the work. I think I should probably let you guys get back to it. Anything else that you wanted to touch on that we didn't get to before we break?

I think we discussed pretty much all the aspects of the paper quite in detail.

Yeah.

Fascinating work. Thought-provoking, maybe more thought-provoking for me than you intend it to be for the average reader. But that's the business that I'm in, is trying to go grab these random things and use them to at least stimulate some creative thoughts about where the future might be headed. And you've certainly helped me do that. So I really appreciate it. Appreciate the time today. Ettore Randazzo and Luca Versari from Google's Paradigms of Intelligence team, thank you both for being part of The Cognitive Revolution.

Thank you.

Thank you very much for having us.

It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.