Stephen Wolfram on Generative AI Space and the Mental Imagery of Alien Minds

Okay. Hello everyone. Well, let's see. Today I wanted to tell you about a project that I've been doing recently. I work on all kinds of unusual projects. This one, during its development was known as the CATS project for reasons that you'll see. But I recently posted something about it you can see here. As I say, I work on all sorts of different things. This one has an image of a cat, and if I click on this, this is the thing I'm going to be talking about today. The topic is generative AI space and the mental imagery of alien minds. So, so first sort of topic is AIs and alien minds. How do alien minds perceive the world? It's an old, often debated question in philosophy and now turns out to be a question that rises to prominence in connection with the concept of the ruliad that's emerged from our Wolfram Physics project. I myself have wondered about alien minds for a long time and tried all sorts of ways to imagine what it might be like to see things from their point of view. But in the past, I've never really had a way to build my intuition about that. That is, until now. So what's changed? Well, it's AI. Because in AI we finally have an accessible form of alien mind. I mean, we typically go to a lot of trouble to train our AIs to produce results that are like we humans would do. But what if we take a human aligned AI and modify it? Well, we get something that's in effect an alien AI. An AI aligned not with us humans, but with an alien mind. So how can we see what such an alien AI or alien mind is thinking? Well, a convenient way is to try to capture its mental imagery, the image it forms in its mind's eye. So let's say we use a typical generative AI to go from a description in human language to like a cat in a party hat to a generated image. This is why this is the cat's piece. It's. It's exactly the kind of image we'd expect, which isn't surprising because it comes from a generative AI that's trained to do as we would. But now let's imagine taking the neural net that implements that generative AI and modifying its inside, say by, let's say by resetting weights that appear in its neural net. By doing this, we're in effect going from a human aligned neural net to some kind of alien one. But this alien net will still produce some kind of image, because that's what a neural net like this does. But what Will that image be? Well, in effect, it's showing us the mental imagery of the alien mind associated with the modified neural net. What does it actually look like? Here's a sequence obtained by modifying the neural nets that we're using before, in effect, making it progressively more alien and at the beginning, still very recognizable as a picture of a cat in a party hat. But it soon becomes more and more alien. The mental image, in effect, diverges further from the human one until it no longer looks like a cat, in the end, looks, at least to us, basically completely random. There are many details of how this works. We'll talk about that later. What's important is that by studying the effects of changing the neural net, we now have a systematic kind of experimental platform for probing at least one kind of alien mind. We can think of what we're doing as kind of artificial neuroscience, proving not actual human brains, but neural net analogs of them. And we'll see many analogs, many parallels to neuroscience experiments. For example, we'll often be knocking out particular parts of our neural net brain, a little like how injuries like strokes can knock out parts of a human brain. But we know that when a human brain suffers a stroke, this can lead to phenomena like hemispatial neglect, in which stroke victim asked to draw a clock, will end up just drawing one side of the clock. A little like the pictures of cats degrade when parts of the neural net brain are knocked out. Of course, there are many differences between real brains and artificial neural nets. But most of the core phenomena that we'll be seeing seem robust and fundamental enough that we can expect them to span very different kinds of brains, human, artificial, and alien. And the result is that we can begin to build up intuition about what the worlds of different and alien minds can be like. Okay, so let's talk about this process of generating images with AIs. How does an AI manage to create a picture, say, of a cat in a party hat? Well, the AI has to be trained on what makes a reasonable picture and how to determine what a picture is of. Then, in some sense, what the AI does is to start generating reasonable pictures at random, in effect, continually checking what the picture it's generating seems to be of and then tweaking it to guide it towards being a picture of what one wants it to be a picture of. So what counts as a reasonable picture? If one looks at billions of pictures, say on the web, there are lots of regularities. For example, the pixels aren't random. Nearby ones are usually highly correlated. If there's A face, it's usually more or less symmetrical. It's more common to have blue at the top of a picture, green at the bottom, and so on. And the important technological point is that it turns out to be possible to use a neural network to capture regularities in images and to generate random images that exhibit them. So, for example, here are examples of images that are generated in that way. And the idea is that these images, while each is random in its specifics, will in general follow the statistics of the billions of images from the web on which the neural net has been trained. So we'll talk more about images like these later. But for now, suffice it to say that while some may look like abstract patterns, others seem to contain things like landscapes and human forms and so on. And what's notable is that none of them just look like random arrays of pixels. They all show some kind of structure. And yes, given that they've been trained on pictures from the web, it's not too surprising that the structure sometimes includes things like human forms. But, okay, let's say we specifically want a picture of a cat in a party hat from all of the almost infinitely large number of possible well structured random images we might generate. How do we get one that's of a cat in a party hat? Well, the first question is, how would we know if we've succeeded? As humans, we can just look and see what our images of. But it turns out we can also train a neural net to do that. And here are some examples. It doesn't always get it exactly right. How is the neural net trained? The basic idea is to take billions of images, say, from the web, for which corresponding captions have been provided. Then one progressively tweaks the parameters of the neural net to make it reproduce these captions when it's fed the corresponding images. But the critical point is that the neural net turns out to do more. It also successfully produces reasonable captions for images it's never seen before. 1 what does reasonable mean? Operationally, it means captions that are similar to what we humans might assign. And yes, it's far from obvious that a computationally constructed neural net will behave at all like us humans. And the fact that it does so is presumably telling us fundamental things about how human brains work. But for now, what's important is that we can use this captioning capability to progressively guide images we produce towards what we want. So we start from pure randomness, Then we try to structure the randomness to make a reasonable picture. But at every step, we see, in effect, what the caption would be. And we try to go in a direction that leads towards a picture with the caption we want, or in other words, progressively try to get a picture that's of what we want. So the way this is set up in practice, one starts from an array of random pixels, then iteratively forms the picture one wants. Different initial arrays lead to different final pictures, though if everything works correctly, the final pictures will all be of what one asked for. In this case, a cat in a party hat. And here are some examples. And yes, there are a few glitches in all of this. So we don't know how mental images are formed in human brains, but it seems conceivable that process is not too different and that in effect, we're trying to conjure up a reasonable image. And as we do that, we're continually checking if it's aligned with what we want. So that, for example, if our checking process is impaired, we can end up with a different image, as in hemispatial neglect. Let me perhaps just show you for a second here what it looks like in practice to generate one of these images. So let's go ahead and pull up a wolf notebook here and let me just go. And the way that my writings are always set up, you can always click any image in there and that will generate Wolfram language computational language code that you can copy in here. Now this particular piece has the complexity that to generate images is very computation intensive and you basically need a GPU in order to be able to do this well, but let's try doing it for this particular case. So this is saying generate, use the stable diffusion synthesize resource function to generate a picture of a cat in a party hat. And you see the picture sort of coming in there and ba boom, there is a cat. So that's roughly what it looks like to actually do this in practice with Wolfram language. And what's important now is that because we're able to do it with Wolfram language, we're able to do all sorts of experiments where that sort of just wouldn't have been possible without that kind of capability. Well, let's come back and talk about the notion of what I'm calling interconcept space. So the fact that everything can ultimately be represented in terms of digital data is foundational to the whole computational paradigm. But what the effectiveness of neural nets relies on is a slightly different idea that it's useful to treat at least many kinds of things as being characterized by arrays of real numbers. In the end One might extract from a neural net that's giving captions to images, the word cat. But inside the neural net, it'll operate with arrays of numbers that correspond in some fairly abstract way to the image you've given and the textual caption it'll finally produce. And in general, neural nets can typically be thought of as associating feature vectors with things, whether those things are images, text or anything else. But whereas words like cat and dog are discrete, the feature vectors associated with them just contain collections of real numbers. And this means that we can think of a whole space of possibilities with cat and dog just corresponding to two specific points. So what's out there in that space of possibilities for the feature vectors we typically deal with? In practice, the space is many thousand dimensional, but we can, for example, look at the sort of nominally straight line from the dog point to the cat point in the space and even generate images of what comes between. And so we see there the sort of cat, dog combo as we go towards the cat point here. And yes, if we want to, we can even keep going beyond cat, and pretty soon things start becoming pretty weird. Well, we can also do things like look at the line from a plane to a cat, and yep, there's pretty weird stuff that goes on in between. It's kind of like a wings, goes to hat, goes to ears kind of sequence. So what about elsewhere? For example, what happens around our standard cat in a party hat image? With the particular setup we're using, there's a 2,304 dimensional space of possibilities. But as an example, we can look at what we get on a particular two dimensional plane through the standard cat point. And our standard cat is right there in the middle. But as we move away from the standard cat point, progressively weirder things happen. For a while there are recognizable, if perhaps somewhat demonic, cats to be seen, but soon there isn't much catness in evidence. Sometimes over here, for example, hats do remain. We might characterize that as a kind of all hat, no cat situation, reminiscent of the kind of Texan all hat, no cattle kind of story. How about if we pick other planes through the standard cat point, all sorts of images appear, but the fundamental story is always the same. There's a kind of cat island, beyond which there are weird and only vaguely cat related images, encircled by an ocean of what seem like purely abstract patterns with no obvious cat connection. And in general, the picture that emerges is that in the immense space of possible statistically reasonable images, there are islands dotted around that correspond to linguistically describable concepts like cats and party hats. The islands normally seem to be roughly spherical in the sense that they extend about the same nominal distance in every direction. But relative to the whole space, each island is absolutely tiny. Something like perhaps a fraction 2 to the minus 2000 or 10 to the minus 600 of the volume of the whole space. And between these islands lie huge expanses of what we might call interconcept space and sort of analogy to interstellar space. So what's out there in interconcept space? It's full of images that are statistically reasonable based on images we humans have put on the web, et cetera, but aren't of things we humans have come up with words for. It's as if in developing our civilization and our human language, we've colonized only certain small islands in the space of all possible concepts, leaving vast amounts of interconcept space unexplored. What's out there is pretty weird, sometimes a bit disturbing. Here's what we see. If we zoom in on the same kind of randomly chosen plane around Cat island that we did before. And let's see, I have a bunch more examples here of what it looks like to zoom in around different cat islands and you see all sorts of strange, as I say, somewhat demonic looking cat like images. More cats, sometimes a bit troubling. So what are all these things? In a sense, words fail us. They're things kind of on the shores of interconcept space, where human experience has not yet taken us and for which human language has not been developed. What if we venture further out into interconcept space and for example, just sample points in space at random. It's just like we already saw above. We'll get images that are somehow statistically typical of what we humans have put on the web and so on and on which our AI was trained. Here are a few more examples. And yes, we can pick out at least two basic classes of images. Ones that seem like sort of pure abstract textures, and ones that seem representational and remind us of real world scenes from human experience. There's intermediate cases like textures with structures that seem like they might represent something, and representational seeming images where we just can't place what they might be representing. But when we do see recognizable real world inspired images, they're a curious reflection of the concepts and general imagery that we humans find interesting enough to put on the web. We're not dealing here with some kind of arbitrary interconcept space. We're dealing with human aligned interconcept Space that's in a sense anchored to human concepts, but extends between and around them. And yes, viewed in these terms, it becomes quite unsurprising that an interconcept space, the interconcept space we're sampling, there are so many images that remind us of human forms and common human situations. But just what were the images that the AI saw from which it formed this model of interconcept space? There are a few billion of them foraged from the web. And like things on the web in general, it's a very motley collection. Here's a kind of random example of them. Some of them can be thought of as sort of capturing life as it is, but many of them are more aspirational, coming from staged, not promotionally oriented photography. And yes, there are lots of kind of net, a Porter style, clothing without heads, images. There are also lots of kinds of images of things like food and so on. But somehow when we sample randomly and interconcept space, it's the human forms that most distinctively stand out, conceivably because things are not particularly consistent in their structure. But human forms always have a certain consistency of head, body, arms and so on. Structure. It's notable though, that even the most real world images we find by randomly sampling into concept space seem to typically be painterly and artistic rather than photorealistic and photographic. It's kind of a different story. Close to concept points like on Cat island, there are more photographic forms that are common. Though as we go away from the actual concept point, there's a tendency towards either a rather toy like appearance or something more like an illustration. By the way, even the most photographic images the AI generates won't be anything that comes directly from the training set, because as we'll talk about later, the AI is not set up to directly store images. Instead, its training process in effect grinds up images to extract their statistical properties. And while statistical features of the original images will show up in what the AI generates, any detailed arrangement of pixels in them is overwhelmingly unlikely to do so. But okay, what happens if we start not at a describable concept like a cat in a party hat, but just at a random point in interconcept space? Here are some of the kinds of things we see and we can show a few more examples here. And well, let's. These images often seem to be a bit more diverse than the kinds of images around known concept points like our cat point. And occasionally there'll be a sort of a flash of something representationally familiar, perhaps like a human form that'll show up, but most of the time we won't be able to say what these images are of. There are things that are somehow statistically like what we've seen, but they're not things that are familiar enough that we've at least so far developed a way to describe them, say, with words. Well, let's, let's go on here. Let's talk a little bit more about these kind of images from interconcept space. And there's something kind of strangely familiar, yet unfamiliar to many of the images in interconcept space. It's fairly common to see pictures that, for example, seem like they're of people, but they're not quite right. And for us humans, being particularly attuned to faces, it's the faces that tend to seem the most wrong, even though the other parts are wrong as well. Perhaps in commentary on our nature as a social species, or maybe it's because we're a social media species, there's a great tendency to see pairs or larger groups of people. There's also a strange preponderance of kind of torso only pictures, presumably the result of kind of fashion shots in the training data. And yes, with some rather wild but interesting fashion statements. Maybe that's the shirt I need. Well, people are by far the most common identifiable elements, but one does sometimes see other kinds of things as well. And there are also things like sort of landscape, like scenes. And some of these look fairly photographically literal, but others build up the impression of landscapes from more abstract elements like here. If you look in detail, they really are not sort of photographically trees or anything like that. Occasionally there are also kind of cityscape like pictures and still more rarely kind of indoor like scenes. Then there are pictures that look like they're kind of exteriors of some kind. And it's pretty common to see pictures that are built up from sort of lines or dots or otherwise kind of impressionistically formed. And then there are lots of images that seem like they're trying to be of something, but it's not at all clear what the thing is and whether indeed it's something we humans would recognize, or whether it's instead something that's somehow fundamentally alien. Here are a few examples of that. It's also quite common to see what look more like pure patterns that don't really seem like they're trying to. To be things, but more come across as kind of decorative textures. But probably the single most common type of images are somewhat uniform textures formed by repeating various simple elements, though usually with kind of dislocations of various kinds. So across interconcept space, there's tremendous variety to the images we see. Many have a certain artistic quality to them and a feeling that they're some kind of mindful interpretation of a perhaps mundane thing in the world, or a simple, essentially mathematical pattern. And to some extent, the mind involved is a collective version of our human one, reflected in a neural net that has experienced some of the many images we humans have put on the web. But in some ways, the mind is also a more alien one, formed from the computational structure of the neural net with its particular features and no doubt in some ways computationally irreducible behavior. And indeed, there are some motifs that show up repeatedly that are presumably reflections of the underlying structure of the neural net. The kind of granulated appearance with alternation between light and dark, for example, is presumably a consequence of the dynamics of the convolutional parts of the neural net and analogous to the results of what amounts to iterated blurring and sharpening with a certain effective pixel scale. Reminiscent, for example, if people still remember that, of video feedback, of what happens when you point a video camera at a video screen. And here's an example of that kind of thing made with very simple repeated blurring and sharpening image processing. We could actually do that. We could just run that just for the sake of showing how one runs something here. Let's just get that. I did a click to copy there. Let's just copy it. Ok, this is just showing. Oh wow, it's giving. There's the key part, the sharpener blur there. And we can just go ahead and run this thing. And there we have our cat made with progressive image processing. Okay, well, we can think of what we've done so far as exploring what a mind trained from human like experiences can, can imagine by generalizing from those experiences. But what might a different kind of mind imagine? It's a very rough approximation. We can think of just taking the trained mind we've created and explicitly modifying it, then seeing what it now imagines. Or more specifically, we can take the neural net we've been using and start making changes to it and seeing what effect that has on the images it produces. So later on, we'll discuss a bit the details of how the network is set up. But suffice it to say here that it involves 391 Distinct internal modules involving altogether nearly a billion numerical weights. When the network is trained, those numerical weights are carefully tuned to achieve the results we want what if we just change them? We'll still normally get a network that can generate images, but in some sense it'll be thinking differently. So potentially the images will be different. So as a very coarse first experiment, reminiscent of many that are done in biology, let's just knock out each successive module in turn of those 391 modules, setting all its weights to zero. So if we ask the resulting network to generate a picture of a cat in a party hat, here's what we'll now get. Well, these are a bit small here. Let me see if I can get you a bigger version of that that we can see more easily. Let me think how to do that. Maybe I have an idea here. Let's see, let's see, let's see, Let me see. I might have an idea. Yeah, let me see whether I can make this work. Yeah, this might work. Here are some bigger cats. Maybe I can make them even a little bigger than that. Okay, these are a bit pixelated, but you can kind of see, get some idea here of what happens when we've kind of knocked out different elements of the, of the neural net here. We're seeing different kinds of sort of alienly imagined cat like things. Okay, so let's look at these images in a bit more detail. Let's see, we can look at. Oh wow. Some of these are missing from my collection here. Well, in quite a few cases, kind of zeroing out a single module doesn't make that much difference. For example, it might basically only change kind of the facial expression of the cat. Let's see whether we can get this image here. Ah, maybe we got that. Hold on. Yes, there we go. And yes, Or perhaps it can more fundamentally change the cat and its hat. It can change the configuration and position of the cat. And yes, some of those cats are not anatomically correct. Zeroing out other modules can change, in effect the rendering of the cat. But in other cases things get much more mixed up and difficult for us to parse. Sometimes there's clearly a cat there, but its presentation is at best odd. And sometimes we get images that have definite structure but don't seem to have anything to do with cats. And then basically there are cases where we just get what amounts to noise, albeit with things superimposed. But much like in neurophysiology, there are some modules, like the very first and very last ones in our original list, where zeroing them out basically makes the system not work at all and just generate pure random noise. As we'll talk about later, the whole neural Net we're using has a fairly complex internal structure, for example, with a few fundamentally different kinds of modules. But we can make a picture that shows an example of what happens when one zeros out modules at different places in the network. And here's that picture. And we see that for the most part there's no obvious correlation between where in the network the module is and what effect zeroing out will have. The data comes in at the left hand side here, and that's kind of where the sort of noise is generated. And the name of what we're trying to get is put in. And then things sort of percolate through this neural net until eventually the final cat emerges on the right hand side here. Okay, well, so far we've looked at what happens if we just zero out a single module at a time. Let's look at some examples of what happens if we randomly. If one zeros out successively more modules, we could kind of call this a HAL experiment. Remembering the fate of the fictional Hal AI in the movie 2001 A Space Odyssey. So we're starting off from sort of the full, the full network at the beginning here. And then we're progressively zeroing out more and more modules until we get to this blob of almost nothing here. In this particular case, as we zero out more and more modules, eventually we fade to cyan, so to speak. Basically, once the catness of the images is lost, things become more and more alien from there on out, either descending into apparent randomness or sometimes that kind of barren zeroness. What about if, rather than zeroing out modules, we instead randomize the weights in them? Perhaps a little bit like the effect of a tumor rather than a stroke in a brain. Well, the results we get are at least qualitatively similar. These are, let's see, here we are looking. There we go. Those are the results from zeroing out, sorry, from randomizing successively more elements in the, in the system. These are different runs showing different randomizations and ending with these kind of very hard to identify what's going on, sort of pseudo random results. Another thing we can do is just to progressively mix randomness uniformly into every weight in the network. It's kind of like globally drugging the brain. And so here are examples where in each case we went from at the beginning here, 0% randomness to 1%, 2%, 3%, et cetera, and eventually things fade away. Typically in a very similar way. Each of these cases a different sort of instance of the randomness that was added. But we see the same Qualitative kind of behavior. The other thing we can do is instead of adding in randomness to the neural net brain, we can just progressively scale all the weights in the network down towards zero, let's say here in 1% increments from 100% to 99% and so on. And we can see things progressively sort of descending into randomness. Or we can progressively increase the numerical values of the weights, just scale them up numerically, in a sense, sort of blowing the mind of the network. And let's see, where do we have that? I think this is that. All right. Ah, there we go. And strangely going sort of a bit psychedelic in the process of blowing the mind by basically just numerically turning up all the weights in the network. Okay, well, we can think of what we've seen so far as sort of exploring some of the natural history of what's out there in generative AI space, or providing a small taste of at least one approximation to the kind of mental imagery one might encounter in alien minds. But how does this all fit into a more general picture of alien minds and what they might be like? With the concept of the ruliad, we finally have kind of a principal way to talk about alien minds, at least at a theoretical level. And the key point is that any alien mind, or for that matter, any mind, can be thought of as observing or sampling the ruliad from its own particular point of view, or, in effect, its own position in ruleial space. So the ruliad is defined to be the entangled limit of all possible computations, a unique object with an inevitable structure. And the idea is that anything, whether one interprets it as a phenomenon or an observer, must be part of the ruliad. And the key to our physics project is then that observers like us have certain general characteristics. We're computationally bounded with finite minds and limited sensory input. And we have a certain coherence that comes from our belief in our persistence in time and our consistent thread of experience. And what we then discover in our physics project is the rather remarkable result that from these characteristics and the general properties of the ruliad alone, it's essentially inevitable that we must perceive the universe to exhibit the fundamental physical laws it does. In particular the three big theories of 20th century physics, general relativity, quantum mechanics, and statistical mechanics. But what about the more detailed aspects of what we perceive? Well, that will depend on more detailed aspects of us as observers and of how our minds are set up. And in a sense, each different possible mind can be thought of as existing in a Certain place in ruleal space, different human minds are mostly close in rulial space, animal minds further away, and more alien minds still further. But how can we characterize what these minds are thinking about or how these minds perceive things? From inside our own minds, we can form a sense of what we perceive, but we don't really have good ways to reliably probe what other minds perceive. But what about what another mind imagines? Well, that's where what we've been doing here comes in. Because with generative AI we've got a mechanism for exposing the mental imagery of an AI mind. We could consider doing this with words and text, say with an LLM, but for us humans, images have a certain fluidity that text does not. Our eyes and brains can perfectly well see and absorb images, even if we don't understand them. But it's very difficult for us to absorb text we don't understand. It usually tends to just seem like a kind of word soup. But okay, so we generate mental imagery from minds that have been made alien by various modifications. How come we humans can understand anything such minds make? Well, it's a bit like one person being able to understand the thoughts of another. Their brains and minds are built differently and their internal view of things will inevitably be different. But the crucial idea that's for example, central to language is that it's possible to package up thoughts into something that can be transported to another mind. So whatever some particular internal thought might be, by the time we can express it with words in a language, it's possible to communicate it to another mind that will unpack it into different internal thoughts. It's a non trivial fact of physics that pure motion and physical space is possible. In other words, that an object can be moved without change from one place in physical space to another. And now in a sense, we're asking about pure motion in rulial space. Can we move something without change from one mind at one place in rulial space to another mind at another place in physical space? Things like particles as well as things like black holes are the fundamental elements that are imagined to move without change. So what's now the analog in rulial space? Well, it seems to be concepts as often, for example, represented by words. So what does that mean for our exploration of generative AI alien minds? We can ask whether when we move from one potentially alien mind to another, concepts are preserved. We don't have a perfect proxy for this, though. We can make a better one by appropriately training neural net classifiers. But as a first approximation this is like asking whether we can. Whether we. Whether as we kind of change the mind or move in real space, we can still recognize the concept the mind produces. Or in other words, if we start with a mind that's generating a cat in a party hat, will we still recognize the concepts of cat or hat in what a modified mind produces? And what we've seen is that sometimes we do and sometimes we don't. For example, when we looked at Cat island, we saw a certain boundary beyond which we could no longer recognize catness in the image that was produced. By studying things like Cat island and particularly its analogs. When not just the prompt, but also the underlying neural net has changed, it should be possible to map out how far concepts extend across alien minds. It's also possible to think about a kind of inverse question. Just what is the extent of a mind in rulial space? Or in other words, what range of points of view ultimately about the ruliad can a mind hold? Will it be narrow minded, able to only think in particular ways with particular concepts? Or will it be more broad minded, encompassing more ways of thinking with more concepts? In a sense, the whole arc of the development, intellectual development of our civilization can be thought of as corresponding to an expansion in rulial space, with us progressively being able to think in new ways about new things. And as we expand in rulial space, we're in effect encompassing more of what previously would have had. We would have had to consider the domain of an alien mind. When we look at images produced by generative AI, away from the specifics of human experience, say, an interconcept space, or with modified rules of generation, we may at first be able to make little from them, like ink blots or arrangements of stars. We'll often find ourselves wanting to say that what we see looks like this or that thing we know. But the real question is whether we can devise some way of describing what we see that allows us to build thoughts on what we see or reason about it. And what's very typical is that we manage to do this when we come up with a general symbolic description of what we see, say, captured with words in natural language or now computational language. Before we have those words or that symbolic description, we'll tend just not to absorb what we see. And so, for example, even though nested patterns have always existed in nature and were even explicitly created by mosaic artisans in the early 1200s, they seem to have never been systematically noticed or discussed at all until the latter part of the 20th century, when finally the framework of fractals was developed for talking about them. And so it may be with many of the forms we're seeing here. As of today, we have no name for them, no systematic framework for thinking about them, and no reason to view them as important. But particularly if the things we do repeatedly show us such forms, we'll eventually come up with names for them and start incorporating them into the domain that our minds cover. In a sense, that's what we've done here. We can think of that as showing us a preview of what's out there in Rule of Space and what's currently the domain of alien minds. In the general exploration of ruliology and the investigation of what arbitrary simple programs in the computational universe do, we're able to jump far across the ruliad. But it's typical that what we see is not something we can connect to things we're familiar with. And what we're doing here, we're moving only much smaller distances in rural space. We're starting from generative AI that's closely aligned with current human development, having been trained on images that we humans have put on the web and so on. But then we're making small changes to our AI mind, and looking at what it now generates. What we see is often surprising, but it's still close enough to where we currently are in rulial space that we can at least to some extent absorb and reason about what we're seeing. Still, the images often don't make sense to us. And yes, quite possibly the AI has invented something that has a rich and meaningful inner structure, but it's just that we don't yet have a way to talk about it. And if we did, it would immediately make perfect sense to us. So if we see something we don't understand, can we just train a translator at some level? The answer must be yes, because the principle of computational equivalence implies that ultimately there's a fundamental uniformity to the ruliad. But the problem is that the translator is likely to have to do an irreducible amount of computational work, and so it won't be implementable by a mind like ours. Still, even though we can't create a general translator, we can expect that certain features of what we see will still be translatable, in effect, by exploiting certain pockets of computational reducibility that must necessarily exist, and even when the system as a whole is full of computational irreducibility. And operationally, what this means in our case is that the AI may in effect, have found certain regularities or Patterns that we don't happen to have noticed, but that are useful in exploring further from our current human point in rulial space. So it's very challenging to get an intuitive understanding of what rulial space is like. But the approach that we're taking here is for me, a promising first effort in kind of humanizing rulial space and seeing just how we might be able to relate to what is so far just the domain of alien minds. So let's talk a little bit. That was kind of the main set of things that I wanted to say about kind of mental imagery of alien minds and its relationship to generative AI. Let me just perhaps quite quickly go through a little talk a little bit more technically about how does generative AI work? So let's see what I've been using here. And what I'm talking about is a method called stable diffusion, and its operation is in many ways both clever and surprising. As it's implemented today, it's steeped in fairly complicated engineering details. To what extent these will ultimately be necessary isn't clear. But any case, I'll talk here mostly about general principles and try and give us a broad outline of how generative AI can be used to produce images. At the core of generative AI is the ability to produce things of some particular type that follow the pattern of known things of that type. So, for example, large language models, LLMs are intended to produce text that follows the patterns of text written by humans, humans, say, on the web. And generative AI systems for images are similarly intended to produce images that follow the pattern of images put on the web. But what kinds of patterns exist in typical images, say, on the web? So here are some examples of typical images scaled down to 32 by 32 pixels and taken from a standard set of 60,000 images. As a very first thing, we can kind of ask what colors show up in these images. Well, they're not uniform in RGB space. What about the positions of different colors adjusting to accentuate color differences? The average image turns out to have a curious kind of Howl's eye type look, presumably with blue for sky at the top and brown for earth at the bottom. But just picking pixels separately, even with the color distribution inferred from actual images, won't produce images that in any way look natural or realistic. There are a few examples, and the immediate issue is that pixels aren't really independent. Most pixels in most images are correlated in color with nearby pixels. And in the first approximation, one can capture this, for example, by fitting the list of colors of all pixels to, let's say a multivariate Gaussian distribution with a covariance matrix that represents their correlation. So sampling from this distribution gives images like these that indeed look somehow statistically natural, even if there isn't appropriate detailed structure in them. So, okay, how can one do better? The basic idea is to use neural nets which can affect encode detailed long range connections between pixels in some ways. See a typo there. It's similar to what's done in LLMs like ChatGPT, where one has to deal with long range correlations, but long range connections between words and text. But for images it's structurally a bit more difficult because in some sense one has to consistently fit together 2D patches rather than just progressively extending a 1D sequence. And the typical way this is done at first seems a bit bizarre. The basic idea is to start with a random array of pixels corresponding in effect to pure noise, and then progressively to reduce the noise to end up with a reasonable image that follows the patterns of typical images, all the while guided by some prompt that says what one wants the reasonable image to be of. So how does one go from randomness to definite reasonable things? The key is to use the notion of attractors. In a very simple case, one might have a system like this kind of mechanical example, where from any randomly chosen initial condition, one always. Evolves to one of here two definite fixed point attractors, the two minima in this surface one has something similar in a neural net that's for example, trained to recognize digits like here. Regardless of exactly how each digit is written or the noise that gets added to it, the network will take this input and evolve to an attractor corresponding to a digit. Sometimes there can be lots of attractors, like in this class 2 cellular automaton. Here evolving down the page. Many different initial conditions can lead to the same attractor. But there are many possible attractors corresponding to different final patterns of stripes. The same can be true for example, in 2D cellular automaton, where now the attractors can be thought of as being different images with structures determined by the cellular automaton rule. But what if one wants to arrange to have particular images as attractors? Here's where the somewhat surprising idea of stable diffusion can be used. Imagine we start with two possible images, an image of an A and an image of a B. And then in progressive steps, we add noise to them. Well, here's the bizarre thing we now want to do. We want to train a neural net to take the image we get at a particular Step here and go backwards, removing noise from it. The neural net we'll use for this is somewhat complicated, with convolutional pieces that basically operate on blocks of nearby pixels and transformers that get applied to certain weights with certain weights to more distant pixels. In Wolfram language, the network kind of looks at a high level like this, schematically like this bunch of things that got a bit overlapped there. But anyway, and roughly what it's doing is to make kind of an informationally compressed version of each image and then to expand it again through what's usually called a U net neural net. We start with an untrained version of this network, say just randomly initialized. Then we feed it a couple of million examples of noisy pictures of A and a B and the denoised outputs we want in each case. Then if we take the trained neural net and successively apply it, for example, to a noised A, the net will correctly determine that the denoised version is a pure A. Let me see if I can show you that. There we go. But what if we apply this network to pure noise? The network has been set up to always eventually evolve either to the A attractor or the B attractor. But which it chooses in a particular case will depend on the details of the initial noise. So in effect, the network will seem to be picking at random to fish either the A or B out of the noise. So how does this apply to our original goal of generating images like those found, for example, on the web? Well, instead of just training our denoising or inverse diffusion network on a couple of target images, let's imagine we train it on billions of images from the web. And let's also assume that our network isn't big enough to store all those images in any kind of explicit way. In the abstract, it's not clear what the network will do, but the remarkable empirical fact is that it seems to manage to successfully generate from noise images that follow the general patterns of the images it was trained from. There isn't any clear way to formally validate the success. It's really just a matter of human perception to use the images. To us, the images generally look right. It could be that with a different, say, alien system of perception, we'd immediately see something wrong with the images. But for purposes of human perception, the neural net seems to give reasonable looking images, perhaps not least because the neural net operates at least approximately like our brains and our processes of perception seem to operate. So what? We've now described how a denoising neural net seems to Be able to start from some configuration of random noise and generate a reasonable looking image. And from any particular configuration of noise, a given neural net will always generate the same image. But there's no way to tell what the image will be of. It's just something to empirically explore, as we did above. But what if we want to guide the neural net to generate an image that we describe as being of a definite thing, like a cat in a party hat? We could imagine continually checking whether the image we're generating will be recognized by a neural net as being of what we wanted. And conceptually, that's what we want to do. That's what we can do. But we also need a way to kind of redirect the image generation if it's not going in the right direction. And a convenient way to do this is to mix a description of what we want right into the denoising training process. In particular, if we're training to recover an A, mix a description of the A right alongside the image of the A. And here we can make use of a key feature of neural nets that ultimately they operate on arrays of real numbers. So whether they're dealing with images composed of pixels or text composed of words, all these things eventually have to be ground up into arrays of real numbers. And when a neural net is trained, what it's ultimately learning is just how to appropriately transform these disembodied arrays of numbers. There's a fairly natural way to generate an array of numbers from an image. Just take triples of red, green, and blue intensity values for each pixel. We could take a different detailed representation, but it's not likely to matter, because the neural net can always effectively learn a conversion. But what about a textual description, like a cat in a party hat? We need to find a way to encode text as an array of numbers. And actually, LLMs face the same issue. And we can solve it in basically the same way as LLMs do. In the end, what we want to is to derive from any piece of text a feature vector consisting of an array of numbers that provide some kind of representation of the effective meaning of the text, or at least the effective meaning relevant to describing images. Let's say we train a neural net to reproduce associations between images and captions, as found, for example, on the web. If we feed this neural net on an image, it'll try to generate a caption for the image. If we feed the neural net a caption, it's not realistic for it to generate a whole image, but we can look at the innards of the neural net and see the array of numbers that derived from the caption, then use this as our feature vector. And the idea is that because captions that mean the same thing should be associated in the training set with the same kind of images, they should have similar feature vectors. So now let's say we want to generate a picture of a cat in a party hat. First we find the feature vector associated with the text, a cat and a party hat. Then this is what we keep mixing in at each stage of denoising, to guide the denoising process and end up with an image that the image captioning network will identify as a cat in a party hat. So the most direct way to do denoising is to operate directly on the pixels and image. But it turns out there's a considerably more efficient approach which operates not on the pixels, but on features of the image, or more specifically, on a feature vector which describes an image in a raw image presented in terms of pixels. There's a lot of redundancy, which is why, for example, image formats like JPEG or PNG manage to capture. Manage to compress raw images so much without even noticeably modifying them for purposes of typical human perception. But with neural nets, it's possible to do much greater compression, particularly if all we want to do is to preserve the meaning of an image without worrying about its precise details. In fact, as part of training a neural net not to associate images with captions, sorry, training neural net to associate images with captions, we can derive a kind of latent representation of images, or in effect, a feature vector that captures the important features of the image. And then we can do everything we've discussed so far directly on this latent representation, decoding it only at the end into the actual pixel representation of the image. So what does it look like to build up the latent representation of an image with the particular setup we're using here? It turns out that the feature vector in the latent representation still preserves the basic spatial arrangement of the image. The latent pixels are much coarser than the visible ones and happen to be characterized by four numbers rather than the three for RGB and. But we can decode things to see the denoising happening in terms of latent pixels. Then we can take the latent representation we get and once again used a trained neural net to fill in a decoding of this in terms of actual pixels, getting out our final generated image. That's basically the story of how the generative AI works to produce images. And it's Perhaps as a last thing, I'll talk a little bit about an analogy to all of this in simple programs where we can kind of see a little bit more the essence of what's going on. So generative AI systems work by having attractors that are carefully constructed through training so that they correspond to reasonable outputs. And a large part of what we've done here is to study what happens to these attractors when we change the internal parameters of the system, like neural net weights and so on. And what we've seen has been complicated, often quite alien looking. But question is, is there perhaps a simpler setup in which we can see similar core phenomena? By the time we're thinking about creating attractors for realistic images and so on, it's inevitable that things are going to be complicated. But what if we look at systems with much simpler setups? For example, consider a dynamical system whose state is characterized just by a single number, like an iterated map on the interval like X goes to AX times one minus X. Well, starting from a uniform array of possible X images, we can kind of show down the page which values of X are achieved at successive iterations. So for the parameter value A equals 2.9, the system evolves from any initial value to a single attractor, which consists of a single fixed final value. But if we change the internal parameter A to 3.1, we now get two distinct final values. And at the bifurcation point A equals 3, there's a sudden change from 1 to 2 distinct values. And indeed, in our generative AI system, it's fairly common to see similar discontinuous changes in behavior, even when an internal parameter is continuously changed. So as another example slightly closer to image generation, consider as we did actually above, a one dimensional cellular automaton that exhibits Class 2 behavior and evolves from any initial state to some fixed final state that one can think of as an attractor for the system. So here's an example of that which attractor one reaches, depends on the initial condition one starts from. But an analogy to our generative AI system, We can think of all the attractors as being reasonable outputs for the system. But now, what happens if we change the parameters of the system, or in this case, the cellular automaton rule in particular, what will happen to the attractors? It's like what we did above in changing weights in the neural net, but it's a lot simpler. The particular rule we're using here has four possible colors for each cell and is defined by just 64 discrete values from 0 to 3. So let's say we randomly change one of those values at a time. Here are some examples of what we get, always starting from the same initial condition as we did before. So with a couple of exceptions, these seem to produce results that are at least roughly similar to what we got without changing the rule in analogy to what we did above. The cat might have changed, but it's still more or less a cat. But let's now try progressive randomization, where we modify successively more values in the definition of a rule of the rule. For a while we get roughly similar results. But then, much like in the cat examples above, things eventually fall apart and we get much more random results. So one important difference between stable diffusion and cellular automata is that while in cellular automata the evolution can lead to continued change forever, in stable diffusion there's an annealing process used that always makes successive steps progressively smaller and essentially forces a fixed point to be reached. But notwithstanding this, we can try to get a closer analogy to image generation by looking again, as we did before, at 2D cellular automata. So here's an example of a well, not too exciting as images final states reached from three different initial states in a particular rule. And here's what happens if one progressively changes that rule. At first one still gets some reasonable, according to the original rule, final states. But if one changes the rule further, things get more alien until they look to us quite random in changing the rule, one is in effect moving in rule of space. By looking at how this works in cellular automata, one can get a certain amount of intuition. Changes to the rule in a cellular automaton seem a bit like changes to the genotype in biology, with the behavior of the cellular automaton representing the corresponding phenotype. But seeing how rulial motion works in a generative AI that's been trained on human style input gives a much more accessible and humanized picture of what's going on, even if it still seems further out of reach in terms of any kind of traditional explicit formalization. So that's what I had to say in the piece that I just posted. I see a number of questions that have come in and I can perhaps try and address these quickly. And then I have to disappear to something else. Let's see. So question here from how can we filter out the human bias towards pattern recognition in terms of the fact that when we generate kind of images and interconcept space, we keep on trying to move them and do what the neural net does and kind of move them to attractors that we know, that are concepts that we know. That's the thing we do. It's an interesting phenomenon. Let's see. Question from Pete here about the embedding of words in high dimensional space. Let's see. And I was showing the line between dogs and cats through interconcept space. The thing to understand is when you're in a very high dimensional space, yes, there is a line that goes through particular points, but there are many, many, many possible lines. So it's very confusing in a sense. It's kind of like if everybody's on a railroad track, all the trains have to be careful not to run into each other. But as soon as you're in, in three dimensions, there's just an awful lot of different paths you can follow. If you choose a particular path, if you're flying between two particular points in the air or something, then yes, the planes can crash into each other because there's just that one line you're restricting yourself to. But in even three dimensional space there's an awful lot of room to maneuver. And by the time you get to these many thousand dimensional spaces, million dimensional spaces, it's a bit unintuitive, but there's just an awful lot of room to maneuver. So between the dog point and the cat point, you can define kind of a straight line in a certain version of that high dimensional space. And there are limited number of points on that line, but as soon as you go off that line, there's kind of a lot of room to maneuver. Let's see, there's a question here from what do about signal to noise ratio? I think probably referring to the way that denoising happens in stable diffusion algorithms. What happens is you start from it's all noise and then what you've done is to make a system which has learnt how to go from something which had just a little bit of noise to something without that noise. And then you say, well, I want you to do even more than that. Go from something that's all noise and see what happens. And that's kind of how one goes from this sort of pure noise initial condition to a quotes reasonable image at the end. And that happens. The kinds of things I was showing the images and interconcept space happen by. You can make those by just sort of starting with noise and then generating an image from that. The slightly more complicated thing, when you have actual prompts where you're saying and I want a thing that conforms to this kind of verbal description or this description, that kind of interpolates between verbal descriptions then you're mixing in some more stuff that kind of guides the way that that denoising will happen. But it's a very weird process that one goes from something that's kind of all noise to something that is actually an image. All right, well if those are all the questions people had we should wrap up here. Thank you very much for meeting these cats and I just want to remind everyone that you can find the text of what I was talking about here on my website, stephenwolfen.com and you can run any of the things, any of the pictures that I showed in that piece and in the, in what I've been talking about today you can run on your own computer though it won't be terribly fast unless you have a fairly fancy gpu. The GPU that I used for these things I think has 25 gigabytes of VRAM. So kind of a mid range GPU. All right, well thanks very much and bye for now.