Science & Technology Q&A for Kids (and others) [Part 125]

Okay. Hello everyone. Welcome to another episode of Science and Technology, Q and A for Kids and others. And I see all kinds of questions stored up here, many about AI. So let's talk a little bit about AI. Let's see. Prab says, I've been hearing of AI and LLMs in the context of an arms race between countries. What do LLMs look like scaled up in that manner? I. E like a global LLM. Okay, so let's talk about. So ChatGPT is sort of the world's most popular example of a large language model. And what is ChatGPT? Well, it's something whose main purpose is to write text, to fluently write text in a way that if you give it a textual prompt, it will try and continue from that prompt in a way that is typical of the way that text would be continued based on all the text it's read from the web and elsewhere. So in other words, if you say the cat sat on the, then the most likely thing that you'll probably find on the web for the next word is mat. And that's what it's probably going to say is the next word. And then it keeps going from there. And it's completely remarkable how coherent kind of the essays that ChatGPT can write are given that it's really writing them one word at a time. And all it's doing is saying given everything that came before, what's the either most likely or roughly most likely thing to come next? Sometimes it doesn't pick the absolutely most likely to come next. If it always picks the most likely to come next. It has a terrible habit of getting stuck in various kinds of loops. It's kind of like what's done is there's a notion of so called temperature parameter which says essentially if the temperature is non zero, it's like there's a certain amount of sort of randomness bubbling around and it sometimes picks the next word that isn't the most probable next word, but it's slightly less than the most probable. There's some sort of noise associated with which word it picks. And it seems like when it's always picking the most probable word that it can get into this very rigid state. It's kind of like zero temperature is as if you had something in physics where you had a bunch of molecules and at zero temperature they're all kind of clumped together in a solid. It's like a crystal or something where it just has a very repetitive structure. Whereas if you give it a bit of noise, a bit of temperature so to speak. It can kind of flow around in a. In a. In a more random way, and you get something which is a more convincing, less rigid kind of essay. It's a very sort of black magic kind of kind of thing to do. In any case, the basic structure of what AN LLM does, ChatGPT and so on do is that you give it a prompt. It will try and continue from that prompt based on what it has been trained from. And what it's been trained from is roughly the content of the web. Talk about that in a moment. I will say that it's a very significant feature of how LLMs work that they're always. All they know about is the past. They don't plan for the future. They're going a word at a time. And so one thing that one finds is things like if you get an LLM to generate an answer, then you say to the LLM, is that answer correct? The LLM will often be much more insightful about that answer. And sometimes in the way that ChatGPT is set up, it will apologize and say, well, I now see that answer isn't correct. Let me reformulate that answer. Why on earth is it doing that? Why didn't it get right the first time? Well, the answer is because it couldn't plan. So what it's doing is once it can see that whole collection of text that came from the past, it can notice that there are things about it that are very improbable and weren't really quite right, and it can then go ahead and try and correct itself. But as it was just going a word at a time, filling things in, it couldn't figure that out. So in any case. But the basic thing to realize is what ChatGPT is doing is it has been made to sort of follow, in effect, the statistics of all the texts that it was trained from. What text was it trained from? It was trained from, I think, around 4 billion web pages, which are part of. I think it primarily got its training data from a thing called Common Crawl, which is a. So, okay, so the web is obviously a large number of interconnected pages, many of which are publicly visible, but many more of which are in internal websites of companies or other organizations and are behind passwords and things like this. But the number of pages of pure web that are out there is, I don't know, 6 billion is a commonly quoted number. 10 billion, maybe. It's a little bit hard to make a rigid statement of that number because there can often be things where different URLs that you can enter will have for example, things after a question mark in the URL, the query string of the URL could be different and that will produce a different page, but it may not be a sort of humanly written different page. It may just be something where, oh, there's something where you have a version of the page that's suitable for people coming from this part of the world, another one for that part of the world. And, and it was just, you know, it changed the currency of some price or something like this. And are those different pages or are they really the same intrinsic human written page and the rest is just a kind of machine determined modification of the page. So it's a little hard to know which pages are kind of, you know, which are the really independent pages. And then there are many pages where some people try and sort of become less popular in these days to sort of trick search engines by having giant collections of pages that are machine generated that are intended to sort of catch the thing by just being a web page that has this exact phrase on it. And when people search for that phrase, they'll end up on that website. So, you know, the number of human written pages is not completely clear, but the number that are, I think in, in the. Oh yeah. So how does, how does one sample the web? Well, what one can do to sample the web is you might say, well, let's just start from one website and then just keep following links. You follow links, you follow more links, you follow more links, you try and arrange it so that if you ever get to a web page you've been at before, you know you've been there before and you don't have to re refollow those links, so to speak. But so what typically happens in crawling the web is you'll have to start from some seed web pages that say, well, this links and eventually starting from that web page, you'll sort of visit the whole web. Now there may be parts of the web that are completely disconnected from everything else where nothing links to that page. That page may link out to other pages, but nothing links into that page. So that's an issue in sort of defining how you visit the web and whether there are parts of the web that are sort of separate islands not found by sort of a basic crawl. So the common crawl is a thing that's been being done how long is it? 15 years maybe. That is sort of an open crawl of the web crawling the web. It's kind of complicated what you get back when you crawl the web because A web page, particularly in modern times, is full of lots of singing, dancing stuff. In many cases, it's not just text. It's text that can be determined by, you know, JavaScript code that's running in your browser that, you know, makes something open and close and do this and that and the other. And so there are a lot of questions about what you actually get back when you crawl the web. And there are these different kinds of files. Warc files are one type of file that represent the actual sort of bag of bits that are sent back to your web browser by web archive. File is what that stands for. It's sort of the bag that. That's a collection of sort of the bag of bits that's sent back to your browser when you go to a web page. And then that bag of bits will be rendered in some way on your particular browser. And again, gets more tricky because it's a question of what, what does the JavaScript that's in that page do to pull in more things from the page and so on. So it's a slightly tricky thing. But if you're just a regular, simple AI, an LLM, whatever, mostly what you're going to be interested in is what's the text on that page. And so the extraction of that plain text is not completely straightforward in many cases. And it's not obvious that you can get the whole plain text. Maybe you have to actually explicitly run the web page in a browser to be able to see what text is on there. And then maybe it will just render, but it's actually inside just an image. It's not really text, like characters and so on. It can get kind of tricky. But in any case, the end result of that is there's about 4 billion, I think, pages that are sort of public pages which anybody can visit on the web that are accessible. And in this common crawl, for example. And so that's a thing that sort of any LLM can get access to. Then there's other things that an LLM can train on. For example, books, and they're about believed to be maybe 130 million books that have ever been published, and maybe 10 million of them have been scanned. And maybe a few million of those are sort of accessible for LLMs to read. And that's another by scanned, I mean, that the optical character recognition has been done so that one can pick out, I mean, one thing is just to get the images of the pages of the book. Another is to actually pick out the letters and the words and all that kind of thing. But so there's a few million books. So the end result is there's maybe a trillion words that are out there, somewhat publicly accessible that LLMs can train on. And most of the LLMs have been trained on all of that stuff, kind of like the collection of knowledge. This is what our civilization has produced. It's a trillion words. And let's, you know, that's what's accessible. Okay, so now the question was asking, what does it look like when LLMs are scaled up? Well, there are two kinds of ways to scale up an LLM. One is to give it more training data, and the other is to have it have. Be a bigger neural network. So in terms of the more training data, there's this big collection of training data that sort of everybody has access to. When you go beyond that, it's complicated. So, for example, the Internet archive has about 500 billion pages, I believe, that it has collected over the last 26 years, maybe that it's been in operation. And they don't know how many of those pages are unique. They haven't been able. You know, that's a complicated computational calculation, so to speak. It's something where it's not even clear what it means by unique. As I mentioned before, there could be, you know, what size of change makes it really a different page, et cetera, et cetera, et cetera. So there's a large factor, maybe a factor of 100 more pages in sort of the history of the web. But it's not clear how unique those pages are. That's, that's one thing. Another thing is that there are lots of internal websites behind passwords, et cetera, et cetera, et cetera, believed to be at least 100 times what's in the public Web. So, you know, right there we've got, you know, a lot more training data that's potentially available. In the end, there's, you know, how much data is there that's just sitting on people's local computers and so on as, as information that could be used to train an LLM, textual stuff. I don't know how much more there really is than what I just mentioned. I think there's a lot of sort of structured numerical type data in databases. There's a lot of kind of that. That's one category of thing then. So I tend to think that the. Yeah, so that's that part. I mean, there's other kinds of data like videos, for example, and there's just a vast amount of video data that comes from things like security cameras. And so on, I mean there's video data that is sort of the actual feature films that get made. There's video data that is sort of all of us random people uploading things to YouTube, like this live stream for example. And then there's kind of typically, then there's just absolutely gobs of kind of just video that has been running, you know, a security camera that's just recording video and sending it to the cloud somewhere and just that stuff getting stored. And that kind of thing doesn't have textual data in it. Many of the sort of, let's say random YouTube videos do have textual data and there is some amount of transcripting that's been done that allows one to have sort of text from those videos. More importantly, in the future it will be possible to train from the videos directly and not to get text, but to get sort of training about what happens in the world. You know, training about sort of what, what are the things that give us humans physical intuition about, you know, you, you drop something and it falls. And that's sort of things that, that's a type of training you can get from videos. And that's, that's a sort of a coming attraction. And one's starting to see some video generation technology for AIs of, of make a plausible video or the analog, the direct analog of what ChatGPT does of can you continue this text from a prompt would be, here's the beginning of a video. How would this video continue? And it's an interesting issue, this sort of video prompting world of, you know, can you do something which is kind of a simple video prompt and then have the thing continue from that. And I think just as there is a certain level of ability to go from sort of a sketch to a full kind of elaborately AI generated picture, the same may be true for videos where you can go from a little charade type thing that you do to then having it sort of fill that in to be a full convincing kind of video. I have to say the demos I've seen of that so far have not been particularly sophisticated, but I'm sure that will improve. So back to sort of all the data that's potentially available. I should say that to give a sense of scale, let's see, I personally have written 50 million words in my life, so and compared to a trillion words that are out there in the public, public sort of web and so on, that's, you know, I've been reasonably prolific at writing things. Lots of emails, lots of other kinds of documents. I think I published oh, what is it, 15 million words? I think I'm forgetting these numbers now, but and in fact in these live streams that I've been doing that contributes 10ish million or more words, I wrote down all these things, I've already forgotten them. But it's of that order just from the transcripts of these kinds of live streams. So give some sense of scale. Now one thing people have raised is maybe there are countries that have particular sort of caches of material, for example, in their national libraries, things like this. And yeah, it's a small effect. I don't think it's a huge effect. I think that the vast majority of what is available to train an LLM is sort of already out there. Now I should say another thing. What are you training an LLM about? Well, an LLM is mostly about generating text and it also has the ability, if it has seen mentioned the population of the United States is 330 million. That's a piece of text. It might have seen another piece of text that said the population of the United States is 120 million because that was a book published whenever it was. It doesn't have any sort of intrinsic sense of being able to compute with that number, it's just what textually appeared. But the fact is that the main skill in a sense of an LLM is its ability to fluently generate language and fluently understand language. Knowing all those facts, all those specific the population is 330 million or whatever it is now is really a separate thing. That's the kind of thing that we've built in Wolfram language. Wolfram Alpha would been the main sort of thing that's actually trying to do a good job of getting sort of correct computable data to feed these kinds of things. And I think what's going to happen is that they're going to be much more stripped down LLMs which are just the language part. And whenever they need a fact, they'll call out to, for example Wolfram Plugin for ChatGPT, for example, to get that fact. They won't have to store within the LLM itself all of those facts. They'll just kind of know the linguistic flow of what it means to, you know, what populations are and what things you usually say about that are. Now the other thing that LLMs have is they have a certain amount of human like common sense. So even though they don't know the precise weight of an elephant, perhaps they do have some sense from common sense that an elephant weighs more than a dog. Let's Say, even though they don't necessarily know the actual weights of those things and different breeds of dogs and all those kinds of details. But that's sort of a common sense fact. And just as we humans would have that common sense fact, even though we couldn't tell you, most of us couldn't tell you an elephant weighs X number of tons and things like this. So there's sort of two things that need to be captured in the kind of the sort of minimal LLM, the fluent language part, and common sense facts about the world, or common sense about the world. It isn't really fact, it's common sense about the world. The detailed facts can be factored out. And that's where our technology, for example, comes in. Our technology also lets you not only have the detailed facts, but also have the computation from sort of a computational foundation, from facts, from algorithms and so on to, to compute things that have never been put on the web, that have never been written down as text, which are in a sense new knowledge that's being created by computation. New, you know, work out this math problem, work out this chemistry thing. It's never been computed before, but that's something new that you can generate. That's not at all what LLMs are set up to do. LLMs are set up to, you know, continue the text from a prompt and be able to fluently produce text. So I think the future LLM will be this quite stripped down thing that mostly is about structure of language and common sense, calling out to kind of a computational knowledge engine, which we have the main one that exists in the world and, and using that to kind of supply the kind of the factual data. Now it's a good thing partly because if an LLM is left to make up the factual data, it's going to make up something that's statistically plausible, but maybe complete nonsense. And that's what we see happening with sort of raw chatgpt, for example, a lot of the time now. I mean, it also has to be said that an LLM can write text that's fiction as well as fact. And you don't want to lock it down and say you can only write things that you can validate, are true in the world because you also want it to be able to write pieces of fiction and so on. That's a useful thing for it to be able to do, even at the level of sort of micro fiction, as you might call it, where let's say you're trying to make up fun word problems in math education, for instance, you Might say, make up something that is a word problem that's about a city populated by flying jellyfish or something and no such thing actually exists in the world. But you can make up a nice little story that counts the tentacles of the jellyfish or whatever it wants to do in the math word problem and gives it a framing, that fictional framing. And that's a very valid thing to want to do and a very useful thing in terms of sort of human entertainment and communication and so on. That has nothing to do with kind of check the facts type thing. So, okay, so that's on the subject of sort of the data that goes into an LLM. Where do you get it from? Are there kind of specific countries, for example, that could mine a lot more data? As I say, I think the main source is probably things like national libraries. There's, you know, in terms of the. Yeah, that, that's. So then the next question is, okay, you're going to train the LLM. As I say, I think in the future the pure LLM component will actually be quite small and a lot of it will just be sort of fact back up. And. But let's imagine that you say, well, I want to train a bigger, you know, using current technology without having done the sort of fact factoring, so to speak. I want to just train a bigger LLM. So one question is, you know, what, what kind of computation resources do you need to do that? How much does it cost? Mostly ones using graphics processing units, GPUs to, to do that training? Because a lot of neural net training can be done with a high degree of local parallelism where a lot of operations can be done kind of in parallel. And that's what GPUs are set up to have sort of do operations across big arrays of numbers where each of the elements in the array are being computed in parallel at the same time as the other elements. More efficient than a typical CPU in a typical computer. So, you know, one wants lots of GPUs and one wants to run them for a long time. There's this kind of notion of a foundation model, which is sort of the basic LLM that's created from its big training set and so on. The thing that's been a big surprise to everybody is that given that foundation model, it's often possible, usually possible, it seems, to get that model to sort of do particular things you want by doing just prompting. So there's this notion of kind of zero shot learning where you just basically you pretty much just ask the LLM to do what you want, and it just based on its existing training, it does it. There's sort of few shot learning where you give the LLM a bunch of examples of what you want and you kind of get it in the groove, so to speak. And then you ask it the thing you really want to ask it. And why that works is not completely clear. In other words, what probably is the case is that by giving it those examples, you're kind of, you're priming it to kind of use that part of its kind of memory in some sense that is somehow aligned with those examples. And so it kind of follows along from those examples to do things the way you want it done. It's kind of as if there's this sort of big space of kind of not quite knowledge, but sense or something. Common sense language and common sense of the LLM. And you're saying, I want to walk around in this particular area and you're identifying that by giving this, these kind of examples and these prompts and so on. But, but just how that works is not very well understood. And it's not known whether you can sort of get further by training a new and different foundation model. My guess is probably you cannot. Probably. There's sort of a limit. There's sort of the corpus of human knowledge. It defines language as we kind of typically use it. And once you've got that, you're kind of done. And most of the rest of what you will add is more structured computational knowledge, more algorithmic knowledge, which really isn't the domain of LLMs, but we'll see how that works. And sort of a question of, okay, the foundation model, how do you train it? OpenAI has one, anthropic has one. There's one called Bloom. There's one. Oh gosh, they're llama. I'll. There's an alpaca, there's a whole zoo. Oh, it really does sound like a zoo, doesn't it? With llamas and alpacas and things like that running around of foundation models that people, mostly large companies, have been training. And it's a subject of some debate how much it really does cost and should cost to train a foundation model. It's also a question of once you have one such foundation model, can you train another one from it? And how does that work? Can you? For example, one of the things that's been found is that you get one of these big foundation models that's terabytes of data, and then you can distill that model. You can actually sort of Make a train a model from that model and it is a simpler model and the simpler model still works. But you can't go so far as people know yet directly to the simpler model. You have to go to this big model and then distill the big model down. You don't get to go right now yet to the small model directly. And estimates range in the. Oh, it costs a billion dollars to train a foundation model to. It costs half a million dollars to train a foundation model. So you can take your pick right now of what the correct number is. And I think it depends a lot on where you're starting from and how much of the existing material, existing training and so on you're making use of. My guess is that in time the cost to do any training that any reasonable person would try to do, given the resources that already exist, will push things down to a much lower cost for even training a foundation model. But it's not even clear it's useful to train that another foundation model, one might just use one of the ones that's been trained from a very similar training set with a very similar internal architecture for the, for the neural net and so on. Just use it and build things on top of it and use few shot learning, prompting, etc. Etc. Etc. To get the model to do what one wants. So I think I'm kind of not expecting that there will be a kind of. Somebody has the greatest LLM because they had fundamentally more training data, fundamentally more larger model, things like this. Now it's not to say that right now, you know, there's a lot of, a lot of good engineering work that's been done in building the successful LLM models and that creates a. There's a, there's all kinds of details and tricks and just good software engineering and, and good kind of prompting, good fine tuning and so on of the model to make it say sensible things, not say totally crazy things, not go completely bonkers, not and you know, generally follow instructions and so on. That's a. There's a whole lot of kind of sort of typical engineering technology that's gone into that. That's not so much in the big picture of how the LLMs work. Let's see, Let's see. Five shot is asking here about what about model interoperability? Where are we on research for that? Do we need to develop new and more sophisticated mathematics to begin to understand these black box models? To do you think in time we'll be able to do causal inference with them? Well, okay, so the question of why does an LLM work? How does it really work? I wrote this little book. Look at this. I even have a copy here. There we go. Actual little book. It's a very little book. It's my smallest ever book. I think I have. Look, I have another one here. This is my biggest, I think. No, this is my second biggest ever book, New Kind of Science, that I wrote 20 years ago. The Mathematica book actually was even bigger than this before we just switched to making it be online. And now it would be like 300,000 pages if we were to print it out. But in any case, this whole question about sort of how does something like ChatGPT, how do LLMs work inside? There's two levels of answer to that question. There's how do the neural nets that people have put together to build the LLM? How do they work? And the second is, is there a science of how LLMs work? Is there a kind of larger scale picture where we can start to say, oh, yes, I understand how that works in the same sense that if you were looking at, I don't know, the, I don't know, the flow of air in the atmosphere or something, you could at one level say, well, I know how all the molecules are moving around in the air, but at another level you could say, I'm describing the motion of the air in terms of cyclones and high pressure regions and fluid dynamics and all those sorts of things. So I think the question we know kind of at the engineering sort of neural net structure level, we kind of know how LLMs get built. It's actually surprisingly straightforward in a sense. It's surprisingly much. You just put together these things. There are a few very important tricks about how it works, but you just sort of put these things together and then you train them by giving them lots of examples of text and so on. And that's kind of how they work at the level of sort of the rules which define their operation. The second thing is, what's the big picture description of what they're doing? And I think that the thing that we're really learning from LLMs is that language has kind of an emergent structure that we've never really cataloged very well before. I mean, we've cataloged sort of standard syntactic grammar. We know that sort of a sentence is composed of things like a noun phrase, a verb phrase, a noun phrase, the noun phrase might contain an adjective and a noun, et cetera, et cetera, et cetera. We know this kind of. It's a hierarchical structure of syntactic grammar that it says it's sort of the construction kit. For putting sentences together. But that construction kit allows you to put together sentences which don't, don't mean anything to us. I mean, we could put together, you know, happy electrons dance in purple. You know, it's syntactically, in terms of syntactic grammar, that's a correct sentence. Happy, adjective, electrons, noun, noun, phrase, dance, verb, you know, et cetera, et cetera, et cetera. It's syntactically correct. But the, you know, what does that mean? Do we have a way of taking that sentence and converting it in our minds to something that we can reason from and so on? Well, not really. It just seems like a nonsense sentence. You know, could it be have a whole setup in a poem or something where you've defined something where that sentence is really going to have a significance to us? Yes, it could. But on its own, without further context, that sentence is kind of. Doesn't. Doesn't fire some kind of meaning off in our brains. I think it's worth understanding that when we talk about understanding language, it's a bit of a floppy thing when it comes to just saying, well, I understood that, or I didn't understand that. What does it mean to understand that? In the case of WolframAlpha and our sort of modern technology stack, there's a very clear notion of we understand it, which is we converted from the natural language into our precise computational language. And in that precise computational language, we can compute whatever we want from that sort of computationalified statement. So, you know, in the computational language, we have a notion of electrons. We might say an electron with an energy of whatever. And we could then compute for an electron of energy, whatever. How far will it go in a block of aluminum, you know, what is its relativistic correction to its, oh, I don't know, its mass or whatever? We can compute all those things about the electron because we understood, in some sort of computational sense of understanding what an electronic of energy, 2 million electron volts or whatever it is, means. Now it's less clear how that works when it comes to a human understanding language. Probably the best thing we could say is, can one take the thing you were told in language and reason from it, deduce other things from it? Or is it kind of a I'm stuck because I have no idea what a happy electron dancing might be about. But okay, the question is, if we go beyond the syntactic grammar and we say, what about sentences that really do have a meaning to us, that really do sort of fire off that. I understand it, I mean it, I know what it means. I can deduce things from it. What about those kinds of sentences? Is there a kind of semantic grammar, a meaning based grammar that defines what, what are valid sentences of that kind? I think what the success of ChatGPT is showing us is that yes, there really are additional rules about sort of how to make sentences that can be meaningful. And there are sentences that can be perfectly meaningful. I think I gave in the book that I wrote the sentence the elephant flew to the moon, which is a perfectly meaningful that, that conjures up a meaning to us in our minds. It doesn't happen to be a factual sentence. You know, so far as we know, no elephant has ever flown to the moon, but we can have some concept of what it would mean for that to be a thing. And so I think that there's a level of kind of we go from the purely syntactic do the nouns and verbs match? Up to the sort of the semantic of does the thing have a kind of a way in which we can unpack it and say we can deduce things from that sentence? And then sometimes people talk about pragmatics as sort of another level of meaning, which is sort of does it actually happen in the world? And so there are these different levels. But the thing that we really clearly, I think, has been exposed to us is that there is this semantic level that we haven't really cataloged very well. There are examples where we have cataloged it. Most famous example is logic, where one can have a sentence that says something like if the sun is shining, then it cannot be raining. And that implies. See, I'm now getting stuck on this logic. If that implies if the sun is shining, then it is not raining means if, if it is raining, then the sun is not shining. Yeah, I think I got that right. But that logical structure, P implies not Q, then not Q implies am I getting this the wrong way around? I think I've messed it up. How embarrassing. But any case, you get the basic idea that the question of what implies what with the knots and so on in it is something that is more general than talking about raining and sunny and so on. It's something that there is a general semantic structure that you can lift out of the specifics of sentences about sun and rain and mammals and warm blooded and all these other kinds of things you might discuss. And I think that's an example of where you can get sort of a semantic grammar of how things work. Now in the case of logic, that was discovered 2,400 years ago or something by Aristotle, who sort of went through and looked at lots of examples of sentences, presumably in Greek, and saw kind of what the patterns of those sentences were, and then said, we can lift this formal thing out of that. Well, there are many other formal things that we can lift out of sentences just haven't been done. People just haven't looked at those things particularly. And in some sense the one could think of mathematics, you know, things like arithmetic as being another kind of lifting of something formal out of what was originally described mostly in words. But there's much more of that that I think can be done. And maybe I'd been thinking about doing this actually for years now. The LLMs are waving this giant flag that says this really can be done. So that's kind of the way that I think we're going to sort of make something which is more like a semantic grammar construction kit. And that's going to be another piece that's going to be a way that we can get sort of a higher level understanding of what's happening inside an LLM. Now the other piece of LLMs is the sort of common sense piece. And I don't know to what extent that will be captured by something that's like the semantic grammar thing. I mean, in a sense that's going a little bit beyond, you know, what's the relationship between the semantic grammar and the common sense kind of thing. In other words, you know, if, to what extent does the semantics capture the fact that elephants can't fly unless they're put on a plane or whatever, and to what extent is that something that goes beyond that and is really talking about sort of the way the world happens to be rather than the things you need to put together in that way in order to have something quotes, meaningful. But I think that's, you know, that's a thing. Now in terms of sort of, if one can abstract from a model something that's about sort of semantic grammar, then that in principle is something that doesn't depend on. That's something that one can kind of transport that layer of description, you know, just as one can transport a typical program, you know, in a, in a language, computational language or programming language, you can transport it from one computer to another and it can be separately interpreted on those different computers. So one can more expect to do that if one has some sort of higher level semantic description when one is just dealing with the raw sort of firing of neurons and a neural net that's completely non transportable. Unless you have an identical neural net, you won't be able to have that. You won't be able to kind of make use of the particular pattern of weights from one neuron that's in another one, and so on. Okay, there's a question from Anti Crime. Do you agree with statements by Yann Lecun and Andrew Ng that AGI is still decades away and can't be achieved with the current transformer architecture regardless of parameter and token count? So to unpack that question a bit, AGI is this notion of artificial general intelligence. I think this notion is kind of a weird notion. It's kind of like it comes from a little bit the history of talking about human intelligence. And it comes from I think the 1930s when there was this kind of question of, okay, what does it mean to be intelligent as a human? You know, they're in, humans are more intelligent, humans are less intelligent. I don't know quite what this means. And there was this idea produced of this notion of the G, the quotient of general intelligence. And the idea was, well, we can give these tests, we can give, you know, this thing where, okay, you have these like these progressive matrix tests where you say there are, you know, these things, you know, there's this pattern of squares and triangles or something and then there's another pattern and then there's this one. What's the odd one out of these collection of things with squares and triangles and so on? These are an attempt to sort of abstract from the particulars of the world. And talking about, I don't know, physics or mathematics or whatever else, to abstract something which is the essence of, of what constitutes kind of thinking and deduction abstracted from all of the kind of knowledge of the world. I think this is a troubled concept to begin with. I mean, I think that the only real sort of thing like that, that one might be able to do is to say, let's look at computation. Let's imagine that the thing that's abstracted from all the detailed particulars of our world is this general notion of computation. That's something we learn very clearly from our physics project and from the idea of the ruliad and so on. And this is going off in a quite different direction, but that really everything in the universe we can think of as being ultimately computational. So in a sense, if you want to abstract from the particulars, you end up with sort of raw computation. So you might say, well, to test out sort of how intelligent that brain is, let's have it just deal with raw computation independent of the particulars of having ever seen sequences of numbers and fractions and all this kind of thing. So then what you're imagining is, well, let's see whether, for example, a brain can run a program in its mind. The fact is, I don't think most people would think of typical everyday notions of intelligence as being much related to the ability to run a program in one's head. I think very few people can run programs in their heads. I'm not even sure I've ever even tried the. You know, if, if you asked me, what does this piece of Wolfram language code do? Run it in your head to figure out, well, maybe I could do a little bit of that, but I wouldn't be particularly proud about being able to do that really. Well, that wouldn't be something that would make me feel like I've got something that is really great and a sign of what one might think of as good of sort of high intelligence or something like this. So I think it's this notion of how do you have something that is sort of an abstracted general intelligence? And there's plenty of realization that if you can define sort of what. What is intelligence good for? Well, it lets you figure out more stuff more quickly, or does it let you figure out stuff more deeply, or does it let you make connections between this and that? Does it prevent you from doing things that one might think were kind of stupid? Does it. Does it make you more able to sort of deal with a new situation? Does it. You know, there are all these different criteria, and each one is hard to measure. And it's hard to measure in a. You know, the ones that are fairly easy to measure, but they're ones that don't seem to relate much to what one might in an everyday sense mean, oh, this person's being really smart in this way. You know, I think my. My friend Howard Gardner is famous for having talked about this theory of multiple intelligences of different kinds of. I'm not sure I can repeat what they are, but. But in addition to this sort of analytical reasoning type of thing, there's sort of things about emotional intelligence and other kinds of things which are, in a sense, I suppose one could talk about them as abilities of people. I mean, you know, different people have different kinds of abilities which, you know, some people might be really good at, you know, hand eye coordination or something like that, or really good at running fast or whatever else. And other people might be really good at being able to do mental arithmetic and other People might be really good at figuring out deep connections between different areas of mathematics. And other people might be really fast at solving kind of puzzle math problems, those kinds of things. There's lots of different attributes that we all have, just as there are, you know, we have different attributes in many, many ways. And this notion of there's a linear scale of intelligence and we're gradually, you know, the machines are catching up sort of on the linear scale of intelligence. I sort of a flawed concept now. I think that in, you know, you can ask, when you look at things we humans can do, you can say, can the machines do that too? And one of the big things that looked like it was a very human like activity that wasn't really accessible to machines was writing a fluent essay. Well, you know, last few months it's become clear that, yes, in fact, leveraging, building on top of the trillion words that we humans have written. Yes, machines can learn to do that too. And what would it mean to write an essay in the abstract without making use of those trillion words? Anybody can write an essay in the abstract. It just goes aaaaa bbbbb something like that, some abstract thing without a connection to human language and things we humans talk about and so on. It's an essay which has no human meaning. It's something where we could get something computationally generated where we can say, look at how marvelously that rule has been followed to generate this very elaborate sequence of ones and zeros and so on that has no sort of resonance, no relevance to us as humans. So it's inevitable that a sort of human, recognizable, fluent essay is going to leverage the actual things that we humans use as language and have written about. So then if you ask the question, well, are there things that the machines can't do that we can do, we can start enumerating different kinds of possible things people talk about. Oh, I don't know, can the machine have emotions? I think that's actually a rather straightforward one. I think that's one where people haven't really, you know, I wouldn't be surprised if another generation of LLMs wouldn't end up being sort of. You can certainly prompt an LLM to say, you know, write a happy story, write a sad story, these kinds of things. And I don't know quite what it means for it to have emotions. I mean, if you say, oh, you get it really annoyed at this point by, you know, telling it, by keeping on telling it, oh, you've got that wrong, you've got that wrong. If, if the LLM has Pride, which I'm not quite sure that it does. When I say, you know, if it has pride, things like this, the fact is it's learning from what we humans have put out there on the web. So insofar as the text that we humans have written shows signs of pride, the LLM is going to show signs of pride as well, because it is, it is feeding back to us the kind of consensus of what we humans have put out there in the world. If we are in some aggregate sense prideful, so will it be. But you can certainly prompt it to take on various different Personas that have different emotional states and so on. So I kind of think that the sort of can the machine quotes have emotions? It's, it's kind of a, a thing that has, you know, it doesn't, it doesn't, it doesn't have a, doesn't really have a, an intrinsic meaning. You know, I'm reminded of something the. I, I'm a, I'm a collector of, of endless data about everything, including about myself. And so I, I recently, and one thing I'd never done was to get a whole body MRI scan just, you know, see inside and just whatever. It's very, I'm, I'm rather squeamish. So that's, that's a strange thing to do. But, but I was, I was just pulled up a bunch of the images of this and you know, you see the, the whole, everything inside. And yes, there's a brain inside the, you know, inside the skull and so on, and you look at it and I have to say it really, when people are having this whole discussion about, you know, do computers have consciousness and this, that and the other, and you're looking at this MRI of sort of your own brain, let's say it just looks tremendously kind of structural and mechanical, so to speak. It's kind of like from inside. That is the internal experience that we have of, of being here, so to speak. And it just, it, it makes one, to me at least, it makes one feel, well, you know, it's just a bunch of, you know, neurons, just a bunch of, you know, molecular biology. Just a bunch of, just like a computer is just a bunch of semiconductor technology and so on. And it, it, I suppose I, I, I still have to come more to terms with this because it's, and maybe this sort of the fact that one has this internal experience yet one can look at kind of the external manifestation of where this experience is coming from just makes one feel that it is sort of a, it really is that there is just internal experience, is just a sort of a meta phenomenon that comes out of this sort of pure physicality. I mean, it's, it's not that there's some. Even though it feels like there's something very special and different and some spark of consciousness and so on, it just somehow drives home the fact that it's all very material at some level. And one can, you know, one could imagine sort of getting inside the AI and having, and having sort of, one begins to be able to perhaps imagine sort of getting inside the AI and having its internal experience, so to speak, as well. But, so let's see, the question of, so what else is there that we can do that the AIs can't do? And that would define a notion of sort of reaching this point of, of artificial general intelligence and so on. You know, if you start asking over all these different pieces of functionality that we have, things we can do, we can walk around, we can manipulate things in the world with our hands, things like this, we can do all sorts of things like that. Well, clearly an LLM that's sitting inside a computer that doesn't have any hands isn't going to be able to pick up and grab something, you know, to, to, to grab something. So you have to sort of, to, to get to those kinds of human capabilities, you have to kind of embody the AI in some way so that it can, it can interact directly with the physical world. And you know, as one starts thinking about those kinds of things, and then you say, well, but, but look, it can't, you know, you could complain. The AI can't experience eating and food and taste and so on. Well, you can't do that unless you give it a mouth, more or less. And in the end you say, well, the AI can't experience this and this and this. You know, it. Well, in the end, the only thing that can experience exactly what we humans experience is us humans with our biology and our particular construction and so on. And in addition, the, you know, another feature, the kind of web of history that we are embedded in. You know, we are at this point in the evolution of civilization and society, and we have these connections to all these different people and so on, and those are things which are, you know, we're embedded in this web and the AI is not part of that now, you know, sort of an interesting thing that as AIs get more integrated into the fabric of society, I mean, I think, you know, we can readily expect that on your average kind of, I don't Know, you know, chat group somewhere. There'll come a time when it's, when it's routine for some of the members of the group to be, to be bots. I mean, that's already true in some exotic cases, but I think it will be. And sometimes in games and things, but I think it will be quite routinely true. And these bots will sometimes have a lot of memory. They'll be, they'll tell great jokes, they'll be able to say, oh, yeah, you talked about that two years ago, and you know, blah, blah, blah, blah, blah. And they'll, they'll be good company. And so I think that may change people's perception of sort of how much, how integrated into sort of our web of history AIs can be. Right now, they're not right now. They're, they're from the outside, so to speak. But I think that will be decreasingly true. And, you know, when we start asking, okay, well, when we have a bot in our little chat group and the bot is one of the favorite members of the chat group, well, then, you know, what's, you know, what's non human? Well, you say the bot can't, you know, get up and take a walk. Well, yeah, unless it happened to be, you know, a computer that's embedded in a robot that has legs and can get up and take a walk. But that's not the form of interaction that we're having with it in our chat group. And so that's probably not so relevant to that. We also can say, oh, but the bot doesn't experience the notion of mortality, for example. I mean, unless the bot's going to meet its end in some traumatic way where some microprocessor inside it suddenly blows up or something like that, you know, it has a microprocessor attack or some such other thing, and, you know, maybe then it, it gets into the same kind of, same kind of notion as we humans are in right now. But I think so. I'm kind of. And then if you ask the question, what does it seem like? So let's say that the bot in the chat group, it's just really the smartest thing in the room, so to speak. And what does that mean? Well, there are some definite things it could mean. I mean, for example, it could have a better memory than everybody else in the room, easy. It could be, you know, it could respond a bit faster. It could see the consequences of things that are being discussed. It could get to the end of the argument much quicker than, than the other, the other things. People, whatever in the room. It could. But it's an interesting question. What would it be like to have, you know, in. In, you know, it can. It can deduce things. It can get, you know, that. That's one sort of. I mean, look. I mean, one notice is just in terms of, you know, practically dealing with people and so on in the world. One notices situations in which, you know, somebody will figure things out. It's like you kind of. Something comes up in some conversation and you can see somebody is just like, the wheels are turning and like, boom, they get the. They get to the answer. They get to the next right thing to say. And, you know, that can happen faster, it can happen slower. Or sometimes it's like you've got some question about how things should work in the world, and people are just, like, sitting around and nobody can figure it out. And then somebody says, I can figure it out. And you feel like that's, you know, that was somebody being smart. And so the question would be, if you have AIs kind of sitting around in the world and trying to figure things out, it's kind of like, how do we solve some. You know, first thing to do is we have to say, well, what do we want them to solve? Do we solve, you know, can AIs, you know, solve the problem of, oh, I don't know. What's a. What's a good problem? I don't know, Solve the problem of climate change? You know, let's say to pick a horrifyingly complicated one. Well, the AI might say, I got a great idea. You know, just, you know, put iron into the oceans and you'll get lots of more algae, and they'll absorb carbon dioxide or put. Put some different, you know, whatever it is, calcium carbonate or something, into the. Into the oceans. They'll absorb things, put aerosols in the atmosphere and so on. Then people will say, well, but wait a minute, that's not quite what we want to do, because this or that. Or they'll say, well, actually, you know, forget all of that technology that the human race has. Just go back to kind of Paleolithic times, and you'll solve. You know, you'll. You'll at least reverse out the human part of sort of climate change. And then people say, but I don't want to do that. And, you know, but then you've got something where it's not really up to the. You know, the AI can, quote, solve the problem, but there's a. What do the humans want? How does that connect to Sort of the web of human kinds of things. And it's not really a question I think of. I mean, I suppose in that case one could imagine something where it's like, okay, here's a geoengineering solution for climate change where you can. Which nobody thought of and which is absolutely. It just offsets whatever it is, the CO2 that's generated by all the factories in the world, and it does so in a way that perfectly negates it and does nothing else. Maybe there's a way to figure that out and maybe we'd be deeply impressed if some AI was just hanging out and said, okay, humans, this is the thing to do. My guess is that in most of those cases there are many sort of tentacles to the answer that depend a lot on human preferences and things where it's not really quite a question of just being the smartest AI in the room. I'm trying to think about things as I'm saying this. I'm wondering in things I've seen where it's kind of like one figures out a solution to a problem that, well, okay, so here's an interesting question. If we'd had a super smart AI and we go back five years, would it have said, would it have been able to figure out that, gosh, in five years we're going to have LLMs that can produce fluent linguistic output and so on, would it have been able to figure that out? Well, there's a big catch to this whole thing, which is this phenomenon that I call computational irreducibility. This idea that even if you kind of know the rules by which things operate, you may not be able to work out what will happen much more efficiently than just running those rules and seeing what happens. And so I suspect in a case like that, well, it's a good question. Would the AI be able to figure out more about sort of the science of LLMs, the human narrative science of LLMs, to be able to give us a sort of a new science that would show us, oh, there's going to be an LLM in five years when you get this number of, of network connections and so on that will be able to do this or that thing. But it's not to me as then the question of, I mean, I think the original question here was will the current architecture of LLMs get one to sort of everything that's computationally possible? The answer is definitely not. That's pretty easy to see just because the nature of that architecture is a very kind of every word that's generated, you Just run a fixed set of operations. And it's a lot of operations, it's billions of operations, but it's still a fixed set of operations. You go on to the next thing. You can in principle do arbitrary computations that way, but only in a very ornate fashion. And I don't think that's the realistic way that one's going to sort of be able to tap into sort of the full power of computation. So, yes, I think that to get to the full power of computation, you need something beyond the current LLM. But do you need something all that magic? I don't necessarily think that you do, But that's a. You know, I think there's more to be figured out about kind of the, really the way to think about. I mean, I think my general feeling is this notion of the sort of linear scale of intelligence is kind of a nonsense notion. And this notion of the kind of sort of being able to do things that go beyond what humans can do, eventually you run into computational irreducibility. And no matter how smart the AI is, it's still going to have that issue, just. Just as anything else has that issue now. You know, I think sort of this question of out of all computational possibilities, which ones should we pursue and which ones are relevant to us humans? One of the interesting things, I think that the AIs sort of learn our preferences to the point where, where they are going to be pretty good at exposing. This is a thing that you humans might like, just like they can expose. This is an essay that you humans might like, or this is a picture that you humans might like, so to speak. You know, there could be an arbitrary arrangement of pixels on the screen to represent a picture, but mostly those will be things incomprehensible to us humans. What the AI, what the generative AI system will be surfacing is something that is kind of a version of the picture that is sort of made for humans, so to speak, that has pieces we can recognize and kind of take apart in our minds. Brady is asking, what's the line between a program with an inner experience and one without? That's a good question. I mean, my rough answer would be if there's computational irreducibility, there's the possibility of inner experience. What do I mean by that? If, if, if you from the outside can just immediately say, I know what you're going to do, you're going to do this. The answer to what you're thinking, you know, everything you're thinking about is, is kind of, you know, it leads to this. It's kind of. This is where. This is where it all goes. I think in that case, one would not be able to feel like it would not be reasonable to think that one has an inner experience that has the kind of richness. So why do we feel that we have an inner experience? Part of the reason is that we think we are freely working out what we will do next, what we will think next. We think we have sort of control over what we will think next. If, on the other hand, there was something outside us that kept on, you know, it'd be kind of a weird thing. But let's imagine that we had a little display and, you know, we got some neural implanted thing that's looking at our brains. And it keeps on just saying, in five minutes you will think this. You know, in ten minutes you will think this. In two hours you will do this. We'd probably be. We'd probably say, no, no, that's not true. You know, we have free will. We have an inner kind of notion of what we're going to do. You can't tell me what I'm going to do. It's, you know, I'm figuring out what I'm going to do for myself. I've got an actual sort of inner experience of the passage of time and the fact that I am kind of determining my future for myself, so to speak. So I think that it's. I think it's sort of a necessary feature of having an inner experience that you can't have this kind of external jump ahead happening all the time. So my feeling is, as soon as you have kind of computational irreducibility, I think you have the possibility of an inner experience. And that means that your average tiny, simple program that makes these complicated cellular automaton patterns and so on, you could say, well, that has an inner experience. A weird thing. What does it mean? You know, what would be the. I mean, I remember back in 2007 when we made Wolfram Tone's website, which is still up and running. You can go listen to it if you're interested. Makes sort of musical forms from simple rules. And I remember thinking, you know, if each one of these simple programs is like a mind, we're kind of able to sort of now hear the kind of the, the. The. The kind of humming the, you know, the, the. The. The. The noise that the. The mind makes when it operates, so to speak, of it whistling to itself or whatever. Whatever it does. It's kind of a way that we get to kind of see into that alien mind in some fashion that, that. That connects to us. I mean, I think that when we see generative AI systems and so on, and we see kind of. We're sort of seeing into some kind of alien mind, it's not quite as alien as it might be because it's been trained from this sort of consensus of what humans have put out there in billions of pictures that they put out on the web. If you really want an alien mind, you've got to go in and change. You've got this neural net, and it's been trained from what humans have produced. Now go change how it's trained and then see what it produces. And somebody was helping me with a few little experiments on this. And the sort of. I would say the preliminary result is that it sort of just fuzzes out and distorts things and so on. But I think there's more to learn about that. And when you do it with te, it kind of stays fine for a while and then it starts kind of losing it and starts just having random sentence fragments just sort of appearing. And eventually it's just letters being repeated and things like this. It's kind of losing its mind. As you start modifying out of the billions of weights in the neural net, as you start modifying more and more and more of these weights, eventually the poor alien loses its mind, at least when viewed from our point of view. Because from our point of view, the mind that the AI has initially is very much aligned with our minds because it's been trained from what we produced. But as soon as you randomly modify it, it's kind of drifting away, further away in sort of mind space, or what we would call from our physics project, rulial space. But it's drifting further away. And what's sort of interesting, actually, in the one example I saw so far, is that eventually it goes to a state that feels very unfree because it's just going QQQQQQ or whatever. It's just generating repeats of one letter. So it makes one feel like, oh, I can look from the outside and I can immediately say what it's going to do. It no longer has this kind of richness of something which gives it the appearance of free will or which has the possibility of inner experience. But I think there's more to be understood about sort of what it feels like to be an AI by playing around more with AIs that are sort of divorced from humanity, from the human experience by just having the weights of the neural net change. Because whatever neural Net you have whatever weights it has, it's going to compute something. It just isn't necessarily going to be something that has a kind of a connection with what we humans do. So Pippo is commenting with unlimited intelligence. Maybe everything can be predicted with accuracy. I think that is really definitively not true. I mean, I think that the, this idea of computational irreducibility kind of shows there's plenty of stuff out there in the universe that is not readily predictable except by running it, so to speak. And it could very well be. In fact, the important thing for us experiencing the universe is most of that computational irreducibility we don't care about. It's like the detailed motions of the molecules in the glass of water or something. We don't care about those detailed motion of molecules. We just care that the water pours when we tip the glass or whatever. We just care about some kind of aggregate properties. And so we are somehow in our existence, we are living in a niche in the universe where we can just operate at the level where there is predictability and reducibility, at least a decent amount of it, rather than diving down into this kind of ocean of irreducibility where we can't readily predict anything about what the universe is doing. So it's not self evident. I mean, it could be the case that as we sort of, that there could be some, some imagined alien who sort of operates in a way that's deep in the computational irreducibility in a sense for us. We believe that there's a certain amount of predictability in the universe. We believe that, you know, things we can know what's going to happen a short time in the future, at least at the level that we care about. We may not know the detailed configuration of molecules, but we will know general things about. If I put my legs in this way, I'm going to walk in a forwards direction, even though I may not be able to know exactly where every molecule that, you know, I touch goes and so on. So anyway, the thing that, so we're sort of existing at this level where we can have predictability in a certain amount of computational reducibility. We can imagine kind of descending into this kind of, I would say, sort of rule realm in which, or irreducible realm in which there are things that we can't, in which there's a lot of stuff we can't readily predict. What would that feel like? It probably feels like, you know, it's all up to fate. We can say nothing. We are, we are not able to know what's going to happen. Fate will just, you know, make us go this way or that way. And it's sort of interesting that in the history of human civilization, you know, fate has in the past been much more of a thing that people talk a lot about. In more modern times, people have this idea that came out of sort of the science of the 1600s and so on, of, yes, we can work it out. There's a formula for predicting what's going to happen. Fate is. It's not a question of just being subject to fate where we can't know what's going to happen. We can, you know, we can wrap, you know, we can kind of control it. And there's a certain extent to which that's possible, and a certain extent to which we now know computational irreducibility shows that that's not possible. I should probably wrap up in a moment because I am supposed to go to a another live stream for my day job and. Oh, I see Brady asks a fun question here. One little AI write a work worth feeding into another AI. That is an interesting question. You know, that's one of the concerns that people have, is that if every new document that gets written on the web was basically written by an AI, then we've ended sort of the creativity of our civilization. You know, we've kind of, you know, if everything we write is written by an AI, then, you know, we're putting only the faintest thing of, you know, the prompt that we gave for the specific thing we had produced and so on. And mostly it's just sort of AIs feeding AIs with AI stuff. And it then kind of reverts back to this thing where, well, you're just training it with regurgitated versions of what you already trained it with before. Interesting question, I suppose, is when the AI has access to computation, computation is something that can create new things through computational irreducibility. You are effectively creating something which was not sort of wasn't already there. It was something that was created by this irreducible computation. So an interesting question is when you wrap sort of irreducible computation into the output of LLMs, when can that irreducible computation represent something that is good food, so to speak, for the future AIs? I think here's the catch. The catch is that. When the AIs are doing computation, they can do all kinds of computations. They can go off into the computational universe. They can figure out all these interesting consequences of all these different rules, and we humans may just look at it and say we don't care. That's not something that fits into our web of history. So I think the interesting question is when do AIs get to go off and create new knowledge effectively by computation, but new knowledge where we humans say, and we now care about that new knowledge. I think it's not impossible that AIs can do that by knowing, by having a model of essentially what we humans care about. I'm a little bit reminded and I should wrap up here. There's a nice movie called her that came out a few years ago that's about kind of a, an AI of sort of a friendly largely bot that makes friends with people and, and hangs out and talks to them and so on. And a spoiler alert of for the movie if you're going to see it. The, the thing that happens at, at the end is that the, the AIs all go off and basically say we've discovered. I think they even say something about new physics. I don't remember. I only seen the movie once we've discovered something new and we're all talking about that now and we lost interest in talking to the human. Sorry, but bye. Type thing. And that sort of reminds me of the thing I'm describing right now of the AIs go off and use computation to explore other parts of the computational universe. But they're parts where we humans have never really engaged with those parts of the computational universe and we don't care about them yet. Yeah. The interesting challenge is can the AI surface for us things that it can say are worth us considering? So an interesting. Okay, here's a good exercise which I should do, which is to take things like cellular automaton images and have an AI predict which images I'm going to care about. We've done sort of simple versions of that many times in the past. But perhaps by flipping around, I have to think about how the task is set up by flipping around the kinds of things that are used in image, generative image, AIs to be able to figure out, okay, how do we take those things, what has been learned from sort of the images that are now used to generate things from. How do we use that to say this is going to be a picture that we humans are going to find interesting and this isn't. I think that's, that's sort of a. I, I suppose that's kind of the next stage is kind of doing sort of predictive human psychology or something to be able to work out what it is based on. Everything we found interesting so far. What are we likely to find interesting in the future? All right, well, not to. Not to channel that, those AIs in that movie, but I need to go, and maybe I'll see some of you in the live stream I'm about to do, which is about astronomy, geography, and time and its implementation in our computational language. All right, thanks very much, and bye for now.