History of Science and Technology Q&A (January 11, 2023)

Hello everyone. Welcome to another episode of History of Science and Technology. Q AND A. I'm happy to try and answer questions about things I've been involved in or things that I might have studied in the history of science and technology. And. Let's see, questions here about the discovery of logic and relationship to modern machine learning and things like that. Well, gosh, let's see. Well, let's start off with talking about the history of logic a little bit. So I think logic dates to antiquity and was part of the kind of philosophical tradition of antiquity. I think that the Stoics had various forms of logic, but I think the real sort of official logic that we know today had sort of its starting point with aristotle in the 4th century BC, I guess. And Aristotle's big idea was take pieces of rhetoric. Somebody says, you know, because Athens is better than Sparta, and this is that, therefore this, these kind of chains of argument. And his goal was to catalog chains of argument. And Aristotle was a big kind of sort of early encyclopedist. He liked to catalog everything from forms of sea creatures to causes of things to all kinds of other types of kind of organization of information, sort of the cataloging of things. And so one of the things he wanted to catalog was the structure of arguments. And I don't know, I don't think he invented this. But by the time of the Middle Ages, when Aristotle's works were kind of the key works of science that were kind of the main thing transmitted from antiquity in terms of science tended to be the works of Aristotle. And Aristotle had some quite sensible things to say and some absolutely crazy things to say that were, you know, I just imagine Aristotle one day just saying, hey, I wonder how this works. And saying, well, I think it might work like this. And he writes it down and that becomes a piece of the canon of science for the next 1500 years or some such. And some of what he concluded was kind of crazy from a modern point of view, and some of it was kind of like, yes, that's the obvious thing to conclude. And it's good somebody wrote it down. And that was the thing that people learned from through the Middle Ages. But so it was with so called syllogistic logic, the typical structure, there are these different structures of syllogisms and they all have the form of things like for all blah there is a blah. For all blah there is a blah. Therefore for all blah there's a blah. And I think the famous one is all men are mortal. Socrates is a man, therefore Socrates is mortal. And Each of those forms of syllogism there were like 15 of them or something which are essentially the combinatorial combinations of for all and their exists and so on. Each of those got a mnemonic. In medieval times, I think famous ones were Barbara and Celerant I think were two famous ones and Barbara somehow has the. Maybe it's in Latin, I'm not sure. Has. Has the pieces of that for all and there exists and so on. And that's kind of the structure of syllogistic logic is this kind of these putting together these pieces and saying there's this pattern. And the idea that there would be more to logic than this kind of structural pattern took a very long time to emerge. That wasn't until Boole in the 1830s, I guess 1830s or 1840s who kind of made this connection between logic and algebra. The idea that you could actually have sort of huge collections of logical expressions, much as you could have sort of huge collections of terms and things like polynomials. Leibniz had had some things to say about this, but I don't think it really formulated that same kind of sort of connection to algebra of logic. The idea that you could take expressions in that were stated things that were stated in terms of. In terms of words and so on and reduce them to something formal that was like algebra and so called formal logic. People like Charles Dodson, AKA Lewis Carroll were involved in this. In the, in the mid-1800s Charles Dodson wrote a famous book about logic and not as famous as Alice in Wonderland, but that was his, His. His sort of day job was. Was studying logic. You know, I was reminded in of the Alice in Wonderland thing. I. I happened to just be finding some books of mine that I hadn't seen in 50 years. I just opened some box and then there were some old books and one of them was a Greek to English dictionary lexicon that was sort of the most famous Greek to English dictionary. That was Liddell and Scott were the two authors of that. And the thing that's notable about that in connection with what I was just talking about is Alex Liddell was the daughter of the Liddell who wrote the lexicon and she was the Alice in Alice in Wonderland, so to speak. Some child of some Oxford classics and other person. Anyway, but back to sort of the progress of logic, I think then this kind of pattern of what does a valid argument look like? Is something that one sees kind of the syllogism idea goes through this long period of time of people learning Aristotle basically as a way to learn science like things in the middle ages and finally gets mathematicized with Boole in the early 1800s. Meanwhile, people like Leibniz in the late 1600s had had things to say about this because Leibniz, for example, his PhD thesis, for instance, which he wrote when he was pretty young, was about resolving legal cases using logic. So I think his idea was you could take a legal case stated in some way or another and you could apply essentially what must have been at the time, syl. Justic logic, to just sort of knit the pieces together to find what the conclusion should be. So that was kind of this idea of what would be a valid argument and how would that be set up. It got mathematicized with Boolean. By the end of the 1800s it was even more mathematicized and these different forms of kind of formal logic came in. So with Boole, he was really dealing with ands and ors and nots and he was dealing with thinking about an and like a multiplication, an or like an addition, a not like a negation, these kinds of things. And you know, his real sort of breakthrough was realizing you could do sort of sloppy algebra where one plus one equals one, so to speak. And it isn't quite modular arithmetic. It's something a little different, it's something a little sloppier. But that that form of what's now called Boolean algebra could represent these logical operations. And what then happened, I think towards the. This must have been probably in the early 1900s, these so called modal logics where you would sort of tag these expressions in different ways. You'd have things like temporal logic where you would have things which specifically talk about time, things that instead of saying for all, oh, I should have explained also predicate logic. So Boole's logic was just ands and ors and nots. There was also the formalization of there exists and for all Leibniz that had formulations of that. In a sense, syllogistic logic has some formulations of that. But the sort of the formal version of that I think was really Gottlob Frege's work in the 1870s, 1880s kind of timeframe of really formalizing what would now be called predicate logic, of being able to say, you know, for all blah, for all X there exists a Y such that, et cetera, et cetera, et cetera. And resolving those kinds of things. I mean, notably one of the things that is sometimes confusing in the history of logic, Godel in 1930, right before he did Godel's theorem, the famous Godel's incompleteness theorem of 1931. Godel proved his completeness theorem, which was that pure predicate logic is complete in the sense that in pure predicate logic, when you've just got for alls and there exists, you don't have any non trivial predicates, like things talking about arithmetic and things like that. Pure predicate logic is complete in the sense that it is the decidable theory where you can just grind through and resolve whether something is true or not. That's also the case with ordinary Boolean algebra that you can just. In ordinary Boolean algebra, you can ultimately just say, you know, p and Q or not, whatever you know. Is that a tautology? Is it always true, for example? Well, one way you can resolve that is just take your P's and Q's which are either true or false, and just fit in all the combinatorial possibilities for the trues and falses. You make a truth table and then you just look, is it all trues in the truth table? It's a strange thing that a truth table might seem like an incredibly obvious idea that you might have thought people like Leibniz or Boole for that matter would have come up with. I think it actually didn't emerge until probably the 1930s or so, and it was. Wittgenstein, for example, talks about truth tables. Whether that was original to him, I'm not sure. Emil Post also talks about truth tables. Emil Post was the person who looked at the Post correspondence problem and the Post tag systems and these kinds of things. He was involved in kind of an early attempt to find a kind of minimal formalization of mathematics and of computation. But any case, he talked about truth tables probably in the 1930s or so. But that idea, which is the thing that sort of shows you that standard Boolean algebra is decidable. You can just figure out everything about it by just enumerating all those possibilities. That took surprisingly long time for people to do what would seem to me to be the sort of obvious computational kind of thing to do. Now, of course, when you say, well, I'm going to find out whether the thing is always true, or another question is whether it's ever true. Is there any collection of P's and q's and so on such that this logic expression is true? That question of whether there's ever a case where it's true is the so called satisfiability problem. And one question that comes up is, is there a faster way to work out whether the answer is true than by just substituting all the exponentially many possibilities and seeing whether one of them turns out to be true. Well, this problem of satisfiability was a problem which in 1970 Stephen Cook figured out was an example of an NP complete problem. A problem for which the figuring out the answer to that is equivalent to looking at the evolution of an arbitrary non deterministic Turing machine. So in other words, basically. You can't work out the answer to this. If you could work out the answer really quickly to knowing is there an assignment of truth values such that the thing is true? If you could work that out really quickly, you would be able to solve all NP problems in polynomial time. You would be able to, and you'd be able to do, you know, factoring in polynomial time and all kinds of other problems in polynomial time. But that satisfiability thing, which is the yes, in principle you can work out the answer. It's a finite computation, but it might be an exponentially large computation in principle. But this whole question about what makes kind of a valid statement, what is logically correct, that is a kind of an idea that goes all the way back to people like Aristotle and even the Stoics perform and so on of kind of what are the rules of construction? That makes something true, that makes something a valid chain of reasoning, so to speak. It's sort of interesting that the interplay between that and Is an equation satisfiable? For example, is there a solution to the equation this, that or the other? These two traditions have evolved somewhat, somewhat connected and somewhat disconnected. The tradition of logic and the tradition of sort of mathematics. They were, they were, you know, they were separate subjects people studied in the Middle Ages, for example. They. It's always a sort of an interesting feature of history that mathematics locked itself into the, the educational curriculum. Logic did not. Mathematics, everybody learns mathematics. Very few people take a class about logic. And mathematics has turned out in a sense to be a much richer field, but for the following reason. It's richer because there's more that one can explicitly do in the case of logic. One gets a certain distance and then one runs into what I would now describe as computational irreducibility. And basically you say you can't work that out with any simple argument. You just have to do a computation and see what happens. And that's a rather non human thing to be having to do. So I think that's sort of why mathematics, mathematics has a lot, much longer on ramp, so to speak, of things that you as a human can do before you hit some boundary of undecidability and computational irreducibility and mathematics sort of successfully defines itself to look at things where that usually isn't the issue. So, but any case, I think this question about sort of the making valid statements. Okay, why is this relevant right now? Okay, so you look at, you know, Chat GPT, for example, and it has this language model, which is a statistical model for the structure of sentences. And it says, well, you know, there is a, this is a likely sentence that one might have produced. And you know, if you look at, well, slightly more primitive versions of, of ChatGPT, I don't know exactly how it's been engineered to avoid this, but there's a. You end up with something where, yes, the most likely sentence to produce is one that eventually just keeps repeating or something like that. You have to add a temperature parameter that kind of, you know, adds some noise to the situation, so that doesn't happen. But the, and by the way, oh, gosh, this is, this is such a, you know, there are these kind of themes that show up of sort of awkward situations that show up in the history of science. And this is one of the, one of the famous awkward situations. It's like you say, well, I'm iteratively producing words from this system. I'm producing the next word, the next word, the next word at every step. Let's say I pick the most probable word. I've got some whole algorithm and it ranks words in terms of which is the most probable. And then do you pick that one? Or do you pick sometimes, do you pick the second or third most probable? Would you just keep pegging it on the most probable word? This is very similar to questions about things like natural selection in biology where you say, okay, you've got these organisms and you're doing an artificial version of natural selection. You're simulating natural selection. Do you always pick the fittest organism or does that lead you to a more brittle situation? Is it important that the not quite winner be actually considered to have a variation that can go to the next generation? Is it always the case, you know, if you have, if, if you kind of select for the toppest, so to speak, in every, in every generation and you only look at variations from the top organism in every generation, do you end up with a worse result than if you say, well, let me go down to the fifth most most successful organism and let's have a variation of that? It's kind of like, like, you know, you have the parents and the. If you have. If you only have the perfect parents, do you get the more perfect children, or do you sometimes have to go to the less perfect parents? Because a variation from the less perfect parent is going to lead to an even more perfect, even fitter, so to speak, child than if you went from the. The. The sort of. The. The. The. The fitter parent. And you can kind of see how this can happen in the structure of these kind of complicated spaces of where fitness occurs. But any case, the thing that then is a feature of. So that's kind of in these statistical language models, you're saying, well, make me a sentence that is a reasonably probable sentence, and hopefully that will be a sentence that makes sense. And usually it seems to do a remarkably good job of having a sentence that somehow makes sense. But there's sort of a question of could. Can one fit together sentences that make sense using some rules, like one does with logic? Logic is a way of making sentences that make logical sense. In some sense, they can be about things that might not even exist. If you say all P's are a Q or et cetera, et cetera, et cetera, you might formally write that down and be able to derive things on the basis of the rules of logic. But it could be the case that there are no P's in the world. It's just a thing where you can formally make this derivation. But it really isn't about anything. And philosophers go to a lot of trouble to distinguish between things like sentences and statements. A sentence is just the thing you write down that has those words in it. A statement maybe is something that you can actually decide is true or false because the things in it have some real existence, so to speak. These are subtle issues of philosophical logic, which people have argued about for centuries, so to speak, for me, as an actual constructor of computational language and so on. The thing that's always interesting about those philosophical debates is in the end, one has to write code. And these philosophical debates have a habit, in my experience, of sort of dissolving when you actually say, we got to write code, it's got to be this or that. We've got to decide what to do. And then the only question is whether the thing you decide to do. People say, oh, that's hopeless. That's not what I mean. Most of the time it's totally confusing. That's a bug in the design. That. That. It's a bug in the design is kind of the modern flag for. There's some issue in philosophical logic, so to speak. But this question of Whether one can sort of decide, like logic puts together a certain kind of structure of things, but without any reference to things, real things in the world, without any reference to things which might exist or not might exist, et cetera. And the question is, is there sort of a semantics where one can some situation where one can sort of feed reality into the logic? And people have tried to do this a variety of times over the course of the 20th century, for example, with various kinds of logics that sort of tag more things about the p's and q's and so on and tag whether they're probable and whether they exist at certain times and all those kinds of things. In a sense, Aristotle had already started talking about this because he starts talking about categories of things and you know, things that are qualities, things that are, oh, I don't know, maybe mental states we might call them today. Things like this, that tradition of kind of having these categories. And then I don't think Aristotle really got too much into what categories can fit together with what other categories. He didn't, he didn't really do that. That would have been a sort of a generalization of his logic to be able to do those kinds of things. That idea of sort of what categories fit with what categories has a pretty lame history. I mean, Leibniz looked at it a bit, Kant looked at it a bit, I think Hasurl looked at it a bit and it had some sort of, you know, some vague presence in the 20th century. There was an, I think a Judewitz in the 1920s and 1930s who was trying to make kind of a, an algebra out of grammar, looked at it a little bit and that, that led to what are now called categorical grammars. Categorical grammars? Combinatory categorical grammars, I guess is the full name. And that's kind of a scheme for asking. So in a standard grammar, a little bit like logic, but of a different character. In a standard grammar, you say you fit together sentences by having a noun, a subject noun, a verb, an object noun, things like this, or a subject verb phrase with adjectives as well as nouns and so on. You're giving a very structural way of the way that the words are assembled and there's no guarantee that the assembled words will actually mean anything. The famous example given by sort of the early AI people is the phrase green ideas sleep furiously, which is syntactically correct. All those parts of speech are right. But what that means non poetically is very unclear. It's. I think so. So the question is, is there sort of a level of that that is more semantic, that is more. Not just we fit any noun with any verb, but we say something about what the verbs and nouns mean. And that's something that there's been a little bit of work done on, but not as much as you might think. I mean, there was a, a theory or I was going to say that the way that those grammars were constructed, one way you can think about it is that those grammars are kind of like an algebraic system. They have. They have rules where you say you put this word in. It's like you add another word. It's kind of like multiplying some, something or making words in a group where you're taking elements of the group, abstract elements of the group, and you're just sort of combining them together by what amounts to multiplication. And then you have rules that say what are possible sequences of words, sort of mathematical rules that define that. That was an early idea that's been reborn multiple times, but it's being even thought about these days in terms of categorical grammars. The sort of. The other idea that came in the 1950s, although it had had precursors back even in antiquity, was the idea of kind of tree structured grammars where you say, I've got a sentence, I'm going to make this diagram where I have, you know, the noun phrase, the verb, and I'm drawing that as a tree. And the noun phrase then has a sub piece that's a noun and two adjectives or something like this. So that is that. That kind of structure of, of how do you put together, put language together syntactically is something that has become decently well understood in the last 50 years or so. People had written grammar books even back in antiquity. But this idea of a more sort of formal computational version of how you put that together is a fairly modern idea. The corresponding thing for semantics really hasn't been done. The. The idea of doing that where you are putting together things which have only a certain meaning. The closest analogy to that is basically what I've done in our computational language, in Wolfram language, trying to represent sort of things in the world in some precise computational way and then say what goes together with what? And there again you have kind of an interesting thing that I have to admit. I'm still grappling a little bit with the philosophical understanding of how this all works. But you know, you can syntactically put together all kinds of things in Wolfm language. You can take, you know, some city and you can take it to the, you know, multiply it by some other city and you can add that to a star and things like that, that is syntactically valid in Wolfram language. Those things exist there. They are syntactically writable down. But what that whole thing means, it doesn't really have a meaning. The way we represent that in Wolfram languages. It just stays unchanged. It's just not transformed in any way by the language. But then there's this notion of evaluation that if you say what is the population of this city, you know, a city, and then it's some population construct there, then that thing can evaluate it has a meaning, which is sort of an actionable meaning, which turns it into something else. But this idea of what sort of are what are valid forms of construction, semantic forms of construction, we have a very well defined notion of that, which is things that can be resolved to other things. But there's also potentially a version of it where one's just talking about things that can be stated and where you can say this is, this is true, so to speak. It is a relation that can be stated as true and then other things deduced from it. It's kind of like you might assert the equation x plus y equals 7. And that's an assertion. That's not actually saying what the values of X and Y are. It is merely asserting x plus y equals 7 is true. And then you can deduce other things from that. And I think there's a similar kind of thing that one can do where one could say something which is syntactic but not semantically meaningful, say that's true. But what that means to say that's true is not very clear. If I just assert, you know, X open bracket, Y comma, such and such, without knowing what these things are, I can assert that. But what we can deduce from that is not clear. It's only if the thing has a well defined understandable meaning that we can expect to make a tower of deductions from it. Now, one of the things that's sort of interesting is that in the case of logic, it took until the 1800s and beyond for people to start sort of building from the original ideas of ans of wars and knots, building these kind of towers of complicated logic which lead now to microprocessors with a billion logic gates on them. Things like this, that where we're making sort of deep use of this kind of stack of logic. And it's an interesting question if we have similar kinds of things, not for sort of the Pure kind of unmeaningful logic. But something which is real semantic content, what that kind of looks like when you have these giant stacks of kind of semantic construction that are being created there. One of the things that I think may happen is that us humans, if we're doing logic, yeah, we can handle basic syllogistic logic, we can all pretty much intuitively handle it. If you try and do something like taking, I don't know, I found the simplest axiom for Boolean algebra back 20 years, 23 years ago now. And you know, the proof of that is this very complicated machine generated proof which is all perfectly logic. It's all just applying logic, but it's incomprehensible to humans. And the, you know, we might think that the semantic chains that we would get would be, oh, that'll be understandable by us humans because it's all about things that mean stuff. I'm sure that won't be true and that we can already see that in the operation of programs to some extent. But I think seeing that about sort of large scale things about everyday life, so to speak, that we can idealize and then make giant stacks of them, that's kind of an interesting case where sort of seeing how that really plays out will be a little bit, I think, interesting how non human that gets how quickly and quite possibly a lot of things that we might say about, oh, I don't know, figuring out kind of the way that a society works given individual people in it with certain kinds of sort of things that they do and so on and ways that they respond to stimuli and so on. It could very well be that when we have this kind of sort of semantic structure that's the analog of logic that we, we may get some slightly different intuition for how that kind of thing works because we've only had, we haven't really had this kind of meaningful to human sort of stack that we can build in a, in a tall way from kind of formal structures so that that's something, something might happen. But you know, this whole question about sort of how do you formalize sort of meaning in a way that is very structured is something that is, as I say, my efforts in computational language are probably the most elaborate effort so far that have been done. I mean, I think the field is somewhat barren and it's kind of like you jump from Aristotle to Leibniz to kind of modern times and computational language design and so on the. I think that there were some efforts in the 1600s along with Leibniz, particularly there was a whole effort to sort of abstract from the mere linguistic, something which people called at the time a philosophical language, something which would represent meaning in a way that was not tied to the specifics of English or Latin, whatever else. And there were some efforts at that time to. To do that. People were very confused about whether the morphology, you know, whether the actual writing of the language was important to this effort. And that made things even more obscure than they would otherwise have been. But, you know, that led in the 1800s to things like Esperanto, sort of simplified artificial language. There was this idea of simple English that was, I think, from the early 1900s of kind of a version of English where you'd strip down the number of number of number of words that were used. And then there were. In modern times, there are all sorts of artificial languages, the. The conlangs, constructed languages. Sometimes they appear in, you know, movies and such like as, you know, the. Whatever it is, Dothraki or whatever the name is, or, or the language. I think Avatar had another constructed language in it. The recent Avatar had another constructed language in it. And usually these constructed languages for the movies, so to speak, are mashups with some modification to known human languages and so on. But I was interested to see that ChatGPT, at least as of a few days ago, thought it could translate to Ithquil, which is sort of the most elaborate of the. Of the constructed languages. And I, I got a line of Ithquil and I, I was sending mail for other reasons to the creator of ITHQ and so I sent him this line of ITHQL and he said it was awful. So, so ChatGPT didn't, didn't learn how to do conjugation of, of. Of verbs and things like that in Ithqil, which is not a terribly big surprise. And it's remarkable that it could do anything in, in that language. Kind of makes me wonder, you know, this, this whole idea of. Of sort of statistical structure of language as sort of the question of what is the essence of what's going on in that statistical structure of language and is that telling one something that is really a feature of human thought, that is something sort of abstracted from the, the mere sort of structure of words. Is it something a little bit more like logic or like general syntax? That is something that is sort of a general feature of how we think about things and how we sort of generate our thoughts as, as language. And it kind of makes me wonder, you know, how. What, what happens when we apply this to the whale Songs, so to speak. And, you know, is there. I. I would suspect that whatever structures there are in our brains that are relevant to the kind of the things that we're doing to generate language, I have to believe that those structures are probably, you know, biology is. Is remarkably conservative and. And the things that we have often are like, oh, yeah, you can recognize that from the. The tree of life from 100 million years ago or a billion years ago. You know, you could. These are not new ideas in biology, so to speak. And my guess is a lot of the linguistics, construction, kinds of things that we have and the things that relate to thinking as revealed in language, probably a pretty old structure and will exist in the whales and the dolphins and maybe the songbirds and things as well. So it's an interesting question to me, at least, whether the things that we're now seeing that are these kind of richer statistical structure of language that things like ChatGPT tap into, whether they have sort of an analog in. A direct analog in the kinds of things that happen in other animals and so on. Now, an interesting question, I suppose, is how would you tell. Well, obviously, the way that something like a large language model like ChatGPT is based on works, what it's trying to do is it's trying to learn how to continue a sentence, basically saying, if you've got a sentence that has these words up to this point, what are the likely continuations of that? And so the obvious thing, which would be an easy thing to try, actually, is to what extent can you do that successfully with, like, whale songs and so on? To what extent can you train, like a transformer net or something like that to be able to predict what will come next? My guess is it will work quite well. Now, the next question is, if you do that, to what extent can you. Is there a way that you can then imagine translation? You see, what happens in human language translation is that you've got some internal, internal representation within the neural net of what amounts to sort of where the sentence is in meaning space. And you're basically saying, well, I've got this sentence in English that was characterized by this list of numbers, but characterizes sort of where it is in meaning space. And now we can render that list of numbers in French or something as, as. As a sentence in French. And that's. And that's how we can think about language translation. You know, on the subject of sort of the general architecture of these kind of networks and how they relate to human thinking, I was a little bit surprised. I. I've Thought about this off and on for many, many years. And I was just looking at my book, New Kind of Science, which came out in 2002, and there's a short section about human thinking that's kind of tucked away in the chapter about perception and analysis. And actually, I have to say, it's surprisingly sensible. And although it doesn't use the term embeddings for this way of turning sort of concepts into numbers, it talks about hash codes, because that was the closest analogous concept that I had at the time. It's actually a surprisingly decent description of how this stuff has ended up working. So I was interested to see that at least. But in any case, so the question of whether there is a way to the way that human languages work, we have this notion of these, we can construct these embeddings and we can make these connections between, you know, because we have parallel corpora which say, you know, have things in English and things in French. We can connect together these, these things and actually make a translation, which is something we don't get to do for, presumably for, you know, for the whale songs or whatever else. Let's see other questions here. 0d is commenting. Didn't Noam Chomsky also do work in the intersection between math, logic and language? I wondered if language models are based on that at all. I mean, I once had a conversation with Noam Chomsky about kind of the concept of creating a computational language which is sort of a precise representation of linguistic ideas, or at least ideas about the meaning of things. And I have to say his main position was I was saying that I'm trying to create a language that represents things in the world and is saying, but are you doing linguistics? Are you solving the problem of how language works? And I'm like, no, I'm just doing technology and engineering. I'm trying to make a practical computational language that can represent things in the world, which is a different problem than the problem of explaining kind of how language works in our human brains. It is not self evident that there is a way to explain how language works in our human brains. It could be the case that there's 100 billion neurons and they're all firing in a certain way and the collective effect happens to produce words. And that there's no better narrative description of what's going on than just to say, I've got 100 billion neurons firing. And you have to look at all, you know, those detailed firings in a kind of computationally reducible way. I suspect there is a better description and I think even what we're seeing with the success of things like ChatGPT is showing us that there is. Is giving us even more of a clue that there is a definite kind of narrative theory of how language works in brains. But there's this question of language in the abstract. Language as it relates to talking about things in the world, and language as it is implemented in brains. And then I suppose language as it might be implemented on computers, these are all separate pieces, and what the correspondence between them is is not clear. I mean, it's like when you study the theory of flight, you can study birds or you can build planes, and those are slightly different branches. And the things you understand about the theory of flight from. From building planes, there are certain cases you're interested in that. In that situation, you know, if you want to study the flight of insects with all their elaborate kind of, you know, the wings meet when they have their sort of backstroke and so on, that. That's all, you know, it's all a different kind of thing than studying. Oh, we just want to know about fixed wings and, you know, the flow of air over fixed wings and so on. So you. You end up having to ask about different kinds of things. I think one of the things, actually, I was just looking because I've been just looking at a bunch of books about the sort of theory of language and semantics and so on. The. Noam Chomsky's original advisor, Zelig Harris, has an interesting book about kind of the mathematical structure of language that really is, I think, more mathematical than the things that, for example, the early work that was done that Chomsky did in the mid-1950s. I haven't really parsed, not to make too much of a pun of that. You know, exactly what the direction of Zellig Harris was relative to what Chomsky's ended up doing, et cetera, et cetera, et cetera. I don't really know. The mathematicization of language is something that Billy sort of happened in the 1950s, although there were precursors and things like the Sjukowitz, what came into category of grammars, which I think was the 1920s, 1930s kind of time frame. So there were precursors of that. There were deep precursors in antiquity of the construction of these grammars, famously by Panini for Sanskrit. But, you know, when you look at these grammars, it's a little bit hard to tell that they really are formally specified grammars. Because, you know, one thing that happens in early grammar books from antiquity is it'll Be like, well, this is a valid sentence here, and these are examples of valid sentences. The cat sat on the mat, the mat was sat on by the cat. These kinds of things, these are valid sentences and other ones are not valid sentences. Notice that grammatically, the mat set on the sat on the cat is a perfectly valid sentence. It just is not likely to be semantically meaningful. And by the way, this whole question about what is semantically meaningful and what is not, you end up getting, well, of course there could be a mat that is sitting on a cat, maybe in some version of mat, in some version of sat, and so on. But this is where. What's interesting about the construction of computational language is that in building Wolfram language, for example, what we've done is to take general concepts and lock them into very definite things that have a documented meaning and actually evaluate. So the function plus in Wolfram language has a definite meaning where you can actually. It actually adds numbers together and so on. It isn't something that is just a word plus that might be, you know, the plus sized version of something or whatever else it is. Using the word plus as a kind of a flag to say as an indicator of what the actual operation of this thing that we call plus is. This really hard computationally defined thing that's called plus is we're leveraging the fact that there is an English word plus, but we're then making a more precise meaning to that word. And I think that's a general approach which one can expect to use. And in kind of going from the merely, you know, this is from the merely linguistic, so to speak, where the word plus can have many meanings to the truly kind of computational and precise, but still making use of the fact that, that we're using a word that is sort of recognizable to people. That's interesting. It's. Commenting. Will the next generation of chat GPT, voice GPT have any negative recourse, especially when it comes to impersonation? Well, I mean, you know, I haven't yet heard really good. I mean, you know, even maybe three years ago, there were voice style transfer networks, just as there have been image style transfer networks for probably four years or so now. The voice ones were still very CPU intensive and not very good. I haven't, I haven't heard one recently. I'm sure that voice impersonation will become something that can readily be done. It will be tremendously surprising if it couldn't. I mean, the truth is, even whenever you are decompressing an MP3 file, you're making use of these things which are essentially models of the human vocal tract. You're essentially, you know, you're representing a sound by kind of parameters that correspond to physical features of the human vocal tract. That's not, that doesn't need to be that way, but that's a convenient way to do it. And so that's, that's sort of a compression that's analogous to some kind of, some kind of sort of narrowing that happens in neural nets and so on. And that is the way that one basically gets a style transfer as you sort of narrow things down so that you've captured the essential essence of the thing and then you, then you pull it back out again in the new style. And that's the thing that one's already doing in some sense not for that purpose quite in some of audio compression kinds of things. So I fully expect that that will work. The question of how realistic the kind of the bot version of me or whatever will sound. Well, the voice I'm sure will be nailed. But you know, that will be, that will be gettable. The, the kind of the, the, you know, the how would, how would I answer this question and you know, generating the words. I don't know how well that will work. People have shown me things from maybe six months ago from earlier versions of GPT3 which were not particularly good for as representations of me. And we'll see how long it takes to get how good and how long it can kind of keep going, you know, for a sentence or two. I think it will not be too hard for a whole long monologue. Well, I don't know. I'd like to think that there are sort of features of that that are not just the sort of the mean of the population plus a little bit added from the particular ways that I do things. But I don't know. A comment from Igor here about can a submarine swim? Which apparently in English those words don't work that way. Apparently in Japanese they do. I don't know that there are words in English. A single word can have many meanings. In fact, one can even see in some machine learning kinds of things with rather nice. One can actually see the different sort of places where the word can be embedded in a sentence and one can actually sort of draw a feature space plot. In Wolfram language it would be a feature space plot. And then you can label it with. You can, you can cluster it. You can actually see here's a blob of meanings of this word that are over here. And here's a completely different Meaning over here, that's visibly a different blob in feature space. So you get to see that with English words that there will be a single word and it has multiple different, multiple distinct meanings. There's a question from 0d about do I think AI represents an existential risk? How could we mitigate it? You know, I think that. What can I say about this? I mean, AI is a form of automation. And the. There are many things at all stages in sort of the tower of automation that we built in the world, we can end up with sort of bad things happening. I think in the case of AI, it's sort of a question of, I mean, lots of bad things could happen with AI and lots of stupid decisions could be made where we say, let's just have a machine decide that. It's a complicated thing because one is automating human judgment and, and often one is doing a reasonably good job of reproducing what human judgments would be. Now you can ask questions like, do we like the way that. That thousand times human judgment, you know, that, that all those iterations of human judgment, do we like it that it always does this, this sort of, this thing that is a reflection of actual observed human judgment. Do we like that? We might say something societally, we don't like that. Even though it is making the right judgment in the sense that it is taking the input and it is coming to the correct conclusion, even to a conclusion that would be the majority conclusion. You know, the typical conclusion of humans given the same task, we might say, well, actually, when we can actually see it plain and it's a machine that's doing it, we don't like what it's doing there. And I think, you know, the, the ultimate issue is always that in the end the machine doesn't have an intrinsic. This is what I'm trying to do that has to be provided by the. The setter of goals, which for the time being is us. I mean, there can be a time, it's worth understanding. There can be a time where there is a society of the machines, where the machines are all doing their thing. There are plenty of machines just sort of doing their thing right now. And that's happening quite independent of what we humans are doing. But we have an incredibly good analogy for that, which is what happens in the natural world. The natural world just does what it does according to the rules it follows. And we humans, sometimes we interact with it, sometimes we perturb it, sometimes we are subject to what happens in it, and so on. But it's its own thing. And we might say, well, we're going to create this society of the machines. And you know, we have control over that. It's, it's, it, it is true up to a point. But once we've set these things in motion, we've defined their rules, their rules say they're going to do this and this and this. That's what they'll do. Just like natural laws say that the natural world works in this or that way. And I think the issue is we will certainly get to a point. We are already at a point where many things that happen in the machine world, so to speak, are not readily understandable to us humans. But, you know, it's the same thing in the natural world. Plenty of stuff that happens in the natural world is not understandable to us humans. And so I think that's a somewhat analogous situation. Now, you know, it's really a question in some cases. Well, do you want to set up, you know, how much, if you have a choice, are you setting up some aspect of society to work according to kind of the, the whims of the society of machines or not? And you think, oh, we put our own, you know, we put ourselves into these machines so we will be able to know what comes back and we'll be able to say we're going to like it and so on. That's really not realistically what's going to happen. You know, computational irreducibility makes it completely clear that whatever, you know, you set up the rules for the machines, we will never be able to foresee all the consequences of these rules. And so then, you know, we can say, well, we don't like the fact that, you know, we're going to put some, some limiter on where the machines are able to operate as compared to where we want sort of humans in the loop. And that gets even more complicated when you have kind of the, the humans with the neural implants that are, that are, you know, providing machine like auto suggestions about what to do and so on. It becomes even more complicated to sort of untangle how that works, I think. Let's see, the, The. Yeah, I mean, these questions about. Look, I think there are many questions which I certainly don't know the answer to, but they're, they're quite in many cases immediate questions about sort of the, the relationship between AI and society. And the, you know, immediate questions are things like you're trying to decide what guardrails to put around chat GPT when it's talking about things, you know, should you tell it to not, you know, should you tell it. You. It might be a reasonable thing to say, only produce G rated, you know, essays might be another reasonable thing to say. You know, don't say things that will upset people. That gets very complicated. You know, who are you going to upset? You might upset people by not saying things that would upset other people. It's a, it's a, you know, it's, it's a hopeless chain of, you know, how do you make it? You know, in the end, the only chatbot that will offend nobody is the chatbot that says nothing. Maybe even that would offend people. The. So, you know, it, I think it's, it's an unrealistic, you know, you can try and make it, you can try and make it bland, but I, I don't even think, I think it will be even across different cultures, blandness is not constant. So what might seem bland in one culture might seem utterly offensive in, in another one. You know, sort of the same words and so on. So, you know, I think that, that this question of kind of, how do we, how do we define what it is we want? And you can pick any one of a whole collection of different things. I think, you know, a search engine, you know, what links should it put up there, A. A social media platform, what, what things should it put in a feed? These are all sort of, what are you choosing? Now? The chatbot case is a little bit more extreme, but because it is more from its own mind, so to speak, it is less like, let's pick content that people have already produced, or let's say it's a generative AI image system. What kind of things should it be able to produce? Should it be able to produce what if it produces some incredibly shocking picture, should it be able to do that? What should you tell it about producing shocking or non shocking pictures? What should you tell it about producing pictures that have, you know, sort of real people in the, in them doing terrible things or whatever else? You know, what, what, what should its principles be and how do you define that? I think it's really hard, but I think, you know, one place that you start to get to is let's just sort of write down laws, write down constraints that we want these things to operate under. And maybe you come in at the end and you say, let's look at this picture. Do you want to consider it shocking or not going in at the beginning and saying, let's tweak the rules to never make a shocking picture? I think that's that's hopeless. I think that's a sort of computational irreducibility hopelessness case when it comes to stating the rules. Well, you know, where do you get the rules from? In a sense, you know, what we have learned in the history of the last 4,000 years or so, more than that of human lawmaking, so to speak, is that, you know, in the end, you write down kind of in some sense what you want the constraints to be, whether it's the code of Hammurabi or modern, you know, US code or a continental legal system or whatever. The. It's, you know, there is a certain thing that you describe. Now, one of the things that's nice and something we've been interested in with Wolfram language for a while is kind of the computational language description of sort of what you want to have happen. The, you know, having a computational language that can talk enough about the what's. What's really there in the world to be able to specify those constraints in that computational language. The feature of that is once you have the thing written in this computational language, it allows you to have something which you can say, you can hold it up and you can say, this is our code, so to speak. And code not in the sense of, well, it would also be computer code. But these are the things that we're operating according to now. There have been experiments and blockchains and so on of things like DAOs, distributed autonomous organizations, where people try and say, let's just define all the rules for how this is going to work in code. And that's ended up very badly in certain cases because it was not foreseen what all the possibilities would be. And it was not foreseen that there would be some particular way that somebody could, you know, transfer lots of money to themselves in a way that everybody thought was shocking and so on. So I think that, you know, I consider this, as far as I'm concerned, this very unresolved issues. And, you know, in a sense you can ask questions like one I've been curious about. I have no good answer. So let me say that up front is sort of, now that we have this kind of ability potentially to be able to write computational language that describes human preferences. To what extent can we. Are there new forms of governance that we can imagine given that capability? Just like in the case of, oh, I don't know, you know, your average democracy situation, it's like, you know, a multiple choice thing. You're saying, pick this or that or that in some vote that you might give. Well, you could imagine a much more nuanced situation where everybody is writing a computational essay about what they want, and you're saying to some AI, take all these computational essays, combine them, and figure out what should happen. I think this is a. I mean, you just get thrust immediately into issues of very old issues of political philosophy. As you think about how is that AI supposed to work? What's its governing principle for how it deals with all those preferences? Does it make, you know, 1% of the people super happy and 95% of the people only medium happy and 3 and 4% of the people super unhappy? What does it do? How does it decide what to do? You know, what are its overarching governing principles? And I think we don't know that. And there is no. That. That is not something which came with the package for us humans is the, is the statement of. And this is how that has to work. You know, the only thing that sort of comes with the package is don't do stuff which makes the species go extinct, because then there's nothing else to discuss about it. You know, if you do, if you, if you do things badly enough that, you know, eventually there's, there's no, there's nobody left to discuss what's going on, then, then you can say, well, that's a probably. Say that's a bad outcome. Maybe you don't say that's a bad outcome. Maybe it's like the, the chat bot that says nothing. Is these the only good chat bot? The, you know, the, the civilization that doesn't exist is the only civilization which will do nothing wrong. You know, that's a, that's a, that could be the. That could be true. Not very useful, but true. So let's see, Let's see. Kat is asking, how do I think that Einstein or Stephen Hawking would react to ChatGPT? Are there figures in science who predicted this development? Look, I mean, everybody who's thought about this stuff has thought this will eventually happen. I have to say, it's happened more suddenly than I expected. I, I think language and linguistic communication may be a little thinner than we had imagined. I mean, the idea that it would be eventually possible and the path to get to kind of statistically generated plausible text, it's not been clear how well that will work. And even a year ago, it really didn't work very well. I mean, ChatGPT, I think, is a mixture of an interesting underlying language model together with some pretty good engineering about how to actually present it so that it doesn't, like, get stuck in a loop and has patterns of output and so on. It's a well packaged example of sort of that core methodology. You know, I think this whole idea of generative AI systems and generative computational systems, well, this is a certainly something, you know, that this is an old story. I mean, in, in my own work in like cellular automata in the early 80s of generating images which are surprisingly kind of naturalistic, even though they come from simple rules, even though they're very much a pure computational origin. That's kind of an example then in, in the early 2000s when we made this Wolfram Tones website that produces some kind of musical forms. Again, that's something where it is not okay. So in the case of something like ChatGPT, it is sort of taking some kind of statistical average of all of those billions of documents that it found on the web. In the case of cellular automata or Wolfram tones, what one is more doing is creating from scratch using the same principles as, for example, the natural world uses to create structure, but not using the big corpus of kind of human memory and human, and sort of human examples of language. But it was perfectly possible to produce things which at least locally were sort of plausible. Small scale musical form, for example, sort of just plucked from this computational universe of possibilities without a big loop that makes use of kind of human experience and so on. The only place the human experience comes in is in the picking of styles and so on, once one has generated some underlying structure. So I think the, the idea that sort of the statistical language, statistical use of language could produce something that will be coherent over the length of a small essay is interesting. I mean, it's sort of ironic that Alan Turing in 1950 talked about this kind of Turing Test of can we talk to a machine like we talk to a person? I think he thought about that as a very rule based kind of thing. I mean, he'd been thinking about, you know, chess playing machines and he was very much, and much earlier he'd been thinking about Turing machines and very much this kind of rule based structure for what the thing should do. Ironically enough, during the Second World War when Alan Turing was working on cryptanalysis, one of his bigger contributions had to do with the statistics of language, of knowing, you know, of all the sort of code, all the passwords that might be being used. How do you tell whether the, the, how do you automatically tell whether with that password the thing that's being generated is, is actually a genuine piece of German, or whether it's just random gibberish and what he noted, and that sort of, I think, turned into, in some ways, information theory and Claude Shannon's stuff and so on. I don't know. We don't know what the precise causal chain there was. But, you know, Turing invented this kind of these ideas of sort of the statistics of language as used for cryptanalysis and so on. And it's sort of ironic that I don't think he made any connection between that work, which was probably still secret, which was still secret at the time, among other things, and the things he was thinking about, about sort of AI and so on. And I think it remains unclear whether there is a. The statistical character of things like large language models and so on. It's interesting that it works as well as it does. Whether that will be the way that we can really take advantage of sort of the computational paradigm, I'm not sure. I think that that's a way of taking advantage of some aspects of computation. It's not clear that we can build as big a tower there as we can in kind of thinking about things in sort of a precise computational way. I mean, I just wrote this thing a couple of days ago about using WolframAlpha, which is based on kind of actually turning, having a symbolic representation of the world and computing things precisely from that, connecting that to ChatGPT. And it's rather lovely because CHAT GPT can formulate questions that can be answered by the natural language understanding system of WolframAlpha, or can be turned by the natural language understanding system into precise symbolic representation of things where the answers can be got. And then CHAT GPT is. Seems to be mostly rather dignified in, if you tell it, oh, here's a thing you might use to correct this essay or whatever else it very much. It almost says, you know, okay, thanks, and I'm going to use it. And then it produces another version of the essay. It is, however, a little bit remarkable when you think to yourself that what it's actually doing underneath, just running this neural net, a word at a time or a token at a time, which is sometimes a piece of a word, and it's just saying, what's the statistically likely thing to come next? And that's it. And it's kind of shocking to think that, you know, brains probably work very much like that too. And I think, you know, there's not sort of more magic to it. Let's see. Huh. Interesting comment from William here. Given what we've learned from AI models, does learning from history allow us to better predict the future? Does modeling the past Imply modeling the future. Interesting question. That's an interesting question. So I mean, the fact that a transformer net will successfully learn the pattern of lots of things on the web and be able to make reasonable essay statements. A good question is, that's a good question. If you feed it the knowledge of human history and you ask it to predict the future based on its knowledge of human history, to what extent will the patterns that it finds be such that it will successfully predict history? You know, the thing that I suppose is in a sense what I'm asking about when talking about Aristotle and logic and that kind of explainable pattern, there's sort of a question of whether, you know, a training of a transformer net, for example, can discover patterns that we have never noticed before and that are sort of perhaps even undescribable by any of the words in the narrative that we currently have. And the answer is it probably is finding some such patterns. There may be a future logic in quotes that is kind of that, that captures those patterns that it's found in a way that is now narratively explainable. So now the question is, is there sort of a theory of history? Is there a pattern of history? I mean, I think, I think Hegel is famous for having talked about sort of theories of history and the sort of necessary flow of history and that got, that got picked up by both very sort of left wing and very right wing political efforts. But I think that this question of is there sort of a flow to human history, a predictable flow to human history? Interesting question. And the answer is probably yes. I mean, I know people talk about, you know, various kinds of, you know, theories of, oh, when a particular society gets sort of quite old, it builds up more and more bureaucracy and it's governance and things like this and eventually this and that happens. I mean, this is kind of the concept of Isaac Asimov's kind of psychohistory idea of kind of there is a predictable flow to history. You know, I think computational irreducibility kind of tells you on a smaller scale, at least there isn't a flow to history. You know, I was just looking at, I was just looking actually at a book by Charles Darwin's son, Grandson, not sure, maybe grandson. That is a book called the Next Million Years. It was published in the 1950s. And I just got as far as the introduction to the book. But what it says is predicting the next 10 years is really hard. It says, but it says maybe just like we can discuss gas molecules bouncing around in a box in a gas. And we can't necessarily say where the individual gas molecules are going to go, but we can say collectively what's going to happen to all the molecules in the box. We can use the gas laws to do that. So maybe if we try and predict history on a timescale of a million years, it is more predictable, says this Darwin, than it might be on a 10 year timescale. And I suppose that's an interesting question is whether the sort of the overall arc of human history has a certain predictability to it. You know, it's interesting. We have, in Wolven language we have data on the borders of all known historical countries. And you can kind of make an animation of kind of particularly Europe where there's a lot of data and a lot of structure there of what's happened to the borders of countries in Europe over the last couple of thousand years. And what you see is something where it's very, it looks very physical or maybe biological. You know, you'll see some region, it gets bigger and bigger and bigger, somebody's making a bigger and bigger empire and then pop, that bubble bursts, that disappears. Other things move in its place. It kind of reminds one of, you know, a froth of soap bubbles and things of all the borders of these countries where every border, every motion of every border was a lot of human lives and a lot of, you know, a lot of kind of human experience there. But when you look at it in the large scale aggregate, it looks kind of like something where you'd say, oh, there might be an equation that describes the motion of all these things with a certain amount of unpredictable randomness to seems like there might be sort of a, a global theory of the kind of the ebb and flow of countries. Whether. And that that kind of supports the, this is this kind of question of whether on a large scale, on sort of the large statistical scale, are there things that can be said about the patterns of human history. And you know, to us living day to day it seems implausible that such a thing would happen. But you know, you imagine you're a gas molecule and you're kind of colliding with other ones and so on. And it might be implausible that you could predict the very large scale structure of the whole gas, but maybe you can. I would suspect that there isn't enough human history to be able to use those kinds of, you know, very large scale kind of arguments and conclusions. My guess is that, that there just aren't enough, you know, there's a couple hundred countries in the world today There have been comparable numbers throughout most of history. Well, when it was in the kind of city state world, they were probably, probably more, maybe more of them, maybe not. Maybe they're about the same number because the population was a lot smaller, I'm not sure. But, you know, being able to predict what will happen on a very large time scale, we may just not have a large enough timescale. We may still be operating in the. Well, it really mattered that the Roman Empire extended this far. These specific features of particulars of history may still have enough effect. And, you know, hence the, you know, can we predict the next million years? Well, I think, I think we generally believe that the human experience will not remain the same, you know, as we, as we reach the point where we have, you know, in the time, in the, even the 1950s, it wasn't clear the notion of kind of a virtual existence and digital, you know, sort of souls and digital form and, and, you know, effective immortality and so on. Those were not really on the horizon at that time. They are now. And you know, exactly what consequences that will have that the future of history doesn't feel like it will be similar to the past of history. But there may be certain patterns of any, any sort of, any consciousness, any sort of entity that's kind of like us in certain ways will have some necessary pattern of large scale history. That could very well be the case. And one of the things that has come out in our physics project is the realization that entities that observers like us who are computationally bounded, who believe that we're persistent in time, that it is a necessary feature of being an observer like us, that you will observe that the universe follows the laws of gravity and quantum mechanics and so on. So it could be the case that if we can tighten up that idea of observers like us and go beyond computationally bounded and persistent in time, but have some other aspects of the essence of the human condition, I don't know what those would be. I mean, these are sort of some kind of, you know, base ethical something or other thing or some other feature of the human condition, that when you put those into the mix, that it will be inevitable that the very large scale flow of history has some particular form, that there will be sort of an inevitable bulk theory of history that will always be the case for any organisms that have a particular form. It kind of makes me wonder if you look at it for organisms, I don't know, like ants or something like this or some even lower organism where you have an ecology of those organisms, what are the Large scale things, you can say. And generally in ecology there are pretty weak things that people ending up being able to say about total population numbers and ratios of this and that and so on. You know, is there, one might ask a more detailed thing to say, if you were a little microorganism living your life in this colony, big colony of microorganisms, would there be, is there, is there more detail that one can give than to just say it's this number of microorganisms. And I suppose that's an interesting question of kind of what the analog, what the bulk limit of individual sort of lives in an ecology actually end up being. All right, well, let's see. I should probably wrap up here, but that's an interesting, interesting question. What the. What, what one can kind of learn, you know, is there a theory of history? I think this whole question of is there a theory of X? There's sort of two levels of that. One is, are there rules one can write down that describe the microscopic underlying structure of X? That's one thing. And then when one looks at the large scale kind of contours of what happens? Are there humanly understandable versions? I mean, in a sense, the whole enterprise of natural science has to do with saying, are there kind of ways to connect what we see happening in nature with what, with what we can think about in our minds, with what we can kind of stuff into our minds, the whole story of kind of computational irreducibility. There are things that happen in nature. We just have to essentially watch them happen and see how they unfold. That's kind of not a. That, that is a necessary. That is something that's necessary. That is a paradigmatically correct kind of thing, but it's something that kind of denies the possibility of a certain kind of natural science. And when one says, is there a theory of such and such a thing? It's really asking, come one go, can one sort of break out of computational irreducibility and find some slice of reducibility? The. Yeah, Emman is commenting about ants that they're, that they're structured distinctly enough that one could lead to conclusions on many levels. Yeah, I mean, certainly one, you know, one watches ants doing their thing and it's, you know, there are some theories I know of kind of overall sort of ant behavior and so on of, you know, the ant sees the neighboring ants and it has certain behavior based on that. I just don't know what statements can be made that go beyond kind of the almost mechanical statement about, you know, when you have a collection of ants or a flock of birds or a school of fish. We know that there are certain characteristics that those collective characteristics that those groups of organisms will have that in which we know that there's, you know, various kinds of flocking behavior where, where things will all arrange themselves in a definite sort of lump and then, then something will happen and the lump will turn and, and how that turn works and so on. That's all stuff we have some understanding of. And, and the main understanding is there's only local information that, you know, you're just. The fish is just looking at somewhat neighboring fish. And yet we, once the leader fish decides we're going to make a turn, then all the other fish kind of cascade down and see they're going to make a turn and so on. And we can describe something about that. But it will be interesting to understand. Is there a larger scale thing? One could say, for example, yes, the school of fish is schooling in this way, but what can you say about the large scale set of decisions that it makes about where to go? Is it a purely random walk? Is it something else? What can be said? Is there a kind of a larger scale theory of what's going on there? And you know, I think again, I don't know what that's really like or whether we reach the point. I mean, we're kind of spoiled with gas molecules because they're an awful lot. You know, there are a trillion, trillion gas molecules in a little box this size. And you know, when it comes to the kinds of things we've been thinking about in our physics project, and we look at sort of the atoms of space, there are just absurdly large numbers of atoms of space in some fairly small region. And so we really can talk about these large scale limits. It's less obvious whether that will really work in these, in these kind of issues of human affairs and so on. All right, we should wrap up there. Well, thanks for raising all kinds of interesting topics. And I see questions about completely different kinds of topics which I look forward to addressing another time. Well, thanks for joining me and see you another time.