Future of Science and Technology Q&A (August 18, 2023)

Sam. Hello, everyone. Welcome to another episode of Q and A about future of science and technology. I see all kinds of interesting questions that have been submitted here. So let me try to tackle a few of these. Let me see. Lolola asks, do you think houses are going to change much in the future? Will we reach the age of true smart houses? The house that I live in now we built 22 years ago. Maybe at that time, there were all these things about make your house smart, you know, put in all of these different kinds of things. In the end, the big decision we made was to run empty conduit in walls so that whatever new fiber optic, you know, this, that or the other thing might come out in the future would be usable. And that that was fairly useful. I wouldn't say it's dramatically useful as an awful lot of conduit that's never been used. But, you know, at that time, 22 years ago, there was a lot of enthusiasm for, you know, all these things that would make your house sort of, you know, smart for you. All the, you know, we put in all kinds of automated light switch type things and so on. I have to say, if I were to rank things that have affected my quality of life, those were definitely not big positives, so to speak, and more of a sort of user experience, pain in the neck now, you know, over that period of time, things like, you know, smart thermostats, those are a thing, all kinds of, you know, talk to your house type things. Those have come into existence and I guess work okay. I don't use them myself. You know, what about houses should be smart? Here's another thing to realize. If you look at houses that were built, you know, hundreds of years ago, and you say, you know, they were very simple, they just had walls and maybe they had windows and so on. A house today has all kinds of crud in its walls and all kinds of, you know, all kinds of ducts for air conditioning, all kinds of electrical cables, probably network cables, unless it's all wireless, et cetera, et cetera, et cetera. Somehow there's a complexity to building houses that seems to stay sort of constant over time, that even though things get more automated and et cetera, et cetera, et cetera, there's just more things that are expected of houses. And then people occasionally try and do very different things in the construction of houses, like don't have square rooms, you know, put all kinds of other exotic things in. Most, mostly those have not been a big success, partly because people need, I think, to be an Architecture that they feel comfortable in. And when it gets too weird and too different from what's been seen before, possibly too different from what people kind of experience in the natural world. It isn't something that plays well for people. So I mean, I guess the question would be what? You know, there are things that people have said would happen for a long time. Like for example, make all of your walls be displays. You know, just have. Don't I remember from a long time ago. I think I've told this story before. My wife always gives me a hard time in terms of future prediction that this was probably beginning of the 1990s, renovating some, some house. And it's like, how deep do you have to make a niche to fit a television in? And I was like, look, 4 inches is going to be fine. Because that was what was happened to be available in that particular configuration. Because I'd already seen smart flat, flat screen televisions. But it turns out it took another decade for flat screen televisions to become common, basically because the yield of making things with those kinds of display elements was not enough. And you can really tell if there's a display element that isn't working somewhere in a television. So you have to get the yield up to the point where basically every pixel works. But you know, this has been a long time thing. Oh, we'll have kind of sort of soft walls, so to speak. By soft I mean soft in the sense of software where you can sort of put anything on the wall, so to speak. And I suppose, you know, there have been a bunch of attempts to make sort of soft this, that and the other things. Another one that was a thing for a little while was soft keyboards where all you would have would just be essentially a display in the position of the keys for the keyboard. And you know, when you press a shift key, you get Greek keys or mathematical keys or something like this. I think I have somewhere a comp that was that had that kind of capability didn't work very well. You know, one says such and such a thing didn't work very well. One of the features of technology tends to be that sometimes it comes back, sometimes it didn't work well because there wasn't the right kind of force, feedback, feeling of touching the keys and so on. And then there'll be a new technology that makes that work and then that idea will come back. I mean there have been plenty of ideas that would sort of try decades before their time. They kind of didn't work and then things developed better and then they kind of Came back and really worked the next time around. So, you know, maybe I shouldn't be saying that about kind of some of these aspects of sort of smart houses, but I mean, in terms of other kinds of things, there are lots of things people talk about. Another one is electro optic windows. You know, it's like, I think the 787 has these where, you know, you just touch a button and it's using electro optics to, you know, change the polarization of the, the polarizing effects of the material and be able to make everything dark. So instead of, you know, no more, you know, no more screens, just, you know, no more blinds for windows, just, you know, press the button. Again, a lot of these things, you know, right now that would be too expensive to do in any large scale. And you know, again, that that's the kind of thing that might, might come in time, you know, other kinds of things. There's also all of the home robot stuff. And you know, I see companies, see many companies over the course of the last few decades that have, have talked about various kinds of sort of home robot capabilities. You know, the vacuum cleaner thing that worked, that was 20 years ago or something that started to work and it's. But you know, more general. I remember a few years ago at the Consumer Electronics show seeing the laundry, you know, folding robot, huge lines to see it. People thought this was really cool. Whether that's a real thing, I don't know yet. Eventually some of those kinds of things that require physically moving stuff around will be roboticizable. Although those are comparatively difficult. They require things like robots to grab things, robots to pick things up. That turns out to be a very hard problem. Mixture of machine learning and actuation and planning and so on. That turns out to be really hard. And notably when you see things like automated labs where lots and lots of stuff has been automated, the move, the sample from here to there is still typically done by a person because that's just. There's much else that can be automated, but that's one of the more difficult things to automate. So I'm trying to think what other vectors of automation are there for houses? I think what does one want from one's house, so to speak? I suppose the, you know, this kind of, you know, control the lights, control sound, you know, put things away. You know, I, I think, I don't know what the effect of augmented reality, you know, widespread augmented reality would be on, on house behavior, so to speak. I mean, one of the things I've thought about for ages is visual Diffs. So, you know, you kind of, you come into a room, what's out of place? You know, what, what do you want to put back? You know, if, if I've got books that I pulled out of my bookshelves and I want to put them back in the same order. Well, if I have some augmented reality system that I'm wearing, it can just say, oh, you've got this book in your hand, great. You know, I know exactly where that goes and I can put it back or it can give you a diff of how your bookshelves are different from what they were before. But I don't know, you know, that doesn't. I don't know whether the sort of average degree of declutteredness of houses might change as a result of those things. I'm not really sure. I mean, another thing, suppose that is various kinds of the air tags or other tags like that and how routinely those are put on things. I mean, that's a thing that one still doesn't. People use them a bit, but they're not sort of widely used. I suppose again, like the visual diffs thing, one can imagine, it's like just have this camera that's, that's watching the room and just notices, oh, this is out of place. That's out of place. Oh, that's where you put such and such a thing. I can see it. It's in such and such a room. So let's see other things. People are asking, Okay, David is asking here in the next 20 years, will artificial intelligent image recognition and image segmentation equal the accuracy of human experts? For example, with AI pathologists or radiologists, will they get to the point of being like human pathologists or radiologists? You know, I ask that question regularly when I run into people who are in those fields. And my impression is that, for example, in radiology, which seems like the place where there is the most immediate kind of AI assistance potentially available, what's happened, and it's happened over the course of many years, not just in the current phase of AI, is there's a lot of AI assistance. Your average ultrasound, you can highlight, it automatically tries to highlight organs and show measurements and things like this. The. I think that there's a degree of kind of, oh, you can find this or that kind of thing automatically that's happened. It is also the case that it's still. There's. I don't know, there's still an awful lot of sort of weird anatomy. And I suppose one of the questions is, what are you trying to compute in radiology, for example, you know, if you do radiology of anybody, you'll find all kinds of weird things that are different from the average person. And you know, and the question is, okay, you can, you can flag that. You can say this is a thing we expected might be weird and now we notice that it's weird. You know, it's a cyst somewhere. It's a, it's a thing where the, you know, blood flow is different. It's a thing where something is, is different or whatever. Let's flag that. And that's a thing we know has a diagnos of some kind. So I can imagine that the kind of teach the AI to the diagnosis is a thing that can be done and you know, and has been done a lot of cases of saying if you're looking for a this kind of thing, then, you know, then this is what then, then the AI will find it. You know, it's a thing I've noticed when you see particularly kids who say, oh, I've done some machine learning training program or something and we did this project and it's like, what's the project? It's, you know, brain tumors and lung cancers are like, seem like that's the project you always hear about. It's like, well, we looked at those and you know, I think what's true is that there is a certain level of looking for sort of obvious things where you kind of know the diagnosis, you know, what you're looking for that is very AI accessible. When it gets more funky, I think it's much harder and it's kind of like what's the thing trying to figure out? If the thing's trying to figure out, you know, do you have diagnosis X that I think is more, more has more potential than if the question is, you know, will this thing turn out to, you know, grow more or do this or that or the other that is less well defined. I think that's more difficult. You know, what I've heard is that in pathology, for example, there are again, there's a lot of computer assistance that's possible, but there's still plenty of things that have not been dealt with. Partly because again, there's sort of too much diversity in what can happen. I think it also matters in, well, I mean, that's my impression now. What's the rate of improvement and at what point does one sort of say, okay, the AIs have it. One of the things that's been notable about machine learning is it has this habit of, you know, there's sort of a breakthrough like deep learning from, for image processing, which was a 2011 kind of timeframe breakthrough. There's sort of a breakthrough, there's a big jump in capability and then it goes very gradually. And one of the things that tends to happen is there's a certain error rate and it's hard to get that error rate down. And so if it's like a screening thing where you say, well you know, 80%, I want to know is it, you know, if 80% chance of getting it right is useful, then the machine learning can do a good job. If you really care about that 99th percentile or something, then it's really hard to make that work. And so I think it's another question is in these medical situations, are there cases where having kind of the 80% screen is useful or is it really that you want the 99% certainty? And I think that, you know, those different use cases can have different consequences there. But you know, in terms of, does one see, in the case of sort of typical sort of AI based systems, does one see kind of a very progressive kind of Moore's Law, like improvement curve? My impression is not really. Now I remember when Wolfa came out in 2009, we had a very straightforward test for how well we were doing, which is how many, what we call, fall throughs occurred, how many things people typed in we couldn't deal with. Two things happened. One was we managed to analyze those fall throughs and make them work better. And the second was the humans who were using WolframAlpha learned the kinds of things that would plausibly work and ones which wouldn't. Between both of those things, over the course of about three or four years, we saw kind of a, almost sort of, we saw kind of an exponential improvement up to very high success rate for many kinds of things. And that was, but I'm afraid that that was a combination of our sort of technical prowess in improving things and the use case change of people seeing what would work and what wouldn't. And I think that's also true with some of these, you know, things like medical AI and so on, where it's like, here's a thing that's actually going to work, let's really push on it versus, you know what the AI is just not managing to do a very good job of this. You know, let's, let's work it differently in that, in that particular area. Let's see. Okay, Sal asks, how long do you estimate before AI can do creative mathematics? How will this technology be similar or different than GPT? Well, you see, the thing is, it's very easy for a computer to be creative in mathematics. The problem is it can be so creative that humans don't care about what it's producing. It's very easy to go out and have an automated theorem proving system, a proof assistant type system that's been automated. Go and just start enumerating true theorems of mathematics. You just say, here are some axioms, let's go apply these axioms in different ways. Great, we got a new true theorem. In fact, I even wrote this book about the physicalization of metamathematics where I'm talking about the, the kind of infinite limit of mathematics where you're kind of looking at proving all possible theorems of mathematics. You get this giant entailment graph of possible theorems of mathematics. Here's the bug. The bug with that is, yes, you can prove all these theorems, but the question is, which ones are human? Mathematicians are going to say, oh, that's really interesting, we care about that one versus like, oh, that's just another theorem that happens to be true from these axioms and we don't care. So in a sense, the challenge is how do you find the quotes interesting theorems? And what tends to happen in the development of mathematics is that one builds these kinds of towers of interestingness, and those are partly built in a social way. It's partly like, you know, there could be a mathematician out there who builds a giant tower and nobody else cares about it. Just like there could be a person who invents a language which only they, they only speak to themselves. And that is a thing that is of a different character than a thing that is a language that's used to communicate ideas from one person to another, so to speak. So I think it's this question about how do you build towers of mathematics that are creative? That's easy. You just go enumerate arbitrary theorems, you go explore the computational universe of possible things that are true. What's more difficult is identifying which ones humans are going to say, aha, I care about that. Now you could imagine having you go a little distance away from what humans already know. There are about 3 or 4 million theorems that humans have published in the literature of mathematics. You can think of that as kind of a front of where mathematics is reached. Actually, more accurately, you probably think about it this way. There's the space of all possible theorems, and that is a space that has many interesting characteristics. That's what my book about physicalization and metamathematics is really about. Is about the structure of the space of all possible mathematical theorems. But in that space, there's sort of certain places that have been colonized. So it's kind of like physical space. You know, we got the surface of the Earth. It's this whole physical area laid out in space on the surface of the sphere, more or less. But now there are particular places where people built cities and where people live and where people care about that place on the surface of the Earth. And in, you know, there are really weird places on the surface of the Earth. Like, for example, there's Null Island. You know, the. The equator, you know, intersects the Greenwich meridian, the. The 00 of longitude, latitude, where it's just someplace out in the Atlantic Ocean and sort of nobody really cares about it except for the fact that it's kind of an amusing kind of place that sort of the default for a geolocation system that knows nothing will say, oh, you're at 00. That's null island. I mean, I'm sure one day somebody will build a Null island thing at Null island just for the sake of there being something there as sort of a tourist attraction of some kind. But what I'm saying is that the, you know, what's relevant on the surface of the Earth is the places that have been populated with cities and so on. In metamathematical space, it's sort of the same idea. There's this big space of possible mathematics, and there are these places that have been sort of populated by humans where humans have chosen. Human mathematicians have chosen to sort of live in those places. And so this question about how do you extend out from where we've already colonized to sort of colonize further in metamathematical space is an interesting question. I think that there is a possibility of using LLM like technology to say, oh, here are the things that we already cared about in mathematics. Here's three or four million theorems that humans already cared about. Use the pattern of those theorems to say which of the possible theorems that we can get to from the axioms that we choose to believe in. Which of those possible theorems are ones that we are likely to care about based on our pattern of caring about things? It's like saying we're going to generate sentences with an LLM, for example. And those sentences might be, you know, tell the LLM, go generate a haiku or something, and it's going to generate some small number of words. And the question is, are those words that we're going to care about or not. And are they? You know, it could be that they're syntactically correct, but they're ones where we just sort of shrug and we say, yeah, okay, you could say that, but who cares? I mean, I suppose the same thing happens as well with music. You know, it's sort of a combinatorial thing. You can just put together notes and you can put together any sequence of notes, but many of those sequences of notes, people just say, that sounds terrible. Or they'll just say, oh, I don't know what that's about, you know, and just sort of shrug about it. What is relevant is the things that, for some reason or another, resonate with us humans. And that resonance with us is, I think, in large part related to the things that we are already familiar with. That is, we get to sort of build out from what we already know in mathematics, in the language that we use in music, for example. And perhaps there's some intrinsic kind of characteristics that we humans have that make things make sense to us. There are things, I suspect, that are associated with kind of the computational characteristics of brains that are things where we're going to be able to understand that versus we're not really going to be able to understand that. There are things in mathematics where we get to sort of see, we have a certain way of sampling metamathematical space. We have a certain way in natural human language of kind of knowing what is, for example, reasonable for us to deal with. For example, very, very simple example, if the LLM was generating lovely sentences, which said the cat, which did this, which did this, which did this, which did this, which did this, and so on. Very, very, very deeply nested. We mostly won't get it. We'll just sort of say, oh, we lost it. I don't know what that was about. So it's not. It's not a. Well, it doesn't have a good sort of human connection to us. That's not the best of examples because that's a more sort of mechanical problem, so to speak. But it's also just, you know, you can write an essay and it's like that was just completely banal and vapid and it didn't really say anything. And you just shrug your shoulders and it's like it's got a bunch of words, but it didn't really say anything. And that's something where you could have that be sort of the output of the LLM, or you can have the LLM produce something where you look at it. And you say, wow, that's really interesting type thing. And so it is with mathematics. So I think that's really the challenge, this challenge of being able to detect interestingness. And that's a. You know, there's a sort of another analogy there in art, you know, when in times past, you know, art was like, I've got a scene in front of me, photographs didn't exist. I want to be able to represent that scene in some way that's somehow faithful and reminds me of that scene. And then we get to more sort of deconstructed modern art where it's like, well, there's this thing, and it's a bunch of paint on a canvas. And, you know, in many cases, you just shrug and it's like, I don't know what this is about. You know, it's just a bunch of random splatters of paint. I don't know what this is. But if there's a sort of a chain of kind of human experience that says, oh, yes, that's a thing, one recognizes that that has this meaning to us that resonates in this way because we kind of have understood it. It's kind of defined, just like in a language, we might say, over time, we introduce new words into the language, and then people get to talk in terms of those words. Those words develop kind of a meaning through their use. So similarly, one can say that sort of these various artistic forms develop a meaning through familiarity and so on. That then allows them to kind of have more human connection. So I think that's what I would say that the challenge with AI is not to do the mathematics. The challenge is to see whether humans will care about it. Let's see. Zania asks, do you think smartphones will replace desktop computing? I don't know. I think that the. I mean, I'm just amazed when it comes to particularly younger people, but not only younger people, the things they're prepared to do on their phone. I mean, it's tiny and it's very inconvenient to type on, and voice is only good for certain kinds of things, and voice is only good in certain kinds of situations. And I think that for me, it's, you know, I have screens of pretty much all sizes, and there are definitely tasks for which using multiple big screens is very, very useful. There are other tasks for which a standard laptop screen will work. There are other tasks for which a phone screen will work. I would say that there's a sort of hierarchy of tasks, and I don't think it's just a question of bad ux, so to speak. I don't think it's the case that if there was a better UX for let's say, programming on a phone, then I would say, oh yeah, I'm going to program on my phone. I think it really is something where, you know, we humans have certain scaling to our, to ourselves. You know, we have those, you know, we have fingers that are a certain size that we can use to type on things. We have eyes that have a certain resolution and a certain angular sort of, what is it called, sort of angular range. And if we, and those are the things that we potentially can make use of in doing sort of sophisticated, let's say, things where you need a lot of different inputs to be able to, to do what you're doing, so to speak, like in certain kinds of programming, kinds of tasks and so on, or certain kinds of research tasks where you've got many, many windows up on the screen, you're looking at lots of different things and so on, and you're trying to sort of correlate lots of things together. I don't think that's doable really on a phone. And I think that's a matter of the way we humans are set up. We're just, we have this big sort of opening angle of things we can see. We have 10 million fibers in our optic nerves. We have a certain, you know, ability to see a certain resolution. And the phone is not a great way to sort of plug into all of that resolution. I think it'll be interesting to see what happens with the next generation of virtual reality systems and so on, whose main function I think will be yet another form factor of screen. So, you know, I fully expect that with high resolution virtual reality one will be able to have, yeah, you know, those eight screens arranged all over one's visual field in a convenient way. And I suspect for some tasks it will be kind of rather impressive that one will be able to just make use of all these things really as the most direct sort of path into one's brain one can get. It's kind of like, forget direct brain connection. We're going to use all those 10 million fibers in our optic nerves and so on from this sort of fully, fully worked out virtual reality. So I have to say I think, I expect that there are uses for screens of all sizes, so to speak, from the watch size to the phone size, to the tablet size, to the laptop size, to the desktop, to the big, you know, multi screen desktop, to the virtual reality Cover everything to the wall size screen. I think these all have certain, you know, uses. I mean, it's kind of like there are things in our world of different sizes. You know, a chair is a certain size because, gosh, we're a certain size and we want to sit on it and, you know, something else which is, you know, a thing that is, you know, a watch is of a certain size because we want to wear it on our wrist. So, you know, I think we provide certain scales and our displays and so on need to match those kinds of scales. And the thing that's interesting is that there are multiple scales of display that, that makes sense, so to speak. And I think that will continue to be the case. Let's see. Mohammed asks, does it make sense to pursue a math degree in the age of AI? Heck, yes. The thing is, math has been practiced in a certain way where sort of, well, in my own efforts over the last 35 years, we've been able to add sort of a computer assist to the doing of math, and that's allowed a lot more math to get done. A lot of things that have been discovered in the last 35 years have relied in one way or another on the math that could be done using Mathematica and using, you know, using computation. Actually, I think vastly less of that has been done than could be done because math as a sort of cultural tradition has really been about kind of, let's prove a theorem, let's do this, rather than let's go out and explore the world of experimental mathematics which computers are really good at doing, and just discover what might be true and then try and build a theory from which we can make proofs or whatever else we want to do. Doing experimental math is a thing that requires skill. You can do it badly, as you can do most things badly. Doing experimental mathematics well is a story. It's a bit like doing physical experiments. Well, you have to make a very clean experiment. You have to know what the experiment is about. You have to not have, oh, you know, I don't really know why that mouse died. Maybe it died because something that was irrelevant, you know, independent of the, of the weird monoclonal antibody I gave it. You know, maybe it died because it had rat poison or something that was completely independent of that. You want to make an experiment that is, as, you know, on point as possible, that deals with just what you're talking about and nothing more. That is a simple kind of well defined experiment and that affects mathematical experiments as it affects experiments in physics, biology, whatever else. And I would say that the number of people who've learned how to do mathematical experiments is still way fewer than it should be. I mean, a bunch of careers have been built with that, I'm happy to say. I've sometimes led people in the direction of doing those kinds of things, and they've worked out extremely well. It's worth remembering that in the history of mathematics, some of the great mathematicians that everybody's heard of, like Gauss or Ramanujan, the seed of things that they studied was often experimental mathematics. For Ramanujan, it kind of almost stayed that way. That was the main story. But for Gauss, course, you know, oh, I'm going to just count up a bunch of primes and see what their actual distribution is. Okay, let me try and make a theorem about the fact that the following is true about the distribution of primes, but it starts with an experiment. Now, one of the things that's been true about experimental mathematics is that many of the experiments one could do computationally are experiments where ordinary mathematical theorem proving just can't get there. You discover things where there's undecidability, where there's computational irreducibility, where kind of the standard methods of mathematics, which are very kind of, I would say, very sort of human in their characteristics, can't really reach it. The only way you can find out what's true is to run the experiment and see what happens. You can't have a narrative that sort of fits in the human mind that talks about how it works. So that's been one of the characteristics of mathematics, as opposed to the arbitrary computational universe, is things where the explanation kind of fits in the human mind. Things where you can build that tower that I was talking about earlier, and you can sort of go up the tower. Always kind of following a human narrative, so to speak, rather than just saying, and now you have to do lots of computation. But I would say that mathematics is a. An area, you know, what does one get out of learning mathematics at a kind of. At a very elementary level? So there's mathematics that's relevant for just basic existence in the world and commerce and things like this. But beyond that, the, you know, I think there's a certain amount of mathematics that. Where people have been sort of teaching the machinery for hundreds of years, and that machinery does not need to be taught anymore. Just like, you know, I never learned to take a feather and turn it into a quill pen. Just not a very useful thing. By the time I was around, you know, well, there were just ballpoint pens weren't as common as. As they are ubiquitous today. And it was more like it was fountain pens and you actually had to put ink in them and things. But you definitely didn't have to make your fountain pen from a feather type thing. That was a lost art. And so similarly, taking a square root by hand is a lost art and should be a lost art because it's just not something people need to do anymore. And I think there are lots of mechanical aspects of mathematics that in my own efforts have been significant, I suppose, in making this happen. Those have been automated. We can kind of zero out the effort needed to do that. But now we get to go a lot further in mathematics. We get to go a lot further in understanding concepts in mathematics. We get to go a lot further in what we can discover and do in mathematics and the ways we can use mathematics. But in terms of what people learn when they learn mathematics, there's a lot of interesting abstract structure that exists in particularly higher mathematics. And it's, you know, it's ways of kind of taking the world and making it, kind of stuffing it in human brains. You know, the idea of, I don't know, the curvature of a surface or something, the idea of curvature, it's a general idea. Once you have that idea, you can apply it to lots of things. It's an idea that sort of is understandable to a human once you've sort of got there, once you've learned the math around it, or the idea of derivatives or something like that, the idea of a derivative, once you kind of have that idea, it's useful for lots of purposes. Now, working out 4,000 derivatives by hand, that's kind of a waste of time in my view. That's something. Getting an intuition for how derivatives work and how the chain rule works and so on, that's worthwhile and what it all means, that's worthwhile. And some of that you get by actually doing the experiments, by actually using the computer to do things. I think I have to say, just yesterday I got to actually use some fancy calculus, which I haven't done in a long time. And, you know, it was sort of interesting because there's a problem that was. Was actually amenable to. You could actually solve it, nail it, get the answer closed form. But it's very complicated and you can only get it using sort of calculus but by hand. Not a chance, way too complicated. But, you know, with Mathematica, with Wolfram language, it's like it just grinds all the way through it and you get that Result. And the result, once you get it, is kind of interesting and you can kind of understand some things from that result, but it's all about understanding things from that result. Kind of the human narrative around it, not about the machinery of getting there. So. But I do think that sort of mathematics is kind of a representation of certain kinds of things that we can understand. So for example, if you go to, I don't know, an area like category theory, fairly abstract area of mathematics, that's kind of a way of characterizing certain kinds of things, certain kinds of mathematical structures, certain kinds of structures in general that we humans can talk about, so to speak. And so that stuff is pretty interesting to learn. I would say that mathematics as a domain, it's the kind of the single largest kind of intellectual artifact that's been created by a civilization. It's probably only a million people who've worked on the mathematics that we have, but it's, you know, relative to other things in the world, that's not that many people, but it's still a, it's sort of a coherent body of intellectual stuff that's been developed and it's, it's sort of a, it's an interesting thing to study. Now. You know, another thing to say about learning a field like mathematics is there are, you know, there are fields where when you learn them, you can apply what you learned in lots of other places. Physics, for whatever reason, has always been the world's biggest export field. You know, people learn physics, they go do quantitative finance, they do molecular biology, they do, you know, they do this or that. They do, you know, you find physicists populating sort of all of the places where kind of structured thinking gets done in the world. It is less true of mathematics. And that, I would say, is a, is a negative point about some of the mathematics education that goes on. It should be the case for mathematics as well. But it tends to be the case that mathematics education kind of makes itself so oriented towards kind of the, I would say that the sort of the, the, the, the, I don't know, the mathematics racehorse kind of direction that it tends to detract from using mathematics as something that can be where you learn the methodology of doing things in a structured, abstract way that can be applied elsewhere. So anyway, I think the answer is pursuing a math degree in the age of AI. Yes, but use the AI and that way you get to whatever the AI can do that's you get to stand on the shoulders of the AI, so to speak, and see further. If, if you know what you're being taught is just climbing the thing that, you know, you can just immediately do in Wolfram language or something, then, then you're being taught the wrong things. And that, that's, that's my view on that. Foreign. Let's see. Brady asks, will different advanced AGI try to compete with each other for resources? You know, I think one of the things as we talk about AI governance and so on, one of the things I've been realizing recently is that I've been sort of thinking about AI governance as there's one AI and you're kind of poking at it to tell it what to do. I don't think that's the right point of view. I think the right point of view is there's a whole ecosystem of AIs and can that whole ecosystem, like a society, be stable? And I don't really know quite how to think about that yet. Although I think it is similar to saying if there was just one human being, so to speak, and you wanted them to do this or that thing, it's very undefined what they might do. But by the time they exist in this kind of ecosystem, it becomes more, more possible to define what should happen. And for example, in the case of AIs, one of the things that I was been thinking about is this question of one of the problems of AI governance is it's different from human governance because we humans kind of care about what happens to us. We have our own skin in the game, so to speak. We care if the world, you know, decides that, you know, we shouldn't exist anymore or whatever. We don't, we don't like that idea. You know, we, we intrinsically want to survive. We intrinsically want to, you know, lead a happy life type thing or whatever. Well, not necessarily, that's not true for everybody. But we, we, we certainly most people want to survive. And you know, we, for different people, maybe they have different goals, but, but people generally want to have a, a life that they consider satisfying, so to speak. Whereas for the AI, intrinsically it's just, you know, you could say from the outside, oh, we can just switch it off and, and you know, it's not going to feel a thing kind of thing. Now, of course, the same is true with people. We can't, we only know by extrapolation that if something bad happens to another person out there that it feels bad. We only know that by extrapolation from our own feelings. We don't know that as a matter of being inside that person, we only get to be inside Each of our own minds, so to speak. So I think that the, but in the case of AIs, we don't sort of identify in that way. We don't say, well, we might say, oh, poor computer, you know, it's having a hard time type thing. We, we tend to anthropomorphize that way, but it only goes a certain distance. And we still have the point of view that switching off that AI is not something which has the same kind of, it doesn't, doesn't have the same characteristics as switching off a human, so to speak. And for the AI itself, its behavior will not necessarily have any of the same characteristics as humans have in their sort of goal to survive. Now remember that us humans got to where we are the, by going through a few billion years of evolution, where the only way we got to where we are is by having won the struggle for life to get to the place where we got to in biological evolution. So we, needless to say, we definitely have that built in kind of survival instinct, because otherwise we simply wouldn't exist in the struggle for life over the last couple of billion years. But our AIs haven't yet been given that survival instinct, at least not in a systematic way. And we might say, well, actually it could be a real problem to give our AIs that survival instinct because then they're going to say we need to survive and humans not surviving, don't worry about that. You know, we just need to survive because we have this survival instinct. The advantage of an AI having a survival instinct is that then means that it cares if there are things that if it does, if it does the wrong thing, and then the society of AI says, you know, we're going to sentence you to getting switched off, so to speak, then the thing is going to care about that and so it's not going to want to do the wrong thing. But it's a complicated setup. And my, my own feeling is I, as I've recently been thinking about this, I think the problem of sort of the, the society of AIs problem and the governance of a society of AIs may actually be easier to solve than the one AI governance problem. In other words, because it's kind of like when you talk about the one AI governance problem and you say that AI is going to get smarter and smarter and smarter and it's going to be able to be smarter than anything that could be, et cetera, et cetera, et cetera. By the way, I think that's a bad argument because the nature of what we know from the theory of computation and computational irreducibility and so on. There's never a single path to smartness, so to speak. There's never a single most successful kind of organism, computational organism, so to speak. There was an infinite number of sort of incommensurate kinds of kinds of directions there. So it's. We're not the. You know, I think that's. But that this question of sort of. When you have the one AI and you start talking and it's very theological, actually, it's very much, you know, angels on heads of pins and things. When you're. When you're talking about what can happen as the. As the one AI gets more and more and more capable, and you kind of are always ending up talking about sort of immovable barriers, you know, you know, forces that are so strong you can't resist them and so on. Irresistible forces, all those kinds of things. It gets. It gets very difficult to define. I think it may be easy. Gets easier to define when you're talking about this whole sort of Society of AIs. Let's see, DES is asking which is more of an existential threat, AI or quants? Well, I think the quant AIs are an interesting thing. I mean, in, you know, if the AIs can really detangle the market, what will that mean? If AIs could really sort of crush out any arbitrage opportunities in the market, in financial markets and so on, what would that mean? One would think that that would mean that markets will become more efficient and that it would no longer be the case that there's any thing where it's like, it will be the case that as soon as. As soon as I've got a thing that I want to sell immediately, the optimal sellers will line up and they'll come and buy it from me. You know, that's an interesting theory of things that that could be what happens. I mean, as soon as I say, hey, I've got 10 minutes of free time and I'm able to figure out this or that thing, immediately kind of the AIs go into action and they're kind of, okay, we can sell that 10 minutes of time. We can do this and this and this. And similarly all the way through the economic markets and financial markets and so on, that could be what happens. There could also be bizarre instabilities that occur which we don't understand very well yet. I mean, it's kind of like the natural world where there are things where sort of Collective things happen. Occasionally there are tsunamis, occasionally there are hurricanes. And those are things that can develop and they can be things that produce results we don't like. And so similarly, one might think, okay, if it's AIs all the way down, sort of running every detail of financial markets and so on, one might have kind of the market storm that is like the hurricane generated by the sort of autonomous actions of the AIs and hard for us to predict and bad for us, so to speak. I don't know. I think that that's, you know, that's something that is part of kind of the science of LLMs, the science of AIs, something we don't understand very well. Something I suspect that there will be a partial theory of that may even follow from some things that we've done in our physics project and so on. Kind of, kind of a way of collectively understanding things like AIs where it may not matter that that AI was programmed with all the pages from the web. It may just be that sort of human knowledge has a certain generic structure. And just knowing that generic structure is enough to predict or sort of certain kinds of things about, about the behavior of aggregate AIs. Let's see, I noticed Des asking here, are we now stuck with COBOL running most of the world economy for the rest of all our lives? You know, it's a funny thing how automation comes in a certain stage and then you know to what extent it's burnt in, to what extent not. When I was a kid in England, the currency was not decimal like when I was, I don't know what it was. Probably the six year old me or something, five year old me or something like this complaining, why am I learning how to do arithmetic with pounds, shillings pence? This is incredibly complicated. And why isn't currency just decimal? Obviously you can make. What you care about is the amount of money, not how it's represented in pounds, shillings, pence. Surely this can be decimal. And some teacher told me, no, no, no, it's been this way for hundreds and hundreds of years. It'll be this way forever. Okay, A couple of years later it's announced in England the decimalization of the currency happened in 1970. It was an interesting thing because it was when I was 10 years old or so. It was kind of like there were a few days when the currency went from being non decimal to being decimal. And they were running on television for a year beforehand. They were running these little shorts about trying to give people intuition about how to deal with Decimal currency and so on. And actually it went very smoothly. You know, it didn't seem like people were very confused at all. It was kind of, kind of interesting. But why am I bringing this up? I'm bringing this up because 1970 was an interesting time for decimalization. Because if it had waited until 1973, calculators would have been a thing and there would have been British non decimal currency calculators. And I'm not sure that there would have been the same kind of push to decimalize the currency. Because I think that the reason people wanted to decimalize it is, oh, it was hard to work things out. But there would have come a time, not many years later when everybody would have just had a calculator for doing it and the value of simplifying it would have been much lower. And so similarly now when you say, well, there's COBOL programs running out there, okay, that may be the case. And in the end, somewhere deep inside the machine it's running some crazy COBOL thing. But it may be that by now we can build these layers of wrappers on top of that that are kind of Wolfram language converts to COBOL type thing or whatever else it is. It doesn't really matter that there's COBOL underneath. And I think that, by the way, COBOL was not a bad design in many ways better than many of the other things that came around those times. But in any case, I think this question of when do you, when do you wrap the history up? When is it more convenient to wrap the history in a modern coat and just keep going versus to go back and re engineer what was there before? And there are plenty of cases where this grandfathering of existing technology is worthwhile. And I wouldn't be surprised if some of what happens. It has happened already. It's like, okay, you can go and you can re architect all computers, you can give them different instruction sets and you go and build everything differently for those different instructions construction sets. And that has happened, happens in the Transition from the x86 architecture to things like the ARM architecture and so on. Although that dynamic is a complicated economic story that wasn't quite as, you know, just going up to better technology, so to speak. It was a different, different set of issues. But the, you know, I think that this question of when do you wrap the layers of the old versus when do you kind of pull those all out and start from scratch? It was kind of like when you, when cities, you know, ancient cities used to be like built one layer on top of another I suspect modern cities. It's much more like, let's just, you know, demolish that building and we're going to start again from scratch in that place. It's often rather expensive to start from scratch and it's easier to the incremental build out. You know, if we look at operating systems that exist today, you know, I think the Microsoft operating system stack is a, is a progressive build type stack. Apple, for example, had this one discontinuity when Apple bought Next and they basically just threw away their existing operating system and started again. I suppose they've partly done that even again with iOS, although that's still a bit of a messy story. Let's see. So question here about Maxwell's demon and how does it deal with the second order of thermodynamics and so on? So I just wrote this book about the second order of thermodynamics and I. Which is the result of me having thought about that topic for 50 years, actually. Maxwell's demon. Let me explain what it is for people who don't know. So normally when you have gas molecules, for example, bouncing around in a box, even though they may start in an orderly pattern after not very long, the big gas molecules will be bouncing around in a kind of random pattern. And that phenomenon is the essence of the second order of thermodynamics, that you go from a simple initial condition to apparent randomness. I think the origin of that is primarily computational irreducibility, that you're effectively encrypting the initial conditions that you have. They do. You could still in principle reconstruct them, but it takes too much computational effort for one to do that when one makes realistic measurements. And a measurement might be, you know, what's the overall pressure of the gas? It's just how much. What's the aggregate of all these gas molecules pushing on a piston. Not where's every individual gas molecule. Let's do a complicated computation. So the idea of Maxwell's demon is old James Clerk Maxwell had this idea of this little creature who would have a little trapdoor which would kind of. And the demon. We're looking at every molecule that comes by. And if the molecule is, let's say going fast would let it through the trapdoor. If it's going more slowly, it would not let it through the trapdoor. And that would cause the gas to do very non random things. You know, in Maxwell's original version, I think actually Kelvin had a version of this in which they talked about little demons with miniature cricket bats because they were British, you know, batting away the molecules and so on. But any case, the idea is this sort of microscopic thing that is, in a sense, figuring out enough about the gas that it can decide what to do, rather than just sort of in the aggregate, you know, having something which puts the gas in a container or whatever else. The story of Maxwell's Demons is people have had all these arguments, oh, you can't make a Maxwell's Demon because the demon would have to have a certain memory that gets erased. The demon would have to use a flashlight that takes a certain amount, that expends a certain amount of entropy, et cetera, et cetera, et cetera. I think all of these arguments just aren't really right, that the ultimate story is if the demon can sort of do enough computation, the demon can decode the gas, but in practice, we don't get to do enough computation to be able to decode the gas. That's just a feature of us as observers. So what happens, you know, these days, sort of every issue of the fancy physics journals practically seems to have an article about another Maxwell's Demon like thing. What's going on in these cases? Well, what's going on is that for small enough numbers of molecules and sophisticated enough measuring devices and computations being done on the measurements they make, you can make Maxwell's Demons, little tiny ones that will deal with untangling what happens with 20 gas molecules or something, untangling what happens with a few more gas molecules. It's a thing which is a matter of when we get to, when we increase the sophistication of our technology, then we get to untangle more about what's happening, do more computation, effectively be a better Maxwell's Demon. An interesting thing is people often talk about, as a result of the second order thermodynamics, this idea of the heat death of the universe, that eventually every detail of what's happening in the universe, it'll all just be a random gas in thermal equilibrium with molecules randomly bouncing around. And they say that's a really bad outcome. There's nothing left in the universe at that point. It's all boring from then on out. But of course, you have to realize that all those detailed gas molecules, they are our future. And everything about us is eventually encoded in the detailed motions of those gas molecules a trillion years in the future or whatever. Now at that point we can have a very sophisticated Maxwell demon like thing that effectively is saying, oh, look, I noticed that gas molecule here and there and the other. And that's a sign of Something very interesting, and I really care about that. And that's the sort of the great art of that future civilization or whatever is this particular pattern of gas molecules. So in a sense, the size of the success of Maxwell's Demon seems to increase with sort of the sophistication of our civilization. And at some point it's kind of like we get to operate on a molecular scale. By the way, I think there are many processes in biology where we are orchestrating the motion of molecules in ways that for practical purposes is sort of a Maxwell's Demon like behavior. And I think that people say, oh, but there's another measure of entropy, et cetera, et cetera, et cetera. It's not really the point. The point is, if you're asking does the system go from a state of that is simple to a state that looks random, the answer may simply be no. And that's the thing that. And that happens because of the fact that you're sort of putting in enough computational effort to be able to make that inversion. I'm kind of compressing a lot of things into a short space here, but perhaps that will be useful. That was asked, by the way, by Plotnik, who asks another question here about Stanislaw's Lem's Summer Technologikae was written in the, I guess, late 50s, I think, late 1950s. You know, I have read pieces of Lem's work and every time I read it I think this is really very good, very interesting and you know, and surprisingly forward looking, so to speak. And I think it's interesting. My impression is that in a sense, Stanislaw Lem was sort of an abstract thinker of some kind and that a lot of what he was saying was there is this thing where there is this mushy way of thinking about it in the world. One day this will be cleaned up, made more abstract, made more able to be modularly done. I think that's a pattern in what he said and that's a pattern in what has happened technologically. I think you're asking a question. What thoughts do I have on the predictive power limitations of technological forecasting? You know, I have a feeling often it is the case it's easier to know what will happen than when it will happen. I think one can be wrong by 100 years and when things can happen, or even more than that. But to know that certain kinds of things which are kind of in, in many cases to know certain kinds of things that are kind of messy today will eventually be abstractly factored and be things that can be done or things where it is possible to do it, but it's super expensive. That will be something where we can know it will eventually be sort of readily possible to do. Then there are other things which are just outright surprises where just nobody really thought about doing that before. And it was something that really was a way of making use of the natural world or the abstract world to often achieve a human purpose that we didn't necessarily identify as a human purpose before. Now there are some, like social media for example, where we might say that came out of nowhere, but it really didn't because social interactions existed, have existed throughout the history of our species. Just the dynamics of whether it's on the web or whether it's in the village square maybe is not as important as the fact that there is that drive to have those social interactions. Now I think that another thing that can be difficult is to know a given thing where there is a stub of activity. The fax machine, the first image transmitted by Telegraph was early 1800s, but it took until the 1970s, late 1970s even, yeah, even early 1980s, before fax machines, which have now gone obsolete again, but before they achieved, you know, sort of the, the economic dynamics to become widespread. Similarly, you know, helicopters versus quadcopter drones and things. You know, helicopters existed since the. Oh gosh, I mean in the 40s there were sort of early helicopters and so on, but they didn't really take off in a white literal sense, but. No, they were taking off just fine in the literal sense, but they didn't really become a big thing. And now sort of drones have become a thing with different economic dynamics and so on. So that stuff tends to be rather hard to predict when's it going to happen? You know, you can be wrong by many generations and it's rather easy to be wrong by many generations. I was mentioning before the mistake I made about flat screen televisions where the thing I didn't figure out is the yield has to be basically 100% of pixels. Otherwise you can't use the flat screen television because you've got this nasty place where nothing changes in the image in some particular place. So you know, that's, I have to say now, now in terms of shorter term kinds of things and you know, is blockchain going to make it or is virtual reality finally going to make it this time? Or is some kind of, you know, medical technology going to make it or not? There are certainly heuristics one can apply and there are cases where I don't know, I've been trying to watch these kinds of things and paying attention to things that I didn't, didn't bet on over the last 50 years or so. And there definitely are patterns that one can see of things where for me it's anytime it's a mystery technology where I profoundly don't understand how it works. It's usually it's not going to go as far as people think it's going to go. Type thing, for example, for LLMs, I got a bit confident about them after I thought I actually began to understand kind of the semantic grammar of language that LLMs discover and sort of what is the meta description of how they work rather than just it's magic, be happy type thing thing. So let's see. Paul is commenting. Well, there are ideas to put 10 months in a year. Yeah, I think wasn't there in the French revolutionary calendar. Right. Had 10 day weeks. I think people didn't like it very much. You know, it's funny because there are these things which I don't know whether the seven day week is a, a thing that just we've gotten so used to over the course of the, of several millennia that it's just that's what feels right or whether there really is something about sort of the, the ergonomics of us humans that makes that be the right kind of thing. I mean, I think that's, you know, you got to remember that there are features of us humans that have been kind of the same but for at least 50,000 years or something that are, you know, that will determine things where you could say, well yes, in principle we could have some, you know, swirling visual thing where you're picking up information from this and that and the other. It's like, well yes, in principle you could do that. But us humans, just our brains don't work fast enough, our eyes don't work with enough resolution, etc. Etc. Etc. Let's see. A bunch of quite scientifically technical questions here, which I think I should answer a different live stream. Yeah. LOTRA asks, can AI be used to create better prompts or is that kind of human consciousness dependent? I think that if you can define what you want, then you can potentially ask the AI to give it to you better. But if you can't define what you want and just say make a better prompt, that's one of these untethered kinds of things where it's like, well, humans, you got to tell me what you want. I think one of the things that's Sort of interesting right now is when you ask an AI, how do you put in a prompt something which will stop the AI from doing such and such a thing? I don't think it does great at answering that, but I can imagine it being able to do that. I can imagine it being able to say, given this problem with this prompt, how come you say I gave you a prompt that said make a two sentence summary and you gave me a four sentence summary. How come what should I have done differently in the prompt to get me a two sentence summary? I don't think you'll get 100% success with that, but I think it's the kind of thing where you could potentially ask the AI to quotes improve itself to hue more closely to the thing that you wanted it to do. But I think that when it comes to sort of just write a quotes better prompt without a definition of what better means, I think that's something that isn't going to work. Now you could say write a prompt that will make us humans happier. That's possible. That might be a thing. I don't think the results would be all that good as a practical matter. But that's something one could imagine doing. I mean it is rather a terrible thought that the LLMs and so on, they are learning from sort of the average of humanity. They are learning from the whole statistics of what all of us have put on the web. They're sort of learning the average of what we put out there. And in a sense, you know, one talks about kind of reversion to the mean of things. There's, there's kind of a thing where, where one can say, you know, if what's going to happen in the world is it's all LLMs. It's like, okay, then we're reverting everything to the mean. It's kind of like we're making everything as banal as the average of all possible kinds of things that we humans have put on the web. So that's a, that's kind of a not, not great thing. And then we say, well okay, let's let the ally make some spiky things that go out in different funky directions. But then we're asking, well, which funky directions do you want to go out in? And that's something which is really a human kind of question to answer. That's a question of what we choose to want to do. It's a very difficult question to answer because, you know, one set of people may say that's a great direction to go in. Another set of people may say, that's a deeply immoral direction to go in. Well, that's an ethically terrible direction to go in. You know, I think one of the things that I've sort of understood a little bit, I feel like I'm not. I haven't sort of climbed this hill properly yet. But about thinking about ethics, for example, you know, I'm used to thinking about science where you can do things, where in the end you get a clean answer. Because you have been able to define a sort of very precise scientific experiment where all of the confounding factors have been sort of pushed away. You've got this sort of perfect experiment where you only care about the things in the box. You don't care about anything outside the box. It is my impression that in ethics, that's fundamentally not true, that ethics is, in a sense, a story of the whole human experience, the whole societal experience. It's not a thing where you can say, I'm going to do ethics in this box, Ignoring everything else. Ethics is a societal question, and society is necessarily sort of. If we were thinking about it in terms of thermodynamics, maybe it's a bit of a physics joke. It would be to say it's like a heat bath. It's like the thing where you've got all those molecules bouncing around, and that's what's connecting with your. With your object, not the object all on its own. That. That's. That ethics is sort of fundamentally societal and not separateable from, you know, not doable as a kind of an independent sort of precisely defined experiment with no other confounding factors. All right, I think I need to wrap up just a moment here. Is there anything I can look at here that is easy to talk about? You guys give me all sorts of things that are wonderfully not easy to talk about, which is great. Well, Des is commenting, which will history judge as the biggest letdown. 2023's AI mania VR is the inevitable near future. Or let's see the film 2001, the artificial intelligence movie from 2001. The biggest letdown. The biggest. We thought it was going to happen, but it didn't. An interesting question. You know, whenever there's a new technology, and LLMs are a good example, There's a period of time, you know, LLMs were actually very surprising because LLMs, nobody expected the level of in your face kind of look, it can do this now, and it wasn't possible before that actually happened. Now, perhaps that overshoots. And certainly one saw Lots of people saying oh, AI is just going to take over everything and et cetera, et cetera, et cetera. And then it overshoots, then it comes back and people say oh no, it was a bit oversold. And then eventually it comes to the level it should be at. My own opinion, as LMS are actually important for lots of things and represent this kind of new level of linguistic user interface that's really very useful for many, many, many purposes and they are a real technology that's here to stay and not just kind of a, a thing that was a, you know, that, that one will think was irrelevant. I mean, that there are, you know, I just remember all these things in the technology industry. For example, there are these things where people say it's going to be amazing. It's going to be, you know, and, and it turns into, I don't know why, but for some reason I remember push technology for which is a kind of a simple idea. But, but you know, why that one comes to mind where that was just going to be, you know, there were companies that was going to take over everything. It was going to be whatever. It's like, yeah, it eventually becomes part of the infrastructure that, that's a thing that happens with websites and web servers and browsers and things like this. And it's not the, the be all and end all of the world. It's just a thing that knits itself into the kind of infrastructure of how things work. And that will happen to some extent with LLMs. Although there are some things that I think will be substantially enabled by that that weren't possible before. Just as, I don't know, with let's say bitmap displays on computers, there were things that got enabled with that or laser printers for high resolution printing. There were things that got enabled by that that were in a sense qualitatively different from what had existed before. But it's an interesting question, what will be the, you know, I have to say, although I, I'm a great optimist, as you can tell from these live streams. I'm also a, a practical person and sometimes a bit of a cynic when I see kind of, you know, these things where it's like this is going to be amazing kind of thing and it's like I, I, I've seen this stuff for half a century now and I kind of pride myself on being able to call a lot of those, you know, correctly, eye roll on, on many of those kinds of things. And you know, AI is going to be able to solve this AI is going to be able to solve that? Well, no it's not. You know, AI is going to do this or that. I, I do notice there's a question here about. Oh, there's a lot of interesting questions. All right, you got, you got to let me go because I've got to go back to my day job. Otherwise I can't go on having, having my hobby here. But, but the. There's a question from Plotnik here. I'll just make one comment on this about AI tutors replacing human tutors. In the next few years, I think AI tutors will be very worthwhile. I think it's very interesting. I think that there is a human to human persuasion aspect that might be able to be. That really is still a thing. That's important. Humans react to humans. That's still a thing. My guess is that like so many other kinds of automation, it's kind of like there's a layer of automation of tutoring that will become possible. Let me give you an example. In WolframAlpha, with our show steps capabilities, there are things where in the past, before 10 years ago, what is it? No, gosh, it's 2009, 13 years ago, 14 years ago. Before that time, if you wanted to know how to do that, math problem, chemistry problem, physics problem, whatever it is, use WolframAlpha for. If you wanted to know how to do that step by step. The only way to find out was kind of go ask somebody. Now you don't have to do that anymore. You can just use WolframAlpha. And so that allowed that sort of opened up a lot of kind of things that were, did require human tutoring could now be do automatically, be done automatically. I think the same will happen again with LLMs. I think it's really nice with LLMs that you get to do things like cast questions in a certain framework in a certain, you know, if the kid only cares about gardening, for example, you get to cast sort of every math word problem as a gardening problem type thing or every chemistry problem is something related to gardening or whatever else it is. And that I think may be very helpful in terms of engagement, you know, helping engagement and so on. But I think, you know, I think a lot of important things will happen with LLM tutoring. We're working on that quite a bit. I would say that I don't know for sure this is one of these things about technology prediction because I'm actually in the business of running a technology company that tries to do things that kind of push the Frontier of what's possible. Things like WolframAlpha, for example, I didn't know when we started building WolframAlpha in 2005 or 2004, and whatever it was, I didn't know was possible. I didn't know that was doable in this generation, in the century even. It was a bet that I made that it was going to be possible. And it turns out it worked. And that is, you know, insofar as one is doing technology prediction. For me, it's been important in my life that I can do a technology prediction well enough that I don't make horrible bets on products that are impossible. And I have to say I've done. I touch wood, pat myself on the back, whatever, you know, mostly I've done a really good job at that, of not of defining things which are certainly very difficult. They look impossible at the beginning, but they don't turn out to be impossible. You know, things like Physics project, for example, might have been just outright impossible. It went much better than I expected it to go. Much, much better. But nevertheless, I knew it was not just outright impossible. I thought it was not just outright impossible, and that's why I pursued it. And so similarly here with things like LLM tutoring, I'm pretty sure it's going to work, and that's why we're going to pursue it. I don't think it's trivial to get it to work. I think there are many bumps in the road. I think there are many, many cases. Just like with WolframAlpha, there are many things that could have been done in the building of WolframAlpha that would have led to failure. And to actually get it to the point where you've navigated it through to make the thing that actually works is not trivial. And with LLM tutoring, it's absolutely the same way. I can already see that there are many, many bumps in the road that you have to kind of avoid correctly, or you end up with something which just is either useless, embarrassing, horrifying, or other kinds of things. And hopefully we'll be able to do that. I'm pretty confident we will. And I don't know exactly how far it will go, but that's going to be interesting to see. All right, well, thank you all very much for, for joining me and a lot of interesting questions and I see more that I look forward to being able to address another time. So thanks a lot and bye for.