Ask Me Anything with Conrad Wolfram: The Math(s) Fix, ChatGPT and Education in the AI Age

Well, hello everyone. For my ama, my name is Conrad Wolfram and my role has been for over 30 years now to try and bring computational intelligence to everyone. Through Wolfram, we've kind of been pushing the boundaries of computation and the good it can do, we hope in society for moving us forward. And for the last 10 or 12 years I have been also pushing the human end of that spectrum as well as the technology end to try and figure out how do we educate our people better for the, you know, for the, for the AI age so that we are for the future as people in charge of, of the machines, not the other way around. And that's, I expect both of those topics will, will come up in the session. The computational knowledge is an area in which people think broadly using the system of computation. And it's not just computer science, not just getting the computer, programming the computer, but in a broader sense. But what I thought I would do in the session is spend a few minutes just outlining how, you know, some, some parts of what I've been thinking about for the last few years and some of the things that have happened this year in changes to the world. And then, you know, most of the session will be throwing it open and look forward to you asking me what, whatever you like. And you can paste those into your sessions, whichever platform you're, you're using for this. So look, it's very good to be here. Virtually this year has seen a shock to the system for education in particular and many other areas, and that's ChatGPT's launch. Amongst many effects immediately questions what we're learning and how in the starkest possible of ways, it appears to be able to answer highly valued assessments better than most human students do coursework as well, and as well as teachers to teach us humans. So in many ways this has been a dramatic year for what possibly has changed in education. Exciting for us at Wolfram, we were announced as one of the first 10 or so plugins for ChatGPT. And that means sort of even better range of what it can do, and particularly injecting facts to ChatGPT's impressive but sometimes fanciful prose or poetry. In AI terms, I suppose Wolfram is the sort of aspergic end of the spectrum to chatgpts, not on the spectrum. But as in real life decisions, the hybrid of both can be an absolutely ideal solution to getting good answers, good decisions, good ideas. So it was with a recent. One of the little experiments we did was we took a British A level paper. That's the exam that people take before they go to college in the UK or at least in England and Wales. And we got Wolfram plus ChatGPT to do the paper and it got 96% so that's way into the A territory. ChatGPT alone got 43% which actually isn't bad if you consider everything. And this was just pasting the questions in. There's no we're not interpreting them. This is straight doing that. And by the way, there's a thing called Further Maths which is sort of harder and I think we got 94% on on the paper we tried there. So all of these things bring up many questions for education and many of the questions that have come to me around these topics. I realized I'd already answered my book the Maths Fix and see it behind and on screen. This was published in 2020 so so well ahead of the latest AI innovations was ostensibly about the problem with mainstream school maths education, the fix to solve it and how we make that change when the real world has changed but education hasn't reacted. So I say ostensibly because I'd slotted in a byline of an education blueprint for the AI age, a rapidly changing outside world, a mainstream subject. At the heart of the chains driven by new technology is the story of mass. But it's now the story of what we have to do across education to transform for our latest fastest ever industrial revolution this AI age. So quite a bit of what we've learned from maths problems are characterizing the issues. Providing answers are not limited to maths but key to making changes across all of education. So that's why I've relaunched the Maths Fix with a forward to explain how it is a blueprint for Raih. And ChatGPT's sudden bursting onto the scene has given us the moment hopefully to spur change like we've not had in education at least for a very long time. So I'd love for this AMA to be about any AI age topic or computational talk or anything really. But from the future of education, learning, teaching, assessment etc to maths to technology. Anyway, thanks for attending, looking forward to digging into these questions here. Really appreciate a lively discussion and ideas. So let's over over to. Let's let's start off some questions here and let's see, we've got one question. I'm curious how AI can assist teachers and tutors better to better help student master subtexts and as a sort of follow on question to that is will it be important for my two year old to have multiple different AI teachers in his early life. Both very interesting questions, I think. I mean, first thing to say about, about the recent innovations with, with ChatGPT and other LLMs is we don't yet know their effect in the real world. So just to step back a moment, I mean, when we talk about the math education crisis, in a sense for many years, possibly decades, the crisis has been building because what's in the real world, which is computer based mathematics and computation where the computer does the calculating and the human does the other stats, that has yet to be replicated in mainstream education. So we've got a complete divergence of subject with ChatGPT and other LLMs so far. We don't yet know the permanent effect in the real world. We have good clues, we've got the start of a change. So we don't quite know what real world we're aiming at quite yet. And so there are sort of two pieces of this you need to think about a little bit. The subject, what is the subject that we're trying to set up for people in this new world and how then do we get there? What's the pedagogical approach to that subject? And potentially chatgpt and the like can help with both of those. So how can assist teachers and tutors better help students master subjects? I, I think it can pose new questions in the sense that you're asking it questions and you can see what kinds of answers it comes out with. You can see how it structures the answers and those may be different to the structure that teachers can give. And I think teachers should use it as a tool in the same way as you'd use a pencil as a tool, in the same way as you'd use Google as a tool or Wikipedia. I think these are, it's a useful tool for getting a different perspective and I think we should do as much of that as possible while we're figuring out what the real world result of that is. And in the same lines, you know, I think it's beneficial often for particularly young children to have many different viewpoints with us within a certain range. And AIs offer some of those viewpoints. So I think that, I think having different AI teachers with different ways to look at things as well as human teachers is a positive. But you know, as with all new technologies, there will be certain negatives that come along as, as, as it, as we spot how it works. But I think overridingly this will, will be positive. So I hope that's a, that's a, perhaps a An answer to a little bit of an answer to. To that question. Another question we've got here is wondering if ChatGPT might assist in checking theorem proof and attempts, attempts. Look, I think one of the, I suppose, things to understand about the computational process. Let me see if I can just quickly put up a slide here. I hope everyone can see this. This is a slide of what I count as the sort of four steps of computational thinking. And in a sense, when we talk about maths or computational thinking, we're talking more or less about, in my view, four steps that you're going through. You're defining a question or questions that you want to answer. This is the thing you want an answer or decision to. You're then abstracting it. You're then saying, well, instead of talking about this, let's say in English, I'm going to try and talk about it in math. And traditionally, this means writing out fine mathematical expressions. More recently, this means writing code, often to represent the question that you've got now, why do we do this? We do this because you can be much more precise. We can put hundreds, in fact, thousands of years of computational experience into taking that question and turning it into an answer that have been born over those years through the fantastic process, this computational process. And so that's why it's worth abstracting. And then the third step is we compute. So this is the magical step. We take the question, you know, here's an equation, for example, and we work out that x equals 3. And that's this step 3, using all the different algorithms and processes we might have to do that. And the fourth step is we say, okay, x equals three. Does that answer the original question we defined in a human way? And if not, do we need to redefine it? We need to go around this whole setup again. Now, up until this year, I was basically saying the major problem in math education is we're spending almost all our time doing step three by hand, whereas in the real world, we are spending this. Step three basically gets done almost entirely by computer at a much better setup than humans can imagine now themselves. But actually, this year, we are starting to see the AI impinge on step two. So it's not so much. You don't always have to manually set up the abstraction. Sometimes ChatGPT or the like can do that for you. And we've seen, for example, Great examples of ChatGPT writing Wolfram language code. So we're saying, here's the problem. CHAT GPT is then saying, okay, here's some code that might help you may not 100% work. Sometimes it does. Maybe we need human intervention there, but that's allowing us then to compute answers and then potentially we can interpret the result. So I think 1 and 4 remain very human steps for the most part. 2 is starting to be a slightly less human only step than it was a few months ago. And it's. So in terms of back to the theorem proving question, I think that I, I can't answer very specifically for you, but I think that this will. The fact that we can go through and get assistance on these different steps will help us in deploying computers for proving theorems. I think it'll be a different take on how we abstract and that itself will help the computational part of this and the interpretation that follows that. Let me move on to another question. So if we don't embrace the use of AIs and LLMs in education, what do you think will be the implications? Will there be any benefits? Look, here's my big. The view I have on this is look at history and previous technology transformations. In the end we don't have a choice. I mean, we have a choice over the short term, but this technology has been invented and is being invented in front of our eyes. It's not going away. And all we do if we try to, if we try to ignore it in education is we will end up sort of postponing an inevitable of having to embrace it, but by which time we'll have caused a lot of damage to that generation who won't understand how to operate with it and will be learning suboptimally many things that they could have learned better or different types of skills. I think, I mean, the implication in the end is, I suppose the big implication in the end if we don't embrace this quickly is that, you know, AIs will take over managing humans rather than humans taking over managing AIs and getting the benefits from them. So on the large scale, that's sort of what will happen. On the smaller scale, I think what will happen is we will have a crisis which we're already seeing of assessments where we can't understand whether something's cheating or not cheating because the real world has something new. But we assess that as not something that is allowed in education much of the time. And so therefore all sorts of people get castigated for apparently doing sensible things that are actually not allowed. So we sort of, in a sense we're in this position where we're training humans to be third rate AIs and that seems highly suboptimal. So I think there are some immediate implications of not embracing AIs, but I think that we should learn to live with them. By learning to live with them, we will also get experience of how to work with them. And also what goes wrong. I think that will be highly beneficial for our societies in understanding how to manage AIs. I think not doing that will postpone those lessons until an awful lot of damage has been done to us and our democracies and our setups. I think the big question I have about the recent releases, the LLMs, is what I brought up a few minutes ago. It's very difficult to see the full extent of. Of the real ramifications in the real world, yet these are emerging. Will the format of discourse change? So, for example, essays and reports are standard formats of discourse at this point. Those are born of a mixture of how easy it is to write something up, how easy it is for the human to look at them, how you look at information. Will those formats persistent when the printing press came out, that changed discourse in terms of how you tended to, you know, put things out. The scientific paper was born some way after that, for example, which is still in. In circulation as a format. So in a sense, these changes I think we have yet to understand fully. And, you know, when we do, we'll be able to better talk about, in a sense, that the full implications and what we ought to be doing in schools, how we ought to be preparing students better. You know, in terms of the sort of. Another question here, in terms of the sort of thoughts on teachers and schools being hesitant, I think any change like this is complicated. We don't know the. We don't know the effects. We don't know how best to deploy it. We don't know what goes wrong. We don't know who'll be benefiting from this, who'll be left behind. You know, change is difficult. Now. One of the problems I've often criticized in education right now is that the whole ecosystem has become very resistant to change. I don't mean this as individual teachers. I mean that most countries and jurisdictions have assessments that lock the subject matter of education. And in order to change the content of the assessment, you know, it depends on the place. But you've got to. You've got to align universities, if it's an end of school assessment to admit people on that, you've got to align teachers, schools, parents, governments, you know, to all agree that this change should happen. And that's very, very difficult. Right? You need Kind of a month of Sundays to achieve that. And so subjects that are considered in the center, maths being a critical. I mean, maths is probably the central problem subject for this because it's very quantifiable, apparently, although that's really calculating, not maths. That's highly quantifiable in assessments. It's considered extremely important and increasingly important as we enter this AI age because it thinks it appears to be the bedrock, bedrock subject and, you know, it's taken by everyone. So for all those reasons, making any change to the subject matter there is very, very difficult. And that has been a problem for moving the subject forward and I think has meant it being left further and further behind. So it's natural for people to be hesitant. I think what we've got to do is have a landscape where we can make a change in the subject of education, leave space to test changes, changes that have a higher chance of benefiting the students, as in fact a higher chance, I'd say, benefiting the students than wasting the time lowering actually the risk for what we're doing with them not raising it, which is what's often believed about doing such tests. And we don't have that space in most curricula. As people want higher and higher grades in these traditional subjects, less and less space is left for experimentation and we need to fix that urgently. And I'm hoping that the LLM chatgpt, so to speak, you know, revolution of this year is a reason for trying to. Is a. Is a sort of thing that's been fired in here that says we need urgent change the like of which we haven't seen education for a very long time. So I think being hesitant is fine. But I think the first question you need to ask is, given that I've got lessons to teach, how am I going to use CHAT GPT to enrich those lessons to help me to help students to get further than they would otherwise have got. And I mean, you know, for me, very many years ago, when I was studying maths at school, graphics calculators were very new. I believe I had a Casio 7000G, which was at that point very new technology. And in fact, I think it was so new they hadn't got around to banning them in maths A level papers in England. So I had a rather enlightened teacher at the time who wanted to show me how to help the calculator, help me to do more. So we did more and more difficult graph sketching, which was a requirement at the time, but we did it on more complicated functions. We saw where we could do it, we did it quickly, we got more experience. So that was an example where new technology had come and my teacher immediately rolled it in even to helping the traditional curriculum. And I think they're very similar things that can be done with, with ChatGPT at this time. So another question here. How do you think current AI technology is going to help self taught students? Do you think the person who knows how to use these tools properly could compete with someone from a top? Yeah, it's an interesting set of, I would say this is a set of questions. There are several questions sort of inside this undoubtedly. Look, one of the problems with, with self teaching, I mean one of the problems when people are alone is, is some people this allows them to really accelerate and some people, it enables them to fall behind. And I think the pandemic showed us that in stark reality and part of whether you, you know, go forwards or fall behind it depends on many facets of the student's personality, their, their experience up until then, the assistance that's on call, etc, etc when they ask for assistance, their confidence, confidence is very important in this. So it's, I think it's difficult to answer. I think there are tremendous possibilities that can be born of all the online technologies and most, most lately the new AI technologies. But I think we also have a danger there that some people will, if we assume that we're going to just leave it up to students, so to speak, to run themselves, we have a problem that some students will be badly left behind, will get into all sorts of difficulties. So it's very important that we very much help with this process. But I think what all the online technologies have allowed, which, you know, one of the funny things we've got at the moment, we've got very mainstream subjects at most schools, maths being an obvious example of that. And in the sense we forced more and more of the time to be spent on these mainstream subjects as people are trying to optimize grades. To put it bluntly, I think it's a real shame. I think we've got an opportunity at this point to expose more students to more different subjects than we ever had before. Very simple example. I was lucky enough to go to a fancy British school and on offer there were all sorts of, you know, subjects like for example, Ancient Greek. So I took Ancient Greek for a time. Now Ancient Greek is not on offer at many schools. I certainly don't think people should be forced to do it or anything like that. But I think if they get interested in it or, you know, they want to give it a try, it's great if they can try a subject just to broaden their, you know, the reach of things they're trying and experiencing. Why not? In the past you'd have to have an actual classicist to teach it in person. There, right there. Now, of course, you don't. That opportunity is right out there. So I think we need to have space to encourage much more of that, maybe even some is to give a little push to our students to try new things and not always to focus on these quotes, mainstream subjects. I think a lot of those, so to speak, outer subjects or outer disciplines or projects, they can really help a student's development in a way that the mainstream subjects really, really can't. So, you know, I think that it's important to allow. I think there's great, you know, things that we can achieve from students being left to do more by themselves. But we've absolutely got to have a system of checking in. I think the role of teachers needs to change a bit. As I've often said in the past, I think teachers should think of themselves often as the CEO of the classroom, not as the fountain of knowledge, if I can put it that way. As a CEO, I don't know everything that happens in my organization. For sure. There are many people who are much more specialized than I am. My job in a sense is to guide both high level and a more granular level what happens and to assist that process, to deploy the resources we've got optimally. And I feel a lot that's what teachers increasingly need to do with a whole plethora of resources. And that's hard to do. It involves knowing what's out there, knowing how to use it, swapping best practice and things. And hopefully more of that, you know, will, will come as we, as we go forward. Let me, let me take some more questions here. It doesn't, as a comment here does seem strange to potentially have only one teacher throughout your life. Yes, I, I would certainly agree with that. I think the. One of the things to understand, I, I certainly a lot of what I learned from teachers was nothing to do with the subject they were teaching. It was to do with their characters. The, the way they did things, the way they thought about things. I hope we can have more of that returning to what the human does, maybe even the AI does. Maybe we'll see different AI characters that will be very clear to people and that will be enriching. But I think we should. There's been A lot of talk in recent times about soft skills. I think it's kind of almost full circle because we've seen often in formal education some of those things being kind of removed. It's kind of like we've got to make people more generic, more standardized, because somehow we want to cut off all the slightly old edges that everybody has. And in a sense, we want people to experience more different types of characters and things at the same time as learning the subjects. I think that's a critical aspect of education, and maybe AIs can help with that. We'll see, you know, we'll see how we can sort of go forward with that. Another question here is, you know, do we have advice for a teacher that wants to use new technologies in their classroom, but the administration is not supportive? Look, I'm no expert on the politics inside individual schools. I mean, there are two ends of this. In the old, in, in the end, there's a, you know, there are cold, hard arguments. I hope my book the Maths Fix does help somewhat with some of those arguments. I do go through, you know, the, the sort of arguments for why this is a necessity, why it's important to involve technology in general. I'm mostly talking about maths there rather than chatgpt, but it's, it's very similar in argument and, you know, it's. I think that what we need to do is have examples of where this seems so beneficial to students. And I think the more one can show that in schools, I hope, the more the argument can be won. But as with every change, there'll be a irrationality with the change as well as good rationality about, you know, what can go wrong in the end, I suppose the, the overall arching argument, in my view, I mean, what is education for? I mean, we got to step right back. And in my view, if you want to answer that in, in sort of one line, the purpose of education is to enrich life. And I mean human life in that, how do we enrich life? Well, I think one key way is to accelerate experience of what you're actually going to meet in the real world so that you can gain more experience, not make mistakes that you would otherwise make, have a greater breadth to what you do, operate at a higher level. Those are way in which I think you can enrich life. And by the way, by enrich life, I don't just mean riches, I don't mean just money. I mean, you know, all aspects of life. So if you're asking how do we do that, the fact of the Matter is the real world has these, this new technology. I mean there's no way and it's going to grow further. We all know that we are entering and we have entered the AI age and that's scary because all industrial revolutions, I think have been scary. I think this one, possibly it's hard to judge previous generations and epochs, but I think this, this is scarier possibly for two reasons. Firstly, it's incredibly fast moving. I mean the rate of progress and change is just unbelievable. Secondly, you could argue that it is hitting at quintessentially human skills. Things that have been considered quintessentially human, you know, thinking most pre the previous industrial revolutions are mostly between sort of what I would call brawn rather than brain. They've got to do with physically replacing humans with machines that can do it faster, you know, lift bigger things, manufacture things, go faster, et cetera. This time it feels like the thing that's so marked humans out from, from the rest of the animal kingdom. Everything else is kind of thinking or thinking in a, in a very sort of open ended way. And it now seems like kind of computers have even encroached on that. So this is scary. And I think that whenever things are scary, administrations and things tend to not quite know what to do with that and there will be faults along the way. So it's, you know, so, so I think in the end, I suppose I'm somebody who talks through the logic in some detail and I give lots of arguments for how to do that and some of the politics around it in the, in the third section of my book. And I think it'll be interesting to hear some of the comments perhaps and suggestions as to what, what the individual questions are that people face. Let me take a couple of other things that follow on from this. The there's a. Or this was slightly different question actually. Critics of AI and large language models say they are totally incapable of thinking and have no way to be creative. It's just plagiarizing. What is your take on creativity now and for future AI? I address this somewhat in my book about, so to speak, thinking, what it means to think. I rather sidestep because I find it less interesting talking about sort of philosophy of what it means to do that than the practicality of what I count as important to me. There are a set of things that AIs may or may not be able to do at this point and I think what's exciting is the set of things that humans still can do rather better than AIs. Which I think there is a style of creativity. I think we'll see that at the moment the current round of AIs at least doesn't seem to be the. Doesn't seem to be doing. But it's very hard to sort of, to sort of follow. I don't think it's simple as plagiarizing what the current AIs are doing. It's a bit more sophisticated than that. But I mean, you can say, if you really want to go to an extreme, you can say that to some extent human experience is a sort of reconfiguration and to a large extent plagiarising of what's been heard before in most people's cases. Some people really can change that. Many people can't. So but what I think we'll see is, we'll see a style of sort of AI in a sense, how AIs are thinking about things that humans may be able to dart around more. Well, I don't know. I think that in the, in the end. So it's a little bit like, it's like airplanes don't fly the same way as birds do. And here's what I mean. The machinery doesn't think the same way as the human thinks. Not exactly. And so what we'll see from that is different things that, that emerge as being good and bad about the machinery versus the human. And I think it's very hard to predict exactly what those will be. I think there will be something that you might count as creativity or might look like creativity to a traditional eyes that isn't the same as what a normal human creativity would look like. But humans can somehow do differently from that in many cases in the AIs. So I wouldn't draw very firm lines here, but I think humans will still have something to offer, if that makes sense. And let me answer a few more, a few more pieces here. What current teacher specialty do you think is most able to take on the new computational thinking teaching? Like when keyboard typing teachers had to pivot to computer teachers? Yeah. In the 90s. Well, I mean the obvious answer to that, which I'm not sure is 100% correct, is math teachers. Because I would think math has had a quotient of computational thinking. But I don't think that's the whole story here. I think that many people who have taught science, for example, STEM subjects, and certainly coding absolutely could, will be very good for teaching the sort of computational thinking I think should be mainstream in our curriculum. Here's the way I think about this A little bit, which is that, you know, if we're going to claim that there is a core computational thinking subject that people should learn for 10 plus years of their life, it better be applicable to other subjects other than itself. So for example, in countries that speak English, the first language, you typically spend a long time learning how to learning literacy, what's called literacy early on, often it's called English language or sometimes English literature and other things later on in your schooling. But nobody suggests that's just for English lessons. You use English in, you know, math and history and geography and all the other subjects you might do across the curriculum. Math is a bit different, right? Right now what we have is primary school math is mostly used, is used somewhat across the curriculum. Other math is above about late primary, early secondary, you hardly see it's very rarely used anywhere else. A bit is used in physics, it's not quite aligned. Biology, not very aligned. So that has to change. And in a sense what we want is a math that underlies the subjects. We want computational X. We want every subject to have a computational way of thinking of about it. Just like you have a literacy, you have reading and writing, you do in most subjects. But even if they're not English itself, you want computational literacy to be across all subjects. So if you think about teachers, one of the ways we want to teach this is by having problems that are solved in from different subject areas, but happen to deploy computation as part of that solution. Instead of just talking about them in English, we want to actually talk about them computationally, want to think about them computationally. So I think teachers from each of those subject areas, whether it's, you know, geography or history or English, I think each of those, particularly some of those teachers who perhaps are a little bit more technical in those areas than some of the others, may well be able to feel more comfortable teaching, let's say, a computational view of history. If they're originally history teachers with some help. One of the things we've been doing at computer based math is to build modules and teaching materials that really try to help with that. You know, it's funny thing, but if I, I've looked at some of the math, traditional math teaching materials that I could teach from if I were a teacher now I'm a mathematician, I've, you know, done math to a reasonable level, et cetera, et cetera, I would find it really tricky to understand what I was teaching from the teaching materials that on offer. You really need to know about the process of teaching and what is the standard discussion in, in the current traditional math subject to be able to teach that, not just about the subject, that seems like a shame for anything new. So one of things we've been pioneering is building the actual materials to help teachers as well as students who are not familiar with the latest computational thinking ideas. And that's something that I think is critical to making this transition. You know, when I hear jurisdictions say, well, you know, we should let teachers go and work out what to teach. You know, with computers in a new way, it's very difficult. I mean, we spent a decade or more, we are right at the center of this where people who have built computational Systems for over 30 years, actually 35 years coming up this June, we deal with math people in more ways than virtually any other organization on the planet. You know, we, we've interacted with more math people because of our technology. We, we use math, we hire math people, we educate people with math, et cetera, et cetera, et cetera. And it's really hard for us, we find it really hard to be able to just say, here's what you would do in a lesson to help somebody teach computational thinking. So the idea that any teacher, however good a teacher, can just jump into this with sort of not, not much help is, is in my view crazy. You know, the curriculum for traditional math has been built up over over a hundred years. So the idea that you can just step in and sort of make your own curriculum easily is crazy. So what we're trying to do, what we believe is the right thing, is literally to map out what you might do in every minute of every lesson. And it's absolutely up to the teacher whether they follow that or do something different. But that give that confidence as a background. And I think that's sort of the way to help more teachers from different specialties come into this. And interesting point here about when Google came, teachers were, the role of teacher was changing. And I agree with that. Teachers emerged. When Google emerged, teachers were encouraged to ask better questions. Now the work is encouraging teachers to in a sense, ask more questions. What part of the current mainstream curriculum do you think should continue? These are great questions. Yeah. I think one thing we have to do is give confidence to teachers to do much more open ended lessons where they're working alongside the student with the AI to discover its limits, to discover where you can take it, discover what we can do around it, including what questions we might think of asking, including how we challenge it when we don't think it's necessarily right, including what machinery to use when how do we get teachers help with that? I think firstly they need to experiment. I think we need to share experimenting. How do we get more confidence? I think some of the ways I've just talked about, which is, you know, building background materials that you can fall back on, I think that's a pretty helpful way. I think teachers actually trying to do things for themselves, if they were trying to do it themselves in real life is a, is a real help for this. And in terms of the mainstream curriculum on math, as I've as in my book and in many other things I've written over the last 12 or 13 years, my TED talk and so forth, I think math is a very special case at this exact moment. But let me come on to that where over a number of decades now it is completely clear that the subject matter is completely out of alignment with the real world. And that's for a very simple reason that whenever you calculate something in the real world today almost in every case you use a computer to do the calculating. And in almost every case in education you use your yourself a human. And those two are increasingly divergent. And that's been obvious for ages. It's just become more and more and more divergent. But there are really bad consequences of this. I mean I cannot over over emphasize the, the problems this is causing in society. So let me just spend a couple of minutes on that. You might think, well, you know, so what? So people are learning some how calculated, maybe it's not used so much at the moment, but you know, hey ho, they're learning some other skills from that. Not so bad. Okay, step problem one. You are learning the wrong process by which to use computation with typically the wrong algorithms. Why? Because. Well, I'll tell you an anecdote. When I was at school I had a teacher who was a friend of Alan Turing's. Alan Turing was the inventor of computers in often said to be the inventor of computers. I think widely claimed to be so during second World War and was a mathematician in the uk. One of the sayings my teacher had was math is the art maths because we're in the uk Maths is the art of avoiding calculation. That's very interesting. Right? He was right. The whole. One of the key issues for mathematics at the time was calculating was the step in the four step process I showed earlier that was most expensive and most stopped you using maths in real life if you couldn't calculate something, you mostly couldn't use math. So therefore and calculating was hard, difficult. So you spent a lot of Your time very deliberately abstracting to make the calculating easy because you, the human couldn't do too much of it. That has fundamentally been turned on its head since computers. We built machines that are possibly the most successful ever machines in human history to automate something that was in a sense, or mechanize, I should say, something that was thought to be very human. And we do it billions and billions of times better than humans do. So math is no longer the art of avoiding calculation. That means you can deploy many new forms of algorithm that use mass calculations to do things. And yahoo. Yaho Machine learning, great example. Machine learning. The idea of that has been around for some time. The reality of that actually using producing useful results has been relatively recent because of very high powered computers. There's no point in learning machine learning by hand because you can't do it in practice. So that's an example where in the traditional math curriculum you learn, you don't learn machine learning, it's not there, it's nowhere there. Right. Because it isn't something amenable to hand calculating. And it possibly also too recent. But actually when you look at the whole math curriculum, every piece of it, you find things on their heads. I wrote a fairly aggressive piece during the pandemic. Find them. Conradwolfram.com about the British government's attempt to work out how to assign grades to people who hadn't taken exams because they couldn't take the exams because of COVID What an awful job they did. And one of the reasons they did an awful job was that they threw out most of the data they could have used before they started figuring out how to assign the grades. Why did they do that? Well, basically because they'd all been schooled in how you got to get rid of most of the data to make the calculating simple. That's what you do because you need to avoid calculation. Except that's 100% wrong. That's not what we're worrying about. We have big computers, they do lots of calculating. You don't need to throw out most of the data. What you need to do is in the abstract step you need to think of different ways you might use as much of the data as rich a picture as possible to get a better result. So the complete wrong thinking is being often taught in our mainstream math education. So it is damaging. We're not talking about. It's just like a problem on the edge. The other problem that I think is absolutely critical from this we are depressing many, many students unnecessarily. You know There was a time certainly in uk, when, for example, you, if you didn't know how to recite a Latin verb in its perfect subjunctive, you didn't do well in your exams. Right. Seems crazy. And maybe there was some correlation between how well you were good, you know, how diligent you were at learning that and how diligent you were at doing some other subject. But in the end it's not very correlated. And some of what we're doing with math at the moment is a bit like that. The fact that you can or can't do a quadratic equation quickly by hand doesn't correlate that well with whether you're a good computational thinker in the modern world. And we've therefore telling a lot of students you failed because you're not, you don't happen to be good at this hand calculated skill, when that skill isn't the predominant skill we need is a disaster in terms of having everyone perform as well as they can and be as satisfied with their skills that can be magnified as we have. So it's very, very important that we don't confuse that subject with it. So math is a particular subject which I think is completely out of align has been obvious. How are other subjects going to change? Well, I think one of the things we need to learn is how we work to optimize results in a hybrid world of AIs and humans. So it may be that the history essay survives, but that your first draft of a history essay might get written by the AI, and then you learn how to make that interesting beyond what everybody else's AI writes and how to hone that to you rather than what the AI says, that might be an example of how things are going to change. I think that we need to learn to deploy many new sort of tool sets on what we're doing and not just, you know, I think the idea that you need to memorize sort of, you know, in a sense, facts which are immediately off online is not quite right. But people say this, so do you need knowledge or do you need skills? We need both. The question is knowledge of what you still need knowledge of what's out there, how to operate it, what goes wrong, how you assess it, how to be skeptical in the right place. Those are pieces of experience and knowledge you need, which needs some facts. So I can't predict exactly how subjects will change, but I think the way to look at them is with outcomes. And one of the things we've done, which I was interested when sort of chatgpt in a sense came along was to look at the outcomes list that we at Computer Based Math have built for computational thinking. And let me see if I can share it here for a second. The this is an outcomes list that we built for core computational thinking. And what I find fascinating about this list, it's in 11 dimensions. Many of the headings here and some of these specific details are actually very to do with, in a sense, these new skills that I think we need around a sort of chatgpt AI age world independent of computational thinking directly. So confidence to tackle new problems that seems a major outcome we need to achieve. You know, instinctive feel in this case for computational thinking, but it might be for history, it might be for geography, it might be for any other slicing of subjects. How you critique and verify this is absolutely critical. Do you understand, you know, when somebody presents you something, do you understand what might have gone wrong? Do you do, do you have your set of things to ask about it and so forth. So I was excited to see how related many of these outcomes are with what we need across the whole curriculum, not just in mathematics. And that's the sort of sense in which in my book, the byline here says, you know, an education blueprint for the AI age. It wasn't just about math and computation, it's really about these more general subjects. The Let me, let me move a bit further here. The Will students be used to train AI or perhaps used to build AGI? The I'm sure at some point, I mean, there are very many ways things you can teach ChatGPT and things as it is by feeding things into, you know, in terms of sort of giving it hints and things. So there's that sort of teaching. There are deeper ways to learn how to train AI. I think the idea of training AI is a bit like saying training a human. There are many different levels of that, from sort of very deep down to individual little things that you can show. And so absolutely, I think that will be part of it. And I think that's a very beneficial educational process as well. Very interesting question here. Will it still be worthwhile to learn how to code? What is the future of coding if AIs can do it better and much faster? Fascinating question. So where to start on this one? I think that the idea that the only way to sort of instruct a computer in a very new area is to write base level code, syntactic code, I think that is over. And I think quite a lot of what's being done, for example, in computer science Courses in university or some of the people who are going into computer science courses are doing it because they want to use computers, let's say, to do data science and to use modern computation, things like that. I think a lot of that will at least be fronted by increasingly sophisticated AI. So instead of granularly writing, you know, code, you will write in a query, so to speak, you know, you will start to ask a question, the AI will write the code. And of course, we've seen the beginning of this with ChatGPT, where it can write code in many languages quite well to a certain scale. Now, having said, which, I do think there is a precision of syntactic coding that will persist and one will need some connection with. I don't think it's true. You know, it's funny, when we had what was called at the time, wysiwyg, I think, which is, you know, windowing systems on computers, people said that's the end of, so to speak, typing things with a keyboard. Most of the time it's like that's the one interface we're going to have. Well, hasn't proved to be the case. Most interfaces remain over time. I mean, paper tape has gone. But actually the idea of typing into a computer to get to code it, to tell it what to do is still with us, even though there are other ways to do that. And in some cases, the other ways impinge that. Now, ChatGPT and the like are majorly impinging on this, right? I mean, there's a lot more you can do. By the way, our WolframAlpha was doing this some years ago. Just, just to point this out, you could type utterances in WolframAlpha and you could get Wolfram language code out from those utterances. And. And indeed, what's really exciting now is to see the mixture ChatGPT, turning it into a WolframAlpha type thing with a Wolfram language output. This is a fantastic set of things that one can do. But I do think that actually being able to deal with syntactic language, computer language will remain important and particularly for some people, but it will become a more specialist skill. Just like knowing, you know, there are many, many more people who nowadays deploy mathematics or computation than make new mathematics and computation. That's the nature of automating and mechanization. Now, one very important part of this is what language do you get LLMs to write in? And I have to say, Wolfram language is really fantastic for this. So here's why. So you utter something into your LLM and it writes you an initial go at code. The great news in Wolfram language is the code will typically be much shorter, much higher level, much more systematic because we have a huge breadth built in. It will cover standardized data as well as standardized functional code, etc, all built in the chance that you as a human can then go and learn how to edit that to exactly what you want with the precision of syntactic code and to do that quickly and effectively is much higher than with some of the other traditional languages. So I think that allows Wolfram language to be really beneficial and you know, very much in educational spaces. One of the problems we've got at the moment with coding in its traditional way is if you take it at school level and at university level, it's slightly got the problem that maths has in the sense that it's in the eye of the beholder a bit. People who are already interested in coding in its own right get excited by it and that's wonderful. But there are many cohorts of people who could really benefit from code coding and somehow it doesn't really come very naturally to them. And they kind of, you know, learn it when they absolutely need it, but they are less, less using it than they might. I think that LLMs and particularly with high level language like Wolfram language for example, really shifts that dramatically. So to come back to the question, I think that just like in the past writing assembly language code was important for many people because of the speed of computers. Now virtually nobody writes assembly language code. It's not worth it. I think that we've gone to higher level languages. I think that, so again there's a big cohort of people, including new people who will come in and be able to code. I think the number of people who need to go off and traditionally do computer science in the way that it's manifested right now will shrink. But I think what we need to replace that with is modern LLM enabled coding, so to speak. And you know that that includes many more people and will get us much further and particularly with a sort of very high level language to that. And so I think that's, that's an important sort of direction that, that we need to head in, in moving things forward. Let me, let me take a couple more things here. If we fast forward to the year 2043 and when all 18 year old school leavers have been through a new curriculum in line with the maths fix, that would be wonderful. What benefits will there be to society, the environment, personal well being? Okay, let me, let Me, let me try and take that for a moment. Important question. I think the best reference to this is if you look back in the early 19th century, I think it's fair to say that's when the movement for mass literacy started to get going. And there were many detractors at the time. Many people said, you know, most people in society were a bit too dumb to learn how to read and write. It was for the few. What we basically had at the time roughly was high priests and aristocrats and a few other educated people telling the rest of society what to think. And the rest of society had a hard job being able to sort of check up on that in their own right. And I think that was very dangerous. I think that was a way in which you get bifurcated societies where you have, you know, a big divide between the haves and have nots. I think you tend to have a lot of problems because experts emerge who tell you things. And it's hard for most people if they're not trained somehow in how to understand what's going on there, to assess those experts. And some of the experts are very good and well intentioned, some of them are very bad and badly intentioned and everything in between. But the point is, what you want is those abilities to be spread to a much wider base of the population. So that happened in literacy from, I think the early 19th century onwards. And the benefits I think it's probably been, if you pick out things in education, concepts in education. The idea of mass literacy, I think, has been one of the most empowering ever. And it's now recognized, whether it's in the UN or wherever else, as a right across the world, in almost everywhere. So it is a major change and I think something that very few people would argue hadn't been anything but entirely beneficial. I think we are now in a need for mass computational literacy, which is the ability to work a level up, use the power of computation with the machinery we've built for ourselves to make better decisions about our life and our societies. And this is true. So I think the benefits are several fold. The benefits are to do with individual well being. And I almost think, you know, there are several levels to think about this. There's survival, I mean, to be blunt, you are going to survive longer. Now if you understand more about how to think about data, particularly with respect to, say, health, you are likely to survive longer than if you don't, because a lot of the decisions relate to data, or at least your analysis on that. So it really makes a Difference even to survival. And then I would step up a level to what I would call subsistence. Can you not only just survive, but can you sort of prosper at a reasonable level in society? And what do you need to be able to do that, you know, to be a reasonable level of. Get a reasonable level of job, to make reasonable decisions about things, you have to make decisions about money, risks of other sorts, et cetera. And then there's what I call value add. Value add. You know, the people who are the specialists, the top people in societies, in our society, really drive things forward in a, in a beneficial way using this modern, modern approaches. And so those are the three levels I think one sort of wants to look at. And I think we can improve, therefore all of those levels. I think that we can have a society potentially that, that is able to question the basis on which decisions are made better. I mean, at the moment we've got a situation in some, you know, some forms where we have quotes, the experts standing up. Some of them are very good, as I said, some of them are not very good. Society feels, many people in society feel totally disenfranchised to query what the decision basis is for, for a lot of those decisions. So they start rejecting all experts in all cases because it's kind of like, well, some of them talk nonsense and that's correct. And so therefore I don't really know how to evaluate the other ones. So therefore we're sort of going to throw them all in the same bucket. And that's also disastrous to a sort of almost pre Enlightenment era when we didn't trust reason of any sort. So that's very negative too. But I understand how that has arisen as well, and I think that's part down to it. So if everybody studied the math fix and the approach that I like to see there, I think we would see individuals who are more empowered. I think just like for example, the UK benefited a lot in the industrial revolution in the 19th century, from early, being early on, mass literacy. So I think that the first countries and jurisdictions and organizations who really deploy computational literacy effectively will, will benefit fastest and most and you know, just economically. So I think that's a major benefit. I also think that societally we need a certain binding between people who have a certain base of knowledge that we can discuss without going back to the, in a sense, assuming no knowledge. So that is where I sometimes believe that some core subjects across the whole population are important. And I think computational literacy is one of those subjects. So I think a more empowered society A better enfranchised society and less dislocation from in a sense the computational haves and computational have nots would be all benefits of deploying the maths fix and the yes, let's, let's take a few more questions at this time. Is the current hype presently around chatgpt and things helpful or harmful? Will the fearful rhetoric AI take over sci fi of some of some lead to regulation and government controls that will destroy open access to these tools? I don't know is the answer. I think that we will need certain kinds of regulation and we don't know what yet and there will be some fantastic consequences and there will be some fairly horrific consequences all wrapped up together. I think hype is usually not terribly helpful because I think what happens is people make knee jerk decisions that aren't necessarily very good. Except I think the hype as I mentioned earlier is somewhat helpful because I mean just to give you an example, I many years ago met a defense a person who was an education minister in a country who had previously been the defense minister in that country and they said that they were attacked by another country, cyber attacked and they said in some ways it was the best thing that had happened to them because it forced them to reorganize and rethink how their defense worked. It was a sudden shock to the system. So I think in this sense the ChatGPT, you know excitement of this year may be helpful if it's treated the right way. It is a shock to the educational system and we should use that shock to rethink what we're doing. And some of this, much of this conversation has been about that and I think other conversations. So such a shock and therefore the hype that led to that shock partly I think is therefore useful. So but I think we've got to be very careful then to sort of calm down and to look at the things but in a positive light of these are coming there is no choice. We can't uninvent the thing that's been invented. And yes, we need to have systems of thinking about whenever we've had very powerful technologies in history. You know, it's very scary. I mean, you know, obvious case is nuclear, nuclear, you know, understanding nuclear physics. It has some very very positive benefits. It has some very, very negative potential benefits. We are still dealing with those benefits at this time. I think that this is such a powerful thing but it will take time and it will sort of oscillate between regulation that's overbearing I think and is wrong down to things that weren't spotted. And we need to like all forms and all forms of badness in humankind, whether it's crimes or other ways these things happen. When there's new technology and you need to fight against it, I'm fairly confident that in the end will do a good job, but there'll be some pain in between. A question here about. Let's see, there was something potentially to point to. Yeah. Just to point out here in discussions of all of this, that of course, one big improvement potentially, just to point out that this is what's coming down the track, is the wolfram plugin for ChatGPT. It's going to be very, very important for people not just to trust ChatGPT as it is, or other LLMs, but to have facts injected to see what the best tools are. And I think that will help to calm down and to sort of orientate people a bit more as to what's really possible here and what should be avoided. So that's all part of this picture of learning sort of how to. How to move forward, you know, as we go, as we go into the future here. Another question here. History is written by the winners. It seems like we will give AI much more power over us and we will believe it is true. How do we filter fact from fiction? That's a nice way to, you know. Yeah. As Churchill put it, history will be kind to me, for I intend to write it. So I guess that's the. That's the AI equivalent here. That's. That's being suggested. You know, we can't foretell the future on this. I think that the best way to get there with humans on top of is as much experience as possible of what we have in the current technology and therefore what its bounds are and where we can go with it. And I think that will allow us to understand some of the issues with the history it might write and how we sort of avoid that history being detrimental. The IT as we know history has different sort of colors to it over time, and we look back and see things that didn't seem accurate and now seem more accurate, et cetera, et cetera. It has different scopes of timing as well. There's very recent history, there are longer time periods and things. I think we will see the current AI age revolution as, you know, another industrial revolution where in the end, you know, I mean, there were all these predictions before. For example, humans would become idle because machinery would just take over everything that didn't happen before. In fact, humans are probably more in Employment than they've ever been. Machinery hasn't done that. What's happened is we've moved up to the next level now. Can we do that again? I hope so. I think that's where the education comes in and that's why it's so critical that we now is the time when we've got to have what I call computational literacy, AI literacy introduced sort of from beginning to end. And to that point actually there's a question here. At what age or stage do I think the math fix should start in education? So look, the idea of the maths fix is that in a sense you should use the best machinery for the job. If in deploying the computational process, usually for calculating, that's a computer. And as we've discussed, that may also impinge back on abstraction for, you know, computers doing AI and so forth, for the abstraction stage as well as the straight computation stage. Where does that hit? Well, it may hit at any time in education from a very young age, but I think where it's gone wrong so badly today is in what I would call late primary education through right the way through to the end of school and beyond. So if you were to pick a time when I would really focus on the change, it would be late primary. And what I mean by late primary depends on jurisdiction a bit. But by the time you are solving quadratic equations and doing trigonometric identities and things like that, things are pretty abstracted away from what you do in any life, real life for most people. And that's where most people, many, many of our populations get lost and disenfranchised. So I think that is a very critical time at which to do this. But I mean, you should be asking the questions earlier. Why isn't machine learning being done in primary schools now? I don't mean building neural networks, that's a much harder proposition. But I mean just using it, getting experience with. When machine learning can help me solve a problem, I think that should be as early as practical. I've said many times, why don't we teach calculus to 10 year olds? Well, it's because we wait to do calculus until their algebraic skills are very good, because it's hard doing the calculations. But actually the basic concept of calculus, the idea that, you know, if I make something, you know, if I, if I make a rectangle and I make it small, smaller, smaller, and I add up tons of them under a curve, I can approximate the area under that curve, you know, I can make, make smaller, smaller, until it's really infinitesimally. Small. That idea is amenable to much younger children and is exciting and interesting and a good way to think about things. And why aren't we teaching it earlier? We're teaching it late because we have deployed the curriculum by, in, in the order of computational complexity, not conceptual complexity. So because the calculating is hard, it's late, not because it's hard to understand what you're trying to do. That's the error. We can reverse that. And so that in a sense goes through the whole of the time for math education. But I think, as I say, it's really critical as you go to late primary. I mean, just to say this doesn't stop at college, just to embarrass her. My daughter is a student of biology at Cambridge University in the uk, a first year student and I happen to oversee her. I happened to see she was deciding to compute some eigenvectors, she was doing it with a calculator for some problem she was supposed to be solving. And I asked her what she was doing and why. And eigenvectors, for those of you don't know, are something you learn certainly in the UK you learn in sort of further maths A level and they're pretty useful for certain kinds of things. But the question I asked her is, you know, when would you use these and why on earth are you using a calculator and computing them by hand? And it's a very good question, right, because she didn't really know why she would use them for her biology. In fact they used quite a lot, as I understand it, in the internals of things. Like some of these AI systems, they're quite important. They're also used in physics for things like sense of gravity and how you work with those things. But you'd never really compute them by hand. You compute them on a computer, but the crucial thing is when you'd use them, how you'd use them, what you do with them. Why on earth was she doing this? Good question. So this percolates through our education system and is a higher level example of where we've kind of gone backwards in a sense. Another question here. If the majority of learning is done using a computer, how do you ensure learners responses submissions are their own. Will there. And I'll follow up. So will there need to be AI schools to train AIs how to teach humans? Yeah, that's a nice, a nice twist. I mean the question of what responses are your own is a very complicated question with all machinery. I mean if you drive a car today along A road. Is your driving your own or is it your cars? Well, it's complicated. Your car does all sorts of complicated corrections that a car from 50 years ago wouldn't have done, hopefully to improve your driving. It's not straightforward. When you're using machine and particularly the more automated, the machinery is what you count as being your own and so to speak, the machineries. The way I think about this is in the end you want the optimal sort of result and you want that human to be being the optimal manager of that process, including themselves and the machinery they have to deploy. Take my example about driving. If you drive very safely and effectively and you deploy all the most modern technology in your car to improve that, that's okay. In fact, it's a lot more sensible than saying you're going to lock it all out because somehow you think it's more August not to do that. I think the question of whether it's your own work, as in, are you actually. And I think this really goes down to assessment. When you're assessing a student, what are we doing there? And can we tell whether it's that student and not their computer that was good or some other student? I think that really the, the issue there is the forms of assessment that we need to change. And I have to say that assessment has gone traditional form, you know, traditional formal assessment I think is increasing trouble. I think that's where we should be looking on this. And you know, that's also true sort of within a school environment. Now if a student, you know, one way you assess this is by actually talking to the student. I mean, it's an interesting how full circle this has come. A little story here that my mother was a philosophy professor at Oxford in I guess 70s, 1970s, and she was responsible for admitting students for reading philosophy in one of the colleges in Oxford. And at the time there was an Oxford entrance exam in which the student had to take the exam, typically write an essay on simple philosophical question, and they would get a grade just like they would today. But there was a difference against many admissions procedures today, which is that my mother would get a copy of the exam paper and she would then interview the student on what they'd written. So there were some students who got better grades and it wasn't actually cheating because they were in an exam room. But let's say they'd been very well coached for the exam and some students who perhaps hadn't been as well coached, maybe the school wasn't as good, but actually were pretty sharp and were very good. That became much more obvious when you talk to them about what they'd written. So I think in the end, what do we want out of the human? I mean, we need to start from what do we want in the real world for the human's own enrichment of life. To come back to my definition of what education's for. What do we want? One of the discourses we want is that humans can communicate well with other humans and have real ideas and. But let's actually be talking to humans as humans and figuring that out alongside what they might have produced to assess them. I think what's gone wrong with assessments? One of the terrible things that's gone wrong with assessments is that it's a sort of funny as the world has got more. I mean this is one of the weird things in the world that quantification of which I'm part making numbers. Numbers and particularly simplistic numbers have assumed an importance beyond their ability to judge. And we see this in many parts of society. A simple number has huge marketing panache. So that if I can produce a number for something, it somehow does much better than me just talking about it. And people rush to the number. And nowhere is that more true than in assessments. You know, I got an A or I got this grade or I got this, and that's very important to people. And so what's happened that then is people say the number's everything, so let's make the number work. And they've then made it so that if everybody took the same student, you know, essay or whatever it is and they marked it, they would all agree about what the number was that came out. Okay, so what goes wrong with that? What goes wrong is that you then make the questions more and more closed ended because you have to have all the markers agree on what the mark is because. Because otherwise everybody gets unhappy because they say, well, it's unfair. But there's another sort of unfair, which is that the question, the exam, the assessment didn't match what was needed in the real world. And what's needed in the real world today is open endedness. We get difficult open end questions. Everything that's very closed ended and procedural. Computers do very well and increasingly well. And now we're finding computers do some of the open ended things better as well. So what we don't need is to examine people on how they pretend to be a computer. So I don't have a full answer, but I think one of the crucial things here is we need to understand what we're assessing how and stop Believing that we can, in a sense, have a very simplistic number and formulaic assessment that will expose what the student's done. And I think the more we do now, funny enough, I think modern AI ChatGPT LLMs can help with this because in the past they can help assessment and structuring of what the student's written and actually help the student to learn and help potentially teachers to be able to see what's been done. And we at Wolfram are potentially working to help that quite a bit, and some of our ed tech is really in that direction. So I think that this will be a. I think a sort of chicken and egg race here, which is at the moment seems very much like it's with, oh, my God, everybody's cheating, we don't know what students are going to learn, panic. I think there'll be some great advantages where we end up with much more sensible assessments to help us figure out how students are doing as well as just sort of decide whether they've reached some milestone or not. But just to come back for a second to this point, that by the time you're at a very advanced level of automation, you get very big questions over who it is that did it. And it's a good example in race car driving. You look at Formula one, that's a huge issue. To what extent is the car and not the driver determining who wins the race? And the way that's got handled over years. Right, is that there are rules for the race which try to allow technology to develop because it's beneficial not only for Formula One, but for normal people's cars. A lot of technology has been handed down from what you need in racing, but. But then set a rule set that excludes some of that, so that then you have to sort of make the playing field more lab, more level between what's available for the technology. So. So there are interesting thing parallels there, I think, with what's happened. What's happened before. There was another question here. Yeah. I mean, the. The question about AI schools to train AI is how to teach humans. Well, I think that's an interesting possibility. I hadn't thought about that before, but there's certainly a space to be developed there, I think, and it'll be interesting. I suspect it won't. There will be more edges to that than you might think, but perhaps. So a great question. If such a proportion of society agree that the maths education is broken, why isn't it changing? Has anyone begun the change? So I do talk very extensively about this in my book, We've got a funny, funny mixture of what's happened here. At one level, many people in many countries think there's a problem, but they don't necessarily, in a sense, they can't crystallize the problem. And during the Math fix work, I've been amazed the extent to which that's the case. So you'd expect sort of an average member of the public, you know, maybe didn't like maths at school, doesn't really know what they're doing with it, finds it a little bit confusing, hasn't crystallized what the issue is. I would expect that it's a complicated subject, took me many years to do it, and I'm right in the middle of it. What I find a bit more tragic is that people who are in many people who are in the field of education or maths or teaching maths or government policy around education or the like don't seem to have understood this at all. It seems to be news to them that, for example, computers are used for calculating in the real world and we ain't doing it in education. So that I find much, much more shocking. So I think we've been doing a number of things. So one of the problems, I think so what I mean, there are many answers. Question one answer is that the traditionalists who want to keep things the same are more united than reformers because the reformers in this haven't got their. Haven't got the story straight, if I can put it that way, as to exactly what the nub of the reform needed actually is. And one of the things I you'll find at the, at the, in the Math fix is I started a campaign to try and if you like, unify reformers at least, you know, because I mean, this is a tricky thing. You know, most people read the Math Fix, I expect if they're good skeptics won't agree with everything I've written in the Math Fix, they'll be skeptical of it. On the other hand, you know, we have to have something definite to say. And so we started this, the Math Fix campaign for core computational curriculum change, hopefully to lay out these five points that we thought were many people who were interested in this could agree about, but which also said something definite that is obviously different to what we're doing at the moment. And I was keen to get people to rally around these five points so that we can then take that to policymakers and others to try and get a change really to happen. And you'll see here there's please do lend your support to this if you agree with them. So question answer one as to why this hasn't changed is because people don't haven't got crystallized what the issue is. And so if you're a policymaker, you know, you have a thousand people telling you different things and you become confused as to what to do. And the safest thing is to do nothing. Apparently I say safest, safest for you. So that's another problem we've got. We've got policymakers who have the wrong risk profile. So what's beneficial? It appears to be less risky in a sense to make few changes because the risk of making a change if you're an education minister appears higher than the risk of not making a change to the subject matter. Because, you know, lots of things can go wrong. You can have people protesting, you can have people saying, I don't like it, you can have mistakes that are made in the new assessments, etc. Etc. Etc. Unfortunately, over time this means a much bigger risk, which is we disenfranchise a huge fraction of our society, which is what I believe has happened. So we need to change the ecosystem of education so that we are incentivizing intelligent change. We're allowing that to appear less risky. We're setting the risk profile for the people who need to make the change so that they get benefit from it rather than just seeing it as a huge negative, when in fact for society it isn't a negative. So I think the, in a sense, one of the main problems is, and one of the locking linchpins of this is assessment. We have these traditional assessments and they lock subject matter. And in fact it's even worse than that because we now have international bodies like PISA who assign what it means to do maths well, for example. And so even if you're a country, if you care about pisa, as many countries, smaller countries certainly do around the world, how they're competing on PISA compared to other countries, it actually kind of locks you together with the subjects other countries are doing rather than allow you to try new things because then you'll score worse on PISA potentially. And actually in my book, I compare this to the fiasco many years ago of the Lloyd's insurance market in London, where people insured in round in a circle. There were lots of reasons for this, including, including some bad behavior, but some of it was incompetence in the sense that people ensured round in a loop and they believed that was mitigating their risk. And I think the same things Happen somewhat in education. We've ensured ourselves with, you know, comparing our mathematical subject across the world between different countries. But in ensuring ourselves, we've ensured ourselves around a loop potentially to the wrong subject. And so those kinds of things all put us together in not making change. And I think that's a key reason why it's been incredibly slow to make change happen. And there's much more about this in section three of my book. Is change beginning? Well, look, I think there are some promising signs. Firstly, the discourse around maths has very much changed in many countries, including the US and the uk. I think if you look back a decade ago, people's discussion over using technology in math, understanding that math calculating wasn't the same as math, that has very significantly changed. Has the correct action been taken in formal education around this? No, in most cases not. We've got a few places where this has happened. We have various countries and jurisdictions around the world. There's been a lot of interest. One of the problems is that the funding model in many countries is wrong. Governments fund the development of quotes curricula. They also separately pay for, either directly or indirectly, materials for teachers and students to use, either in the private sector because schools are buying them through the public sector or directly. They also separately and again in different ways around the country have curriculum, I'm sorry, assessment authorities that decide what to assess and they spend a lot of money in the process. I mean, this is not a problem of necessarily spending more money, it's a problem of putting those things together so that we can change the subject, not just how we teach it, but the subject. And that is what we critically need now. And a lot of what I talk about, which I think relates to what's changed with ChatGPT, is how you change a subject. And a lot of in the maths fix, although it's about maths changing maths, the subject of maths, it's very much also about how you in general think about formulating a new curriculum, how you allow teachers who weren't schooled in that subject to teach that subject, how we build materials for that, how we make the change. So there is progress, I mean, even in the last few weeks in the uk and we'll see where this goes. The Prime Minister, Rishi Sunak, as one of his main policies, wants to make maths more for 16 to 18 year olds. But he is interesting, I understand, in investigating alternative maths, not just traditional maths. So we'll see how that pans out and whether it goes down a good route. Or not. But I think the fact that that's center stage, the fact that we've got reforms in other areas, and I think that ChatGPT is also driving us forward. So we've had very many positive conversations. But please, one of the things I would ask two things of people listening to this. If you know who policymakers, people who can are empowered to make change. If you know people who can help with that in your jurisdiction, your area, please, if you agree with this, put pressure on them to understand the vision of this. By all means, put them in touch with us. And let's try and get this moving because it is critical for our societies. The second thing I would say much more directly, we have adults who have gone through the current education system. Let's not throw them, throw them under the bus here. Adults can absolutely learn the sort of computational literacy, computational thinking I'm talking about in organizations. And of course in organizations, governments aren't telling you what to do, and there's not a big edifice here to change. And we have been working hard on trying to understand how what we've been thinking about can be deployed for adults. And we are very interested and keen to connect with those who want to build such a capability in their organization or more generally across many organizations. And that, by the way, applies not only to commercial companies, but also to universities and also within government. How decision making One of the problems of computational literacy, as I pointed out in my my blog post about the UK exam fiasco, one of the problems is lack of computational literacy in our very own governments. I mean, as I put in that blog post that I mentioned earlier that's on conradwolfram.com, the Department for Education in Britain were hoisted by their own petard of lack of computational thinking, the exact thing they needed there that they've been promoting as the education for everyone showed itself lacking in how they made decisions. So we need this at all levels and I think the fastest place to deploy this is with adults and we will hopefully be part of working on that. And the. Let me maybe round out. We're getting to the end of our time here, so in the last few minutes, let me round out, perhaps. Pull a few of the questions together I see at the end here and round out with some things to say. I think what I'm describing in computational thinking is a different subject to what we've seen before. And in the end the question we need to ask is what is the core subject, the core computational subject that needs to prepare us for a hybrid AI human age. And I believe that's very connected with this computational literacy that I've been talking about and there are other pieces of experience we need to learn. But I believe when you look at what we've been talking about, mass fixing is actually very connected with this new world we find ourselves in and is very disconnected with the traditional sort of world we've got. So it's different from coding. Coding is one piece of that. And as we discussed earlier, coding in its traditional sense may. May rather change in character as we we get coding assistance in a sense llama systems to coding. The. I think to round out here, you know what I ask people to do. I think it's that a lot of this conversation has been about the future of education, particularly around computational thinking and chatgpt and what that changes. I think maybe we can do a follow on to the session where people have had time to absorb some of the of what we've talked about in this session and you know, please do, you know, post things about what you'd like to hear further and which things you've enjoyed and found less useful in this session. But what I would say is think about try and see where decisions look out for where decisions need an element of data science modeling, computation, where that's cropping up and you'll see it all over the place. Places where you didn't imagine, places where there isn't explicit calculation involved. Many passion in my book relate to this and maybe that's helpful. If you have a chance please do look at Both computer based maths.org and computationalthinking.org on linked from those sites and on Wolf from you. We have interesting courses and particularly ones on computational think. We've got four of our sort of trial modules from computer based Maths and those may give you insert they're very much beta test modules at this time. We don't necessarily have all the formulation right but they show one view into how you might get started with a particular level of computational thinking. So those are some of the ways I would sort of get started at thinking about this for, for those of you who are interested in that. But it will be lovely to build this as a, a bigger movement and I think the time is ripe. You know, I really do view ChatGPT's announcement and excitement about that and the earlier question about is it hype? Is that useful? I do think that's useful. I do think we can use that now to drive a very important discussion about the purpose of education and how it needs to change. And I hope we, you can, we can drive some of that. I hope you can drive some of that. I hope you can help us, you know, to drive that forward. So please do engage with us. And, you know, yeah, if it's helpful, do a follow up on this. I would love to do that. Just a reminder that I relaunched my book, the Maths Fix, and with a new forward, at least for the Kindle version. But you can also read it on my conradwolfen.com blog. So I do recommend that. And that has a lot of the themes that we were talking about in this and otherwise. Look, I much appreciate you coming to this session and thanks very much indeed. Thank you.