How Fast AI Learned Math | Phillip Kerger, Professor @ UC Berkeley, ex-NASA

In December of last year, AI could not help me on my research at all. And then in March, it started being helpful. And then in July, it solved a problem that I'd been thinking about for 2 years. And then now it is solving Millennium Prize problems, going from zero to solving problems that people have thought about for 60 years.

So the existential crisis is applicable to probably not just the Mathematicians, probably to everybody. What do you think is left for you as a scientist and mathematician?

A lot of us just do applied math, and we're all in an existential crisis because of AI doing our work much better than us. Then there's the nuclear engineers over there.

They have to build some kind of device.

What about quantum? Like, how's quantum gonna affect all of these?

My hot take would be that quantum AI is pretty much dead, at least dead As a practical proposition, you're in a better place to be working in a space where in the end there's something physical about it, and that is protecting you a little bit. It's protecting your usefulness.

Everything that's in person becomes more valuable because there's a bigger constraint on these things. Wow, we should all go after those 60-year-old, 100-year-old problems to solve.

In Principle of Freshmen, Could, or anyone could solve an important problem in some field.

Today I have Philip Kuehger. He's a professor at UC Berkeley Industrial Engineering Department, and he's formerly a NASA quantum researcher. Today we're going to hang out and talk about what he's teaching and maybe what his students are actually doing right now and how's AI changing the classroom in Berkeley. How's that sound?

Thank you. Thanks so much for the intro. Yeah, so there's a lot going on. In how AI is affecting education, how it's affecting the classroom, how it's affecting math and the type of research, even especially the type of research that I've been doing earlier this week, I'm sure.

What research are you doing?

So my research has mostly been in optimization complexity theory. So that's the question of how much time or how much computation, how many computational steps Do we need to solve certain optimization problems? And part of maybe an intro about me also was if anyone saw maybe a month and a half ago or so, there was a very viral Reddit and Reddit post also made the rounds on Twitter that Sebastian Bubeck from OpenAI retweeted also, which was a result of a 30-year-long gap in optimization complexity. being solved by GPT-5.6 SOL. And I was the author of that post and author of that work, but author of that work comes definitely with floating quotation marks because I was not the one who did that work. That was really GPT doing the heavy lifting work.

Sorry, I interrupted you earlier. You were saying earlier this week?

Oh yeah, in terms of the impacts Of AI on math research. Earlier this week, there was the result on the solution to the Navier-Stokes Millennium Problem, which to put into context of someone coming directly from someone— I don't work in fluid dynamics, computational fluid dynamics stuff, but I'm close enough to the space and know enough people who work in there that when I saw that on Tuesday morning, I was properly shook a little bit. It was a semi— the math researcher in me had an existential moment of this was extremely unexpected. And I saw even the days before there were rumors on Twitter, people talking about, oh, whispers of something being solved. And I thought it was going to be something maybe a month out or a couple months out or something, and not the next day they announced the solution. And then there's all the controversy around it. Uh, did they scoop results from other people?

Data leakage.

Yeah, exactly. All those things. So I think the way they handled that, it's amazing that OpenAI—

Can I— what is the millennial problem for people who are not familiar with it?

One of the ways to think about it, especially in context of the solution That OpenAI gave, I should also say I'm not an expert in this particular field, but I think I can still explain what is that. The Navier-Stokes equations are equations that are used for almost all modeling of incompressible fluids. So that's mostly water. Anything that's like water is mostly modeled in 3D using Navier-Stokes equations. And those equations assume an actual continuous fluid. So that's not a perfect model of reality because once you go to a small enough scale, you have, you end up with water molecules, of course. So you can't, so it's not actually a perfectly continuous fluid, but the model works extremely well because at any scales of water in my water bottle here, there's so many molecules in there that the Navier-Stokes model works pretty much perfectly. And one of the questions or What kind of the millennium problem boils down to in what OpenAI showed. Are there any configurations or is there any kind of system of flow and water movement under a finite amount of external force that can create in the model some kind of something like infinite velocity or infinite force? And we call those things singularities as imagine some vortex where the center of it Is distorted in a way that the model says the water there is moving at infinite speed. So the resolution to the question was that in the model, there is a solution that in finite time achieves infinite velocity of water through this. Maybe you've seen pictures of this weird vortex and there's animations of this vortex too, which is like a construction of vortex where stretches and then in the center, this infinite velocity of the fluid is achieved, which can't happen in reality. So it's basically a solution that kind of verifies that these equations can't model the—

The reality.

They're not exactly, they can't model actual reality because they lead to basically this impossible construction. You can make this construction, which is impossible to actually happen in real life. So we knew all along, in a sense, that this is— we're modeling things as a continuous fluid, and actually water is not a perfectly continuous fluid. Actually, it's made up of molecules and stuff like that. But the solution is the mathematical proof. Of there is a solution that infinite time has this velocity singularity that we know is impossible to have happen physically.

So how does GPT solve this problem?

Yeah, that's a really good question. I do think with this problem, it's still a construction of something, which there's, there's differences in mathematical proofs of things of how do you show some result? And this result is still through Constructing something. And that's where having 10,000 agents that are all trying all kinds of different constructions, which is, that's, they had at their peak 10,000 concurrent agents working, is something of, if there is one construction that works, if you throw enough intelligent, enough power at it, then if that type of construction exists, Which we didn't know whether it exists or not, but it does exist. So it's almost like the paper is 160 pages or something. So of course I have not had the time to read it, but I think my first impression is a little bit, it's like an intelligence-based needle in a haystack type of finding the needle in the haystack, but the haystack is all of these possible things that you need a lot of intelligence to be able to understand how they would develop and things like that. Yeah. Which is not to take away from the feat because a lot of things like that are constructing the right solution. If we think about algorithms for important optimization problems, it's about putting together the steps in a right way to get to the solution and then proving something about why those steps get you the best possible solution very quickly. There are a lot of important problems like that. There's still, I would say there's still some room for math AI doubters. Yeah. Would you say that some fundamentally new type of mathematical arguments that are completely new and creative are in there? I think there's at least room to say, I don't think they're in there.

On the surface, the solution just sounds, if you throw enough experimentation, then you hit something. It's statistics, right? You increase the surface of the achieving the goal, and then you're more likely to achieve the goal compared with if you have a really smart person and then just doing the experiments in a linear way, then for humans, we just prioritize. We just find what's the most probable solution, right? We go from there. And if it's within a human achievable amount, then maybe a person can do it and they're going to win the Nobel Prize, right?

Yeah.

But with agents, it's brute force compute plus some level of intelligence.

Yeah. I think it is the right way to think about how do people come up with good solutions to things is we have this level of intelligence and creativity that we can narrow down to maybe I have a couple of different approaches that I think are reasonable for how I might solve this problem. So let me explore each of those in depth. Until I hit something. So the, the reason that I can solve, or I have been able to solve problems in the past, is because of that type of approach where I'm not just brute force evaluating all the different possible things that I could do, which would take me forever. I have enough intuition and I guess reasoning or something like that and knowledge about the field to be able to narrow down those approaches. Now, if you have 10,000 agents, Then it means as long as they're intelligent enough to evaluate whether one approach is going to work, and as long as they're intelligent enough to narrow down to somewhat reasonable number of approaches, then you can do this brute force looking for a solution type of thing because you don't have— if you can get around having an exponential blowup of the number of things that you're trying to check, then You will at least have, it'll be viable to do this type of search, basically.

So the existential crisis is applicable to probably not just the mathematicians, probably to everybody if this is the solution to everybody. So what do you think is left for you as a scientist and mathematician? Does creativity still matter? Intuition still matters? What if those 10,000 agents are To the level of at least your PhD students?

Yeah. Yeah. I think there's a lot— math is in a unique spot because of just, if the solution is correct, it's correct. So there's no need for human, or there's much less need for human taste or evaluation of the solution. And the question will be crystal clearly posed where there's enough things in the world, even practical questions that are more on even the engineering side, where maybe what question you're asking is not exactly clear. And maybe there's, for some practical problem of implementation of a real-world system, maybe there's 10 different possible reasonable things that are all correct in some degree. And then you have to choose what is your preference between these things. I mean, think about something like designing a bike. There's no correct way to design a bike. And more in that type of a case, I would still be very much be able to get assisted by AI, but that's more, it can give me 10 or 20 proposal designs and 3D models and cost estimates of all these things. And then when I have to actually make the choice of something to go in the real world, then I have to choose from all of these existing designs. Given approaches. And even in applied, in more applied research, there's a similar type of problem. Let's say I'm a researcher in cognitive neuroscience and it takes me a month to collect data from behavioral experiment that I'm running because every session with every subject takes 1 hour because the experiment is actually recording behavioral data from the patient. So it takes me a month to do the data collection. And now if I want to either build some system, maybe some predictive system out of this, or I just want to even write a research paper about it, then AI can help me get approaches for, okay, what models should I use to model this? Or what ways could I possibly, what types of, maybe there's some product that I want to build that catches when someone's getting distracted. If I have to actually in the end choose something to implement, I have to choose something to use and there's no single correct answer, there will still be metrics of performance metrics of different methods or things like that. If I'm actually building something that's using real data and going to go into the real world, then I, even if I get a ton of great solutions, great possible solutions from AI, I still have to choose one of them because I can't do all of them. And at that point, I don't see people just saying, oh, GPT, just choose the one that you think would be best. Because that would be very similar.

That would eliminate us.

Exactly. Like you could, if I say I'm launching a bike company and I want to like launch my flagship electric bike that I want to start with, that I could have GPT do it end to end. But would I not at least take a couple days to really myself strategically evaluate all these choices? And that comes also with choices of those design choices. They depend on like, how do I want to brand myself? What is the story? What's the vibe? What is my target audience? And things like that. And those are questions that you can also get assisted by AI. I could, with AI, I could say, you know, look at all my competitors. What would be a good target audience? Or what are some target audiences that I should consider? But that would be like, that's like using AI as a superpowered web search. It's like, do the research for me on my competitors and stuff like that. But I would still be in the loop and in control.

If you think about this pipeline we just discussed, We're building a bike from the research scientist who's thinking on what can be plausible to build a more advanced version of electric bike. And the use of AI becomes a super data collector, right? And a very powerful experimentalist who's doing a lot of experiments, design partner. But I think those 2 are the most time-consuming thing if we humans were to do that.

Yeah, absolutely. And one of the Major factors, I think, going back to the idea of if I'm doing a math problem, I can have the agents just try out all of the approaches and see if they work. I can't do the same thing in any kind of applied context because if it's something interacting with the real world, that has cost and time if I have to build and deploy this thing. So I can only, I have to choose one approach in the end to follow. And I can't have the agent, okay, maybe you can do simulations or predictions and stuff like that, but those are never going to be 100% reliable. So I can't do the thing of just 10,000 agents, try every approach in depth and see how it works out. Because if I have to build 10,000 prototype bikes, I'm going to go broke.

But then it also helps with design downstream, customer branding and all those kinds of things. Let's assume that I just take you completely out of this equation. And I think at some point AI will be able to do something like that. That's what I'm trying to push on. Like, what's the role for humans? One thing that you mentioned I think is really important is at the end of the day, AI is not a consumer. We— you're building this bike company because we're human, we ride bikes. It doesn't matter if a robot comes out and starts riding bikes, right? Do you really care? I think we are the consumers. That's why you have to be in the loop.

Yeah, agreed.

Maybe it is not an existential crisis in a way. We're just no longer doing what we were doing, right? Trying to narrow down the experiments and trying to create what we want to create from that perspective. Instead, we're just working with AI and doing more large-scale Scale, you know, test. And if our role is to choose, then that's where the creativity and intuition goes, maybe.

Yeah, I think I totally agree. And I think along with that comes also some idea of personality and taste and curation, which I see, for example, in dashboards that people make. I see when people make dashboards with Claude or GPT-Sol, they all look pretty much the same. They look very similar. They love this one type of font. I don't know what it's called, but it's not Times New Roman. It's not Arial. It's not, it's not one of those ones that you see very commonly. And they love these color palettes of dark greens and off-whites, like the wall behind me. They love those types of color palettes. And they'll then, for anyone who asks for a dashboard without infusing their own design choices and specific design and taste instructions, The dashboard will always look roughly the same, or it'll have a similar vibe. And that's because ChatGPT doesn't know that it's made for 10,000 other users. It doesn't know that it's made the same look of a dashboard. It just says, because it's working from zero and as if you are the first user to ever ask for a dashboard, and it's gonna make pretty much the same design choices every single time unless you instruct. And the same thing goes for writing and things like that. If ChatGPT uses some dramatic wording and em dashes and stuff like that, it doesn't know that it's done that for 10,000 other people and that then everything ends up looking the same and all the writing has the same personality. No. So that's something where then infusing your own personality and vibe and taste into whatever you're building with AI, I think, is so important. I do applications. I have For various different things, I have reviewed applications that included resumes and things like cover letters. And whenever I see some kind of statement that's written where I can immediately tell that it's not because it actually has personality behind it, it has a much stronger like impact. And it's like this immediate flag. A lot of times it's hard to—it's of course like hard to tell whether something is AI written. And there's a lot of in between of like maybe. This person did use AI. Maybe this person just has been working with AI for so long that their own writing style has become the AI writing style. But sometimes there's things I read that are like so clearly a real, like, human personality behind it. I find like really refreshing, and especially for something like applications, if there's things like personal statements or cover letters, it's that like infusion of the personality and taste that I think Makes a difference.

Do you think this personal vibe and personal taste will be encoded into AI soon enough? I actually interviewed several companies who are building digital twins. So there, you want to create a version of you that can make decisions for you or write for you, for instance. And some founders predicted that this is going to happen, that at one point, they don't know how soon, but maybe my agent will talk with your agent that will sound like me and yours and sound like you, but this scales you in a sense. You can talk with 1,000 people, right? I think there's research talking about how much human relationship we can manage as one person. I think it's 100 or something like that.

Is it?

Yeah. Number, right? It's a village. And then with AI, if that happens, think about how many people you can communicate with. And then it's large-scale experiments, and then you'll be able to sift through, okay, there are opportunities among all these conversations, but you get to talk with more.

Yeah, will we have personalized agents that are like self-digital twins? I think I could see it. And even with OpenCLAW agents and stuff like that, I think people are trying already to do that with give it a whole sample, like writing samples of things you've written and maybe communication style with maybe just giving your whole email inbox as context or something like that. I'm not totally sure in either way. how well that works for personality copying, because I haven't, I don't know anyone who's really done this beyond personal agent to respond to quick administrative emails or stuff like that. But it would be very strange. I think that's the, where we also get into the territory of what am I happy to outsource to AI? And for me, that includes especially things like writing my code for me, for even experiments and research and stuff like that. And Anything that has to do with what I feel like defines me as personality and taste and stuff like that, I would prefer to keep myself. And it would be interesting if some agent, like my OpenClaw agent, emails that I've had it draft for me when I was playing around with that. I very frequently, I was just like, this is absolutely not what I'm trying to make. Granted, this is a tiny agent running. Yeah. Consumer-grade GPU. So this is not a super advanced model or anything like that. But still, I'm like, this is nothing of how I want to communicate. So in that sense, I would lean towards, I guess if that was like a very well-made product, I might still not want it. Yeah. I might still not want it. I would at least want to check off reply responses or stuff like that. I wouldn't even want to, with any like friends or other people, or even people in professional relationships, like Anything that's not some very simple— yeah, like anything that's not a very simple administrative thing. If you ask me, oh, could we push the meeting to later, whatever, something like that. If my AI agent automatically said Philip is available from 1 to 3 PM, I checked his calendar, we can do that. Then I would be totally fine with, but anything beyond that actually involves me in some more meaningful way, like something—

I guess. Interview your agents. Exactly. Offer them real promotions.

Yeah. I would think about it as if I had a really good secretary, like if I had the best secretary in the world, like what would I have them do?

So it doesn't seem like you're gonna let them be you.

Exactly. Yeah.

They cannot replace you.

Yeah.

But they'll do a bunch of other stuff for you.

They'll do a bunch of other stuff to make my life easier. And then on top of that, I also have a super talented PhD student who knows math honestly better than I do. So with all those things, and I have a coder who's better at coding than I would ever be and all that stuff. But even with all those things, it's more like there are certain things and it's especially communication with other people. Like anything that has to do with things that I enjoy, like personality and taste and things like that would be something that I would maintain doing myself. In terms of personality taste, one of the things I was thinking about Recently also would be, I have no idea and I haven't had time to actually research this, but how good LLM frontier models are at generating actually good standup comedy. So I think that's one of the things where the personality of the personality and taste and the way it works with the individual who's performing it is what makes the entire routine. 2 people can tell the same joke. And you can't just remix old jokes that people already know. It would be interesting to have standup comedy competition where some of them are genuinely human entries and some of them are standup comics, but they're doing a routine that was written by an LLM and see how the audiences score them. I don't know how, it's probably difficult to actually implement and people wouldn't—

Would hate it.

The standup comedy community would definitely be like, we hate this.

Screw you.

You don't wanna do this experiment.

Yeah.

But it would be interesting. So not saying I'm not endorsing to do this, but it would be cool to see. And I would suspect that the LLMs would do terrible, but you never know. I don't think comedy is what they excel in.

What about a robot standing there talking jokes?

A robot?

Yeah, a robot.

Oh yeah.

Yeah. I saw some news about in China, they're having this sprint or marathon tournament among robots. It's just, do you as a human really care about how fast a robot can run? Other than maybe there's security issues behind it, right? If every policeman becomes robots and they all run. Everybody, then that's a different question. It's not entertaining to me. I enjoy seeing people, human beings, athletes excel in what they do, but who cares about robots? But maybe some people do.

Yep. Chess is a great example because chess has been solved forever, but in over the last 10 years, chess has grown massively as a viewer sport. I don't know if it still is, but a couple years ago, maybe 5 or 6 years ago, I think it was even the most watched content on Twitch was chess. Chess streamers became huge.

Okay. What about quantum? How is quantum going to affect all of these? Isn't that going to scale these experiments even faster, data collection even faster?

Yeah. How is quantum going to affect AI? My hot take would be that quantum AI is pretty much dead and at least dead as a practical proposition. So there's A lot of, or there was excitement in the 2010s, maybe especially 2015 through started ending 2020. What about quantum neural networks or quantum SVMs for machine learning classification problems or image problems or problems where you have massive datasets? And pretty much nothing really panned out.

Why is that?

Great question. Why is quantum AI not really working out is that Problems, in order to have a quantum speedup or quantum advantage, they need to have this like very specific structure. Some structure that like the specific weird nature of what quantum computing is can exploit. The number one thing that quantum computing can exploit is simulating a quantum system of some sort. So simulating actual quantum interactions, that's the best thing that a quantum computer will be good for. Because if I'm doing like quantum chemistry or like maybe some kind of material science where there's actually quantum effects I have to model, those are extremely difficult to model on classical hardware, but quantum hardware is perfect. And then there's the very weird unicorn of a problem of integer factorization, which underlies RSA encryption, which has a very, very weird way of getting this exponential speedup. So that's an, that's an absolute outlier of a problem in terms of much exponentially faster quantum algorithm to solve that. But it has some, there's some special things about that problem, which is A, the problem input is really small because it's just 2 large integers. So I don't need a lot of, I don't have practical problems with like, I need like quantum that I need some huge amount of like quantum RAM or like quantum data management. On the other hand, modern AI models, even like a tiny LLM, needs a lot of RAM to run. And modern machine learning is running on massive datasets. And the types of operations that modern machine learning algorithms use, and like classical machine learning algorithms, simple things like linear regression and so on, are not things that have been able to, that are able to run much faster on quantum hardware. Even if you had like theoretically perfect quantum hardware, there's not really anything that you can do with it in the AI and machine learning space. So what's the like practical proposition of quantum for AI? At least my view is if you really dig into it, As tell me what problem, what method and algorithm, what dataset, and what hardware you need to get an advantage with quantum computing in AI. There's no answer for all of those together. There's some like, oh, things that are close. There's a paper on quantum transformers that would be like for LLMs, but it's not practically implementable. There's a lot of problems that even the authors are like very aware of these problems too. It's not like they're just like selling this as, oh, we have quantum LLMs. They know there are very clear problems with like actually implementing this. Those are problems of things like quantum memory management. There's issues with quantum computing. For example, you can't copy perfectly a quantum state. So there's even within that quantum transformer problem, there's something where you have to have, you have to recompute and read out states many times, and that makes it already very impractical. So these are potentially interesting theoretical algorithms and results. But they are highly impractical. And there's some work that I have done with the NASA quantum group that was also showing impracticality of a lot of quantum algorithms. Basically what happened there is I was researching optimization methods for some type of optimization problems over networks. And I used an existing quantum algorithm to solve a subproblem. That I broke this network optimization problem into. And then at the end of later in the research, when I had an actual algorithm to solve this problem, I wanted to figure out whether it was practical. And then it turns out that even that subroutine that I was using for solving a subproblem, which was using an existing quantum algorithm, that existing quantum algorithm was completely impractical. Impractical, meaning that the size of this network, there had to be more Like nodes in the network than grains of sand on Earth. And every node in the network had to be itself a quantum computer. That was— there is no way at all that would ever be practical, even though like in the like complexity theory way, the algorithm looks a little bit faster. So it's a little bit faster in some ways on the surface of scaling for mega-sized problems, but The amount of scaling where you would see speedup is completely impractical. And that's something where quantum computing has a huge challenge of the hardware is just so much harder to deal with. It's so much harder to scale than classical hardware. Quantum computing suffers exactly from those problems.

So if quantum is practically useless for AI, at least as of today, where is quantum genuinely useful right now?

Yeah, where is quantum genuinely useful goes back a little bit to anything that is in some way or another simulating or needing to simulate quantum dynamics. So if we think about quantum chemistry, like interactions of like individual molecules and even like subpieces of molecules, those can have quantum effects with each other. Materials, potentially material science can also have like quantum effects. And then There's some which I know less about is the drug discovery. So drug discovery is like potentially somewhere in the middle. There's potentially also some, if there are any like quantum interactions, but that can also go back to like quantum chemistry. And like anyone who's doing quantum physics, it's of course like excited because now you can simulate, you can actually have quantum, you can study quantum effects. In the quantum computer itself, but those are benefits to a relatively smaller group of people than at least mainstream discourse about quantum computing, I think, is. Within optimization, as an optimization researcher, I don't think quantum computing is helping with any of the optimization problems that We are solving for large-scale industry applications. That is maybe a little more disputed, but there is sometimes what I perceive as lack of classical optimization expertise on some of the industry side. I can give an example, which was an Airbus executive that gave a talk at a conference I was at some years ago, talked about using quantum computing and starting to integrate quantum computing for some problems that are like as efficiently as possible loading their airplanes. And this is a variation of something that's called a knapsack problem. And people who are experts and knowledgeable about these types of problems would tell you that you can solve very large instances of these problems with the right classical methods. In seconds on a regular consumer-grade laptop. So Airbus actually put out a quantum computing challenge that had the types of problems that they're interested in. And they had a particular instance that I actually ran on my laptop and it took 1.1 seconds to run on actually a Microsoft Surface. So it's like a tablet. It's not even a laptop. A powerful laptop. So that is, even if quantum computing can solve your optimization problems for you, some of them, maybe at some point, the competition is against classical hardware and classical algorithms and classical solvers, which are so strong. They're so good. And they can solve these massive problems where the size of problems that you can solve efficiently, solve quickly, classically, is already too big to even fit on a quantum computer. So it almost takes away even the room for quantum to be helpful in those scenarios, unless we have some totally crazy breakthrough on the engineering side for building quantum computers.

Sounds like it's doing the labs. Maybe that's why you left NASA and now you're teaching at Berkeley. Tell me about your classroom. What's happening?

Yeah, what's happening in the classroom? I think a lot of questions of what is the best way to use AI in the classroom and what is the best way to guide students to use AI well. The truth is that people look at their professors for guidance and expertise and stuff like that, but we're still figuring this out.

What do you tell your students?

I tell them to try to use AI in ways that will help them learn. As much as possible. One of the things I've done in my classes now is homework is not graded on correctness. So that's because you can always get 100% homework score if you want by just asking GPT to do it for you. So I think that scoring homework by correctness, it misaligns the incentive because now the incentive becomes to get 100%, and then that incentive pushes you towards just having GPT solve it. Instead, the incentive should be for you to learn, which is then I should incentivize you to just give an honest attempt, an honest effort, and engagement on the problem. So I say, as long as you have an honest attempt, honest effort to solve the problem, you just get full credit. And then your incentive is just pretty much hopefully to learn. So then you should— why would you just ask GPT for what the answer is? In that type of case, so those are things that I also know a lot of other people are implementing similar things for how to manage homework and projects are like the two big things because those can be like highly AI assisted. Everyone's going back to paper and pencil exams. That's pretty much we had a very short spell of people liking take home exams that started during COVID Pretty much all back to. Paper and pencil.

So you sit in the classroom and take the test there?

Exactly. Yeah. You do the test in person and we're testing, we're back to the way we did tests 20 years ago. I think for testing the foundational knowledge of these are the things that you need to know for this class, I think it's relatively reasonable. But in other ways, there's a lot of experimentation that is happening because we don't just know what the best thing to do is. I know that there was someone. Berkeley, who actually experimented with open internet, open book, and open ChatGPT exams. So on the exam, yeah, on the exam, which was in person, students were allowed to use ChatGPT. I think the exams were in person. I'm not 100% sure, but I think they were in person. And then the average on the exams was like 99%. So then it was like, okay, we tried this thing. The reasoning was like, in the real world, you'll have ChatGPT, so I should allow you access to them. And then the scores were like 99%. 99% average or something like that. And there's the question of what's the point of this? And everyone has an A+, but they don't actually, they might not actually know anything themselves about the content, but still someone has to experiment with these different approaches. And so we're all collectively figuring out what works best. And that is, I do wish there was a more formal collection or discourse, which does come Through conferences and stuff like that. And there's just, there's a lot of conversation about if I'm trying something new in my class that's AI integration or AI related, other professors are also trying new things that they think might work well. And then in the end, we're all talking to each other and exchanging ideas and then implementing the things that work best. So I think eventually we'll converge on at least some set of strategies for like how we manage this and in the end, just how we maximize student learning in the presence of this.

How do you measure if a student actually spends reasonable amount of effort on something?

Yeah, that's a good, good question. How do we check? Does a student really put the effort in? To be honest, it's pretty much impossible to measure because they could have asked ChatGPT to do it for them and give thought Answer, but that's where then testing is at least because there's then if there's a test that's in person and on paper, then that incentivizes you to really use the homework as learning exactly to study as really and what am I doing in the end as a professor? I am trying to teach you and I'm trying to guide you through studying. To learn a material, to learn something, you have to do exercises and practice to figure out what do you know, what do you not know, and so on. So the homework really serves as what it really is. It's a curated set of problems to help you understand, did you understand the material? Do you get, are you able to do what you will have to do in the real world when you're solving these types of problems? Okay. You'll be AI-assisted. In that case, but still you have to have the foundational knowledge. And even just very practically, anyone who hires you for a job where you're, I don't know, doing supply chain optimization, they're going to want you to know about supply chain optimization and not just only work with GPT and say, I don't know anything about supply chain optimization, but I'm really good at asking ChatGPT about it. So you should hire me. I think just practically you're not going to get a job in that way.

Do you have to convince your students you should still make effort to learn even with the AI-assisted style of working today?

Yeah. Is there some convincing that has to be done? Convincing even though AI can do all these things, you should learn? I think there is a little bit. I think there can be a lack of motivation from students, which I Very much sympathize with. It's like, if ChatGPT can just do all these things, why should I learn these skills? And it goes into something that I would think about as the AI learning paradox, which I think about as there's no single individual skill or piece of knowledge that you actually need to have. Because any individual specific skill or piece of knowledge, you can ask ChatGPT for, or you can look it up. And even for a skill like I have to fix my flat tire on my bike, I can watch a YouTube video or YouTube tutorial, or I can take a picture of the flat tire, send it to my favorite AI and tell them to tell me how to fix it. No individual skill or individual fact of knowledge actually matters, actually is necessary. Yet, if you have No skills and you have no knowledge, then you will be like hopelessly lost in pretty much everything. Or you'll be like hyper-reliant on, you'll be too hyper-reliant on AI models and you won't have like judgment. If there's multiple different possible approaches, you won't have judgment for like which one should you pick. You might not think about like external other factors or other context. That maybe weren't directly part of the problem, if you don't have some kind of subject expertise or knowledge. So it puts students into this position where if you forget about like a specific individual thing, it doesn't matter. But still you have to build like a critical mass of skills and knowledge so that you can be like properly informed and so that you can actually like even know like, what should I do and what kind of things should I consider?

So what does that mean to the younger generation if they ask, what kind of skills should I learn because AI can assist almost everything and how do I acquire that?

Yeah. What kind of skills should I learn? How do I acquire them?

Talking about skills and you still need skills and a massive knowledge. What would that be?

I think there really, it's a combination of would be mostly the foundations of whatever they're interested in and making sure that you really know the foundations very thoroughly yourself. Because even for, if I ask AI to explain something to me, if I have a certain amount of foundational knowledge, I'll be able to understand it much faster. And same thing with factual knowledge. If I have all this factual knowledge about like some maybe biology or something like that. I know all about the organization of the brain and the neurons and stuff like that. Then it'll make my interactions with AI much more efficient because the AI can talk to me as an expert. And then it maybe just, it's enough to, for it to say a couple of key points. And I know exactly what's going on because I know that the frontal cortex does this and this thing and the hippocampus does this and this thing. So I don't need like all that. Additional explanation. So I'll be able to work much faster if I'm, I don't know, building some mind reading device or something like that. If I have that critical mass of foundational knowledge, I'll be able to work much more efficiently with AI. And if I forget an individual piece of knowledge, that's okay. I'll just ask for it. And even to acquire that foundational knowledge, AI can certainly help you with that. AI is certainly a very helpful and powerful learning tool. Students and just people in general, I think, are the AI savvy people are very much taking advantage of and making use of. It's really useful to learn about science of learning, like human learning. For example, retrieval practice is retrieving, reciting a fact, remembering a fact. It makes it stick in your head much more. Or explaining a concept and so on. So quizzing is actually a highly effective studying strategy. So if I want to learn something, then I should practice retrieving it. I should do retrieval practice. And that's easier than ever with AI because I can just ask AI to quiz me. And that goes from any kind of, that goes from language learning. I can give it a list of vocab and I can just say, quiz me on this. And you can do that with voice AI, so you can do like Language vocab practice while you're driving. And for students, like feed in your lecture notes from the day at the end of the day and ask AI to quiz you on it. Because if you get quizzed on it, you'll just, like all the learning science says, you'll remember it better.

I feel some young people, I think they could be lost. It almost feels like what they are relying on is some sort of self-motivation. I don't know how you land on the subject of math and quantum computing, but you appear to me that you have intrinsic interest in these subjects. A lot of my friends, including myself, how we grew up is a lot of driven by how our parents are teaching us. You know, being Asian, parents telling us you have to learn, say something like accounting, medicine, or More so that you make more money, right? A lot of drive comes from those— External pressures. Yeah, exactly. But I think that's really bad in a sense that we were told to do so and we weren't self-driven. And a lot of people, even when they're in middle career, they might regret or they're reflecting on the career path they chose. But with AI, I think this problem is amplified in a way. I don't know if your students are thinking, why am I studying industrial engineering? Because everyone is going to use AI. Your students who will show genuine efforts made, they will be the ones who are self-motivated. And you don't have problem with that, right? Yeah. And I don't think they'll have any problems for their careers because they're self-motivated. They can learn, they can find ways to learn and find ways to use AI in the right way. But the others, Let's say you ask them to make efforts on their homework and they just pass it through ChatGPT. It's just, you can't really tell. I think there are a large amount of students or younger people in that group.

Absolutely.

They're more lost. Do you have any advice for those people?

Yeah, I think that's a, it's a really valid and large concern of, in some way, the, I think at the moment, the biggest blessing that you can have Is having some intrinsic self-motivation and drive to learn something, because then you don't care if AI can do it better than you. If you are like motivated to do this and you wanna learn about this because you're curious, and that is an intrinsic motivation, then you'll be happy studying this, whatever it is, even if AI can do it better. It doesn't matter. I think there's Definitely a motivation issue in some subset of students. If you, something like data science, for example, is something that as a field, 4 years ago, people would've told you if you were graduating high school, would've told you like, oh, data science, like fantastic. And computer science, software engineering, also like fantastic. And like, you'll be like on top of the field and you'll get a great job, earn a lot of money. There's still a lot of great jobs in like data science and computer science, software engineering and stuff like that, of course. But it's nowhere near like what was promised to those, like to college students who started their bachelor's degree 4 years ago. And then you do have students who like Would've chosen this as something that was not necessarily like pure intrinsic motivation, but could be through external pressures. It's like in the past it was always like studying business was something that like stereotypically like people studied if they didn't really know what to study, but they were like, oh, smart. And their parents or someone told them that like studying business, you would get a good job and make good money and have a good life. Yep. And computer science, data science has Turned a little bit into that. Maybe like computer science, maybe a little bit less so than like data science type of fields. And then if you're not intrinsically motivated to study something and AI can do it better than you, then I think those are students to worry about. So what's the best thing that I can try to do as an instructor is try to like instill, kind of instill motivation through really communicating as much as possible, like why this is interesting. And maybe I can get a couple students who maybe I can actually get them motivated by showing them applications and more interesting, more like fun applications. Then if someone cares about sports, then maybe now they can also care about some kind of optimization or machine learning or something like that through seeing those methods active in sports analytics or something like that. And I think it goes a lot through motivating what you're studying. Every topic that you introduce as an instructor, I've always felt like motivating a method or motivating with some real-world interesting example has always been helpful. But I think for those students, It's even more important. Just communicating, I, as an instructor, am obviously gonna be passionate and motivated about the things that I'm teaching, but I want to also instill that on you so that hopefully you can find— I get it. Hopefully you can find some satisfaction and maybe even a little bit of motivation and purpose and stuff like that through studying this. Interesting thing.

With AI, do you now question about the purpose of education or college education?

I probably less than others. I think there is some problem of we think about what is the purpose of education now that AI can do almost everything so well. There's definitely less need for specific technical skills and specific technical details, but there's some form of higher order thinking and higher order problem solving and reasoning and strategizing and things like that you learn through studying a lot of different fields. Probably matters much less what exactly you studied now, as long as you're engaging your brain to basically become smarter, you're doing— you're good. Really, the goal is become smarter, level up, and gain more skills and knowledge so that you can understand the world around you in a better way, and that you can find ways to contribute, and so on. Because that will always still be necessary. Even if we want to go all the way to AI, your context window as a human is way bigger than even frontier models right now. You have context of the whole world in all available to you pretty much. So I think there's still a lot of value in the pure education sense. And then of course you need it to pass your interviews. That's the pragmatic answer, but even beyond that, I think there's still a lot of value and purpose. Yeah. Has it diminished? Probably.

What's diminished?

The value of a degree. Now someone who's a smart person with good judgment and stuff like that, if they were starting their own company, like Whether they have a degree or not and whether they have specific knowledge about different things matters much less. So in that sense.

If you were to choose again, if you were going to college right now this year, do you have a suggestion? Would you choose something different, like a different major?

I don't think I would choose a different major. So I was, as an undergrad, I was applied math. I think after that in grad school, my work was Very theoretical, I think you're in a better place to be working in a space where in the end, there's something physical about it. I think that's a good place to be. So if you're thinking about pure math versus some kind of engineering, in the engineering side of things, you're still gonna be doing a ton of math, but at the end, there's some kind of a physical thing. Yeah. If you're doing material science or mechanical engineering or even industrial engineering, there's a connection to the physical world and that is protecting you a little bit. It's protecting your usefulness because it's not just developing software or things like that. Like AI can be in the loop for the entire process. Once something is involved with the physical world, yeah, there's robotics and stuff like that. But that's totally different world of so much more expensive, hard to train, and who knows really what's happening there. It's at least much slower also to progress. I would probably have told myself to consider pivoting into something where there's more something physical involved. We share a floor with nuclear engineering and over here, industrial engineers, a lot of us are, we really just do applied math. And we're all in an existential crisis because of AI doing our work much better than us. Then there's the nuclear engineers over there. They have to, in the end, they have to build some kind of device. They want to do fusion better or stuff like that. Even if AI can help you in that process, which also I'm assuming that if you ask ChatGPT anything about nuclear engineering, probably it just says, I can't answer that.

I don't know. Yeah.

So they're probably cut out. But they have this nature of at the end of the day, they want to produce a physical device and they can't just try all different possible device configurations or solutions. They have to choose one to build and test and collect all this data on. And then if AI can help you analyze the data and organize the data from your experiments faster, fantastic, but it's not doing the whole thing for you. for you. So any connection to a physical thing, I think, is a huge— is a big benefit for whatever you're studying at the moment.

Thank you so much for all the tips, by the way. Maybe we should revisit next year again to see where AI is. Yeah, that

would be

awesome. Yeah, see if we're seeing stand-up comedians speaking LLM jokes, but I don't think they're gonna tell us that. I don't think they're gonna tell us that. But if robots are standing on stage and telling jokes, We'll definitely see that, right? So

yeah,

but in summary, I completely agree with you. I think everything related to the physical world, it's not just a device, the physical device, it's also the human-to-human relationships.

I agree. Yeah.

Everything that's in person becomes more valuable because there's a bigger constraint on these things, right?

Yeah.

It's also fascinating to see how education is going to evolve over the next few years as well. Maybe next year. You'll be different teaching different things.

Very possible. Yeah. It's evolving. It's keeping me on my toes all the time. And yeah, next year, I feel like even one year away, it feels so— what will the frontier AI models next year be able to do and look like? Given even in December of last year, AI could not help me on my research at all. And then in March. It started being helpful where I think 5.5 and I think it was Opus 4.6 or something like that. Like those 2 models were for one of my projects. They were actually, it was the first time where I felt like, oh, okay. They now understand what the purpose of this research project in this repository is. And then there was the first time where I was really using those models to contribute. And then in July, it solved a problem that I'd been thinking about for 2 years for me. And then now it is solving Millennium Prize problems. So it's like, what, 9 months from, in terms of research, going from zero to a little bit helpful to solving the problems I'm thinking about to solving problems that people have thought about for 60 years. And so, yeah, we'll see where we are in a year.

You know, maybe we should all, and including the younger generation as well, we should all go after those 60-year-old, 100-year-old problems to solve. Maybe that's what we should do.

I have students ask me to do research very frequently, to get involved in research. And it is tempting to tell someone, a student, be like, look, just go crazy on any problems and I can give you ideas and fields and things to look at. And if you think that you've found something, then just come to me and I can evaluate it as an expert. And if you have a real result, then we can just write a paper about it. That would be totally reasonable because if it's a real result, and then if you— I think it's really important to also communicate and write the result very well, which lots of controversy around the OpenAI Navier-Stokes solution and all that stuff. So that's something that I would put a lot of value on and okay, make sure that the result is communicated properly, that we're hyper-transparent with how AI was used and stuff like that. But in principle, a freshman or anyone could potentially solve an important problem in some field if they are good with their like prompting and using the right models in the right way, I think is realistic.

Yeah. Way to go. We'll see you next year.

Yeah, I'm looking forward to it. It'll be great.