OpenAI’s IMO Team on Why Models Are Finally Solving Elite-Level Math

The pace of progress is really, I think you see it so clearly in math. And I think Alex tweeted about this, where even a few years ago these models were struggling with grade school math. And then I remember even in 2024 that Gsmak was used as the standard eval when everybody would release a model and then it was math for a short period of time and then it became Amy and then it became Usamo. And the pace that it's just gone, blown through all of these math benchmarks is really astonishing.

Today we're joined by Alex Way, Cheryl Hsu and Noam Brown, the Trio behind the OpenAI model that just achieved gold medal performance at the International Math Olympiad. The IMO gold is one of the most important milestones in the race to artificial superintelligence. And what makes this breakthrough particularly fascinating isn't just the mathematical chops, but the underlying architecture. General purpose techniques for scaling test time, compute and handling hard to verify tests that extend far beyond competition math. We've now gone from models that can reason about math for a tenth of a minute just a year ago to systems that can reason and concentrate on the order of 100 minutes. The hope for superintelligence is that as we scale reasoning to thousands or hundreds of thousands of hours with can begin to solve humanity's greatest unsolved problems in math, the sciences and more. Alex, Cheryl and Noam joined us on Training Data to talk about their approach and share some of the behind the scenes fun and learnings behind this historic result. Enjoy the show. Alex, Cheryl, Noem, thank you so much for joining us Today we have with us the team behind OpenAI's first gold medal at the IMO. Congratulations to you all. It's a momentous achievement.

Thanks.

Thank you.

I'd love to get into a little bit of the origin story behind this. I know that the IMO gold has just been this elusive thing that everyone in AI has been chasing for a long time. I remember back when Sam pitched us in 2021, it was on the slides and I remember thinking, oh, that seems really far away. I'd love to understand the more immediate origin story for this specific effort. When did you guys start thinking about this and how did it come about?

Yeah, I think it's something that we've been thinking about for a long time. I remember in my first week at OpenAI, Noam asked me, when do you think the model will get IMO gold? And I thought it was really unlikely in 2025, but I feel like it's something that's always been on our minds, as you said, Sam, many years ago as well. But this specific effort, I think it was really only maybe a couple months since.

Just a couple months.

The last sprint to get everything ready for this year's imo. And of course we've been working on improving our RL algorithms. The ideas for this started coming together maybe six months ago, but really the last push, like we're going to try to do something for this year's IMO was only a couple months long.

It's amazing. And how big is the team involved?

So we're definitely building on a lot of folks work at OpenAI. This is not possible without a lot of help from people, from the people working on Inference and the Scaling org, the people who do the pre training and the RL training. But in terms of the core team, I would say it's just three of us. So it was a super small, scrappy effort here.

That's crazy. Just the three of you.

Also, it was mostly Alex. Alex had been working on this technique for a while and Cheryl and I were happy to help out as we were getting closer to the IMO to make it a reality.

That's so cool. And how does this even come about? Do you self direct and self choose? I want to work on IMO gold and I'm going to get us there. How do you even raise your hand to work on something like this?

I think it was something where it just felt like maybe it's possible. Maybe if we push a bit for a couple months we can just get there.

One of the nice things about OpenAI is that I think the researchers are really empowered to do the kinds of research that they think is impactful. So Alex had this pitch that, hey, there's this new technique that I think could help out a lot. And, and honestly, there's a decent amount of skepticism. I think some people were supportive, but everybody felt like we should give them the freedom to be able to explore this and pursue it. And then it started showing some strong evidence and I think people still were a little skeptical, but more people were getting excited about it. And eventually it turns into something more substantial. And I think now people are obviously very excited about it.

Can you say a little bit more about the strong evidence? What were some of the early signs that you all were seeing that made you really lean in?

I think it's just like, you know, like progress on hard to verify tasks where I think previously we, you know, a lot of RL was more focused around just like, you know, if you have, like, these verifiable rewards, like, you know, what can you do? We were just seeing more improvement on these, like, harder to verify tasks is I think, what made us excited maybe on that front.

How did you even verify that the results you had were right? And I saw that you published the proofs on GitHub, but can you just say a little bit more about how you even know that you've discovered the answers? Because my understanding is that they're done a bit differently from how a human might answer them.

Yeah, I do think the style of the model outputs is a little atrocious.

Atrocious isn't the word I was going to use. I was going to say creative, like an alien language.

Yeah, it's a little. Yeah, I think we could have. I think it was a very small, scrappy effort. And so we didn't optimize as hard for human readability. But that's something that we know how to do. We can do the same stuff in the same way that ChatGPT is very readable, we can do the same things here.

Do you even need to optimize for human readability? Is that even important?

I think if you're showing this to humans, they prefer readability. We were actually discussing. We got the proofs because you could actually just run them through ChatGPT and ask ChatGPT to rewrite them in a more readable way and the proofs are still correct. They're just a little bit more readable. And we were like, oh, when you post these online, should we post the more readable version that's run through ChatGPT, or should we just post the raw version? And we decided, I think for full transparency, we'll just post the originals and people will figure it out.

You guys have a bunch of IMO medalists and participants in the staff at OpenAI, right? Do you guys moonlight in your spare time grading the answers that the model produces?

During the testing, we read a lot of samples, but for grading these specifically, we hired external former IMO medalists. So each proof was graded by three medalists, and for each one, they reached unanimous consensus on the correctness.

I should also say that for me, I don't know about Cheryl, but for me, the proofs are beyond my ability to comprehend. I was a math major and I never really did competition math, and already the stuff that this model is writing about is beyond my ability to grade.

Yeah, same. I think that's what makes it, like, even more amazing, just, like, how smart the model is.

Totally. What about problem 6, how come none of the models at this year's IMO had a solution and your model didn't even attempt problem six? Can you say more about what makes that problem? And traditionally problem six is always the hardest at the imo. Is that right?

Yeah, I think problem three or problem six usually.

Okay, just say a bit more about what made problem six different and what you learned from. I think you tweeted that the fact that your model knew that it could couldn't solve problem 6 was one of the things that gave you hope. So just say a bit more about.

That as well for problem six. It's just a really tough problem. I think if you gave me months to think about it, if you even gave me a big hint about the main idea to solve problem six, I don't think I'd be able to get there. It's just a crazy tough problem where there's so many things you can do and there's a very narrow path to finding the proof. I think it's one of those things. I think math is just hard.

Yeah. And we threw a lot of compute at problem six. But I think it was good to see the model doesn't try to hallucinate or try to just make up some solution, but instead we'll say no answer. It is kind of disappointing when you feel like it's done so much work just to say no answer. But I think it's good that actually acknowledges that.

Yeah, that's an amazing level of self awareness of your own kind of ceiling. Because, I mean, I remember at least a couple years ago with these models, they'd always try to be helpful and make up an answer. Right. And so to see this is just like, I think an amazing level of self awareness from these models.

When we released the reasoning models, I talked to some professors at mathematicians, computer scientists, and I was asking them, you know, are you finding value in these models? And the answer is, you know, frequently, yes. But the one thing that they would complain about is like, if they would ever ask the model a question that it didn't know the answer to, it would just like output a very convincing but wrong answer. And they would have to like go through it very carefully to figure out was it exactly correct or was there like, you know, some flip of an inequality or something that the model snuck in there. And it's nice to see that this model, like, if it doesn't know, it will just acknowledge that it doesn't know.

At least more frequently, I guess, internally. Did you guys have a betting like a Poly market or something going on, whether you guys were going to win IMO gold this year. And what was the internal vibe?

I think we felt like we had a strong shot, but I think we also felt that it wasn't like a lock where there's definitely a distribution of questions where the models would probably struggle more than the humans. But then there's another distribution of questions where the models would be really, really strong. And I think this year was somewhere in the middle where problem six I think is just out of reach of state of the art models today. And I think maybe in general these hard combinatorics problems, which problem 6 wise I think more challenging and that's still something that the models struggle with.

What is it about combinatorics that makes it challenging versus geometry, for example, which seems like you guys do well at.

I think for combinatorics it's probably because it's a little more abstract, a little more high dimensional. And I think oftentimes combinatorics problems require leaps of faith or leaps of insight that the models aren't as good at. I think the models are more good at problems that require a bunch of smaller steps.

For example, what about from your guys perspective? Was the internal vibe optimistic or not that you all were going to hit gold?

I feel like it wasn't super optimistic. I think they definitely knew that it could happen. But I think even a month or two months back it definitely felt it would have to improve quite a bit, which I guess we did.

I remember I was talking to another researcher at OpenAI maybe two months before the competition and we were like saying like, okay, if we were to bet, I'm a betting man, I've happened to bet.

Yes, you are.

And I was saying like, what odds would you take? Because I was willing to bet on like we were going to get gold here. And he was like, there's really no chance. And you know, and you know he said that he would gladly take like 2 to 1 odds against like the model winning. So like, you know, less than 1/3 chance. But he didn't want to bet against us so you know, he thought it'd be bad, bad vibes to bet against the team winning. So he didn't go for the bet.

So did you make some, some pocket change to him?

I wish, I wish I. I mean.

You need it so because I mean you guys were at, I think you tweeted 12% on Amy like 15 months ago. Right. So even though you never want to bet against scale and OpenAI, it's just an astounding slope of what you all have accomplished here.

The pace of progress is really, I think you see it so clearly in math, and I think Alex tweeted about this, where even a few years ago, these models were struggling with grade school math. And then I remember even in 2024, that Gsmak was used as the standard eval when everybody would release a model, and then it was like math for a short period of time, and then it became Amy and then it became Usamo. And the pace that it's just gone, blown through all of these math benchmarks is really astonishing.

Yeah, I remember training a model on GSMA 8K two years ago.

Yeah, we're past those days, huh? Saturated the evals. What's next, do you think? I mean, at this point next year, you think we'll be solving Millennium Prizes?

I think those are still very far away. I think on one hand, you think about how much math progress has been made since GSM8K, which is just two years ago, was sort of a standard that people were trying to push on. That's an astounding level of progress. But also you think about how much time it takes for people like GSM 8K problems. They, like grade school math, takes someone good at math like, a couple seconds. And now we've gone from like a couple seconds to something that takes, like, you know, these brilliant students an hour and a half per problem on average. Or like, you know, the IMO is three problems, four and a half hours. And then your research math is going to be like, you know, these same, you know, brilliant students, they've grown up, they're researchers. It's going to take them like 1,500 hours. So there's 1,000x of more thinking time. And then Millennium Prize problems have taken entire fields, like people's lifetimes of thinking, and we still don't have much progress on most of those. And so it's on one hand super exciting that we've made so much progress. On the other hand, it's also humbling to see how much further progress has to go from an hour and a half to tens of thousands. Hundreds of thousands of hours of human thinking.

Totally. Noam, I think you deserve a lot of credit for seeing the future on this. I remember you visited us before you even joined OpenAI, talking about the results from gameplay and what happens if you let a model think for hours and tens of hours and credit you. You've really seen the future on this.

Thank you. Yeah, I mean, it's exciting to see it actually happen.

What are the hard things that happen as you scale compute time, inference time, from the order of 0.1 minutes to the order of 100 minutes, I guess at a high level, because not everyone, most of our listeners are not AI researchers. But what are the hard things that happen to keep the model on the rails, so to speak?

One thing I think we can point to is pretty clearly a challenge, is that if you have the model thinking for 1500 hours, then in order to eval it, you have to have it think for 1500 hours. And so eventually the evaluation of the models becomes a significant speed bump on progress. So we're not really at that point. If we have the model think for an hour and a half, it's no big deal. We can run those tests. But to run a test where the model is thinking for a month, it takes a month to finish that test. And so progress can only advance so fast if you want to wait for those kinds of results.

I think both of you are on the multi agent team. Help me understand what the role that multi agent systems play in this is.

Yeah, so in addition to having the model think for a very long time and make a lot of progress on hard to verify tasks, this also involved scaling up parallel computer. And so there's a multi agent component to that. We're probably not going to be able to go into too much detail about the exact techniques, but that was certainly one way that we were able to scale up test time compute for the imo. By the way, one thing I'll add for the multi agents scaling parallel compute thing is that the way that we did it, we really tried to prioritize generality in our techniques. For example, I worked on AI for poker. Alex and I actually both worked on AI for diplomacy. So Alex was on the team that worked on Cicero. Yeah, nice. And those were projects that I'm really proud of. But they were also projects that we spent years working on to achieve that result. And with the pace of AI progress being so fast, it felt like that wasn't the best use of time to develop a very bespoke system that could only do that one task. And so we all really prioritized general purpose techniques and all this. And the techniques that we used for everything for scaling up the thinking time, for working on hard to verify tasks, and for the parallel compute are all general purpose techniques that we're either planning or have used for other systems as well.

And is that the reason you all chose not to do this in Lean? My understanding is the official kind of IMO AI track was a Lean interpretation this year. Is that why you guys chose not to go with Lean?

Yeah, that's right. I mean, there is certainly, I think there is a lot of value in Lean as a tool. You know, mathematicians find it useful, for example, but the priority for us is really general purpose reasoning capabilities and Lean has its limitations and so that's why we wanted to prioritize natural language.

My layman's understanding is Lean is a formal verification tool. Does your result here basically say that informal verification with scale can perform at the same level or even surpass formal verification? Is that the right takeaway?

I would not say that's the right takeaway. I don't know. Alex, do you have thoughts?

I say that these are just two orthogonal components here where I think we found the informal math sort of an interesting problem because it represents sort of like a kernel of difficulty around scaling up test time, compute hard to verify tasks that represented something like difficulties from a very broad set of tasks that we were interested in. From a general purpose standpoint, I think Lean is a little bit more narrow where I think a lot more of the world can be approached with informal reasoning than is formalizable.

I don't think there's anything wrong with narrow AI. Narrow AI can be very effective and obviously far surpass general purpose AI in certain domains. And I think the right way to think about it is in the same way that humans, human mathematicians, find a lot of value in Lean. General AI can be compatible with a more narrow system that's focused on formal mathematics. And the combination I think can be better because of it.

I think I saw on Twitter from multiple folks at OpenAI and I think you guys have mentioned this as well, that this system was built with a very similar approach and infrastructure to many of the recent launches from OpenAI. We had ESA from the ChatGPT agent launch on the podcast last week. Can you say a little bit more about what the similar kind of foundation and approaches?

I think infrastructure wise, I mean we all kind of just use the same infrastructure. But I think as far as the core of this question, like Noam Alex said, there's nothing that's very bespoke to IMO here. And the hope is really that we can use the techniques that Alex worked on as far as nonverifiable tasks and as far as just scaling up test time, compute and be able to apply this to other areas of reasoning or other areas of model capabilities in general and just build stronger models, keep improving agent, keep improving ChatGPT and everything else.

Tell me about the actual experience of IMO day. What was it like?

Yeah, I mean, we were waiting for the problems to come through because once the participants finish the exam then they get posted. And so we plug the problems into our model. And that was around, I guess pretty late at night, maybe 1am or something. And honestly I went to sleep because it's like, you know, it's 1:00am, I not going to stay up for four and a half hours to see the output. I'll just wake up in the morning and see. But I think these two actually stayed up and got to watch the model and see it come in in real time.

It was a lot of fun.

Did anyone call Noam? Like wake up, wake up. We got this.

There were a couple moments where Alex was so exhausted that he decided to take a nap. But we told him, okay, just make sure your phone is on silence so that if we need to wake you up we can call you. And at one we did actually have to call him, but I don't think he woke up.

That's awesome. It must have been such a thrill and such a high especially for that to come through at like. So you got you started at 1am so you must have known at like 9am Then.

Oh, it's four and a half hours.

Four and a half hours for the first day.

Yeah. I don't know. I mean we can kind of see the problems come in. So I'd just be like making sure the systems are like staying stable. And Alex is like over there reading and seeing whether or not how the model's doing.

So you were doing the, you were doing the live human proof, checking to.

See if it was actually.

You're naturally very anxious about the results.

Totally.

So I was just looking at the partial progress the model was making. We can sort of observe that. And then I also hand check things. We were going to send these out to the graders, but I was also just hand checked them because I was so curious.

Okay, well call me next time I want to come hang out there for that. I'm not going to sleep. That sounds awesome.

One of the cool things about these models is like, you know, I can't understand the proofs, but when you see the model, like thinking about will express its uncertainty or its confidence in natural language throughout the process. And it will just kind of say words that will like hint at. It's like, you know, if it's really confident that it figured out, I'll say good a lot, you know, and if it's like unsure, it'll throw in a lot of question marks. And so it's like cool that I can kind of follow along and see how the model is feeling about its progress, even though I can't really tell if it's got it correct or not.

Yeah, you get the dreaded Seems hard.

You got that on problem six, Scott.

That a lot.

No progress. Hard. Same.

Sorry.

Keep going.

Too bad. Wonderful. I guess, looking ahead, you've gotten the pinnacle results in competition math. I guess you can go do Putnam next year, but you're basically at the top, Right. And so what's next?

Yeah, so actually for Putnam, the problems, I think since the exam is like less time per problem than the IMO and it's a little more knowledge heavy, we actually found in our evals that the model was really, really good at putting in problems better than it was at IMO problems. And so I think the frontiers here are really not about these very time boxed competition problems anymore, but it's about problems that really take longer periods of time and more deep thinking to solve.

It's really cool. Okay, so you're going to start proving novel theorems now.

Again, I think there's this very intimidating gap between these very time box competition problems. Then a real research breakthrough, which takes a year's worth of work that's on the order of 1,500 hours instead of 1.5.

Yeah, totally. I guess. Relatedly, I was listening to the Demis podcast last night and he mentions that the hardest thing is actually coming up with the interesting problems to solve. And I'm curious if you all agree with that.

I think there's some truth to that, that these models are really good now at solving these problems. Coming up with them is still a challenge, but I think it's also worth noting the incredible pace of progress that we're seeing. And there's always a next hurdle. And originally when LLMs came out, it was like, well, how do we get them to reason? And then we got them to reason, but then how do we get them to reason on hard to verify tasks? And now they can reason on hard to verify tasks. And I think the next hurdle is going to be like, okay, well, how do we get them to come up with these novel questions? Even creating an IMO question is a challenge and it takes a lot of extra mathematicians a lot of work to do that. But I don't see any fundamental barriers that block us from getting there.

I love that. Do your results in math, do they just fully generalize to you're just going to be better at scientific reasoning, you're going to be better at general reasoning. Does being great at competition math make you be great at everything else?

I think how we approached this was not like we should be like great at competition math, but really I think it's like we were focused on developing general purpose techniques to make reinforcement learning better. And I think those. We are very excited to improve our models in other domains beyond math and hopefully make models more useful for us in everyday usage.

This is like a, it's a pretty late breaking result. It's. Honestly, it was a surprise even to people internally at OpenAI. And so the next step is to incorporate this more broadly into our models and improve the reasoning capabilities across the board. But it's going to take some time to go through that process and deploy it to the world. So I think it's going to come. But yeah, it'll just take a little bit more time.

Is it harder for these models to do the IMO or the Physics Olympiad?

I think definitely the Physics Olympiad, because the Physics Olympiad has, I think like an experimental section or you have to.

We need to solve robotics first.

I didn't realize that. Okay. I thought it was just on a piece of paper.

Yeah. So I think it's. The model will probably be good at the on the paper part, but yeah, I think there'll be a little bit of time before it can do the.

Experiments, not with like a world model. Okay, cool. Are you going to release this model for customers to play with? Roloff's son is a Math Olympiad kid and he's like, I want access to the Math Olympiad model. Will people be able to play with this?

So we want to make this accessible to mathematicians to use. We're still trying to figure out the exact details of how we make that happen. But I think it's really cool that we've developed this system that is incredibly good at math and it makes sense that we want to see what mathematicians can do with it. I've actually already been emailing with the Stanford professor, mathematics professor. He actually emailed me about a year ago before we announced 01 and he was like, hey, do you want to do a collaboration on solving hard math problems? And basically what I told him is I think we just got to advance general reasoning capabilities and eventually they're going to be able to help you with your hard math problems. And I think that's actually the most promising route to getting there. He was a little skeptical, but every model release, like every reasoning model release, he's like emailed me with a follow up and is like, can it solve this problem now? And I've been plugging them in and I don't know what the output is, but I like email it back to him and he says, like, yeah, that's wrong. And, and he emailed me like, a follow up this time with like, you know, the same problem, like, asking like, hey, can it solve it now? It still can't solve it, but at least this time it like, recognizes that it can't solve it. So I think that's like a big step. But we're curious to see if there's like, a lot of other problems out there that mathematicians want to challenge this model with and see if it can take them on.

Amazing. Congratulations to you all. I think this is a momentous result that the entire field has been waiting for for a very long time. And the fact that it was accomplished by a team of three people in a span of two months, it's extraordinary. Congratulations and thanks for joining us on training data.

Thank you.

Thanks for having us.