DeepMind's AlphaGeometry was trained on 100 million synthetic math problems and solves geometry proofs by generating creative insights then doing mechanical calculations. It performs near average International Mathematical Olympiad level and is roughly 100 times smaller than GPT-4.
Questions this video answers
January 24, 2024.
Approximately 100,000,000 (one hundred million) synthetic mathematical problems.
The speaker states the model is about 100× smaller than GPT-4.
Yes — the speaker states the project is open source from day one.
Chapters
My goodness. Alpha Geometry. An amazing Google DeepMind paper fresh out of the oven. And I think this paper might be history in the making. So how is an AI that learned on a hundred million mathematical problems able to compete in the International Mathematical Olympiad, that is perhaps the most prestigious math competition in, in the world. So did they achieve a breakthrough? Well, let's see together. Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Now, competing on such problems is trouble. This requires not just a huge glorified calculator for adding numbers, but it requires planning, your solution, logic and reasoning, and more. But of course, GPT4 can do all of that. So just give the problems to it and off you go. End of the video. Right, well, just as a demonstration, GPT4 can ace the bar exam, would be hired to Amazon for its coding skills, and is better than almost all of humans on the Biological Olympiad. So it can sweep all of these. Easy. Right? Well, let's see. If you try to use GPT4 and give it 30 of these tasks, it can solve. Exactly. Wow. It can solve zero of them. I hope this gives you a feel of how difficult these problems are. Devilishly difficult. And scientists at DeepMind just proposed a system that is about a hundred times smaller. I can't believe it. But to start, how do you even solve such a problem? Let's see an easy example. For instance, if we ask a human mathematician to prove that there is an infinite number of primes, we could prove it by listing them. But no one can list an infinite amount of numbers. That is impossible. No. So what do we do instead? How do we do the impossible? Well, instead we start with the assumption that there is a finite number of primes, and then find a contradiction that says this assumption cannot possibly be true. This part of the proof requires thinking out of the box. It requires a moment of brilliance, and without it, the problem is intractable. This is akin to pulling a rabbit out of your hat. And when you have the rabbit, mechanically finishing the problem is relatively easy. And the strategy could be that if the rabbit didn't work, try to pick a new one. We can do that, but can an AI, can a machine pull a rabbit out of a hat? And the initial answer is no, mostly not, but let's try anyway. This is how a human would do it. And here is their proposed AI that could hopefully try to do it. So does it work? Well, let's have a look. When given a problem with blue, it first creates the key ideas, the rabbit. And then the green part is the remainder of the calculation that leads to the solution. But this was easy peasy. Now give me some proper problems. Oh my. Now we're talking. So little AI, can you solve this? And whoa. It pulls the blue rabbit out of the hat and then runs the green calculations until it solves it. And make no mistake, this is just an excerpt of the solution. Now hold on to your papers, fellow scholars, because the full solution looks more like this. My goodness, look at that. More than a hundred steps have been concealed here. And it had done all of this correctly. And that is not even the longest proof it is capable of writing. Not even close. Wow. So how good is it? Well, a previous technique was this good, and the new technique without the rabbit is this good. So it is almost as good as the average mathematical Olympiad contestant. Note that these are really smart people, so the average of those is also really smart and it can compete with that. But wait a second. This can run the mechanical calculations, but it only takes you so far. You also need the brilliance to pull that rabbit out of the hat. And as soon as we add the rabbit part of the solution, what happens? What are you seeing? What I am seeing? It is nearly as good as the smartest of these super smart people. And fellow scholars, if you think that is impressive, hold on to your papers because we are just getting started. Here are two mind blowing facts about the paper. One, it learned from scratch by itself, without any human demonstrations. Yes, the proposed system does all this without human intervention. This is essentially an AI implementation of the two modes of human thinking, and that is thinking fast and slow. Thinking fast is about quick instinctive responses like reading something, while thinking slow involves deliberate, logical and calculated decision making. This can do both as well as some of the smartest humans can do. So when we are worried that this cannot possibly get any better because there are no humans good enough to teach it anymore. Now we know that it only needs synthetic training data so it can learn by itself. And get this, it has already found more general, more elegant solutions for some of the tasks than humans did. Two, this project is open source from day one and every piece of the solution is out there for you for free. Yes, you fellow scholars can run your own experiments with it. And all this with a model that is about a hundred times smaller than GPT4. An absolute slam dunk of a paper. We are still early, but I think it might be fair to say that this is a breakthrough. As incredible as this AI is, note that it is still relatively narrow. Even it can do geometry, but it cannot play Starcraft or do anything else. However, the ideas and concepts described in the paper are general enough to make sure that this can be applied to other problem domains as well, and that, fellow scholars, is going to be a series of incredible breakthroughs. By the way, it is a possibility that I will visit San Francisco around mid April for the first time ever if this is the case. If you are a local lab like DeepMind and OpenAI and you would like me to visit, or if there is someone who I should really meet, please let me know on Twitter x Here you see a set of incredible courses on generative AI LLMs, all free for everyone. You can learn more about these things that we talk about here in the series, hear from other experts, follow the code they prepared for you, and get a certificate. Make sure to sign up at WNB Me papercourse or click the link in the description below.
