DeepMind's AlphaProof and AlphaGeometry systems trained on 100 million examples solved geometry problems that stumped ChatGPT entirely. AlphaGeometry v2 solved 83% of historical Olympiad geometry problems, some in seconds, reaching silver medal level performance.
0 of 30 problems.
53%.
83%.
19 seconds.
Equivalent of a silver medal (one point shy of gold).
Not so long ago we talked about AIs competing on mathematical Olympiad problems made for people and found something really surprising. ChatGPT like AIs are generally really good at so many things. They can ace the bar exam, get hired to Amazon for their coding skills. They can also solve pretty hard math problems. So a Mathematical Olympiad should be easy, right? Well, when they tried ChatGPT on 30 of these problems, it could solve, well, exactly zero of them. Wow, that is unexpected. Why is that? There is something about these problems that this generation of AI techniques just cannot grasp. And the problem is that they would have to perform not just calculations, but reasoning logical deductions over many, many steps on hard, novel problems. This requires not just a huge glorified calculator for adding numbers, but it requires planning your solution, logic and reasoning. And Google DeepMind released two new techniques that are hopefully better in this area. So how good are they? Can they solve any of these problems? Or can only humans do that? Or maybe are they as good as humans? Dear fellow scholars, this is two minute papers with Dr. Zsolnai. So Google DeepMind showcased their alpha proof and alpha geometry, two techniques. Here you see alpha proof at work and goodness, just look at it go. At the Mathematical Olympiad, humans are given two sessions of four and a half hours each to do all the proofs for a total of six problems. So here's how it went down with the AIs. They got a little help, a formalized mathematical language. And now hold on to your papers fellow scholars, because after this it was able to solve most of these difficult problems, some of them in minutes. The rest took a great deal longer, three days. It was even able to solve the hardest problem that could only be solved by five of the best contestants. And these fellow scholars are all incredibly good. Some of these humans later go on to win the Fields Medal, which is almost like a Nobel Prize in mathematics if there was one. Wow. So how good is it? What is the final score? An equivalent of a silver medal, just one point half of the gold among some of the best mathematicians on the planet. Goodness. Don't forget we started out with a big zero for ChatGPT. And now silver medal. I always say just imagine what we will be capable of. Two more papers down the line and we are exactly two more papers down the line and this happens. What a time to be alive. I am loving that we can witness this in our lifetimes. I also really liked how a gold medalist said that he was impressed by how it can come up with really non obvious solutions. I mean, just look at that. Alpha geometry 2 can take this super challenging problem and solve it in 19 seconds. We can't even look through the calculations that quickly. And it gets better. When given all historical geometry problems at the olympiad, alpha geometry one could solve 53% of these problems. Now second version 83% alpha geometry works by looking at millions of problems and learning how they are solved. And then when given a new problem, it generates a bunch of new ideas and starts developing them to see which one of them has promise. And these results are absolute insanity. And I ask you to once again just imagine what we will be capable of just two more papers down the line. They are already working on a version that doesn't require this help and who knows what else is in the works. Isn't this super exciting? So what do you think? Let me know in the comments below. We need new tools for the era of LLMs and weights and biases. Now has weave a lightweight toolkit to confidently iterate on LLM applications, use traces to debug how data flows through each step of your app and use evaluations to measure your progress. It is the best. Try it out now at WNB Me papersllm or click the link in the description below.
