A Two Minute Papers video (Mar 3, 2022) by Dr. Zsolnai reviewing recent OpenAI-related work (Oct 2021–Feb 2022) on using large language models to solve math problems. The host contrasts GPT-3 (175B parameters) performance on grade-school puzzles (~35% correct under some setups) with follow-up work that fine-tunes Codex and uses automated additional guidance to solve MIT introductory exam questions (near-perfect on selected items) and generate new problems. He also notes OpenAI progress toward high-school Olympiad problems (≈30–40% solved), emphasizes limitations/trade-offs, and invites viewer discussion.
Chapters
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today we are going to have a little taste of how smart an AI can be these days. And it turns out these new AIs are not only smart enough to solve some grade school math problems, but get this, a new development can perhaps even take a crack at university level problems. Is that even possible or is this science fiction? Well the answer is yes, it is possible, kind of. So why kind of? Well, let me try to explain. This is OpenAI's work from October 2021. The goal is to have their AI understand these questions, understand the mathematics and reason about a possible solution. 4 grade school problems alright, so this means that the GPT3AI might be suitable for the substrate of the solution. What is that? GPT3 is a technique that can understand text, try to finish your sentences, even build websites and more. So can it even deal with these test questions? Lets see it together. Hold on to your papers because in goes a grade school level question, a little math brain teaser if you will and out comes. My goodness, is that right? Here out comes not only the correct solution to the question, but even the thought process that led to this solution. Imagine someone claiming that they had developed an AI that this capable. 10 years ago this person would have been logged into an asylum. And now it is all there, right in front of our eyes. Absolutely amazing. Okay, but how amazing? Well, it can't get everything right all the time, not even close. If we do everything right, we can expect it to be correct about 35% of the time. Not perfect, not even close. But it is an amazing step forward. So what is the key here? Well yes, you guessed it right. The usual suspects. A big neural network and lots of training data. The key numbers are 175 billion model parameters and it needs to read a few thousand problems and their solutions as training samples. That is a big rocket and lots of rocket fuel if you will. But this is nothing compared to what is to come. Now believe it or not, here is a follow up paper from just a few months later, January 2022 that claims to do something even better. This is not from OpenAI, but it piggybacks on OpenAI technology as you will see in a moment. And this work promises that it can solve university level problems. And when I saw this in the paper I really now grade school materials. Ok, that is a great leap forward. But solving university level math exams, that's where the gloves come off. I am really curious to see what this can do. Let's have a look together hmm. Some of these brain teasers smell very much like MIT to me. Surprisingly short and elegant questions that often seem much easier than they are. However, all of these require a solid understanding of fundamentals and sometimes even a hint of creativity. Let's see. Yes, that is indeed right. These are MIT introductory course questions. I love it. So can it answer them? Now, if you have been holding onto your paper so far, squeeze that paper and let's see the results together. My goodness, these are all correct flying colors. Perfect accuracy at least on these questions. This is swift progress in just a few months. Absolutely amazing. So how is this black magic done? Yes, I know that's what you're waiting for, so let's pop the hood and look inside together. Alright. Two key differences from OpenAI's GPT3 based solution. One, it gets additional guidance. For instance, it is told what topic we are talking about, what code library to reach out for, and what is the definition of mathematical concepts. For instance, what is a singular value decomposition? I would argue that this is not too bad. Students typically get taught these things before the exam too. In my opinion. The key is that this additional guidance is done in an automated manner. The more automated, the better. The substrate here is not GPT3, at least not directly, but codecs. Codex is OpenAI's GPT language model that was fine tuned to be excellent at one thing and that is writing computer programs or finishing your code. And as we've seen in a previous episode, it really is excellent. For instance, it can not only be asked to explain a piece of code in, even if it is written in assembly, or create a pong game in 30 seconds, but we can also give it plain text descriptions about a space game and it will write it. Codex is super powerful and now it can be used to solve previously unseen university level math problems. Now that is really something. And it can even generate a bunch of new questions. And these are bona fide real questions, not just exchanging the numbers. The new questions often require completely different insights to solve these problems. A little creativity, I see. Well done little AI. So how good are these? Well, according to human evaluators, they are almost as good as the ones written by other humans. And thus this can even be used to provide more and more training data for such an AI. More fuel for that rocket and good kind of fuel. Excellent. And it doesn't end there. In the Meantime, as of February 2022, scientists at OpenAI are already working on a follow up paper that solves no less than high school mathematical Olympiad problems. These problems require a solid understanding of fundamentals, proper reasoning, and often even that is not enough. Many of these tasks put up a seemingly impenetrable wall, and climbing the wall typically requires a real creative spark. Yes, this means that these can get quite tough and their new method is doing really well at these Once again, not perfect, not even close, but it can solve about 30 to 40% of these tasks and that is a remarkable hit rate. Now we see that all of these works are amazing and they all have their own trade offs, they are good and bad at different things and have different requirements and most of all they all have their own limitations. Thus none of these works should be thought of as an AI that just automatically does human level math. What we see now is that there is swift progress in this area and amazing new papers are popping up. Not every year, but pretty much every month. And this is an excellent place to apply the first law of papers which says that research is a process. Do not look at where we are, look at where we will be. Two more papers down the line. So what would you use this for? Please let me know in the comments below. I'd love to hear your ideas. And also if you are excited by this kind of incredible progress in AI research, make sure to subscribe and hit the bell icon to not miss it when we cover these amazing new papers. This video has been supported by weights and biases. Look at this. They have a great community forum that aims to make you the best machine learning engineer you can be. You see, I always get messages from you fellow scholars telling me that you have been inspired by the series but don't really know where to start. And here it is. In this forum you can share your projects, ask for advice, look for collaborators and more. Make sure to visit wnb Me paperforum and say hi or just click the link in the video description. Our thanks to Weights and Biases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
