A Two Minute Papers YouTube episode by Dr. Zsolt Zsolnai (LARRUZ channel) reviewing a new Google/DeepMind AI model and testing whether it performs at human levels on several subjects. The presenter highlights five notable properties: (1) reliable math solving including diagram questions, (2) strong performance on physics, (3) broad generality across biology, engineering, chemistry, astronomy and ML, (4) substantially better performance with larger models, and (5) high scores on real-world exams (e.g., 2022 Poland national math exam) and some MIT undergraduate problems. The video explains the approach (generate many candidate solutions and vote) and notes remaining failure cases; it includes a sponsor segment by Weights & Biases.
Questions this video answers
August 26, 2022.
2022 Poland national math exam.
Weights & Biases.
Chapters
Dear fellow scholars, and this is Two Minute Papers with Dr. Zsolnai. Today we are going to have a look at Google's new AI and find out whether it is as smart as a human, at least in some test subjects. Not so long ago DeepMind published an AI technique and its knowledge in the area of humanities, social sciences and medicine was fantastic. But at mathematics of all things, not so much. Interestingly, it also thought that it is a genius and then amusingly, it got quite confused by very simple questions. Look at that. Do geniuses mess up this multiplication? I think not. And that is a bit of a problem. After all, if this genius or non genius AIs are too unreliable, we ask them something and might have no idea whether their solutions are correct or not. So our question today is can we improve the consistency of these AIs? Well, let's have a look at Google's new technique and put it to the real test. And I mean it. Yes, it is going to write a math test and so much more today and we will see how it performs. Is it as smart as a human? Can that really be? We will find out today together. And it turns out it has five amazing properties that I found really surprising. One, it can solve math problems reliably and sometimes even in surprising ways. Look here, it solved the problem in a way that is not only correct, but is different from the typical reference solution. And the surprises don't stop there. This one can even deal with questions that include drawings, which is stunning. Two, A proper AI today has to be relatively general and this means that it shouldn't only be good at math, it should be good at other subjects too. And its solutions to these physics questions are are not only correct, but clear, crisp and beautiful at the same time. I absolutely love it. So yes, it can do not only math, but physics too. But three, I hope you know what's coming. Oh yes, Physics and math are not the only two subjects it can deal with. The generality does not stop there. This AI is a true polymath as we can ask it about excretory organs in biology or even electrical engineering, chemistry, astronomy, and hold on to your papers because it even knows some machine learning. Yes, a machine that learns about itself. How cool is that? What a time to be alive. Four, it significantly outperforms previous techniques, provided that we use the larger AI model. Here is a beautiful example where we can prompt it with a question and the smaller model fails. I wonder if the larger, more capable version of it would be able to get this right? Does it?
Yes.
So good. And 5. Here you see an example question from the 2022 Poland's National Math exam. And now we are finally at the point where we can ask the question, is it as smart as a human? Well, if you have been holding onto your paper so far, now squeeze that paper because the answer is yes, kind of. It not only solved this question correctly, but adding everything up, it scored above the national average. What's more, it can even solve some undergrad math problems straight from mit. So yes, at least on this particular test, it is at the very least as smart as a human. I am truly out of words. This means that the consistency of these solutions is finally at a point where we could use this as an AI science assistant. A junior assistant who messes up from time to time, mind you, but an assistant nonetheless. What a time to be alive. Interestingly, if we look under the hood, we see that it works by generating many candidate solutions and starts a vote between these solutions and the ones with the most votes end up being the solutions that you've seen throughout this video. Now, not even this technique is perfect. You see some failure cases here, but once again, this is so much improvement. Just one paper down the line. I wonder what happens if we apply the first law of papers which says that research is a process. Do not look at where we are, look at where we will be. Two more papers down the line. So what do you think these will be capable of? One more paper down the line? Does this get your mind going? Let me know in the comments below.
This video has been supported by Weights and Biases. Check out their recent offering Fully Connected, a place where they bring machine learning practitioners together to share and discuss their ideas, learn from industry leaders and even collaborate on projects together. You see, I get messages from you fellow scholars telling me that you have been inspired by the series but don't really know where to start. And here it is. Fully Connected is a great way to learn about the fundamentals, how to reproduce experiments, get your papers accepted to a conference and more. Make sure to visit them through WNB Me papers or just click the link in the video description. Our thanks to Weights and Biases for their long standing support and for helping us make better videos for you.
Thanks for watching and for your generous support and I'll see you next time.
