Googles ALPHACODE-2 Just SHOCKED The ENTIRE INDUSTRY! Full Breakdown + Technical Report

A quick announcement. I did recently make a Patreon and that's for those of you who want to just give some extra support because sometimes the one man operation can be a bit daunting. So if you do want to join the discord, join the Patreon, get featured in some videos. For your comments, vote on what videos I should do next. Don't forget to check out the link in the description, otherwise, enjoy the rest of the video. So AlphaCode 2 by Gemini this was something that I wanted to cover in yesterday's video, but I'm sure it deserved its own video because it was a bombshell that was dropped that unfortunately went under the radar and kind of slipped by in terms of people missing this crazy announcement. So what we have here guys, is a very impressive coding system and this is built on top of a fine tuned version of gemini Pro. So AlphaCode 2 is essentially built on a fine tuned version of Gemini Pro. And what you need to understand about AlphaCode 2 is that AlphaCode 2 is an entirely separate system and currently isn't available in Bard. So as you know, many people are using Bard and looking at their advanced coding capabilities. But please understand that this is not AlphaCode2 as AlphaCode2 has not yet been released and they are working on doing that. So even if right now you do think Bard's coding capabilities have increased, please understand that you're not seeing the capabilities of AlphaCode 2 as it is a separate system. So I want to quickly dive into the Technical Reports first page, then I want to show you the actual demo from Google and then I'm going to go into some further details about AlphaCode 2 that I think you should know. The first paragraph of the AlphaCode 2 technical report actually houses some very fascinating stuff, so it actually does mention the fact that their previous version of AlphaCode, if you didn't know, AlphaCode was essentially the first AI system to perform at the level of the median competitor in competitive programming, a difficult reasoning task involving advanced maths, logic and computer science. And essentially what they've introduced is their second iteration of this model. And essentially what they've introduced was AlphaCode 2, their second iteration of this model. And it's a new and enhanced system with massively improved performance. Powered by Gemini, AlphaGo 2 relies on the combination of powerful language models and a bespoke search and re ranking mechanism, which is really cool and we're going to get into that later. And when evaluated on the same platform as the original Alpha code, We found that AlphaCode 2 solved 1.7 times more problems and performed better than 85% of competition participants. So let's take a look at that Google demo and then I'm going to go back and do a deep dive into the paper because there's some really cool things that you should know about AlphaCode 2 in terms of where Google is going to take the system.

We built Gemini from the ground up to be natively multimodal, including something quite important for both of us, programming code. Gemini is able to consistently understand, explain and generate code that is correct and well written in most programming languages. That includes Python, Java, C and Go. It substantially improves coding abilities over previous Palm 2 models. From a benchmark around 200 programming functions in Python, it consistently solves about 75% of them in the first try, versus around 45% on Palm 2. If you allow Gemini to check and repair its own answers, this number jumps to over 90%, which is a huge step forward. It can help you create and prototype new ideas in seconds. Let's give it a try. I really like trains, and if I wanted to create a trains spotting location web app, I can simply ask and get a working prototype in less than a minute. While the code isn't perfect, it's really helpful to have a first draft. Gemini on its own has the ability to transform software development as we understand it, but it can also be deployed as a key component of more sophisticated systems.

Gemini is great at coding, but we've been able to take it even further, creating a specialized version that performs remarkably well at competitive programming. Now, why do we care about competitive programming? Well, it is one of the ultimate litmus tests of algorithmic coding abilities. So you have thousands of talented programmers from all over the world that come together to compete and try to solve incredibly complex problems that require not only coding, but also math and reasoning. Two years ago we presented AlphaCode and it was the first AI system that could compete roughly at the level of the average human competitor. Today, I'm delighted to introduce AlphaCode 2, a new and enhanced system with massively improved performance, powered by Gemini. When we evaluate AlphaCode 2 on the same platform as the original AlphaCode, we solve almost twice as many problems, while AlphaCode broke through the top half of human competitors. On average, we estimate that AlphaCode 2 performs better than 8.85percent of competition participants. Let's have a look at our model in action on one of the hardest problems that we faced. And I say hard because in the original contest in which the problem appeared, less than 0.2% of participants actually solved it. The problem is quite difficult, it's very abstract, so I can't get into too many details. But the basic gist of it is that we are tasked with computing aggregate statistics that account for what appears to be an impossibly large amount of random arrays. The really cool thing is that to solve it, AlphaCOD 2 makes use of dynamic programming. Dynamic programming is an advanced algorithmic technique which basically simplifies a complicated problem by breaking it down into easier subproblems again and again. And what's really impressive is, is that not only AlphaCode 2 knows how to properly implement this strategy, but also when and where to use it. What the example shows us is that competitive programming is not just about implementation, it's also about understanding maths, computer science, and indeed coding. And that makes it an extremely hard reasoning task. So it's not very surprising that up till now, generally available large language models have scored very poorly on this benchmark. These models are really, really good at following instructions. But AlphaCode needs to do more than that. It needs to show some level of understanding, some level of reasoning, designing of code solutions before it can actually get to the actual implementation to solve the problem. And it does all that on problems that it's never seen before. Another thing that is great about AlphaCode is that it performs even better when it collaborates with human coders who can provide grounding. Basically, developers can specify properties that the code samples have to obey, and when we do that, we see performance increase significantly. We think of this kind of interaction between programmers and AIs as the future of programming, where coders will not just give instructions, but actually collaborate with highly capable AI models that can reason about their problems, program that can propose code designs and that can even help with the actual implementation. AlphaCAD 2 was built for competitive programming, but we're already working on bringing some of its unique capabilities right into the general Gemini models as a first step towards making this new programming paradigm available for everyone.

So that was a very nice comprehensive overview of the AlphaCode 2 capabilities and I found it really fascinating how well, well it does on coding and just how much it seems to understand the task due to the fact that it is on a fine tuned version of Gemini. So one thing I also did find was the interesting overall system. So essentially what we have Here is the AlphaCode 2 system and it's quite different to how normal systems are done. So essentially AlphaCode 2 has a team of mini brains or model and, and each one is really good at coming up with different ways to solve the Coding problems. And essentially they all work together to create tons and tons of possible solutions. In effect, I'm pretty sure later on in the technical report they state that it is actually millions of possible solutions that they do create. Now, essentially, when they create many, many different of these solutions in this sampling mechanism, what they do next is they then essentially get rid of the bad ideas. And this is of course the filtering mechanism. Some of the solutions that they do come up with obviously won't work. So they have a special checker that throws out any ideas that don't actually solve the puzzle. Then of course we have a clustering grouping algorithm where they notice which ideas are kind of similar and to make things easier, they group the similar ideas together. This way they don't waste time on ideas that are almost the same. Then of course what they do is they essentially for each group of ideas, they then pick the very best one and then have a special way to score each idea to figure out which one is the most likely to solve each puzzle coding problem. And overall, what we have here is a very effective system. So the one thing I wanted to expand a bit upon was their sampling system, because I found out that this is where things do get interesting. So essentially, AlphaCodes 2 sampling, you can see that it says we generate up to a million code samples per per problem using a randomized temperature parameter for each sample to encourage diversity. So essentially they create 1 million different solutions for one single coding problem. And of course to make sure that each solution is different from the last, they use a special trick. So essentially they adjust the robot's imagination to come up with a wide range of ideas. And of course this is done by using something called a randomized test temperature parameter. So think of it like a turning dial that changes how creative or different each solution is. Then of course we have adding more details. So they add and change small details about each solution based on things like how hard the puzzle is or what kind of puzzle is, and that makes each solution even more unique. Then of course they have the fine tuned models or their brains essentially that are really good at coming up with the solutions and it uses all of them equally to get a good mix of ideas. So al alpha code used to think in both Python and C+ languages. It now only uses C+ because it makes better quality solutions in this language. Then of course we have the picking slash filtering. And because they have so many solutions, they have to pick the best ones. It looks at all the solutions, throws out the best ones, and then ranks them to find the top 10 and in the end it chooses up the best top 10 solutions to try and solve this problem problem. So it's a really, really interesting deep dive to see exactly how the model thinks its entire process and how it gets from problem to solution. Now with that, I did want to look at the future of AlphaCode because although the system was just released, it is interesting to see where this could go. Now, one of the problems that AlphaCode does suffer from, just like the other advanced AI systems like GPT4, it does suffer from cost to run. So it says our system requires a lot of trial and error and remains too costly to operate at scale. Further, it relies heavily on being able to filter out obviously bad code samples. So it seems that right now I don't think we're going to be getting alpha code because it's too expensive to run and too costly to operate at the scale that they would need to if they wanted to deploy it worldwide. And of course, they also do state that a lot more remains to be done before we see systems that can reliably reach the performance of the best human coders. So it seems like we are moving towards a future where coders aren't going to be replaced, but more so we're going to get coding assistance in the form of AI rather than complete replacements. So one thing I did really want to include was of course, the evaluation, because it's really important to see how it scores up compared to AlphaCode 1. So essentially, AlphaCode 2 was tested on CodeForces, the same platform that used the original AlphaCode. They chose 12 recent contests with over 8,000 participants from two difficulty levels, Division 2, the harder Division 1 and 2, covering 77 problems in total. So for each problem, AlphaCode 2 generated 1 million potential solutions which were their candidates. And it then submitted up to 10 solutions per problem for following a specific selection process. This continued until it found the correct solution or ran out of candidates. And in terms of the performance results, AlphaCode 2 solved 43% of these competition problems. This is almost double the performance of the original AlphaCode, which solved 25%. In terms of competition rankings, AlphaCode 2 is estimated to be in the 85th percentile, performing better than 85% of the participants in comparison with human competitors. It ranks between expert and and candidate master categories on codeforces. And the estimated ranking does take into account a simulated time penalty in efficiency problem solving. In some contestants, AlphaCode 2 performed better than 99.5% of the participants and it was found to be 10,000 times more sample efficient compared to the original alpha code and it requires about 100 samples to reach what AlphaCode achieved with a million samples. In addition, it does seem like AlphaCode 2 could get even better because in addition it does seem like AlphaCode could actually get quite better because they do use Gemini Pro as their foundation model and using this foundation model led to significant increases in performance on two critical components of the system, the policy models generating the code samples and the scoring model used to select them. So essentially they basically say that we suspect using Gemini Ultra as the foundation model instead with its improved coding and reasoning capabilities would lead to further improvements in the overall Alpha Code 2 approach. So now it's simply a waiting game to see whether or not Alpha Code two is going to be made more efficiently so that it can be scaled up and deployed globally or they're going to just simply improve on it like they did with AlphaGo type systems and see if they can potentially get 100% on some of these benchmarks.