OpenAi's New Q* (Qstar) Breakthrough Explained For Beginners (GPT- 5)

So this video will get into the exact specifics of how Q learning works, and it's going to try and break it down in the easiest way possible so you can gain an understanding of why OpenAI's potential breakthrough could be the next evolution in large language models and AI models. So let's waste no time and jump right in. So what is Q learning? And one of the main things that we want to talk about is where does the name Q come from? So the name Q likely comes from two sources. Okay, so firstly, the Q could be a reference to the Q learning, which we will discuss later. And essentially it's a type of machine learning used in reinforcement learning. Okay, so that's where the Q is from. I'm guessing they're trying to merge this. And I'm going to talk about the second part. So the second part. So for the second part, essentially the start comes from the A star search. There was a research paper, I think, written in 2019, and. And the A search algorithm is a path finding and graph travel. And the second part is essentially the star. So the star comes from the A search. And the A search algorithm is a path finding and graph traversal algorithm, which is widely used in computer science for a variety of problems, especially in games and AI for finding the shortest path between two points. Okay, so I'm going to do that again, but I'm going to show you guys in more simpler terms how exactly that works. So essentially a simpler definition of Q learning is essentially you can think of the name Q like a nickname for a super smart robot. And then the cue part is basically like saying this robot is really good at making decisions and that it learns from his experiences, just like you would learn if you played a video game a bunch of time. And of course, the more you play, the better it gets at figuring out how to win. Then of course, we have the simpler definition for a star search, and essentially you just need to think of it like this. So imagine you're in a maze and you need to find the quickest way out. There's a classic method in computer science, kind of like a set of instructions that, that help you find the shortest paths in a maze. And that is exactly what we call a star search. And of course, once you mix this with deep learning, and then you get the computers to learn and improve from the experience, you get a really, really smart system. And it's not just finding the shortest path in the maze. It can solve much more trickier problems by finding the best solutions, just like how you might Figure out the best way to beat a video game. So now we're going to look at six steps to actually understanding Q learning. Because there are six key parts, and they're really simple once they're broken down into these parts. And overall Q learning, before we get into the six parts, it's basically like training a pet. If the pet does something good, like sitting on command, you give it a treat. And if it does something not so good, like chewing on your shoes, you say no or ignore it. So that's how the basic of this reinforcement learning actually does work. You reward them for the good decisions, and then you penalize them for the bad decisions. So step one in Q learning is the environment and the agent. In Q learning, you have an environment like a video game or potentially like a maze, and an agent and the AI or computer program that needs to learn how to navigate this environment. So that's just the basis. We have the agent, and then we have the environment that the agent is going to be in. Then, of course, we have the states and actions. So with the states and actions, this is where we have the environment. It's going to be made up of different states and different actions that the agent can take. So essentially, the agent may be able to move left or right, and of course, the different positions that they can take on the board or in said game, which is fairly simple to understand. And of course, we have something called the Q table. So the Q table is basically like the big cheat sheet that tells the agent what action is best to take in each state. And at first, this table is filled with guesses because the agent doesn't know the environment yet. So, of course, this isn't going to have all the correct data because it doesn't have the right movements, because it hasn't done it yet. Then, of course, we have step three, four, which is learning. By doing so, the agent starts to explore the environment. And every time it takes an action in a state, it gets feedback from the environment. You get rewarded for the positive points, and you get penalties for the negative points. So this feedback loop helps the agent update the Q table, essentially learning from the experience. So it goes out, it tries to figure out which way it's going to go. And of course, it updates that. And of course, that's what we have at step five, which is where you update the Q table. So the Q table is going to be updated using a formula that considers the current award and also the potential future awards. Make sure you pay attention to this part, because the potential future awards Is of course one of the key things that separates Q learning from many of the others. Okay, so that this way the agent doesn't just learn to maximize the immediate rewards, but also to consider the long term consequences of its actions. Because think about it like this, if you had an AI system which didn't think about long term rewards, every time it got a reward for doing something good, it would just keep doing that same good thing over and over again and it would just kind of be like this, you know, spiral that wouldn't lead you to future and long term better goals. So that's why this algorithm is really, really cool, because it has long term consequences planned into it. Then of course we have number six, which is over time, with enough exploration and learning, the Q table gets more and more accurate. The agent becomes better at predicting which actions will yield the highest rewards in different states and eventually it can navigate the environment very, very effectively. Which is why we have this image of an AI that is pretty much a God and is able to do it in the fastest way possible. So overall you can think of Q learning like playing a complex video game where over time you learn the best moves and strategies to get the highest score. Initially you're not going to know the best actions to take, but, but as you play more and more, you can learn from your experience and get better at the game. That's what this AI is doing with Q learning. It's learning from experiences to make the best decisions in different scenarios. Then of course, we do have the most likely future of LLMs. Because one thing that I did want to add was that LLMs do have current limitations. And that's why I do believe that Q is currently being explored as a viable option for the future of large language models. So, so please watch this clip from Someone at Google DeepMind who talks about how LLMs have these limitations and why these kinds of styles that we're starting to implement and starting to look in are going to be the future of large language models.

These foundation models are world models of a kind. And to do really creative problem solving, you need to start searching. So if I think about something like AlphaGo in the move 37, famous move 37, where did that come from? Did that come from all its data that it's seen of human games or something like that? No, it didn't. It came from it identifying a move as being quite unlikely but possible, and then via a process of search coming to understand that that was actually a very, very good move. So you need to get real creativity, you need to search through spaces of possibilities and find these sort of hidden gems. That's what creativity is. I think current language models, they don't really do that kind of a thing. They really are mimicking the data. They are mimicking all the human ingenuity and everything which they have seen from all this data that's coming from the Internet that's originally derived from humans. If you want a system that can go truly beyond that and not just generalize in novel ways. So these models can blend things. They can do Harry Potter in the style of a Kanye west rap or something, even though it's never happened, they can blend things together. But to do something that's truly creative, that is not just a blending of existing things, that requires searching through a space of possibilities and finding these hidden gems that, that are. That are sort of hidden away in there somewhere, and that requires search. So I don't think we'll see systems that truly step beyond their training data until we have powerful search in the process.

So in this part of the video, I do want to talk about some of the limitations of large language models, because there are quite a few. So one of the biggest things that you didn't know about LLMs is that. And we're going to get into the benefits of Q learning and why Q learning and how it compares to LLMs. And one of the biggest things is, of course, the data dependency. So traditional LLMs require massive amounts of data for training. They learn from examples in this data, which means their knowledge and abilities are limited to what's present in the training set. There was even a recent paper, I can't find it. If I do find it, I will leave a link in the description because it's going to be one entire article on the website. And essentially in that paper, they talk about how large language models cannot generalize on training, cannot generalize on information that they haven't seen in their training data, which basically just means that these large language models are only good as their training data. And essentially we've explored this concept before with Microsoft's Phione. Essentially it was a very, very small model and it was able to do coding much better than some of the large language models. And it was trained on only specific coding stuff and it was able to excel at that. And basically what this means is that if you don't have good data, your LLM is going to do horrible, but if you have good data, it's going to do really good. But of course, that comes with some other limitations. Okay. Of course we have Static knowledge. Okay, so static knowledge is once trained, LLMs have a fixed knowledge base, so they can't learn or update their knowledge after training, which means they can become outdated as the world changes. So of course you can see here knowledge cutoff September 2023. And that means that currently it can't get any more data because we're now in November. Not sure when you're watching this, but. But if OpenAI doesn't decide to update it, it means you're going to be stuck without that new update. So static knowledge isn't entirely great because as you know, things change day by day, every second, every minute. The world is changing. And if these AI algorithms are going to be really good, they need to be able to adapt rapidly to that changing world. So that's why traditional LLMs, this is of course a bottleneck limitation. Then we have context understanding. While they're good at understanding and generating human like text, they sometimes struggle with understanding the deeper context or intent behind a query, especially if it's complex or very specific. And that is something that happens when you're dealing with LLMs. In addition, we do have bias and fairness, which is something that is really, really prevalent in AI. And essentially the problem is, is that the bottleneck is the data. So when you have data on an LLM and you train it on that specific dataset, it's going to be geared to that data set. So, for example, if you train it on data that you only shows it a certain type of car, and every time it's seen that car, it thinks the car is orange because it's only ever seen that car in an orange color. It's going to be really hard to get that AI model to think of the car in any other color. So just think of that in that kind of bias. So the AI systems can have two kinds of biases, the cognitive biases and the lack of complete data. So if data isn't complete, it's not going to be representative. So like I said, if you don't have all the colors of the car, then it's not going to be representative of all the colors that it could potentially have. And of course there are cognitive biases, which are things that could seep into the machine learning algorithms via the designers unknowingly introducing them to the model or a training data set which introduces those biases. So bias is something that is really hard, but that, you know, it's really a big problem and it's something that people are trying to solve by making LLMs as unbiased. As possible, but it's not something that is easy to solve. And of course we have the lack of adaptation which I've already discussed. And of course now we need to get into the pros of Q learning or Q which could be GPT5. So of course we have dynamic learning. So Q learning can continuously learn and adapt based on new data or interactions. That means it can update its knowledge and strategies over time, staying more and more relevant, which is of course what we talked about before, something that we're going to need to do. Then of course we have the optimization of decisions. Learning is always about finding the best decisions to achieve a goal which can lead to more effective and efficient decision making processes in various applications. And with Q learning that's clearly what it's going to be able to do over time. Then of course this is the main thing about Q learning is the fact that we have specific goal achievement and Q learning models are goal orientated, making them suitable for tasks where a clear objective needs to be achieved, unlike the general purpose of traditional LLMs. So essentially the reason that this is going to be really good is because when you apply this to other things that require goals. So for example, maybe we could apply it to self driving, maybe we could apply it to AI agents on computers that are actually going to be able to have a complete end goal. Maybe the end goal is going to be a video, maybe the end goal is going to be an entire art school, maybe the end goal is going to be building an entire business. That's where we have the specific goal achievement, that's where you get that next leap up in ability in AI systems, which is why this could be really, really next level. And of course we have something about the systems in which companies are already on this. So on 28 June 2023, Demis Hassabis says that the company is working on a system called Gemini. If you haven't heard of Gemini before, it's Google's next huge large language model air company that is going predicted to be beating GPT4 across all benchmarks and it's going to be using a method called tree search which is going to be able to explore and remember possible scenarios which is quite similar to Q learning. So they're moving away from the standard methods and they're now trying to think about advanced techniques where they can essentially explore and remember multiple different things. Now if you found that a bit confusing, you should take a look at AlphaGo and how it's going to impact the future of AI. Because AlphaGo was essentially something that researchers thought that they couldn't predict because essentially with AI. What the problem was was that the moves on AlphaGo were essentially uncomputable. They couldn't just remember every single move, they had to get the system to think. And essentially the problem was, was that there are more AlphaGo moves than there are, I think, atoms in the universe or grains of sand on the beach. It's something absolutely crazy when you look at the statistics. So this was something that researchers thought that they were never going to solve, but of course the AI managed to solve it. So I would say take a look at the quick trailer which I'm going to show you guys. I don't want to spoil it for you. It is honestly riveting to see this kind of content, but it was something that happened a while back that people do forget if they aren't particularly plugged.

Into the AI space standing challenge of artificial intelligence.

Everything we've ever tried in AI just falls over when you try the game of Go.

The number of possible configurations of the board is more than the number of atoms in the universe.

AlphaGo found a way to learn how to play Go.

So far, AlphaGo has beaten every challenge we've given it, but we won't know its true strength until we play somebody who is at the top of the world likely to die.

Then of course we had the significance of move 37 was, I think it was a 1 in a 10,000 move that nobody expected from an AI where it seemed to exhibit some creativity and many people weren't expecting this. So there's also multiple videos where they talk about move 37, which was something that of course we didn't expect. And of course this brings us to this point. Google delays the release of Gemini AI to Q1 of 2024. So this might be in response to the fact that it might be harder than they think. Maybe they're changing their angle, maybe they just want to perf it. Currently, we don't know what the reason is for them delaying this model. But what we do know is that Gemini is going to be currently delayed. And if this model does come out and it does possess these capabilities, it will be interesting to see how it compares to GPT4 and if it's going to be similar to Q Learning or how different it's going to be. Of course we have one of the main questions and that is, will it be in GPT5? Many sources have already shown us that Sam Altman has already started training the next level in LLMs or AI systems. So will GPT5 contain this cute star, or is it just going to be something that is in Future models like GPT6? Either way, it's going to be interesting to see how this entire thing pans out. And if this video did help you understand, don't forget to leave a like subscribe all that good stuff. Check out the full article in the comment section below.