GPT 4 Can NOW Self Reflect Just Like Humans!

What's more powerful than a GPT4 model? A GPT4 model that can self reflect, learn from its mistakes and self correct. With this new paper, the ability of these models to learn is not restricted to their training data cut off. But now they can improve themselves over time and learn from their own mistakes. It's fascinating. And now it gives AI models human like reasoning abilities. Paper in question is called Reflection, an autonomous agent with dynamic memory and self reflection. So you might ask what is reflection and why it's significant? One of the biggest weakness of the current AI models are they're not able to learn from their mistakes. But with reflection it gives an agent with dynamic memory and self reflection capabilities the ability to enhance existing reasoning, trace and task specific action choice abilities in simple words. Now the model has the ability to memorize what action it has taken and then it can go back and look at those actions and correct its mistakes. Now the great thing about this approach is it's not really limited to ChatGPT or GPT4 models. It can work with any large language model and you don't really have to fine tune the model at all. It can work with existing models with just added reflection on top of it. So for example, in this case you have two large language models. The first one generate a sequence of tasks, evaluates its rewards and then you have the reflection model which simply based on these heuristics, look at the reward function and updates the action that needs to be taken by the original large language model. This added layer seems to give the models a huge boost in performance. So for example, the normal ChatGPT4 model, which can get a 67% accuracy on a code generation tasks, can go up to 88% accuracy with simply adding reflection on top of it. But these results are not part of the original paper. We're going to come back to them and see how they're different from the original paper. The original reflection paper give results on two different data sets and both of them are significant for its ability to reason. The first one is this hot pot QA dataset which is a data set for diverse explainable multi hop question answering. Now in this case the language model has to go through multiple documents in order to find an answer. So it's not just an example of simple information retrieval, but it has to reason through multiple documents. The second data set is ELF World Aligning Text and Embodied Environments for Interactive Learning. Now I found this one a lot more interesting because they're combining text inputs and outputs with the physical world. So the model is able to interact with the physical world using these text prompts. Now, just by adding reflection, the model was able to do great on both these data sets. So for example, in the first case on the Alpha world data set, without reflection, the performance is around 70%, but with a reflection added it goes to almost 97%. And keep in mind we are not fine tuning the model again, so it's the same base model. Similarly, on the Hotpot QA dataset, the performance boost is around 20%, which is very significant. Before showing you reflection in action with an example, let's look at what people are getting wrong about this paper. A lot of people are thinking that this paper uses GPT4, but in reality this paper is using GPT3 and 3.5, which is ChatGPT. And that's why the this block false is significant because it added some more results which we are going to look at. I want to show you how it works with an example and then we'll talk about how you can tie this with Auto GPT which is going to be very significant. I have a video on that if you haven't watched it, so watch that after this. So this example task is based on the Hotpot QA dataset. The question at hand is Grown Ups starred the actor who was best known for his role on Holo olo. So now the model divides this into different tasks and comes up with the first thought experiment. This is what the model is thinking. I need to search Grown Ups in Olo olo, find the actor who starred in the Grown Ups, then find the role he was best known for in olo. So initially in the initial query we want it to find out the name of the character. Now it goes through different searches and observations and it couldn't really figure out the name because the initial starting assumption is wrong. Then it reflects I searched the wrong title for the show Olo olo, which resulted in no results. I should have searched for the show's main character, that's Gordon Cain, to find the role he was best known for in the show. And with this reflection it changes strategy. So now in the second case it comes up with this thought process. I need to search the actor who was best known for the role on Olo olo, and find out what role they were best known for. And using this new strategy, it comes up with the right answer. This is exactly how a human being will approach a problem if they made mistake in the first place. So we as a human would think what we did wrong and try to correct that in the previous video I showed you AutoGPT where you give it an objective and it comes up with your its own task and execute them. But now if you are able to add this reflection on top of those tasks, then it will be able to modify the task on the fly. So rather than coming up with the task in the stand and just execute them sequentially and at the end it says that oh, it couldn't figure out anything. Now it will be able to execute a task, see what mistake it made, then update the task and re execute it. That is true intelligence. This paper is very significant. However, there is one major limitation to this paper. This paper needs ground truth to work. However, several real world situations don't have a definitive ground truth or a single optimum location or a single optimum solution. And that's where this blog post comes in. In the blog post they have this section reflection without definitive ground truth. So this is significant. And don't worry, I'm going to break it down if it doesn't make sense. So to address such situation they are talking about when there is no ground truth or optimal ground truth available, we propose a method that again mirrors human problem solving. When given a task whose solutions are not clearly defined, we usually take time to plan and create an internal test suit based on our construction understanding, either consciously or unconsciously. So think about the escape room example. In that case, you will create a plan of what objects you're going to look at, right? So we evaluate various potential solutions against these tests and assign a confidence level to each. Adjustments are made until a solution that is likely to satisfy all or most of the tests will, which then becomes the proposed solution to be executed. So in this scenario, the solution that meets all or most internal test cases is accepted as the one likely to result in the ground truth. And the chance of success relies on the probability of an erroneous test design. Now why it's significant, they're actually not trying to solve the problem, rather they come up with their own test. So the model will actually come up with its own test and try to satisfy or pass most of the test. So it's indirectly solving the problem, not direct. Here is a visual representation of how this would look like. So you have a task, the model generates different tests and then see whether it can pass those tests or not. Like if it's able to pass those tests, that becomes a proposed solution. And if the test design is good enough, then that will be close to the ground truth solution. Now the takeaway is that we don't care about accuracy bottleneck anymore. So we are not trying to increase the accuracy. So we are shifting the accuracy bottleneck from correct synthetic and semantic code generation to correct synthetic and semantic test generation. So in theory, the test generation should be much easier to accomplish than code generation. In order to explain this, let's look at an example. So if you are a software development usually use follow these steps in test driven development. So first, as a human or as a software engineer, you receive a description of the technical problem and list of goals. So those are your requirements or design documents, right? Then the human designs a number of unit tests that capture the ideal behavior of the program. You come up with unit tests, then you write code for the actual problem to solve, right? And then you run the unit test against your code, right? And if it passes most of these, then you say that okay, it's good enough, you send it to production or whatever code base you're working with. So the authors of the paper actually tried this code generation data set. And following the same example of a software developer approach, they asked the AI agent to solve some programming tasks based on just the description of the task on this coding generation task, this new model with reflection. So the GPT4 model with reflection blew every previous results out of the water. So GPT4 model without reflection gives you around 67% accuracy. And this one reflection plus GPT4 gives you around 88% accuracy. I have seen somebody reporting it goes up to 91% accuracy as well, and that is very significant. Now how does it do it? So let's look at how the AI agent was implemented. So the approach that they adopted is listed here on this human evolved data set. In summary, all you have to do is simply provide it with the functional details and requirements. Then it will come up with a few unit tests based on the requirements that it receives. It will implement the code, test it against those unit tests and see if it satisfies those or not. If it's not, and it will go back, redo the functional implementation, and it will repeat the process. And based on the functional requirements, it's actually a lot easier to simply come up with these unit tests rather than the whole code correctly in a single flow in the near future, say a couple of months, rather than defining step by step code generation programs, now you will be able to simply give it a whole bunch of instructions in a one go and then let the program iteratively improve its coding and give you the results. So this is going to be really fascinating to see in action. I want to close it. With this thought, we as a human needs to reflect on what we are developing and what direction we want to take it. Thanks for watching. See you in the next one.