O1’s Chain of Thought: I Built a System to Mimic It—Here’s How It Went!

As you can see, in the first stage it made a mistake, but in the second step it was able to recover from the previous mistake. And the answer that it comes up with is correct. Is it possible to replicate Owen's behavior with other models using chain of thought and reflection? Over the last few hours I put together this code base which tries to do exactly that. I'll share some of my preliminary results in the approach that I'm using. Before showing you how this system works, let's look at some example outputs for the LLM. I am using Gemini Flash, but you can potentially replace it with any other model of your choice. I am using Flash for a very specific reason which I am going to explain later in the video. Now, before showing you how it works, let's look at a couple of example outputs. So we will start with the mandatory Strawberry test. This is a multi stage sequential system where the output from the first step is analyzed and critiqued by the agent in the second step and so on and so forth. So we can look at a quick output here before I explain how the whole system works. So here the first agent says, okay, I'm ready to solve this problem using chain of thought reasoning. So through prompting we enable chain of thought. Now here it comes up with a plan. So it's going to visually inspect the word strawberry and then it will count the number of times the letter R appears. Now as you can see, it makes a mistake. So it says the first R is in the second position, the second R is in the fifth position, and then there are no R's. So the first stage makes a mistake by saying there are only two R's. And even during its own reflection stage it doesn't find any issues. Now here's where the second stage is very important. So it analyzes the results from the first stage. It talks about some of the strengths of the first stage. This is not where we are interested. We are actually interested in the weaknesses of the first stage output. Now here you can see that it says while the solution is accurate, the counting process is ill described in somewhat informal manner and the solution could benefit from more systematic and programmatic approach. And the improvements that it suggest is introduce a formal counting method. Instead of simply describing the position of each R, consider iterating through the word and incrementing a counter each time R is encountered. Now in this case it comes up with a Python code that is going to be used to count the number of hours. And this is actually the reason that I'm using Gemini Flash, because the Gemini Flash API can not only come up with a Python code, but there is a code executioner behind the API and can actually execute or run the code that it creates and give us the output. You can do the same with other models as well by enabling function calling and executing the code yourself, the one generated by the model. But since Flash provides that ability, so I decided to use Gemini Flash. We're going to look at the detail of the actual approach later. But here's the final reflection on the solution that is provided. And at the end it comes up with the right approach or the right answer that there are three Rs. Okay, this was a very simple question, but let's ask a little bit more involved. So if you ask GPT4O which states have the letter A at third spot in their name. So here's the list that it comes up with at the, at the beginning it's Alaska, they're in Arkansas, Hawaii, Kansas. Right. And you can see that it's already making mistakes. Now, what happens if I use the same prompt? With the system that we have built Again, for this prompt, it decides that I'm going to use Python code because it's looking at the output of the fourth agent or fourth sequential step. So it simply writes a Python script for that. And by running that Python scripts, it's able to give us the correct answer, which A, which is these four states. So Alabama, Alaska, Idaho and Utah. And if you ask this, if you use the same prompt with O1, in this case, I'm using O1 mini because I ran out of the print review credits. It gives us the same answer in this case. Again, it goes through the list of all the names, then checking the third letter of each state, it comes up with the same answer that we have. Here is a problem that I found, I think on X, which is an interesting one. So write me a poem with 12 lines. The first letter of the odd lines should spell out Mehran. The first letter of the even lines should spell out Jalari. The first letter of the word in each line should spell out a country's name. Now, the O1 mini is not able to solve this. O1 preview might be able to solve it. However, GPT4O has no trouble coming up with a solution. Now, here's how our system performs. So here at the first stage, it comes up with this approach. It is like generate a list of countries. I will need a list of countries to choose from. So this is basically the first step. And then it wants to make sure that the list of Countries that come out comes up with actually follows these set of requirements. So even in the first step we are able to solve this problem. Then the subsequent agents are just looking at the solution and making sure that the solution is right. There are cases where our system struggles a lot. So here's an example. Write a 10 word coherent sentence where each word has exactly one letter more than the previous word starting with the first word having one letter and so on till the 10th word having 10 letters. I think I also found this. This prompt on X Look at GPT4's response. It does make some mistakes because this specific word is supposed to have six letters but it has seven letters and dedication has 10 letters. So it's a pretty hard problem for these LLMs. Now this is the one where the system that I built has a lot of trouble. So for example, it comes up with this solution A. It may help limit stories. I think this is where it goes off the rails. So let me explain how this whole thing works. But before that I want to show you one more example which will really explain why this system works for certain types of problems. So here's another prompt that I found I think in Sam's video. Sam has a pretty nice video on explaining the O reasoning models. I'll highly recommend to check it out. And the prompt is what is the value of last 5 numbers in in the first hundred digits of PI added together? The O1 mini gets this wrong and according to Sam's video, these should be the last five digits and the answer should be 29. Now first let me show you how our system did this specific problem and then I'll explain how this whole thing is put together and we'll also talk about what are the failure cases. So in here at the first step it decides that it's going to come up with a solution using this plan. So it says obtain the first hundred digits of PI, then isolate the five digits from the sequence and add these five digits together. Which is a pretty accurate plan. It came up with this Python code which seems to be right when you look at it. But then the agent decided to assign a value without running the code, decided to basically remove the last five digits. If you add those digits together that is going to become 36. And by mistake it says the sum of the last five digits in the first hundred digits of PI is 36. During its own reflection it thinks that is a pretty accurate answer. And that is why you want to have actually these multi multiple stages or multiple steps. So the confidence level that it comes up with is 10 by 10. Then when the second stage was initially looking at it, it thinks that the previous solution correctly identifies steps needed to solve the problem, which is actually correct and the code is well organized. The solution uses appropriate Python liabilities. Verification section demonstrates a good understanding of the logic and potential. However, then it started looking at the weaknesses and then it says incorrect PI digits. The previous solution claims to obtain the first hundred digits of the PI, but it actually only retrieves the first 16 digits due to the limited precision of math PI. And then it says the slicing is incorrect as it includes three. It will give you one or two numbers out of it anyways. So based on this observation, it actually comes up with another plan which says that, okay, we will directly use the first hundred digits of PI as a string and it puts the PI into a string up to 100 digits and then writes a Python program which is able to retrieve those last five digits and then do a summation of those last five digits. So as you can see, in the first stage it made a mistake, but in the second step it was able to recover from the previous mistake and the answer that it comes up with is correct. Here's the whole code that I put together with the help of Claude and O Preview. So you saw from some examples that in some cases this system works really well. It can recover from mistakes that it has made. In other cases it fails miserably. So I'll try to explain why these things happen and hopefully that will give us a window into how this O also works. Now, so far throughout the video I have been saying that it's a multi stage system, so essentially it has multiple agents and they are put together sequentially. I'm also working on another system where these agents are going to be put together in a parallel fashion. But for this specific example we have a multi stage system where the output from the first stage becomes the input to the second stage, and so on and so forth. Now, at the moment there are four agents or four steps in the current implementation that I hacked. In a few hours. It actually needs to run through all these four stages. I think a better approach is to figure out dynamically how many stages to use and when to stop. Now, each of these agents have very detailed system prompts, which really tells it to analyze the problem that it's supposed to solve, as well as to look at the previous solutions, criticize it, and come up with a better solution. So we're going to quickly look at some of the system prompts, put this in a GitHub repo. It's not really well refined code base, but I think it's something that you can start experimenting with. So for the first agent, it's like you are agent one, an expert problem solver specializing in providing initial solutions using thorough chain of thought reasoning. Then we give it an objective. So it needs to understand the problem deeply, come up with a clear and detailed initial solution, and it has to think step by step. That's kind of the chain of thought part of it. Now it has specific instructions how to actually formulate it. So understanding the problem. And that's why you see that every time in the output it kind of repeats the problem. Then it has to come up with come up with a plan to solve it, then solving the problem. And it also has access to code executor or code interpreter which can run Python code behind the Gemini API and return the solution right. And at the end it also has to reflect on what it thinks it did in terms of the solution itself. Now I think this part needs a lot of work because in the current LLMs that we have that can't really think about what they have produced so far, it's just like next word prediction. So if it started off wrong, it will still think that it was the best possible solution. Now, second stage is a critical analyst which will receive a solution and then it has to improve upon that solution. Third and fourth step agents are pretty much the same, except we have the final reviewer in the fourth step which ensure that the solution is correct, comprehensive and well presented. The rest of the system prompt is very similar to what we have seen up to in terms of the agents. So there are two different types of agents. Basically one is the first stage which just receives the problem, the problem. And you can see that it follows a very similar pattern to what we give it, right. In the second stages, if there are previous solutions, so that means like second and third one. So this will determine if it has to critique the previous step output or not. Now, based on my experiments, this system works really well if the problem can be solved using coding or analytical reasoning. If it doesn't involve a coding solution, that's where this specific system fails. Now for O also, it's really good at coding and reasoning, but it's really bad at general tasks such as writing. Now let's talk about a few limitations that I think I need to work on. So first of all, it's a sequential system at the moment. Another approach would be to create a more parallel solution in the beginning, so something along these lines that whenever you receive an input from the user, you can generate multiple parallel solutions and then you have an evaluator stage which actually analyzes these parallels, these multiple solutions and figure out what is the best solution among the provided solutions and then use a sequential system like the one that I have to improve upon one of those selected solutions. The second thing which I noticed with this current implementation is since I'm using Gemini Flash for all of these agents, there's no really diversity of opinions since it's the same model. So it probably is going to gravitate towards the same solution even if it's used in multiple different stages. So one idea would be that you can have multiple different types of LLMs at different stages, whether they are executing in parallel or sequentially. But that way these diverse LLMs can produce diverse solutions and hence you might be able to get a better solution out of it. This is in no way closer to O is a very different system. OpenAI saying that it's a single model which can do chain of thought because it's baked in during the pre training and then it does RL at inference time. So it's a completely different approach. This just tries to mimic the thinking process. This system works really well if the solution can be achieved through coding or through a programming solution. This code is going to be available in a few days after the release of this video. I hope you found this video useful. Thanks for watching and as always, see you in the next one.