OpenAI O1 Can Reason—But How Good Is It?

OpenAI released a new family of models which they are calling OpenAI1. They are pretty impressive, but not for the responses they generate, but rather for the thinking that happens before responding to the user query. This family contains two models currently. One is called O Preview and the other one is a smaller sibling called O1 Mini. I was able to go from this to this with the new models, but came across some very unexpected issues. More on this later in the video. But can it solve the Strawberry problem? The answer is no. Along with the model announcement, they also released the System card which has this very interesting section. The model pursued the goal it was given and when the goal proved impossible, it gathered more resources access to the Docker host and used them to achieve the goal in an unexpected way. Now, if you read the System card, it's more benign than it sounds. Okay, so here's the blog post introducing OpenAI 01 Preview, a new series of reasoning models for solving hard problems. And the main development is that these models are designed to spend more time thinking before they respond. So there is a reinforcement learning along with test, time, compute. And with this release they are releasing the first of this series in chat, GPT and also on their API. So the models that we are seeing are preview models and hopefully there are going to be regular updates and improvements. These are specialized models specifically trained for reasoning tasks. They are trained with reinforcement learning to perform complex reasoning. O thinks before answer and it can produce a long internal chain of thought before responding to the user. And as a result of this chain of thought at test time, you're going to see that these models respond a lot slower compared to GPT4O or ChatGPT, and it's pretty impressive on the evolves that they have shared. So for example, on computation math, GPT4O was able to get an accuracy of 13%. But O, which is the model still under training, is going to be able to achieve 83%. The one that we currently have available is 56.7%. Similarly, on coding it's much better compared to GPT4. Okay. And I have actually seen the improvement in my coding project which I'm going to share later in the video. Similarly, it seems to be doing much better on PhD level science questions even compared to human experts. Now these are very bold and impressive claims and I think we'll have to wait for them to see whether they are true or not on real world problems. If you look at these reasoning benchmarks, it does much better compared to GPT4O, one of the innovations that they are using is chain of thought. So they described it similar to how human may think for a long time before responding to a difficult question. O uses chain of thought when attempting to solve a problem and this chain of thought is happening during test time or infants time and using that chain of thought it refines its strategies before answering the questions. Now this is what reflection was supposed to do for that 70 billion model which we never got. Now overall it's a very impressive model. I'll put link to these blog posts if you are interested. I'll highly recommend to go through them. But something to keep in mind and this is coming from Boris Power who is head of applied research at OpenAI. So in this tweet he says this release is much closer to the original GPT3 release than ChatGPT release a new paradigm which some will find incredibly valuable for things Even we at OpenAI can't predict. But it's not a mass product that just works and unlocks new value for everyone effortlessly. I'm confident we'll have another chatgpt moment so soon. It's a preview model and the reasoning improvement that we are seeing currently is not as big as what we saw from going from GPT 3.43 to chat GPT. That means something very exciting is in words. Although the model is impressive, but it still struggles with some very basic questions. So here's an example. If you ask how many teams keys are in Strawberry, it actually struggles a lot. So I wanted to test this myself. I use the same prompt. Now here is the thought process and it took about six seconds to generate this response. Thanking the user for the question. I am noting that Strawberry contains no tea. Symphony of possibilities I'm mapping out the best approach by evaluating different answers for how many teas are in the Strawberry and its phrase the option range from 0 to 7. I don't know how did it came up with a 7 because there are more than 7 characters in Strawberry. So the final answer that it comes up with is the word Strawberry has zero T's and I am using the bigger preview model here. Now that being said, if you use it for more reasoning tasks or coding you can actually see the improvement and it's substantial. Next I'm going to show you an example of a complicated project that I did with the O1 model over the last few hours. But before that something that you need to know and that is going to be how they Recommend to use O1. So ChatGPT plus and team users will be able to access O1 models in ChatGPT starting today, hopefully everybody will have the access. By now both O Preview and O1 mini can be selected manually in the model picker and at launch weekly rate limits will be 30 messages for O Preview and 50 messages for O1 mini and that is weekly limits, not daily. And they are working towards increasing these rate limits and enable enabling Chat GBD to automatically choose the right model for a given prompt. Now with these limits in mind, you really need to be cautious about how do you use each message. Now you probably have seen a lot of videos testing it on different prompts or discussing the demos, but I thought given the authority prompts limit per week I will use it for a much more complicated project and along the way I I noticed a lot of different things and learned some new stuff about this new model. So for the rest of the video I'm going to walk you through the project itself and then explain what were the different issues that I faced along the way. Working on a real world Problem A few days ago I created a video on call poly for efficient document retrieval with Vision language models. I showed you how to use bialdi, which is a new Python package for orchestrating the call Poly model along with the Quinn model which can generate answers based on input image and input text and the goal was to basically build an end to end retrieval system with a nice looking ui. For this project I was initially using cursor with cloud 3.5 sonnet. So here I provided all the documentation links for these different libraries and explained what I want Cursor to do. It came up with a solution but I ran into a lot of issues and after a few hours I just gave up. In fact I was just feeling very sleepy since O1 was announced today. So I thought why not give it a shot and see if it can help me create this project. So I decided to use it standalone and if you look at the list of projects or list of models from in ChatGPT you can see O1. I did not use it inside Cursor yet since there is a rate limit of 30 prompts per week. So I decided to give it detailed instructions. So here is how it looks like. So create a Python project that utilizes a Vision model for retrieval augmented generation to interact with the PDF and image documents. Then you will build complete pipeline where the user provides a full folder of documents. The system will index and retrieve relevant documents using by LD package, then generate responses to queries using Quint to the Vision model and then retrieve the documents. The retrieved documents will be converted to images and An HTML based UI will allow interaction with the system. Now it has no clue what call Poly is or this package or it's not even aware of the release of this new Qin model. So basically I provided a list of requirements, then some description regarding what the Baldi package does, what the QUIN model does. It has to detect the different types of hardware. And since it doesn't really have access to the Internet, I actually went ahead and copied the documentation from here and added those in a separate tags where I told it that it's part of the documentation. And I did the same thing for the Quinn's documentation. So if you look at the hugging face repo for Quinn, they actually have some quick start guides. Then how do you use multiple images at inference time? So I copied all of this information as a part of the initial user prompt. Now here is how my prompt looks like. It's a very detailed prompt with a lot of information, probably one of the longest prompts that I have provided to an LLM. Okay, so it actually took about 26 seconds to process it. So first it thought about concept implementation. Then what is the task at hand? Then mapping out the process. So it says, okay, let me see. OpenAI code generation include high level project Open overview document indexing with val. The image based response generation with this model, right? Then mapping out the process, building the system, it goes through a step by step process and each, at each steps it kind of reflect on what it has done so far. This is kind of its internal monologue. Now based on that it generates this response. So it says suddenly below is a step by step guide to create Python project that meets the requirements. Right? So here there's going to be an indexing part retriever, then generation provide an HTML based UI and then automatically detects available hardware and configured accordingly. Here is how the project structure is going to look like. What are different files that we will need? Right? And it started generating all these different files that were needed. Now at the end after conclusion it put this section which I did not ask for but usually if you're working with ChatGPT or even Claude, you will see some of these things that it will suggest some improvements, for example error handling, security performance related issues and UI UX improvements, right? So nothing that really stands out but it was able to deliver code. Now when I ran that code there was an error, so it actually was not processing the path correctly. So I just provided the error and was able to fix that. So here is how the error looked like and it simply said understanding the error the thought process again took about 10 seconds, but at the end of this, it actually gave me a pretty solid solution. All right, so then I ran into another issue which had to do with the buffer size or the size of the image that we're trying to process, and it was able to update the code. Now, it does provide explanation just like gpt4o or chatgpt, but the explanation that it provides is usually much better compared to what GPT4O is able to do. Now, the interesting thing is that when I provided this error related to the size of the images and actually picked something from the documentation, so it says ensure image sizes are multiple of 28. That's what Quinn recommends to you to use. And that is the reason I think it picked this specific multiple size. So this was actually a pretty nice thing to notice. Okay, so we went back and forth multiple times. And this is the first version of the ui. So you can select a file here and then click Index Documents. It index the documents to create a vector representation and then gives you the ability to chat with those documents. Now, a couple of things that actually started occurring when I was improving this project with the help of O Preview, it started giving me incomplete codes, and I started asking it to give me complete codes multiple times. So here is an example. Always give me the complete code. It started giving complete code again, but after a while, after I think two or three different interactions, it gave me incomplete code. So here are a couple of examples. Here it is referring to existing code for quinmodel, even though I specifically asked the model to give me complete code for each file. Here is another example. So this can happen, but I noticed this towards the end of our conversation, and it is a pretty large conversation, so it does get a little lazy if the conversation is too long. It also did change the structure of the project. So initially it had a very simplified directory structure, but all of a sudden it basically changed the directory structure and I had to rearrange some of the files. Now, if the conversation is too long, you might see that it starts missing out on some pieces of the code that it has generated before. Now there are instances in which I was really surprised by its thinking ability. So, for example, here's an error that we encountered because of the path errors. It had made this error multiple times and I told it, but out of nowhere it referred back to the documentation, which I thought it had completely forgot about. So here it says that the input underscore path parameter should be either a single file name or directory. And then it was able to connect itself. So there are these moments in which it really surprises you by the way it thinks about certain problems. So here is what we are able to achieve at the end of this coding session. So we created a new app. Here you can choose a file Now I uploaded the file. If I click on upload an index it will create an index using Call Poly. Now the indexing process is complete and we can ask a question. So I'm going to ask how do you steal a part of a network and send this request? And here is an answer from the model along with the exact pages from which it is extracting the information. For example, when I asked who are the authors of this paper? It is able to retrieve the author information because it's looking at the first page. And here's the actual structure of the project. So it is complex with a lot of moving parts. It's able to have a ui, is able to create indices, keep track of different indices across different chat sessions. It took me a couple of hours to create this using using models that it has not seen before just by providing the documentation and some hand holding along the way. Now it's definitely a huge improvement compared to the previous models, but it's not really an AGI or anything close to that. But to be honest, I'm actually pretty impressed so far for what is able to do. Specifically it has a very narrow focused on STEM reasoning and coding and it does a pretty decent job at that. I hope you found this video useful. Thanks for watching and as always, see you in the next one.