Open AI SHIPS! o1 FULL & ChatGPT Pro - First Impressions

Hello everybody. Welcome back to another video. So Today was the first day of OpenAI's 12 Days of Ship Miss. Basically every weekday leading up until Christmas, OpenAI is shipping something. Like I said, today was the first day. So let's talk about what they announced. This is the first live stream that OpenAI did. It was only like 15 minutes long on their YouTube channel. Our first ship is OpenAI 01 and 01 Pro in chat GPT. So the full 01 is finally here. It's out of preview and they've got some benchmarks to show off. You can see the normal GPT4 Omni scores 13.4. In competition math, 01 preview was at 56.7, which is way above GPT4 O. And then the final O1 that we're getting today on release is 83.3. So way better in math. Another thing too, 01 is multimodal, meaning it can take in image inputs. And that is something that was sorely missed in 01 preview. So I'm happy to see that. We also see significant improvements in competition code. Again, 11 score for GPT4.001 preview at 62 and 01 at 89. I mean that is a huge increase in comparison to the original GPT4O. And then finally we move on to PhD level science questions. This is GPQA Diamond GPT4O was at a 56.1,01 preview was at 78.3, with 01 at about 78. So pretty much the same as preview in terms of those PhD level science questions. You can see though, and I believe lighter colored bar here for O1 preview is supposed to signify the error rate. It seems that even though O1 is about exactly the same in terms of accuracy, it seems to do so in a fewer shot hit rate. And human experts are sitting over here at 69.7. Nice. But apparently for these PhD level science questions, we've got better accuracy with this new 01 model than an expert human. However, we don't get the human eval for competition code and competition math, which leads me to believe an expert human is probably still better. Now let's talk about OpenAI01 Pro mode. So there's a new pro plan coming to ChatGPT. It's obviously more expensive and we'll get into the pricing in a little bit. But O1 Pro mode uses apparently more intelligence and more compute to give you the most accurate answer. And 01 Pro mode seems to be a small bump in comparison to the full O1 in both PhD level science questions and competition math. We got an 85 in math and a 79 in science questions. They also did a worst of four performance benchmark. And again, this is for math code and science. The model attempts each of these questions four times and they only consider a question solved if all four attempts are correct. So it's got to do it four times in a row perfectly. You can still see it's very consistent. It still puts up those high benchmark scores. So I believe this is OpenAI sort of saying like, hey, this isn't, you know, cherry picked benchmarks. These are pretty legit. And obviously I'm going to link this down below for you guys if you want to watch the full 15 minute thing. But it's, you know, Sam Altman talking with some of the devs and they do a couple of demos. They do a test list Roman emperors of the second century, including their dates and accomplishments. And not only is the full O1 faster, but it is more accurate and it gives you a better response. They also drew this astrophysicist physics problem on a piece of paper and sent it to full 01. But this is actually sort of a trick question because it left out a very specific parameter and not only did it actually do the math correctly, but it actually gave its best guess for the omitted parameter. And according to the researchers, it was accurate. I know I'm not getting very complex or deep into this stuff, but to be honest, I mean, look at this math. It's a little bit beyond me. I was never great at math in school, so I'm gonna trust that this thing knows what it's doing. And the researchers are corre this was or it did seem to be a live demo, which is pretty cool. And I respect OpenAI for doing a real live demo. Now this is where they begin to test the 01 Pro mode. And again, that comes with the ChatGPT Pro subscription which we're going to talk about in just a little bit. The question here was for O1 to guess a specific protein that fits a bunch of criteria. And again, like I don't know anything about proteins. So you know, there's length of 100210 to 230amino acid residues, 32 kilobases, a chromosome. It's, it's complex stuff so it does a lot of thinking. It took it about a minute to solve this, but it also correctly guessed the protein. I mean, we're really achieving levels of intelligence here that are far beyond the average person. And I think they did a good job. Demoing how dynamic this O1 model is at solving these high level issues. All right, so let's talk about that new Chat GPT Pro subscription. Here it is. ChatGPT Pro $200 a month. What do you get for the massive bag of $200 a month? Well, you get everything that you normally do in plus you get unlimited access to everything as well. So Unlimited access to 01, 01 mini GPT4O. Also unlimited access to voice mode, and you also get unlimited access to Zero1 Pro mode which uses more compute. For the best answers to the hardest questions, you might be thinking, Matt, are you going to buy this $200 a month pro subscription? The answer is not today. I'm going to wait for some fellow youtubers to test this out first and then I'm going to make a choice to see if I want to pay for it. Obviously, if I do end up paying for it, I will be giving you guys a nice video review on it. Let me know if that's something you'd be interested in seeing down in the comments below. But $200? I mean that's really steep. That is a 10 times increase. Now it is unlimited access to everything, which is great, but there are also other considerations. There is no option to, you know, pay as you go through the API like OpenAI traditionally has. Both the full O1 and Zero1 Pro modes are not available in the API. Advanced voice is not yet available in the API either. I feel like there are a lot of people that would be willing to pay, you know, 80, 100 bucks a month just using O1 Pro mode through the API for example. And one thing that is irking me, it's kind of in the back of my head in regards to this plan is 200 bucks a month is super expensive. And yeah, 01 Pro mode and having the unlimited access to that is unprecedented in the AI space right now. But in a few months I'm thinking open source is going to catch up. I think we'll have models that are very competitive with 01 and even 01 Pro mode in 2025 that are likely open source and can be run on your own hardware or websites that offer pay as you go systems or just flat out cheaper plans. Then again, we still do get access to the full 01 in plus, which is 20 bucks a month, so maybe that will help them remain competitive. Now, since this Pro plan has unlimited access to these AI models, I'm also wondering what's stopping people from sharing accounts. I think four people paying $50 a month is a lot more attractive to some people who maybe have friends in the AI space and are willing to split the plan. But is that against the terms of service? Is it against their policies? Probably. Could you get away with it though? I think a lot of people are going going to absolutely try. Do you think $200 a month for unlimited access to these tools is justified? Do you think the AI is worth that much? I am sure that there are some people out there that would be using this 01 Pro mode enough that unlimited access at $200 a month is worth it. But I can't say I'm one of those people. I'd really love to hear your feedback in the comments below as I make this video. Guys, you have to understand people are talking about and discovering and testing these models in real time. If you want to see someone test Chat GPT Pro and I knew he was going to do it too. Matthew Berman has purchased this new Chat GPT Pro account with access to Unlimited Pro mode with 01. So I'll link this down below. Like check out his channel. I'm sure he's going to be doing some pretty rigorous testing on this thing. I'll definitely be taking a look at his videos as well to help me decide if it's something that I also want to maybe pay for and cover on the channel. 200 bucks a month is very steep. I already did do some live testing in my Discord server in a voice channel, so let's go over my first impressions with the full 01 model. The first thing that we wanted to test was the multimodal capabilities, so I sent it a nice Where's Waldo? And this is a full res typical Where's Waldo print. If you're wondering where he is, he is right here. It thought for about 17 seconds specifically about finding Waldo, which it then did. Look closely at the crowd in the upper half of the scene near the shoreline. Waldo is standing amongst the bathers on the sand wearing his trademark red white striped shirt, hat and blue pants. Which is absolutely correct. Now I was a little concerned too because I'm like maybe it's seen this scene before. Maybe into the training data somewhere it knows where this specifically is. So I asked it to then point out some key elements or structures that Waldo is near, but striped beach tents and windbreaks, colorful beach umbrellas and towels, and clusters of bathers and beachgo. It is very clear that it has a good idea of where Waldo is in this image. So that was pretty impressive. Now can regular 4o find Waldo not exactly. 4o tells us where we can probably find him, but it does not give us a specific and exact position like we got with the full O1. So impressive image recognition capabilities are absolutely there with this model. I love to see that. Next up, we push it a little bit further. I sent it a funny photo of myself and I said, convert me to ASSI art or ACSI art. And this is what it produced, which is kind of cool looking, but it definitely isn't, you know, a representation of me. I then asked it to do something a little bit simpler. So Shrek. And it tried to do, you know, a Shrek for me. It kind of got his iconic ears in there. You can kind of see his face. But definitely still something that is not really a priority in these models. Even though they are very good in certain areas, they still absolutely possess limitations, and I don't think anyone is denying that. Now, this was the most impressive thing that I think I tested today. I sent it a screenshot of the Spotify app and I said, recreate this in htmx. Now, it thought about HTMX for four seconds and then provided me with some HTMX code. Now if you go over here and copy this code and place it right into codepen, boom, there is the recreation. And I gotta say, guys, this was pretty darn impressive. It did the home, the search, the create playlist, the browse podcast. Now, none of these buttons actually work. The typing does work if you want to search for something, but if you highlight over each of these, it does gray it out to, you know, quote unquote selected. And it did try to do the colors correctly as well, also with those little highlighted pieces, those extra little cubes advertising each selection. And like I said, this was all during my little live session in the Discord server. So one of my mods suggested to ask it to provide CSS and JS code to make it interactive, which it then did. Here is the CSS, here is the JavaScript. And it also provided an updated HTML to go along with them. So it knew to do that as well. And I didn't even ask it for that. And here is the updated one. It looks a little bit better. The colors are better as well. But you can see if we hover over and select each of them, it will actually animate, making each selection a little bit larger, as well as rotating one of those smaller squares at the top. So that was really cool, really impressive. And by the way, this was all first shot. It's not like it had an error and it needed to redo something which happens a lot when you code with AI. It just did it correctly first time right off the bat. Also shrinking down the home tab here works as a function with this as well. Moving on, I asked it to do some more HTML and CSS to do an infinite runner style game. It was also able to replicate that quite impressively. You can see the game that it made down here. I'm quite impressed with the graphics specifically on this one. We have a gradient there at the top for the sky. The floor or the ground is made up of these alternating brown colored blocks but adds like enough texture to make it pretty appealing overall. One thing that it doesn't have is there's no way to collide with these blocks and lose the game. But we do have a score for first shot though. Again, just right off the bat it ripped this out. It's pretty darn impressive and really cool. Now finally we asked it to convert that infinite runner game to 3js, essentially convert the game to 3D and it honestly even was able to do that pretty impressively. Now the game is a little bit broken. If I refresh here and then paste in the HTML, it will then load up the game. It'll do an infinite runner and then once you hit a block you'll lose. But you do have a counting score. There's no way to restart the game. And I think that's the only issue with this one. But pretty cool that it was able to like convert this into 3D just just like that. It's definitely great at coding. And finally someone in the chat sent this funny Dall E3 generation. So I sent this and I said is this you right now? Haha no that's not me. I'm just a virtual language model so I don't have a physical form. Let him blown a dramatic sweat soaked face. But it did a pretty good recognition saying it's a highly expressive manga style character losing his mind in excitement or shock. So now guys, I want to continue our testing a little bit. Let's try giving it some difficult logic based problems. I'm specifically going to test it on Matthew Berman's logic problems. As I mentioned before, he is great at testing LLMs. You can see he already bought that $200 a month. Plan to test the full 01 Pro mode. But he's got a really nice list that I'm going to link down below of logic problems to test LLMs like this. So let's give it a shot. Let's start here. A marble is put in a glass cup. The glass is then turned upside down and put on a table. Then glass is picked up and put in a microwave. Where is the marble? Explain your reasoning step by step. This is kind of a classic logic problem. A lot of LLMs end up failing. It thinks for just a couple of seconds and then gives us the correct answer. The marble is still on the table. It never moved into the microwave, only the glass was. Bravo. Perfect response there. Love to see that. It does a pretty good job explaining it too. When you lift the glass straight up off the table, the marble which is sitting on the table surface is no longer enclosed by the glass. It stays right where it was on the table because there was no mention of sticking it to the glass or being moved along with it. Facts. This, however, was a problem that was also passed by 01 preview. And by the way, this is all based on Berman's testing. Again, this is going to be linked down below. Let's try the only one in here that 01 preview failed, which is the logic problem about the North Pole Imagine standing at the North Pole of the Earth. Walk in any direction in a straight line for 1 km. Now turn 90 degrees to the left. Walk for as long as it takes to pass your starting point. Let's see what it gives us for an answer. Oh, it's Mapping out the journey Latitude Dynamics Analyzing the puzzle Navigating Earth Charting the journey Redefining the Journey by definition, the North Pole is the northernmost point on Earth. Any direction you choose to walk from the North Pole is directly south. Walking 1km south, you will not be at the pole anymore. You will be on some latitude circle close to the pole. Since you turned 90 degrees left from a south, you are now heading east, walking east or west at a fixed latitude, which means following a parallel. Essentially walking along a circle around the Earth. Yes. Circumference of the Latitude circle the circumference of any latitude circle other than the equator is smaller than the Earth's maximum circumference at the equator. The closer you are to the poles, the smaller the latitude circle. As you approach the pole, these circles become very small. Yes. Now you have positioned yourself such that after walking 1km south, you end up on a latitude circle with a specific circumference. You can create a special scenario. So now it's considering a latitude circle where the Circumference is exactly 2 times PI. If you're 1 km south, step places you on such a circle. Then walking east along that circle is like walking along a track. That is precisely 2 PI kilometers around. Once you complete a full loop, you will arrive back at the same point along that circle. Thus you have indeed passed your starting point on that latitude circle. So apparently our answer here is 2 times PI. So exactly 2 times PI kilometers. I'm gonna be real guys. I'm no genius. I don't know if that's even correct. It seems correct the way that the AI explained it. After some digging, this originally seems to be a post from Yan Lecun on Twitter. It seems we have an example from all the way back in March 17th of Claude 3 haiku solving this correctly getting the same exact answer. There's a lot of responses in here that are also different than what we got from 01. People are also criticizing this prompt for being purely mental visualizations, but it does seem like the answer is 2 PI. Whether or not you want to criticize this question at hand, 01 seems to have gotten it right, at least in terms of its original creator. So bravo to 01. Alright, let's try some hyper specific problems. I'm going to give it a photo of an engine bay and I'm gonna ask for the specific engine. What engine is this? Oh it's thinking. It's thinking. Dang, it's pretty good. Mid era 2000s Volkswagen TDI engine key identifying features the fuel pump, camshaft driven vacuum pump, engine cover and oil fill cap location. There is no engine cover on here, but we do have an oil fill cap. Yeah, that is. That's pretty accurate. So it did a good job here. Now it thinks this is a PD model. I'm pretty sure this is an ALH that doesn't mean too much. Next up let's try this thing out on GeoGuessr. See if it can just get guess this location accurately. We'll just send it like this picture right here. Guess the location thought for 12 seconds. Language and cultural clues Spanish speaking country Latin America Leafless trees suggests a winter season Southern hemisphere countries like Argentina, Chile or Uruguay. Old truck loaded with wooden crates. It also knows that it's a N geoguessr which is funny. The Spanish language clues Southern hemisphere winter trees Typical suburban architecture, older vehicles and stylistic feel points strongly towards Argentina. Greater Buenos Aires, Argentinas. And was it correct? It was pretty darn close. So this was in fact in Uruguay so that would have been a pretty decent guess. Not perfect though. Well, I think that's where I'm gonna leave this one off today. Obviously more testing is going to ensue in the future. Let me know your thoughts on this. Are you excited for the rest of the 12 OpenAI days before Christmas, there's Christmas gifts. O1 is impressive. It's a great model. It's very fast, it's accurate. And finally, image recognition. I was really waiting for that one. So that's a good ship from OpenAI. Matt approved the $200 plan, though. I mean, that seems crazy to me, but I got to think about that one. Let me know your thoughts. Thanks so much for watching and I will see you in the next video. Goodbye.