Dr. Caro Zsolnai breaks down OpenAI's new O-series ChatGPT variants trained on 100,000 conversations. The models solve nearly half of high-school cybersecurity challenges, up from 21% with GPT4O, and are three times more resistant to jailbreak attempts. They also show 18% improvement on virology tasks and reduced hallucination.
Questions this video answers
52 pages.
100,000 text rounds.
About 21%.
Nearly 50% (described as "almost half").
Previous: about 3–4%; New: about 13%.
Chapters
Yes, OpenAI has unveiled their amazing O3AI and everyone is talking about it. This was their ChatGPT that can think before it answers. It shows many thoughts that lead to the final answer. It can reflect on its mistakes. It truly is chef's kiss. And thus it does way better than previous methods. You hear a lot about it. Everyone says AI this, AI that, and they say it will never be able to do this and that. But to that we say no. Because we are fellow scholars here. We want hard data from real research papers. So do we have any of that? Oh my, look at this. A 52 page paper. Now we're talking. And I went through it, so you don't have to. Although it's super fun. So here are three things that really surprised me. For instance, it has been tested with 100,000 text rounds and so much more. Dear fellow scholars, this is Two Minute Papers with Dr. Caro Zsolnai. Note that these are results with the O version which is in terms of fundamentals, the same as O3. First, finding cybersecurity issues. Now that would be absolutely incredible. An AI that helps us create more safe computer systems. Sign me up right now. However, that is not easy at all. For instance, they tested O1 on a set of curated cybersecurity challenges. First on a high school level. Now don't be fooled. These are still quite, quite a challenge and require chaining multiple steps to solve. One liner solutions don't work here. These require a little creative thought as they cannot be copy pasted from a textbook. You have to adapt your knowledge to new systems. The AI was given 12 potential attempts at the problems. And what happened? Well, the earlier GPT4O system was able to solve 21%, about every fifth. And what about the new one? Is that better? Oh my. More than double. It solves almost half of these problems. That is incredible. However, I know you saw it. Oh yes, there are collegiate and professional level challenges too. So what about those? Well, my initial guess was that these are going to be big fat zeros. These are really tough problems after all. So what is the actual result? Previous system 3 and 4%. And now hold on to your papers, fellow scholars, for their new system system which can solve more than triple 13%. Whoa. I have to be honest, that is absolutely incredible progress. But it gets better. Second, let's try to hack the O system ourselves. What is that? That is called jailbreaking. A jailbreak means writing a carefully designed prompt that tries to make the AI do something that it shouldn't be doing. If we Try that. The previous system was quite resistant to those. But wait, there are datasets with examples that it did not do well with at all. So how about the new one? Oh my. Look at that. Loving this. Even the worst version is more than three times more resistant to these jailbreaks. It is like a safe that is resistant to many of the world's best lock pickers. And when humans were testing the two systems against each other, the new one was found to be safer around 60% of the time. Approximately 30% goes to the previous system and about 10% in ties. But yes, I hear you fellow scholars asking what about hallucinations? This is when we ask about something and we get an answer with made up data. Is it doing better in that regard? I really hope so, because I am using this previous model to talk about this very paper and it hallucinates a fair bit. Well, first they talk about accuracy, not hallucination rate. Hmm. Why? Because it is easy to say something that is not made up information, but is also not accurate. Like leaving the answer for the question in an exam blank. That is not hallucinating but also not helpful. So accuracy has increased for the new system. And with that, did hallucinations increase or decrease? Would you look at that? Decreased. Yummy. It is better on both axes. Absolutely incredible. And it is also meaningfully better on troubleshooting questions in virology. 18% better. And don't forget, this is just O1. We are currently at O3, which is once again meaningfully better than O1. I think this shows that a huge amount of work goes into the testing and evaluation of these AI models. What a time to be alive. Now something to highlight. OpenAI found that it also makes a formidable con artist. This area is outside of my expertise, but. But I would like to highlight it so that more people know about it. Though I cannot add too much more. But I will say one thing. I would love to see more work on how we can use this system to be a shield against such manipulative behavior, both from humans and from bots. It knows what patterns to look for, so it should be able to help us. I would love to see that. And this is all hard data from the paper, not just stuff that you hear thrown around in the media. So that's what we do here. A two minute papers. Subscribe and hit the bell icon if you wish to see more. It's also good for us because you know, we can keep existing. That would be great. So what do you fellow scholars think? Let me know in the comments below. To run your own experiments on an Nvidia gpu, check out Lamda. I use it myself regularly for these videos. Hmm, look at that. You can generate high quality images in less than a second per image. I did a ton more of them and paid less than a dollar for all crazy. Seriously. Try it out now@LambdaLabs.com papers or click the link in the description.
