OpenAI Just Supercharged GPT-4o and It’s Faster Than Ever Before

OpenAI just rolled out a feature called predicted outputs for their GPT4O and GPT4O mini models, and it's already making a big difference in how fast we can get things done with AI. Let me break down what this feature does and why it's such a game changer, especially if you're into AI powered coding, content creation, or even just looking to speed up workflows. So you're working with AI to edit code or update documents, Right? Normally, even the latest models have to generate each token, the words or bits of code, one by one, which takes time. But here's where predicted output steps in. With this new feature, you can tell the model what you expect part of the response to be in advance. That way it doesn't have to start from scratch every time, it can jump straight to what you've predicted, generating fewer tokens, which means you get your answer faster. Now. How much faster? Well, according to OpenAI, early tests show that code editing with these predictions is coming in two to four times faster than before. If you're dealing with large files like a big code file or a lengthy document, something that used to take about 70 seconds can now be done in roughly 20 seconds. So essentially we're talking about cutting the time by more than half in some cases. Now, the idea is pretty straightforward. Instead of just asking the AI to do something and waiting for it to come up with everything on its own, you can give it a hint. Basically a prediction of what part of the output should be. Let's say you're tweaking some code and most of the code is going to stay the same, except for one or two lines. You can input the existing code as your prediction for the model to base its response on. The model then only needs to fill in the gaps instead of rewriting everything from scratch. So where would you use this? Well, predicted outputs shine in situations where the tasks are repetitive or predictable. For example, you're updating a blog post, modifying small parts of a script, or making tiny changes in a huge data set. Instead of waiting for the AI to rehash all the parts that are unchanged, you let it know what's already there, and it only focuses on the updates. You're essentially giving it a shortcut so it spends less time figuring out the whole response and more time delivering exactly what you need. But you but here's a key Predicted outputs aren't ideal for generating brand new unique content. If the model has no frame of reference, like when you're creating something from scratch with no predictable elements predicted Outputs won't help you there. This feature works best when there's a baseline, something repetitive or partially known that the AI can rely on. Now, OpenAI didn't just leave it there. They've tested this feature across multiple programming languages like Python, JavaScript, Go, and C, and found that it's effective in each one. So if you're a developer working in one of these languages, this could really be a time saver. But if you're wondering whether you can use this with every model, here's the deal. Predicted outputs are only available with the GPT4O and GPT4O Mini, and there are a few restrictions you should know about. For starters, this feature doesn't support some advanced API parameters that you might use in other situations. So if you're used to calling for multiple outputs at once, using function calls, or getting probabilities with log probs, this feature won't play nice with those settings. OpenAI even says that if you want to maximize efficiency, try using predicted outputs for controlled tasks, where the response is somewhat predictable. Let's go a little deeper with an example. Say you're working on refactoring some code in TypeScript. Let's imagine you have a class called user, with a property labeled username, but you need to change that property to email. The code structure remains almost entirely the same, so you can feed the existing code to GPT4O as the predicted output. Then, when you prompt it to make that change, it will only update what's necessary, just that single line, instead of rewriting the whole thing from scratch. This brings up a few neat technical details too. When the model processes your prediction, it actually keeps track of how many tokens it accepted and how many it had to reject. Tokens that match your prediction get processed quickly, while rejected tokens, ones the model doesn't end up using, are discarded but still charged at the regular completion rate. So there's a bit of strategy involved here. If your prediction is accurate, you save time and potentially cut down on tokens. If it's way off, you'll still pay for the extra effort. And to go even faster, OpenAI's also rolled out a way to use this feature with streaming, which can boost performance even more. Let's say you're doing that same code refactoring example with streaming. Instead of waiting for the entire response to load at once, the API sends it to you in chunks. So as each part of the response is generated, it's streamed in real time. You can display it immediately or process it on the fly, which really minimizes any delay. Another useful thing about this feature is that it doesn't matter where your predicted text appears with in the response. You could have a chunk of code at the beginning, some new additions in the middle, or at the end. It all gets factored in. The AI can latch onto those predictions no matter where they're positioned, speeding up the process even further. Here's an example to illustrate. Let's say you're working with a simple server setup in JavaScript using a library called Hano, and you want to add a new route. You could give the model a version of the code with most of the server structure in place. Then, when you ask it to add a new route that responds with hello World, it doesn't need to think about the rest of the file. It can just drop in that one new line while saving time by using the structure from your prediction. So what are the limitations? Besides the restrictions on the models, certain settings just aren't compatible with predicted outputs. Right now, for example, you can't set maxcompletiontokens when using predictions, and if you're planning on generating audio, mixing modalities, or calling functions within the API, you'll have to look elsewhere. Predicted outputs don't support those features. One more thing to keep in mind is that even if the prediction isn't spot on, it'll still have to process all the tokens you've provided. So if your prediction tax includes 100 tokens but the model ends up rejecting 20, you'll still be billed for the full 100. Essentially, it'll save you time, but not necessarily money, unless you're strategic with how you use it for developers. OpenAI's documentation provides more details on how to implement this feature, but here's a basic idea of how it works with OpenAI's SDKs. Let's say you're using JavaScript and want to refactor a TypeScript class. You'd input your code snippet as a prediction parameter. Then, using a chat completion prompt, you'd specify what changes you want, like replacing the username property with email. The AI takes it from there, updating just that line, thanks to the prediction you provided. In terms of practical application, predicted outputs will likely appeal to anyone dealing with repetitive tasks. Think bloggers updating old posts, developers tweaking chunks of code, or data scientists working on small adjustments within datasets. Imagine you've got a list of entries in a blog post that need minor updates. Rather than regenerating everything, you simply pre fill what's staying the same and the AI only needs to add the new bits. The more predictable the task, the better this feature performs. OpenAI also advises starting small, testing it with controlled, repeatable tasks. This lets you figure out just how much time you can save while getting used to the feature. And since predicted outputs require a bit of trial and error, starting with simpler predictions might help you avoid extra charges from too many rejected tokens. So what do you think? Is this something you try in your workflow, or do you see a specific use for it? Drop your thoughts below. And if you enjoyed this dive, don't forget to like and subscribe for more AI insights. Thanks for watching, and I'll see you in the next one.