OpenAI's DevDay And A Preview Of Our Agentic Future

Today on the AI Daily Brief, what we learned about our agentic future at OpenAI's Dev Day. The AI Daily Brief is a daily podcast and video about the most important news and discussions in AI. To join the conversation, follow the Discord link in our show. Notes. Hello friends, Quick note. There ended up being so much to Talk about with OpenAI Dev Day that today's episode is just just that it is just a main episode. We are not doing the headlines and frankly part of the other reason for that is that the headlines are in fact other really cool product announcements. Pika, Microsoft and 11 Labs all announced really interesting products yesterday and there was also an XAI meetup. So there was a lot going on outside of just OpenAI's Dev Day and we will get to all of that in a future show this week. But for today it is just going to be all about OpenAI and it's pretty interesting. Welcome back to the AI Daily Brief. Today, of course, we are talking about OpenAI's Dev Day. This is their second annual Dev Day. It's a chance for them to give a bunch of updates, share a bunch of new products, and as you'll see, one of the big themes from this particular Dev Day was that this one really was for developers. This did not feel like an event that was focused on consumers. There was, as you'll see, very little talk of GPT5 and instead what we got was something of a preview of the future. There is a picture that is starting to become clear that's not necessarily new, but is really rounding out in its vision of the agentic future that is hurtling down the pipeline at ever increasing speed. So let's do a little bit of what was actually announced and then let's talk about how different people reacted to it and what some of the big themes were to start. As much as they would have liked to. There was no way to avoid the recent departures of CTO Mira Moradi and Chief research officer Bob McGrew. In a briefing with reporters ahead of the event, new Chief Product Officer Kevin Weil said, I'll start with saying Bob and Mira have been awesome leaders. I've learned a lot from them and they are a huge part of getting us to where we are today. And also we're not going to slow down. Like I said, a lot of the discussion was about developers. In this session. OpenAI said they currently have 3 million developers building on top of their models, which is triple the number of active apps on the platform compared to last year. One of the things that came up in discussions a lot was the decreasing cost of intelligence. OpenAI said that they had cut costs for developers to access their APIs by 99% over the past two years and obviously this was driven by incredible competition from Google and Meta, who are also pushing the price down now. In terms of what was announced, there were four big things. The RealTime API, which probably generated the most discussion on places like Twitter, vision, fine tuning, prompt catching and distillation. Let's talk about the Realtime API first. The Realtime API gives developers nearly real time speech to speech experiences in their apps. The really important thing here, which seems like a small detail but really isn't, is that this is native speech to speech, as they put it. There is no text intermediary, which means low latency nuanced output. So of course when you're thinking about applications like customer service, this is a total game changer for that sort of experience. In terms of other details, there is a choice of six voices provided by OpenAI. These are different than the ones used by ChatGPT, and currently there's no ability to integrate third party voices, probably in order to avoid copyright issues. Head of Developer experience Romain Hewitt demoed a trip planning app using the Realtime API. He verbally discussed a trip to London with the AI assistant, got low latency responses and the app was also able to annotate a map with restaurant locations as it answered questions. A second demo showed how the Realtime API could speak on the phone to order food for an event. Unlike Google's duo, the API can't make calls natively, but can integrate with APIs like Twilio to do so. OpenAI is not currently forcing the AI models to identify themselves as such on phone calls, something which could be illegal under new California laws. The Real Time API returns recorded responses, transcript and function calls in real time. Now, once again, this is not a hey, this is coming in the future product. The public beta is now open. In terms of costs, one of the big conversations was about how at $0.06 per minute of audio input and $0.24 per minute of audio output for an evenly mixed use case of about 15 cents per minute, this was not only more expensive than 11 labs speech to speech products, which cost 11 cents per minute, but might not actually in the short term at least be cost savings relative to human labor. Many people pointed out that the cost for one hour of this would be around $18, which is more than what companies would pay for many international call centers now. Others pointed out that that would assume constant talking, which isn't necessarily going to be the case. And I also think more broadly than that, trying to understand the prices now as some sort of static thing is probably not really going to give you a good picture of what the future is actually going to look like. Today's episode is brought to you by Plum. Generative AI promises to supercharge your productivity and give you superpowers. But if you're not an engineer, trying to harness AI can be incredibly frustrating. Hours wasted wrestling with complex tools only to give up when they don't work. We all have tedious tasks we'd love to automate and challenges AI could solve, but few of us have the skills to fully leverage these game changing technologies. That's where Plum comes in. The mission to make automating your work feel like magic. Imagine typing out AI Read my Gmail and ping me in Slack when something critical comes in and watching it come to life before your eyes. No coding required. Whether you're a marketer, salesperson or founder, Plum enables you to create custom AI workflows in minutes, not hours. Check out useplum.com that's Plum with a B for early access to the future of workflow automation. Today's episode is brought to you by Venice. Venice is a private, uncensored, generative AI app. It accesses open source models to enable text, image and code generation without the fear of being spied on or having your data exploited. Discuss anything with Venice without concern about it being monitored, sold, or given to advertisers and governments. Venice is different because your conversations and creations are kept securely within the browser, never stored or accessible by Venice. Unlike other AI apps, Venice won't tell you what's okay to say or not. Venice won't patronize you. It simply provides direct access to machine intelligence. No topics are off limits, no ideas are taboo. With Venice, you're in control of the AI as you should be. Pro subscriptions are available for $49 a year or $8 per month. AI Daily Brief listeners receive a 20% discount on Venice Pro. Visit Venice AI NLW and enter the discount code NLWDAILYBRIEF. That's NLW Daily Brief, all one word. Next up was Vision fine tuning. This is a really cool one that I think is going to open up a lot of new use cases as well. Developers can now use images as well as text to fine tune applications of GPT4O using the API. This should help massively improve performance of tasks that require visual understanding. Some pointed out that this could be significant for autonomous driving think traffic sign detection and the OpenAI team said that in general this was the top feature request to the fine tuning team. Now when it comes to safety, head of Product API olivier Godman told TechCrunch that safety policies are in place to prevent uploading copyrighted and violent images. The third notable feature was prompt caching. This is a similar feature to the one that was introduced by Anthropic several months ago, but basically it allows developers to cache frequently used context between API calls, which reduces costs and latency. OpenAI say developers can save 50% using the feature for repeated API calls. For what it's worth, Anthropic promoted theirs as capable of reducing costs by up to 90%. This is clearly part of the deflation of API costs, but some wonder if this is going to cause issues with Microsoft. Dan Shipper from Every writes, it's great for developers, but it also creates an interesting dynamic with its biggest partner, Microsoft. I heard from Dev Day attendees that Microsoft has been pushing large enterprises to commit upfront to buy a certain amount of GPT4 API calls in order to guarantee capacity. One wonders how Microsoft and its customers who have already committed feel about these price reductions. Last big update was model distillation. Developers can now use larger models like O1 preview and GPT 4.0 to fine tune smaller models like GPT 4.0 mini using the API. The idea here is basically to allow developers to get better performance out of smaller models. OpenAI are also launching this feature alongside a beta of an evaluation tool so developers can compare performance now in terms of what wasn't announced. No, there was no update on the GPT Store, there was nothing about Sora, there was no full size 01, and there certainly wasn't mention of GPT5. That said, the buzz around 01 was still palpable and what this session really reinforced was that this is a different category of product than GPT4O. That's exactly how Head of Product API Olivier Godman described it, basically saying that this was a new family of models distinct from GPT4O. You're starting to see then a cleave between on the one hand general purpose LLM models and on the other hand reasoning models and a sense that there's going to be different use cases that fit each of these different categories. Head of Developer Relations Roman Hewitt again did a live demo of owan, where he used it to build an iPhone app with a single prompted 30 seconds. He also prompted a web app to control a drone that was present on stage and then use the app to pilot the drone as shipper. Again, from every points out, it would have been possible to do these demos with previous GPT models, but they would have taken much longer to build and probably wouldn't have been suitable for a live audience. Now one of the big culminations of the event was a fireside conversation between Sam Altman and Chief Product Officer Kevin Weil. One thing that's notable is that this was not live streamed and so what we got was basically summaries from people like Greg Kamrat who was there in the room, a few of the highlights One of the big topics of discussion was how close we are to AGI. Sam said that they would finish a system and ask in what way is this not an AGI? Basically arguing that the word is overloaded and that O1 is level 2 AGI. For Sam though, the rate of scientific discovery is the benchmark. He said that the fact that definitions matter this much means we're getting close. He also said we're in this period where it's going to feel blurry for a while. If we can make an AI system that is better at AI research than OpenAI is, then that feels like a real milestone. Another part of the discussion was alignment. Altman said, it's true we have a different take on alignment than whatever that Internet forum is presumably talking about. Less wrong. We want to figure out how to build capable models that get safer and safer over time. We have an approach to figure out where the capabilities are going to work, then make it safe. Oh one is our most capable model and it's our most aligned model too. Sam reinforced the idea that iterative deployment is the best safety system they have because as Kevin Weil put it, no matter how many smart people you have inside your walls, there are way more smart people outside your walls. There was more discussion of 01 before the end of the year. They said that 01 would support function calling along with system prompts and structured output. Altman said the model is going to get so much better so fast. It's at GPT2 level. We know how to get it to GPT4 level. Plan for the model to get rapidly smarter. At one point Sam asked the crowd if they thought they were smarter than 01. A few hands went up and he said, do you think you'll still think this by 02? No one wants to take that bet. This was yet again the company really reinforcing that they're going to be the best at reasoning models, which I think is one of the biggest takeaways of this event. There are a bunch of other little things that were interesting too. Sam, for example, said that they thought that infinite context length would happen within the decade. And finally there was the discussion of agents. Basically, 01 was presented as the path to making agents actually happen, altman said. People get used to any new tech quickly, but agents will be a big deal. People will ask an agent to do something that would have taken them a month and it'll take an hour. Then they'll have 10x agents, then they'll have 1000x agents. And because of O1 for the first time, the agent conversation doesn't feel like it's some super far in the future type of thing. And so let's talk about takeaways. There were two kind of big themes that stood out to me. One is just the speed of progress. As Dan Schipper put it, it's easy to forget that just a few years ago none of the things that were shown today were possible or even on anyone's radar. Today, a single developer making an app in their spare time can build things that entire teams of developers wouldn't have been capable of previously. However, I think in many ways the even bigger theme was summed up by developer Nick Dobos, who wrote OpenAI is challenging what a computer can do. Everyone else is playing with LLMs. It really does feel like there's this fork that's starting to happen between LLMs and reasoning AIs, and that O1 is where these two genetic lines diverge. They still feel really close right now, but it seems like they will be fundamentally different in the future. Ethan Mollick put it this way, OpenAI put a lot of interesting pieces on the board today. With the API you can get good planning, real time, two way voice, and a few ways to get cheaper specialized answer when you don't need an expensive generalist. A lot of the key components of AI agents coming into view Shipper again reinforced that OpenAI believes that 01 is an important step towards agents. Agents have long been one of the sexiest AI applications, but previous GPT models were likely to get off track if they were left to figure out a task by themselves. 01, because of its ability to reflect on its own thought processes and plan next steps, is a key pillar in making agents that are actually autonomous. That leads to another conclusion for Dan. OpenAI is leading into the race to build different kinds of models for different use cases. The company believes that the most effective applications are going to string together multiple models rather than use one for everything. Now this might seem like a small thing, but I actually think it might be the beginning of a fundamentally different positioning. Everything in AI since ChatGPT launched was just about benchmarking the most state of the art models or doing a different version of that, constrained by size or constrained by the fact that a model was open source. Now we're talking about a new family, a new category even of models. And OpenAI is clearly planting a flag that says leading the AI space isn't just about GPT5, it's about how far you are in this fundamentally new branch on the AI family tree. I think it's going to take a little bit of time for us to really grok the implications of that, and to the extent that they're right, OpenAI will have once again pushed the Gen AI space into its next evolution. Really interesting stuff. Appreciate everyone who was tweeting and sharing their thoughts from the actual event so we could share it here as well. Appreciate you guys listening or watching as always and until next time. Peace. Sam.