Gemini's Next Frontier: 2.0 Flash, Flash Lite Strategy & Real-Time APIs with Logan K from Google Deepmind

We sort of released the experimental first iteration of Gemini 2.0 Flash back in December. Today, we brought Gemini 2.0 Flash, an updated version of it, into production so that developers can actually continue to build with it. We announced pricing, $0.10 per million input tokens, $0.40 per million output tokens, which is, I think, a huge accomplishment for us to pull that off. We're going to have the world's best coding model at Google. And I still believe this deeply. And I think, like, Pro is going to be that model. And a bunch of the reasoning work that we're doing is going to be that model that continues to push the frontier for us. The world needs a platform in which it's hosting all of the sort of publicly available benchmarks and sort of leaderboards and stuff like that. I find it incredibly difficult to just like navigate and get a snapshot of like, how good is this model? There's like 20 random benchmarks here and 50 random ones here. They're all split out over the place and it's just like hard to keep track as a developer.

Logan Kilpatrick from Google DeepMind, product manager of the Gemini API and AI Studio. Welcome back to the Cognitive Revolution.

Thank you for having me, Nathan. I'm excited. I'm hopeful that I'm getting close to the record for the most times on your podcast.

So I appreciate you for always— I think this might be setting the record at 4 if my count is correct. So yes, congratulations. That's rare air and well deserved. So it's launch day. We'll get to everything that you've launched and what we should be thinking about building with it. Quick little detour though, before we get there. You're now part of DeepMind. So Google obviously is a vast company and is continuing to, I don't know, align, restructure, streamline itself to focus more and more on AI. What's the story from the inside on what it's like to be at DeepMind now specifically?

Yeah, I'm super excited about this. So we've been— I joined Google 10 or 11 months ago. Literally from day one, it's been a deep collaboration with DeepMind. DeepMind's gone through all these evolutions over the last few years, transitioning from an organization doing fundamental research to sort of actually productionizing models. And then within the last 3 months with the Gemini app moving over and then AI Studio and the Gemini API is now actually an organization that like end to end does the research, creates models, and then actually brings them to products inside of Google. And I think that's been a shift for them. But from my personal vantage point, like, I think this is the thing that makes the most sense. Like, Being really close to research, and we already were really close to research through this collaboration we've had, but removing as much friction as possible for us to bring the researchers who like actually know how to bring, in many cases, like get the most capabilities out of the models, like bringing those 2 things together makes a lot of sense and it's going to be a ton of fun. So as an external person, like who doesn't care about Google reorgs, which is most of the world, the thing that you'll hopefully see is an acceleration of model progress, but also an acceleration of product progress because we bring these 2 teams together.

Well, it sure seems from my vantage point on the outside that everything is accelerating, and we've had previews of some of the stuff that is now going general availability today over the last few months. And of course, there's been just one advance after another from DeepMind and others over the last few months. Looking back a little bit, what would you say are the customer success stories and/or just like coolest apps that you have seen come online that have been built with the Gemini API in recent time?

Yeah, I think the thing that I'm most excited about and also feels like we have the biggest opportunity here still is around all these like text-to-app creation softwares. And like there's a bunch of examples of these, like Bolt.new just went live, I think yesterday with Gemini support. Cursor has Gemini support now and is using 2.0 Flash. Hopefully we'll see others like Lovable and v0, et cetera, et cetera, have, have that support as well. If you look at just like the economics of running those products, it's extremely cost intensive, especially as like the number of people who know that you can actually do that use case today of put in a text prompt and like get a basically working app slash website for free essentially is like a very small number of people. And it feels like that's like this new frontier. Use case slash product paradigm that I think is going to be picked up across, like, all the big players are going to do this. But I think also there's going to be a ton of startup activity in this. Like, how do you just build domain-specific software for people without those people actually having to know how to code? So I'm really excited about that use case. And I think we have a lot more model progress to still do to, like, become the world's best model at doing coding. But I think even for 2.0 Flash and 2.0 Pro, from where we were 6 months ago, like, it's just an incredible amount of progress. And I think— Trying to keep pushing on progress in the context of Flash without like increasing the price in any dramatic way, I think has been the biggest win for us. Like I've seen tweets of like the LLM usage cost for some of these startups and, you know, you can imagine what those, and it's like on the order of like $40,000 or $50,000 a month. And you can imagine what those costs would be with Flash. It's like probably $1,000 or less or something like that. It's like a 40x cost reduction, which is just crazy. So that's one of them that I'm excited about. I think we also previewed in December the Multimodal Live API, which allows you to do this sort of collaborative, real-time conversational video text interface with the models. And that feels like it is getting us closer to the future that I think we are sort of all been promised with AI, which is this like co-presence that's able to see the things that you do and interact with the services you do. So I'm really interested and I've gotten like the entire spectrum of outreach from people who are using it to help them do coding to like people who aren't developers who are just using this product experience and they're blind and they're actually just, you know, trying to navigate their daily lives using this tool. And it's like crazy that, you know, the product that we're building sort of as a demo for developers to drive adoption of the API is actually helping people who are just trying to live their daily lives with this product. So I think it speaks to where we are. in the adoption of this technology, which is like everyone is sort of waking up and like trying to find the best way to use these tools, which is really interesting to see happen.

Let's put a pin in the Lovable and Bolt discussion because I actually— you're teaming me up perfectly because I think immediately after this episode, the next 2 episodes are going to be with the founders of those 2 companies. I love that. And I am— we're calling it Software Supernova. Because it does seem to be, we're kind of mid-transition, mid-tipping point right now, I feel like, where it is becoming like pretty realistic for people that don't know how to code to create at least basic full-stack applications. Obviously they're not going to create enterprise platforms just yet, but the models continue to advance. So it feels like we're in a very different world quite soon with products and paradigms like that. In terms of living my daily life, I have tried both the AI Studio version, which is the desktop experience where I can share my screen with the multimodal Gemini, and I've also been using on the mobile. And I don't know why, but for whatever reason, OpenAI only has that experience enabled on the mobile app, as far as I know. So I've been trying both of them. This is too stupid, but it's also too good of an example. So I got my kids the Nintendo Switch for the holidays, And we're going through historical video games, right? Because I feel like they're young and the modern games are like, I feel like too overstimulating. And plus, like, let's burn through the old catalog first. We'll work our way up. So we're doing, you know, original Nintendo games, Nintendo 64. So we're playing Mario 64 for the original Nintendo 64, which is like this open-world game where you like go around and hunt stars and whatever. But the frustration for me is I often don't know what to do. So I've been sitting there with the advanced voice mode on and sometimes, you know, also like showing it the screen and just telling it what level I'm in and having it tell me what to do in the game. Like, what is my objective? You know, where should I go to find the star? And my kids are really getting used to this. And my, even my 1-year-old, you know, sometimes now like comes and wants to take the phone and he's like trying to talk to AI. So it is, I think, and for seniors too, you know, my grandmother, I think about all the time. This switch from a you type to it and it gives you text back to it really can be copresent with you everywhere. I feel like I've only still only dipped a toe in that world, but man, does it feel like a very different world for people that are not anchored to their desk all day doing computer work. It's like, wow, that could really just be— imagine putting it in glasses too. I'm sure from the company that made Google Glass, I'm sure there's a lot of thought going into that sort of thing. So I can, yeah, that's my model.

Yeah. I feel like, and my guess is we're actually going to see a lot of this, you know, I think if folks have been watching this closely, like, you know, it takes time for this progression to actually happen. I think like text was the best example of this. Like it was kind of a toy demo a year and a half ago. And it's like now at large scale, like text LLM applications are like broadly being used throughout the world at billion user scale. I think this co-presence, like there's, it's going to take us like to be sort of humble of the place that we're in today. Like, we're going to have to iterate on the API. We're going to have to bring the cost down. We're going to have to make it so that it's like actually something that developers can build. Because you could imagine the like co-presence cost of having AI with you all the time is like probably pretty expensive. Like, there's a reason that we'd limit the sessions to only 10 minutes right now, because there's a whole lot of challenges to like scale this up beyond that and have it maintain memory and state and context of all the things that you were just talking to it about. But it, to me, it's like very clear that that's the direction that we're going to go in. And I think it'll be interesting. It does feel like everyone, you know, again, like 2 years ago was talking about like, oh, everything's a wrapper. There's not a lot of value to be created. And it just feels like that continues to be wrong. Like it feels like there's a new thing. And then like all of a sudden, like all of these new things that were not possible before can now be created. Which continues to just get me excited about like the future and also like making sure that we're enabling those next things so that people show up and create the experiences that don't yet exist today. So it's going to be fun. I'm excited. And hopefully, yeah, send, send over feedback as you keep using the real-time mode.

Cool. Yeah, will do. Hey, we'll continue our interview in a moment after a word from our sponsors. Just taking one more beat on the, like, Cool stuff with the Gemini API. Last time we talked about a couple of things I wanted to check in on the status of. One was really long context. Another is just insane affordability, as you mentioned, like a huge drop compared to other options. And then there's sort of unclear exactly how sort of natively the video is being consumed, but you can just feed video. audio multimodal inputs into the Gemini API as well. What would you say are kind of the status of those things right now? Like, are you seeing use cases where people are like, 200,000 tokens is not enough, I really need to provide more, and therefore, like, Gemini is the one thing I can use? Or, you know, cost-wise, this just wouldn't be affordable if I didn't have this 10 cents per million input token pricing?

Yeah, I, this is a great question. I had a long conversation with Jack Ray yesterday, who's one of the co-leads for the reasoning effort. And he previously worked on sort of the long context breakthroughs with Gemini and enabling that from a research perspective and is now one of the co-leads with Noam on the reasoning models. And we sort of opined for a long time about how It is funny that like maybe the real unlock for long context ends up being reasoning because like long context is extremely impressive and like it's really useful and we do see people using it in production. And I do think there's cases where like 200,000 makes a lot of difference versus like a million or 2 million. But like one of the inherent challenges is just like how many things can the model attend to in the context window? And like it works really, really well if you're just asking questions about a couple of things that are in the context window, but if you're trying to Put together 1,000 different things that are in the 2 million context window. It gets really hard to do that just because of the inherent nature of how the models are trained and set up. I think reasoning is maybe where this starts to change. So I think like us having this really long context window where the models can actually just like think through and perhaps in the future, like use tools and like bring information in and out of the context window, I think is where long context is going to start to make a lot more sense.

And like—

Truly become like an enabler. That's where my head's at from a long context point of view. I think video stuff still, yeah, video, audio still happening natively, image still happening natively inside of the models. With the 2.0 release, we showcased some of the like next steps of this native multimodality, which is the models actually being able to output images and audio. And that's available to early testers. I don't know if you're in the early access program or not, Nathan, if you've played around with it yourself.

I have.

It's, it's pretty good. I think we still, there's like more quality work that needs to happen still. But I think the, where this really starts to, and, and we're about to actually, at the time of this recording, tomorrow we'll roll out Imagen 3 in the Gemini API, which I'm super excited about. And if you sort of look at like, why do we still have state-of-the-art image generation models when we know that the models can sort of natively have this capability themselves? Like, there's definitely this quality trade-off in some domains where you trade off quality for world knowledge. And I think the world knowledge case is like actually where this is really interesting. Like there's a ton of models out there that generate like really, really pretty pictures and can do all types of cool stuff, but they lack this world knowledge that the Gemini models have because it's this native capability that's coming as part of the training process. So I think there's going to be this whole new, you know, onslaught of use cases, which like didn't work before because the models weren't smart. They were just good at generating images that we'll see happen with native image generation. So. Hopefully we'll get that out soon as we continue to hill climb on quality.

Cool. Yeah, that's interesting. I think your point about the need to have a long chain of thought in order to really fully take advantage of super long context is quite interesting. I have been noticing for myself in like application development, I do find like I want the hardest thought, right? So I'll go to the model that's going to think for me the longest, and then I sometimes have to contort myself or contort my inputs to get it to fit into the context. But I do feel like, yeah, I wouldn't 10x dump context into a model that is going to immediately jump to the answer. And I hadn't really put together why that might be, but I think that's a pretty interesting— hypothesis that I could imagine. I look forward to dumping, you know, my full million token codebases into the Flash reasoning pseudo-rev later.

Yeah, I'd love to see if you have use cases that haven't worked well historically for long context. Like, I'm super curious if like also folks in the audience have, yeah, like you can do the compare mode right in AI Studio, try it with like 2.0 Pro with long context and try it with reasoning with long context and see whether or not like the, the extra reasoning steps actually make a difference and My intuition is that it is going to, which is exciting.

Yeah. Cool. Okay. I like that. Have you seen anything passive? This is another area where I feel like Flash, you know, very minimally famously said more people should try to spend a dollar a day on Flash. And, you know, that is a lot of tokens. And so it's kind of like you need passive applications to for most people to get there. So I've been thinking about— a couple of people have asked me about, could I use a vision language model to monitor my factory floor for safety incidents or policy violations? Or my grandmother lives in a senior living community, and the seniors hate to wear their fall monitors. So that's a constant battle. And they probably, a lot of them would, to the degree that they have, and my grandmother does have the ability to make her own decisions on this, Probably would accept not having to wear that thing if there was another sort of visual monitor that could keep track of her state at any given time. Have you seen anything like that where people are just kind of truly passively, almost Internet of Things-like sending signals into the API?

This is a great question. I think it is so core to my thesis of what's going to happen for a lot of these domain-specific Can, because like, and if you take a step back, like, how would people solve that problem today? Like, you would need to go and buy some custom software that does that, which probably is expensive. It might not generalize well. Like in my past life, I was a machine learning engineer and we did a bunch of stuff with security cameras and like what it would look like. How would you track someone moving from one frame where a camera is visible to another frame and how to keep object permanence of that person and like. It's incredibly hard. Like, it is not an easy problem to solve with traditional computer vision technologies. And I think vision language models, like, just do this task incredibly well. And the cost basis is now with Flash, like, so low. I think I haven't talked to anyone who actually has this in production, but I have to imagine that this is like the, the opportunity that people are going after. And it's not, it's actually not just the, like, bounding box use case and image understanding use case, which I think is really, really powerful. Like being able to know like, here's where an object is. And we have a good demo of this in AI Studio. If folks haven't tried the bounding box capabilities, if you go to starter apps, there's like, I think it's spatial understanding or maybe it's just called bounding boxes. I forgot what the name of it is, but there's an example in there and you can put in images and like ask, you know, to identify the objects and it'll throw bounding boxes pretty much just like you would get out of like one of those custom bounding box models that you could probably find on open source or something like that. I think these use cases just take time, but I also think my guess is as vision becomes more and more prominent, like we're going to see the whole like YC startup wave go after all of these ecosystems and industries where they're using domain-specific vision models and not using like a general purpose model. And your cost basis is just going to be wildly different. And also you unlock all these use cases, which those models are just like not actually capable of doing. They're very, very rigid and like can't, you know, be fault tolerant in a lot of those cases. So I'm super excited about this.

So let's get into then what you're launching today. I saw an interesting tweet and you mentioned YC and the idea was basically that every time the frontier advances, the, some, you know, all the, all the YC companies go and kind of see like, can this work for my use case? And the report was every time a new model comes out, some subset of the current YC batch company's products start to work.

I saw this.

And others are kind of just like waiting and continuing to build all the other stuff with the expectation that they're going to get a model that's going to tip them from kind of not quite working to working. So what are you launching? And to the degree you can speculate, what is it going to make work that wasn't previously working?

Yeah, it's a great question. I think one of the— and just to draw a broader point, I think one of the really interesting observations I've had is thinking about, like, because of how much excitement there is about AI, how the, like, resource constraint has not made people think as deeply as they need to about this problem. And I think, like, maybe this is me somewhat, You know, just because of like our Gemini models, like being at the frontier of cost per intelligence, if you look at that as a ratio. But it is really like the, the, the YC companies are just like so well funded that like we bring the cost of intelligence down some reasonable factor. And like it actually, in a lot of cases, like doesn't move the needle for these startups, like just because they have millions of dollars. And I think it's interesting to see what the outcome of that trend is going to be. And like, if I had to guess, actually, I think it gives a lot of power to these individual developers who don't have this large amount of financial backing from tier 1 VCs, as an example, that can actually push the frontier of some of these use cases and capabilities, which I think is a really interesting and cool phenomenon. But to answer your question specifically about what we're launching, it's a whole suite of Gemini 2.0 models. So we released the experimental first iteration of Gemini 2.0 Flash back in December. Today we brought Gemini 2.0 Flash, an updated version of it, into production so that developers can actually continue to build with it. We announced pricing, 10 cents per million input tokens, 40 cents per million output tokens, which is, I think, a huge accomplishment for us to pull that off. And then announced a preview of Flash Lite, which is the smaller variant of the Flash model, which we'll intend to make available for production use very soon, and the pricing for that. And then released the, the sort of experimental variant of 2.0 Pro, which is the most capable Frontier model we have, joining, rounding out the full offering with the Flash reasoning model. So now we have the sort of reasoning Flash model. We have Flashlight, the smallest model, Flash, which is sort of the most, you know, performant cost trade-off-wise, and then Pro, which is the most capable model.

And only the— well, wait, so let's go down the availability too, because You guys have high— in my typical work, which is usually focused on proof of concept type stuff, I mean, honestly, I love my life these days. I never have to really worry about the harder work of making something actually production ready and much more get to focus on that easier, faster ascent portion of make the proof of concept work. So when I go to the AI Studio and I grab code and go do stuff with it, With one exception, which I'll mention maybe in a minute, I basically never hit rate limits. So even when something's still in kind of preview, for my purposes, the limit is there's enough headroom there for me to do usually all the testing that I want to do. But if I'm Bolt or Lovable or certainly Cursor or whatever, then they would be hitting those limits. So what is super scalable now versus is still in kind of experimental access and yeah, just give us like the concrete on how much of these different models we can use.

Yeah. So 2.0 Flash during experimental, I think the AI Studio UI has different, has like some, we don't publish the limits because it can change dynamically based on how much capacity we have. We don't usually change it, but we sort of try not to publish it just in case we need to and don't want to make people sad. But in the API, I think it's 10 or 15 requests per minute on the free tier. And then I think like 4 million tokens per minute, which is a lot of tokens per minute, to be honest with you. So I think that's probably why a lot of people don't hit. And unless you have like users or are trying to run a, like people, a bunch of people have like internal evals and things, or like people who are running leaderboards as another example. Those are usually the use cases where people reach out and are getting rate limited. So with the production availability, you can now, if you're on the paid tier, the free tier stuff all stays the same. You can keep using the model, keep tinkering, doing all that. If you're on the paid tier, you can get, there's, there's no daily RPD limit. So you can send as many requests every day as you want. I think it defaults to 2,000 requests per minute and still stays at 4 million tokens per minute. And then we're rolling out tonight at midnight, we're rolling out new quota tiers. So As you continue to scale usage, you can unlock things like 10 million tokens per minute, 10,000 requests per minute as well, to help those who continue to need to keep scaling up.

It's wild. The infrastructure behind that is truly an incredible accomplishment.

Yeah, lots of TPUs to make all this stuff happen. And lots of like, it's, I think for people who aren't at LLM companies, you don't think a lot about this, but it is, there's just a lot of complexity in like how many models there are. And like, we, you know, for, you know, I see a lot of memes online about, you know, the bad naming conventions that we have with our models and stuff like that, which I appreciate. But like, I think it actually hits at a different point, which is like, there's, there's just so many different model variants. And it's really difficult. And this is why, like, a lot of the feedback about our experimental model release chain has been like, hey, we love these models, let us use them in production. And the challenge is we have to be really picky about what model do we use in production, just because like, The compute footprint that it takes to like actually make it so that, you know, the cursors, the bolts, the, the, you know, other YC startups, the developers trying to scale and build companies can get the compute they need is just like, it's hard and it takes a lot of compute. So we have to be a little bit more intentional about how we do it. Ideally, we would just GA every model and like everyone would take every model to production and we wouldn't need to worry about it. But yeah, there's just a lot of constraints with, with doing that.

Hey, we'll continue our interview in a moment after a word from our sponsors. Yeah, if you're going to support millions of tokens a minute, I can imagine why you'd need to be prepared for it. So maybe help me develop my intuition for how should I think about Flashlight as it relates to Flash? Because my totally candid initial reaction was Flash is already so cheap, And pretty fast. Flashlight comes in 25% cheaper and presumably faster, but also like slightly weaker. You know, you've got the table, of course, of all the benchmarks, and it's like a little less on most of them. Have you been getting demand for like an even cheaper, faster model than Flash? That seems like almost— that's like hard for me to wrap my head around, to be honest.

Yeah. So I think the positioning is around twofold. One, we wanted to give Like technically by default, the 2.0 flash price is more expensive than the 1.5 flash price. So, and especially given how much we had leaned into the sort of the low cost per intelligence of the models historically, like we wanted to give people a 1-to-1, like if you were, if, you know, 7.5 cents per million tokens was like the thing that was enabling your business and we showed up and said, hey, by the way, now it's 10 cents. It just didn't feel like a great story for developers given how much we leaned into that narrative. So being able to have an option. Where, you know, it really was a direct, it was not only a better model, but was the same exact cost as you were getting before. And I think for the 2.0 flash model, it just like, because of a bunch of constraints, like just did, it wasn't going to be possible for us to keep that same level of price. So really about that, like wanting to make sure that we didn't sort of mislead customers into thinking that they were going to, you know, be able to continue to push on whatever the Pareto frontier is that they care about with cost, cost intelligence. And yeah, like, I think there's also like some other things that as we think about how do we make the cost lower for these models, like there's a bunch of features that aren't supported. Um, like more of the like high-end things. Um, like that, those flashlight models, for example, will never be able to do, uh, native image generation or native audio generation. There's a lot of things that we can do to keep the cost down as far as like serving those models at scale, uh, which the 2.0 Flash default version Doesn't have. It's also kind of similar to, I don't remember when we talked last about this, but we also put out that Flash 8 billion parameter model. So you can kind of think about, uh, we're not releasing the size of the Flashlight model publicly, but like you can think about that as like another version of the sort of small model train that we'd previously done with Flash and with Flash AB. Yeah, Flash AB was like the most, if folks have looked at OpenRouter before, like it was the most high token volume usage model, like on, on OpenRouter. router, which again is like a reasonable proxy for model usage in some contexts. So like the clear feedback from us was developers love low-cost models and like there's a huge amount of new use cases you can unlock by continuing to reduce the cost down. So I think, you know, if we could have made an even cheaper model, I would have pushed for that as well.

Yeah. Interesting. I would love to hear if anybody's listening who fits that description of a 7.5 to 10 cent price change per million tokens would have made a meaningful difference for what you're trying to do in the world, reach out to me. I want to hear that story. I really, I find that pretty hard to imagine. I can easily imagine how people might just look at a menu and say, I'll take the cheapest one because I'm just processing whatever and extracting addresses out of whatever stream of data. So I can imagine choosing the cheapest by default, but I have a hard time imagining how a business model gets disrupted by that kind of change. But I'd welcome that story.

I think you're right about this. I think a lot of this is just like about how we tell the story to the world of how we show up. And I think like it'd be easy for that to become like the narrative we don't want, especially given how much we've pushed on reducing the cost for developers is like, You know, Google's raising the price for developers. Like, that's not the narrative we want. So it was really important to us to like preserve the continuity of making sure that we have that low price point available for developers who care about this. But I think generally I agree with you, like, especially given how much the other signal that we've gotten is like, if you have better models, like the story of a lot of the models in the ecosystem is if you have great models, people will pay for them. Like, that's actually not the limiter in a lot of cases. Yeah, for sure. For sure.

They're all cheap compared to human labor. That's I think, you know, such a striking 2x4 to the forehead, the fact that it is often kind of glossed over. Let's do the Pro side. So how should I think about Pro? You could compare it to Flash, you could compare it to other Frontier models that are out there, but how should I kind of understand it in the increasingly busy constellation of available models?

Yeah, I think Pro is where you're sort of not Bound by costs in a lot of ways. Like we, we didn't release the price of the pro model yet, uh, because it's still experimental, but it's gonna probably roughly follow the similar patterns to what some of the pro models have been in the past, which is like just a lot more expensive. So it has to be. And, and I think there's actually, you know, if you look at like what is the traditional advice for developers as they're building sort of the frontier applications is go with the model that's best. Even if it's a premium, make your use case work and then figure out a way to sort of bring the cost down over time by switching to a smaller model or optimizing, doing fine-tuning, whatever it is. So I think it's important for us to like continue to honor that flow, which I think actually works. Like there's a lot of that. This is the default experimentation path that developers go on today. I think specifically like the use case where we're seeing the best performance relative to the other domains is encoding. And I think like we're going to continue to, and I had a glib tweet you know, a while, I don't even remember when it was, the day before O3, I think got announced about how we're going to have the world's best coding model at Google. And I still believe this deeply. And I think like Pro is going to be that model and a bunch of the reasoning work that we're doing is going to be that model that continues to push the frontier for us in coding. And like, it's a domain we need to win, especially if you believe in, and we were talking about the sort of text to app creation stuff before, especially if you believe in that sort of trend continuing and the acceleration of developers continuing from like an internal software engineering productivity standpoint. So I think that's probably the best use case for it. It still has 2 million contexts. Maybe it'll have longer contexts in the future. So that might be something as well.

Yeah, we've heard whispers of infinite context.

You and I were sitting in that room together, I'm pretty sure, when Jeff Dean said we were getting infinite context at some point in the future. He didn't put a date on it at I/O last year in that session we were in together. Yeah, hopefully. I think that's going to be a huge unlock for folks once we land it.

How would you guide people to think about the non-release of— it was Ultra, right? Was the largest scale. We've kind of seen the same thing, obviously, from Anthropic, where it's like, wait a second, what happened to Opus? Is there anything you can share to help people just generally understand Why we seem to have gone from small, medium, and large to now just small and medium?

Yeah, that's a good question. I think so. The historical context on Ultra was basically trying to prove out the research direction that this scaling was going to continue. The reality is they sort of proved with Gemini 1.0 when the original Ultra sort of model candidate came out that that was the case. But also then like there was just all of this like continual rapid innovation of like making like all of a sudden the, the Pro model was better. Then all of a sudden, like now I'm pretty sure Flashlight is better than the original Ultra model. So it just becomes like this question of what are the cost trade-off, um, infrastructure, uh, equation to keep in mind. And like, you know, you could imagine a world where We have an Ultra model and it's 5% better on every benchmark and it's, you know, 5 times larger and costs 5 times even more than Pro. And then like, you just start to think about, like, do the math and think about like, what, from a research perspective, like, where does it make sense to spend our time and energy? And also from an infrastructure footprint perspective, where does it make sense to spend our time and energy? And like, we continue to see gains on like making models. Much higher quality at the same size or even a factor of a size less than previous models. And I think now, especially with reasoning, there's even more question marks in my mind of like how much, like if we could get a model that was even 10% better on every benchmark, like, does that make sense given a world where reasoning has like so much scaling and like there's just a lot of low-hanging fruit work that we can do there?

I think.

Back to the conversation I had with Jack yesterday, we talked a lot about this pre-training scaling and reasoning scaling. I think Google is still scaling pre-training. So this isn't that we're not going to keep scaling pre-training. I think whether or not we officially release an ultra model, I think, is an open technical research question just because of all of those constraints. And it's also kind of tongue-in-cheek where, like, you know, We could rename all the models and say that we have the Ultra model if we wanted to. And like, maybe that would have been the right thing to do historically and make Pro Ultra and then remap all the other names. But I think trying to honor like the, the essence of what was intended through that Ultra model, I think is part of the thinking.

Yeah, I'm on the side of anything that maintains any clarity in naming schemes. So I'm with the consistency. That's the only reason I even know to ask this question, right? I mean, if it was all renamed, I would even just be even more confused than I am. So, okay, let's go back to coding and maybe think about the likes of Bolt and Lovable and Cursor and Devin, increasingly many options. I can't even keep up with all my AI coding paradigms these days, and they probably can't keep up with all the models. So the question I have for you is like, okay, we got all these benchmarks. Probably the new one versus the old one kind of wins on some benchmarks, loses on some benchmarks, whatever. An interesting thread that I wouldn't say is dominant, but has definitely got some weight behind it over the last, say, month or two has been, yeah, O1 is great, even O3 Mini is great, potentially Gemini Pro is great, but for some reason people still seem to think Claude 3.5 Sonnet new is the best coder and the best coding assistant. And they haven't, in fairness, had time to compare it against necessarily everything. But how would you suggest people think about this? The simplest thing to do would be like swap in a model in exactly the same situation that they currently have another model performing the best. But you could very easily say, well, you're probably leaving some performance on the table because these 2 models are probably not going to have maximum performance under the exact same conditions. They're each gonna have their different conditions that elicit the best from them. But how do you find it? You know, and how do you know how much time to invest in that? And like, yeah, I don't know. It's, it seems very difficult and it's, it's even difficult for me as just like an obsessive hobbyist. So what's the, what's the best practice or how, how should one think about absorbing a new Gemini Pro and comparing it to a, a Claude 3 5C and an O3 Mini in today's world?

Yeah, this is such a tough problem space and I just have an incredible amount of empathy for developers and founders and people who are building stuff because like there is no silver bullet. I'll give a couple of reactions to this like general thread. First, like my own personal example that sort of, I think underscores that the second point that I'll make, uh, I was doing a bunch of, you know, I was doing the normal web developer thing the other day of trying to get the corners of a table rounded. And I was, I was smacking my head on, on this problem, uh, because of a bunch of weird constraints in the environment that I was working in, uh, and using the Gemini models. And, you know, I became fed up at one point and I was like, you know what, maybe the Gemini models just aren't that good. And like, I'm gonna go try Claude and see what it does. And I, I, you know, went outta the environment that I went out of AI Studio, went into Cursor, tried it on this like very simple prompt and it worked. Single shot, everything just worked. And I, I started to ping a bunch of people frustrated and I was like, this is, you know, we need to keep making coding better, yada, yada, yada. And then I went in just for my own sanity. I went and reran the exact same prompt and stuff that I did again with the Gemini models. And it also worked. And I think like, this was a good example of like, I think just like, like the prompt was really like, I was just doing a bad job prompting and I pulled myself out of this environment that I was in of this like iterative loop with the model. And I started over from scratch again and formatted the question and the context in a different way. And like, it worked with both models and I think This point underscores this like vibe slash like incredibly unstructured scientific way in which people make these decisions today. And there's like a lot of the, I won't name names of products, but like work super closely with lots of teams who have LLMs in production. And you would be surprised if like insert your favorite LLM product, like in a lot of cases, how few evals people actually have to understand, like, what are the metrics that matter for us as we build our product and service? And I think there's like 2 ends of this coin. I think like one, the world needs a platform in which it's hosting all of the sort of publicly available benchmarks and sort of leaderboards and stuff like that. I find it incredibly difficult to just like navigate and get a snapshot of like, how good is this model? There's like 20 random benchmarks here and 50 random ones here. And I got to go, you know, look at Minecraft because that benchmark's really cool. I love that benchmark, but like they're all split out over the place and it's just like hard to keep track as a developer and like to To your point, like, this is my job. This is what I spend every waking moment of my life doing. And like, it's still difficult to keep track of this. And you could imagine for people who have much less time and are, you know, doing other things, it's just hard to keep up with it. So I think someone needs to build this platform. And maybe this is like the, you know, you and I should do the Y Combinator, like, call for startups thing. And like, that is the call for startups. Like, build this platform to help people really bring all this stuff together. And then separately, I think And the folks at Kaggle actually are pushing on this notion of having personal evals and being able to build a platform in which, as new models are released and made available to the world, you can sort of just run your personal eval, will just run behind the scenes on those new models. And you'll just get an email that says, hey, this model, based on what you've told us, might actually be one that you should spend your time checking out because it's Really good at these things. And, you know, those are the things that you indicated you care a lot about. So I think that type of platform and product experience, like taking the burden off of developers to sort of just do this is going to be awesome. The challenge in that case is like, you still have to make your personal eval to begin with. But I think that like one-time cost is a lot less than like having to do all the spin-up cost every time a new model hits the market.

Yeah, it almost seems like if I. we're going to take one sort of practical rec out of all that for developers, I might say let your users choose, because at least that way you can get some data and they can sort of feel a little more agency. And maybe there's something good to find there. I like the personal benchmarks too, but the things I care about typically don't have a right answer. Like at Waymark, it's always the same challenge. Like, what makes a good video? We can have a language model judge, but now we're in Hall of Mirrors territory, and we just don't have any. I'll cop to being like, we don't really have evals that work for us. We can detect outright failure to follow the structure, or it's too long. So we have some guardrails that could be like the clear thou shalt nots, I call them the thou shalt nots of this task. Like we can, yeah, we can detect those, but past that it is really still vibes. And yeah, it's interesting. We don't actually let our users choose, but that could be an interesting— our users also don't even know, you know, they're not in this world. So the developer's use case is more what I had in mind when I was thinking, let your users choose.

I think that's a, it's a trend that you, you hit the nail on the head. Like, and I don't even know how much of this is like a conscious decision that founders are making, but if you look across like Like, again, some of the, you know, choose your favorite LLM products, uh, with the exception of Waymark, uh, like developers are, uh, and, and like actually end users have this choice today. Like you go in, there's a model dropdown and, and most of those products and like a lot of them that are not created by large model providers have most, if not all of the models available. And like the, the sort of timely example of this is Copilot. Like Copilot now, you know, was historically just powered by GPT models 2 years ago. And I think like the developer community and like where the world has moved, it now has a model dropdown and you can choose the Gemini models or the Anthropic models or the OpenAI models. And like, I think more and more products are going to go that route. Also, as like the— this is maybe corollary to a point that a lot of people make, but like, I think, you know, people talk all the time about like the commoditization or like the contraction of like the, the delta between these models. I think that's actually not true. I think there's like a lot of this weird nuance that is going to continue to sprawl out over time where like You will get substantively different answers from, you know, insert whoever your favorite model provider is over time. Even if like the capability is similar on paper, there's still all of this stuff that will be different. And I think it'll be important that people continue to have the choice because I think that, that subtle difference makes a lot of impact on like the end product experience people are going to have.

Yeah, no doubt about that. Even DeepSeek over the last couple of weeks has, I think, shown a very different profile and is probably the most, and perhaps not surprisingly given its source, is it's like closest to base model is probably my best description of it. And for that reason, it can write sometimes in ways that are extremely compelling, but it's also much less behaviourally refined in the way that the Western, at least top-tier model provider-created models are. So, yeah, a lot is— and that's all with very similar benchmark results, right? So there's a lot beneath the surface that remains to be unpacked with any major new model release. Well, let's talk for a minute. I know you got to go before too long. Last things I wanted to cover were briefly fine-tuning, because as a developer, that's something I'm always interested in and would love to know the status there. Then maybe what's up with reinforcement learning at Google? We've seen the thinking model. It struck me that there wasn't much actual mention of reinforcement learning, whereas the other developers are coming out and saying, we did a reinforcement learning model. It seems like DeepMind has kind of positioned it differently, although I assume that a lot of the same techniques are happening under the hood, but I don't know. And then I was actually going to ask for your call for startups too, because I know that you have recently raised a Solo venture fund and we'll see if we can get you some deal flow. So yeah, fine tuning, reinforcement learning, your fund, take as long as you have.

Yeah. Fine tuning is something like, I, I'm, I continue to be incredibly bullish on fine tuning. I think the future where everyone in the world is using their own fine tuned model, a version of the model that's, that's sort of has the context that they need it to have. And the context isn't being like overly Overly impactful on the priors of like how the model makes decisions, I feel like is like the rough estimate of like how I, or the rough explanation of how I think about fine-tuning. So we don't have it yet for 2.0 Flash. I think we need to. I think we, you know, we've been having a lot of internal discussions about like, what is the, you know, size of investment we want to make in fine-tuning. To me personally, I think this is like one of the biggest opportunities. Your point about developers wanting this, I completely agree. So we'll keep pushing on it. It's not available yet. Hopefully it will be available soon. And like, even more than it is, like, there's a lot of limitations on how we do fine-tuning with 1.5 Flash today. You can't do images. Yeah, there's a bunch of rough edges. So we need to solve all those things and make it so that it's really a first-class experience. Your second question?

Reinforcement learning.

Reinforcement learning. Yeah, there's— RL is part of making those reasoning models. Part of this historically is that, like, why have we not talked a lot about it? We did the sort of normal low-key approach to doing the release, which is what we've historically done for our experimental models. I think we're actually, as the world is getting more and more excited about what's happening with reasoning models, we're going to start talking a lot more about that work. And yeah, I'm excited for us to tell more of the story about the work that we're doing. It's actually like the reasoning narrative inside of Google is one of the narratives that gets me most excited about the direction that we're going in. There's just been so much progress and like a ton of breakthroughs. So it'll be hopefully soon we'll tell that story to the world and yeah, we'll get you some people on the podcast to talk about it hopefully soon. And then yeah, startups, like I still, this, you know, this is the moment to build impactful, interesting companies. Vision is one of those things that I'm really excited about. Like I still think like take the entire ecosystem that's built on domain-specific computer vision models. I think all of that is up for grabs with vision language models. I think I think there's gonna be a huge amount of startup activity in that space. I think reasoning is going to make agents work, which I'm excited about just because of like how many, like I haven't, you know, I, I'm not an agent investor in a lot of ways, but like I think there's just so many companies that are trying to tackle problems with agents and like a lot of them just don't work today. And I think that's actually gonna be the biggest unlock we were talking about before, the like new models enabling some percentage of Y Combinator startups to like actually work all of a sudden. And I think reasoning is going to be that continual breakthrough that, like, more than anything else that happens in the next 2 years is going to make those companies, their products actually work, which I think is really exciting for them. I think it's exciting for the world as we sort of figure these things out. I think there's a whole class of startups which I think don't yet exist. And I've started to talk to some of the people building these companies around, like, how the fundamental nature of the internet changes in a world where you can't assume the only thing that's visiting your website is a human, with the exception of like, you know, index crawlers to make your content more discoverable, which historically was like the only non-human thing that was visiting your website. The like social contract of the internet is not set up for that. And I think it's going to be very, very interesting to see how like fundamentally things change with like websites being locked down or like what the protections that are in place are, or all of that type of stuff. I think there's just like all of this new, like very basic how we engage with the internet experience that is going to change over the next few years. And I think there's a lot of people who have businesses and websites and companies that are going to have to, like, they're not going to be able to solve that problem themselves. They're going to need somebody else to solve that problem for them. And I think there's some interesting companies that can be built to enable that sort of Yeah, how your website talks to agents, attribution happens, how you're able to capture your slice of the value creation that's taking place. So I'm interested to see what happens in that space.

Yeah, that seems like a really good candidate for what's going to change the world most in the not too distant future. First of all, it seems like it's really likely to work with a reinforcement learning paradigm because you're going to get a pretty clear signal from a lot of tasks as to whether or not it succeeded. And yeah, I've been watching like Payman a little bit recently. Any, any like specific companies you would suggest people look at or anything? Any specific problems that you think are like most come pitch Logan if you're working on this?

Yeah, I think evals is the other one. Like actually going back to this point, like you, you making the comment that like even for you as somebody who I would classify as like incredibly sophisticated in this space, like you understand what's happening in the ecosystem. You understand why evals might be important. Like the fact that you can't, the fact that it's so hard for humans to articulate like what is the taste and your perspective on some of these problems into something that can programmatically happen, I think is really, really interesting. I don't know how that problem gets solved, but actually if you, if you've spent too much time in the evals rabbit hole like I have, then one of the big things is actually like Like most of the problems in life end up being eval problems. Like if you look at like, you know, it's just very interesting if you like follow that chain of thought, how, yeah, how things end up happening.

Yeah. For now I just do expert demonstrations. Like for Waymark, we just go and say, creative team, write us a bunch of good stuff, fine tune on that, you know, or put those into a few-shot examples and hope for the best. And, you know, from there it becomes vibes. But yeah, it would be nice to have something better. So yeah. Well, where can folks find you if they want to pitch you or if they have— and you're incredible, as everybody knows, online with responding to questions, concerns, issues around the API. So I don't want to bring more of that to you than you already have, but where should folks find you if they want to either point out an issue or pitch you a startup?

I saw someone respond to one of my tweets the other day and they said, I miss the old Logan. He used to reply to all of his replies on Twitter. And I was then went and looked at my, how, how much time every week I spend on Twitter. And I was like, it's the, my, my time doesn't scale here. I'm already putting in—

Yeah. You're gonna need an agent.

Yeah. Way too many, way too many hours. But yeah, Twitter, LinkedIn, everywhere that the internet exists. Uh, hopefully I'm, I'm there helping people with Gemini stuff.

Cool. Well, Gemini 2.0 Flash is out today. General availability. It is good, fast, and cheap. So definitely one to check out. Logan Kilpatrick from Google DeepMind. Now, thank you again for being part of the Cognitive Revolution.

Thanks for having me, Nathan. This was fun.

Always a pleasure. It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.