Google’s New Stack: Gemini On-Prem, ADK, Open Models -- Interview

Hey everyone, this is Mohammad. This video is going to be very different than my normal videos. In fact, this is the first time I am talking on camera on this specific channel. So I'm trying something new and hope you all are going to like this. So in this new series, we're going to be talking to industry experts, those who are building with LLMs, those who are enabling developers to build on top of these systems. And the goal is very simple. I want to bring industry insights to developers and enable developers to build thoughtful systems without all the hype that we are seeing all over the place. So a few weeks ago I was at Google Next and I had the opportunity to sit down with Matt Thompson, who is the director of Developer Advocacy at Google Cloud. We talked about Google Cloud, Gemini, Gemini on prem, which for some reason nobody's talking about, but I think it's one of the most fascinating ideas that we have seen in a while. We also discuss ADKN a lot more. So I hope you're going to like this conversation. Hopefully this is going to be first of many conversations that I'm going to have with industry experts and would love your feedback on that. Also, if you have any specific guest in mind, please do let me know. Anyways, I hope you enjoy this conversation and I'll see you all on the other side. Thanks.

How are you doing, Matt?

Today I'm doing really well. It's been a great week. I got to admit, it's Las Vegas. There are a lot of people here. I'm getting a little tired of people. I will always love developers, but I'm a little tired of all the other people that are here. It's a lot of walking. We've had some good food, we've had some good weather.

Nice.

Had some great demos. So I've enjoyed it, but I'm tired.

Okay, well, it's the last day, right? So you get to go home after this.

That's exactly right.

Yeah. So there were like a lot of stuff released or announced. Right. When it comes to developers, like, what do you think what developers should be excited about?

So there, there are a lot of things that developers should be excited about. But I'll. I'm going to. Because the list is very long. I'll pick two to start with.

Okay.

So the agent developer kit, adk is actually a huge opportunity for everyone to get started with agents. And I say this because we're going to. It's going to change so much of what we do. So if you think of enterprise apps today, the really old model was you put up like an XML schema, somebody goes and queries it, figures out what you do as a service and then, you know, queries it to go figure out, you know, what they're going to do to call you. That's, that's old school. So then we add in, you know, we're starting to, okay, we use the web and use rest APIs for that. But it's still the same notion of I got to go figure out what you do and then be able to do it. With agents, you're actually going to go with a purpose. Yeah. So you're going to build software and say, I need you to go do X. And you know, the classic ones that we've been talking about is you're going to book an airline ticket, you're going to go buy something in retail. Or interestingly, the B2B scenario is if I get, you know, components from 500 different suppliers, I now no longer have to query 500 different, you know, things, I can actually build an agent smart enough to be able to say, here's the manifesto, what you need to go get for us on this kind of timing, once a week, once a month, whatever, when a certain inventory level gets a certain amount, and building an agent then could go do that proactively for us, change tremendous time and then turn it around to the supplier and go, why wouldn't you build an agent to actually then allow agent to agent interaction? So the reason why I think ADK is so interesting is it's the very basis of that. It allows people to start playing with agents. I can't tell you what the greatest winning combination of agents are going to be. Nobody can right now. But allowing developers to start playing with Azure, so that's number one. The second thing, and this is less of a product announcement, but more of kind of what we signaled is the platforms getting more intelligent. So the notion is we actually want people to be building applications without having to worry about a bunch of the things the platform's going to take care of for you. One of those that we actually talked about here, which I think is really interesting, is imagine if you didn't have to even query the platform for what your cost is. It just told you once a week, once a day, once a month. That way if you're using models, it's not going to run out of control for you. Like you can say, I want to know when I'm at 90% of my, of my capacity or 90% of my, you know, whatever I put in for cost that to Me is the signal that's most important for developers to listen to. Because when we think about the hyperscalers, everybody can kind of figure out in their minds what AWS is going to be. Everybody can kind of figure out in their minds what Azure is going to be. What Google Cloud's going to be is the most intelligent cloud partner. And there's just nothing else that's to going be close to that. And that's what we signal. That's what I'm super excited about.

Coming back to the Agent Development Kit, what's kind of the philosophy behind making it open source? Is it more of a. So that it's widely adoptable or getting feedback from the community?

So you're a friend, so I'm going to be honest with you here. Sure. We want adoption, we want feedback. All those things are true. The reality is there will probably be 1001 different agent frameworks and different ways for agents to engage. What you're seeing Google do is go out early, what you think is early and try to get momentum around one way of doing this. We think it's the right way. There could be other ways. One of the things that we've said on stage here is that we expect to interact with a lot of different agent frameworks. So part of the thinking behind ADK was we're going to give you not an abstract but a fairly generalized view of how agents can be built, how they interact, how they work other deployed and managed, and allow that to then work with any other framework that people come up with. So one of the reasons for putting it out in open source is to enable that early adoption. Get people playing. The most important piece for us right now is get people thinking about what this world's going to look like. I can name 500 different scenarios on how agents are going to work. I don't know any of them are right. So what's going to happen is we'd like the developer ecosystem to go play, then we'll start seeing what's winning.

Okay, okay. But is it going to be like a. More of a opinionated implementation at this point?

I think there are pieces where we're opinionated, so agent interaction, how they communicate, how they can be managed. Security is a big one. We haven't talked about a lot here. But one of the things to understand is, and this is where we start getting into kind of the ethereal stuff, rogue agents, like how does the system manage a rogue agent? Can. Can a rogue agent be built such that it can go destroy other services? Of course Not. And so the notion is, at least on Google Cloud, agents will interact with agents in a well defined and secure way. And so part of the ADK going out there is starting to get developers to think about one, we don't want anybody building malicious agents. That's not the goal. Right. But we also don't want people to have to suffer from that. So we're putting constraints around agent capabilities. Agents can't destroy a port, for example. Right. Things like that. And so that's a, that's part of the thinking behind this.

Okay. Yeah, no, it's actually really good that you brought on the security because like, I think some of the other frameworks probably haven't really paid attention to it. Right. And it's, that's a really big concern now. Something which everybody is discussing or asking when you guys announce the agent to agent, how does it play with agent mcps?

So, and, and this is a great question. So in the developer keynote, we talked about two different levels of engagement. There's, there's MCP and there's a. To a. Yeah. So agent to agent. I would love to give you all the greatest thinking that we've got on this, but the reality is we're still thinking some of this through. So is there a time in the future this is not an announcement by Google? Is there a time in the future where we see agent to MCP engagement or MCP to agent engagement? Possibly. Right now we see those as kind of, I don't want to say different, but maybe two different layers of engagement. So to answer your question, I would say hold tight, let's see what happens.

Okay. Okay. Okay. I want to switch gears towards the cloud and one of the announcement, which I think is very important, but it hasn't really got a lot of traction or notice. Is Google distributed cloud split, specifically Gemini on prem. Are there any concerns about leakage of the model weights and things like that?

Right.

Because I think that, that, that's, that's a huge thing to enable.

Yeah. So one, I'm going to say this is a great question. Like, like you, you, I don't know how you've listened so well this week, but that was you. You've already internalized what we announced and now are asking really good questions about this. And I'm going to give you, I think there's some interesting answers here. I'm also going to tell you there's a lot of this. We have to go figure out stuff. Right. But let's talk a little bit about the philosophy. So if we Go back a little bit. We had a product called Anthos, which is now something. We use it for something else. But the notion is, could we deploy Google Cloud on other infrastructure, your own infrastructure on prem, et cetera? The reason why we did that and the reason why we had that program was because we see the need to take Google Cloud services and either take them to the edge or take them to other infrastructure. Okay, so that's a thought that's now five years old with us. Okay, so when we start thinking about Gemini, and again, it has nothing to do with this other infrastructure, but when we start thinking about Gemini, that thought was already native to us. And so the question was, is Gemini, let's say, popular enough, or is the price performance good enough so that we'd want to take it to other platforms? And the reality is it is now. So we have a number of kind of, I'll say, leading indicators. It's not statistically significant, but we have a number of people out in the community that are now telling us Gemini's price performance through API calls is the best in the industry. So if you have that, why wouldn't you want to make it available to more people? And so what you're seeing is clearly we have an opportunity to take Gemini to other platforms on prem, et cetera. There's a whole bunch of stuff we still have to work out there, but that's the philosophy behind it. I also think, I mean, we're talking among friends here. There's a play here where we want Gemini to be available to more developers, so why limit it to just the platform? And so how popular is it going to be on some of these other platforms? I don't know. And the security piece is a really interesting thing. Right, because we clearly don't want, if you have your own model weights for these things, you don't want that suddenly, can we secure it on another platform, for example? I think some of those questions are still open and how we're going to do that. I would say from a philosophy standpoint, we don't want to put a model out there that is easily mishandled. Yeah, yeah, yeah.

Makes sense. Makes sense. And another question which I think, like, everybody's curious about, most of the other frontier Labs are releasing, like, more, more and more expensive packages per month. Right. How is Google able to serve some of these models for free?

This is a great question, so I'm going to give you a developer answer in a minute. I want to give you the Google answer first. So the Google answer is, and for all of you that have heard our answer 10 times before, I apologize. But we're really the only hyperscaler at this moment that can go from chip all the way to the full stack. And that gives us an economy of scale in places that nobody else has. Now we have a number of partners, whether it's Nvidia, whether it's AMD, et cetera, as well as our own GPUs that allow us to build infrastructure at the core that nobody else can touch. So the first thing is we know it's a math equation.

Right.

We know our economy of scale, if we build the stack right, will be better than everybody else's. It has to be because we're the only ones actually building things at the bottom, building things at the top. Now that's not a great answer for a developer because, okay, so that makes sense. That's how we're going to do, how we manage the cost. The reality is this is very much a scale business. The more developers that we have using these resources, the better price performance we can get out of this. Clearly serving the largest model to one customer is going to be the single most expensive thing you can do. Yeah, serving the largest model to the most developers out there. There's a cost effective measure that we can get to some of it. You're seeing us being very aggressive as well. So can we do it cost effectively for the next 10 years? I think there's economy of scale that we have to get in there. One thing, you know, just as an example, we, we talked about our new TPUs here. The Nvidia chips, the AMD chips, these are all amazing things. We'd probably like to see more developers building workloads for TPUs as well. Yeah, because then there's another economy of scale that we can deliver. What does, what do some of these open Source models on TPUs look like from performance perspective? That becomes a really interesting point. We have the ability to build hardware specific to the model that you want to run. Now we're not going to do that. It's not like pick your GPU tpu, pick your model to that. It works. But certainly we can start looking at patterns of this. So I think you're seeing, excuse me, Google being very aggressive in this space.

Okay, interesting. Yeah. And I think like one of the announcement was vllm support for TPUs. Right. So that kind of, I think gives us an indication of Google does want to support some of the open weight models. Right?

Yeah, absolutely true.

And yeah, and you're hosting some of those Models in Google Cloud.

Yes.

Already, right?

Yep. So, and I see where you're going with this. So the answer for us is choice matters, Right? You're not going to. I don't think you're going to hear Google start saying model is a choice. But the notion is we're trying to provide what we think will be the most open, most complete and most performant cloud in the industry. And that means you do have to offer up solutions for other. As great as Gemini is, and as great as Gemma is, there are other models that people want.

Yeah, that's a good point. I think that's source of confusion for a lot of people. Like, okay, when do you use AI Studio versus when do you use Vertex?

So for. For everybody listening to this, that's a brilliant question. Thank you for asking. The. The answer I'm going to give you is going to leave everybody wishing I say more. Let's start with we think a great way to get started. And let's start. So I'm a developer guy. For me, let's start with Persona. If I'm a student, if I'm in, you know, if I'm in the US, it would be K through 12 or even going into college. I'm going to start with AI Studio. I want to get the knowledge of playing with learning prompts, having Gemini do something for me as a learning thing, and I'll share. So I have two kids in college in the US right now, both studying computer science, both started their AI journey with AI Studio. And so now there's a point, we heard it in the developer keynote here yesterday, where if you want to deploy that, if you want to build an application out of it that you then want to serve up to other people, AI Studio is not the platform for that.

Okay. Yeah.

So there is a point where you are going to want to deploy this. And we have. So I sit in Silicon Valley, I'm in San Francisco all the time. Part of my job is actually to go talk to startups and listen to what they're doing. And without naming a certain startup, one of the startups that we've been working with like 5 weeks ago hated, literally wrote on X. We hate Vertex, we hate Gemini. Okay. And okay, woke us up, we went and talked to them, we worked with them. They've now published a number of YouTube videos that say they love Gemini. Vertex is not their first choice. And so what we want to do is be able to give people choice. The one that you didn't put in there is what about running Gemini on Cloud run.

Yeah. Right.

So, and again, from a startup perspective, if I, if I had a startup, if, if I was talking to a startup founder and they were just starting out, but they knew they were going to build an application that they had to serve to people, and usually startups don't have time to play with something for three months and then go build something. Like they're starting from day zero and they have to be getting something out. They need to get to an MVP within weeks or months. Starting with Vertex is not a bad answer because they're going to be able to scale up really easily. But it's hard getting that infrastructure hard, relatively hard to get infrastructure set up. Running that stuff on cloud runs, a really interesting opportunity for them. Yet if I'm in an enterprise and I know I need that infrastructure, I need that managed process, I'd start with Vertex and just take the hit up front. We're making it much easier. There are, we know through work that we've done recently that people can get up and running Vertex in, you know, in a few days.

Okay.

But it's not in a few hours.

Yeah, yeah. So, so, so like the, the reason is a lot of people, or especially developers, they will start at AI Studio.

Yeah.

Because that's the, I think the simplest interface. It is. Right. Then, like, I don't see a simple transition to Vertex on there.

It's see your. So let's talk about one of the things that we've done. So there is a path forward that we're trying to implement. So the first thing was the common SDK.

Oh, yeah, yeah, yeah.

So realize if we talk back, if we were meeting, talking a year ago, you actually could have said, why do you have two SDKs too? Right? Two platforms, two SDKs. So what you saw with us is that we published a common SDK that allows you to get started with an API that's common across both. So that's the first piece. Now, how do we actually enable migrating what you might start constructing in the loose sense, something like AI Studio, to easily deploy into Vertex? Not announcing anything here, but you should assume that we understand the problem and that's something that we have in our future as well.

Okay.

But it's a great question and for all your listeners. Yes, we know this is an issue.

That's good, that's good. And thanks for bringing up the common SDK now. Yes, I think that makes the transition relatively smoother.

It makes it smoother. We know there's still work to be done.

I also wanted to ask you About Google being in a very interesting position because you're not only releasing state of the art frontier models, but you're also releasing open weight models like Gemma. Right. So I'm curious, like, what is the philosophy there?

That's great. So I've got to step out of my job here for a second because if I look at what Google's doing, I need to bring in DeepMind. Okay. So for all of you that know. So DeepMind is our research side of the business and they are, as a research institute, they are phenomenal. Many people are aware that Demis just won a Nobel Prize for chemistry. Right. I mean there aren't many tech companies that have a leader that has a Nobel Prize for chemistry. Right. But that came out of work that they were doing to forward the, you know, the research in, in biology. And so having a research arm that is working with, you know, you name it, the top thousand, the top ten thousand researchers in the world, they have different needs than a developer working in an enterprise developer working in a startup. And so when you start seeing some of the open wave models or things like Gemma, there are many, many research arms for companies and universities that need to have unfettered access to the model itself. Changing weights, maybe even changing some of the distillation that happens. And so from our research side, we want to enable that because. And this is not their philosophy, it's my philosophy. I've always had a belief that innovation happens elsewhere. And so what that means is you have to have strategy. Not all the smart people are going to work for you.

Yeah.

So how do you then enable all those other smart people to go do interesting things? And so that's the strategy you're seeing from DeepMind is that they want to create both a infrastructure tooling and capability for all those other smart people to go do interesting things. Yeah. And so that is at the foundation of why we do closed models, open models, open weights, et cetera. And how does that. So on the, I'm going to say the enterprise commercial side, we benefit from that. Why? Because there are lots of open source developers and lots of other companies that want to play with those stuff as well. So that's probably the best answer I can give. That is my personal philosophy. If you talk to the DeepMind folks, they'll tell you that their primary audience are those researchers.

Yeah.

And if you talk to people in Google Cloud, our primary audience are obviously developers and customers.

So yeah, makes sense. Makes sense. And I don't know if you have an answer for this or not, but do you think Google is going to continue these two parallel paths of smaller open weight models versus these state of the art closed source models? Or there might be some sort of like, okay, you start open sourcing some of the previous generation.

Yeah, really interesting. So I will say so I do a set of predictions in the kind of the IT space every year and I'm really bad at it. So I don't know that I can predict what the future of Google will be for what we're going to do with the larger models. I do think you'll see, continue to see us serving the research institutes, the universities, students, et cetera, with open models because there's so much value there. I, I don't know what happens with the commercial side other than to say, and I. This is, this is common knowledge now that one of the most interesting things that we've learned as a company, and this was a year ago, is that you can actually derive a smaller model from the larger model to gain price performance.

Yeah.

With all the knowledge of the larger model.

Yeah, the description.

And that's. But that thinking about that from a philosophy standpoint means that there's value in continuing to push the boundaries on that larger model and then distilling what you want from, for each market, each opportunity. And if you open those things that might be okay.

Yeah.

So I'll stop there. I can't predict what we're going to do in the future.

That makes sense. Makes sense. This is great. Now coming back to I think the developer point of view. Right. So one question I would ask you is there are a number of different cloud providers available.

Right.

As a developer, why should I even care about Google or like what's, what's something different?

Yeah. So I love the question. I've got to be careful because I think some of my arrogance will come in. So I will, let me just say. And people can look this up if they want. I'm not going to say the name of the company. So I worked for another hyperscaler for 10 years. I chose to come to Google because of Google Cloud. The other hyperscaler that I worked for was either number one or number two in the market, depending how you want to look at it. I chose Google Cloud five years ago because I thought it had the best opportunity specifically for innovation. So I already mentioned that I have this philosophy on you have to be able to enable innovation happening elsewhere. That happens in Google too. So we hire. I'm taking you on a little story. I like telling stories. No worries. I, I came to Google, in part because Google has many intelligent people, super intelligent people. It's one of the smartest companies I've ever worked for. But not everybody who's smart works for Google. Matter of fact, I would argue we're a very small percentage in a very large ecosystem. So how do we enable all those people to do interesting things? That's my philosophy for developers. My goal is to allow them to innovate on a platform doing whatever they're excited about doing. It could be gaming, it could be, you know, classic it, it could be SaaS applications, it could be AI based stuff, it could be, you know, my, both my kids are big into social and they're like, what's an AI? What's the new AI Social engagement. I'm like, yeah, right. So for me, the reason why I think Google, if we are winning, the reason why we're winning is because we've got the best balance between performance, capability and openness. So some, some of them are more open, some of them might even be a little more performant. Some make you, you know, like Legos, build every little piece together. Our goal is really simple. Enable customers to be really effective on our platform and then for me personally, enable developers to innovate. Whatever they want to do, we do not want. And this, everybody in your audience is going to laugh when I say this, but our goal is actually to remove every hurdle that developers have developing on our platform.

Okay, okay.

And we're not there yet. Yeah.

I think this also comes down to the tooling as well, right?

Oh yeah, yeah.

Like what type of tools you build for developers in there.

Thank you, thank you for the prompt, I really appreciate that. Yeah. So we believe that one way you enable developers to be more innovative is by allowing them to abstract away some of the complexity that, you know, one of the things that's very common is the complexity. My kids tell me all the time that when I was studying CS it was easy compared to what they're doing. I'm like, that's not true. You guys are doing Python. I help deliver Java to the world. Like you get to use all these high level languages where all the runtimes are easy to use and memory is managed for you and everything else. That's easy. No, no, no. We're building much more complex applications than you were building when you began. That's what they say to me. Tooling is the way we abstract that complexity. Right. So what you heard me say earlier is that we're trying to build the intelligence into the platform that's A form of tooling. Right. But on the tool side for developers specifically, I'm going to be a little bombastic here. I actually believe we're moving towards a world, I'm not going to put a date on this, where the IDE is going to fade away into the background.

Interesting.

So I don't think we're going to be living in an IDE per se, but I think the services that an IDE provides will be available to you both from tools. So things like if you're using GitLab or any of the management for your code, there's services that come there. As a matter of fact, you could argue that many of the software development lifecycle services will come from where your source is. Right. You also want code assistance. Code assistance is. I keep asking my kids how much of your work at university is being done by using code assistance versus your brain? And they go, oh, we only use it a little bit. And I'm like, eh, I don't know about that.

Wife coding.

No. Vibe coding. That's right. Actually, without going too far into this, one of the big things in universities in the US right now are hackathons.

Right.

They do student led hackathons. My son helps organize the hackathon at his university. They ran a vibe coding kind of bracket for people who wanted to, you know, just take ideas and just go off on them. And I was like, well so it's, it's, you know, whether you believe it or not, it's being used especially by people getting started. But tooling I think is really important. So for us, code assistance means a couple different things. So we announced here Firebase Studio. Right. So for developers that want to get started in kind of a more complete environment, including deployment, Firebase Studio is brilliant. And code assistance is built in for you. Yeah. So and then if you're more working in more of the traditional space, you know, we have, we have our code assistance built in there and then we can plug it into Most of the IDs that you're going to want to use. Even though I say in the future we may not be moving IDs, the other things that we talked about here is by building intelligence to the platform, we're going to be able to help you at runtime, at deployment time and later at management time. So.

Okay, okay.

Now this is really insightful. Any tips or tricks for upcoming developers in the wife coding era?

I was gonna say do it with friends, do it with people you like, because one of the notions. More stories. But collaboration is really important. We all know this. In this industry, my team, we collaborate on everything. We measure the ability and the interest that people have in collaboration. The lone wolf model has disappeared. So vibe coding works really well if you get along with people you're doing it with. If you don't like the people you're doing with, I don't think it's vibe coding. It might be something else.

That is true. That is true. I think we're going to wrap it up on that. Thanks, Matt, a lot for the opportunity. It was a great discussion and hope everybody enjoyed. Thanks for watching. See you in the next one.

Thank you very much.