Snowflake CEO Sridhar Ramaswamy on Using Data to Create Simple, Reliable AI for Businesses

The product that makes even the people that go, I have GPT4, I have an army of software engineers. The thing that even they struggle with is things like a reliable talk to your data application. Because even with GPT4 out of the box, you end up getting 45 odd percent reliability, meaning it gets half the questions wrong when it tries to answer it. We are well in the 90s and we are racing to get to like 99% reliability on talk to your data applications. Obviously we restrict the domain and turn this into more of a software engineering problem than just like a pure AI model problem. But that's the thing that makes every Snowflake customer perk up and go, like, I want that. Because even the people with the money and the resources to spend on software engineering teams very quickly realize that this is a wall that they are likely not going to break.

Today we're excited to welcome Sridhar Ramaswamy, CEO of Snowflake. Snowflake is one of the most important enterprise companies in the public markets. It's the default cloud data platform. But today the question of what role does Snowflake have to play in the world of AI looms large. Sridhar is somebody we've known for a couple decades. He actually started on the very same day as our partner Bill Korn at Google. Back In April of 2003, we backed Sridhar and his own startup, Neva, which was an AI driven search engine. Snowflake acquired Neva, which is how Sridhar became the successor to Frank Slootman. Rarely have we encountered somebody who is as in the weeds on the technology, but also as commercially savvy as Sridhar. And he will join us today to talk about what AI means for Snowflake. The importance of safety nets, the open source community, the competitive landscape and the practical applications of AI that he's seeing in the enterprise through his lens as CEO of Snowflake. We hope you enjoy. All right, Sridhar, we're excited to have you here with us today. You're a technologist by trade, you've spent a lot of time in the consumer world and you are now at the helm of one of the most important enterprise companies of our generation. So before we jump in, we have a lot that we want to know about enterprise AI, what Snowflake is up to, some of your predictions on the world of AI. Before we jump in though, just the level set, can you give us a couple words on your personal background? And then just for people who aren't familiar, which is probably not a lot, but just for fun, for people who aren't familiar, what's Snowflake? So who's to read our what's Snowflake? Let's start there.

That's great. Pat. Sonia, super excited to be here at iconic Sequoia, home to many, many legends I admire. Yeah, I'm a computer scientist by training, early career as an academic. I joke to people that I'm a reformed academic because I was like, I wanted to do things with more impact. Super lucky to be an early part of Google, where I joined one of the greatest businesses ever invented by humanity, which is the search ads business. I ran that for close to a decade. All of ads and commerce at Google for five years helped grow that business from a billion and a half to over $120 billion in revenue. And then funded by Sequoia, did an ambitious startup called Niva, which wanted to modestly rethink what search meant before getting acquired by Snowflake and becoming its CEO. And Snowflake is the AI data cloud. Our core thesis is that a cloud computing platform that puts data at its center is going to be way better for enterprise customers to act on data than a generic cloud. And AI, of course, we think of as a transformational technology that is going to change every aspect of, of how data is stored, how it gets moved around and of course how it's accessed. We have over 10,000 customers, made 2.6 bill last year. But at the center of everything enterprise and data, that's a super quick blurb.

Perfect, thank you. And so you have 10,000 or so customers. I know you've met at least 100, probably hundreds of them since you took.

Over hundreds of them by now.

So there you go. So I'm guessing you have a pretty decent read on what's going on in the world of enterprise AI, so maybe we'll just start there. What's going on in the world of enterprise AI? What are you seeing at your customers?

First of all, people get that this is going to be transformational. You know, lots of technologies have skeptics. I'm sure you have run into folks who are like, ah, mobile, it's not going to be a thing. This browser, like, so lame. It takes a while for people to absorb. I think what's different about AI, first and foremost is people are like, I get what this can do. I think some of the power is just like, honestly looking at the magic that ChatGPT is, anyone that has interacted with it, asked it to write a poem, asked it to create an image, knows like, wow, this is Something that's very special. So the level of awareness is incredibly high. And we have thousands of customers that are in various stages of implementing AI solutions. They span the gamut from people like Bayer that are very excited by the idea of giving business users access to business data without going through like an elaborate, you need an analyst, you need a BI tool, you need blah, blah, blah. You need a week before a change can be made. They're like, I just want to put data into the hands of people that need it right now. But we also have dozens of people that are using AI as a transformation engineering. So, for example, if you have unstructured data, whether it's an image or let's say, like a transcript, previously you had to run a software engineering project to figure out what's this image about. Now you feed it into a model, ask it a question, and you get the answer. And so people are super excited by things like that. We have a product called Document AI, which extracts structured information from documents. Say, like contracts. All of us have contracts sitting around in our company folders that have all kinds of magic numbers that ideally you want to do analysis on. So there's a wide variety of cases that people are implementing and sending into production. But I would say stuff at the bottom, which is how do you transform data more effectively, more flexibly, and stuff at the top, which is how do you make data easily available to all kinds of business users in new ways, in interactive ways. I would say that's the sandwich in terms of what are people wanting to do with data. Mm.

And can you say a couple words on Snowflake's right to win? So some of the things you mentioned, like data transformation, for example, feels like that is very close to the core business of Snowflake. But then there are some things that are maybe a bit further afield. You know, if somebody wants to deploy an enterprise agent of some sort, they can use Snowflake to do it. But what's Snowflake's right to win in that situation? So can you just say a couple words about how Snowflake fits into this overall landscape and sort of the rate to win?

So, first and foremost, the basic approach that we took to AI sort of enabling or infusing AI into Snowflake is it should be an accelerant for everything that you do with Snowflake. That's what Cortex AI is. It's a model garden. But it's more than that. Snowflake prides itself on super tight integration of its various product features. And this is not Another service that's part of Snowflake, it's built into Snowflake. This means that any analyst that has access to SQL has access to AI. And so it's a massive democratizing mechanism. And then the early applications that we have built like Document AI are a very natural next in the progression of what people want to do, which is, hey, I want to act on the data that is within Snowflake's purview by both expanding the data that Snowflake has access to via things like Iceberg, which is basically an interoperable storage format for cloud storage, but then providing things like Document AI. We just make a whole bunch of AI applications that previously used to be software engineering projects into two commands that an analyst can issue. And so our first lens very much is that AI should become easy, trivial for data that is sitting in Snowflake 100%. There are going to be applications that are cutting edge, are going to involve many, many different services. But the angle that we bring to all of those customers is we make reliable AI. And this is a topic that we can get into. So, for example, I tell people you have no business believing the raw output of a language model for anything. You can't actually do any business with that because it's ungrounded. It doesn't understand truth from falsehood, doesn't understand authority. So we make things like creating a grounded chatbot. Again, as I said, two commands, not a software engineering project. Similarly, with Cortex Analyst, which is our talk to your data API, we bring the full power off. We know everything about the schema, all the queries that have been run on the schema, the semantic context on the schema. We can produce a reliable application that others are going to struggle to create. So we are leveraging our strengths in data to make AI products better. Are there going to be specialist applications that can only be done with GPT photo and a custom integration with a bunch of other stuff? Absolutely. But that's not what we are after. The bulk of our customers want to get work done. They're not in the business of doing research with AI.

And are you seeing customers bring net new data that maybe didn't sit inside Snowflake historically into Snowflake because of your AI services. And how do you think about your right to win as it comes to the data that's not in Snowflake yet?

This is a broader question. I think one of the things that I've actually been a good part of is in expanding the lens of, of data that Snowflake should Play in Snowflake, as you know, is first of all, it's closed source software. For the most part, the code engine is closed source, just like Search. But we also had a proprietary storage format where data was ingested into Snowflake and kept in this format. But what we consistently heard from customers, and I'm sure like you hear all the time, is there is 100 or 1000 times more data sitting in cloud storage than there is inside a specialized player like Snowflake. And more and more industry trends have been towards interoperable data. People want their data to be accessible from multiple places. So for example, if they want to write their own bespoke applications, most people don't want to do that. But the biggest ones do. They want the data to sit in cloud storage, where, yes, Snowflake perhaps can write it and read it, but other applications should also be able to read it. So we made a big push around Iceberg, which is the interoperable format. We also announced a cloud catalog recently. The idea is that in 10 years, data is going to be sitting mostly in the cloud, mostly in cloud storage, which is very cheap, mostly in interoperable formats accessible via open catalogs. This is the place where we see there being so much more access to data from Snowflake. So everything from data engineering and AI now comes into our purview. We have customers that, for example, are doing things like, oh, let's run a video model using Snowflake's container services on data that is sitting in S3, extract transcripts and stick it into Snowflake. So it's just a very different world we are playing in.

Makes sense. And then let's say for the data that's currently sitting in one of the hyperscalers, for example, you start the conversation by saying the core tenet of the company is that when you build your infrastructure kind of all around the processing of data, you can do better things. What are some of the ways that you're able to kind of offer better AI services around the data that doesn't currently sit in Snowflake, but that you're hoping customers will bring in versus you know, what the hyperscalers are doing already?

Yeah, and can I add onto that real quick? Because one of the things that we have heard from customers is at either end of the spectrum, you've got at one end of the spectrum, work directly with OpenAI, send your data, you know, into their, into their cloud, and maybe have some nervousness around whether that data is going to leak into the model or whether they have the right security and privacy sort of governance around it. At the other end of the spectrum, you can just do everything yourself. Grab a model off a hugging face, build it internally, you know, super safe, super secure, but pretty painful to do all that. And then the middle ground, you've got Amazon Bedrock or you've got a Snowflake and they both kind of have a value prop of best of both worlds going to make it easy for you, but it's also safe and trusted and secure and all that good stuff. And so I think my angle on Sonia's question is like, for somebody who's making a practical decision about sort of what should I build in Snowflake versus what should I build on Bedrock or a comparable cloud service, what leans people in the direction of Snowflake?

It's the fact that everything that you want, whether it is data security, data governance, ecosystem, ease of use, all come out of the box. The incredible power that comes with core Snowflakes platform, including things like collaboration, other third party applications, we make AI simple 100%. There are those people that will say, I want to take data that's sitting in cloud storage or even in another application, I want to bring it into cloud storage, I want to recreate ACL's access control list and then I want to create a vector index using a bespoke vector indexing solution. And then I will stitch together, I'll figure out which model that I want to use, whether it's an API or something that I host myself. And then I will use LangChain and write custom routing logic for my application. I can assure you that, you know, 99.9% of our customers want no part of this. You know, that's just the reality. All those poor people wanted was a chatbot to run on 100,000 docs that they have so that they can replace the annoying search box for FAQs on their site with, you know, here's a solution that just works and our take is yes, whatever governance you have had before works out of the box and your data does not go anywhere else. You have the same rock solid guarantee that Snowflake will never use your data to train any cross customer model. And we will be very efficient and cost effective from just like overall cost of running the solution. But Snowflake's magic honestly is we make the hard simple and it's things like total cost of ownership. Many of our customers are banks, they are healthcare institutions, they are finance or other kind. Like we play a lot in the media space as well. Most of our customers want to solve problems, not solve technology for the sake of technology. We have a foundation model team. They're very focused on things like how do we get models that have better grounded generation, how do we get them to follow directions well, how do we get them to say no to questions that they should not be answering when it comes to, let's say, like talk to your data, you know, so we focus on specialized areas like that. But the biggest reason to use Snowflake for a lot of Our customers is 10% software engineering project with a whole lot of risk about data and security and what else can happen turns into six hours of work for an analyst. We are good at that. We are proud of that.

So it sounds like the one liner might be it's kind of the level or the layer at which you're intersecting these products. If you're working with one of the public clouds, you're still very much at the infrastructure layer. Building a lot yourself. Snowflake, you're at the platform layer. A lot of the hard work's been done for you.

And our long term bet, Pat and Sonia, is that ecosystems move upstream. There was a time not so long ago where, I don't know, our parents, our grandparents knew every part of a car. Yeah. They're like, oh, so manly to change a carburetor and get oil in between your nails.

I gotta be honest with you, I'm still impressed every time my dad knows exactly what is wrong with the car.

Yes. You know, while I'm willing to go to, you know, go to strength training every day, getting oil in between my fingers, you know, with my car does not sound so attractive anymore. And so 100% you can work with CSPs and you can be like, I have a model guard in here, I have a caching service there, I have a database here. I will stitch all of this together. As I said, everything turns into a software engineering project for us. You're like, no, that's just a little data pipeline that you set up. And here is a beautiful UI that you get if you want a chatbot. Obviously you can do more, but you don't have to.

Yeah.

Whether your customers building on Snowflake and are there certain types of AI applications that are better suited to be built on Snowflake than others?

As I said, the categories of AI applications come naturally from the kind of data that are already there. I would say the broadest, broadest use case is really using Cortex AI via SQL in either interactive queries and dashboards or in Jobs that people are running. And so these span the gamut from oh, let's do sentiment detection with a small model. It doesn't really have to even be that expensive. So that's just like literally it's one function call or let's do other kinds of data extraction where as I said, you have things like a transcript or maybe clinician notes, you take that out and you get structured data from from it. Or the other thing that I talked about, document AI, which is you extract structured data from things like receipts, from contracts, so on and so forth. That's kind of our sweet spot. But I have to say the product that makes even the people that go, I have GPT4, I have an army of software engineers. The thing that even they struggle with is things like a reliable talk to your data application. Because even with GPT4 out of the box, you end up getting 45 odd percent reliability, meaning it gets half the questions wrong when it tries to answer it. We are well in the 90s and we are racing to get to like 99% reliability on talk to your data applications. Obviously we restrict the domain and turn this into more of a software engineering problem than just like a pure AI model problem. But that's the thing that makes every snowflake customer perk up and go like I want that. Because even the people with the money and the resources to spend on software engineering teams very quickly realize that this is a wall that they are likely not going to break through.

And how do you accomplish that? Like maybe peel back for us how you're able to get to the 90s%. Are you training your own models? Are you just tell us about how this all becomes possible.

It's systems design. Just like the magic of how you make a coding agent or an effect less a coding agent, more an effective copilot work. In practice, it's not always the giant models. It is carefully breaking problems down so that you present the right context to the model. It's in deciding things like, oh, I see the problem of answering a code whether to answer a question is different from how to answer the question. So you can specialize and have different models for these different subtasks. And also what's the basically the. I call this like a problem definition, a product structure question. We structure the product of Cortex analysts so that it is more restricted than a free flow domain. What I mean by that is schemas are weird things. People do random stuff. They have horrible column names that mean completely the opposite. Every company has its own definition for revenue and if you like, take the best model on the planet and let it loose on an arbitrary schema, the likelihood that it's actually going to understand the nuance of what's in there, close to zero. Like our big deployments, for example, our customers have 200,000 tables and you can bet that there are several tens of thousands of tables with the word revenue in it. They just don't have the same meaning. So it's really like problem definition to me. By the way, this goes back to the magic of product. I think of like any amazing founder, any amazing product manager, as someone that can visualize what's like the right trade off to be making in order to create something that has broad applicability. And that's the thing that we have done here. We constrain the problem. But as I said, we also explicitly train for things like when to refuse questions, as opposed to trying to pretend that you can answer every question. But obviously there's a precision recall trade off there. You can get 100% precision by answering no question. That's not the goal. You want to be useful, but still be precise. But it's a lot of software engineering.

I want to go in a slightly different direction. Sure. Okay. Which, that, that reminded me of this and I don't know why, but, but you guys, you seem. The product velocity at Snowflake seems to have inflected to the positive.

Yeah.

Even in the last six months or so. And we've worked with a lot of founders where, you know, the bigger the company gets, the slower and slower the velocity becomes. And so I guess I'm curious, what have you guys done to positively inflect product velocity? Because that's hard to do when you're dealing with an organization at the scale of Snowflake.

I've done this many times before and the formula is always roughly the same. Which is first and foremost you make sure that you have a safety net that you believe in, which is you have regression tests so you don't blow up big functionality. But if you're pushing hard enough, you will make mistakes. And so you have to distinguish between different kinds of mistakes. For a database company, there are catastrophic mistakes. Like if you write data badly, it's going to take you months to get out of that. So you know, so you need to understand like what is risk? And then you build a safety net for things like as I said, to detect problems before they happen. But in case you do have problems, how you get out quickly. At Google, for example, we built auto experiment scaling framework, basically you would come up with a new experiment. All changes went through this experiment framework. And this thing would automatically say, I'm going to run this on a machine, watch it for 15 minutes, make sure that the machine doesn't crash. And then it roll it out to 0.1%, 1%, 10% with measurement all along the way. All of a sudden you have velocity because someone can design. People can design a whole bunch of experiments. They're sort of now pushed out. So, as I said, the first part is the safety network, and so we spend a lot of time on that. The second part is the inner loop productivity, which is how quickly can you get like a single change in quickly, because ultimately it ends up being the decider for how many changes are you going to get through. Another system design, Snowflake actually went through a process that predates me, starting about two years ago, of how to make the system extensible. As I said, at Snowflake, we are very proud of the single unified product. But that can become like something that gets in the way of speed. And so you have to design carefully for how do you make things extensible. So things like AI basically took advantage of that framework. And then to a certain extent, to be honest with you, it is also the focus that leadership needs to bring on what is important. How do you drive clarity at all times? With all teams, there is an infinity of work to be done and driving that clarity, driving a sense of accountability. With AI team, for example, I force every team to make promises for, yes, over three months, but also, what are you going to do the next two weeks and calibrate yourself on? Did you deliver on the things that you were doing that you said you were going to be be doing? It's pretty much in my mind, if you want to get better and better, life boils down to say what you're going to do and do what you said you would do and examine and make things better. And so it's a bunch of things that have been there that I've been building up at Snowflake. But certainly I bring this sense of quality and speed are both requirements in what we do. It's a change, but people like the idea of just getting more things done. You and I have never met a software engineer that says, yep, I want to release that day after tomorrow. It's like, no, you want to get it done today. And so that itself builds momentum. When you release a bunch of products and you have a lot of customers that are using it, that becomes positive energy for the team. To build on the good behavior that kind of got you there. And so I would say the team has responded very, very well. And I tell them, hey, listen, this is the world of AI. Stuff changes every week and you need to build with that speed. I'm very happy with how the team has responded.

Is there anything in particular that you're most proud of in terms of what you guys have done in AI thus far?

I'd say Cortex Analyst is probably the hardest product that we have designed and launched. Things like Cortex AI, which is like our platform layer, I'm proud of it. But it is sort of predictable infrastructure work, even though there's a lot underneath in terms of, hey, should you use Vllm or something else? How do you optimize for inference? How do you get capacity in, like, this annoyingly crazy world where it's very hard to get your hands on GPUs? There's a bunch of stuff, but to me, that is a unique. That things like that things like Document AI are a unique combination of our strengths being applied to new areas in ways that can make a big difference to our customers. Yeah, but you also know, Pat, that this is a little bit of like, who's your favorite child? So I can't really do that. And so there's a bunch of stuff like, even if you take Polaris, which is our cloud catalog, you know, done in a matter of three months. And so I think there's a lot of energy within the team because, you know, it's a slow message, but it's getting through that. You can have speed and quality. They're just different aspects of the same problem. And my firm belief all through my life is that virtuosity trumps strategy all day long.

What does that mean?

Your speed of execution, your speed of reacting to situations is going to trump strategy very, very quickly. Yes, you need strategy, but life is never about fixed strategy. Because we live in a very, very dynamic world. It's hard to predict which product is going to be wildly successful, what your competitor is going to do. Like, we're going to talk about, like, GPT5. It's like it's a big unknown whether it's going to come out and what impact that's going to have. So I place a huge amount of emphasis on you just need to be really, really quick at what you do. And I would say, like, that's the message that I'm trying to convey to the team.

That's very. I see nice continuity from the Slootman era into the Sridhar ERA because I know I've heard Frank say at least a few times the general patent quotes, a good plan executed violently today is better than a perfect plan tomorrow.

100%. 100%. And I said that. Adaptability. Napoleon has a famous quote which roughly, I mean it's not his, it predates him. It roughly translates into I commit and I adapt, which is you go into an important area knowing that you're not going to know everything and then you're adaptive to the situation that actually presents itself.

Yeah.

Are there any misconceptions about Snowflake and AI that you want to debunk?

We are a real player. It used to be that Snowflake used to be thought of as somebody that didn't really get AI. But like early on we relied on things like more of a partnership oriented strategy for AI. But my big sort of observation realization is that AI is a like it's a platform change in the sense that it is a new way in which you and I and everybody else in the world is going to get to software, is going to get to applications. And so once we had that realization, out came a bunch of product consequences which is AI needs to be central to Snowflake. We need to make it super easy to both build applications, but also build the most important applications ourselves. Cortex Analyst, for example, is a direct to business user application. We have never really done things like that before. It is driven by a strong belief that AI is going to disrupt how information is going to be consumed very, very broadly. And I am proud of having a world class team from bottom to top, from foundation models to inference experts to product engineers that integrate the AI. Plus also the product engineers that are creating applications on top of AI. That combined with things like broad data access, which is Polaris and Iceberg, I think puts us in a very, very good position.

Can we zoom out and ask a little bit about your, I guess, your hypotheses and your hot takes on the future of AI?

Absolutely.

I just think you are so well positioned. You probably built one of the first, if not the first kind of LLM native consumer applications at niva. And now obviously from your seat at Snowflake you see so much. Maybe first on the LLM kind of race to scale, like what do you think about all that? Are we reaching the limits of scale? Like what's next for those guys?

I mean obviously this can go in a couple of different directions. I talk to a lot of experts and there is a collective belief that There is a GPT5 in the horizon. What I don't think anyone has a clear bar far is what that's going to represent. Yeah, GPT 4.0 was very cool, much faster. It also integrated multimodality natively in a way that's pretty amazing. But when you think about reasoning capabilities, the ability to come up with plans for how to execute stuff, it didn't feel like it represented a step change. And while agents are very hot, similar to Cortex, until Cortex analysts came along, people didn't really believe that you could build reliable talk to your data application. They were always kind of hit and miss. And remember, the bar is very high. If you're giving data to a business user, like 75% accuracy is like one out of four wrong. And so I think the big unknown is whether these models are going to represent a big step forward in things like multi step reasoning. And if they can, they're going to unleash like a whole new class of applications that you and I just cannot imagine right now. You know, on the other hand, I think when it comes to driving broad adoption, there is a lot that can be done with existing models. So many things that are useful for you and me every single day, whether it's a piece of mail that we are looking at or looking through a PDF, just think about all the tedium that all of us have to go through. And so I think there is huge impact to be had simply in AI technology just permeating software as we know it, especially the user input part of software. So unlike other technologies, I think there is enough that AI has already delivered that is going to have a meaningfully large impact on society. It's just going to take a while to run out. I sincerely hope we don't get to a phase where you need a billion dollars to train a great new model. I actually think that while what that model can do is cool, I think it also reduces the number of people that can have models like that to a very small number. And I think competition is just overall healthy, but it's very hard to make a call.

You've mentioned this a little bit, but I'm curious to get your take on it a bit more. If GPT5 is delayed or not a big step up or whatever the case might be, or if you just imagine a world in which the current capabilities of the foundation models, that's what we've got and it comes down to how do we implement those, how do we optimize those, how do we tune those? One of the things that we hear from a lot of people Building an AI. The first couple weeks are like magic. Everything is amazing. This is great. And then the next few months are pretty painful. Oh, shoot. It can't do this corner case. It can't do that corner case. It's not quite accurate enough and people get really frustrated and sometimes they can engineer their way out of it, sometimes they can't. But sometimes it leaves people feeling kind of disillusioned, like, yeah, this stuff's not as good as I thought it was. You know, maybe the time's not right. And so I'd love to get your take. If we froze the capabilities of the foundation models today, what sort of changes will we see in the enterprise landscape over the next handful of years? What sort of stuff will we not see because we're just not ready for it yet?

To me, this is honestly the magic of software engineering. Part of what I feel we have implicitly accepted with ChatGPT is it's sort of like is it's omniscience. You're like, it can do everything. They don't say it. In fact, they take pains to not say it. But just like Google search never tells you that's a dumb query. Think about it, right? Be kind of fun if it did, right? But there are lots of dumb queries that people type into it. Google's like, oh, yeah, I type lots of dumb queries. Yeah, they're like, oh, here are 100 million pages on the web and here are the 10 best pages for you, Pat, for your dumb query. And so I think it's like, some of it is good old fashioned AI enthusiasm. It can do everything. But some of it is just also plain dumb. You should not be doing that to me. This is where things like, okay, let's actually make grounded chatbots the norm for interacting with information. The model is, this application should tell you where it got the information from. It should be very easy for you to verify said piece of information and feel good that you're actually getting something. Similarly, you need a test framework, like Harrison talked about, an observability framework to do this on an ongoing basis. But I think sometimes when it comes to things like chatbots, people forget, wait, there is such a thing as a set of regression tests. There is such a thing as acceptance criteria for software. Everything that we have, like, if somebody were to build a new application, like one of your founders, your expectation is that, you know, they got their clue together and are actually testing stuff before they give it to customers.

Yeah.

And somehow in the world of AI, we're like, no, no, no, no, no, it doesn't matter. And these models react pretty violently to the addition of a period in a prompt. And so I think there needs to be this idea that you need good old fashioned software engineering and you need to measure the performance of these things. And so I think this is where it goes away from. These are hobby projects that can be hit or miss to. Here's somebody that can actually software engineer this for you. And we think of that as a core strength of what we bring to the table, which is like you should be able to have a predictable way to say this chatbot is going to work or this agent like application. This is the success rate that it's going to have, or this is what Cortex analyst is going to do for you in your domain so that you're like, okay, I feel good about deploying it. So even if GPT5 did not happen, I think there is a lot of magic to be done, but it's also just work.

Yeah, yeah, yeah, well put. Well, what's the. I forget who said it. There's a quote that we use every now and then. People miss most great opportunities because they tend to be wearing coveralls and they look like work. You know, I think this is one of those where like anything else, if you want it to be great, you gotta work pretty hard on it.

You got to sweat it out. And to me this is also the place where the thinking of recall as something that you should tune. Thinking of recall as an important part of how you think about these applications. Any ML engineer worth their salt will promptly come and tell you it's like, okay, I have an AUC curve for you. What are they trying to say? They're basically trying to say there is a trade off between how much you squeeze the model to do and how good it is. There's no perfect answer. That's really what the AUC curve represents. And the more we think of AI applications as also having this AUC curve, there are trade offs to be made between reliability and ability to respond. And that's a very conscious factor in how you should think about things. I think the better off we are going to be in terms of where can they deliver value.

Yeah, I'm going to go back to the point you said a little bit earlier about reasoning and kind of that delivering the next big leap. Hopefully for GPT5 and Claude, et cetera, it seems like the approach that most folks are taking is kind of bringing in search at inference time. And so a lot of more inference time compute and kind of this alphago style search stuff, I'm curious just given you are one of the best people in the world at search, do you think that is the path to the promised land on the research side for bringing reasoning into these general models?

Give me a little bit more context. I can certainly see how search plays a role in how these models operate, but can you just tell me a little bit more? Yeah.

So I mean if you take the example of if you take AlphaGo and you're trying to decide what move to do next, if you can kind of create a branching tree of here are all the possible moves from here and do a search kind of over that of like here's what move I should do next. I think people are trying to bring that logic into, out of the gaming world and into domains like, I don't know if you saw Devin's cognition where they're effectively searching over different things that you can do in your coding as well. And so just like at inference time, just giving the model kind of the ability to like search possible paths to decide what to do.

Yeah, there have been a number of papers, you know, on this. I think even Neurips had a bunch of papers about searching over domains as you, as, as you come up with a plan. What I don't have to me it's important to understand, I'm forgetting the name of the new ribs paper, but it also had the same problem they were doing tree search is that they fundamentally rely on a model, typically a neural network, being able to do things like grade a particular point in a state space. Yeah, basically like AlphaGo for example, has pretty solid ideas about what is an advantageous position versus what is not and the search is guided by that. What isn't clear in sort of very open ended questions is as you come up with alternatives for the search space, can you actually grade them effectively if it's like an open ended plan? Certainly number of these techniques work well for games that have structure in which you can actually learn what does optimal mean and you can begin to optimize towards it. What I don't have as good a feel for is let's take like, you know, something as simple as cooking. You would think it's, you know, it's simple but if you take, I don't know, 10 ingredients and 20 steps that you can take along the way and various things that you can do in each of these 20 steps and the steps themselves can be short, they can be long, you quickly end up with like this crazy combinatorial explosion of different ways of doing things. And yet there is just one perfect recipe or two or three. That's the part honestly I don't have a good feel for. In terms of like how do you even begin to measure the jump in terms of cognitive ability? It's easy in structured environments, but like out in the real world where you're trying to do some pretty complex things, I think it becomes trickier. We've built prototypes for basically like agent analysts, but it's again a structured space. So what we do one thing. I've done numbers like pretty much all my life. I used to do whatever household finances for my dad when I was 10. Same like we did in a notebook. And over the past 20 years every day I get like this email that tells me how my company did the previous day. Used to be called bean counters at Google. Every day you got a report card. Every few weeks something would go wrong, like you made less money somewhere. And we would start this predictable problem, like predictable exercise of some poor analyst would go drill down into a bunch of different things, blah blah, blah, blah blah. Look at sliced stuff. And then they would come back with like oh Sridhar. It was like Easter in Germany and Ascension Day in Brazil and that's why our numbers were off. And it took like a decade to model all of these complex things in the world into like a prediction model. So you're like, okay, I can begin to predict. But if you think about it, the analysis that they do is constraint. It's pretty much if a metric is wrong, go slice it by 10 different dimensions, go look at the results, see where likely the problem is. Certainly we have built prototypes of this AI analysts that can remove 60, 70% of the work that is needed in actually diagnosing problem. It's pretty free form, but you can make a language, you can tell a language model. These are my attributes. Oh, go call Cortex analyst with all of these parameters, get the output, take a look at it and then tell me what I should do next. So you can begin to automate some of it so that this is actually useful so you can do things like that. But a much more open ended problem of here are 100 different things, incomparable things you can do and how do you judge and how do you prune? I think that's the part I honestly don't have good intuition for. Totally.

I'm going to ask about search in a different sense if that's okay. You obviously have a incredible point of view on search given your time at Google and at Niva and it Seems like right now the consumer world is watching excitedly and nervously about is there going to be a new kind of search king crowned? I'm curious your take on the whole kind of AI search space right now.

How about a hot take on perplexity? Do you have a hot take on perplexity?

Like look, I'm happy for perplexity. And it reminds you again that right time, you know, right time, right place matters a lot at niva, you know, which converged onto a view of what search should be that was very similar to perplexity. We were just two, three years early and timing ends up being everything. You can think of perplexity as like a consumer manifestation of how we want to deal with information. Let's face it, I want to look through an eight page doc to find the two lines that I really care about said no one. But that's search. And so in that sense it's absolutely the right place. I think the more important question is whether the business of search, which is carefully preserved with business contracts, not with consumer choice. Consumer choice is fiction in a whole bunch of things that we do. We eat what's put in front of us and we will search with the default search engine that came in our browsers. Okay, we might resist it, but on aggregate with humanity, that's the reality of the world. And so I would say that that is the bigger challenge because search is mostly locked up by a few players that control the entry points. But I think that's the fundamental problem, which is it is very difficult to break into the business of search. Consumers don't like doing stuff.

And this also gets to one of the kind of broader questions in the world of AI right now, which is incumbents versus startups. And historically the battle is can the incumbents with distribution build cool products before the startups with cool products build distribution. And I think search is a great example of that. You might have the coolest products in the world. It's awfully hard to change consumer behavior. AI is an interesting test case for this because so much of the coolness of the products is available through the open source world or through third party models. And so it feels like it might be a scenario in which incumbents are advantaged versus the startups. But do you have a point of view on that?

I would take two different lens to this one. One is what you said about models, open source models plus players like Meta that basically have infinite budgets under willing to open source models. I think the world of creating models from scratch, unless you have an attached Hyperscaler, an attached business looks very, very hard.

Yeah.

And you know, so I think, as I said, I hope this doesn't go to like an Ergo 3 GPT5 class models that the world has because I think that's a bad ending for the world. So I would definitely say that foundation model companies without a strong business to accompany them, it can be a product like I think OpenAI has created a pretty solid product product. It's not just a foundation model. Yeah, I think that's one thing to keep in mind. I'd answer your second question of sort of disruption innovation from a historical lens. I think of every generation of Silicon Valley companies as learning from the previous ones. They are smarter, they know the ways in which things can be disrupted and they lean in pretty heavily. We all know, for example the IBM to the mid range computer sort of disruption and then The Decks and SGIs of the world, then getting disrupted by the Microsofts of the world and then the web coming along leading to the rise of companies like Google or mobile. I would say that in each and every one of these transitions, powerful incumbents with very large pockets have shown an ability to lean in sooner, lean in faster. At Google, for example, when I was there, we leaned in very heavily into the home assistants because Alexa was going to take over the world. That was going to be the way in which you and I and everybody else searched. We were terrified and we put a pile of money into it and nothing came of it and it didn't matter. Why? Because the cost of a disruption is way higher than the amount of investment that you have to make. I would say now this is generation five or something to that effect. I'd say all the incumbents are very aware of what can be disrupted and they lean into it. There's a bunch of strategic thinkers, as I told you. I think of AI as basically shuffling the tiles on enterprise software. And a part of me goes like no way Snowflake is going to be leading the charge when it comes to AI, not waiting for it to develop. But I think you see every enterprise AI company lean in the same way. And so this to me would be the question about how much disruption is AI going to drive in consumer software. Certainly there'll be new categories. To me, if I were a startup I'd feel a lot more comfortable that I'm creating a new category. Image creation like done in a mass scale, clearly amazing. But the same goes for videos, same goes for voice. There's a bunch of specialization that you can do here, adapt them to marketing new things feel like a much safer bet in the AI world than, you know, take your pick. I can do XYZ faster because I am AI enabled. I don't think of that as having a whole lot of legs. Yeah.

Do you think ChatGPT has a chance of becoming the next Google? And to your point on consumer choice being a mirage and, like, you know, business deals are where this stuff gets locked down. Like, I'm curious what you think of the Apple ChatGPT deal.

I think ChatGPT, I mean, the phone is a pretty interesting place. To me, the phone, because it's a controlled environment, actually offers enormous potential for consumers. I tell people something as ridiculous as copying, I don't know, an address like from your calendar or a piece of email or to Uber. So dumb, so hard. You know, like, you would think Citi would like, you know, do this, copy the address from this email from Pat and stick it into Uber so I can get an Uber. So to me, I think, like, there's a huge amount of potential, again, in, like, mundane applications. And because the mobile ecosystem is a pretty closed one where Apple can mandate things like you must have APIs that make it possible to access your functionality using language models or else you might not get any traffic, that sounds like a pretty good incentive for everybody to kind of get in line. So I think there's a huge amount of potential there. I honestly wish there was more innovation in this space because, again, all of this is super doable technology. You and I can argue about, should this be done in the cloud, what can be done on the phone? But, like, as a consumer, do you care? Like, we have great connections. I'm kind of like, if this thing works only when I'm connected to the Internet, I'll take it. And so to me, those are sort of, you know, those are details. I actually think ChatGPT is an amazing product. There's underlying technology, but in so many different ways. They've actually created a stunningly beautiful product experience. Yeah. That spans the gamut from, you know, they've turned pretty much like, visually illiterate people like me into budding artists. I tell people, it's like, I'm good with words. I can talk all day long, I can write all day long. And the magic that I can do with ChatGPT is truly amazing. That or even things like I, you know, for example, like, I'm on this language kick, I'm learning Hindi, and I, at some point I was like, oh, I'm struggling with these numbers. But off comes A prompt that says, hey, I want a CSV that translates numbers, just a string of numbers to Hindi. And can you do that? Can you just give me a CSV file that I can import into Quizlet that literally is faster for me to type than to describe to you. I type it in, out comes the CSV file. In 10 seconds, I download it into Quizlet, I have a quiz. And so. And pretty much everything that I used to do with Python scripts on structured data, I just do, like, with English, you just upload the CSV file and you're like, oh, add these two columns, do this other thing, format it into this nice table and get it out for me. It's magic. So I think there's absolutely a there there in terms of, like, is it a great product and a great business? But, you know, being the king of search is like a few more zeros. They don't come easy to people.

Yeah. All right, should we close with a couple of quick fire questions? Rapid fire questions?

Yeah, let's do it.

Okay. Who do you admire most in the world of AI?

Who do I admire most in the world of AI? I admire the people that are, you know, like, working on things like foundation models that are able to do it on. On. On the cheap without the infinity of resources. So, for example, people like Arthur or Danny, I think they've gotten Danny Yogatama from Rekha. I think they've gotten just like a remarkable amount of things done or from our own team folks like Samyaman Yushang. To me, they represent so much creativity because I go and tell them, ah, limited budget and what can you do? I think there are a set of just amazing, earnest people that are driving research under tight constraints. So there's obviously lots and lots of people, but it's the doers that are doing the work, imagining our future that I'm a huge fan of.

What's your favorite AI application?

ChatGPT. By far.

Easy one.

Easy one. Just the utility that I get from it day in and day out is just truly remarkable.

Okay, follow up then. What's an AI app that you wish existed?

Like an actual talk to your phone that can actually mediate between apps? That would be super cool because remember, as I said, like, just flipping between applications, doing very little things. Such a pain. Yeah.

All right, we're going to end on an optimistic question. What is the best thing that can happen in the world of AI over the next five or ten years? What would you be most excited to see coming out of the world of.

AI software, which you can think of as encoding our thinking, capturing our ability to think and act in real world situation clearly has been transformational over the past 50 plus, you know, years. To me, AI as a, as an enabler of access both to the act of creating software, but using software to all of the people in the world would be a significant step up. And as I said, I don't think it's like lots of fancy new technology that you need. The newer technology can certainly help newer classes of applications. You know, I was very proud of the fact that we put Google Search, thanks to things like Android, into the hands of pretty much every human being on the planet. It is a, you know, you can be cynical about technology but it's a genuine step forward for humanity. To me, just like AI models as like the new layer between humans and software and software and software is actually a significant step forward just in having this functionality be vastly more accessible to lots more people. As I said, both in the creation aspects but also in the consumption aspect. I think that's a pretty cool thing to look forward to.

Awesome. Thank you Sridhar. Thanks for doing this.

Thank you Pat. Thank you Sonia.

Thank you. Sa.