Turning Academic Open Source into Startup Success ft Databricks Founder Ion Stoica

In general, this was our approach. We are going to be aggressive also about partnerships, even though the partners could compete and overlap, because you have to trust yourself that at least when it comes to Spark, you can build the best products. We are, you know, kind of saying internally, well, you know, if someone else is building a better product for Spark, then we deserve to lose. Right. So that's kind of always was confidence that we can build the best product for Spark. And eventually, if Spark wins, we are going to win.

Hi everyone. Welcome to Training Data. Today we're excited to welcome Jan Stoica, professor of computer science at UC Berkeley and co founder of both Databricks and Anyscale. He has a uniquely exceptional career as a leading professor and founder of companies at truly legendaryscale. Today we dig into questions like databricks positioning in AI, how research projects like Spark and Ray have led to the founding of Databricks in Anyscale, how he ties his research projects closely to industry from day one. New projects out of his lab like Vllm, memgpt, lmsis and Vicuna and what research fields he's thinking about next. Jan, thank you so much for joining us today. We're really excited to have you on the pod. To kick things off, we'd love to hear a little bit about where databricks aspires to fit into the overall ecosystem, especially with some of the recent launches. What are you personally most excited about?

First, thanks for having me here. So I think with databricks, always we wanted to provide the platform an end to end platform which helps our customers get most of the value out of their data. And one of the best ways today to get value out of the data, it's using these kind of new developments in AI, including large language model and everything else. And the one thing is that to note that this was always our vision from day one. Actually Spark was created. One of the main reason to create it, Mantec, when built it, it was to solve, to speed up classical machine learning algorithms, right. And to scale them up. And so in some sense for us it's full circle. We started with AI machine learning. It's classic machine learning. And right now we are back and doing more and more AI because to take advantage and to create value out of the data.

So you mentioned that you were founded around enabling classical machine learning. How do you see the current AI moment as same and how do you see it as different? And I'm curious, what are the specific things you're doing for this moment in time, like the Mosaic acquisition, things like that.

Yeah, so, so certainly the momentum around AI, it's, it's on a different level today, right. You just look at the investments out there, right. That should tell the, you know, a big part of the story. So I think that obviously is like what, what the way we are looking at is that taking advantage and being successful with AI is not easy. Right. It's like the AI ecosystem is growing in complexity. It's not just a simple call to a model. Right. You have so many techniques now like rag and raft and to improve the accuracy of your application using AI. Right. Obviously everyone is excited about AI because it solves so many problems and there are so many, you know, generate so many headlines, but still it's not what you want. You want a product of AI. A lot of things still we are seeing today are fantastic demos, right? And demos are inspirational. And when people see demos, what, you know, whatever, chatgpt, all this kind of problems, Olympics, math problem, it's very easy to think that, wow, doing that is going to do everything, right? But going from demo to production, like I said, it's a big step. A demo means that you need to find an instance, a listen instance to be really impressive in production. It's like there exists such instance, but when you go from the demo to the product, when you have a product, it has to work for all cases, right? So that's kind of the big gap. So that's why the effort is to improve the accuracy, improve reliability, of course, eliminate hallucinations as much as you can and really also to find where it provides the best value. Because you can apply AI to 1000 use cases, but which are the use cases which are going to provide you the biggest value? I think that's what we are also trying to help our customers with. So to navigate that how to successfully apply AI to their product, to their, to improve their services, to their business.

In that vein, one of the most interesting things that I thought came with the Databricks AI launch was the new Open General Purpose LLM created by Databricks DBricks. What was the reasoning behind training your own models open, sourcing them and what do you think are some of the best use cases for that model?

Yeah, so I think the main, if you look like our main market, it's enterprise and enterprise customers, they do have a lot of concerns about data privacy, confidential ID and obviously they kind of want control. It's not only that they want auditability, right. They want to be able to audit what data you use to add results have been used and what decision have been made, with what data the decision have been made. And so that's why in general the enterprises, everything being equal, migrate towards the open source models we can host on their machines or in their VPC and so forth. So it's one of the reason for releasing DBricks is to help our customers. And then the customers say in many of these cases they start from this model and then fine tune with their own data and to optimize for their particular use case. Right. And it's again so our enterprise customers, they want open source models, they want kind of control, as much visibility as they can, they want privacy and confidentiality, especially in the recent light of the breaches which are widely published. The other thing is that the DNA of data databricks has been open source. It's not only Spark, but Delta and MLflow and many others.

Yeah, the other thing I thought was fascinating about Deep Brex is its excellent programming abilities. What do you think has made it such a capable code model, especially even compared to something like code llama 70B.

Look, I think it's about obviously it's about data and how you train it. And it's one thing, I think one of the major databricks advantage is Mosaic. Right. We have also the entire infrastructure for training. We have not only about the data, but the entire infrastructure of training and fine tuning and that make it much easier and most cost effective to optimize models for different use cases. And obviously the copilot programmability, it's one of the very important use cases in this enterprises because software engineers are still very expensive.

Yes.

So making the people you have, and it's not only about that, it's like hiring top software engineering is very difficult. Right. If you are a large company like maybe, I don't know, 4G or something like that. And so making those people productive, it's extremely important and critical for their business.

You mentioned Mosaic as a key part of the strategy. I guess. What do you think are your most important chess pieces in this kind of AI battleground? I imagine Mosaic is one of them. And are most of your enterprise customers looking to train their own models? And how does Mosaic and your other acquisitions fit in with your customer needs?

Yeah, so I think there are a few enterprises who would like. So it's again it's pre training and then it's fine tuning and using the model on your own hardware, in your own vpc, on your own machines, you rent it your own. So as you may expect, there are a few which are doing Pre training, but there are still a few doing one to pre training on their data. You have enough data, a lot of them they want to do fine tuning. Because look, if you are an enterprise and a company and you want to improve your business, what do you have? What do you have? And which other don't we think you have? Which other don't? It's the data, the data about your business, about your users. Right? Right. So therefore, because this is something you have and other do not, you want to take advantage of that. Right. So how do you take advantage of that? Again, there are many ways and you try different ways to do it. Right. One is about fine tuning user data and you have an open source model, you fine tune on that. Right. The other one to use rack. Right. And things like that. But and so you are going to do any of those. But then you want to do, like I said in your own VPC, right to preserve security, to have 10, four security perimeter. You want to do it in order to protect confidentiality of your users. And obviously right now with gdpr, California Consumer Privacy act and so forth, there are a lot of regulation and the number of regulation is going to increase. So the fact that you can having an open source model you can fine tune, you can use rag in your own vpc, it's very compelling value proposition.

And are you finding that most of your customers want to go that approach versus OpenAI and the very powerful closed source models in the market?

I think enterprises and still we are still early on and there are many enterprises obviously using OpenAI for different use cases, different applications. And I'm sure OpenAI and Microsoft Azure will come with new products to provide better confidentiality, better security. But at the end of the day what I was saying is that everything being equal, right. As an enterprise I will prefer more control, security and strategic. Right. It's like it's less kind of locking or things like that. Right. So I think that's what we are saying. So if as open source models are going to catch up with the proprietary models in particular in the use cases which matter for the enterprises, it doesn't even to be perfect for all the particular use cases. It just need to be very competitive where it matters, then enterprises will prefer these solutions on which they have more control and more secure.

And how far away do you think we are from that moment? Do you think we're there today where all else is equal or when do you think we cross that in terms.

Of the open source versus Proprietary?

Yeah, open source being on par. All else being equal for the core use cases.

So you have a lot of use cases. It's again it's the open source plus data and right now the application are more complex. It's not just a call to a large language model. You have this kind of. It's what it called, you compose, you have application which build from many components and that's what we call compound AI. So actually it turns out that if you can build an applications for doing recommendation or things like that, or for programming for a particular copilot for a particular tool, you can actually do better than Even something like OpenAI on the latest ChatGPT because you have more data. And the other things about databricks which I think there are also announcements about that is that with a kind of unity catalog and things like that, you have access and you know also about the structure of the data which helps you tremendously to improve the accuracy of your applications. Right. So it's not only the model, right. It's everything else you have around it and the quality of the data you feed it in. So I think that is a data stupid like you say, right? At the end of the day it.

Sounds like control and security are two primary areas of things that enterprises really care about, that you've noticed. And databricks obviously has a tremendous advantage with having access to data as well to help these companies use controls and.

Security for the customers.

Exactly. What about some other factors have you noticed that they also care about? How much does cost matter? How much does diversity of other models matter?

I think obviously cost is important and it's like this, right? First of all, you know, you want to, you know it's initially the importance, think it's about the value. Can you provide the value, right? So that's the first thing. And at that stage cost is not as important, right. And in here actually in some of these early stages where people also try to use the most powerful model like OpenAI and things like that, but once you cross that and you have a use case which you conclude that is good, add value to your business. Now you want to scale it up, right? And now you are talking about having more control and more security and all of these things protect confidentiality of the data, privacy of your users and so forth. And now basically people consider how they are going to deploy it. And this is where having multiple choices and having like you said, more control, security, it matters. And where open source models and platforms like databricks are very valuable. And again it's also all the other components that you're together in databricks, like I mentioned, like Unity Catalog and everything else to increase the value of your applications.

Yeah, super interesting.

I'd love to talk to you about compound AI systems. I think you guys probably coined or popularized the term and it seems like that's a lot of what the industry is latching onto now. Maybe for. Can you explain what is a compound AI system and what are enterprises thinking about when they're building these?

Yeah. So a compound AI system basically consists of multiple components, multiple calls to large language models or agents. And it's very much you can think about. When you write a program, you have multiple components, you have different function procedures to do different things, and then you then put them together to. To create the program. The same is very similar here, right? You can use maybe one model to parse the data to extract the data you can use. And then you can use, for instance, depending of the prompt, you may use a model if the prompt is about mass problems. Right. It's like you can use one model if the prompt is about, maybe about programming, you may use a different. Different model, Right. And then you may use, for instance, about formatting the result. Right. That's another one. You may use that for instance. Now more and more you are talking about agents. And with agents you have to, you call external services or functions like search or you can use a calculator and things like that. So now you have different models, you can do a better job. And they are small models actually to take the prompt and convert it to a function call. Right. So that's kind of what it is. But the way to think about is like the conceptually is like you write a program has different components and make it easier to develop and deploy and manage the same thing you want to apply to AI applications.

So like a collection of smaller components that work together where the, you know, the sum of the parts is greater than the one monolith that you're placing.

I'd love to dive a little bit into the databricks story quickly, which I think is an incredible legendary journey over the last decade, but with a lot of nuances that I think maybe many folks don't yet understand. From today, it looks like you are building the right company at the right place at the right time. But the actual nuance is that databricks was originally started originally for data scientists. It happened to cater well to machine learning workloads because of all the work that you were doing in data. And over time, you made the right strategic decisions to actually really grow with the AI market. Can you share a little bit of some of the learnings and journey that got you to where databricks was today?

Yeah, I do think that very happy too. I do think that a lot of it's also about being the right time and the right place and things like that. It's being lucky or I think that's all of this is true. And there are many things needs to go right to be successful. Some of the things you control, some of the things you do not control. When we started, indeed we are focusing to build a product, a cloud product hosted product for data scientists. We have this kind of notebook and we have provided hostage spark and we target data scientists and with targeted data scientists. It was one of the reasons we targeted data scientists because again we all have like I mentioned spark early on also targeting machine learning, application machine learning workloads. And at that time there were not many data scientists. It was like 2013. However, when you look around, most of the universities already have data science programs, right degrees. You're starting to offer data science degrees and you are saying okay, we are. It seems a good, you know, path forward, a good market. And I remember you're looking at LinkedIn to see how many data scientists are there. Because our users, no, not initially. They're not as many thousands. Especially when you compare the data with database analytics analysts and engineers and so forth. He thought we build and I think it was a reasonable good product. And then we started to grow quickly in terms of the number of customers. Initially we had small customers, but then after. And still the interactive analysis the data science has been for a long time. It's actually one of the biggest workloads in particular in terms of revenue because the interactive workloads are priced higher than batch workloads. And I remember that we sold to these companies to use databricks and they are aspirationally buying it to do data to the data science AI. And when after a few months we go to them because we didn't go much earlier because we saw that they are doing well, their usage was growing and everything seems good, so no reason to worry. So we went back to them and said to see what they are doing, maybe we can write a blog post or whatever they're doing. It's like marketing and everything. The surprise is that very few are doing actually machine learning at that time. And we asked them what happens, right? Well, it turns out that obviously that in order to do machine learning you need data. Like we discussed early on. And they realized that for the particular applications they Wanted they don't have the data they need. So they need to now to put maybe to add new logs, to collect new logs in their products and things like that. And also they need to clean up the data, curate the data and so forth. So now we are lucky because Spark was very good also for data engineering for data processing. It's a data processing tool at the end of the day. And so they are using Spark for data engineering. And then obviously we start focusing also on serving data engineers much more than before. And that's how we started. And then obviously later now got the data engineering and still have all these data scientists exploring and starting to build models. And then it has a very natural extension to start to add more products for our user, for our customers, like I mentioned, to get more value out of their data. And that meant building machine learning models and using the open source models.

Very interesting.

I want to talk about the Databricks Microsoft partnership. I think that was the stuff of legends and I think still probably one of the only case studies of a successful, truly transformative partnership. Maybe walk us through what that partnership was. Was it a bet the company moment at the time. Do you think databricks would have become what it became if you hadn't struck that partnership? Maybe just talk about that.

Look, obviously the partnership with Microsoft was a great partnership for us. The one thing I want to say that and this is most visible, we are very from the day one, we are actually very focused on the partnerships. Our idea was always we make Spark successful, hopefully de facto standard for data processing and then we make databricks the best place to run Spark. So early on actually we started a few months in the life of the company we had this kind of partnership with Cloudera and then we had with Hortonox. And this partnership was mainly because we are in a to advance Spark, right? Because Spark, it was created in Hadoop ecosystem, right. And these are the Hadoop companies, right? Right. And so we had this partnership or data stacks and so forth. So this was like in the first year or so we had all this partnership despite the fact that some of these companies, we knew that they could become our competitors. Right? Because you know, just making them is making and helping them to deploy, to manage and to sell Spark based services. Right. It's like. So in some sense, you know, the Microsoft is like, it fits our kind of approach of trying to strike do partnerships with other, being aggressive, doing partnership with other organizations in the ecosystem. Even though again in some cases it was not a clear cut whether they are going to Compete with us or not. So just growing the ecosystem and growing Spark, that was the priority. We even had at some point a partnership with Snowflake. So it was very. It requires a lot of heavy lifting. So we always look for partnership which are meaningful. Right. And I think great negotiation, you know, Ali and so forth did a fantastic job there. But at the end of the day we also need to commit to it. It's like it took to do to build the Azure databricks. We are an aws. Before that it took tens of engineer one year and you are a small company at that point. So it was a huge commitment and a huge bet also from our perspective to do that. And yeah, and we, I think engineering and everyone executed very well and it was a successful product. Right. And Microsoft was great partners. And yeah, this is what happens. We are obviously a bit lucky, but in general this was our approach. We are going to be aggressive also about partnerships, even though the partners could compete and overlap. Because you have to trust yourself that at least when it comes to Spark, you can build the best products. We are kind of saying internally, well, if someone else is building a better product for Spark than we deserve to lose. Right. So that kind of always was the confidence that we can build the best product for Spark and eventually if Spark wins, we are going to win.

Do you think that databricks would have become the company it is today without that Microsoft partnership?

I think so. It maybe could have taken a little bit longer, but yeah, I think so. Right. We would still have a good, very good offering on Azure like we have with gcp. You have taken a bit longer, but I don't see any fundamental change in dynamics. Right. Because one of the advantage of databricks of course is one Spark, one and we could provide the best product for Spark. We are in a very strong position and compared with other Clouds, remember that one of our advantage like everyone else, advantage like confluent and so forth, is that you can provide a service on multiple clouds. Right. And multiple clouds is, you know, multi cloud has been, you know, more and more very strategic for, especially for large enterprises. We do not want necessarily to be locked in or to want the one choice.

So yeah, I love the confidence and conviction that you have in Spark and your own execution abilities internally. But that married with the practicality and aggressiveness of winning as a business and pursuing the right partnerships and doing whatever it takes to win.

Yeah, yeah. I mean you try to simplify things. That's what they were saying, you know, you like initially when he said look, we have to make Spark win. There are many. I remember, I look at all these combinations. It's like Spark wins, the product fails or Spark loses. But we have a product which is successful or both fail. No, that's not very interesting. Or both are successful and we convince ourselves that, you know, we need to bet on Spark to win because that's the most likely way also for the product to win again, for better or worse, right? But sometimes, you know, there could be many ways to success, right? And in retrospect, you cannot go back and you know, and try other alternatives. Maybe there are better alternatives at that point. But it's important to commit to unsync, which hopefully it's a reasonably good solution, a good path forward, right? Again, there are many passes to the peak. The most important is to commit to one which leads to the peak, right? It's like not the shortest one or the easiest one. It may not be, but it has to be one to get there and.

To the highest peak.

And that's why I said, okay, Spark has to win, right? And we have to build. Need to be the best place for Spark. And then we are saying to be the best place for data and AI, we need to. Eventually we knew and we assumed that if you are to be hugely successful, you are going to go beyond Spark, right? It's like that's why the name of the company, Databricks, not Spark Labs or something like that. So that's kind of. You try to simplify once you do that and you start to execute, okay? So you want to make Ray successful as an open source, so you want everyone to use it. So that's why you do Cloudera and trihorton orgs and so forth to do this partnership. Because at times there are other solutions. You know, people knew that Hadoop, MapReduce, you know, these times have passed, so to speak. So they are talking about new systems. There are like Tez, it's actually Horton or Tez, this project and so forth. So that was very important. And then it was also for data science, it was kind of when we bet, it's like it's a niche and you know, and we saw that we can build the best product for it. So, you know, so that's kind of need. And you need to have ultimately confidence in what you bet on, right? You have to bet, right, right. Because you are a small company, right? If you don't bet, how you are going to win, right? Everyone, right? And then you need to have some level of conviction, right. To do it. And yeah, I'd love to kind of.

Pull on that thread and switch gears a little bit into tying a lot of the entrepreneurial path that you have with a lot of the academic and research background that were the roots for the beginnings of these companies. You have a very unique career as both a leading professor and a founder of multiple Unicorn and Decacorn companies. I don't think there's anyone who comes close to pursuing both disciplines at the scale of success that you have. Maybe specifically Ray with Anyscale. Ray to Anyscale Spark 2 Databricks. Take us into your head. What, what is the process in which these research fields start to ruminate in your mind? When do you kind of continue to give them resources to develop and then when do you know that it's the time to then start a company to pursue that in a better, more open and fast way?

That's a good but hard question. So I think that, and by the way, I like to preface to say that, to start this saying that it's obviously also a lot of luck involved and you know, being in a place like Berkeley and having fantastic students and colleagues around you, you couldn't do without that, right? It's like, is their merit probably more than mine. But one thing I think is that I've been always trying to focus on, it's on the problem. And actually when even to my students I'm telling them it's like one of the most important things you need to do is to figure out what problems you are going to work on. Right? Because it's like everyone which comes to Berkeley or this top schools, one thing they have in common, they are good problem solvers, right? They have good grade, good scores, write papers. Paper is about solving a problem. So therefore if all of them are good problem solvers, the differentiator is a problem you are working on, right?

Yes.

So you start with that. And also I think it's like especially at Berkeley, you get, you get exposed of kind of not only new ideas, but willingness to take risk and get into new areas in some sense. And this is what I like about Berkeley. If you look traditionally Berkeley among top schools, they are the first to open new areas is like of course the risk processor, it was with Stanford as well. But databases, networking, sensor networks, even in open source with a Unix, bsd, you know, tcp, ip. Right. Part of the ctab. So they are always kind of more, you know, a little bit trying to experiment. So I think that's kind of, you know, that kind of culture. I really resonated with it. And then the other thing what happened is vertically we have these slabs which are like five year slabs which basically each lab has kind of a vision and it's a group of faculty coming together which believe in that vision and with their students and try to make it happen over five years. And this has a lot of great impact all the way. This tradition started 40, 50 years ago with Dave Patterson, Randy Katz and others. And they built in know risk rate, redundant area of inexpensive disk. Now network of forestation is commodity. You know, this is. Everyone is building now this huge cluster of commodity machines, servers and again and many more. So there are these kind of elements and these labs are. It's a very strong relation with industry, right? Connections. They are funded since I came, when I came actually. So as one change happened before, these labs are also supported, supported by government in particular darpa. But it was a point in which that kind of, at least that particular DARPA funding dried.

Yeah, yeah, yeah.

So kind of when I came to Berkeley, zero this kind of. Now you need to go and remember, you know, you know, getting more money from industry. I remember from Google first time we got. And it was unheard by then because you are asking for 500,000 per year, right? Life for four years. So but you know, we got. And this is, we got. So now you have also this kind of very tight connection with the industry. It's a very good environment to see the problems, right? To understand the problems, right? And then you can see that and you try to think about, also about obviously about trends, right? Because trends are important, right? You have to be aligned with these secular trends, right? You need to bet on the right trends because these are things you cannot change, right? Or it's very hard to change. So you are not aligned. It's not good. And these trends actually there are multiple trends and the multiple trends kind of open gaps between them. And these are kind of opportunities for problems. Like for instance, for big data, it was clear you have more and more data and the amount of data people collecting were just growing, right? It was pretty clear, right? It's like Google has seen that years before and they built all these systems, but now everyone want to emulate that, right? That's why Hadoop was created, right? And then you see, you start to see again, you look at that and you are working in that area and we are, you know, we have all these Hadoop people coming to our retreats on these kind of labs and we are friends with them and we started to see problems. And then you try to use them. And then you just like for instance, there are two things which happened with Hadoop. One thing is about a group in our lab. Oh, by the way. And the other thing, what happens in these labs, they are interdisciplinary. It was people. We are in this kind of rat lab. It was people from machine learning systems, databases, networking. So there are these groups of Michael Jordan students which wanted to compete to this Netflix challenge that Netflix released some data basically and asked for people to provide recommendations, come up with recommendations to build recommendation algorithm systems so that, you know, to beat their own recommendation. So they come to us and okay, it's a lot of data. How can we, what can we do about it? Right. We want to. And you know, told them to do. Yes, Hadoop. But Hadoop was very slow, right? Because, you know, and then, you know, Matei put together something quickly for solving this problem in which the data was kept in memory. Memory. The other thing I've seen is about like I have a previous company, Conviva, and it's about its own analytics company and kind of. It was very slow and we tried to do it, you know, it's like for ad hoc queries and there is no way to do it. And again, keeping the data in memory was a solution. It's one solution. That's kind of how we started. That's one thing, right. And you look at the trends. Yeah, it's obvious, right? It's like on one hand you have more and more data growing faster than the more slow. So you need to have. It's not going to fit on one machine and therefore you need to use multiple machines. And then the only other question is that are you going to have data sets that are going to fit, important data sets are going to fit in memory. Right. It's like that's kind of the first question. And they are. Because people, even for when they are doing, for instance, a whole query, we notice that when you look at the data from different clusters, Hadoop Cluster from Yahoo and Microsoft and others, and you notice that in a lot of cases, actually when you do queries and you're doing analytics, they are doing not only very rare on all the data you do say for the most recent data, you want to see what happens yesterday, what happens last week, something like that. So once you get that and you have a lot of cases in which the data is fitting in memory and memory is still growing quite quickly at that time. You know, that's. You kind of. You are pretty, you know, you connect the dots. Yeah, right. And then it's about solving the problem. And the other thing is happens and why they are related. You know, I'm talking about academy and industry because I'm telling, you know, people, you know, some people, you know, there are people in academia push back and say, you know, it's a lot of engineering here, this is not you should do in academia. But one thing is what I've always found it is very satisfying. If you build a system in a new area and that system it's used by other people, then you can, you are in the in the best position to understand the new problems in that area because people are going to use your system in different ways, right? And then, you know, you understand that, right? And, and going back, if you know the problem, you are also in a good position to solve it, right? So actually it directly helps you with your research, right? To be ahead. Because otherwise what is the choice where you are finding the problems, right? Of course there are problems, very good problems in theory which are not solved by decades and so forth. But other things, what people do, they go to Google and Microsoft and so forth, spend time and to understand what are the problems they have, right? Because to solve. But that's kind of a little bit unsatisfying, right? Because you go to someone to learn about their problems. But the question is why don't people solve those problems? Maybe they don't solve the problems because maybe, you know, they're not as important as they're given time and maybe, you know, maybe for the right reasons they are, you know, too much to do in the future to. But this is the thing, right? You have to focus about the problem, on the problem and you have to focus on the trust and the way they're connected. You want to solve a problem ideally, which is going to be more important tomorrow than today.

What are the problems that you're most excited about right now? Like Spark Ray, what's going to be the next databricks or any scale.

So a few things. So I still think that it will going to be a lot of work right now what happens is that need to resync most of the software stack. Why? Because it's again if going back to the trends is that the demands of this application in particular AI application and so forth growing much quicker than capabilities of a single process or a single node. Even if you considered accelerators, right? So on one hand this happens. On the other hand, the infrastructure becomes much more complex. You need to run that application not only on one node, but on many nodes. It's distributed, but it's not only that, it's becoming very heterogeneous, right? Because in order to bridge a gap between the demand and capabilities of hardware, people build accelerators. That's why Nvidia is trillion dollar company, right? But now the infrastructure become even more complex, right? Is not only distributed, heterogeneous. When we started Spark, it was homogeneous. All the nodes are the same. Some storage, some CPUs, that's all. But right now, look at the Hydrogen 8. You have Nvidia and you have many other. You have TPUs from Google. Every cloud, it's having their own chip. Now you have MD and Intel say that it's everything, it's about AI, right? So that's kind of what happens. So now you have a huge gap between this application and this very complex infrastructure just growing in complexity. And then it's not only about the compute, it's about networking. You have Infiniband and you have all of this RDMA and so forth, right? So huge heterogeneity. And the software stack has to abstract away that complexity for the developers. There is no way around you want a single machine or you have this operating system to abstract away the complexity, right? That's why, that's what makes it easy, right, to develop all this application. Now it's extremely hard. So something is going to happen there. I think the other one is about this building application. You're talking about compound AI or talking about compound AI and things like that. Right now, this application, AI application in particular, large language models. Everyone is talking about large language models. The application are like assistants for humans. Humans are in the loop. If you are thinking about customer support, if you are thinking about copilot, if you are thinking about Q and A question and answering even summarization, you have a human helping a human to be much more productive, which is a fantastic application. But they cannot be autonomous. They are not theatre autonomous. And to go from having the human in the loop to being autonomous is a huge gap, right? Because with autonomy, to have something autonomous, you need to have someone which is running kind of. It's more deterministic, it's more reliable, like far more accurate, right? And you know, you need to get there, because if you don't get there, you know, you are still limited to having the system where the human is in the loop and the human is a bottleneck will become the bottleneck. It's, you know, it's. It's just certain number of people on the planet. And I think that there is about a lot of work will be about how to make I would say at least some aspects of this lag language. Building the lag language model application more like an engineering discipline. Right. Where you can build much easier systems from smaller components.

Okay, so the two next databricks will be distributed compute across heterogeneous hardware and autonomous compound AI systems. Noted. I'd love to touch on another thread that you mentioned just now of kind of how funding constraints drive what you're working on. I'm curious what you think right now of. There's been a lot spoken about kind of the. Almost the brain drain in AI right now because the universities just don't have the funding that you could get if you went to work at one of the big research labs. How do you think about that? What do you think is ideal? Does operating under constraints force creativity for you? What do you make of all that?

Yeah, that's a great question. It's true. I mean it's challenging. It's very challenging. And when I came to United States, I came to do my PhD and I graduated from Kennedy Milanon University. I am originally from Romania. So one thing was admired, people admired about United States is that how well this kind of this, you know, the collaboration and the partnership, the three way partnership between academia and government and industry are working. Right. And prime examples at that time was obviously the Internet, right. Which was DARPA project. And of course academia had a huge impact and also industry was, you know, it's whatever third industrial revolution people were saying and that when it comes to AI today that kind of partnership is broken. It's like industries, every company is doing this research in silos, right. Even you know, they don't talk to each other as much. Academia like you said, doesn't have resources and the government doesn't invest as much. Right. So I think that's something to be very concerned about. Right. And that's why I'm also a big proponent of open source models. And US and California kind of will lose in long term if this doesn't is not fixed. So what happens now in unfortunately in academia one thing that happens is that of course there are, are some, you know, bigger universities and labs which can still afford to maybe train, spend 1,2 millions maybe to train some models still they are not in the same league like OpenAI. Right. It's like OpenAI and Microsoft are. They are talking about building the Data centers about $100 billion. Right. And I think there is the danger is that you are going to. Some of the academics are going to give up and try to Innovate around the edges. Now you can still innovate in this application and things like that and you know, I think there's a lot of innovations there but clearly, you know, you know it will be harder to come up with new model architectures and you know, innovate, you know, have. It will be harder. It's not impossible. Nothing is impossible. So yes, it's, it's, it's a, it's, it's, it's a challenge right now. I mean there are people like, you know and which are fortunate, more fortunate position maybe I'm one of them and which have access to resources outside academia. Right.

But.

You do want to level the playing field in order to maximize the innovation. The innovation comes from everywhere. Now this being said, it's true that scarcity and always in the past scarcity, sparse innovation as well. But the concerns about not having access to resources to play the same game like in industry, it's a concept.

I'd love to switch gears into some rapid fire questions if you're ready for it.

Yeah, go ahead.

Will anyone take meaningful market share from Nvidia over the next five years?

I think they, they will be at the minimum because they are going to. It's probably that. It's because Nvidia will not like to be accused about monopoly behavior. So their market share has to decrease under some percentage, whatever 70, 80%. So that one will be one of the reasons. But I think that if I have to name one company to. Of course there are clouds and for strategic reasons they are going to push their agenda to build their chips. Like still probably the biggest competitor right now in terms of the market share. It's Google with TPOs. Yeah, probably that will continue for a while.

What's one project or student in your lab right now that you'd want to highlight?

You know I'm going to cheat here because I think that both Vllm and Chatbot arena has been tremendous. It's like I'm not talking about Skypilot because Skypilot is like they started a company so I'm not talking about. But I think Vllm has been amazing. It's like it's one year old project and I haven't, I've never seen such a rapid growth and of course it's also part of the AI. AI is kind of compresses the time so it's that there is something about, about that and I think the other one is Chatbot arena because it just is fascinating to see the development in the space and to see how these different models where they are strong or they are weaker. And I think that having a front seat at that to see that kind of development of the ecosystem and in the space. It's fascinating.

Do you think the foundation models will commoditize?

The foundation model was a rare like.

A GPT4 or a Claude. Do you think, do you think there's a market to be made in, you know, providing these models over time or do you think it commoditizes?

I think people will continue to build larger and larger models. I think when it comes to serving is it looks like model distillations works quite well. Well by the way the model is where you train a smaller model on the outputs from the bigger model has been a lot of success with that. By the way, this in some way it says that it shows how important is the data for training a model. Going back early on in our conversation because you have higher quality data from the big model and use that to train the small model and it's working very well. So I think using multiple distillation models to reduce the cost of inference is going to be a way forward. But yes, I think for advancing and pushing the frontier, so to speak. Right. Pun intended. You are still going to see a lot of effort on bigger. On bigger models.

What are you most excited to see in the world of AI in 1, 5 and 10 years?

But I'm most excited about AI. Look, there is no question it's transformation know. Right. I think that and it'll change a lot of. A lot of things, everything maybe I think the most excited I am about it. It's about how do you make this AI more C systems, more predictable, accurate, verifiable, how you can can debug these systems. All of this kind of in realm of software engineering like techniques. This is what I think it's exciting.

What advice do you have for founders building an AI?

It's the same thing. Focus on the problem, don't focus on the hype. Hype is emotional, it's not reliable. Just look at the facts. It's like and look at the problem. Try to understand the problem and try to be truthful to yourself. It's about if you build an application, it's about production. It's not about the demo, it's beyond the demo. Of course the demo are important, don't get me wrong, are very important. But the production, this is the way the mindset you have to have. And the dangerous thing is there's so much hype and you think you can solve everything and you can do everything probably in certain number of years. But now just focus on exactly what problems you are going to solve. Convince yourself it's a good problem, convince yourself that you can solve it or at least you can have an mvp, right? You can solve a smaller version of that problem which is still very valuable for your customers. That's what I will say. Nothing. No silver bullet.

Amazing. Thank you so much, Jan for joining us today. I've loved hearing a lot about your own thinking and reasoning behind your own journey. A lot of the thought process behind finding the right problem to solve, building the right systems to actually be in a position to understand the best problems and then applying that even to many of the bold decisions that you've had to make in founding multiple companies from research into commercialization and the incredible success of databricks today. Thank you.

Thank you for having me, Sam.