The Future of AI, Open Source, and Enterprise SaaS with Databricks CEO Ali Ghodsi

Hi everyone. How is everyone doing today? Are you staying warm? Okay, so I'm Nithya Ruff and this is my super guest, Ali Goatse. So let me go back to the beginning to introduce myself. I head the Open Source office at Comcast. We connect people to incredible entertainment experiences as well as to Internet services. And I also chair the board of the Linux foundation, which is the home of Linus Orval's and Linux and hundreds and hundreds of open source projects. And it's kind of at the center of open source. So what we'll do is I'll go through a few slides to kind of set the stage of open source momentum that's happening both in terms of adoption as well as money going into open source. And then I'll discuss some of what databricks is doing, which is so magical, managing open source in a very scalable and enterprise way. And then we'll go into a lot of great questions and answers so you can actually get a chance to know what to do if you are an open source based company. So I really don't need to tell you that this is the 30th year of the birth of Linux. So about 30 years ago, a young Finnish student called Linus Torvalds released the microkernel for Linux through a mailing list and people started adopting it, contributing to it, et cetera. To the point where today Linux is everywhere, right?

Yeah.

And frankly it's in all of the different markets. It's all the way from supercomputing to AI to Internet of things, to embedded devices, name it and it's there. And Open Source has really become one of the most successful enablers of global innovation in history. In every part of infrastructure, in every part of user experience, you have some amazing open source contributions. Then you look at the kind of money that's going into Open Source and you realize that back in the day, in 1990, 2000, you only talked about companies like Red Hat. And Red Hat used to dominate the valuations for open source based companies. And then you Fast forward to 2010 and 2020 and you find that red hat is only 10% of valuations. Valuations today of open source based companies are close to 300 billion, 400 billion and climbing. And you have some amazing exits like Confluent. And then you have incredible unicorns in the open source based business like Databricks, like Confluent, like Hashicorp. And it goes on and on, and it's just beginning. And I'm so glad to speak to databricks because databricks not only started out in the AMP labs of UC Berkeley based on the Apache Spark, but they continue to innovate, continue to open source their projects like Delta Lake, like mlflow and Colas. And it's so important to discuss how do you balance growing a business for the enterprise in the data and the ML space, whilst also working with community and open sourcing and keeping that innovation engine going. So with that framework in mind, I'm going to. I'm going too fast here, so I'm going to ask Ali some questions here. Ali, you know it's very important to start a company, right? And you all did so many things right when you started an open source based company like databricks. Can you share some of the things you did which are still helping you succeed today?

Yeah, I mean, for us, one thing that was really important in the first couple years was focusing on the community around open source projects. At that time, Spark wasn't very well known. So really what we were doing, the first five, six people in the company and then honestly first 20 employees, was we basically were doing developer relationships. We would rent a car from Enterprise and we would get on one on one and we would go to all kinds of startups and companies here in the South Bay and elsewhere and we would give talks on Spark and sort of evangelize the technology and build basically ground up community around these open source projects. And that was super important. So we weren't focused on revenue, we weren't focused on making money or anything like that. We just focused on how do we get Twitter or Facebook or Airbnb or one of these companies to adopt the technology. And we were really going after the developers. We weren't going after the suits or the people high up with fancy titles, it was just the developers. And we would win them one by one and we'd answer the questions and they would contribute and join these projects. That was really, really important. That was sort of the beginnings of the whole figuring out how you get these projects to succeed. And we've carried that with us today.

That's incredible because community is so important. We. Why is community so important to the success of an open source based business?

Yeah, that's a great question. Basically the idea is that you create these. Basically it's a moat for the business model in some sense, right? What happens is you get millions and millions of developers around the world. We actually have 30 million downloads of these projects just alone every month. So it's 30 million downloads. And of course that's not people but it's almost like a B2C thing, right. In business, to business enterprise companies like us, you don't see millions of customers or so on. Right. But with open source, you have that ability to do that, and it's the community that does that. And once you have the community, it's kind of. You're a little bit unbeatable. And if you can figure out how to do it again and again, then you have something that's unique. Unfortunately, in many cases it ends up being founders create an open source project, they evangelize it, lots of people use it, then the founder becomes CEO or CTO or maybe retires and so on. And then the company only has that one open source project that they're monetizing. But in our case, we wanted to continue doing it. We wanted to figure out what was it we did right in the early days so we can repeat this and continue building this momentum behind these projects.

And often the developers, the focus on developers was so important because developers often become your first adopters, then they become evangelists and they are the ones who kind of scale it in an enterprise. Right. What is that process from developer adoption to enterprises paying attention to it?

Yeah. Before open source in enterprise business, what you would do is you would get the meetings, you would sort of explain what you have, and then they would say, well, okay, let's do a proof of concept. Let's get the legal team in, let's get the security team. And then it could take three months before you even have the POC started. And then once the POC starts, they're starting to evaluate it. With open source, you have people in the room that will raise their hand and say, I know what Spark is. I've already used it. I downloaded it on my laptop at home. I know how this Delta Pro, I'm a big fan. I know what this is. So that moves things along much faster. And also internally, let's call them the people in suits who maybe have the budget and make the decision on, are we gonna really work with databricks? When they look internally and say, can we trust these guys? You'll have that person internally say, yeah, yeah, yeah, I know who they are. I know these projects. You know, here's how that works. Well, how does this compare with that other thing? Well, I'll tell you because I know I've used it already because I downloaded a free version of it. So that's why that's important. That's one. But there's also, you know, there'll be ebooks There'll be books, there'll be courses, massive online courses, There'll be experts out there. Also, if a company needs to hire someone that has know how about any of these projects, they don't have to just come to databricks and say, hey, can we buy professional services? They can just go and hire from the community. So that's another reason why this is important. The final is the final reason, which is more egotistical for databricks. If you want to hire really amazing engineers. It's hard now, right? There's tough competition for, you know, good engineers from Google, Facebook and these companies. If you're doing open source development within your company, you can attract much better engineers because you can tell them, come here, work here. You can work on these projects and you can put these on your cv and when your next employer asks you what did you work on, instead of saying, hey, secret confidential, I can't break my NDA, you can just say, oh, I actually worked on Delta Lake, you can check it out, it's online, go look at my contributions. So it's also a way to attract talent.

I think it's magical, frankly, because of all the stuff that you said, the millions of people who adopted, the advocacy that they do for you on your behalf and then the fact that you have a ready made skilled workforce that you can bring in and the trust and the reputation that precedes the company coming before it. You also did something very, very good, which is you chose a name for the company that's different than the project. And why?

Yeah, actually there was big debate internally. We were engineers so we actually made long list of different names we would pick and we scored them and we did a bunch of research. But one of the things was several of the founders wanted to call it Spark something. At that time it was kind of in vogue to pick the same name as the project. Yes, there was Docker and they had the project Docker. And because the project was known, Spark was well known. The idea was, well, we'll get the lift from the community. They'll know we're the Spark folks. But from early on we said, look, technology comes and goes. Spark in 10 years will have probably aged and hopefully we come up with other innovations. So let's pick a name that separates the company from the open source technology and hopefully we can continue coming up with, maybe we'll come up with something that's better than Spark and we can even, you know, we don't need to sort of just be attached to it. In fact, I think One of the examples someone used in the early days was Google. At the time when Google started, they had an algorithm called PageRank that was how they would sort the, you know, how they would rank the different pages. And they could do that better than the competition AltaVista and Yahoo and so on. But we're saying nobody knows Google as PageRank or the PageRank company these days. It's something much bigger. So we said let's pick something that's independent. And we picked their bricks because we thought, you know, we'll build one brick at a time, you know, in the data stack as we're building it. So it was actually one of the important decisions early on I think.

So it separates you from the project, but yet and gives you a lot of Runway and an umbrella term to kind of innovate continuously, which you have done. I mean you've gone from data warehousing all the way to an entire workflow for machine learning, which is such a hard problem to solve. I know because I talked to our machine learning folks at Comcast and they love working with you all. So another thing you did right from the start was you went with a managed service or spark as SaaS. Why? I mean at those days on Prem was so popular. Why did you choose to do SaaS based model?

Yeah, that's a great question. We thought a lot about this. We were not particularly fond of the open source on Prem model which is largely services and support around open source projects. We thought there was one example of a good one which was Red Hat, but there weren't many other examples that were doing well. And now we've seen Red Hat also did not survive as an independent company. And the reason for it was, I mean we analyzed it and what we saw at that time Hadoop was really popular. And we were analyzing the Hadoop vendors and their business model was like the Red Hat model on premise. And what discouraged us was we saw that they would, the first vendor at the time, Cloudera would go in, they would sell this support and services around Hadoop. The customer would build these big data lakes with data in it, but then they would find that a year later they can get cheaper support and services from a next vendor. At that time it's called hortonworks, they're not around anymore, so they would get cheaper support from them and then two years after that they would maybe hire some people to do support themselves or maybe they would just outsource it to some other company, but they would keep the software because just open source free. So the business Models were not robust in the sense that really you were really selling services masqueraded AS software. The SaaS model we were super excited about, we looked a lot at aws, Amazon Web Services and we thought they had done an amazing job. They basically offered lots of different open source technologies as a service. And what you do then instead of just as a customer, just buying, you know, getting free software, installing it, managing it yourself on your own data center, you're actually renting it and you're delegating the security, the reliability, all the difficult things to that vendor. And that's just much better business model. You have less people turning and you're not in the business of doing support and services. You're in the business of automating problems away instead of just adding people to it. Frankly speaking, we thought that some of these vendors, they might not be incentivized to make the software great because they make a lot of money by sending professional services. So we wanted to make sure we're incentivizing ourselves to do things in an automated so sort of way. So this became super important to us. So we host this software for you. We'll take care of all the difficult parts, like we'll make sure it's secure. If you have a data breach, it's our fault. That's not the case on premises. It's the IT team that's responsible for data breach. We make sure it has good SLAs, we make sure that it's super fast and optimized and we'll take care of the upgrades. Which actually is really painful.

Yes, right.

It's like, you know, people take that for granted, but upgrading to the next version of the software could take a year. You could have outages. We'll take care of that for you. And you kind of almost lease the software from us.

I recently started a solution first on prem, and it took us forever just to operationalize it and install it and get the configuration right and manage it. And we didn't have the team to do it. So I then moved to a SaaS based model and it was incredibly helpful. So this is the age of the public cloud and a lot of ISVs really struggle to compete with public cloud. How did you do so well? I mean, how did you manage to compete successfully and also partner with all the public cloud vendors?

Yeah, I mean, it's the fact that we started in the cloud from the beginning. I kind of joke and say that running software in the cloud as a service, it's kind of like if someone Told you you had to stand in your hands for the rest of your life, right? It's really hard, like most of us can do it, but if your life depended on it, you'd figure it out and you get good at it. But it would take you a while to learn, right, to not use your legs at all and just walk in your hands. That's how it is running software in the cloud. So what we think is a lot of vendors develop great Open source software. They sell it on prem, they haven't yet learned how to walk on their hands and then they hear from their customers that oh, we want to run this stuff in the cloud. So then they try to maybe offer the cloud version that is trying to explore walking on their hands. But then they realized, wow, this is really, really hard. And by the way, the cloud vendors have been doing it for decades and they're really good at it and they're having a hard time then competing. We started from the beginning just walking on our hands. So we only had a SaaS offering in the cloud and we were good at it from the get go. So we can compete with the cloud vendors making sure that the reliability, the performance, the security, the upgrades, all that are seamless. And that's really, really hard to do. So that's why I use this analogy. So I think that's really the fundamental difference. And then that's why you see people saying, oh, this is so hard in the cloud, maybe we should lock down the license or something. Right?

Yes.

To prevent the cloud vendors from doing this. But you don't have to if you're actually good at this.

And I mean, you solved a very difficult problem for customers and you added great value to by offering it as a cloud service. And in the age of cloud, do you think Open Source is dead or gone away or will it diminish in importance?

Not at all. I think Open Source is here to stay. In fact, I think Open Source has the potential to replace pretty much any software that's, any proprietary software that exists on the planet today is, you know, in, in a really vulnerable state in the sense that it can be disrupted by an Open Source version. Back in the day people would say Operating systems are really complicated. You have to have PhDs and really skilled people to write an operating system and nobody's going to be able to. So you can't open source that. But Linux showed that you can actually even open Source the hardest, most complicated software. We're seeing that the same thing is happening in the data stack. So there's A modern data stack appearing where we're open, sourcing it, and bit by bit, I think the proprietary pieces that exist in the modern data stack are going to be disrupted by open source. So I think it's on the contrary. And if you look at companies getting funded now in Silicon Valley, most successful projects get funding immediately. Most successful open source projects get funding really, really easy. And I had a close friend of mine who was actually trying to raise money recently and they have amazing people, they had amazing idea and I introduced them to some VCs and they did not get funding. And when I asked them, hey, what happened? These people are really, really good. And they said, well, they didn't have an open source project, they don't really have any sort of customers yet. We didn't want to take a risk. So with so much investment going towards these open source projects that are successful, I think we're going to see much more of it in the future.

I completely agree with you. I think people will innovate on open source and, and as the projects get more popular, that's where the commercial support for the project or even the cloud vendors are picking winners from open source. But someone has to start the innovation and open source innovates so fast and globally and it's very, very exciting to me. You have to start somewhere and then you can.

Yeah, yeah. In fact, you can think of it this way. AWS took a lot of open source technology, offered it as a service. Yes, Right. But someone had created those open source projects.

Exactly.

And we've shown you can actually do that. We hosted our own Spark service in the cloud and we competed well with the cloud vendors. So I think there's actually a huge opportunity in the future. All these different services you need in the cloud, you can start them and you can offer them as a service in the cloud. That's what's gonna happen in the future. And the new companies that are coming along now won't have some of the struggles of the companies of the past that created open source software because they'll be cloud native, they'll start in the cloud and they already know how to walk on their hands.

Exactly. And I think you did some other things right. And there are other advantages for you, which is your multi cloud and you support hybrid cloud and you're not just stuck with one public cloud. And that helps customers who are all kind of going hybrid and adopting multiple clouds.

Yeah, that's an important trend today.

Yeah.

Two years ago people would ask about it. Now sort of the requirements are becoming louder and louder, that we don't want to just be in one cloud, we want to be able to run it anywhere. And I think this is a secular trend. So in a couple of years, I think it's going to be an absolute requirement. If your software only works on this cloud, I'm not sure I'm just going to be in this cloud. Some of my customers might not want to run in this cloud. They might mandate to run over there. Or if you have an international business, the clouds are not all equally popular in different parts of the world. So in Asia, I might want to use one cloud, in Europe, another one, in the US, a third one. So I want to make sure the software that you provide works anywhere. So that's why I think multicloud is going to become more and more and you're going to see huge tailwind for companies that are multi cloud, like us and others that are going to also pop up actually.

And you also gave some really good advice to companies that didn't start in the cloud. You can't really play with licenses and expect to win. What I've seen is it doesn't work. People just don't like the fact that you have polluted a license or you're not setting the right expectations. You have to innovate and you have to deliver services on the cloud. We have about four minutes before we go to the conclusion. I really love what you're doing with MLflow and call us and the whole machine learning problem set. What is the problem that enterprises are facing in machine learning with all the massive data sets that they have? And how are you helping solve that whole workflow?

Yeah, the problem is really to get to machine learning, to get to those really interesting use cases where you automate everything. There's a lot of hard work you have to do on the data side. And actually you have to do data and AI together really well. That's what Google, Facebook, Airbnb, Twitter and these guys did really well. They built custom platforms internally using thousands and thousands of engineers that really enabled not just one data team in the company to succeed. They enabled the whole company to use data and AI. If you look at an Uber or a Facebook or a Google, they wouldn't be here today if it wasn't for AI. And the whole company uses it, right? Enterprises, data, they're not quite there. They have one data team and they sort of, you know, they're trying to either build it themselves, but they don't have the resources, or they're trying to buy software from vendors and unfortunately, the space today is divided into two types of software. AI software that's not so great for data. So it doesn't really have data capabilities. There's lots of companies that provide machine learning capabilities, but they have no data processing capabilities. And then you have data vendors that provide you data processing and they're good at that, but they have no AI capabilities and you really need both. So what we're trying to do is give one platform where you can do all of these things in one place. That way you can bring the whole organization to the platform. Because in a large organization you'll have data engineers, you'll have data scientists, you have business analysts, and then you have sort of domain experts in the business, you have administrators, you want all of them to be able to use the platform. So we're trying to sort of build that stack and open source each piece of it as we go along. That's kind of the vision. So when we started with Spark, that's just the way to get the data. And then we moved to Delta. Delta enabled you to basically get reliability and the kind of enterprise capabilities you need to make the data really fast. MLflow enabled you to do machine learning. And there's actually other projects like Redash and so on that we've open sourced for visualization and other things. So we're trying to build up this whole analytics stack in an open way for enterprises.

That's fantastic. I know at Comcast we have tons of voice commands, millions and millions of voice commands where people are using natural language to do queries and say, hey, can you show me all the movies that Pierce Brosnan did? Or whatever. And then I think your magic at the back helps us kind of get to the right results.

Yeah, I really like the Comcast example. Basically you speak to the voice remote, the audio goes in, it goes into a databricks lakehouse, it actually gets processed by MLflow, then it's stored in the Delta lake and then they do the synthesizing and figuring out what the commands are. And so the whole thing gets orchestrated. And you might take it for granted, but it just works. You can just speak to it and it leverages actually all these different components. That's what I like about it, is that it actually, it's not just. Yeah, it goes through all of the whole pipeline. So very excited about it. Actually, there's a really great talk that I think Jim Forsyth at Comcast gave at Spark Summit a couple of years ago on how that exactly works. So I can encourage people to watch it.

I Love that example because customers want to speak naturally and not have to kind of remember which channel to go to or which movies to go to, et cetera. So that's a really good example. And I know you've also helped pharmaceutical companies arrive at better drug results, which are so expensive. And shell company help them kind of find oil resources and just really accelerate the path to results and predictive.

Yeah, last year was pretty cool with the vaccine development. The they don't want to talk openly about it, but most of those vaccine teams behind the scenes built basically what we call lake house. We stole all your data, store all your data and do AI on it. And it was really used for these sort of vaccine development both here and in Europe. So it's exciting to see how the technology is actually now starting to impact these businesses in a serious way. That was not the case 2012 when we started, AI was sort of a cool idea, but enterprises were so far away from getting these use cases in production and now it's becoming super strategic for really large companies that you would not think.

I love that you democratized AI if you will. Only the really big companies like Google and Facebook could afford it now. All enterprises can have access to those kinds of magic. So that's really fantastic. I am going to actually go to the takeaways and summarize the takeaways since we have about three and a half minutes. First of all, I think open source is here to stay. That's where all the innovation happens and then it moves forward and is used to productize by various companies and put commercial and enterprise grade support and SaaS behind it. There's been billions of dollars, I think anywhere between I'm hearing numbers like 15 to 30 billion. So I use the larger number here. There's a lot of money going into open source based companies, as Ali was also saying, and it's just growing. The category that someone has created is called commercial open source and that's what they call it. Successful exits are happening every day, which really is a remarkable thing. It says that this model is mature and there are strong business models like SaaS. SaaS I think is one of the biggest business models because there's a tremendous value add and the software problems of installing and running open source is taken away. And the last thing is I think Ali and a lot of companies that do it well really do a good job of working with community because community helps them adopt the solution, evangelize the solution, bring it into enterprises and frankly recommend the solution to enterprises. It also helps do More innovations and contributions. And what I like about what databricks has done is most of these projects are in a neutral home like the Apache foundation or the Linux foundation, where a lot of companies can contribute. I think companies, even competitors, right, contribute and kind of contribute, collaborate at the project level and then you can compete separately at the product level in a different way. Any other conclusions or points to add here to the conclusions?

No, this makes a lot of sense. I think the SaaS model applied to open source is going to be really disruptive because of this transition period. We had these on prem open source vendors. People think maybe open source won't work in the future in the cloud, but I think people are gonna see that slowly you're gonna find lots of companies like databricks that open source technology and then offer it as a SaaS service and they're gonna be the ones that are gonna be dominant in this new sort of SaaS era.

So I think so too. I think that's the way to, I mean, open source, let's face it, it's highly, highly innovative, but it often has rough edges in terms of operationalizing it or doing it enterprise scale or enterprise grade. And the SaaS model takes away all of those rough edges and makes it easy to use and go straight to the functionality and straight to that. And to be honest, we need to continue investing in a SaaS model to compete in a very cloud world. So I love what you're doing. If I just want to thank you, Ali. It's been fantastic talking with you this morning and really showcasing a company that's doing open source, right, is working with community, is also innovating continuously and is serving enterprise customers like my company. And it's a pleasure to be here this morning.

Thank you so much. Thanks for having me.

Thank you, thank you.