#302 Making AI Applications like Greased Lightning with William Falcon, CEO at Lightning AI

Welcome to Data Framed.

This is Richie.

The AI boom has resulted in an explosion of new tools for creating yet more AI. That means that the AI engineers technology stack has become vastly more powerful, but also a lot more complex. It's the modern data stack conundrum all over again. Today we're talking about how you can streamline your AI tech stack in order to develop AI products faster. Our guest is Will Falcon. He's the creator of Pytorch lightning and the CEO of Lightning AI. PyTorch Lightning is kind of Ruby on Rails for AI development and it's one of the hottest open source AI frameworks right now. I want to know why Will created it when open source is important in AI development and how to just stop fretting about tools and get on with building things. Let's find out how to develop AI.

Applications like Grease Lightning.

For AI to really take off, the focus has to be on how to deliver a business outcome to people, not what tools to build. Because people don't necessarily care so much about the tools, they care about what it lets you do and what the outcomes are of using those tools. And this is coming from a tool builder, Right? But like, we focus a lot on how can we enable you to build like an end product. I think that models should be open source. Like the thing that's interesting about the models is not the model itself, but what you do with the models. Right? You don't regulate technology, you regulate applications of technology. Right. That's kind of where the government really should be focused on.

I want to pick up on something that you said at a meetup I attended. So you said that you think artificial general intelligence is going to come out of New York, not San Francisco. Can you talk me through why you think that?

I think that in New York you have people from different backgrounds that are coming at problems from a different perspective. You have lawyers, you have doctors, you have market marketing people, et cetera. And so I think that variety or breadth in viewpoints is actually what I think will happen, will create the next generation of companies. And probably to some example, to some extent, like a Harvey or something like that is, I think, a good example of that where it's like lawyers that are involved, they're doing something very specific. Not that there aren't lawyers in San Francisco, but I think in San Francisco people tend to think more uniformly about what they built. And it tends to be kind of solving their own problems. And most people are in tech. And so that's kind of the issue that I see there okay, so there's.

A difference between just thinking about the technology side of things versus thinking about the domain side of things and then how the technology can help the domain.

Yeah, because I mean for AI to really take off, the focus has to be on how to deliver a business outcome to people, not what tools to build. Right. Because people don't necessarily care so much about the tools, they care about what it lets you do and what the outcomes are of using those tools. Right. And this is coming from a tool builder. Right. But like we focus a lot on how can we enable you to build like an end product.

Okay, that's interesting. Do you want to expand on that? So how do you get closer to solving domain problems with lightning?

Yeah, I mean, you know, there's, there's like the journey that people go through from idea to like a real product in production and there's like a path. Right. And it always follows something like the ML life cycle. Right. You are like, sorry, you like label data, you collect. Well, you first collect it, you label it, you prepare, there's a bunch of things you do there. Then you might do some sort of like, you know, distributed workflows where you might do embeddings and then at some point you end up with like a clean data set, I guess. And then now you need to decide am I going to fine tune or pre train a model? Am I going to like, what approach am I going to take there? And sometimes you might not. You might just start with an API and then you use rag anyways there's like so many decisions along, along the way. And then ultimately I think where most people land Is at a PoC deployment, like something where they're like, wow, look at this cool thing that works for like seven people. But the real answer is, well, how do you get something that's actually live in production and like really reliable and so on. And so the kind of approach that we have is really more like we have AI Hub, which is like the end of the journey. So you don't have to go through that. If you just want something that is an agent or an expert, or if you want a fully built Rack system or you want an actual vertical end application. It's funny because if you think about what it actually takes to build. So let's say something like a Harvey, that's an and application. Right. So in AI Hub you can find those kind of things. They're like very specialized to domain, you just hit them, hit one click deploy and then you got that. But to build something like that you need a workflow usually, which is kind of where most ML companies are sitting around. They're like, well, we'll do it with rag, we'll do with fine tuning. There's like 20 million different approaches on how to do it. So those workflows are things you can find in the AI help as well. So they kind of give you that final step. And workflows themselves are made up of atomic particles that are models, vector, dbs, et cetera, which you can also find as atomic isotomic things on the AI hub. And then if you don't like what's on there, then we have a whole platform of tools that you can use to build those things ultimately, which is studios. Right. So you want to train a model, go for it, you want to embed some data, go for it. Whatever it is, there's like a suite of tools. So that's kind of the approach we take and we hope that we get. Our goal is to really solve that last mile problem for people. And I think that's where a lot of the value lies.

Wonderful. So basically they're saying that a lot.

Of the hard bit is just getting.

These things into production and getting that end user application. Is that the biggest challenge you think is with development?

Well, it's hard. Once you already decided what the problem we wanted to solve was, you took the creativity to decide how to apply AI in your enterprise, then you hire the people who could do the work, then those people decided what tools to build, what techniques to use, and, and yes, if you had all that, then the hard thing was getting into production. But like there was so much hard stuff before that.

Okay, I want to pick on something you said. So you're talking about there are lots of different types of tooling. So you're saying you've got the end user applications, but for developers there's different layers of things. So you've got workflow tools, you got tools for rag, you've got tools for fine tuning. There's so many different tools. So I try and follow the space, I can't keep up. Can you give me an overview of what all the different sort of tasks and tools are and what does a tech stack look like?

I want to start with the outcome, so the solution. So for example, if you go to AI hub, you're going to find something called the stock research expert, right? Or sorry, stock research agent. And this agent, literally you can give it a ticker and it'll go do research on that stock and give you a report that's the kind of thing you build in a financial, like a investment bank, right?

Okay.

So that's the ultimate application that in itself is like a whole company potentially, right. Where like, it's like all they do is make that thing really, really good. How did you get to that? Well, you step one level down, you take a workflow, like a fine tuning and deploying workflow that takes some model, fine tunes it on that data and then deploys it and then teaches it how to use tools, right? So that second level is something that you can build on the platform with the tools you have. So we have a way for you to like create for example, a job that does fine tun, and then a way for you to create a different thing which is like a deployment that's like a Docker base image. You know, these are two like individual things usually where you need two different products to do it. So with us it's really more like a single platform that will let you do both at once, right. With the same workflow. It's very, very straightforward. And so that gives you the workflows, right? And then, and then if you wanted to build each particular piece of that for the fine tuning, you actually might need something like data generation and so on. And so ultimately for when you're building those particular tools. Sorry, I mean, it's really confusing because we're talking about so many layers here, right? When you're building those, I guess you can call them, I don't know, atomic pieces of us AI system, you're going to need a set of tools. So it all starts with the studio, right? And the studio is like a cloud development environment. So think about it like you have your VS code on your laptop, except that it's not on your laptop, it's on the cloud and you've got GPUs there, and whatever you could do in your laptop, you can do there. And so if you say, look, I'm fine tuning a model, you're going to need some GPUs. Okay? So get GPUs, takes a few seconds to get it. And now you can clone your GitHub, repo, whatever you're going to do, start developing on it, testing it, iterating, swap GPUs if you want to test a different type of GPU or not, and it's going to persist, so it's not going to lose your progress, right? So you don't have to install anything again or whatever. And then you can do a lot of the iteration, the development side of that, right? And then you say, okay, I'm ready to train the model. And then from there you like package it up and you submit it as a job and then it goes and trains and then you can say, okay, I now need to deploy a model. So you start a separate studio now and in there you set up your model checkpoint, you set up your server, you debug it, you like run the server live, you can hit it because it's on the, on the web, right. So it's got ports, you can kind of really load test them, do all that stuff. And then you ultimately package that up into a Docker container and then you ship it and it gets deployed serverless as well. Right. So really, no matter what the workflow is, the kind of studio is kind of like the base foundation for it usually.

Okay. So essentially that there are really discrete tasks within this AI application development space or within agent development. So maybe fine tune is a task, maybe deployment is a task, and you can deal with them one at a time. So it sounds like with Lightning Studio you're trying to have the entire workflow in a single tool, which is quite different to a lot of other companies where they're focused on one segment of development. So what's your motivation for that approach?

Well, that was the number one thing we had to do from day one. So I was, when, when I was working, when I started doing AI, like more professionally, right. So I started as an undergrad, like missing out in 2015, doing research. And then when I did my PhD, I was at Med, I was at Facebook AI at the time. And you could already tell there were a lot of tools that were coming out. There was like a lot of different things. You had different types of GPUs, you had different, like experiment management stuff, things that we take for granted today. But like it was like a lot of these different things. And so when, when I built Pytorch Lightning, the first thing I focused on was how do you get all those tools to work together nicely in the training framework. Right. So which is like a very, as one level below kind of where the platform sits, so the training framework. And from that day, that was like 2019, you know, I was like. And I had come from like web development, other stuff. And I knew, you know, you have an explosion of libraries. And I was like, I think one day there's just going to be a million tools that are happening and they're going to come, come at us, you know, fast. And I want to build a platform that basically is kind of tool agnostic. And like it can pivot like that very quickly. And, and the value to a customer is like when a new tool comes out that we curate it, put it on there for you and show you how to use it so you don't have to figure this out. And like, we'll also tell you if it's not a good tool. Right. So like, to your point, how do you keep up? I don't think you do. I think we keep up and we tell you what's interesting. Right. And so that, that was really the genesis of the idea and then it just took about five years to figure out how to do that. Well, because it's a very hard problem, of course.

Okay, so not having to worry about keeping up with all the latest tools and having to use them in different manners, that does seem like an incredibly powerful thing. So I guess the advantage of Lightning here is just giving you some sort of consistent interface to whatever tool you want in order to develop your AI applications.

Yeah, I mean, look, it's a natural progression in things, right? Like your phone here, this thing was a replacement for you carrying a calculator on a bag and like a separate device for note taking and like a flashlight and a camera and all these other things. And so it became a platform now that has all those things in it. Right. Something like a Slack. Like a Slack was a replacement for a special tool that you could send images on and a special tool for texting, a special tool for sending videos and this and that. So I think all roads lead to an end to end platform for pretty much any type of new technology. And for AI, it's going to happen. Like you're not going to have specialized tools for training or serving. There's just point solutions. Those are band aids today for the real products. Right. And so that's kind of the approach we took. And so to us, all those things are plugins ultimately.

Right.

Like, I don't know, they're just built into the thing. Okay.

So it's probably starting fights with a fair few other startup companies saying, all right, your company's just a plugin. But yeah, this is an interesting approach. All right, so we talked a bit about the tooling side of things and I like to know who's going to work on this. So tell me what sort of talent you need in order to build these applications, this sort of general role of AI engineer. But can one person do everything or do you need different people with different skills?

Well, with the right tools you can. Right. So I think, I think like, you know, It's a good analogy. Like people like, you know, car mechanics maybe, or like construction, right? Like, you know, could you work on a car today? Could you, could you like, fix stuff on your, in your house? Like, you're not a specialized. Maybe you are, I don't know. But I'm not like a specialized mechanic or construction builder. But like, I'm. I guarantee I could go to Home Depot and find the tools that would actually help me do something that I couldn't have done 50 years ago without being very, very specialized. Cutting wood or something. Right? And so, so I think that the right tools help you level up the people that you work with. And so our goal is to make it accessible for anyone in the world to build AI products ultimately. And I think right now a huge part of our users are actually developers who know nothing about AI. And they're coming on here and they're like, great. I don't really care about AI. I care about building a product and I don't care how that gets done. But they're familiar with things like Kubernetes or whatever else, and so they can deploy things on the platform, they can grab a Docker container, they can put it serverless. And then when they train a model, it's kind of a no code experience. So maybe that's how we do it. It's like for us, the no code stuff is like, if you're not an expert, then go for it. If you're an expert, we give you the full code experience and you should be able to tweak your models and do whatever you want. Right? But I think if we're successful, anyone in the world, no matter what who you are, should be able to build AI in whatever shape that means to you. So like, AI to my grandparents means like a thing that I can chat with and they should be able to do that. Like, what does that look like to you? Right? And AI to a PhD means a model with 24 layers in here in this particular way, and as a transformer with this type of attention or whatever, then do it as well. Right?

Okay. So you need different skills for different types of AI. You also mentioned that a lot of these people who are developing AI things come from a software development background rather than from a data or machine learning background. So can you talk me through what are the most important skills you need in order to work with AI to develop things?

You know, I came from the PhD background in AI, right? And like the researchers I think are, they love the research, they love to get caught up in the type of model and the way that you fine tune and this and that and like, like, you know, to build AI products, it really makes marginal difference most of the time. Like those kind of decisions, right, they're actually been made for you already in the open source. Like we've all done it. Like we, the models are there, they're already set with the standards, you just have to run them. So I think software developers in general are best equipped to do this because they're treating this stuff as a tool to achieve an actual business goal and so they actually will not get caught up in like all these details. And they actually have pretty solid like software engineering techniques like how to, how to CICD workflows, et cetera, et cetera. And that's actually what you mostly need. So I don't know, I think you mostly just need to be a fairly reasonable software engineer who's not afraid of it, who's willing to run something and see if it fails and be okay with having something that's working and you don't know why it's doing what it's doing and that's okay. I don't think anyone here drives a car with knowing every detail about everything that happens within that car or learning physics to drive a car. You don't do that today, so there's no reason to do that to hold your to that level. So I think it's open mindedness and being okay with not knowing everything probably is the core stuff. Okay.

It's definitely an important skill to be okay with not knowing everything because I mean at this point it's just impossible to keep up with everything there is. And I think that for the data framed audience, a lot of them come from a data or machine learning background. Are there any particular software engineering things that you think are good to know if you want to work with AI?

We use notebooks for scratch work, things you're going to throw away, right. But when you're finished doing the work that you're doing, put it into Python scripts, like actual Python scripts with like real entry points and this and that, so that you can put it into production, meaning you can train it, you can deploy it, et cetera. We provide a lot of tools to kind of bridge that gap. So like if you for example wanted to deploy a model and that model sitting on a notebook, you know, like, you know why I think it comes from? It's because in legacy data science you could train an XG Boost model on like a CPU notebook and like three lines of code and it was fine. But in AI you need GPUs, you need distributed processing, you need all this stuff that like a notebook cannot do because it's literally a single Python process. It cannot go and do multi process, multi machine things because it's like impossible. Right. And so, and so that's why we have to break out of the notebook thing. If you're doing AI, if you're doing data science, stick to notebooks, it's great. But so if you want to grab that model, then we have something called let's Serve, for example, where you can grab that model checkpoint, drop it in into this. It's kind of like Pytorch lighting, but for serving like a harness, I guess. And then you have kind of the few lines of code that you need to do and then the MLAPS is kind of handled for you automatically behind the scenes. Right. It doesn't have to be a Pytorch model, it can be an XG Boost model, it can be whatever. But there are these types of tools that are helping bridge people out of these things. So I would say just use the right tools for the right job. If you're experimenting and if you're doing analysis, stay in a notebook because you need the interactiveness of it, the ability to visualize things a hundred percent. But if you're going to train or deploy things for real, then you need to learn Python without a notebook.

Okay, that's interesting. And probably fighting talk to a lot of data scientists. So, yeah, I think there's definitely some problems with notebooks. But does that mean there's a gap in the market for a notebook that does scale to distributed computing and working with GPUs, or is it just that notebooks are fundamentally unsolvable and everyone should switch to writing scripts?

No, I think people have tried. There have been plenty of products that have tried to do that. I'm not saying never, maybe there will be something, but I think people can, I don't know, they should really just switch to writing the scripts. It's not even the distributed part is like the out of order of a notebook. How can you guarantee that this code was executed in exactly the same order every time? You can't, because people, notebooks, they'll do like cell one, cell four, cell two, cell three and then cell seven. And that path is what led you to the model. And that's not reproducible. Right?

So yeah, this sounds like it's a fundamental problem then with notebooks. So if users aren't constrained to executing things in order I can see that that's going to cause problems in production.

Don't get me wrong, I use notebooks, but I use them for the right thing, which is where order doesn't matter and it's exploratory work. But when I'm going to do serious work, I'm going to use Python scripts.

Right, okay, all right, that seems like a good thing to know about. So Pytorch is very much the dominant player in terms of frameworks for doing machine learning research. So why did you feel the need to create lightning? So what was missing from the base Pytorch?

I equate pytorch to like JavaScript. So would you build a website using JavaScript or would you use something like React?

Right, okay. So yeah, it's going to take you a lot longer to do raw JavaScript everywhere.

Exactly. And like you're going to reinvent the wheel every time. Do you really need to reinvent state management for every single website you're going to build? It's a cool experiment if you've never done it. It's the same thing in Pytorch. Like, do you really need to reinvent the way to save checkpoints, the way to do distributed training, the way to plug into fstp, the way to do multi GPU training, the way to log things, manage experiments, collaborate? Why would you do that every time? It doesn't make sense.

Right.

So yeah, I mean, that's really why, ultimately, because if you're doing anything serious, you're going to need a framework. Right. It's only the purists who are like, ah, let me do this. I'm like, yeah, maybe like, cool story, you wrote a JavaScript website. But like, you know, Facebook isn't written in JavaScript, it's written in React. Right. Google isn't written in JavaScript, it'S written in Angular. If the best people in the world aren't using JavaScript directly, there's a reason for it.

Right, okay, cool. So it seems very sensible having a framework just to speed things up. So are there any particular tasks you think lightning makes easier compared to lower level Pytorch?

Yeah, I mean, generally like just the ability to experiment. Right. But I would say the thing that it shines at the best is distributed training specifically. So with Lightning, you can write your code in a way where it can run on a CPU or you can run it in one GPU or four GPS or a thousand GPUs without changing anything about the code and it'll just work out of the box. Right. And you can do it even on other chips, like trainium chips or TPU's or whatever. Right. And I think that's really what's going to give you the power to kind of go between laptop, cloud, whatever you want to do as well. And not to mention all the best practices that have been embedded in there. Right. So I'll give you context. So Lightning has had probably close to a thousand contributors at this point. We did the math recently. It's something like 400,000 developer hours. So if you could, you know, and that's going to get you stability, correctness, like, you know that the code is not going to have an issue when you run. You didn't swap the backward with the way when you released the gradients, or you didn't do the stub before the backward or whatever. All that stuff is sorted out for you already. And like it's very, very robust. So unless you can tell me that you're going to invest 400,000 hours into your own code base to make sure you have no bugs, then it seems a bit silly to not leverage the things that people have built. Right. So those are interesting stats there, but there are so many small optimizations there as well. Like the ability to go. Not even small, I guess, to mean that they're small now, but like 8 precision, 8 bit, 4 bit, 2 bit, the ability to just test one batch of data and see what happens, try to overfit one batch. All these, all these things you Learned during your PhD that are embedded in there, and they were embedded in there by me, by people at Meta, by people in the Pytorch team, by people at Facebook, AI, by a lot of amazing researchers all over the world who put the best tricks of the trade into this thing. So you don't have to learn what they are anymore. Right.

That seems very useful. So just building on, I guess the shoulders of giants is the big point of open source. So can you talk me through what the relationship is between the Pytorch Lightning developers and the vanilla Pytorch? I think it's mostly people at Meta. Right. Developed Pytorch.

Yeah, I mean, it's, it's over the years, it's varied. Right. So Pytorch is, is something that came from Meta. Right. So it's not something that we developed. And, and, but you know, when, when Pytorch was coming out, there were a few of us who were like very early. So my CTO Luca was actually much earlier than I was working on Pytorch like very, very early. He was actually on the Pytorch paper itself, right. When it was published, and the autograph paper before, which led to Pytorch as well. If you guys ever read this book, Deep Learning with Pytorch, he's one of the authors on there as well. And then another one of the authors is Thomas Viennman. So Thomas works with us as well. Thomas wrote Torch Script. Right. And so over the years we've had people who come for Meta like we. We had for a while. We had a handful of people from the Pytorch core team as well here working on optimizing things. And there's kind of revolving door to some extent, right. So some of us go there, some of us, they come back and forth and so on. And now serious organizations have lightning teams internally. So like Meta has a lightning team internally who works on their Pytorch Lightning equivalent internally because at some point they were using Pytorch lighting for all of their production stuff. And so all these things have derivatives. Like Nvidia has a lighting team. There are these companies where they'll have something internally. We collaborate with all of them today as well. So it is a big effort. And, you know, we support Pytorch beyond just Pytorch lighting. So we serve on the Pytorch foundation today. And then, you know, Luca, who I was mentioning, our cto, he's the, he's the chair of the Pytorch foundation this year as well. So we try to stay involved in the governance of how we shape the product and the project as well. And it's not just us, it's a handful of companies who are doing that today. But, you know, I believe Pytorch is a community project, as is Pytorch Lighting today. And we try to do that for, for everyone as well.

Um, yeah, okay, so that seems pretty fantastic that both of these projects have got good collaboration going and you're working closely with people who are working on vanilla Pytorch. Now, both Pytorch and Pytorch Lightning are open source projects. And it seems like for large language models, whether or not these should be open source is quite a divisive issue. So you've got the Metalama models that kind of semi open source, they've got open weights and a lot of the competitors are closed source. So we're yet to see a completely open model. Do you have a position on how open these large language models should be?

Yes, actually, before I answer that, I forgot to mention one thing this month, it was an interesting thing that we surpassed. So most people are familiar with keras on the TensorFlow side, right? So you've got TensorFlow and PyTorch that sit at kind of the same level. You've got Keras and Lightning who sit a little bit above that and they kind of automate a lot of stuff. Although, you know, I think Lightning is kind of built more for research and like high performance stuff. But Keras is one of the most popular projects in the world. Right. This was, this is the first month where Lightning was used more than Keras in the world, had more downloads. Right. And so that's been really interesting from our perspective because I think what it just, you know, it's hard to tell what their relationship is. But like Pytorch is growing, but is it growing? Because you know, we make it easier to use as well and then because they're growing it makes us easier to use. Like it's like a two way relationship as well. So it's just been really cool to see like that, that pair grow. So that was just an interesting tidbit that we saw this month. But in terms of open models, I think in the, in the asymptote it all becomes open, right? Like in the, in the limit it's all going to be open source at the end of the day. And whether we do it or China does it at some point we'll do it. Right. And the best models already I think are the open source models and like Deep Seq proved that. And you're going to see a few more models come out. I'm surprised, you know, we haven't, this is March 28th, but like we haven't heard from Meta yet and I'm sure very soon you'll hear something about it. I have no insider knowledge, I'm just guessing. Right. And, and I'm sure the model will change things as well for the industry. But no, I think that models should be open source. The thing that's interesting about the models is not the model itself, but what you do with the models. Right? So you don't regulate technology, you regulate applications of technology. That's kind of where the government really should be focused on. What's a good example of that? You don't regulate cars, I guess, or maybe weapons is a good one. You wouldn't regulate like the ability to build a weapon or use a weapon. You would regulate when and how you can use it and who can use it, for example.

Right.

And same thing for A car. Like you wouldn't regulate the building of a car. You would just say, hey, you can't use a car for certain things that are like illegal, like running into stuff or whatever. Right. And so, and so it's really, you know, time has already said that that's kind of how you manage technology. Ultimately it just feels scary, but I don't actually think it's, I don't know, it's that scary. Okay. Yeah.

So in this case you would want to regulate what people use the AI for, not necessarily putting regulations on what the model contains or the capabilities of the model. Is that about right?

Yeah. Like make it illegal to generate deep fake images, for example. Right. Make it illegal to, to use AI for whatever, like I don't know, for trading, smuggling, like those kind of things.

Right.

But don't make it the actual use of AI itself illegal. Right. Okay.

And I guess the other part of it is that at the moment a lot of the open models, the data sets aren't publicly available, so it's not completely possible to reproduce these things. Do you think we'll see more openness in that area in the future?

Yeah, good question. I hope so. Maybe someone will, but it'll be like an activist type who does that. I, I don't think that they can open the data because it's definitely not obtained legally, probably. Or like the data that they've used is not. No one wants to get sued. Right. So that's, that's ultimately what it is. So no company will tell you what it is because it'll open it up to lawsuits and. But yeah, if you had the data, I think that would be a lot better to really understand what went into these things. But I don't think it's feasible. Like no one's incentivized to do it. Yeah, I don't think it's feasible.

Yeah, I don't think it's necessarily feasible, at least in the short term. So we've got some economic and legal challenges having true openness then.

But the question is, do we have to be so specific about this true openness? This isn't like a, I don't know, like a moral stance, like ah, every, like a purist stance, an open source. Right. Like everything must be open. Like look, the model weights are open, the architecture is open. Sometimes they give you the way that it was trained, but they don't tell you what, what data used potentially, but they tell you high level, like what is. Did we use books or whatever? Yeah, I don't know. You Know, that's pretty good, I have to say.

All right, so what we have, we get the benefits for researchers being able to share ideas, but not necessarily the commercial competition that everyone can just completely reproduce what you've done.

You know, it's funny because for the longest time there was this debate of like even before AI came out and like you were in data science, right? And people are like, what's the value? Is the model the value? Like people remember, I think data scientists back in like 2020 were like, why are all these deep learning people open sourcing their models? Like they spent $10 million, you know, open sourcing this model weight and this and that. Well, it's funny because now people are understanding it's not the model that matters, it's the data. And we've always been saying that forever. The mode is the data. Ultimately for an enterprise, it's not the model, it's the fact that you have unique, perfect data that no one else has in your industry. The way you train your employees, your customer activity, like all that stuff is the moat.

Okay, so in this case the data is more important than the actual model in this case. So in most cases you can tweak the model. You're going to get slightly better performance, but better data that's going to give you dramatically improved performance for your predictions or for whatever the AI is generating.

Yeah, I think the most humbling thing for people working in alums is them changing models Phi 3 and then you know, llama 2 and nothing really changes. And then they go change their data or their prompts and they're like, wow, this is way better. Like yes, the data is the main thing that matters. Interesting. Context is king is what they say, right?

Okay, so does that mean organizations need to be investing in their data capabilities before they start worrying too much about AI performance?

Yeah, I think that you put you, you grab something like a lightning, you put a V1 of a product out there very quickly in a few days, right? Deploy something, no code, whatever, it's out there, you get it working in your production system and you start measuring the success of that and you now treat it like an a B test going forward. Right? Okay, what if I change my data? If I fine tune it this way, this way, what's the effect of my customers? And you just keep iterating on that over and over.

Okay, so this seems incredibly important for a lot of companies to deal with. So you mentioned before about Chinese AI and of course at the start of the year we had deep seq released the R1 model and it caused a lot of controversy. So it seemed to be very cheap and very powerful. And it's quite a wide range of views about how this is going to impact the AI ecosystem. So we've had a month or two for this news to settle. So what's your takeaway like having had time to digest it.

So I think it's a, it's a great thing actually, in my opinion because you know, this is goes back to the question of open source. A lot of these foundation model companies want you to think that it is really hard to pre train a model, that it's millions of dollars and it's too expensive. It's not. I have people on the platform doing this for thousands of dollars literally. Sometimes people fine tune for 12 cents because it's that quick. Right, but why do they do that? Because the harder they make you feel like it is, the more that you're going to pay them to just use their, their black box, non open source thing. Right. And so I think it's great that a research lab from China put this out and was like, hey, this could be done cheaper. Now is that a real claim or not? I don't know. They may have used a thousand GPUs behind the scenes, who knows. But from a researcher perspective, I can tell you that the techniques that they use are real and they do like they can give you the improvements and performance. And I think probably worst of all is those techniques have been around for years. They're not new techniques. Right. They've been around for a while and they just put them together in like a, in a good way that kind of yielded their results. So I've always said that my prediction last year for the new year was a 1 billion parameter model will do just as well as a 70 billion parameter model. And you're seeing a lot of that stuff start to happen now into smaller models. And then ultimately what I was kind of saying with that as well is saying smarter science will beat engineering. And at Facebook AI, like a lot of what we did, and I spent Most of my PhD on was smarter science on how to train a model, like foundational methods to training a model. So like what do I mean by that? A model can take a year to train if you're calculating like an MSC loss for example, which you wouldn't do, right. But like let's just make that up, then you can move into like a cross entropy loss and the computational time of that might be slower, right? And then there's all these techniques like Negative sampling, right. NCE loss, for example, where it like takes positive pairs and negative pairs and the way does that is it computes an N by N multiply and it gets you all these scores and then you sum them up and you normalize and stuff. And that multiplication can be pretty hard. And so what people started doing was distributed training to do that, where the negative samples go on one gpu, positive on the other, so that competition gets faster. But making that thing better, like the quality of the model better came down to what math function are you using? What is the loss? Like, how did you structure the loss? How do you regularize it, how do you sample your data? That's the science element, the engineering of making it better. It's just like, how can you scale out to more GPUs in a smarter way, or this or that? That's like being dumb about it, right? Like anyone can do that. But how can you come up with better science to do that? That's I think, where we're going to see the biggest leaps. And so to me, it's a good step because it's exactly what should happen. And someone else will do it. Someone else will come up with better science. They're going to pre train a model in like seven days or something crazy. And it'll be like, wow, okay, that's crazy. We don't know we could do that. It's like, yes, because you had the wrong loss function or you didn't regularize, or there was something smarter you could have done. Right. It is not the first time this has happened. I mean, remember Alexnet kicked off deep learning and it was literally a single PhD students who just happened to grab this thing, put on a GPU and like blew away 80 years of computer vision research. Very, very simple. It was done once already. I think the second time was OpenAI itself. Like they took something that already existed. Transformers. They didn't invent Transformers. That was already there. They took that and then they just scaled it out to more GPUs. And guess what? This new you get, you get this crazy model that works really well. That's chatgpt. It happened twice. Like, is it going to happen again? Yeah, 100%. Could it be one single person? Yeah, it's usually been one single person.

That's very opportunistic. And I like that it's possible to get a major scientific breakthrough and it just improves the state of what everyone's doing. So as long as everyone finds out about it and the research, it's clear what they did. And the research is open and shared, so yeah, it becomes available for everyone.

Yeah. And that's why the importance of open source is big, because I personally care less about the literal model. I care more about what did you learn, what worked for you, what didn't work. So we can try that out. Right.

Okay. I guess this brings us to another topical point, is that there's a lot of money being thrown into AI infrastructure, so data centers and things like that. So we've seen the first big IPO from one of their recent AI infrastructure companies with CoreWeave going public. So do you think there's a bubble in AI infrastructure?

Oh, 100%, no doubt. Yeah. I mean, look, GPU prices drop all the time. So on Lightning, you're able to get GPUs from different cloud providers. Right. So we kind of give you the lowest across a bunch of them. And you see this trend, like H100 is dropping consistently, so prices will continue to go down. I think that there is a delay effect from COVID So Covid made GPU production a little bit bottlenecked, and then AI came on top of that during, like the year after, and it just compounded the whole thing. And so you had this massive demand then Nvidia produced so many GPUs, probably overproduced GPUs, I would bet. That's my, like, unpopular opinion. And then you got all these people with all these GPUs all over and, like, most of them are sitting there idle. Like, most people can't use those GPUs today. Right. And so a lot of what we do is we partner with those cloud providers to give access to, you know, our whole community of users and everyone else so that they have someone who actually can use their GPUs. And that also benefits our users because users get the cheapest GPU prices at the same time. Right. So there's this massive imbalance in the market that we're seeing as a platform that kind of serves both of these sides. And I think black gold is going to exacerbate that more because now people are able to use these black gold chips. So what's going to happen to these older chips? So I do think there's a massive bubble happening there. And I think the financials of a core weaver are interesting. I'm not going to judge it. Maybe it works out, maybe it doesn't. But taking a lot of debt to buy things and then high interest rates, it compounds and it becomes really hard to deal with as well. We'll see, but there can't be a thousand cloud providers, let's just be clear about that. And there's a new one popping out every three weeks.

So what happens then? If GPUs get really cheap, what happens to everyone else then?

Well, I mean, I think it allows people to actually do their AI stuff easier. Right. Because a lot of it, for many years you couldn't even get the GPUs for people. Right. And so they couldn't even try to do the workflows. So actually I think it'll accelerate AI quite a bit in my opinion. If GPUs are cheaper, I think that the stock market will have a massive problem. Right. I think that Nvidia and all these other companies will have issues. Now Nvidia sells so many GPUs like it probably will not affect them that much, but I don't know, when process of things drop, it hurts some group of people and it benefits a lot of other people. I think it's a net plus benefit for everyone in the market and that'll just accelerate the ability of who can build AI. It'll democratize it. So I think it's overall a great thing for people.

Yeah. Okay, so maybe the infrastructure providers are not so happy, but for everyone else, low GPU prices are going to be good because you can just do more AI with them.

Exactly. And you know, from our perspective, like we're, we're kind of sitting here giving customers access to this. So like my goal is to give people the cheapest GPUs possible. Right. And give them the infrastructure and the tooling that they can use to build stuff. And so, you know, I guess cheaper prices are great for us. Nice.

So I guess related to this, I think the two big trends recently have been about reasoning models and AI agents. And both of these, the consume a lot of tokens and so they get quite expensive and sometimes in unpredictable ways. So do you have any advice for organizations who want to make use of these technologies?

Yeah, stop paying for tokens. It's not a good business model for you. Right. So you can exchange tokens for compute ultimately. Right. You can, you can either pay by the token and not run the compute yourself, but then you're limited by the throttling, arbitrary throttling that they have and the pricing that they set up. Or you can go to like, I'll say, like Lightning AI, for example, you can grab R1, deploy yourself in about five minutes. You'll have your own private dedicated endpoint and you're not paying anyone by the tokens there, you're paying for the compute only.

Right.

And we do a lot of work to make sure they fit in cheaper machines, H1 hundreds, et cetera. But yeah, and then when you're not using it, they go to sleep and there's no cost for the infrastructure. Right. So I think if you're paying this token tax, it's kind of self inflicted at this point.

Okay, so your recommendation is just to build things yourself, have it hosted in a way you control and then you control costs a bit more. Is that the idea?

Yeah. Why wouldn't you like why would you want to pay buy the token to a black box that you can't control, you can't control their uptime, you're sending your data to someone else, like on something like Lightning, you can get your own private dedicated endpoint on your own private cloud with all fully secure. And you're not paying by the tokens, you're just paying by the compute that it's being used under the hood and it's been optimized like to be pretty good. You know, the other thing is like that model won't change if you're hitting an API. They might change the model under the hood and then your prompt won't work anymore. Right. And so you had built this whole system on top of this, relying like a deterministic output that you just won't get anymore. Here you have guarantee that like your model is not going to change.

Okay, yeah. Certainly if your model changes and then you're getting a completely different output for your use case, then that's going to be a very difficult maintenance problem. Okay, so just to give me some more motivation, can you talk about some of the cooler things that people have built on top of your platform?

What have people built? So we have a ton of people developing drugs on the platform. So a lot of science companies, people sequencing DNA, people trying to. We have a lot of people like testing games for example, so using reinforcement, learning to test what could happen in a game, for example. What else do we have? I mean I'm finding the most interesting. There's a ton of LLMs. People are like fine tuning, training, deploying LLMs, R1s, hosting your models, that kind of stuff. But I personally think the most exciting stuff is things in the sciences. Right. There's recommendation engines as well. But a lot of big companies use our products today. So like there was kind of the last thing that we did was we worked with LinkedIn. So they pre trained a hundred billion parameter LLM and so we helped them do a bit of that as well. And that was all done using Pytorch lighting as well. And. Yeah, I don't know. I mean, for me the sciences are the most interesting stuff. We work with a lot of foundation model companies as well. Nowadays robotics is a big thing that we work on. And so they're like simulating all these things, they're training these models. I mean, it's pretty wild. I think the future is going to be crazy in the next few years. I don't think we're crazy yet. I think it's like just the beginning.

Nice. When you said that people are developing new drugs on the platform, is this some kind of sketchy dark web thing or. Well, I'm presuming it's pharmaceuticals.

Yeah, yeah, Pharma, like they're like submitting, you know, they're trying to find proteins to synthesize, submit to clinical trials, that kind of thing.

Right, okay.

All right.

So legit stuff. Now it's interesting you said that you scale to using large language models for LinkedIn's AI applications. So I presume the scaling is, is pretty much taken care of.

Yeah. And we routinely deploy 600 billion parameter models. Like you can find full R1 right now on the platform without quantization. 670 billion parameters you can deploy like that. It'll do it. Multi node inference, it's scalable. It works like, you know, people use us for distributed training ultimately. And like I think we're, our team has so much experience distributing models across the world. So it is, it is our bread and butter, I would say, today.

Okay, nice just to wrap up. What are you most excited about in the world of AI?

Yeah. What is going to have. I think, I don't know, robotics, I guess. I think that's really interesting. I don't think it's going to replace workers. I think it's really more around automating a lot of this factory stuff that people are doing actually, I mean, I'll say it again, but I think the ability to find better drugs that like can actually cure things like cancer and so on very quickly because you have to, you have to think about how this works. Like if you want a cancer vaccine, right. I'm not a. I'm gonna butcher this probably. So sorry if you guys are like at one of these companies. But if you wanna have a cancer vaccine, for example, like there's a very specific protein sequence that you need to generate. Like there, there is a combination of letters, right? That, that like will get you this particular protein and you know, back to like Biology 101. There's like four amino acids, I guess, right. And you mix them up, you have how you connect them, et cetera. So it's four, it's like what, like a hundred potential spots on that, on that drug. And there's 100 to the four because there's four options, right, of these things and probably even longer. How many possibilities is that? Like trillions. Right. And so, and so it gets pretty wild. So if you didn't have AI, you would have to kind of like loop through all of them and be like, okay, sequence one, sequence two, sequence three. And testing a sequence can take years because it's got to go through clinical trials. You have to synthesize the drugs. So imagine if you could reduce the time that it takes and find high probability sequences that have like a high probability of being an actual drug. Well, you could actually solve most diseases because those sequences do exist. We just need to find what they are. Right. So AI has a massive ability to speed that up, the drug discovery.

Absolutely. And we're certainly working on these big healthcare problems and these big social problems. It's just one of the great reasons of having AI. So we've got real problems that are currently very difficult to solve right now. And hopefully in the future we'll be able to solve them with some technology assistance. So yeah, it's a bright future. Now, final question. I always need follow recommendations. Whose work are you most excited about? Who should I follow?

I mean, you gotta follow my PhD advisors. So Kyung Hyun Cho, he was one of the original authors on this paper called Seek to Seek with Attention. So it was, if you guys know well, Attention, I guess. I think it was created in that paper or like another paper by Ilya. I think they were both kind of working on this thing at the same time. So they have very similar papers. Actually, Ilya Sutskever, who was the Chief Scientist at OpenAI and then that led to many years later and he was at Yasha Benjio's lab when he was working on this that eventually led to, you know, attention Transformers, attention is all you need, et cetera. So they're doing super interesting work and they also happen to be working on, on a lot of this like cancer stuff at Oppression Genentech, for example.

Right.

So they're interesting. And then Yann Nakun, you know, like he's been at the, at the head of a lot of this. But like Jan, it's funny because he's been doing the same thing. So like, even the things that I was doing a PhD with him. Like, I saw a paper recently where it was like, kind of that works. I had a continue my PhD. That would have been roughly what we were doing, but like, it was learning from videos. Right. And so I think that I would say a lot of people I met, like Facebook, AI, have a lot of really good ideas, and they probably have the tooling nowadays to start executing them. And they're probably one of the last few labs that are like, actually focused on advancing science and AI research. I can't speak too much for DeepMind or Google, but Facebook has really still pushed a lot of this. And like, you know, Llama is a good example of something that they push to really open source. So I don't know, I think they're still pioneering a lot and yeah, it's been good. Who else? Kevin Murphy at Google. He's great. Love his books. He does a lot of really good work. And I mean, I follow my researchers. I guess I don't really follow, like the influencer types, so I can't speak for them. No, no.

Research is definitely good to follow. So it sounds like I've got some reading to do. All right, wonderful. Thank you for your time.

We'll. Yeah, thank you for having me. This was fun.