Investing in MLOps // Leigh Marie Braswell and Davis Treybig // MLOps Coffee Sessions #81

All right, y'all, we got a special MLOps Community Coffee Session podcast for you today. I'm talking with Davis Trebig of Innovation Endeavors and Leigh Marie Braswell of Founders Fund. Both of these 2 incredible guests are specifically focused on data products. Davis was a PM at Google before he started working at Innovation Endeavors, and Leigh Marie was actually a PM at Scale AI before she went over to Founders Fund. So this was a cool conversation. I'm pretty excited about what and where we went. We talked about what kind of things they are looking for in the market, the bigger macro picture of MLOps, where they think consolidation is ripe for happening in the next year or so, different parts of the ML workflow and what tools could potentially be combined or what tools might go and eat up other tools. And then we talked even about open source, how they look at deals when there is an open source product and the community around it, the structure, and also the momentum that it has. A hot take from that was, for me, kind of eye-opening that you don't necessarily always have to to build an enterprise offering on top of an open source tool. So if you're an open source team that's just got an incredible amount of traction with your open source product, Davis mentioned that you can create an adjacent enterprise tool that is very, very complementary to the open source tool that you have this huge following around. I appreciated this conversation so much because it is a way for me to get out of the nitty-gritty of MLOps and also see their macro perspectives and look at what they are investing in, look at their vision for the future, and also how they go about doing deals. So that's it for me and this intro. So in case anyone would like to reach out to these 2 because you've got a hot new MLOps product that you want to showcase to them, You can find their information in the description below. For now, I will leave it at that. Enjoy the conversation. Welcome to Davis and Leigh Marie. This is awesome to have you both here. I think this is the first time that I've been on a podcast where I've actually met the 2 people that I am interviewing in person. And luckily enough—

Oh, wow.

Yeah.

This rare occasion for what we have been living in the last, what, 3 years? And so, we got to both meet up on separate meetings, but I got to meet both of you. And after meeting both of you, I thought it would be super cool to have you both on a podcast like this and just talking about what you're seeing in the market because both of you are Investing in companies that are going to be changing the landscape that we look at in the MLOps space. I know that you have your, you're probably up to your nose in seeing MLOps pitches and all of that good stuff. So I think it's good to start out with just like, where are we now in MLOps? What are we looking at? What have you been seeing? What are some things that you want to call out as trends or different things? And maybe Leigh Marie, can you start us off?

Absolutely. Yeah. So I guess to kind of clarify, when I'm talking about MLOps, it's the act of putting machine learning in production. And typically this is, or the term tends to apply when it's more complex. So when it's deep learning or something like that. And kind of the reality of the situation is, you know, not that many companies are using or urgently need deep learning. There are quite a few. There are now well-known use cases of deep learning where that is what you need. You can't substitute with something more simple. But kind of given how early it is on the adoption curve, this MLOps sort of infra startup market is extremely competitive. Extremely crowded, which is kind of an interesting dynamic. And I think that's because, you know, if you are an ML engineer and you're trying to put a model into production, it is very complicated. And there are currently no sort of consensus choices for multiple things that you need to do. And so tons of startups have popped up to try to fill that need. But yeah, right now, very crowded. I would say there's multiple categories where there's not sort of a de facto solution. Like, what's the de facto infra active learning tool or tool for collaborating with, you know, non-technical stakeholders? There's just no consensus and a lot of startups.

Yeah, it's definitely a crowded space. I think tons of point solutions. I think the core categories that are going to end up existing in this space still need to be figured out. Like you see, I think to build an MLOps stack today, it's 6, 7, 8 tools. I talk to many data scientists who are just overwhelmed. It's so hard to understand what to do and how to stitch these things together. And so I think you're going to see some consolidation. I think there's also some segments where terms are very ill-defined and people don't really even know what they're looking for. I think feature stores are a good example of that. There's others as well. And so it's interesting that we've seen such an explosion in this space, kind of this wave 1, wave 2 of startups, but probably In spite of that, there's a lot more to be figured out.

And speaking about like consolidation, where do you think is ripe for consolidation? Where do you think there are currently a few tools that are happening right now and could potentially get eaten by a bigger Databricks or DataRobot? And you see that happening or combining, a new tool comes out and it combines a few different pieces. And Davis, maybe you can go on this one and then we'll let Leigh Marie.

Yeah, a couple of things that immediately come to mind. One is that today, if you look at, I guess, like model orchestration and all the components associated with that, so tracking experiments, tracking models, et cetera, I see a lot of companies, they're stitching together, like they're doing a lot of the orchestration with Airflow. They're using something like MLflow or Weights and Biases, a couple of these other tools for basically tracking all the metadata associated with managing these pipelines. That to me feels like it's kind of all the same thing, right? Which is like I'm running these trainings, running these hyperparameter tunings, orchestrating all these models. And you kind of want, I think, all of that packaged together. So like a newer startup or newer project that's kind of going more in that direction is something like Metaflow, where you're starting to get a lot more of that bundled together. So I think that's one area where you'll probably see consolidation. I've wondered for a while if you'll see more of it on the data prep side. Today, labeling is very distinct from tools like Aquarium Learning or similar, which are more about, you know, understanding skew in my data, understanding where I need to augment the data, etc. I feel like data prep in general, especially it's also tied very closely to feature engineering. I'm wondering if that should be more consolidated or tightly linked, though there's reasons that some of it should be separate. And the last is on like more of the the monitoring kind of post-production, post-deployment side. One notable thing I find weird there is like the separation of some explainability tools versus monitoring tools. And more broadly, I'm seeing more monitoring tools move towards thinking about monitoring kind of like the end-to-end performance. So across training, testing, validation performance and understanding skew or drift across those and what that means for kind of iterating on the model. So those are 3 kind of immediate categories that come to my mind.

Yeah, definitely agree with those categories. I think just to add maybe a little bit of color to 2 of them. So the most recent one that you mentioned was explainability monitoring, which totally makes sense. I mean, you want to monitor, you know, and another part of this is also the data. So like, there's right now the split between like data quality monitoring and then ML monitoring, and there's probably a lot of overlap. Like when your model goes off the rails, it might be because of bad data. And right now you have tools that are kind of ML-specific or non-ML-specific, like a Monte Carlo versus like a, I don't know, a RISE. I wonder if there's like some potential for consolidation there. And then also on the monitoring side, you have tools that their sort of tagline is, detect when your model's misbehaving. Well, what's the next immediate thing that you wanna do? You wanna fix it, right? So I think there is a lot of work that could happen when it comes to, okay, the monitoring tools flag something, what do you need to do? Like, how do you need to retrain? Or, you know, like, can the model still be going in production? Like, what's the failsafe? You know, I think that's, that's sort of a category that, that's ripe for, for consolidation. And, you know, startup like, you know, Gantry, I think has written a little bit about this. And then of course, I'd be remiss not to mention on the sort of data prep quality side, I spent 4 years as an engineer and product manager at Scale. And I think Scale is already starting to see this area as ripe for consolidation. So Scale started out labeling training data for autonomous vehicle companies. They quickly realized that the same infrastructure that you can use to label training data for AV companies, you can used to label training data for all types of use cases like NLP or, you know, 3D, different types of 3D data because that same core quality infrastructure is the same. So they kind of consolidated that view and now you see them expanding to, I think recently they announced a big push into synthetic data. So data that mirrors the quality of real-world data to augment your training set in a cheaper way than just collecting a bunch more real-world data to find edge cases. And then also they have the Scale Nucleus product, which is somewhat of an Aquarium competitor where you can see all of your data and see if your dataset is balanced and figure out what data you need to label and send to Scale.

Leigh Marie, one thing you mentioned that I'd be curious your broader take on is like you mentioned the dichotomy between more classic like analytical data infrastructure tools, which is all these quality tools, there's transformation tools, And that stack is so different than kind of ML tooling stack today. I'd be curious more broadly your take on besides quality, which is one area that makes sense to me. Do you see other areas where it would make more sense for some consolidation versus the stacks being so bifurcated?

Yeah, it's interesting. I mean, I think when it comes to data pipelines, you'll potentially see some consolidation as more companies Realized, let's— I don't only just need the bread and butter ETL workflow orchestrator. I also want now to deploy multiple machine learning models in production. And so I do think that there's a bit more, at least brainstorming around the idea of, yeah, how do we make a workflow orchestrator for, you know, ML and data? I think that's one area. I think there's also kind of an area of analytics now where, you know, I guess the best way to describe it would be Like, for an ML model to get made, you want to have a lot of input from not only like the ML engineers, but also, you know, probably the data scientists who have built models before, the, you know, analysts who have to like deal with the results of these models or interpret the results of these models, or non-technical users that have to, that might have, you know, either deep learning or regular models as an integral part of their workflow. I think you're starting to see a rise of like lots of analytical tools that realize the need to accommodate stakeholders of various technical complexity. So like, you know, Hex Data or like, what's it called? Oh my God, I'm totally blanking. The ML—

Streamlit?

Oh my God. Yeah, yeah, yeah, exactly. The Streamlit companies like that.

Yeah.

That accommodate, you know, easy sharing of data apps, whether they're ML or not.

Yeah. And I've heard that called, which I liked, I stole this from somebody that I can't remember now, but like the last mile of machine learning or even MLOps where you're taking the model and then you want to get some, you want to get that value out of it. And it's that last bit from where you have it in production making predictions to actually like seeing, What is it? Is it helping us? Is what are— what kind of value are we actually getting from this machine learning? And so, I completely understand that. Now, as you both look at deals and Leigh Marie, like when you are analyzing something to invest in, what kind of pain does it need to go through? What kind of pain does it need to solve? And how do you recognize if that's a real pain or it's just like a hammer looking for a nail?

Yeah, I mean, I think this is a great question. And I kind of want to zoom out a bit. I think it's really easy, especially for somebody with like Davis or my background or your background to just kind of be like, oh, there are all these holes in the MLOps landscape and I want to find a company in each one of these buckets. I even wrote a blog post that, that kind of, if you skimmed it, that would be like the conclusion. But I mean, and this kind of echoes Founders Fund's really only thesis that we have. We are looking for extremely strong teams that are building a truly defensible product. And I think that means that whatever they're building, their strategy needs to have a moat in some way, whether that's like an incredible tech insight or some sort of accumulating data advantage. That's ultimately what I'm going to be in the back of my mind trying to figure out when I hear any MLOps pitch. I think, you know, if someone's pitching and it's a pain point that I experienced as an ML engineer, it automatically, like the credibility of that pitch goes up, but still there needs to be a real business plan there and a real sort of strategy. And that's what I, Sometimes, especially now, we see this common paradigm, which there's many examples of it working, of like just very technical teams, maybe spinning out of a company or building on top of an open source language. And I think, you know, you still need to pay attention to sort of the 101 of starting a business.

Excellent. Davis, how do you see it when you're trying to invest in companies and And the pains that they solve?

Yeah, I mean, I think there's all the basics that apply to any startup, so I won't go into those. But I think some of the unique facets of this market that I guess I think more about, one is like, how do you stand out? What's the wedge, especially in a market that's as crowded or noisy as this? I think, you know, there's a lot of companies that I think they're solving a real problem. It's a real pain point. It kind of is relevant to certain people, but it's not like building on a substantial enough like nuance or difference in the way things are being done or some market shift that it's not going to be able to get really huge. And I think honestly, for a lot of companies, you want to think about how to over time you become a platform, right? So, you know, as Leigh Marie mentioned, like Scale is kind of becoming a platform at this point where they start with a really specific labeling use case, but they were the first to kind of take this API-driven approach to labeling, which I think was like unique. Some users truly appreciated that. And that gave them this early foothold to expand, expand, expand. I think I typically want to see something like this, especially in this space, right? So nowadays, maybe that's okay. There's some interesting challenges around edge models, right? And if I can start by solving one or two of those, maybe eventually become the framework for deploying kind of end-to-end in the kind of edge or IoT devices or something like that, right? I think other versions of this could be targeting a new user type, right? So A lot of MLOps startups target more of this like Silicon Valley data science plus ML engineer startup. I actually think there's a ton of opportunity in targeting companies that mostly have people who know SQL or mostly have people who know, are just more like business analysts, right? And so is there like a vertical or a user segment where I can start with this one acute pain point but grow over time? I think I think about those facets far more in this market than a lot of other areas that I look at. given just sort of the noisy, crowded nature of it.

That's huge.

Yeah, I love the sort of, yeah, point solution to platform. Can you go across personas? I think another thing that I think about a lot in terms of the MLOps space, just kind of the knowledge that with ML, you have like a few people, autonomous vehicle companies, big tech companies, growth stage startups where ML is mission critical. that are spending a lot on these problems. And then you have like sort of this long tail, people getting started with machine learning, you know, start, you know, startups where maybe it's not as important, but, you know, it's helpful for their business that are maybe spending a little.

Right.

And so, you know, which sort of group is this company going after? Like I'd say Scaled was going after the former, maybe now they've expanded a bit, but. AV was the focus for a long time, autonomous vehicles. Weights and Biases, I would say, is going after the latter. You know, Weights and Biases, tracking experiments, anyone getting started with ML needs something like that, either buying them or building something internally. And so thinking about, yeah, how, and if a startup doesn't fit in either one of those buckets, if it's a tool for everybody or something like that, I'm a little bit more skeptical just given like the difference in sophistication. Tooling-wise that these 2 different groups need.

Such a good point. So we talked a little bit about, I feel like there's, there's a question to be had here that I'm trying to formulate as I speak, which generally doesn't work out so well, but I'm gonna try it anyway. And it's around the, the idea that You have the modern data stack, which got really popular a while back. And I've been talking with a lot of people about the modern data stack, or people ask about the modern ML stack. And my go-to default answer for that is there's no such thing, and nor will there ever be such thing. And the reason for that, I've been trying to write a blog for ages about it, but I don't want to just call out the problem and then basically shit all over it and not give any answers as to what you can do instead. And so the what you can do instead, I've gotten a little blocked on, but I promise that I will get a blog post out about this. And the reason that I say that there will never be a modern ML stack is because there are so many different use cases. And you have like computer vision that is for healthcare is much different than computer vision that is for autonomous vehicles. And they're still computer vision, but the SLAs of one are much different than the SLAs of other. And then you have something like structured data, or you have a fraud detection model and then REXIS, and those kind of go in the same bucket, but maybe not exactly. And there's specific things that you need for each. And then you have robotics, which is a whole nother ballgame altogether. And so can we even call that ML? I don't know. That probably should be called like something different, maybe like robotics. And so the idea there is because there's such a broad market and such a broad thing when we talk about ML and use cases for ML, how I really appreciate this point that you're bringing up. Like, yes, find a point solution and then expand out and try and eat larger and larger pieces. I don't necessarily see a company being able to expand out all the way to where it's like, oh, we need this one for sure if we're doing any kind of ML. Do you all see that happening? Maybe Leigh Marie, can you tell us, like, if you had to look ahead in the future 5 years, will there be one that will be useful for any kind of ML?

Yeah, I mean, I think it's an interesting Question. I mean, I guess in terms of let's just kind of look at the ML lifecycle. I agree with you. I don't think there will be like one tool to rule them all for every part of your ML training and deployment. You're going to go to this tool no matter like what sort of company you work at. I think potentially for data, there could be one of them. I mean, I just think you're already starting to see that. Obviously, you know, a bit biased, but you're already starting to see that with scale becoming sort of the de facto, I need training data. And, you know, I think they have, in all of the examples you mentioned, they have customers. So theoretically, you could see that becoming sort of a universal training platform. But then in terms of the complexity of like deployment for all these customers, it's just so different. Like AV, you literally have multiple stacks that you need, perception, planning, You have multiple types of sensor data. For robotics, you need like, yeah, fleet management and all of this other like, yeah, debugging and these huge tensor logs and all of that. And then for like, yeah, healthcare, there's a lot of, let's say, privacy concerns that maybe you wouldn't see in some of the other categories. So I, yeah, I just think the problem is way too complex. And then also with like just data right now, you know, it's not, Even the things that we think are kind of solved aren't even 100% solved, right? You see like Airbyte coming up and challenging Fivetran. And so like, you know, there's still, even though ETL is pretty standardized, there's still multiple companies in the space. And I think ultimately just reflects like, I mean, I think it's an exciting time to be an investor in the space because there's just so much data. People are now trying to make more and more sense of it. make smarter decisions and a lot of companies that can be huge generational companies can be built.

Yeah, I think I agree that it's maybe a little bit of a false dichotomy, right? Like what is the quote unquote modern data stack? I feel like maybe the simplest definition is just a set of data tools that are kind of warehouse native, cloud data warehouse native. Maybe you'd incorporate like you have some kind of core ELT type tool like dbt is part of that. But aside from that, I think you see actually a large heterogeneity of tools around that, right? Are you using notebooks? Are you using BI tools? You know, are you creating some kind of embedded analytics product? The way you build this periphery actually changes a lot based on companies, you know, and some companies have to have very niche or specific tools that are relevant only to one or two use cases. And I think it's actually very similar probably in the ML space, right? Where some of this upstream stuff around data feature engineering is fairly common across modalities of machine learning deployment, types of models, et cetera. But as you go later in the stack, it changes a lot. Even simple cases, I think, vary a lot as you go towards the end of the stack. There's some interesting companies in machine learning serving, which I think have interesting architectures that are really useful for very specific serving requirements, certain latency requirements, throughput requirements, more large deep learning models, et cetera. But there's a lot of serving cases where you can just get by with a Flask server or something like that. And so I actually don't know if the spaces are that different. I just think there's so much branding and marketing around the modern data stack that it feels more unified than I think it realistically is.

Yeah, my favorite thing to joke about with like data or ML founders is like, yeah, what's your marketing term going to be? Because you're going to have something catchy to describe like your little part of the data stack. So I always like to come up with a marketing term.

It's just ML Ops. I, you can just say whatever you want and then append store to it and you'll be good. So everything is a store these days. You've got a model store, you've got the feature store, you've got the whatever. What is it from the evaluation store? So yeah, that's the common one that people have. But yeah, I do like that. I think That's very true. It's marketing at its best. I've heard Joe Reese say that what happens when you're doing streaming data and you need Kafka on that modern data stack? Is that like the postmodern data stack? What are we dealing with here? And then like the next one's gonna be the Renaissance data stack that we have. And so, it is funny to think about that. Now, there was something else that I wanted to call out before we move to the next question. And that was, Leigh Marie, you talked about how you have some really strong use cases for machine learning that people are basically, they're all in on using machine learning and they know that no matter what, either the business is gonna go outta business or we're going to continue using machine learning, whether that's like the FAANG, the gigantic tech companies, right? the AV companies.

Yeah.

And then you have those tech startups that are using machine learning as part of their business model. And then you have all the rest and it feels like there's this huge sea of that long tail that is kind of like exploring but not necessarily all in. And Yeah. I just wonder like, what are all your thoughts on that sea, the long tail, like, And it's so, it's kind of what we've been talking about. It's so unorganized and so chaotic in those companies. Like, how do you even look at those and tools that can serve that market? Or do you try and stay out of it?

Yeah, I mean, that's, that's, it's a great question. And I think it's something I'm still trying to figure out because it's all about, I think, growth rate, you know, like how fast are these companies going to graduate into spending more and more money, you know, on whatever tools that they're using. Maybe they're really small today, but a few of them will be huge in the future. And this is just like a common infra thesis in general, right? Like, you know, why cloud providers are successful, why, you know, like these observability tools like, you know, Datadog are successful or whatever. You have these few customers that end up like graduating and staying with your platform. So it's like, Yeah, when is that going to happen? Honestly, I'm still trying to figure out the sort of timeline on my end. I mean, I think you're starting to see like tools that certain types of like ML engineers just kind of default to, but then you're just trying to figure out, okay, well, how much are they actually willing to spend to continue using these tools? You know, one that comes to mind is like Hugging Face. So Hugging Face, you ask any NLP practitioner, you know, where do you go to find the latest models? Where do you go to, you know, maybe even like play around in a sandbox, deploy them very simply? They'd say Hugging Face.

Mm-hmm.

But yeah, how much, like how much are these NLP researchers willing to pay to do that? And like, will they continue to use that platform as they like, you know, finish up their PhD and go start a company or go to a company or their startup gets bigger or whatever. So I think, yeah, it's something that I think every investor is thinking about. But I mean, I think as we've seen time and time again with open source companies, you know, like if, for example, like a GitHub or something like that, lots of usage, usually you can figure out a way to monetize. Or to be, or to create some sort of like uncopiable advantage that's extremely valuable to other companies.

I saw a funny meme that was talking about exactly what you were saying with like Hugging Face. And I can't remember the exact picture, but it was something along the lines of, now the time has come where you have to choose. And it's like, Do you pay for Hugging Face?

Yeah, well, I mean, people love it. So, you know, you should just see the responses to that meme.

Yeah, yeah, totally.

The point around paying for Hugging Face tied to, I think part of the context of the question was like this long tail of the ML market. One of the things I actually find interesting about that is I think the more you target the long tail, the more you go up abstraction levels. So it's going to be less like, open source, bottoms up, really sophisticated ML engineers adopting tools naturally, I actually think it becomes a little bit easier to have a good business model in those cases, or you have less of these concerns around build versus buy because the build is actually just actually impossible for customers like that. And so at least on my end, I've been actually really interested in platforms that target more this long tail of less sophisticated companies, whereas nonetheless there's a lot of just obvious simple use cases. It's not complex models, but like Everyone in retail has inventory management problems, revenue forecasting problems, things like this. In supply chain, they all have a lot of the same issues. And so one company I find pretty interesting in this space is Continual, which is kind of a SQL-first declarative ML platform. It just sits on your data warehouse. But anyone, if you just know SQL, you can basically deploy a production model. So it's kind of also, you could think of it as like a next-gen data robot that's more production-oriented and fits more of the DBT-style model of working. I actually think there's a huge market for tools like that. Similarly, I think there's a pretty big market for— you have all this growth of these mega transformer-based language models or pre-trained models or systems like this nowadays, where with just a little bit of fine-tuning, it can actually mostly solve the use case for a lot of forms of data and images, text, et cetera. I think that's going to get to the point where you can offer really nice abstractions to less sophisticated teams that allow them to do really interesting use cases on top of data types like that. And so I feel like we're starting to get to the point where maybe there's interesting companies to be built that really focus primarily actually on the long tail of the market, but there's challenges too.

For sure. Like, I mean, I think my, I'm sort of, yeah, very optimistic about those types of companies as well. Well, I think then the question just becomes like, yeah, you have to build a product that's really easy for these people to use, which looks very different than your typical like ML infra product. And then you need to have like a real sales motion. You know, you're not like having people go on GitHub and, you know, adopting the open source version and then, you know, graduating to the enterprise paid version. You're like targeting a very different type of customer.

Yeah.

Or maybe types of customer. So then it almost becomes like, you know, and do these customers even really need ML or do they just need like better analytics, right? Like—

Yeah, yeah.

There's always that sort of like, sort of complexity, like simplicity trade-off and what's kind of the optimal ROI for these different like personas and, you know, I think there's a lot of unanswered questions in my mind. I'm confident that really great founders can figure it out and they can pivot and they can create the sort of product and sales motion that works. But yeah, I definitely don't have all the answers there.

That's huge. That's huge. Speaking about open source and how you look at open source tools when, and Davis, I remember you said this earlier, Like there has to be that wedge. There has to be some way to garner attention. When you're looking at open source tools that don't have a business model figured out yet, like a Hugging Face, well, I mean, they're a little bit further along, but I imagine the ones who have just spun out of the gigantic company, name whatever one you want, but how do you look at that as an investment? And you're kind of sitting there going like, okay, we can get a lot of users, the community's strong here, but will that translate into dollars?

Yeah, it's an interesting question. I mean, open source undoubtedly has a ton of benefits in this space, especially if you're targeting more technical user personas. It lets them try things on their own. You can typically create more of a community around open source projects, which matters a lot in a space where it's important to stand out. Also, for products that are more infrastructure-oriented, you know, it solves this lock-in objection, especially if you're still an early-stage startup. You know, even if you go under, they can still keep running it, and that makes it a lot easier to get your foot in the door in the early days. So it makes sense as a model for many companies in this space. I think what I personally look for is it's okay if they don't plan to make revenue or monetize for a while, But I want to see some kind of strategy or thinking about what that will eventually be, right? Because the problem that can emerge is if you just build open source and say, I'm not going to think about monetization for a while, you may eventually realize, oh God, I really put a little bit too much in the open source, but I can't pull it back now because my community will revolt and then I'm dead. But then you're kind of stuck with nowhere to go. And so it's really important early on to, I think, be strategic about What do I want to be kind of free and open? And what do I want to be in an enterprise or paid tier, even if I'm not going to build the enterprise or paid tier for 3, 4, 5 years? And what I have a harder time with is founders who are like, oh yeah, we're just building this and we'll figure that out later. Sometimes it works. Sometimes it's more obvious what monetization can be, but it's not always clear.

Yeah, totally agree with that. And then also just, yeah, I don't really like the mindset of like, Oh, we'll just completely figure it out later. Like, typically having at least some ideas or like some sort of a strategy, just like being able to, obviously, like the first strategy is not going to work or like you're gonna have to pivot. Like every great company has so many crazy pivots. But yeah, just having like an if-then sort of going down the rabbit hole of all the ways that this could play out and the game theory of it. Yeah.

One of the areas where this gets interesting is I do think, you know, in the MLOps space in particular, you see a lot of these cases where it started with a team who was building at a large tech company where probably when they first built the open source, you know, they weren't thinking about starting a company around it. Or sometimes you see these really long-running open source projects which eventually get monetized, like maybe Dask is an example of that, right? Where there's so much inertia and community behind the project, but it wasn't tactical, you know, you know, tactically curated around the idea of startup formation. I think in those cases, it's interesting because on one hand, it's like, wow, this is a legendary founding team. You know, there's such strong community behind the project, but at the same time, they may not have that same clarity. And I think sometimes in those cases, it ends up making more sense actually to build an enterprise company around the open source that's more adjacent rather than open core. Like an analogy I would make is in like the kind of front-end web stack, like Vercel has done a good job of this on top of Next.js where it's, It's like not open core, it's just kind of a different product, but very complementary. So that can sometimes make too, depending on the, I think the legacy of the open source project in the space.

Wow. Yeah, I hadn't thought about that at all. And that's a great way of looking at it because I was stuck in that like, oh, well, if you have this open source product, you just have to figure out a way to do something on top of it, as opposed to doing something that is very complementary to it because you know that project inside and out and you know that there is a huge community around it that's got momentum. It has all of that. So now changing gears a little bit again, I want to talk about something that is my— so this is my theory and people who listen to the show probably have heard me say it a few times in the last couple of weeks, but it feels like there are a ton of startups happening right now, not just in the MLOps space, although there are a lot in the MLOps space. And because of that, because of all this money that is going into the market, it's become really difficult to hire people. And that is almost in a way like, because it's easier to go and found your own company than it is to go and like build a company for someone else. And so I think a lot of incentives are for great engineers to go and build their companies. themselves, how they want to make it. How do you all look at that? How do you coach different founders through that if they have to go through it? And are there tips and tricks that you've found for hiring, for creating a magnet that you want to get people to rally behind and where engineers want to go and work with a team and a vision?

I really love this question. I love that you asked this question. Because yeah, this is, I think, on the top of every founder's mind right now. Like, I've not spoken to a founder where this is not just like something that they're thinking about constantly. And I think also it should be on the top of every investor's mind when they're deciding whether or not to invest in a company. Yeah, like, how will this company be able to hire? Because it is such a huge question right now. You're even seeing, I think, a lot more acquisitions of companies at early stages to bootstrap hiring, which is like, you know, another symptom of this just sort of, you know, do I— It's a very competitive job market, a lot of macro stuff. But I think one thing that I see a lot that's helpful for founders hiring their first team members, honestly, like their personal networks and the investor networks of the very, like of the very early stages of the company. Just kind of, if they come out of a startup where there's a lot of strong tech talent, I think that that can really bootstrap hiring or a school with a lot of strong tech talent. It was funny at Scale, many of the first hires were MIT Poker Club because that's where we all met the CEO of Scale. But that's kind of like maybe To get the first few hires. And then I think after that, you wanna create some sort of unique community, or at least this is something that I've seen work really well. So whether it's having a podcast or doing a newsletter or like some sort of competition or hackathon or an in-person event, which, I mean, these days, especially in like San Francisco, can be rather unique. Or maybe even if you're bigger, throwing a conference or having a really active Twitter account, having a programming language that you kind of have a monopoly on, like Jane Street, which is a quant trading firm, essentially has somewhat of a monopoly on OCaml developers because that's what their stack is. And it's like such a unique language. So I think there's all these sorts of strategies that I think founders should be and great founders are thinking about constantly. Like, how do I curate this brand for the type of people that I want to attract. And this is quite frankly something that, you know, I think, yeah, I think investors can be particularly helpful for like thinking through the strategy around these different sorts of channels.

Yeah, I can, I can say that in MLOps in particular, relative to other categories, we specifically look for founders who can just be natural talent magnets, right? Like they are kind of some form of luminary. They're famous in the field in some way. Because while hiring overall is an issue for, I think, every single startup I work with, the pool of talent who know how to build machine learning systems and infrastructure is so small, so highly paid, you know, so easily drawn to the many companies that want to hire them that I think it's dramatically exacerbated in this market. So the first step, I think, is be a— if you're going to be a founder, you know, feel like you have a network you can pull from easily. And be a good storyteller so you can recruit people. I think in terms of tips, I don't know if I have too many more aside from what we already mentioned. Maybe one other thing would be, I think if you can create some like aspirational feeling around your startup that is tied to some concept that really gets some people excited or going, that can be another way to attract talent. So I'm just going to make something up, but Maybe you believe that the future of machine learning is all sort of like declarative systems, right? It's like this data-first, like, Andrew Ng model or something like that, right? If you're like the first startup to really brand around that, regardless of what you do, maybe you just find some people who it's like they believe in that concept, that idea, they want to be part of that movement. And so they want to come work for you. I think that type of emotional framing around your startup can be powerful if done right. But of course, You need to be smart about doing that in a unique, authentic way. If you just, you know, throw us some random term, it won't work well.

Which we've all seen.

Yeah, like democratizing ML maybe a few years ago would be like more unique. Democratizing anything these days, I mean, I just feel like it's what I see on every single company's website. And not saying it's a bad mission, but it's just, yeah, not something, as you said, that's very unique and willing to like, or going to probably stir up a very emotional response.

Yeah, it's not cutting edge anymore. That's so true.

Not anymore.

Yeah, it's interesting. I think this is a broader problem or challenge for infrastructure startups. You know, I know a lot of people who are very mission-driven, and, you know, the mission of a startup can really attract them to work there. You know, even though the comp is lower and other things, you know, the work is harder, etc. But a lot of people, you know, it's easy to associate that with like a climate startup or a more like application layer startup. It's harder to focus that if you're at this infrastructure layer. And so I think part of what you need to do as a founder is figure out how to tell that mission-driven story or that aspirational story in spite of the fact that you are a tool, you know, rather than solving some very specific use case. But I think if you do that well, you can help people realize that actually, you know, these tools still have impact, they have power, you know, they're important in the world, but it's a little bit harder of a sell. And so it's more important to think about how you craft and frame that story.

Amazing point. And I'm actually reading this book called Storyworthy right now, which I don't know if you all have read it. It's an awesome story or a book about how to tell better stories. And I figured it's probably good for me because I talk to people all day. And so I don't think anybody would ever regret learning how to tell better stories. And I think it's great what you said, Davis. Have something that can inspire someone and try to make it so it's a little bit outside of the box. Make sure that you have that ability to be a bit avant-garde when you're inspiring people. Now, let's close here with this one. What are some things that have surprised you over the last year? Whether or not that is something that is a. company that has blown up. Leigh Marie, you mentioned, I think, Airbyte before and how we're still looking at Airbyte. We thought it was Fivetran that had dominated that market, and now Airbyte is all of a sudden a competitor. Or maybe it's a company that you thought was going to do great and they've tanked. What were some surprises in your eyes? And Davis, maybe you can go first on this one.

You know, it's an interesting question. I think the first thing is maybe just the raw volume of startups I continue to see crop up in this space continues to exceed my expectations and be an interesting thing to think about as an investor. Maybe one thing I'll mention is I've gotten the impression that a couple of these companies that are focused on more unique deployment modalities in the machine learning space, like robotics, edge deployments, etc., seem to have gotten a lot more rapid interest or traction than I maybe would have expected a priori. I think, you know, I talked to so many companies who can barely get the most basic model deployed in the cloud. And so from an investor's lens, I've been really focusing more on like the basics before you go to more exotic situations where you need to do, you know, quantization and figure out how to optimize for the hardware target and all these things. But it seems like some of these companies are doing very well. And so maybe that was a little bit of a surprise to me.

Yeah, definitely a surprise to me. You know, I think I said earlier, there's like these 2 buckets that I put ML startups in. Either you're serving like these very few use cases that are willing to pay a lot, or, you know, lots of different use cases or personas, and they're not, you know, willing to pay some, but not nearly the amount in the first bucket. Like, I have to confess earlier, I really just kind of thought it was very hard, if not impossible, to build an ML company. that like went after the second bucket, the long tail. And now I've seen multiple examples of companies that have been quite successful doing that. So, you know, like Weights and Biases, Hugging Face. I think these companies have really, you know, broken out, seen real traction. And so that's something that over the past few years has surprised me. Now there's a ton of startups trying to go after it. So I don't know like how much room there is or, you know, like how fast growing to accommodate all these startups, but who knows. And then one thing that surprised me in the negative sense is, I guess, work, you know, working at scale or like, you know, just kind of being in college. I thought by now we'd just see so many examples of like robotics just out in the world, like, you know, at restaurants and things like that. And it still feels very like there are definitely some and, you know, you can call a self-driving cruise car an SF now. if you're on like a very limited list. So there's definitely like some proof points that things are happening, but it's unfolding a bit slower than I thought it would be. Like, you know, I think though robotics infra companies are very promising, you know, how, like, when is that market gonna fully be there? And so that's something that has surprised me.

I wonder if that's a lot about society and like the societal norms that we have and our acceptance of those things as much as it is about the companies themselves being able to prove out value?

Yeah, I think it's a combination. Like, I mean, I still do think it is like pretty expensive if you just wanted to, for example, I'm a warehouse and I want to replace all of my, or, you know, augment all of my processes with robots. I do think it is like a large upfront hardware cost and You know, like I'm not 100% sure that the ROI is currently there. Yeah. But yeah, you know, maybe I'm wrong.

Well, this has been great. This has been so cool. If anyone wants to reach out to you all and talk to you about their new MLOps startups, can they?

Yeah, definitely.

Oh my God. That's it.

We'll leave. Yeah.

Okay.

What's the best way to talk to you? I know you're both on the community Slack, But you're also on LinkedIn and Twitter and all that good stuff. So, we'll leave all the details.

For me, the best is just davis@innovationendeavors.com. Happy to chat with anyone exploring the space.

There we go. Cool. Well, it's been a pleasure. I really appreciate both of you coming on here and showing me how you look at things. I'm super excited about this space, obviously, and I love hearing from investors on how they are gauging it, what the temperature looks like from their point of view, and So thanks again. This has been awesome.

Yeah, thanks, Demetrios.

Thanks for having us.