Investing in ML startups, building ML products, startup opportunities in ML | Leigh Marie Braswell

Hello and welcome to Infinite Machine Learning. This is your host, Prateek Joshi. In this podcast, we talk to amazing machine learning practitioners and dive deep into a specific topic. We have Leigh Marie Braswell on the show today. She's an investor at Founders Fund, one of the most successful VC firms in the world. Prior to that, she was one of the first 10 team members at Scale AI, a startup which is currently valued at more than $7 billion. Before that, she was a machine learning engineer at Google. She has a degree from MIT and loves to play poker. In this episode, we'll talk about key learnings during her time at Scale AI, how she identifies promising startups, startup opportunities in the machine learning infrastructure, large language models, applications of machine learning, and product management at AI startups. This is going to be a fantastic discussion. So, Ready for liftoff? And 3, and 2, and 1, let's go! Leigh Marie, thank you so much for joining me today.

Thanks for having me.

Let's dive straight in. You were one of the first 10 team members at SCaLE. You know, it has seen blockbuster growth. Can you share your top 2 to 3 key learnings during this time, especially experiencing hypergrowth, building V1, and then just, just kind of working through that process?

For sure. Well, I definitely learned a ton while I was at Scale. So yeah, joined, joined as one of the first engineers, and then about a year later, sort of transitioned more into a product management role. I was mainly working on our LiDAR annotation product, which is used by self-driving car companies, robotics companies. AR/VR companies, et cetera. Yeah, I guess if I had to distill sort of the key learnings down into a few, I think one, my time at Scale really showed me just kind of the power of moving fast. Like, even if you are a small team, if you're rapidly iterating, you're learning from customer feedback, you're building something that people actually want and will pay for, and that's de-risked very early on, that's extremely powerful. So I think I didn't appreciate fully before I joined Scale, just kind of like the power of small teams and the advantages that startups inherently have being more scrappy. Secondly, and I think this became more clear maybe a few years in, just the importance of having a moat as a startup. So, I think one thing that a lot of people didn't see when they first looked at Scale was that Scale was very different from all the other labeling companies out there at the time. And I could argue still is. There was a big focus at Scale on sort of owning the end-to-end experience. So not only sort of the annotation software, but also taking control of the workforce, the quality, essentially like abstracting all of that beneath an API versus like opening up parts of it and having customers bring their own labelers to the platform or have to manage the operations and SLAs themselves. And so that was a big differentiator earlier on. And of course, it made scale much harder to build. Like insanely challenging. The labeling interface, you know, that's software and it's somewhat challenging, especially if you're loading, you know, tons and tons of LiDAR points in the browser. But it's nothing compared to making sure that tens of thousands of people are annotating things correctly with no bias in the wrong direction that will then bias the ML models. So yeah, automatically sorting out the quality control system became Scale's moat very early on. And, you know, I think you could argue they have a few moats, but that one in particular. Yeah, I guess those are probably the top 2 learnings. Of course, I learned a lot also about being a product manager. But yeah, I can dive into that.

No, that's amazing. And what you said about moats, it's such an important point. So I want to dive into that. You've been an angel investor, you're now at Founders Fund investing in some amazing companies. So When it comes to the 0 to 1 phase of building a company, right? So obviously, you experienced that at scale. What best practices do you recommend founders? Like when you work with founders, obviously, they make some— they make mistakes, there are some best practices. So what's your framework on guiding founders who have an idea, no product yet, they're Trying to figure out who are my first few customers. So yeah, I would love your perspective on that.

Definitely different on kind of a case-by-case basis. And I mean, ultimately, Founders Fund, we're here to find founders that can go through the idea maze, you know, of course, with our help. But that idea maze looks kind of different for different ideas and different industries. I mean, I think some— maybe since this audience is more skewed towards machine learning problems, I do think one thing that, you know, I think it's easy to kind of get caught up in how cool the tech is in the areas that we look at, like, especially in ML infra and ML applications. It's just, you're literally working on sort of the future, and the tech is so cool. I think it's easy to get caught up in sort of like theoretical land where it's like, in theory, we need all these different things, but then in practice, ultimately, you gotta create a business that people will want to pay for, like they won't want to pay for the product. And so I think that's something that in general, and this is something that I learned at scale, just getting something in front of customers as soon as possible that they can actually make paying decisions on, I think it sort of validates, that's the best way to validate your hypothesis versus just like building something alone, locked in a room for a really long time that you think people want. I think it's better to get out. a less than perfect version in front of somebody ASAP. And that's the only— and that's one of the, you know, reasons that we chose to double down on LiDAR annotation. We've coded a beta really quickly, got in front of somebody, and then they, you know, ended up wanting to pay a lot for it. And we were very surprised. And so then that was just kind of like a moment that we had early on that was very important in sort of the journey of the company.

Right. I think that's a great way of looking at it. And now, as you come across investment opportunities, right? So you see many startups come to you, they pitch you. What gets you excited about a company, right, when they come in, and especially on the early stage? But by the time it's Series A or B, yeah, I mean, there is this proof. But when it's like super early, what gets you excited about a company?

Absolutely. So definitely, I think the thing that gets me the most excited is when it's clear that this is like the founder's life's work. Like, this is what the founder was made to do. They have to do it. You can hear it in their voice. You can hear it in how they talk about the plans of the company. They're just so excited to work on this company. And they have a plan A, they have a plan B, a plan C, a plan D. They're able to kind of go down whatever rabbit hole you poke and prod, and they aren't defensive or anything like that. They're just genuinely curious and wanting feedback and asking you questions, and it's more of a conversation.

Yeah.

I think that's the first thing that gets me really excited, especially with an early-stage company. And it's even better if for some reason they have just some other unfair advantage or secret that no one else believes. They have some very contrarian opinion about how the future's going to play out and this is why they're doing this, or they have a set of connections or a set of knowledge that's very N of 1 or very proprietary. That's all extremely exciting as an early-stage investor.

Yeah.

And then, and then, you know, as we kind of talked about earlier, I think past that, it's like, can you articulate how this business has moats? Because, you know, ultimately, like, if you're building software, it's kind of all about getting a head start. Ultimately, if you're successful, there's going to be a lot of people that try to copy you. Like, very rarely, you know, if there's not competition in an area, there should be a reason. And so you should have kind of like a, Why hasn't this happened until now? And then also you should have a, how do you prevent this from becoming a commodity?

And I think the last point, I think it's actually really spot on. Like, how do you prevent this from becoming a commodity? And if it's democratized, then you just lose your edge, right? You wrote a great post on your Substack last year titled Startup Ops in ML/Infra, right? I actually frequently go back to it. It's just nice, it's clear, and I like the mapping.

Thank you.

Yeah, yeah, yeah, yeah. It's funny. I've been— I was like, I always thought if I do a podcast, I'm gonna invite her. It says I have to. It's a nice map. So when you look— and also, listeners, if you haven't read the Substack, go read it. It's amazing. And when you look at it, right, you did it last year. And there's this amazing map. How has your perspective changed since you wrote that? And also today, what trends are you seeing in the ML infra space. Within that, like, what— where is the future going? And for founders who kind of want to build an ML infra but they don't know what to do yet, so what themes are exciting to you?

For sure. Yeah, I guess my thinking hasn't changed too much since I wrote the post. I do think, you know, we're continuing to see how crowded all these sort of— they kind of divide up into different areas of opportunities. And I think You know, after I wrote the post, even last year, there were more and more companies that popped up in all of those areas. And then 2, I think we're observing like a change in the macro environment that maybe we didn't expect as soon as right now that it would happen so significantly, that it's very clear that, you know, we're on— I don't know, don't want to say anything too extreme, but it's clear that the next few years might be more challenging for startups, just kind of given like You know, an economic downturn of some sort. So I do think, especially in an environment like this, you have to, you know, ultimately, like, there will probably be some amount of consolidation in the space. There will be some shrinking in certain types of budgeting for, for especially for ML, if it's not in sort of a mission-critical use case at a company. So yeah, you have to just be really careful about, like, of course, some of these companies will be successful, but making sure that Yours is one of the ones that has a true sort of expansion story that will allow you to not only capture the small part of the ML stack where you start, but then also capture adjacent areas.

Mm-hmm.

So, you know, like, I do think there are plenty of opportunities to build, like, a tool that makes ML engineers much more productive. But you have to be very careful about what's the initial wedge. Like, why is that going to be the thing that your initial persona is going to choose? And then, How do you like expand past that to capture like budgets from other teams, for instance?

Right, that's important to understand. And your job as an investor to identify good opportunities, how do you put yourself in a position where you come across these great companies, right? So, is it your personal network, Slack communities? Like, what do you do to make sure that, you know, these, you come across these great companies before they're great?

That's the name of the game. That's the question. I think investors are always, and this is the area that people are always the most paranoid about, like, are we seeing everything? Are we seeing things as early as we can? I think in general, we feel confident about our diligence process. And then obviously we feel as Founders Fund, we have a lot to bring with a partnership, but it's always about like, can you just figure out all the different channels that you should be sourcing from? I mean, I'd say me personally, I mean, part of the reason I do these podcasts and write Substack and all of that is that, you know, ultimately, like, you want to be one of the first people that people think about, especially in the areas where you're most interested. And for me, those areas are dev tools, ML applications, and infra data, other types of infrastructure, even in crypto. So, you know, just making sure that my name is out there associated with those things. And I've had people come inbound to me, both founders and people looking for jobs, Because they've seen content or stuff that I've written. So that's definitely an important channel. There's a lot of cold outbound, a lot of just like, look at all the companies in the space, look at all the people that are thought leaders in the space, just try to talk to them, try to see, you know, like, are they starting a company or what's sort of their plan? And then, yeah, let's see, a lot of networks, a lot of going to events. A lot of, yeah, just trying to learn about all the different experts in a space and follow up with them.

Right. So for people who think the job is easy, this is a, this is the list, right? And that is coming from someone who's at a very, like the topmost tier of venture funds. So good luck if you think it's a cushy job. So thank Leigh Marie for sharing that.

It can be a cushy job if you don't try very hard, but yeah. You want to maximize the probability that you'll be successful.

It's weird.

I don't know any VCs at top firms that aren't working constantly. I really don't.

Right. And I thought your recent tweet about the poker game was funny. It was like, if you try to invite someone to poker in SF, that was hilarious. I was like, that's so spot on. And also, I think someone else, I think it's Matt Turk, he posted like trying to get a hold of investors in August. That's hilarious.

Hilarious. I thought I, I thought I cracked the code. Like, I thought I figured out like how to do these events with VCs because multiple times I've been having these poker nights quite a bit. And every time, if I invite 1.5 times the number of people that I want to attend, exactly a third of them will fall out like the day of basically. And so I'm like, I've got it under control. If I want an 8-person poker night, you know, I invite 12 people or whatever. My last poker night, everybody came and it was like super crowded. And so now I'm having to reevaluate what the ratio is. I thought I had it.

Yeah, that's hilarious. I think it's, yeah, you never know how that turns out. All right. Coming to the topic. So you wrote another post on large language models on your Substack. And I feel like over the last couple of years, LLMs They've become the mainstream option. It's almost like zeitgeist in AI. A lot of talk, a lot of research, a lot of applications too. And I think you listed some really good use cases, right? Just how do you bring it to practice? So when a founder approaches you with, hey, I built this amazing, cool application using a language model, how do you determine that they're onto something? Because it just looks like I can take—

Yeah.

one of the open-source LLMs and just do stuff, build a chatbot. But how do you do it? Because it's your recent research, I would love to dive in a little bit to understand that.

Yeah, I mean, I think why I'm excited about large language models generally is that they unlock this sort of potential 10x improvement in the technology for a certain use case than like without LLMs. So basically, like, that's one aspect that you need. I think in Zero to One, there's like 7 questions for product innovation. And one aspect is your tech needs to be 10x better than everybody else's. And so I think the introduction of LLMs, basically, that can be your opportunity to, for example, make a copywriting tool 10x better than previous copywriting tools in the past. But that's only like one of the things That you need, right? Like needs to be the right time for that that use case to take off. There needs to be like an opportunity for you to like get the best in class team together. Like you need to figure out distribution, blah blah blah blah blah. So there's just that's one part of it, and then obviously like there's the other part of like actually building a business. And so it's like yeah, is the particular market you're going after like going to be receptive to this tool? Like what else would you have to build to like take away market share from the incumbents? Like, how well does your tool actually work? Like, I think LLMs, there are certain use cases where it's fine when they make certain types of mistakes repeatedly. Like, the lightbulb moment that I had to even write the post in the first place was seeing all of my friends using GitHub's Copilot. And yes, there are problems with GitHub's Copilot, but it's useful enough for engineers to use it to complete their code. So it's like, where else can you find use cases where It is useful, the technology is useful enough, and you have like the right data to train the large language model. And, you know, you can also sort of like create moats to make sure that it doesn't become like, you know, a commodity. So, you know, I think it's one aspect. I'm really excited about the wave, but at the same time, you know, I have to like keep telling myself, don't get too excited by just the technology. There's so many other things that are required to like build a generational business.

Yeah. All right, let's shift to the product aspect of building machine learning companies. Now, historically, the self-serve part of machine learning companies has been notoriously hard, mostly because the data is not in a good shape. The users don't know how to bring it to the right format, use your product. And before you know it, it becomes an implementation-heavy, top-down product. But more recently, there are some really good companies who have figured it out. So what's your viewpoint on if somebody wants to build a self-serve machine learning company? Like, what guidelines or framework can they use to make sure they don't fall into the trap?

Yeah, I think that it's a really good question because I think, you know, now there have been many examples of companies that have built large businesses on the not self-serve, the like Target the whales, serve them, you know, like make— it's pretty challenging to get started, but then once you get started with the tool, then, you know, you're gonna use it for a really long time and pay a lot.

Right.

So that's kind of been de-risk, I guess, in ML. And then as you said, like, there's this other business model of get, you know, the long tail of people that are either just getting started with ML or aren't spending that much. And you need to create a self-serve experience because ultimately they're not gonna pay you that much. So, You can't spend a lot to onboard them onto the platform. Yeah, I mean, I think we've seen Weights and Biases do this really well. I think Scale is actually starting to do this really well with their new Rapid product, among other things. You can just start using it right out of the box. And so, yeah, it's like, how do you create a self-serve product? Yeah, I mean, I think depending on exactly what you want to do, you've got to play nice With all the different frameworks and types of data that people are gonna use. You've gotta create something that's user-friendly to whatever persona you're wanting to onboard. So, obviously, if it's a machine learning engineer, you know, you can assume some degree of, like, being technical. But if it's like, let's say I want— I'm trying to make an AutoML product that allows data analysts to, like, use deep learning to figure out churn or something like that. That's a different set of skills that you have to assume. And so, your onboarding experience should look pretty different. And then also with self-serve, you've got to kind of figure out pricing sooner rather than later. So, what's your cost to acquire users? And then how are you going to charge them to make that effective? Versus with the enterprise stuff, pricing can be a little bit more of a discussion, and it can be a little bit more bespoke. And they're big contracts, so you only have to make the decisions a few times, maybe. Self-serve, it's a different game. And so you have to really be disciplined about like, okay, this is how much it costs to get users and this is how much they're going to pay us so that you don't get into like a scenario where you're losing a lot of money.

Right, right. That's a great point. Now, when you look at the ML startups coming to you, let's think of 2 big buckets. One, a product that's built for ML practitioners. And in the second bucket, we have a product where ML is infused into another product, like a CRM tool. So between the two broad themes, what excites you more? Like, where do you find yourself leaning more towards?

I look at both. I don't know. I mean, I think they're both ML infra and then ML applications. I think they're both needed. One, I mean, infra to build the applications. I feel like I go back and forth like every few months. You know, I think right now maybe I'm looking more at applications, but a few months ago I was looking more at infra. It just, it really depends on, on what I see.

Right.

So yeah, with the LLM stuff, like very bullish on these applications right now and, you know, just invested in one and, you know, excited to talk about that when that comes out. But yeah. Yeah.

Application, right? So let's talk about maybe recently publicly announced that you can talk about. Talk us through how that application product kind of— why that excited you and what made you say, okay, this is a useful application. Because many times I see like, oh, AI-powered, yada, yada. And it's like, no, tell me what it does first. AI, you can use it to accelerate yourself, but it's like, what are you doing? So yeah. So can you talk us through A startup that's building an application that you funded that got you excited?

Absolutely. Yeah. So one that was announced relatively recently, it's called Arena AI. Essentially what they're doing, they're helping companies like large CPG companies, Fortune 500 companies, companies that are serving different types of consumers online. They're helping them with price optimization. So using deep learning, to essentially do dynamic pricing and figure out sort of the right way to price certain things so that you increase your revenue by a lot, basically. And yeah, this really smart team out of Palantir, they're based in New York. I think ultimately what convinced me that they were onto something is ultimately the sort of customer traction. I can't give exact numbers, but it's incredible. When you tell customers, like, hey, you know, this is how much money doing this with ML versus not doing this with ML made you, can we take 10% of that? And that turns out to be a huge number, it's pretty exciting. I think sort of an application where it's very clear to a customer, especially if they're not super ML sophisticated, that it doesn't care if it's ML or not, or what the algorithm is, or whatever. As long as it's like making them a lot of money, right? That's product-market fit if there's anything else. Yeah. Yeah. So, I mean, ultimately just like seeing, and I think this kind of goes back to the first point of like getting a product out in front of customers and seeing like how it behaves in the wild and if people really need it or not. Yeah, I think that's super important.

That's fantastic. And glad that an application is, you know, when it generates ROI, which is like money, customers tend to love it. So that's pretty clear. Yeah. When we look at machine learning products, especially in the early stages, actually before that, the pricing for SaaS products is well understood. Like Slack, you know what you're doing. It's like per-seat model. It's pretty easy. Slack, Zoom, it's easy to understand. But when it comes to machine learning products or data products, right, it gets more nuanced because there is consumption involved. There's compute power, there's performance guarantees. So how do you—

Yeah.

guide early-stage founders to think about pricing mechanics, right? Because if you're selling a data product, and you sell it like per seat, it just doesn't make that much sense. So how do you guide early-stage founders on pricing?

For sure. And this is definitely not an easy question to answer. And so, you know, like, I agree with you, it's like, you want to make sure that sort of your pricing function is like correlated to your cost. You don't want like some costs to like just completely get out of control, whether it's compute or You know, something else on your end, and the customer is not paying any of that. Yeah, that that that would be pretty catastrophic. Yeah, I mean, it's I guess it's somewhat hard to give super general advice, but I mean, one piece of piece of advice is always you can always charge less, but you're never going to be able to charge more. So you know, if there are key customer relationships. Having somebody use your product for free for a really long time, of course, this can happen and this has happened. I mean, I think PayPal is a good example of this or whatever, where people were using it for free and then they started having to pay and it worked. But I mean, it's still much harder to have that discussion of like, you've been using this for free or at this reduced rate for a really long time, now you need to pay. Of course, if it's sticky enough, it's sticky enough, but it's It's much easier to go the other way. And so, like, figuring out closer to, like, you know, like, an amount that's somewhat comparable with the value that they're getting out of the product, and that whether it's, like, dollars from an incumbent solution to you, or dollars from not hiring people to you, like, you're replacing, you know, like, you know, people that could potentially be, like, coding a version of this for them, from them. I think kind of figuring out the value that your product is delivering and trying to charge close to that earlier on is probably the best strategy if you can do it. And yeah, just kind of making sure that all these hidden ML costs, like, you have to annotate data or buy data or compute or all these other things that you might not have to think about as like a SaaS business, like just a regular SaaS business, or like in your sort of pricing function. I do think that's an important point.

Right. It's actually a great topic. And I feel like I could go on and on, but I think we are at the rapid-fire round. It's a wonderful discussion. I want to keep diving in, but we are at the rapid-fire round. So, I'd love to ask a series of questions and love to hear your answers.

Cool.

All right, let's dive in. Question number 1, What's your favorite book?

7 Powers: The Foundations of Business Strategy. It's probably taught me the most.

Amazing. Next question. What tool do you end up using the most for your work? Well, 2 parts. Part A, as a PM at Scale, and now as an investor at Founders Fund.

At Scale, honestly, it was either Quip, which was our documentation platform at the time, or Jira or Clubhouse or, you know, whatever PM tool that we used. Um, because we switched a lot. I think it was eventually Clubhouse. Um, and then at Founders Fund, honestly, it's LinkedIn and Twitter. It's terrible, but Twitter's a business tool for me.

So, uh, so how do you, how do you manage, or how do you keep track of all the conversations? Is it like Excel?

I would love— I would actually love advice on this if, if any of your listeners have like a Twitter tool that they use, like, to, like, categorize things and, like, make your feed only productive and, like, keep up with everybody, that would be, like, amazing. But yeah, I don't know. I, um, I use Affinity to basically, like, if I know— if I talk to somebody and I know that they're gonna, like, start a company or, like, I really want to check back in with them, like, I'll just set a ton of reminders. And then, like, every week I'll go through all of my Affinity reminders, like, that, you know, you're supposed to ping such and such so-and-so at this, at this moment. But yeah, I would love like a professional business Twitter tool that like let me categorize like things and could make my feed like just perfect. Like, I still get wacky stuff in my feed from like, this is popular, you might like it. And then it's like something that's just obscene.

Yeah, no, I don't like it. Yeah, I know, I know. It's one of the big problems of Twitter. That hasn't stopped me from using it, but still, they gotta fix it. Somehow. Yeah. All right, next question. What has been the biggest development in AI over the last 3 years?

I'm going to go with transformers and the sort of growth in size of transformers, like in terms of hyperparameters, and then also using more data to do things.

Yeah, definitely. Definitely one of the biggest. Next question. What product do you wish existed today Product I wish existed today.

Well, I mean, it seems like there's just a lot of problems around, like, deployment, compute management, compute cost management. I mean, I think there's some companies that are tackling this today, but I do think there's still a lot to do in, like, terms of data and orchestration and things like that. So, I'm excited to see all the startups develop in that space.

Right. When it comes to educating yourself on a new topic, What type of content do you prefer? Like, what's the process?

Yeah, I mean, definitely first is I have just kind of like a network of trusted friends and people from work, you know, all of my work experiences that I kind of like lean on when I'm learning new things. And then a lot of just like written content, blogs, books. Like at Founders Fund, we have a lot of tools that we can use to educate ourselves, whether it's like a A PitchBook or a Capital IQ, or like even like an expert network, like an AlphaSights or a GLG where you can literally, and this is like such a fun part of the job, you can literally just call expert people and talk to them. They don't even know who you are. Ask them all these questions, which is a lot of fun.

No, that's actually really good. I was about to add another question is like, what type of goods and services do you personally spend money on? Looks like GLG, great example. Like buying knowledge from experts. That's fantastic.

Keep buying Alpha, hopefully.

But there we go. All right, final one for you. What's your number one advice for founders building AI products?

Yeah, I think if I had to distill it down, it's find a mission-critical use case. So something that ultimately people are willing to pay for. I think that's something that I constantly have to remind myself and my portfolio companies.

Love it. I think that's a golden rule for everyone. Like, you got to get— we got to build something that people will pay for. All right. It's a business at the end of the day. You can get lost in your cool tech implementation, but at the end of the day, it's a business. So you got to find people who pay for it. So Leigh Marie, this has been fantastic. Really amazing conversation. Thank you so much for coming onto the show and sharing your insights.

Absolutely. Thanks for having me. Great questions. And it's been fun.

That was a fantastic conversation on how to build ML startups. Thanks to Leigh Marie for coming onto the show. You can visit infinitemachinelearning.com to subscribe to the podcast. Thanks again for tuning in. I'll see you soon with another amazing episode.