E44: How Benchmark Invests in AI with Eric Vishria and Quilter Founder Sergiy Nesterenko

Welcome back to Turpentine VC, a podcast where we discuss the art and science of building successful venture firms, VC to VC. In this episode, we're focusing on Benchmark's investment philosophy in the AI era with General Partner Eric Vishria and Sergey Nesterenko, the founder and CEO of Quilter. Sitting in for me as host is AI Scout and the Cognitive Revolution podcast host, Nathan Labenz. Up ahead, they cover the questions that Benchmark asks to determine whether a new company is solving an enduring or temporary problem, and the reasons why Benchmark hasn't invested in a foundational model to date. Eric Vishria, general partner at Benchmark, and Sergey Nesterenko, founder and CEO at Quilter. Welcome to the show.

Thank you for having us.

Excited.

Yeah, excited for this conversation today. I think kind of a little bit of a broader conversation on just kind of where this whole AI thing is going big picture and, um, how you guys are thinking about both investing in companies that will hopefully stand the test of time and also trying to build a company that will be able to successfully ride the wave and not be crashed over by the wave, which is definitely, I think, a challenging dance for many founders right now. Eric, maybe you want to start off with just giving a little bit of background as to what it was that got you guys interested in Quilter specifically and how that fits into your broader philosophy and portfolio.

Sure.

Yeah, that's— well, super, super excited to be here. So thank you again for having me and having us on the show. There's obviously, there's no shortage of AI startups out there. So we're, you know, each of us, each of the Benchmark partners is probably meeting 4 or 5 of them a week. And so you're kind of perpetually meeting new ones, and a lot of them sound really interesting. They're really interesting ideas. And I would actually even say the revenue traction on a lot of the companies is tremendous. It's just unlike anything that we've seen just in terms of the speed of revenue traction and everything else. But obviously, in the timeline that we're investing as early-stage investors, you're looking on a 5 or 10-year horizon really of the company maturing into a a big business and wanting to be one of these exceptional kind of companies that get created in our industry a few times a decade. And so a big part of what we're looking at is which of these companies is not just riding the wave and not just on a sugar high, as I would describe it, of either revenue or traction or developers or hype investor dollars, but is actually building, has an opportunity to build a really, really big business. And obviously, that's really hard to figure out at the early stages of a company, but that's what we're paid to do. So that's what we try to do. And one of the frameworks that I use quite often is just like, well, what's changing in the world that is going to enable this company to be really, really large? And in the case of, which has nothing to do with the company, actually, it's just like what's happening in the world That is this. And of course there's AI, which is like, okay, that's a wave. But there's actually, you need something more substantive and specific for a company to be able to be created and be durable around that. And I think in the case of Quilter, there were a couple things that really caught my eye, which was first off, obviously electronics are permeating every aspect of our life, right? Like, things that were like very, very simple not that long ago, like a, I don't know, a light switch, now have electronics in them. They have circuit boards in them in ways that just weren't true before. And so, you have a rapid expansion of the number of companies, the number of products that are electronics and have circuit boards in them. And that is a first, very, very fundamental thing that's happening in the world and seems bad. Then, and I didn't really understand this until I met Sergey, but the process of actually designing or laying out these circuit boards, place and route, I guess, as it is called in the parlance, is an incredibly manual process.

Yeah.

and time-consuming and slow process. And one of the things I think we know just industry-wide is the faster you're able to iterate on products, the better and better they get. And so, you know, that kind of friction in the process seems bad in a world where electronics are permeating every kind of everything that we interact with. So that was kind of interesting. But then the real thing where it started to get Where the story started to get really compelling to me was particularly, I don't know, 6 months ago or so, everybody was talking about copilots, like copilot this, copilot that. There's going to be a copilot for lawyers. There's a copilot for doctors. There's a copilot for developers, which is— and in each of those spaces, there's like 15 companies that have emerged that are doing some form of copilots. And Sergey's kind of provocative statement at the time was like, this isn't a copilot problem. It's just not for copilots. Like, you don't want a human-assisted AI to do printed circuit board layout. And I was like, oh, okay, that's different than anything that I've heard in the last few months. Like, that's just a different view.

Yeah.

And so it's like, well, why? And it's like, well, like, you know, if a Copilot or an auto layout does 90% of the job, then by definition, the other 10% of these layouts, and, you know, you can look at any circuit board and see what you're talking about, which is like, there are all these thin lines that are connecting one component to another component, and they're all over the circuit board. And then, and so you can just see, like, it just, as you look at it, you're like, wow, a human laid that out. And then that seems like really tough. And if I had to, draw another line, like, how am I going to draw the line and not mess up everything else that's already laid out? Like, they're pretty dense. And so he's like, this isn't a— it's just not a human-assist problem. It's a problem that should be AI and done. AI should just do it, like, do it entirely. And in order to get there, we have to, you know, have a, like, a clean interface where we kind of get the design and architecture of it. And then we have a thing that can be spit out and go to manufacturing. And so, but those interfaces exist in industry. And so we sit in the middle and we kind of like take on that aspect of it and we just like lay it out. And he's like, we have to obviously start with simpler boards. We can't do a super, super complicated iPhone A14 to start. Like we got to start with something simpler and like work our way up, but we can work our way up and just be AI and done. And so I thought that was a very, that was the first like really provocative, like just makes you think like, oh wait, there's a whole bunch of things that we just are like doing with humans and like Copilot's vogue, but like, is Copilot the right answer? Like in a whole bunch of cases, like Copilot's not the right answer. Like it's just not like we should just like let AI do it, which is cool. And so like, that's a different, that was a different perspective. So that was the first thing. And then the second thing, you know, which, which really kind of got me thinking, Was, okay, you know, everyone's talking about like large language models and like everyone's like LLMs and, you know, GenAI and blah, blah, blah. And he's like, yeah, we're not using any of that. That's not what we're using. I was like, oh, okay, well, that's different too. And I'll come back around in a second, but that was, you know, his, his perspective was like, hey, you know, that's not the right way to, that's not the right AI for this problem. Like we tried all of these other things, but like, this is, that's not the right AI for this problem. The right AI, and Sergey can describe it much better than me, is like, as we design a game, we tell the game what the optimization problem is. And do we want to optimize for price? Do we want to optimize for performance? Do we want to have a really dense board that's expensive to build but like super power efficient? Or do we want to have a really cheap board that's like bigger and has more space on it? And if we actually like give the user these controls, Then the AI will be able to give them a range of solutions and they can kind of pick where they want to be on that solution set. And I was like, oh shit, that's cool. Like, that's a very provocative thing. And so those were kind of the very specific things, like, which is like there's electronics permeating everything. There was a view that this is an AI and done problem, not a Copilot problem, which was different. And there was a view of the right kind of AI. to use for this particular problem, which isn't, again, the kind of thing that everyone's talking about. And as you kind of dug into it, it seemed like really cogent. And, and I know nothing about printed circuit boards and know, know very little about, you know, obviously specific technology. So like, you're kind of like thinking about it and then you're trying to validate it, and then you, you're talking to people and going after it. But the biggest thing that— just to kind of zoom out for a second— for any entrepreneur and anyone kind of thinking through these things, there's a Chris Dixon wrote this short post several years ago called The Idea Maze. And if you haven't read it, you should read it because it'll take you like 30 seconds, but it'll stick with you because it's a short but very powerful post. And I'll summarize it, you know, not as eloquently as Chris said it, but there's, you know, what you want to do when you're talking to an entrepreneur is like you want someone who's been like rolling around in this idea for a long, long time. Like they've been in the idea maze. And then they've kind of gone down a path and they've hit a dead end, and then they've backed up and gone down a different path and hit a different dead end, and then backed up and kind of tried to figure their way through the maze. And as you're talking to them and asking them questions, what you keep getting is you keep getting like, yeah, I thought about that, but here's why that doesn't work. So here's why I think this could work. And like, and you're, you're just like, and what you realize is like, maybe they have a path through or maybe they don't. but they've been exploring and living and rolling around in this idea and turning it over and turning it this way and that way. And so they just— there's so much depth on the problem, there's so much depth on the, um, on the potential solution that you kind of are like, okay, this person— if there's going to be someone who's going to figure it out, the way through the maze, like, this person has a really good shot at it. And obviously with Sergey, his experience at SpaceX designing a bunch of boards, going through that, and then playing with and trying to figure out solutions and applications of AI to solve this problem. First realizing the problem, trying to pursue a whole bunch of different ways to solve the problem, having the realization that it really is an AI and done kind of problem, having the realization that of the right kind of AI techniques to use for it. Those were just like evidence to me that, that like he's like lived in this, in this maze for a long time. And, um, and so that, that was probably like the, the macro thing that gets you excited. So it's this area of that, and we can talk about like so many of these other things, but that, that was just like a very, um, you know, it just ends up being like very exciting when you find someone, um, who has, who has kind of approached it like that. Cool.

I appreciate the backstory. I kind of want to maybe circle back in a minute to this disruptive approach. I mean, this seems like a pretty classic, almost textbook example of a disruptive solution in that it seems like it's coming in at kind of the low end of the market. It's serving people, if I recall from the other episode that we posted, that It was striking to hear, actually, that SpaceX has an internal board team, but they couldn't serve you. And it's like, man, talk about a market that's got to be very broadly underserved if an internal specialist team can't even serve the other team at SpaceX. That's pretty wild. Hey, we'll continue our interview in a moment after a word from our sponsors. So it seems like there's a potentially a very kind of textbook pattern here of starting at the bottom of this market, massively expanding the bottom of the market. I wonder how often that's something that you're seeing across the portfolio or specifically trying to do. But maybe before coming back to that, could I try a little bit of this idea maze stuff? I have a couple of ideas that I'm wondering kind of maybe why they didn't work or why they wouldn't work. Would you be game for a couple of possibly harebrained ideas that you can shoot down?

Yeah, if I can. Happy to.

Okay. These are potentially quite novice ideas, but I guess for starters, what's the data landscape in this space? Right. Typical deep learning approach is predicated on a lot of data. Is there any open source dataset out there that somebody could go tap into any significant scale of of published boards that could be used in that way? Or was sort of just lack of data a forcing function to make you go in another route AI-wise?

Yeah, that's a good question. And probably one of the most common questions that I get about how Quilter is using AI for this problem. I think what most people tend to think is, how does Copilot work? You take all of GitHub and all of Stack Overflow, feed it into an LLM, and it makes good predictions based on the average human behavior. And obviously, we don't do that. So we don't do that for 2 reasons, right? One that you mentioned is there actually just isn't that much data, right? So if you look at all of GitHub, not that many open source boards, there's a few sources here and there, but not a lot. The best ones are locked away behind companies, right? They're in Apple's repositories and Google's repositories and SpaceX's repositories. But there's another reason that's even more compelling to me why that's the wrong approach, which is fundamentally people are not good at designing boards. Just full stop. Right? So if it's a process that takes, you know, for a complicated board, 2, 3, 4 months, you are going to make a whole bunch of like margin on margin on margin decisions that make your resulting board like much bigger than it otherwise could have been, use more layers than it could have, be more expensive than it could have. And if you just use data to do supervised learning and try to predict boards, you're probably going to get roughly that level of performance of like a high-level human designer, even if you could get all of that data. But with reinforcement learning, you have the opportunity to go significantly better than humans, right? So this is the most famous example I always come back to is DeepMind playing Go, right? The AlphaGo problem. They actually started first by training on human data and then creating agents that are based on human expert moves. And they got to a grandmaster level with that. But the best system today starts with no human data at all. It just learns how to play the game, you know, and it determines whether it wins or loses. And then that is what gets you to far, far, far superhuman levels nobody can match.

So did you also start with literally zero human data as input, or did you have some? And especially if you started with none, how do you get over the sparse reward problem? This seems to come up all the time. And I'm reminded also of the eureka. I used to say about AI, no eureka moments, meaning At least from the generalist systems, you wouldn't see them doing legitimately new stuff better than human. With the Eureka Project, actually, I started to have to say precious few eureka moments, because now there's at least some examples that are starting to pop up. In that case, I'm sure you're well aware, they used GPT-4 to write the reward function for the robot hand as it was learning to do all these tasks where in the beginning, its success is so fleeting or even nonexistent that it's really hard to even score what it's doing. So I guess I'm curious, how did you get over the sparse reward problem, especially given how little data you had to go on at the beginning?

Yeah, totally. So in general, the sparse reward problem, kind of broadly stated, is a good reward. So winning the game of creating a circuit board is so rare that as you randomly explore, you never find Right. And that's an issue because if you never get a signal that you've won a game, how do you ever learn anything?

Right.

So the nice thing is that, like, people have this problem, right? Like, people in general have broken up the problem of circuit board design into many different steps and have basically come up with heuristics along the way. So what Quilter can do in terms of not running into the ultimate sparse reward problem of, like, design an entire board, and at the end, only if you get a yes from all the physics simulators that this board is going to work, do you get a 1, and otherwise you get a 0. What we do is just break it up into problems. So we take the first part of the problem, which is placing the components. We can make sure that at that point it's manufacturable, the components don't collide, things of that nature, and then compute some heuristics that basically indicate how likely we are to succeed at the next phase, which is the routing phase, and so on and so forth. Similar things for actually meeting all of the physics constraints. You can simplify the problem to an extent, To compute basic, fast, dense rewards that correlate to your ultimate sparse reward. And that's what we have to do for now. Now, I will say that, like, long term, the dream for this is definitely like a single sparse reward, you know, just a single yes or no, this board will work or not. And when we're really uprooting the typical patterns that humans use and are trying to do much better than them, that's what we'll have to do. But for now, since we're not as good as humans at layout, especially on more complicated boards, we can still use the same heuristics that they would use to just at least automate similar to what they would have done.

Yeah, interesting. It sounds like, if I understand your comment correctly, the answer to this might be yes, but I was wondering if there's an analogy between the reinforcement learning processes that are used on the language models and these sort of evaluator systems, which I understand are like not models, right? They're just sort of either simulators or checkers that are deterministic. But people worry a lot about, in the context of language models, the idea that the human reward signal is not fully reliable, right? Like, we're kind of inconsistent, we're sometimes mistaken. It's been observed in many language models that there's a certain sycophancy tendency where it seems to try to tell you what you want to hear versus the truth in some cases, because maybe that's what got higher reward in the training process. So it sounds like there is kind of a similar problem here where the checkers are— I guess I could imagine that they might be purely physics simulators and could be rock solid, or I could imagine that they're on a foundation of sort of a bunch of heuristics which might in some subtle ways kind of also lead the process a bit astray.

Yeah, one thing, and obviously Sergey can talk about the specifics, but one thing that I've This is an analogy that is imperfect in about 50 ways, but it's kind of supposed to— like, one of the things that I've been thinking about to try to articulate the GenAI limitations, and particularly like the LLM and even stability models, like limitations versus some of these other techniques and what humans are actually good at. Is like, I think humans are very good at extrapolation. So like coming up with novel things and developing and pushing creativity. And if this, then that, like we can, we can also do this and you can like take it and extrapolate. But because of the way these are trained and everything else, it feels like, and of course, who knows, but it feels like The, a lot of the gen AI stuff is good as interpolation, which is within the bounds of things we already know, like what are other points like in that space, right? And it kind of makes sense if you think about it, like it's, they're interpolating, like they're interpolating, they're, they're figuring out like what the next word is or what the next image could look like based on the training set that is like pushed in a bunch of dimensions and humongous and everything else, but it's still like bound. It's bound by all the stuff that humans have prior created. And so, like, this is part of why I think the AI scare stuff is so misguided and not really because it's like, hey, if they're interpolating, like, okay, that seems fine. And it feels like humans will continue to be good at extrapolation, which is developing new novel techniques to do like various things. And That's a very important and valuable thing that we'll be able to continue to do. That was a very macro answer to your question, but I think it's a useful thing. Curious what you guys think.

I have a ton of thoughts about that, but I think our subscribers have heard them in other contexts. I kind of feel like all this stuff is converging, so that's why I was trying to make the connection between the Basically, how much do we trust the reward signal in the context of reinforcement learning for language models? It's like, it certainly helped a lot, but not that much, I'd say, is kind of the consensus answer. I wonder what the situation is in the context of circuit boards. And then we could maybe also speculate about a similar question when it comes to self-play there. It seems like the sort of narrow problem, not just one domain, but in general, narrow problems are better suited to self-play for now. But we are also starting to see some of these self-play techniques be applied to language models. And that's where I'm also like, I don't know if it's going to stay in the bounds of what humans have given it for all that much Longer, but let's take it piece by piece. So Sergey, let's start with the, you know, how much do you trust the reward signal in the context of your problem?

Yeah, so one of the nice things about working on a hard physics problem, right, like humans are not in the loop. Like humans are actually bad at looking at a board and judging if it's going to work or not. That's where like 80% of boards that are built are faulty in some way. And it's because humans are just not good at that task fundamentally. But all of the core physics is computable, right? We care about laws of the Maxwell equations and laws of thermodynamics. And we've had convergent techniques to solve those for over 100 years. So in our case, if you actually use the oracle, actually numerically solving the Maxwell equations for every set of possible considerations and problems, and you're kind of being careful to make sure that you're convergent and approximate everything correctly, It's completely trustworthy, much more trustworthy than the human result by far. Only problem with that is speed, right? So if it takes 20 minutes to compute a single physics solution and you need to do millions of those as you fine-tune on your model, that's problematic. So what you can do is you can make approximate models that are just conservative. And so at that point, maybe you're not doing the best possible arrangement that physics could allow, but you're still doing one that works. still doing one that competes or beats humans, and is still definitely going to work because you've been careful about the physics approximations you've used. So I think that's one of the luxuries that we have is like, we don't have to negotiate with humans, right? We just have to make sure that it's manufacturable and that it's going to work. And physics tells us the answer. And physics is unambiguous about that. Gotcha.

So, and these were tools that existed already in the industry that you're able to build on top of?

Yeah.

When the system is going about the process of designing a board, how iterative and sort of tree searchy is that process? Because I think folks again will know at least the basics of AlphaGo, right? And these are like hyperparameters ultimately that you can sort of turn up or down at runtime, right? But how deep are you going to allow a system like AlphaGo to search the space of possible moves is a huge factor in how well it's going to do. If you just make it do a single raw guess, it's not going to do super well. If you allow it to map out a bunch of possibilities and then get scores on all those possibilities, then it can sometimes land on superhuman results. So give us a sense for kind of what that search iteration and scoring loop looks like in your context.

Yeah, I think actually this is somewhere where a user has meaningful choice. So for a lot of designs, like you just want it fast, right? Like you're maybe primarily focused on making sure that your schematic is okay. You don't care if the board is rather sparse, you know, big, not very dense, maybe it's a little more expensive to manufacture, so on and so forth. And in that case, like you don't really want a whole lot of search. You just want the first answer that's going to faithfully implement the schematic on the board. And so in that case, the ideal thing is to return an answer within a few minutes to an hour. On the other hand, suppose you're looking at a board that's going to have 100 million units produced, like an iPhone motherboard or something like that. And if you save a cent or a dollar on every board, that's a lot of value. In that case, you might let it search for a month and explore in all sorts of directions because that's how long it would've taken a human to do it anyway for a single design, never mind for the billions and billions and billions you could explore with this kind of system. So this isn't like a lever that we have in the tool today. We kind of just, we basically treat overnight as the constraint for us right now, right? So the idea is that at the end of the day, you finish your schematic, you uploaded it, and either immediately or sometime within the next 12 hours, you get a result. And that morning, the next morning, you can look at it and see if it's up to your standards. But in the future, I see that being a lever and it's for you to decide what is more important to you.

Yeah, that makes sense. What is the compute? Like, what are the sizes of the models? What does the compute look like? And how much of it is on the inference side versus on the scoring side in the physics simulations?

Yeah, those are all really good questions. So, like, obviously, we're not dealing with the kind of compute that LLMs deal with, right? Like, we're not using 10,000 GPUs to train across a trillion tokens and whatever crazy numbers are being used nowadays. This is a relatively focused problem. And so we can deal with much smaller models. We can deal with much faster convergence times, much less compute effectively. So right now, the vast majority of the compute is going into actually playing the game. And so a lot of that is still actually CPU cores. And then some of it is GPU cores, depends on which part of the problem. And then the The kind of other part of the majority is going into training. And we actually train during the production runs. So we have fast enough environments and fast enough evaluation that as somebody uploads a board, we're not just doing inference, we're actually training on that board as it was uploaded. Now, the cost of physics is going to increase for us for sure. One of the ways that we are expanding in the market is by enumerating all of the different types of physics considerations you have to look at and basically chipping them off one by one, right? So instead of saying, hey, we're going to attempt all kinds of boards from all types of physics and do only the easy small ones, we're saying, okay, for now we're doing low-speed boards that have up to 4 amps of current or something like that. And like, okay, the physics to compute that is straightforward. The next thing we're working on is high-speed digital. Right. And then we'll step into approximate, maybe bounding methods of computing whether or not those high-speed digital signals are going to be okay or not. That won't be too expensive. But then we'll do 1 or 2 validation runs at the end with like a full wave model. And that's probably, if I had to guess right now, it's probably going to cost us a few GPU days per board, something of that nature.

Interesting. Okay, so it sounds like ultimately more compute. on the validation side than on the generation side, just because of the intensity of simulating. Is it simulating physics or is it just solving? Is it like a closed-form solution of like a ton of crazy differential equations? Or is it a sort of more Wolfram style, like you got to actually play this out in simulation, no shortcuts kind of a thing?

Yeah, the simulations, that's another thing that we're exploring heavily. There's a lot of different ways to solve differential equations, right? So the most brute-force simple way is, at least for electromagnetics, this method called finite difference time domain, where you literally just grid the entire world in 3D. And you basically just apply the differential forms of the Maxwell equations in sequence, right? So you take like, almost like a curl of your local pixels of electric field and do that for the magnetic field, do that for electric field. And you have to do that, you know, for 100 million cells for, you know, 100,000 steps, something like that. So just a lot of raw compute. But there's also much more clever methods, right? So FEM is kind of a different way of approaching this problem within electromagnetics in particular, there's things like method of moments that only look at like the surfaces of your different electrical systems. And then there are approximations. So in the approximation of low speed, low frequency, you can actually factor out time out of the differential equations. And so you can do this thing called the quasi-static approximation where you don't run time at all and you're only looking at capacitance and mutual inductance of the systems. And for certain frequencies, that's a perfectly sufficient approximation. So on and so on and so forth.

Yeah, very interesting. That reality maybe blunts the value of this next idea that I had. But I was thinking, if you did have a lot of data, and I wonder if there presumably are some, like, I guess I don't know the structure of this industry, but just kind of Using chips as a reference point, obviously there's a few manufacturers that take in a lot of designs and output the actual devices. I would assume there's probably some big players in the circuit board space as well who are getting lots of designs. And one might imagine the sort of big data approach, something like a diffusion model seems like it could be an interesting fit for this, where you would kind of run a bunch of loops through whatever. Obviously the models can vary, but say you have a transformer, like the latest Stable Diffusion version, they run a bunch of passes through this transformer. At each step they're denoising from raw visual noise to the image. You could imagine sort of a similar approach to— and this is also happening for proteins now too. It's crazy. So Seems like it could also perhaps apply to a circuit board design. And first you're kind of doing it would sort of even follow the path that you described, where it's like first you kind of lay out the big things and then gradually you're getting more and more into the low-level details of the design. I guess questions there would be like, do you think that would work? And if so, is that something that, you know, worries you from a competitive standpoint at all? And then maybe though, it just wouldn't even be that much of an advantage because the— if all the compute is on the simulation side anyway, then like, maybe it doesn't really matter.

Yeah, so the big issue with that is the quality of your data, right? So one of the just the facts of life in this industry is that, you know, 3 times out of 4 that you submit a board to a manufacturer, like it It looks right. It's manufacturable. The manufacturer can follow all the tolerances and all that stuff, but the physics just doesn't work. So even if you collected all of that data, you still have the problem of going through and identifying which of these is an actual working board and which of these have mistakes, right? Because it's a junior engineer or somebody who didn't see some sort of issue or something like that. Even senior engineers make mistakes on the electromagnetics of boards all the time. I mean, you fundamentally have thousands of components, tens of thousands of traces to look at, all of which impact the other. And so it's just very, very difficult for a human to keep all that in mind. So if you're going to clean the data, you still have to run the simulations to make sure that whichever candidates you're training are actually good. The other problem is that the information that the manufacturers are getting is not sufficient to do this. The manufacturers basically get You can imagine this like a photo. If you're developing film, you get a mask, a set of masks that shows you how to etch copper on all the different layers of the board and then which components to glue down. But that doesn't actually tell you what signals are happening throughout the board. And you need to know that to evaluate the physics. So, I mean, we could generate our own data by just self-play or every time we find a good candidate, save it in a database, then train the diffusion model to just recreate those.

I think that's valid.

But the point is that you still have to kind of come up with those data points that you have verified from physics first principles are actually good designs.

Yeah, that's interesting. I feel like there's some way in which the noise sort of cancels out. Like, no doubt that in— and I'm just kind of porting my intuitions from other domains of AI here, but certainly no doubt that people do spend a lot of time curating data and going for quality, but also like, It does seem that the models are pretty tolerant to at least some amount of kind of wrong stuff in the data, and I guess it kind of regularises out in the training process, one hopes. And in practice, it does seem to work. I don't know if you think that it's just fundamentally not going to work in this case, but it seems like more of your motivation, if I understand correctly, is like, it's more about you want to get to superhuman and You think that the reinforcement learning is obviously like the proven path to get there, and that argument definitely makes a lot of sense to me. What do you think is the timeline to superhuman circuit board designs? And to ask a question Eric might have asked you in the private thread, what's the bottleneck? What would keep you from going faster toward that superhuman board design future?

Yeah. So, I mean, in my perspective, the bottleneck is talent, right? Like finding really great people to work on this kind of problem is the hard part. You have so many different aspects of this that need to be really, really, really amazing, right? You need to have amazing people who are expert at neural nets, expert at cutting-edge reinforcement learning methods. You can't just grab the latest thing and apply it and hope for the best. You know, there's a lot more nuance to this, you know, but also on the C++ side, the CUDA side, the physics side, all of those things have to come together. You know, the timeline, I can't predict it exactly. You know, maybe for small boards, we're a few years away, maybe for something like a motherboard, we're 5 years away, but I'm guessing.

Yeah, my crystal ball also gets very foggy more than a couple months out.

As it should.

In terms of the kind of big problems that you need to solve that you're maybe not sure how you're going to solve, or that you need the talent to come join the company to be able to get over certain humps, I often feel like when I talk to people, especially those in research, and you may say, well, that's the difference, is that it's research versus actual engineering. But I often feel like I get the sense that a lot of things are working, like a very high percentage of things are sort of working, and that a lot of times multiple different approaches probably could have worked. It seems like just in general, we've kind of hit on a couple of architectures that are really working, but it seems like there's a lot more where that came from, and presumably we'll be discovering more and more all the time. Is there something that you're legitimately not sure if it's going to work or really have no idea how you will make it work? Or does it feel like the kind of thing where, of course, there's going to be work and optimization and making it run faster and all that kind of stuff, but basically it feels like you're going to— is there like a sheer face of a mountain that you have to scale vertically, or is it kind of graduated stairs that you're pretty confident you can climb one by one?

I'm confident in the latter. Like, this is 100% a solvable problem. Given enough time, I'm confident we'll solve it. You know, there's a lot of— the nice thing about this problem is there's a lot of steps you can take, right? Like, with something like self-driving, you kind of have like a do or die, right? Like, either you're confident you're not going to crash or you're not, you know. And of course, there's still gradations, but like, the stakes are really, really high. With us, we have a lot of checkpoints along the way, right? Like, we have checkpoints in terms of complexity and size of board. We have checkpoints in terms of the physics that we can solve. We have checkpoints in terms of the ambition to be significantly better than humans that can come over time. And with that kind of, you know, those kinds of stairs ahead of you, you can treat each one as like, okay, like, now it's, you know, we need to make this 20% better, 20% better, 20% better, 20% better, and let it compound over time. I don't see anything fundamentally about this problem that is unsolvable in any way. It's going to be hard. Don't get me wrong, it's going to take a lot of people, it's going to take a lot of effort, but there's nothing about this that seems unsolvable in any way.

When you hit that mature phase where we now have superhuman circuit board design, what does that look like to somebody like me? Can I just show up and say, hey, I'm making a talking stuffed animal and How ignorant can I be and still get a board out?

Right?

Because it's like today to even specify what you want is sort of an expert problem. What do you see the sort of independent entrepreneur tinkerer kind of person who has a product in mind, knows literally nothing? Do they talk to a language model and work their way towards specifications and then put specifications into your system? Or what's that sort of future state round trip look like?

Yeah, sure. So with us, we are focused explicitly on layout, which is kind of one of two problems, right? So to give you an analogy to hang on to, think of creating a schematic for a circuit board as writing code, right? Like a schematic literally looks like a block diagram with inputs, logic in the middle, and outputs. And it's entirely kind of logical and abstract. You're just communicating to other engineers what the inputs, outputs, the functionality of the board will be. Layout is like compiling your code, right? How do you actually make something physical that takes that schematic or takes that code and actually makes the atoms of the world do that, right? So we are very specifically targeting the layout portion because we think a compiler should exist for electronics and it just doesn't. So the nice thing about that is that a lot of the benefits that we now have because compilers exist in software, I think are going to happen and are going to answer your question in In electronics, right? So compilers eventually led to higher-level languages, eventually led to things like Python, eventually led to large language models that allow you to write the Python and automate the whole thing. We will eventually move up that stack and look at schematic and how to make that piece easier so you don't have to learn how to write C++, that you could learn how to write Python or maybe even just deal with block diagrams and make a circuit board.

Yeah, that's really interesting. Reminds me a lot of, we just did an episode on this tiny GPU project where a young guy who set out to design his own GPU from scratch in 2 weeks and ended up taking him 4, which was pretty remarkable. And a big part of why that's possible, I mean, it's still quite an accomplishment in my view, but a big part of why that's possible is that the sort of equivalent of the layout compiler does exist. And so he was able to kind of get to that stage and be able to feed it into an existing system that could do that sort of gnarly work for him. But yeah, that's helpful. So, Eric, how does this kind of compare and contrast? I kind of can't help myself sometimes from going down the rabbit hole on techniques, but just zooming back out to the benchmark portfolio more broadly, What would you say here are kind of like the patterns that are common across the portfolio? What investments have you guys made that sort of have a very different pattern in terms of what part of the market they're going after first that are maybe in or not in this classically disruptive mode? And I guess broadly, how do I make money in AI?

Yeah, that is the right question. You know, I'm very bullish because I think the overall, like, we want to be in areas where there's lots of disruption. When there's lots of disruption and lots of things changing, you know, that creates the kind of primordial soup for there to be like new big things created, right? So it's really valuable in that way. And we've seen that. And there's always a lot of crap.

Yeah.

that gets created and a bunch of stuff that doesn't work. But there are a bunch of good things that get created in the process, and it's all about finding them. Really loosely, I think there's, you kind of have in AI right now, there's, it feels to me like there's like 3 big categories of companies. There's the foundational model companies, right? That are like, they're building foundational models or they're They have some techniques around it. They have proprietary data. Maybe they have other things that they're using and they're trying to do something really special there. Then you have the— there's a set of infrastructure companies in the next category. I'm on the board of Cerebras, which we invested in in 2016. That's an AI chip and systems company that's focused on training. But you have Groq. I'm also on the board of Fireworks, which is an inference provider. And so, and, you know, there's a bunch of things where we're also investors. So there's just, there's like quite a few in that category too, which are their infrastructure, which is enabling some of the other stuff that's happening in the ecosystem. That's been a pretty fruitful area. I think those companies have done well, they've gotten to revenue quickly, and so forth. And then the third category are companies like Sergey's, like Quilter, which are vertical applications of AI. So they're applying AI to try to do something, right? And we see those for lawyers, we see them for doctors, we see them for accountants, we obviously printed circuit board design. So there's just a bunch of those things too. And each of those 3 categories, which is very abstract and loose and they're all kind of different, they all have a set of really big opportunities and really big problems. I would say, The foundational models are incredibly expensive to develop. They are extremely quickly depreciating. I said this before, I think they're the fastest depreciating asset in the history of humankind, which is like you build one, you spend $150 million on it, and 6 months later someone can build the same thing for $5 million. That's not historically a good way to make money with venture capital. Like, it's just like that money tends to get incinerated and then someone has something alternative. And we'll see, I could be totally wrong on that. Maybe someone will build something that defeats it. But the general purpose models have proven to be really expensive and depreciate very quickly. Like, what is cutting edge? The infrastructure companies have done pretty well in a bunch of ways. They have real business opportunity. They're enabling a bunch of things. The question always exists with them, which is like, do those problems exist for very long? To some degree, a lot of AI development is where software development was circa, I don't know, 2002. Right? In terms of, like, the toolsets available and the abstractions that people are working with. And so, some of the things that people are solving, I'm just not sure that they're gonna be around for a long time. And so, like, that's a potential real challenge there, but there's a lot of—

What do you have in mind there? Are you talking, like, LLM observability type?

Yeah, like, that's a great example. That's a great example, which is just, like, Is the LLM observability thing a thing? Like, it's a thing today. I don't know if it's a thing long-term. Like, maybe, maybe not. We have to kind of think through it, right? But that's a pretty challenging— that could be a potentially challenging area. So, you know, that would be one that we think about a lot. And then the vertical stuff, the vertical stuff has proven to be— have in some way the fastest revenue traction. Like that some of the revenue scale of the vertical companies has been insane. But, you know, the kind of common criticisms of like, are there any moats or barriers to entry? Like, you know, if you can build a company very quickly in 3 months or 6 months and get something out to people—

Weekend hackathon, for that matter.

Yeah, weekend hackathon. Then like that also means the 10 other competitors are going to do that. And, um, and so, you know, do you end up with something durable there or not? And I think we just don't know, um, in a bunch of cases. And you have to have a theory of it. Like in, in the case of looking at Quilter or, um, you know, Benchmark's investment in Sierra or, um, or some of these others, like I would say, um, you know, we have a view, a belief that there's something durable there that will be built and compound over time, even though you know, they got quick, quick traction. Um, and so I think that's just something that we have to kind of like look at, and there's a lot of value. But I think there's like— this is what's cool about what's happening right now is there's like lots of traction, there's lots of really interesting ideas in all of these categories, and, and obviously amazing people working on them. But we got to figure it out. And So that's kind of how I've been thinking about it. You know, I'd also say like, there's this other thing that's really interesting to me, where a lot of the AI work has happened with researchers, like you hear it all the time, like this is a researcher, their PhDs, like, they're doing that. And who was telling me this? I forget, one entrepreneur was telling me this was like, you know, you get this, so the researchers are writing something, it works, it kind of works in a research context, which is a proof of context, proof of concept or experiment. And then they take that code and they try to like scale it in production. And it's like, holy shit, that's not good. That doesn't work. It doesn't, it's not written for that or whatever. And this entrepreneur said, she's like, I have a rule when it comes to code written by researchers. And I think she said she was just like, it's just RM*. And it was like, it was such a classic, like, uh, so, you know, it's just— but it was— but there's this bridge and divide, which is quite— between research and proving something out conceptually and actually like turning that into shipping product, um, that can scale and, and work and be And iterated on. And, you know, that's a divide that just didn't exist in software development, at least in my adult lifetime. Maybe it existed in software development, like, back in databases in the '70s and '80s, but, like, that— I don't know enough if it did or not. But it's been like— so, that's when I say that, like, the stack maturity Like the kind of stuff that people have to do to work with PyTorch today or work with CUDA today or something like that. Like, that's just not something that software developers have had to deal with for at least 20 years.

Yeah, that reminds me of comments I heard Demis Hassabis make about why they have taken this step now of merging DeepMind into Google proper after so many years of kind of holding it out as its own thing. He basically just said, We're now at the point where we're not done with research, but certainly a lot of the research has been done. And now, it's becoming the bridge to engineering that is a huge challenge. And, that's where they feel like the time is right. That's right. Yeah.

And, it's more of everything. We do want to continue to do novel research and push the limits of it. But, as we try to bring this stuff into real applications to benefit humanity, we have to engineer it. And, um, and like, that's a— that's its own thing. And it shouldn't be— it's not— it's not so simple, uh, it turns out.

I know we're just about, uh, out of time with you guys today. Do you have any, um, kind of call for startups? Anything that you wish somebody had brought you that you haven't seen or haven't been able to find yet?

You know, I have this thing that if a venture capitalist has the idea, with very few exceptions, like maybe Mike Spicer or someone else like that can pull it off. But if the venture capitalist has an idea, that's a bad situation because this is not what we're— to the idea maze point where we started, that is not our job, nor is the thing that we are generally good at. Look, I would say I continue to be really excited. I think we have a ton of companies across the board. At least in terms of the infrastructure companies and the vertical applications that are super interesting, would love to meet amazing people who have been working on ideas or thinking about something for a long period of time and are obsessed with it. And that's a very broad remit, but our job is to be involved in the best companies as early as possible. And so that's what we want to do.

Yeah, I hear that. With as fast as things are moving today, uh, there's really no substitute for obsession.

Yeah, totally.

All right, great. Well, this has been a lot of fun, guys. Sergey Nesterenko, founder and CEO of Quilter. Eric Vishria, general partner at Benchmark. Thank you both for being part of the Cognitive Revolution.

Thank you so much.

Thank you, Nathan.

Turpentine VC is a podcast from Turpentine, the network behind Moment of Zen and Econ If you liked the episode, please leave a review in the Apple Store or rate us on Spotify.