Investing in AI for Hard Tech, with Eric Vishria of Benchmark and Sergiy Nesterenko of Quilter

Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week we'll explore their revolutionary ideas, and together we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Thornberg. Hello and welcome back to The Cognitive Revolution. Today we have a short but fascinating conversation about investing venture capital into the AI wave and about the application of AI to hardware problems with Eric Vishria, general partner at Benchmark, and Sergey Nesterenko, founder and CEO of Quilter. Regular listeners will recall that we recently cross-posted an episode featuring Sergey from Will Summerlin's Autopilot feed, but rest assured that this episode covers almost entirely new territory, including what amount to my follow-up questions on Quilter's vision for superhuman circuit board design, the reinforcement learning approach that they're taking to get there, and what this could all enable for society more broadly. We also get Eric's point of view on what sorts of investments are likely to pay off in the AI era, starting with the provocative vision of circuit board design as an AI-and-done problem, not another copilot problem, that attracted him to Quilter in the first place. We also consider the questions that Benchmark asks in an attempt to determine whether a new company is solving an enduring or a temporary problem, And we discussed the reasons that Benchmark has not invested in any foundation model companies to date. Eric describes these companies as investing huge amounts of capital to create some of the most remarkable, but also some of the most rapidly depreciating assets in venture capital history. As always, if you're finding value in the show, we'd ask that you take a moment to share it with friends. And at the same time, we invite your constructive feedback and your topic or guest suggestions, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. Finally, a quick reminder that I am looking to connect with aspiring AI engineers and also AI advisors who are excited about helping Main Street businesses take advantage of AI technology. I've mentioned this twice and got good feedback each time, so please keep it coming. And before long, we might have a Cognitive Revolution jobs board on our hands. For now, I hope you enjoy this under-the-hood look at AI investment strategy and this glimpse into the future of AI-powered hardware design. with Eric Vischeria of Benchmark and Sergey Nesterenko of Quilter.

Eric Vischeria, general partner at Benchmark, and Sergey Nesterenko, founder and CEO at Quilter. Welcome to The Cognitive Revolution.

Thank you for having us.

Thanks, Nathan.

Yeah, excited for this conversation. Regular listeners will of course know that we ran a cross-post with Sergey from Will Summerlin's new feed on all things AI in automation. And so for the deep dive on Quilter, that's the episode to check out to get up to speed and learn how the company is approaching the automation of integrated circuit board designs with a reinforcement learning approach, which is fascinating. Today, I think kind of a little bit of a broader conversation on just kind of where this whole AI thing is going big picture and how you guys are thinking about both investing in companies that will hopefully stand the test of time and also try to build a company that will be able to successfully ride the wave and not be crashed over by the wave, which is definitely, I think, a challenging dance for many founders right now. I guess, Eric, maybe you want to start off with just giving a little bit of background as to what it was that got you guys interested in Quilter specifically and how that fits into your broader philosophy and portfolio. Sure.

Super, super excited to be here. So thank you again for having me. And having us on the show. There's no shortage of AI startups out there. So we're, each of us, each of the Benchmark partners is probably meeting 4 or 5 of them a week. And so you're kind of perpetually meeting new ones and a lot of them sound really interesting. You know, they're really interesting ideas. And I would actually even say the revenue traction on a lot of the companies is tremendous. Like it's just unlike anything that we've seen just in terms of the speed of revenue traction and everything else. But obviously, in the kind of timeline that we're investing as early-stage investors, you're looking on a 5 or 10-year horizon, really, of the company maturing into a big business and wanting to be one of these exceptional kind of companies that get created in our industry a few times a decade. And so, a big part of what we're looking at is which of these companies is not just riding the wave and not just on a sugar high, as I would describe it, of either revenue or traction or developers or hype investor dollars, but has an opportunity to build a really big business. And obviously, that's really hard to figure out at the early stages of the company, but that's what we're paid to do. So that's what we try to do. And one of the frameworks that I use quite often is just, well, what's changing in the world that is going to enable this company to be really large? Which has nothing to do with the company, actually. It's just like what's happening in the world. And of course there's AI, which is, okay, that's a wave, but there's actually, you need something more substantive and specific for a company to be able to be created and be durable around that. And I think in the case of Quilter, there were a couple of things that really caught my eye, which was, first off, obviously electronics are permeating every aspect of our life, right? Like, Things that were very simple not that long ago, like a light switch, now have electronics in them. They have circuit boards in them in ways that just weren't true before. And so, you have a rapid expansion of the number of companies, the number of products that are electronics and have circuit boards in them. And that is a first very fundamental thing that's happening in the world and seems bad. Then, and I didn't really understand this until I met Sergey, but the process of actually designing or laying out these circuit boards, place and route, I guess, as it is called in the parlance, is an incredibly manual and time-consuming and slow process. And one of the things I think we know just industry-wide is The faster you're able to iterate on products, the better and better they get. And that kind of friction in the process seems bad in a world where electronics are permeating every kind of everything that we interact with. So that was interesting. But then the real thing where it started to get, where the story started to get really compelling to me was particularly, I don't know, 6 months ago or so, everybody was talking about copilots. Copilot this, Copilot that. There's going to be Copilot for lawyers. There's a Copilot for doctors. There's a Copilot for developers. And in each of those spaces, there's 18 companies that are emerged that are doing some form of Copilots. And Yossi's kind of provocative statement at the time was like, this isn't a Copilot problem. It's just not for Copilots. Like, you don't want a human-assisted AI to do printed circuit board layout. And I was like, oh, okay, that's different than anything that I've heard in the last few months. That's just a different view. And so it's, well, why? And it's, well, if Copilot or an auto layout does 90% of the job, then by definition, the other 10% of these layouts, which is, they're all these thin lines that are connecting one component to another component, and they're all over the circuit board. As you look at it, you're like, wow, a human laid that out. And then that seems like really tough. And if I had to— draw another line, like, how am I going to draw the line and not mess up everything else that's already laid out? Like, they're pretty dense. And so, he's like, this isn't a— it's just not a human-assist problem. It's a problem that should be AI and done. AI should just do it, do it entirely. And in order to get there, we have to have a clean interface where we kind of get the design and architecture of it. And then we have a thing that can be spit out and go to manufacturing. But those interfaces exist in the industry. And so we sit in the middle and we take on that aspect of it and we just lay it out. And we have to obviously start with simpler boards. We can't do a super, super complicated iPhone A14 to start. Like we got to start with something simpler and work our way up, but we can work our way up and just be AI and done. And so that was the first, like really provide, like just makes you think, oh, Wait, there's a whole bunch of things that we just are like doing with humans and like Copilot's vogue, but is Copilot the right answer? Like in a whole bunch of cases, like Copilot's not the right answer. It's just not like we should just let AI do it, which is cool. That was a different perspective. So that was the first thing. And then the second thing, which really kind of got me thinking was, okay, everyone's talking about like large language models. And he's like, yeah, we're not using any of that. That's not what we're using. Oh, okay. Well, that's different too. And I'm going to, and I'll come back around in a second, but that was his perspective was like, hey, you know, that's not the right AI for this problem. The right AI is we design a game, we tell the game what the optimization problem is. And do we want to optimize for price? Do we want to optimize for performance? Do we want to have a really dense board that's expensive to build, but super power efficient? Or do we want to have a really cheap board that's like bigger and has more space on it? And if we actually give the user these controls, then the AI will be able to give them a range of solutions and they can kind of pick where they want to be on that solution set. And I was like, that's cool. That's a very provocative thing. And so those were kind of the very specific things, like, which is there's electronics permeating everything. There was a view that this is an AI and done problem, not a copilot problem, which was different. And there was a view of the right kind of AI to use for this particular problem, which isn't, again, the kind of thing that everyone's talking about. And as you kind of dug into it, it seemed like really cogent. And I know nothing about printed circuit boards and know very little about obviously specific technology. So like, you're kind of like thinking about it and then you're trying to validate it and then you're talking to people and going after it. But the biggest thing that, just to kind of zoom out for a second, For any entrepreneur and anyone kind of thinking through these things, there's a— Chris Dixon wrote this short post several years ago called The Idea Maze. And if you haven't read it, you should read it because it's a short but very powerful post. And I'll summarize it. What you want to do when you're talking to an entrepreneur is like you want someone who's been like rolling around in this idea for a long time, like they've been in the idea maze and then they've kind of gone down a path and they've hit a dead end and then they've backed up. And gone down a different path and hit a different dead end and then backed up and kind of tried to figure their way through the maze. And as you're talking to them and asking them questions, what you keep getting is you keep getting, yeah, I thought about that, but here's why that doesn't work. So here's why I think this could work. And, and you're just like, and what you realize is maybe they have a path through or maybe they don't, but they've been exploring and living and rolling around in this idea and turning it over and turning it. this way and that way. And so they just, there's so much depth on the problem. There's so much depth on the potential solution that you kind of are like, if there's going to be someone who's going to figure it out the way through the maze, like this person has a really good shot at it. And obviously with Sergey, his experience at SpaceX, designing a bunch of boards, going through that, and then playing with and trying to figure out solutions and applications of AI to solve this problem. First realizing the problem, trying to pursue a whole bunch of different ways to solve the problem, having the realization that it really is an AI and done kind of problem, having the realization that of the right kind of AI techniques to use for it. Those were just like evidence to me that he's lived in this maze for a long time. And, and so that was probably like the macro thing that gets you excited. It just ends up being like very exciting when you find someone who has kind of approached it like that. Cool.

I appreciate the backstory. I kind of want to maybe circle back in a minute to this disruptive approach. This seems like a pretty classic, almost textbook example of a disruptive solution in that it seems like it's coming in at kind of the low end of the market. It's serving people. If I recall from the other episode that we posted, that it was striking to hear, actually, that SpaceX has an internal board team, but they couldn't serve you. And it's, man, talk about a market that's got to be very broadly underserved if an internal specialist team can't even serve the other team at SpaceX. That's pretty wild. So it seems like there's potentially a very kind of textbook pattern here of starting at the bottom of this market, massively expanding the bottom of the market. I wonder how often that's something that you're seeing across the portfolio or specifically trying to do. But maybe before coming back to that, could I try a little bit of this idea maze stuff? I have a couple of ideas that I'm wondering kind of maybe why they didn't work or why they wouldn't work. Would you be game for a couple of possibly harebrained ideas that you can shoot down, Sergey?

Yeah, if I can. Happy to.

Okay. These are potentially quite novice ideas, but I guess for starters, what's the data landscape in this space? Right. Typical deep learning approach is predicated on a lot of data. Is there any open source dataset out there that somebody could go tap into, any significant scale of published boards that could be used in that way? Or was sort of just lack of data a forcing function to make you go in another route AI-wise?

Yeah, that's a good question and probably one of the most common questions that I get about how Quilter is using AI for this problem. I think what most people tend to think is, how does Copilot work? You take all of GitHub and all of Stack Overflow, feed it into an LLM, and it makes good predictions based on the average human behaviour. And obviously we don't do that. So we don't do that for 2 reasons, right? One that you mentioned is there actually just isn't that much data, right? So if you look at, if you look at all of the, all of GitHub, not that many open source boards. There's a few sources here and there, but not a lot. The best ones are locked away behind companies, right? They're in Apple's repositories and Google's repositories and SpaceX's repositories. But there's another reason that's even more compelling to me why that's the wrong approach, which is fundamentally. People are not good at designing boards, just full stop. So if it's a process that takes, for a complicated board, 2, 3, 4 months, you are going to make a whole bunch of like margin on margin decisions that make your resulting board like much bigger than it otherwise could have been, use more layers than it could have, be more expensive than it could have. And if you just use data to do supervised learning and try to predict boards, you're probably going to get roughly that level of performance of a high-level human designer, even if you could get all of that data. But with reinforcement learning, you have the opportunity to go significantly better than humans, right? So this is the most famous example I always come back to is DeepMind playing Go, right? The AlphaGo problem. They actually started first by training on human data and then creating agents that are based on human expert moves. And they got to a grandmaster level with that. But the best system today starts with no human data at all. It just learns how to play the game and it determines whether it wins or loses. And then that is what gets you to far, far superhuman levels nobody can match.

So did you also start with literally zero human data as input, or did you have some? And especially if you started with none, how do you get over the sparse reward problem? This seems to come up all the time. And I'm reminded also of the eureka. I used to say about AI, no eureka moments, meaning at least from the generalist systems, you wouldn't see them doing legitimately new stuff better than human. With the Eureka project, actually, I started to have to say precious few eureka moments, because now there's at least some examples that are starting to pop up. In that case, I'm sure you're well aware, they used GPT-4 to write the reward function for the robot hand as it was learning to do all these tasks, where in the beginning its success is so fleeting or even nonexistent that it's really hard to even score what it's doing. So I guess I'm curious, how did you get over the sparse reward problem? Especially given how little data you had to go on at the beginning.

Yeah, totally. So in general, the sparse reward problem kind of broadly stated is your kind of a good reward. So winning the game of creating a circuit board is so rare that as you randomly explore, you never find it. And that's an issue because if you never get a signal that you've won a game, how do you ever learn anything? So the nice thing is that people have this problem, right? Like people in general have broken up the problem of circuit board design into multi diff— many different steps and have basically come up with heuristics along the way. So what Quilter can do in terms of not running into the ultimate sparse reward problem of design an entire board, and at the end, only if you get a yes from all the physics simulators that this board is going to work, do you get a 1, and otherwise you get a 0, right? What Quilter— what we do is just break it up into problems, right? So we take the first part of the problem, which is placing the components. We can make sure that at that point it's manufacturable, the components don't collide, things of that nature, and then compute some heuristics that basically indicate how likely we are to succeed at the next phase, which is the routing phase. And so on and so forth. Similar things for actually meeting all of the physics constraints, right? Like you can simplify the problem to an extent to compute basic, fast, dense rewards that correlate to your ultimate sparse reward. And that's what we have to do for now. Now, I will say that like long-term, the dream for this is definitely like a single sparse reward. There's a single yes or no, this board will work or not. And when we're really uprooting the typical patterns that humans use and are trying to do much better than them, that's what we'll have to do. But for now, since we're not as good as humans at layout, especially on more complicated boards, we can still use the same heuristics that they would use to just at least automate similar to what they would've done.

Hey, we'll continue our interview in a moment after a word from our sponsors. Yeah, interesting. It sounds like you're— if I understand your comment correctly, the answer to this might be yes, but I was wondering if there's an analogy between the reinforcement learning processes that are used on the language models and these sort of evaluator systems, which I understand are not models, right? They're just sort of either simulators or checkers that are deterministic. But people worry a lot about, in the context of language models, the idea that the human reward signal is not fully reliable, right? Like, we're kind of inconsistent, we're sometimes mistaken. It's been observed in many language models that there's a certain sycophancy, tendency where it seems to try to tell you what you want to hear versus the truth in some cases, because maybe that's what got higher reward in the training process. So it sounds like there is a similar problem here where the checkers are that they might be like purely physics simulators and could be like rock solid. Or I could imagine that they're on a foundation of sort of a bunch of heuristics which might in some subtle ways kind of also lead the process a bit astray.

One thing that I've— this is an analogy that is imperfect in about 50 ways, but it's kind of supposed to— like, one of the things that I've been thinking about to try to articulate the GenAI limitations, and particularly like the LLM and even stability models, like limitations versus some of these other techniques and what humans are actually good at, is like, I think humans are very good at extrapolation. So like coming up with novel things and developing and pushing creativity. And if this, then that, like we can also do this and you can take it and extrapolate. But because of the way these are trained and everything else, it feels, and of course, who knows, but it feels like the, a lot of the gen AI stuff is good as interpolation, which is within the bounds of things we already know, like what are other points like in that space? And it kind of makes sense if you think about it. Like it's, they're interpolating. They're figuring out like what the next word is or what the next image could look like based on the training set that is pushed in a bunch of dimensions and humongous and everything else, but it's still bound. It's bound by all the stuff that humans have prior created. And so this is part of why I think the AI scare stuff is so misguided and not really because it's, hey, if they're interpolating, okay, that's, that's like, seems fine. And it feels like humans will continue to be good at extrapolation, which is developing new novel techniques to do like various things. And that's a very important and valuable thing that we'll be able to continue to do. That was a very macro answer to your question. I'm curious what you guys think.

I have a ton of thoughts about that, but I think our subscribers have heard them in other contexts. I mean, I kind of feel like all this stuff is converging. So that's why I was trying to make the connection between basically how much do we trust the reward signal in the context of reinforcement learning for language models? It certainly helped a lot, but not that much, I'd say, is kind of the consensus answer. I wonder what the situation is in the context of circuit boards. And then we could maybe also speculate about a similar question when it comes to self-play there. It seems the sort of narrow problem, not just one domain, but in general, narrow problems are better suited to self-play for now. But we are also starting to see some of these self-play techniques be applied to language models. And that's where I'm also like, I don't know if it's going to stay in the bounds of what humans have given it. For all that much longer. But let's take it piece by piece. Sergey, I, I, let's start with the, how much do you trust the reward signal in the context of your problem?

Yeah. One of the nice things about working on a hard physics problem, like humans are not in the loop. Like humans are actually bad at looking at a board and judging if it's going to work or not. That's where like 80% of boards that are built are faulty in some way. And it's because humans are just not good at that task fundamentally. But all of the core physics is computable, right? Like we care about laws of the Maxwell equations and laws of thermodynamics. And we've had convergent techniques to solve those for over 100 years. So in our case, like if you actually use the oracle, like actually numerically solving the Maxwell equations for every set of possible considerations and problems, and you're kind of being careful to make sure that you're convergent and approximate everything correctly, It's completely trustworthy, like much more trustworthy than the human, human result by far. Only problem with that is speed, right? If it takes 20 minutes to compute a single physics solution and you need to do millions of those as you fine tune on your model, that's problematic. So what you can do is you can make approximate models that are just conservative. And so at that point, maybe you're not doing the best possible arrangement that physics could allow, but you're still doing one that works, still doing one that competes or beats humans, and it's still definitely going to work because you've been careful about the physics approximations you've So I think that's one of the luxuries that we have is we don't have to negotiate with humans, right? We just have to make sure that it's manufacturable and that it's going to work. And physics tells us the answer and physics is unambiguous about that.

Gotcha. And these were tools that existed already in the industry that you're able to build on top of. Yeah. How, when the system is going about the process of designing, how iterative and sort of tree searchy is that? process? Because I think folks, again, will know at least the basics of AlphaGo, right? These are like hyperparameters ultimately that you can sort of turn up or down at runtime, right? But how deep are you going to allow a system like AlphaGo to search the space of possible moves is a huge factor in how well it's going to do. If you just make it do a single raw guess, it's not going to do super well. If you allow it to map out a bunch of possibilities and then get scores on all those possibilities, then it can sometimes land on superhuman, superhuman results. So give us a sense for kind of what that search iteration and scoring loop looks like in your context.

Yeah, I think actually this is somewhere where a user has meaningful choice. So for a lot of designs, like you just want it fast, right? Like you're maybe primarily focused on making sure that your schematic is okay. You don't care if the board is rather sparse. big, not very dense, maybe it's a little more expensive to manufacture, so on and so forth. And in that case, like you don't really want a whole lot of search. You just want the first answer that's going to faithfully implement the schematic on the board. And so in that case, the ideal thing is to return an answer within, you know, a few minutes to an hour. On the other hand, suppose you're looking at a board that's gonna have 100 million units produced, like an iPhone motherboard or something like that. And if you save a cent or a dollar on every board, like that's A lot of value. In that case, you might let it search for a month and explore in all sorts of directions, because that's how long it would've taken a human to do it anyway for a single design, never mind for the billions and billions you could explore with this kind of system. So this isn't like a lever that we have in the tool today. We kind of just, we basically treat overnight as the constraint for us right now, right? So the idea is that at the end of the day, you finish your schematic, you uploaded it, and either immediately or sometime within the next 12 hours, you get a result and That morning, the next morning you can look at it and see if it's, if it's up to standards. But in the future, I see that being a lever. And it's for you to decide what is more important to you.

Yeah, that makes sense. What are the size of the models? What does the compute look like and how much of it is on the inference side versus on the scoring side in the physics simulations?

Yeah, those are all really good questions. So obviously we're not dealing with the kind of compute that LLMs deal with. Like we're not using 10,000 GPUs to train across a trillion tokens and whatever crazy numbers are being used nowadays. This is a relatively focused problem and we can deal with much smaller models. We can deal with much faster convergence times, much less compute effectively. So the, right now the vast majority of the compute is going into kind of the, like actually playing the game. And so a lot of that is still actually CPU cores. And then some of it is GPU cores, depends on which part of the problem. And then the kind of other part of the majority is going into like training. And we actually train during the production runs. So we have fast enough environments and fast enough evaluation that as somebody uploads a board, we're not just doing inference, we're actually training on that board as it was uploaded. Now, the cost of physics is going to increase for us, for sure. One of the ways that we are expanding in the market is by enumerating all of the different types of physics considerations you have to look at and basically chipping them off one by one, right? So instead of saying, hey, we're going to attempt all kinds of boards from all types of physics and do only the easy small ones, we're saying, okay, for now we're doing low-speed boards that have up to 4 amps of current or something like that. And okay, the physics to compute that is straightforward. The next thing we're working on is high-speed digital, and then we'll step into approximate, maybe bounding methods of computing whether or not those high-speed digital signals are going to be okay or not. That won't be too expensive, but then we'll do 1 or 2 validation runs at the end with a full-wave model. And that's probably, if I had to guess right now, it's probably going to cost us a few GPU days per board, something of that nature.

Interesting. Okay. So it sounds like ultimately more compute on the validation side than on the generation side, just because of the intensity of simulating Is it simulating physics or is it just solving? Is it like a closed-form solution of a ton of crazy differential equations or is it a sort of more Wolfram-style, like you got to actually play this out in simulation, no shortcuts kind of a thing?

Yeah, the simulations, that's another thing that we're exploring heavily. There's a lot of different ways to solve differential equations, right? So the most brute-force simple way is, at least for electromagnetics, It's this method called finite difference time domain, where you literally just grid the entire world in 3D, and you basically just apply the differential forms of the Maxwell equations in sequence, right? So you take almost like a curl of your local pixels of electric field and do that for the magnetic field, do that for electric field, and you have to do that for 100 million cells for 100,000 steps, something like that. So just a lot of raw compute. But there's also much more clever methods, right? So FEM is kind of a different way of approaching this problem. Within electromagnetics in particular, there's things like method of moments that only look at like the surfaces of your different electrical systems. And then, then there are approximations. So in the approximation of low speed, low frequency, you can actually factor out time out of the differential equations. And so you can do this thing called the quasi-static approximation where you don't run time at all and you're only looking at capacitance and mutual inductance of the systems. And for certain frequencies, that's a perfectly sufficient approximation, so on and so forth.

Yeah, very interesting. That reality maybe blunts the value of this next idea that I had. But I was thinking, if you did have a lot of data, and I wonder if there presumably are some— I don't know the structure of this industry, but just using chips as a reference point, obviously there's a few manufacturers that take in a lot of designs and output the actual devices. I would assume there's probably some big players in the circuit board space as well who are getting lots of designs. And one might imagine the sort of big data approach, something like a diffusion model seems like it could be an interesting fit for this, where you would kind of run a bunch of loops through whatever inputs of the model can vary. But say you have a transformer, like the latest Stable Diffusion version. They run a bunch of passes through this transformer. At each step, they're denoising from raw visual noise to the image. You could imagine sort of a similar approach to— and this is also happening for proteins now too. It's crazy. So it seems like it could also perhaps apply to a circuit board design And first you're kind of doing it would sort of even follow the path that you described, where first you kind of lay out the big things and then gradually you're getting more and more into the low-level details of the design. I guess questions there would be like, do you think that would work? And if so, is that something that worries you from a competitive standpoint at all? And then maybe though, it just wouldn't even be that much of an advantage because the If all the compute is on the simulation side anyway, then maybe it doesn't really matter.

Yeah, so the big issue with that is the quality of your data, right? So one of the, just the facts of life in this industry is that 3 times out of 4 that you submit a board to a manufacturer, it looks right, it's manufacturable, the manufacturer can follow all the tolerances and all that stuff, but the physics just doesn't work. So even if you collected all of that data, you still have the problem of going through and identifying which of these is an actual working board, And which of these have mistakes, right? Because it's a junior engineer or somebody who didn't see some sort of issue or something like that. Even senior engineers make mistakes on the electromagnetic supports all the time. I mean, you just, you fundamentally have thousands of components, tens of thousands of traces to look at, all of which impact the other. And so it's just very difficult for a human to keep all that in mind. So if you're going to clean the data, you still have to run the simulations to make sure that whichever candidates you're training are actually good. The other problem is that the information that the manufacturers are getting is not sufficient to do this. The manufacturers basically get, you can imagine this like a photo, like if you're developing film, you get a mask, a set of masks that shows you how to etch copper on all the different layers of the board and then which components to glue down. But that doesn't actually tell you what signals are happening throughout the board. And you need to know that to evaluate the physics. So, I mean, we could generate our own data by just self-play or like every time we find a good candidate, save it in a database, then train the diffusion model to just recreate those. I think that's valid. But the point is that you still have to kind of come up with those data points that you have verified from physics first principles are actually good designs.

Yeah, that's interesting. I feel like there's some way in which the noise sort of cancels out. No doubt that in, and I'm just kind of porting my intuitions from other domains of AI here, but Certainly no doubt that people do spend a lot of time curating data and going for quality, but also it does seem that the models are pretty tolerant to at least some amount of kind of wrong stuff in the data. And I guess it kind of regularises out in the training process, one hopes. And in practice, it does seem to work. I don't know if you think that it's just fundamentally not going to work in this case, but it seems like more your motivation, if I understand correctly, is like, It's more about you want to get to superhuman, and you think that the reinforcement learning is obviously like the proven path to get there. And that argument definitely makes a lot of sense to me. What do you think is the timeline to superhuman circuit board designs? What would keep you from going faster toward that superhuman board design future?

Yeah, I mean, in my perspective, the bottleneck is talent, right? Finding really great people to work on this kind of problem is the hard part. You have so many different aspects of this that need to be really amazing, right? You need to have amazing people who are expert at neural nets, expert at cutting-edge reinforcement learning methods. You can't just grab the latest thing and apply it and hope for the best. There's a lot more nuance to this. But also on the C++ side, the CUDA side, the physics side, all of those things have to come together. The timeline, I can't predict it exactly. Maybe for small boards we're a few years away, maybe for something like a motherboard we're 5 years away. But I'm guessing.

Yeah, my crystal ball also gets very foggy more than a couple months out.

As it should.

In terms of the kind of big problems that you need to solve that you're maybe not sure how you're going to solve, or that you need the talent to come join the company to be able to get over certain humps, I often feel like when I talk to people, especially those in research, and you may say, well, that's the difference, is that it's research versus actual engineering. But I often feel like I get the sense that a lot of things are working, like a very high percentage of things are sort of working, and that a lot of times multiple different approaches probably could have worked. It seems like just in general, we've kind of hit on a couple of architectures that are really working, but it seems like there's a lot more where that came from. Presumably we'll be discovering more and more all the time. Is there something that you're legitimately not sure if it's going to work or really have no idea how you will make it work? Or does it feel like the kind of thing where, of course, there's going to be work and optimization and making it run faster and all that kind of stuff, but basically it feels like you're going to climb? Is there like a sheer face of a mountain that you have to scale vertically, or is it kind of graduated stairs that you're pretty confident you can climb one by one?

I'm confident in the latter. Like, this is 100% a solvable problem. Given enough time, I'm confident we'll solve it. There's a lot of— the nice thing about this problem is there's a lot of steps you can take, right? Like, with something like self-driving, you kind of have a do or die, right? Either you're confident you're not going to crash or you're not. And of course there's still gradations, but like the stakes are really high. With us, we have a lot of checkpoints along the way, right? We have checkpoints in terms of complexity and size of board. We have checkpoints in terms of the physics that we can solve. We have checkpoints in terms of the ambition to be significantly better than humans that can come over time. And with that kind of— those kinds of stairs ahead of you, you can treat each one as, okay, like now it's— you need to make this 20% better, 20% better, 20% better, 20% better, and let it compound over time. I don't see anything fundamentally about this problem that is unsolvable in any way. It's going to be hard. Don't get me wrong. It's going to take a lot of people. It's going to take a lot of effort. There's nothing about this that seems unsolvable in any way.

Hey, we'll continue our interview in a moment after a word from our sponsors. When you hit that mature phase where we now have superhuman circuit board design, what does that look like to somebody like me? Can I just show up and say, hey, I'm making a talking stuffed animal? And how ignorant can I be and still get a board out? Because it's like today to even specify what you want is sort of an expert problem. What do you see the sort of independent entrepreneur tinkerer kind of person who has a product in mind, knows literally nothing? Do they talk to a language model and work their way towards specifications and then put specifications into your system? Or what's that sort of future state round trip look like?

Yeah, sure. So with us, we are focused explicitly on layout. Which is kind of one of two problems, right? To give you an analogy to hang onto, think of creating a schematic for a circuit board as writing code, right? Like a schematic literally looks like a block diagram with inputs, logic in the middle, and outputs. And it's entirely kind of logical and abstract. You're just communicating to other engineers what the inputs, outputs, and functionality of the board will be. Layout is like compiling your code, right? Like how do you actually make something physical that takes that schematic or Takes that code and like actually makes the atoms of the world do that, right? So we are very specifically targeting the layout portion because we think a compiler should exist for electronics and it just doesn't. The nice thing about that is that a lot of the benefits that we now have because compilers exist in software, I think are going to happen and are, and are gonna answer your question in electronics, right? So compilers eventually led to higher level languages, eventually led to things like Python. eventually led to large language models that allow you to write the Python and automate the whole thing. We will eventually move up that stack and look at schematic and how to make that piece easier so you don't have to learn how to write C++, that you could learn how to write Python, or maybe even just deal with block diagrams and make a circuit board.

Yeah, that's really interesting. Reminds me a lot of, we just did an episode on this tiny GPU project where a young guy who set out to design his own GPU from scratch in 2 weeks. And it ended up taking him 4, which was pretty remarkable. And a big part of why that's possible, I mean, it's still quite an accomplishment in my view, but a big part of why that's possible is that the sort of equivalent of the layout compiler does exist. And so he was able to kind of get to that stage and be able to feed it into an existing system that could do that sort of gnarly work for him. But yeah, that's helpful. But just zooming back out to the benchmark portfolio more broadly, what would you say here are kind of like the patterns that are common across the portfolio? What investments have you guys made that sort of have a very different pattern in terms of what part of the market they're going after first that are maybe in or not in this classically disruptive mode? And I guess broadly, how do I make money in AI?

Yeah, that is the right question. I'm very bullish because I think The overall, like, we want to be in areas where there's lots of disruption and lots of things changing. That creates the kind of primordial soup for there to be like new big things created. So that's, it's really valuable in that way. And we've seen that. And there's always a lot of crap that gets created and a bunch of stuff that doesn't work. But there are a bunch of good things that get created in the process and it's It's all about finding them. Really loosely, I think there's, you kind of have in AI right now, there's, it feels to me like there's 3 big categories of companies. There's the foundational model companies, right? That are like, they're building foundational models or they're, they have some techniques around it. They have proprietary data. Maybe they have other things that they're using and they're trying to do something really special there. Then there's a set of like infrastructure companies in the next category. I'm on the board of Cerebras, which we invested in in 2016. That's an AI chip and systems company that's focused on training. But you have Groq. I'm also on the board of Fireworks, which is an inference provider. And there's a bunch of things where we're also investors. So there's just, there's quite a few in that category too, which are their infrastructure, which is enabling some of the other stuff that's happening in the ecosystem. That's been a pretty fruitful area. I think those companies have done well. They've gotten to revenue quickly and so forth. And then the third category are companies like Sergey's, like Quilter, which are vertical applications of AI. So they're applying AI to try to do something, right? And we see those for lawyers, we see them for doctors, we see them for accountants. We obviously printed circuit board design. So there's just like a bunch of those things too. And each of those 3 categories, which is, it's very abstract and loose And they're all kind of different. They have a set of really big opportunities and really big problems. I would say the foundational models are incredibly expensive to develop. They are extremely quickly depreciating. I said this before, I think they're the fastest depreciating asset in the history of humankind, which is like you build one, you spend $150 million on it, and 6 months later, someone can build the same thing for $5 million. Like, that's a That's not historically a good way to make money with venture capital. Like, it's just like that money tends to get incinerated and then someone has something alternative. And we'll see, I could be totally wrong on that. Maybe someone will build something that defeats it, but the general purpose models have proven to be really expensive and depreciate very quickly what is cutting edge. The infrastructure companies have done pretty well. In a bunch of ways, like they have real business opportunity, they're enabling a bunch of things. The question always exists with them, which is, do those problems exist for very long? Like to some degree, a lot of AI development is where software development was like circa, I don't know, 2002 in terms of like the toolsets available and the abstractions that people are working with. And so Some of the things that people are solving, just, I'm just not sure that they're going to be around for a long time. And so that's a potential real challenge there, but there's a lot—

What do you have in mind there? Are you talking like LLM observability type?

Yeah, that's a great example. That's a great example, which is just like, is the LLM observability thing a thing? Like it's a thing today. I don't know if it's a thing long-term, like maybe not. We have to kind of think think through it. But that's a pretty challenging— that could be a potentially challenging area. That'd be one that we think about a lot. And then the vertical stuff has proven to be the fastest revenue traction, like some of the revenue scale of the vertical companies has been insane. But the kind of common criticisms of, are there any moats or barriers to entry? If you can build a company very quickly in 3 months or 6 months and get something out to people.

Weekend hackathon for that matter.

Yeah. Weekend hackathon. Then that also means the 10 other competitors are going to do that. And, and, you know, do you end up with something durable there or not? And I think we just don't know in a bunch of cases and you have to have a theory of it. Like in, in the case of looking at Quilter or Benchmark's investment in Sierra or, or some of these others, like I would say we have a view, a belief that there's something durable there that will be built and compound over time, even though they got quick traction. And so I think that's just something that we have to look at. And there's a lot of value. This is what's cool about what's happening right now is there's like lots of traction. There's lots of really interesting ideas in all of these categories and obviously amazing people working on them, but we got to figure it out. And so that's kind of how I've been thinking about it. I'd also say there's this other thing that's really interesting to me where a lot of the AI work has happened with researchers. Like you hear it all the time. This is a researcher, they're PhDs, they're doing that. And who was telling me this? I forget. One entrepreneur was telling me this was like, you get this. So the researchers are writing something, it works. It kind of works in a research context. which is a proof of concept or an experiment. And then they take that code and they try to scale it in production and it's, holy shit, that's not good. That doesn't work. It doesn't, it's not written for that or whatever. And this entrepreneur said, she's like, I have a rule when it comes to code written by researchers. And I think she said she was just like, it's just RM*. There's this bridge and divide, which is quite between research and proving something out conceptually and actually, like, turning that into shipping product that can scale and work and be iterated on. And that's a divide that just didn't exist in software development, at least in my adult lifetime. Maybe it existed in software development, like, back in databases in the '70s and '80s, but, like, that— I don't know enough. if it did or not, but it's been like, it's so that's when I say that like the stack maturity, like the kind of stuff that people have to do to work with PyTorch today or work with CUDA today or something like that. That's just not something that software developers have had to deal with for at least 20 years.

Yeah, that reminds me of comments I heard Demis Hassabis make about why they have taken this step now of merging DeepMind into Google proper after so many years of kind of holding it out as its own thing. He basically just said, we're now at the point where we're not done with research, but certainly a lot of the research has been done. And now it's like becoming the bridge to engineering that is a huge challenge. And that's where they feel like the time is right to—

It's more of everything. We like, we do want to continue to do novel research and push the limits of it. But as we try to bring this stuff into real applications to benefit humanity, we have to engineer it. And that's its own thing. It's not so simple, it turns out.

I know we're just about out of time with you guys today. Do you have any kind of call for startups? Anything that you wish somebody had brought you that you haven't seen or haven't been able to find yet?

I have this thing that if a venture capitalist has the idea, with very few exceptions, like maybe Mike Spicer or someone else like that can pull it off. But if the venture capitalist has an idea, that's a bad situation because this is not what we're— to the idea maze point where we started, that is not our job, nor is the thing that we are generally good at. Look, I would say I continue to be really excited. I think we have a ton of companies across the board in at least in terms of the infrastructure companies and the vertical applications that are super interesting, would love to meet amazing people who have been working on ideas or thinking about something for a long period of time and are like obsessed with it. And that's a very broad remit, but, you know, our job is to be involved in the best companies as early as possible. And so that's what we want to do.

Yeah, I hear that. With as fast as things are moving today, there's really no substitute for obsession.

Yeah, totally.

All right, great. Well, this has been a lot of fun, guys. Sergey Nesterenko, founder and CEO of Quilter. Eric Vischeria, general partner at Benchmark. Thank you both for being part of the Cognitive Revolution.

Thank you so much.

Thank you, Nathan.

It is both energizing and enlightening to hear why people listen and learn what they value about the show.

So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.