How Ricursive Intelligence’s Founders are Using AI to Shape The Future of Chip Design

Right now we can't have much of co design between chips and models because of this asymmetric design cycle for chips because it takes so long, so much, it takes so much time to design chips. The cycle is there's a mismatch between how fast we can create the next generation AI methods and how fast we can build the next generation chips. But if we can make our chips much faster, then we can enable this co design and co evolution of workloads, applications and chips altogether.

This week on training data, we explore the biggest bottleneck holding back AI, compute and the design of chips themselves. Our guests are the founders of Recursive Intelligence, Anna Goldie and Azalia Mihosseni, the team behind Google's pioneering alpha chip project that helped design four generations of TPUs. They're now applying AI to the entire chip design process, transforming the industry from fabless to designless. Anna and Azalia paint a picture of what a Cambrian explosion of custom silicon means. We'll explore the holy grail topic of recursive self improvement. How AI designed chips unlock radically creative new chip designs and why democratizing chip design could accelerate the path to superintelligence. Enjoy the show. Thank you so much for joining us. We're so excited to celebrate the announcement of recursive intelligence and congratulations on embarking on this new adventure.

Thank you, thank you so much.

To kick off, you've said that chip design is the compute bottleneck to advancing AI. Can you share and paint a picture of what's happening now? What, what are all the bottlenecks to chip design today?

I mean, I think when we were saying that, we were kind of alluding to this motivation that we had originally with our moonshot, which was this observation that neural networks, these concepts that have been around for decades, but the AI that came out of it wasn't that effective until we had more powerful computer systems and chips. And we thought that now that we had very powerful AI systems, we could use those to tackle the bottlenecks in chip design and, you know, create basically more effective compute. So if you see these like scaling laws, basically the more compute you apply to training a model or inferring with a model, like, the more intelligence you get out. And we're seeing that there's this mismatch, like we're using, say GPUs are like originally designed for graphics processing, but we're somehow repurposing them for crypto and then for training neural networks models, I guess they're good at like large matrix multiplies, but like if we could more better co optimize the models and the hardware, we could get more effective compute and then we could push ourselves out on those effective scaling laws.

You were the co creators of AlphaChip at Google, which I think is used in four successive generations now of TPUs. Maybe take us back to that project. How did it get started? What were the key results you drove?

Yeah, so we started a project in 2018 and we had some earlier version of placement, that placement work that we were interested in solving and that was not about chip placement, was about compiler and MA basically mapping neural networks to chips. So we did that project, we got great results, we published and we were like what are the highest impact projects that now we can kind of push this further and have real world impact and, and that's where the chip placement came in the picture. And we started the project working very closely with the TPU team at Google because back then we didn't know much at all or at all about how chip design is, how the process is. So we work very closely with them from very early on. And at the time it's interesting because at the time AI was like, was still very talked about but nothing like the scale was nowhere near what it is today. So we had to really work through things and showing them data and constantly iterating over our approach and hearing them what they needed from us, listening to them iterate through the process and eventually we benefit from a research concept on chiplate placement all the way to something that was actually used in product and we could tape out and all that.

I remember one of our very first meals together when it hit me that you were so customer obsessed and had so much empathy for the customer. And I asked you how and why and you said, well I've been working with the TPU team as our customer internally for so many years. So I think that really kind of gave you that other perspective. What were the TPU team's early reactions to what you had created?

I mean they were like super skeptical. For example, Azalia and I, we were researchers, we had read a bunch of research papers and we're like, okay, there's this half perimeter wire length, that's what academics report results on. So we made this RL agent that could optimize half perimeter wire lengths for replacement. And then we were kind of excited about sharing that with them and we showed these results and they were actually kind of angry at us. Why are you showing us these results? We don't care about half perimeter wire length. We want routed wire length congestion Horizontal and vertical congestion timing violations, power consumption area. And so we just kept listening to them and we were like, okay, so what is the thing that matters? I think another part is just to make your customer feel like they're part of it too. So for example, we worked with them to create the cost functions that they cared about, approximations of those. So like Mustafa, who was on the the TPU team, developed these very fast, like congestion cost function, um, and then we could optimizing at the end density and then wire length as well. And then we show results on those. And then we run the commercial tool and we show that this actually correlates to good results on the metrics that they do care about.

Yeah.

So I want to talk about floor planning. And I come from an EDA family, you know, floor planning is, it is the crown jewel of eda. Can you say, just for our listeners that don't come from the chip design industry, what exactly that is and how your technical methods help solve it.

So floor planning is a process where we place the components of a chip onto the silicon. And the complexity of this process comes from the large size of the input problems, like for a block of a chip, not the entire chip, like a single block in a chip. The graph that we are optimizing has mil could have millions of nodes and all of these nodes need to be placed and routed. And we need to make sure that all these constraints that earlier we were talking about like PPA power area performance are optimized and all the physical constraints are met. So the placement problem has to do with how we can do this, such that all those constraints given by these, like very small technology node sizes given by TSMC and others are met. So we are dealing with a combinatorial optimization problem that's large scale, but also all the metrics are hard to evaluate and measure.

Got it.

And this traditionally was done together with EDI software like a cadence or a synopsis. Right. How does your method differ from the classical methods?

So what we developed here was a learning based approach to the problem where we would train a reinforcement learning agent that would try out different ways to approach the placement problem. And it would learn through interactions with this environment that we built to learn from the positive placement and also the negative examples and iteratively improve itself. And so one of the main differences between our approach and all prior approaches were this ability to learn from experience, which would enable the model to self improve just like a human expert. That becomes better as they solve more incentives of the problem and harder problems. Our Agent was able to kind of exhibit similar behavior. And that was a very major kind of delta and change between what we had and prior approaches. Yeah.

And I remember with, I mean if you take AlphaGo as an analogy, move 37 was kind of shocking because it was just so different from how a human player would move. Are you seeing something similar happen with floor planning and ship design?

Yeah, we saw these very strange like curved placements. So there are donut sheeps as well. I think humans would tend to make the macros very. So macros are memory components that are larger and they make them very aligned and then they would put logic maybe in the middle and then the wires would connect all of these components. But if you make the shapes curved, you can reduce the wire lanes which can reduce a power consumption and timing violations. But it's just the complexity of making this curve placement was like beyond what humans would have wanted to take on or it felt risky to them. So yeah, these aliens.

What was the moment you realized that this had the potential to transform the entire end to end process, not just the floor planning stage. And when you decided that it had the potential to really be a company.

There were a few like milestones. That first was like showing that it works, that it gets the superhuman results on some blocks. And then later on the chip was actually taped out and it came back and it was actually working, which is a big, a big kind of milestone because it can be really, really costly if for some reason our AI missed something. So those were big moments, the real impact and also on the algorithmic side when we saw this kind of self improvement, improvement properties and the fact that models were, our models were becoming better as they solve more problems like that, that is like very. Because when we get to that point then there is no stopping for the AI. They can see more data points and solve more instances of the chip optimization problem than any single human can ever do. So that's, that's the moment that you're like, yes, so there's a lot more potential here. But Alpha chip was targeting a module, a part, a part of physical design. But chip design as we know there, there are many more like stages to it and many more components. And right now we think that everything from the state of AI and the capability of AI both on LLM side and other like graph optimizations and other approaches and, and the way that we can really scale up these algorithms and enable synthetic data, very large scale distributed computing, everything is ready to tackle the end to end chip design optimization problem. So that's why we are excited about doing the company right now.

Super cool. How do you think about getting the right training data for this market? I think it's, you know, it is kind of the binding constraint to AI in many ways. And is this a market where you can have, you know, enough synthetic data or self play to be able to bootstrap?

We're excited about synthetic data. So obviously there's some open source data, but it's not that interesting or meaningful. And you know, customers are willing to share data with us, but we don't want to train our models on that because we want to keep their data private and siloed from each other. But synthetic data is where we think there's the most promise. Like as Ali and I, we've been working on synthetic data approaches and for LLMs and various code domain tasks across Claude and Gemini and we see a big opportunity to get a scale of data that would go far beyond what any customer could ever share with us. Like orders and orders of magnitude more.

With the TPU program, I guess. At what moment did the chip experts stop doubting you? And as each successive generation came out, what was the progressive impact of what RL was able to drive in those designs?

I mean, I think that our approach with the TPU team is like every week we would show them data over and over again.

Every week?

Yeah, for every week for like a couple years. Yeah. Because I mean, the stakes are very high here. Like if there's something wrong with this layout, like the setup on the TPU team is like this. There are human. The TPU is quite large and complex, so they divide it up into dozens of blocks and then each block is owned by a human or a team of humans. And that's their responsibility to get this block right. If anything goes wrong, like it's their fault. Right. And so they would generate their own layouts in collaboration with commercial tools and then we would show them our layout, some AI generated layout, it would look weird, it would be curved and they would, you know, they would have to say, okay, like I'm picking this AI generated layout over my own layout and I'm taking responsibility for that. I think, yeah, it requires a lot of trust. We have to be better in every single metric for them to choose that. And we saw across each successive generation of TPU that we were being adopted in more and more of the chip block and more of the area and also that we were getting increasingly superhuman performance. We published an alpha chip blog post, I think September last year. We showed this curves can I say.

A word on that? Yeah, on the blog post. And the superhuman performance.

Yeah, I think just across every single generation that we use, which I think there were three generations shown in the blog post, but it actually were used in another generation after that was published. Every single one, we were seeing more of the area and more like increasingly superhuman performance.

Yeah. So basically the delta between alpha chip layouts and the baseline layouts are also growing. Growing, which is like, it's a property of AI and how it scales with data.

It makes sense, right? Alpha chip was trained on more and more TPU blocks. So it gets better.

Why is the company called Recursive?

Because we're recursive or AI for chip design and chip design for AI. So recursive self improvement in terms of the name, the spelling of the name. So the name itself is recursive. So the initials RI Recursive Intelligence are the first two letters of the company's name.

What does recursive self improvement like why, why does that matter for, you know, the broader AI race? Like just say a word on. I think this is, this is a root, no problem. Just say a word on it.

Yeah. So chips are the fuel for AI and scaling laws are driving much of the progress in AI, whether it's on pre training, post training, test time and all that. And so the faster we can make chips that are more custom or better designed for the AIs that we run, the faster we enable this efficient, more efficient kind of compute and that bends the curve for our scaling law. So that means we get to the next generation of AI faster and we can create design better AI's faster. And so this is a type of recursive self improvement that we want to see because our AIs then can help our chips become better and we can design them faster and so on. So that is the recursive self improvement loop that we are going after.

One of the visions that you have is to transform the industry from just Fabless to designless. What does that mean?

So Fabless was basically this concept of it used to be that people thought that no serious chip maker could exist without their own fab. This was obviously before this multi trillion dollar companies like Nvidia, TSMC basically created this whole new world where we could have these incredibly valuable Fabless companies. So we think that there's an opportunity to create incredibly valuable companies that don't need to have their own in house design teams. So many companies, they serve these models at massive scale. They're spending I think $100 billion plus on AI inference alone this year and it's growing rapidly. So companies would benefit from like maybe custom chips to serve their models or train them, but that requires enormous teams of like hundreds or thousands of human experts in house and we don't think that's necessary. So we want to move towards, yeah, designless paradigm.

It's fascinating. We're seeing it come true with Obviously Google and TPUs, Amazon with Trainium, also OpenAI and Broadcom and maybe even Tesla.

Do you think the future is then even today there's almost like model application, co design. Do you think there's going to be chip, application, co design and even individual companies will have multiple chip architectures as a result?

Yes, we think that we are going to see more and more co design across the deep learning or AI stack, from model and data to software and all the way to the chips and we are going to enable that. And co design is really the secret or the path to more efficiency and more performance going forward. But right now we can't have much of co design between chips and models because of this asymmetric kind of design cycle for chips, because it takes so long, it takes so much time to design chips. The cycle is there's a mismatch between how fast we can create the next generation AI methods and how fast we can build the next generation chips. But if we can make our chips much faster, then we can enable this co design and co evolution of workloads, applications and chips altogether.

One of the things that I loved that we talked about early on was that the value you bring isn't just on reducing the cost it takes to actually design a chip or the speed at which you can accelerate the chip design process, though that is transformative in itself. It's actually to unlock completely new potential applications and maybe custom silicon as well alongside it. Can you share a little bit about that? What is the Cambrian explosion that you might expect?

I mean, we think computing is going to be increasingly ubiquitous in every aspect of our lives. So obviously there are things like ar, VR, there's maybe chips in space, even hearing aids. There are experiences that aren't possible unless you can serve them at sufficiently low inference latency or at low enough power. And we think that custom silicon can enable these applications.

Yeah, we think that AI is going to be everywhere, in every kind of experience, every, every aspect of industry and life going forward. And there are chips that would enable these AIs to run and given the scale, we would want them to run as efficient, very efficiently and low power, high speed, all of that. And Custom Silicon is really going to enable that. And we want to enable Custom Silicon for basically any workload that is being run as sufficient scale.

Since we have the ear of whoever's listening on the podcast, who might some of your ideal customers be?

There are a range of customers that we are envisioning in the first phase of the company when we are building ways to accelerate dramatically, accelerate the chip design process. Our customers would be chip designers like Nvidia, AMD, ARM, MediaTek, and all of these companies that are already designing their chips. And all of them would want a faster design cycle and we can help them with this technology, but we don't want to stop there. We want to enable any customer that has a workload or a family of workloads that they want to run, that they're running it at sufficient scale and they would benefit from Custom Silicon. We want to enable them to have that without having teams of hundreds to thousands of chip designers in their companies. I think that kind of goes back to this Cambrian explosion of chips that we can enable.

I'd love to talk about the kind of existing incumbents in chip design. So like I grew up in a family of Cadence. Mom and dad both worked at Cadence, and I think chip design is a duopoly between Synopsys and Cadence. And I think they're adding AI to their product suites. How do you see your company playing out versus the incumbents adding AI?

I think that we see ourselves as like sort of coming from the opposite direction. Like we're a frontier AI lab. We come from Google Brain or Anthropic, those kind of backgrounds. And we want to rethink or reimagine how chip design can be done. And we think that fundamentally, in order to be able to. This isn't like a point solution thing where we want to replace some module one at a time with AI. We want to reimagine and co optimize different stages. And we think that an AI approach is necessary here.

What do people from the chip design industry think of you guys? I would imagine there's a range of from excitement to extreme skepticism. Extreme excitement to extreme skepticism or extreme fear. What do the chip design folks think of you all?

I think a lot of them are excited to work with us as potential customers and we're excited about them too. And a lot of people are excited to come join us and work together. But yeah, of course, what we're doing is very ambitious. So I'm sure many people are skeptical too.

Let's talk about that for A little bit like there was, you know, there's some Internet uproar. You know, micro niche Internet communities love getting spicy and there is like some spiciness around Alpha Chip and like people from media saying like ah, that's not, it's not really real for X, Y, Z reasons. Like what do you think was behind that? And what do you think was valid in that criticism? And what do you think is the important thing that they weren't realizing?

So whenever AI goes to a new field and does something disruptive, we would see reactions like that. And this is not necessarily just, just for in our case it appears almost in every other the bitter lesson. It's the. And we think the bitter lesson is part of it. Like we are true believers in the bitter lesson and it's usually a little challenging to kind of like accept that a whole new technology or new way of looking into problems that people have been, have worked on for decades is just coming and is solving things more end to end. And, and these people like in this case Anna and I and the Alpha Chip team where we were coming from, not from not an EDA background at least we had not worked on that. That's that problem space and now we could come come up with problems that were, with solutions that were being productionized and all that. So, so there's always this kind of a reaction in the beginning. But somehow the true kind of impact of our work was, was much bigger than just the problem that we solved. And I can mention some of those. For example this bringing like doing reinforcement learning. Looking into this graph graph Neural net optimizations were applied to many other problems across chip design including a best paper award in DAC in 2023. The first author of that paper is now, is now our teammate at Recursive and that was for physical design. We saw there its adoption in other stages of chip design like synthesis and so on. The work brought a lot of like attention to AI back in 2020 in, in the chip design field and that was we think it's like one of the most impactful thing as a result of our project right now there are conferences like we actually we are involved in one of the MLM Aided Design that are just, just focused on AI approaches to chip design which back in the days was not a thing. So there are so many positive things happened as a result that we are grateful for that.

I had a funny interaction at ML for Systems Workshop which is another conference that we had started in 2018 and this person was like oh, Alpha Chip Inspired my entire PhD thesis. And I was like, oh, that's wonderful. And I was like, so I'm just curious, how did you first come across the paper? And he was like, oh, because of the controversy. Yeah, very amusing. I think the thing that surprised me a bit was I thought that the people who would be upset by the work would be like physical designers whose jobs were at risk. Right. And those, and you know, those physical designers were skeptical. It required a lot of data for them to be able to change their mind and accept that our method worked. But they actually weren't the people that were upset. It was people who had developed prior methods in the field. And I think in retrospect that makes sense. There's something painful about like you pour like your like time, your, like your human ingenuity, your like soul a bit into these methods and then people come from outside your field and then they're using sort of in some sense simpler approaches. Like they're using things that scale with data and compute and you're outperforming. It's like a little, it's painful totally. But you know, everyone, we can all, you know, build on top of this work.

Yeah, we can all adapt.

Yeah.

Can I ask you the opposite question? So I love the bitter lessness answer. I then have the opposite question which is while the large language models became so good at coding, for example, why won't they just naturally become good at chip design? Why does there need to be a specific chip design? Frontier Lab.

Yeah, so chip design has a lot of components and some of them are language and related to language and code, but actually a big chunk of it is not related to language and code. And it has to do with these different properties and graph structure of, of the, of the chip and all these constraints that appear like these are like really, really hard, large scale combinatorial optimization problems that required a specific, like the custom way of like specific kind of approaches from an AI perspective. And our approach here is to apply the right method to the, to the, to every problem. And LLMs, we love them so much and we're going to use them extensively. We are going to build LLMs that are really useful for some stages of the process, but they're not going to be sufficient for all of this. So we are going to have our own AIs and specific optimizations for different stages. And those are really important because unless we design them, those, those kind of modules that are really fast, really optimized, we can't iterate around them with LLMs or with other methods. So, so Our approach is going to be a very hybrid kind of multifaceted AI that is gone after this. But we are true believers in AGI and bitter lesson and we just want to be on the frontier of that with better chips.

What do you think a world with AGI looks like? It's so hard to imagine the world in 10 years or even five. But what do you imagine for whenever that future is?

I guess AGI, like what does that mean, human level for everything? Like maybe you just have many employees that are just effectively, you know, a bunch of compute powering them. I think eventually, like, you know, maybe we can all take a vacation. But.

It'S just so it's going to be extreme, like cases of productivity for every single human being. And as a result, we can create a lot of economical value at a scale that is not possible today and hopefully distributed at a scale that is not possible today. So I have a very optimistic view about future. About the future when we have AGI. Just lots of good things are going to happen and if you are careful about it, we can distribute the value to all humans. And that's great.

We're going to have data centers in space.

Yes, why not?

And is that going to require actually, is that going to require custom silicon? Because it's like a different thermal footprint, right?

Yeah. That's great.

Yeah. There are other, there are different requirements in space from temperature, from like kind of resistance of the chip and so on.

Yeah. Even like working on the Pixel phone, they have different corners. So the phone has to be robust to different temperatures, like different voltage settings. And so, yeah, you just have to increase in many more corners, I guess for these chips.

Yeah, that's a good example actually of where the different properties of different heterogeneous compute. You have different chips, different use cases, I think chips in space. Right now the whole data centers in space field is structured around how do you make existing GPUs perform actually in space. But I think if everything for the pre parade for your company works, there's going to be specific compute for space.

Customization for space in this case.

So you've already become incredible talent magnets in just the couple weeks since incorporating the company. We have Jiwoo Yi Chen, Dan Ibrahim. What else are you looking for in terms of talent?

I mean, we're definitely building out on the technical side. So every stage of tip design, we're looking for top talent, but also on the LLM side. So this is like we're, we're a frontier AI lab. So you want people all the way from like pre training, mid training, post training, RL training, you know, experts in evaluations, data and also on the non technical side we want, you know, operations specialists who can help unlock us, like recruiting, like chief of staff, like that kind of thing.

What do you hope to have accomplished in a year?

Oh, we're going to release our first product by a year. But yeah, yes, strong commercial partnerships.

Yeah, strong partnership and we want to have our first product and that timeline that we can offer broadly, not just to our partners, but more broadly to other chip designers.

Anything you're willing to share in terms of what that first product might look like?

Yes, it's going to accelerate the process, it's going to tackle the long poles in chip design and it's going to be more end to end than the products that we have access to today.

Yes.

How do you see the role of a human engineer evolving in a world where the design process is automated? What did they end up doing instead with their time and talent?

I mean I think there's like our company goes through various phases so the answer to that question maybe changes over time. But you know, one way to think about it is like, you know, these engineers can come work with us and help us reimagine the process. So that's one way. Um, the other thing is like it's sort of like as humans start working with like Claude code or cursor, you know, maybe they're not doing as much hands on coding, but they're becoming amplified and becoming much more productive through these AI tools. So something like that, like for example, even just alpha chip, we could generate these like credo optimal curves of like different trade offs. Like every one of these points would have taken humans like weeks to generate, but we could generate many, many of them because it just takes so little time and compute for this model so the human can explore what is it we really want, what trade offs we care about.

So we're going to vibe code chips. That's what I heard.

Maybe that's not quite, but yeah.

So you've worked with some of the greatest of all time, Jeff Dean, Noam Shazir, many, many more, very closely Cochlee and others. What are some of the lessons that you've learned from them or experiences that have really shaped you and how have they shaped your perspective today?

So I think with all of these people that you. And they're extraordinary. So just setting the bar really high. And all of them are also extremely passionate about what they're doing and that's how Anna And I feel about this company like we are so excited to solve this problem no matter what. So that's something that we think is very important to actually be successful. And the other thing about them is just. They're just really nice people.

Yeah.

And that setting, that nice culture, collaborative culture where everyone is heard and everyone can do their best work, that's a very important thing that we want to follow their routes as well.

That's right. I think Jeff, for example, incredible breadth, incredible depth, incredible speed, but also very high integrity. He treats every single person with respect. It doesn't matter if they're president of the universe versus they're an intern or something. It's equal treatment. And I think that's like, we want to embody that too. And I just remember, like, Jeff, he actually gave me, like, a performance review, the corrective feedback or something. He was like, ask for more from other people. And so I wanted to expect more from others, too. I like that.

I like what you shared. I saw all these photos of Jeff at Neurops running with, like, many people, which I think perfectly encapsulates what you just said. He welcomes others in and he, you know, motivates them and spends time with anyone, no matter what.

I remember Kwok also. He was our manager almost for 10 years. He was telling us, like, deep learning makes all of us renaissance people. Like, we can make contributions in many different types of fields. And I think that's true here as well. And he also said, like, have fun in this company. Like, actually enjoy. And I think we think that's important too. We think if everyone's having fun, then we're going to do even better work together.

I love that.

All right, thank you so much. Anne Mazalia. We're so excited for your first year at Recursive, and we're really, really excited about it being the Frontier lab for AI chip design.

Thank you so much. Thank.

You. Sa.