Reinforcement learning for chip design

Bandwidth for Changelog is provided by Fastly. Learn more at fastly. Com we move fast and fix things here at Changelog because of Rollbar. Check them out@rollbar.com and we're hosted on Linode Cloud Servers. Head to linode.com changelog do not underestimate the power of the independent open cloud for developers. Yes, I'm talking about Linode. Linode is our cloud of choice and it's the home of changelog.com what we love most about Linode is their independent and their commitment to open cloud. Open cloud means being unencumbered by outside investment and maximizing value for the community, not shareholders. And that's exactly what Linode represents. No vendor lock in open at every layer. If you want to learn more, head to linode.com open again linode.com open. Welcome to Practical AI, a weekly podcast podcast that makes artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changeaw.com community and follow us on Twitter. We're at Practical AIFN. Okay, take it away guys.

Well, welcome to another episode of Practical. This is Daniel Whitenack. I'm a data scientist with SIL International and I'm joined as always by my co host Chris, who is a principal AI strategist at Lockheed Martin. How are you doing, Chris?

I am doing very well. How's it going today, Daniel?

It's going very well. Staying very busy. As we were talking before the episode, it seems like after the crisis and after being everyone at home right now. So for future listeners, this is still during the COVID 19 crisis. It seems like I'm more busy, more busy work wise now than even before. What about you?

I think it's the same and we're just trying to not be paranoid. We're right in the middle of pollen season and everyone in my family suffers. So you know, you get sore throats and you're coughing and you're like, oh my God, that's not, you know, symptomatic of, of COVID or something. So we're just trying to maintain our calm. Yep. Get through this and. And all is well. Other than that. It is a love here in Atlanta, Georgia.

Yeah, it's a beautiful time to get outside as well. Of course, maintaining distance from others. But we've got something I think pretty interesting to chat about today in the midst of all of the COVID 19 and coronavirus tweets and articles that I've been seeing and reading and all of that stuff. I was able to pick out this one story that seemed really interesting to me that was, was not related to COVID 19 and that was this story about a team at Google who was using reinforcement learning to somehow design chips like, like hardware computing chips. And we were joined today by Anna, Goldie and Azalea. I'm sorry to mess up the name Mir Sa Ni. You can correct me here in a second on that one. Sorry. Azalea, welcome to the show.

Thank you so much for having us.

Yeah, thanks for having us.

Yeah, excited to have you both and excited to chat about this amazing project. It was really interesting when I read it. But before we jump into the project itself, I would love to hear a little bit about both of your backgrounds and how you ended up doing what you're doing now. So maybe we could start with Azalea, could you give us a little bit of information about your background?

Yes. My PhD from Rice University in computer engineering. My work was, or my thesis was focused on co design of hardware software systems for machine learning applications. And then when I joined Google, I joined Google Brain to the residency program. I stayed at Google Brain as a resident for a year and that was the time that they developed passion for the work at the intersection of ML for system. Like how do we develop new machine learning algorithm and deep learning algorithms for system. And ever since then I've been at Google Brain for like almost four years now and enjoy doing research and work on impactful projects.

Awesome. And now you're a senior research scientist at Google Brain, is that correct?

That's correct, yes.

Awesome. Well, thank you for the background. Anna, do you want to give us a little bit of information about your background as well?

Yeah, sure. So I studied computer science and linguistics actually at MIT and I did my Master's basically building a Mandarin speaking dialogue system. I joined Google about eight years ago, Google Research and I've been working mostly before this on natural language processing applications. And about like two and a half, three years ago I started working with Azalea. I actually saw some parallels to natural language processing in some of these systems problems because a lot of them can be formulated as sequential decision making problems. And it's been just wonderful working with Azalea and we have such an awesome team solving basically trying to use machine learning to optimize and automate various problems in computer systems.

So could you give us a little bit more of an idea about this team and how big it is? I know you both work closely together but what's the sort of team like that you're working on?

Actually when we first started it was basically just me and Azalea, and then we've been gradually growing. I think we have something like maybe 8, 10 people on the research side. And then we also partner with chip designers who are building the next generation of TPUSC. Maybe there's about 8, 10 people on that side as well. So it's, you know, pretty substantial effort at this point.

So kind of wondering, you know, I come from a software only background and as we kind of dive in, you know, I'll probably be more comfortable in the reinforcement learning part, but I knew nothing about creating chips and stuff. And I guess if you could just kind of lay out the context of what that means. What does it mean? I've heard the phrase chip placement and I've heard chip floor planning and stuff like that. Could you talk a little bit about the baseline, about what it is you're trying to do and how that is what the context of it from the hardware side, which I know nothing about.

Sure, maybe I can take a stab at that. So basically this is just one of the stages of chip design. There's already been a bunch, basically computer architecture stage and synthesis occurs first. But the problem that we were solving in our research was taking a graph of chip components, which is called a netlist. So it's basically a bunch of SRAMs, which are memory components, macros and standard cells, which are logic gates like nands and nors. All of these are connected by wires. And so it's a graph. And we want to place that graph onto this two dimensional grid such that we minimize various costs like latency of computation, power consumption, wire length, area, while adhering to hard constraints on density and congestion. So that's sort of the core problem that we're trying to solve.

So do I have it right? Like when you say that this is kind of graph structured, you're meaning like there's this component like something physical that has to go on the chip and then there's this other components and they need to be linked by an electrical connection, I guess. Is that like a way of saying. So like the graph is formed of these components and the electrical connections between them. Is that, is that kind of.

Exactly, yeah. There's all these like sort of logic and memory components connected by wires or like. Yeah, electrical connections. And then we physically need to decide where to place them such that we get better performance for that chip.

Gotcha.

And can you talk a little bit about what that Means when you talk about why does physical placement have an impact on performance and what it is about that placement which affects performance.

So one way you could think about it is the timing of computation or the amount of time it takes to compute with this circuit is affected by the lengths of critical paths in this graph, this placed graph. So if the total wire lengths connecting these components is larger, then it's going to tend to be slower, going to consume more power, that sort of thing.

Gotcha. And how big of a graph? Like, how many things are we needing to place and optimize in general?

So it's millions, like millions of standard cells. And then in a chip, there's typically many, many blocks, so hundreds of millions in total of components that you're replacing.

Gotcha. So, Azalea, I'd love to get some context in terms of, like, how has this kind of been this problem of figuring out the placement of all of these components of the graph? How has this been approached in the past and what are the bottlenecks or problems in terms of creating a solution to this?

Well, there are several approaches to this problem in the past. In fact, since 1960s, research in both academic community and industry started on doing the physical design or placement optimization. There are various approaches. For example, there are quantitative approaches. There are approaches based on greedy methods or simulated annealing or hill climbing approaches, genetic algorithms and such. I would say that the way we came in and the way basically deep learning and reinforcement learning is helping us taking a new stab at this problem, is that for the first time we can, like, learn the context of the problem and learn from experience. Meaning we think, unlike all of the previous approaches, what we are doing is training agents that can accumulate experience, and as they are optimizing more chips, they become better and at placing new chips. This is an approach that's different from all the previous existing methods.

Gotcha. And for those who may not be very familiar with reinforcement learning as a technique, before we kind of dive into how you're using it in this, could you take a moment and kind of give listeners, either one of you, whoever would be, or both of you kind of what is reinforcement learning? And why is that in particular a technique which lends itself, but even kind of starting with just the quick run through the fundamentals of what is it if you're not familiar with it?

So basically, it's a way in normal machine learning or supervised learning, you're trying to fit labels to input examples. In this case, you have this additional power. I guess you can take actions in the world, and then you receive feedback from your environment, and then you use that information to try to optimize the parameters of your own policy, which is generating these decisions to do better over time. So basically, it's composed of states, which is sort of the state of the world at a given moment in time. So for us, replacing these chips one at the nodes of this graph, one at a time onto the chip, so the state is kind of what is the placement so far. And then actions are decisions that you make at each point in time, which is for us, like where to place the next node. And then reward is the sort of final key component for reinforcement learning. It's the feedback that we get from our environment. So in our case, after we place all of the nodes, we have approximate signals on wire length, congestion, and now timing, and we use a weighted average of these to tell our policy how well it did. And so it can update itself and generate better placements over time.

So I know a lot of people might have kind of heard about reinforcement learning, maybe with, like, agents that play Atari games or maybe more so in, like, robotics. In those types of scenarios, you have this agent which, you know, may be composed of one or more models, and it's trying to take actions. Like, you know, people like, tend to maybe associate that with like taking actions in the video game or like moving the arm of your robot or something like that. In this case, like the quote unquote game you're you're playing is, is really the placement of these components, right? So you're kind of your agent is placing components and then getting feedback about how well it's placing those components. Is that a good way to put it?

Yeah.

Yeah, that's great.

Yes, exactly.

Okay, great. So in doing this, I'm kind of curious. I don't know if anyone's tried to do this before. I assume maybe not. In terms of reinforcement learning for this problem, how did you come to decide that reinforcement learning might be a good approach in this scenario versus maybe some other methodologies? How did you kind of come to the point where you say, oh, those things that people are doing in robotics or in these games or something else? How did you come to think that those methods, specifically reinforcement learning, might be suitable here?

So before we started this project, we have been working on another project which was doing device placement optimization with reinforcement learning. So that project had to do with taking a computational graph, such as a machine learning, like TensorFlow graph, and mapping it optimally to the hardware devices such as GPUs or TPUs. Such that the runtime or performance of the underlying ML algorithm becomes as fast as possible. That problem was a combinatorial optimization problem and a very complex task and started thinking about how ML and this context ever learning can help doing that optimization problem better than existing ones. And reinforcement learning is really like a natural thing to come in mind if we think about ML. Because this task is not a supervised task, we don't have labels for it. We want to optimize this problem by doing several rounds of exploration exploitations. So we did reinforcement learning for that and we got a lot of interesting, very encouraging results on the device placement tasks. So then we came to a natural next step for us to try. Okay, now what if we tried the same kind of approaches for the chip placement problem, which is a much more complex problem than device placement. So that was the transition for us from devices to chips. But the interesting thing was that chip placement, when we came to it, we realized it's a way like orders of magnitude more complex problem and device placement. So it was very unclear to us in the beginning that we are going to get gains with reinforcement learning for this problem that has been there's so much research on it already, but after some trial and error and several rounds of improving our algorithms, it seems like it actually is helping a lot in this problem as well.

What's up? This is Daniel Whitenack, one of your practical AI co hosts and I hope you're enjoying this episode episode and staying healthy during these crazy times. I'm working on some pretty cool AI stuff here from my home office, but I've also found that I'm having to get a bit creative and be intentional when it comes to honing my AI skills and virtually connecting with the AI community. If you're in a similar situation or you've been inspired by the practical AI we talk about on this show, I want to invite you to a live online AI training event I'm hosting hosting this May called AI Classroom. In AI Classroom I'm going to teach you the practical skills I've learned over the years using the latest open source AI technology. You'll learn AI theory along with practical hands on implementations in both PyTorch and TensorFlow. And after the training you'll be able to understand the latest AI models, implement your own models in code, train computer vision and NLP models, create model inference servers, and experiment with state of the art methods like reinforcement learning. AI Classroom is taking place this May. It'll be taking place live and completely online in a high quality virtual classroom so no travel is required. There'll also be two cohorts with convenient time zones for Eastern and Western hemispheres. Don't miss out. Tickets and more information are available at Datadan I.O. that's Datadan I.O. and Practical AI listeners can use the code PracticalAI10 for 10% off. See you online in AI classroom.

So I am curious, you mentioned a moment ago that there was, you know, like the data itself wasn't labeled lack of labels and you know, that it that reinforcement learning was seemed like a very good technique to lend. I am curious if you had not gone down this route or maybe, you know, you know, not machine learning at all, what some of the other options, whether they be in the realm of machine learning or not, might have been. Just to have a sense of what kind of the technique opportunity cost would have been, you know, how might others have done it had you not gone this down this path?

So we did experiment with some other techniques. They say evolutionary strategies, they tend to be less sample efficient. So it didn't really seem like too promising a path to go farther down. We also experiment with like using supervised learning as a way to basically ground our architecture search the policy that architecture that we were able to achieve generalization with was tuned using a supervised learning objective and then we used that as sort of the encoding stage of our full policy value net and achieve better generalization results.

Yeah, I would love to follow up on a couple of those things. So maybe digging into a couple of those pieces just to break it down for listeners. So when you're talking about this encoding piece and the supervised stage that you did complete, does that have to do with getting the graph structure data into like, into another form, like a sort of embedding or representation that you would use in other things? Could you kind of describe that a little bit more?

Yeah, sure. So I think basically in order to achieve generalization it really, really is about the representations. Like as you said, what is the correct embedding for a given input graph. So basically we created this very large data set of different placements generated by different placement techniques, including reinforcement learning policies, but also like force directed methods, simulated annealing greedy methods. And we used that to try different architectures on the task of predicting the approximate wire lengths and congestion for those placements. And the architectures that were better at this prediction task did a much better job of creating policies that were able to generalize across different chip netlists because they presumably had a better representation.

I am curious, you mentioned a little while ago that the thing that inspired you Guys, to kind of go down this particular path was device placement optimization, I would imagine, and correct me if I'm wrong, I would imagine that this is like a complete, completely different scale in the sense of, you know, working in. In very, very small spaces, I would imagine, compared to the original device placement optimization you were doing, if that's accurate. Did the scale, you know, moving down to such small spaces make a difference or was it fundamentally the same? Did the approach hold up the same as you had experienced in the prior project? Azalea, do you have any thoughts on that one?

Yes. So in both projects, we are still doing reinforcement learning. So the meta approach still remains the same. But like you said, the scale of these two problems are very different. For example, in device placement, we have like a dozen. Our action space is like tens of devices or less or a few devices, few GPU CPUs. But here our action space is the placement or cells of the canvas onto which we are placing the chip. And this canvas can have thousands or even more of locations. So our action space is orders of magnitude larger than the previous problem. At the same time, our input state, which is the graph that we are processing. A chip graph, like Anna mentioned, can have millions of nodes, whereas a computational graph can, could have like tens of thousands or so. So here in this problem, we were dealing with a much more complex state and action space. And to enable RL agents that can optimize this problem, we had to do several changes to the way that we present the environment to the agent. For example, we had to kind of take a hierarchical approach to the way we represent the input graph. So, for example, we grouped certain standard cells, we break down the complexity of the input state to a graph with like thousands of nodes that we were later on placing. And on the representation learning, we had to do a lot more work, because in this problem, not only we were interested in placing one chip, we were also interested in creating agents that become better at placing unseen chips, because that opens new opportunities for chip design optimization if we can quickly given a chip block, can place it, optimize it, and see how well we are doing in it. So this generalization property that we wanted from this problem led us to really heavily focus on representation learning of the graph. And we created a lot of new techniques for creating these generalized representations that we are hoping in future problems and better in other stacks of chip design or other kind of hard ML for combinator optimization that we are dealing with can help us do better in those problems as well.

So I'm really curious I have a follow up from that and as you were talking, I was thinking about how essentially you have all of these different possible arrangements of the graph onto the, the physical canvas, like you said, but also in this problem as you're placing components, there is this like sequential nature to it. And maybe this is where I think it was mentioned earlier that there were kind of even some parallels with natural language processing. And I was wondering how you deal with this situation where you're really not just taking like a one, at least in my understanding, you're not taking a one step approach of like, here's all my components, components and then here's my prediction for the placement of all of those components. You're kind of placing one component and then placing another and then placing another kind of in a more iterative sort of way. So is that the, how do you deal with that sequential nature of this process? And does it involve kind of like sub graphs within the graph and then adding a component to that and kind of taking the last so many components and then trying to figure out how the next component comes in, kind of like placing characters or placing words when you're doing text generation, how does that sort of sequential thing come in?

So the first architecture that we had that worked well, we would actually pass images of the placement so far. And so the model is kind of like a human designer. As they're maybe placing a graph, they could see, you know, what space is left on the canvas and such. And we had a, basically an LSTM model for the policy head basically that sort of stores information about the full like sequence of placement decisions that have been made up to that point. But in the end, actually, I think our current policy head is a deconvolutional neural net that predicts a poly DIS policy decision over this two dimensional grid.

And I'm kind of curious, I'm also following up on the same thing actually, and you may be starting to address that there. But I was kind of curious. You mentioned when you were talking about representation, learning of the graph, that there were some new techniques that you got into and you made the comment, and then I was wondering, is there anything else that you kind of learned to apply to this or did you just cover it right there?

So I think as Aliyah was getting at this graph embeddings that were developed for this project, and I think at a high level the insight there was that for most like sort of graph convolutional neural net type applications, it's really about the features of the Nodes themselves. And so you kind of represent nodes as some kind of average or other aggregation of their neighbors features. But in our case, what really matters is the connections between these nodes because it's about the paths. And so our graph embeddings are much more focused on edge features and kind.

Of diving a little bit more into those embeddings. Again, I'm trying to make connections with maybe things that I've seen or heard about before. I know in like the NLP world with like, you know, these newer language models that are coming out and the word embeddings that they're generating. The thought is like, oh, we're going to train this model or learn this representation based on a one or more tasks, like you know, replacing missing words or something like that. And then you learn this embedding and then kind of apply maybe some new layers onto the network to do a particular task like question answering or whatever it is. Here is, is it similar in that you were talking about how you use some supervised learning to train the. Train the embeddings in my understanding. So you have these certain tasks that are supervised and you learn the graph embeddings and then you were able to apply those in a new scenario. Is that the strategy or do I have that wrong?

Yeah, that's very much right.

Yes, that's correct. I think the way we can describe this is that we trained architectures to capture the representation or encode the embedding of the input by having a supervised model with very easy to produce labels. We call them pseudo labels. Those labels were our proxy cost for the optimization that were very fast and not at all expensive to generate. So the motivation for us to train the architecture this way was if our agent, our policy is to generalize to unseen graphs, it should also have a good understanding of predicting what the actual reward is for a given state, like unless it can. So that's like a prerequisite for generalizing policies to unseen graphics to have an idea how good a current state is. And that's what made us do the supervised approach first, where we predict these pseudo labels for a given graph. And once the architecture is tuned in a way that this prediction task is done at a high accuracy for the test set, then we take that and use that as the encoder part of the policy for further optimization of placement.

Awesome. This is really interesting because we've brought up graph neural networks a couple times on the show, but maybe not in this sort of applied way that we're talking about them here. I was Wondering if you could just before we get too much further, just mention, like, what makes, like, a graph neural network. A graph neural network, instead of just a normal neural network, I guess, and maybe, like, help clarify for people. Because even in this episode in our conversation, we mentioned, like, computational graphics, which people might, that might come to people's mind if they're thinking about TensorFlow. There's like this computational graph in the background. But here, like, for a graph neural network, we're not talking necessarily about the computational graph. What makes the graph neural network a graph neural network? Is it just the input data and this sort of way of representing graph data?

So what makes graph neural nets. Graph neural nets is what, the way they encode information. So in atypical graph neuralness, we are learning representations of the nodes of a graph with respect to the properties of this node and the properties of its neighbor nodes and the neighbor of the neighbors and so on. So graph neural nets have this property that they can encode information about the 1 hop, 2 hop, like k hop adjacency information of a node. And you can also, on top of this adjacency information, like the connectivity graph, you can also add features per node and you can also, in our case, you can add features per edge of the graph. So basically, graph neurons are allowing us to capture all of this information about the graph structure of an input data and generate embeddings of the nodes and edges that kind of relate and can capture those graph structure and graph information.

So, you know, having gone through this, which is fascinating, it's entirely new to me, I'm curious what the results were like. And as, you know, kind of, where did you arrive? What surprised you along the way in the process, you know, what was not what you were expecting to see? And also, how did the larger organization at Google take the results? Is it something that is now becoming kind of standard at Google, or was it just a test or an experiment? Or how did it affect the larger organization in terms of designing chips going forward?

So we have definitely tested this method on chips that Google makes and have gotten superhuman results on a good portion of the, like the complex chips that we tried placing them. But in terms of other questions you asked, I'm not sure if we can answer that at this point.

Okay. But nothing jumped out from a surprise standpoint. Just kind of like, you got something. I was curious.

Yeah, sure. I have something to offer on that. I don't know. Surprising, maybe just exciting. In terms of those generalization results, we would say take a policy and pre Train it on a larger number of chip netlists and then, you know, apply it to a new chip. So sort of surprised and excited us was that a pre trained policy that was fine tuned for say only 12 hours would outperform a policy that was trained from scratch on this netlist for 24 hours or more. So I think it was exciting to us that this new policy architecture generalized so well that it actually does better and it takes less time.

That's pretty amazing.

Yeah. Was, was that having to do? I know when I was looking at the paper you talked about like domain adaptation, which I, I remember we talked about with the OpenAI team and also we've talked about in relation to like robotics and moving hands. So is that key to that sort of generalizability is adapting the domain or the, the environment during this training? If so, did you have to like create a bunch of simulated data for various environment changes and that sort of thing? What was your approach there?

So we actually just used real chip netlists for all of the pre training, but so we stay trained on 20 real chip netlists and then we were able to achieve those results where we have much better and faster results. But we probably could do some kind of data augmentation where we could maybe turn those 20 into many more or, you know, source more netlist in some other way and we would do much better.

And what is your feeling in terms of, you know, how specific this pre trained policy is for the sorts of chips that are included in the set of chip nets, I think you called them, that you use during pre training? How specific do you think that the pre trained version of the policy is for that kind of family of chips or do you think it's generalizable beyond that?

I think it's definitely affected the policy's performance on a new netlist, is definitely affected by the types of netlists that it's trained on in the past. At the same time, it's a pretty general problem. So I think as long as you train on a representative set of netlist, you could do well in a new one.

Gotcha. What are some of the challenges that maybe you faced during this project that you maybe didn't have time to address in the initial version of this project? What are some of the things that you want to explore more going forward?

There's just so many other stages of this process and what's exciting about developing policies that can more quickly generate high quality placements is that we can kind of explore feedback or interactions between say previous stages, like upstream choices, like this choice of sram, basically there's a certain amount of memory that needs to be in this chip, but the choice of how to slice it up into these macros is somewhat arbitrary. And if you can say, try one like slice up the macros a particular way and then see what kind of placement, what level of quality you can get in terms of timing and other properties from that quickly you could do all sorts of explorations upstream.

So I want to follow up on something that you were saying before and just to make sure I understand when you're looking at these different types of chips that you want to apply this to. And going back, I know we had someone talk about some chips from a previous company earlier, but you know, they were talking about basically different types, you know, from gpu, tpu, FPGA and such as that. Do those different architectures dramatically change the problem for you? I know that we were talking about the domain adaptation a moment ago, but I mean, in a practical sense, do you have a substantially different RL approach every time you change out the chips? Given that like I believe like a GPU will have a whole bunch of of things beyond what a TPU might have on it because it's being able to address problems, whereas a TPU is very specific to the matrix multiplication, how does that affect your approach on that? I personally wasn't clear enough on it because of trying to learn this as we go.

Yes, we have tried our method on a bunch of different types of chips that were available inside of Google and also chips that were available open source. And the way we did, our RL approach didn't need to change going from one set of chips to the other. But definitely like Anna was mentioning, the larger if you have a chip that is drastically different from anything you have seen before, then it could affect the performance of the agent. But at the same time, the input space of our problem is very abstract. We don't deal with the specifics of a chip. Rather we are dealing with a generic netlist representation of a chip with these nodes with certain connectivities and the node have different sizes and different shapes and we are placing them optimizing for the cost that we have developed. The problem, if we don't think about what chips they are, is very abstract in a sense that it can really can handle different sorts of input from different chips. And so far we didn't have a chip that was drastically different from our training set that we had to change the RL algorithm for. There's always like modification of the algorithm for improving it overall, but like I said, the input state is pretty standard among different types of chips.

So I'm, I'm kind of curious. The more I, I think about this problem, it seems kind of like we're using an AI method to help design a chip on which AI will hopefully operate or, or be trained or run inference on or, or, or that sort of thing. I'm kind of curious on a more general sense, how you see, you know, AI. As we move into the future and AI development continues to accelerate, are we going to kind of need these sorts of methods more and more because more specialized chips are going to be needed for these types of AI problems moving forward, how do you see kind of AI influencing the hardware that AI runs on? I guess is my question.

So chip design is a really complicated task, and making customized chips is definitely also very complicated. We are witnessing that we are going to need more and more of these customized chips because of various computational demands of especially AI algorithms. Our vision is AI can help the design of these chips. Chips because of its ability to learn and improve over time and its ability to kind of optimize over a very large optimization space. For example, if we look into the chip design process, there are various stages of optimization, from architecture design to logic design to verification and physical design and placement. Each of these stages are very complicated, are combinatorially hard. And so our goal or our vision is AI can help us finding globally optimized solutions across all of these stages, then we are going to have hopefully a lot more performance improvement over what we have right now, where we optimize each stage separately and then just cascade them together. And the reason we think AI can help with it, we mentioned this a couple of times in this conversation, was that AI can improve over time. And this property is something that's very different from what we have seen in any other existing methods. So the policy, the agents can become better, more experienced at doing newer tasks. So if we accumulate this experience over time, then we are dealing with these agents that become much better than any single person or single algorithm that has ever optimized the chip.

Yeah, in that way, it almost seems like so many other areas that we're applying AI techniques to, and that you take it to that superhuman level and just continue on. It makes me wonder as you guys, and I don't know, this may be almost an organizational question to some degree, but I'm curious whether, having pioneered this, you know, being able to apply reinforcement learning to this particular problem, is this something that the two of you are expecting to continue working on for some time, or have you kind of done your experiment and, you know, you got your results and you're going to move on to other problems? If it's the latter, what might those other problems be? Or if it's staying on this, what are you looking to next? What's the next step, whether it be on this problem or doing something else for each of you. Anna, you want to go first?

I think that there are definitely other stages of the chip design process that have a lot of impact. I think getting to your last question a little bit in terms of how can this affect kind of AI for AI chips? The current chip design process takes nearly two years. And so there could be certain types of machine learning architectures that just aren't computationally feasible on today's hardware. But if we could more quickly design chips for them, they might become more viable approaches. But the problem is that, say, chip floor planning is just one of these stages. So if we wanted to say, dramatically accelerate this process, we would have to tackle these other stages, say like architectural exploration or design verification.

Awesome. And you've built one of the building blocks of that process, but you could be exploring some of those other building blocks as well. Is that right?

That's right.

Awesome.

What about you, Azalea?

Yeah, I think I'm in a similar boat. I think I'm very excited about the research on RL and ML for optimization tasks in general. I think chip design is a very critical and important application of optimization, something that's going to enable if you have better chips, we're going to have better next generation AI algorithms as well because chips are key enablers of those algorithms. So I would say both research on RL for optimization and with applications in chip designs is something I'm very excited about and look forward to continue working on.

Awesome. Well, thank you both for taking time to join us. This has been super fascinating and it's been great to dive into some of these subjects like graph neural network and chip design and these things that we haven't talked a lot about on the show. So I really appreciate both of you taking time and joining us for the conversation. It was great to talk.

Thanks for having us.

Thank you so much for having us.

Thank you for listening to this Episode of Practical AI more like this@changelog.com PracticalAI there you'll find our latest as well, well as lists of our most popular episodes and the ones we recommend. If this show has helped you on your AI journey, please leave us a five star review on Apple podcasts, heart us on Spotify, star us on Overcast, and tell a Friend what they're missing out on. Practical AI is hosted by Daniel Whiteneck and Chris Benson. It's produced by me, Jared Santo, and our music is brought to you by the Beat Freak Breakmaster Cylinder. We have awesome sponsors. Please support them. They support us. Thanks again to Fastly, Lynn, Node and Rollbar. If you and your organization could benefit from speaking directly to all the AI practitioners out there, you should sponsor the show. Podcast advertising is one of the most effective ways to spread your message in an authentic way. Plus, you get the added bonus of supporting something you love. That's all for now. We'll talk to you next time.