Phaidra’s Jim Gao on Building the Fourth Industrial Revolution with Reinforcement Learning

A lot of times, like when we talk about AI. Right. Both in the Valley and elsewhere, I think there's a conflation between AI and automation. Right. Like AI can absolutely automate things. There's no doubt about that. Right. Especially like routine things. Right. But I think that honestly undersells the real promise of AI. Right. I think the real promise of AI is what Demis, the CEO of DeepMind, calls like AI creativity. Right. It's the ability to acquire knowledge that did not exist before. Right. And I of course experienced this firsthand. The reason why I'm such a true believer in the technology is because again, I was the expert who helped design the system. But this very AI agent that we created is telling me new things about the system that I didn't know about before. Right. And that's a very, very powerful feel.

Hi and welcome to Training Data. Please welcome Jim gao, founder and CEO of Phaedra. Jim was previously the leader of DeepMind Energy, one of the first and only Alphago style reinforcement learning applications. In the wild, DeepMind Energy used reinforcement learning to manage Google's data centers and drove some staggering metrics, including 40% energy savings. We're excited to ask Jim about reinforcement learning in the industrial world and to learn more from him about what other real world applications are poised to be transformed next by deep reinforcement learning. Thank you so much for joining us. Maybe before we get started, we're going to spend a lot of time today talking about your Deep Mind Energy journey, but maybe can you give everyone one or two sentences on your background and what you're building?

Yeah, of course. So Phaedra is an AI company? Of course. Fundamentally we are an AI automation company. So what we do is we use a type of AI known as reinforcement learning to directly control and operate our customers very large mission critical industrial facilities. So in practice, these AI agents, they act as virtual plan operators, virtual members of the plant operations team.

Let's go back in time and talk about the journey that led to this journey. And I believe that you once sent an email with the subject line, reinforcement learning plus data centers equals awesome. Can you question mark? Yes, awesome. Sorry, sorry, let me say differently. Reinforcement learning plus data centers equals awesome. Can you tell us who did you send that email to? Why did you send that email? What was on your mind at the time? And then of course, what did that lead to?

Yeah, of course. So the reason why there was a question mark is because it was generally an unknown if the combination of reinforcement learning with industrial facilities would actually be awesome. So that was an Email that I had sent to a person named Mustafa Suleiman, who would later become my boss at DeepMind. And really the impetus was something called AlphaGo. So to set the stage properly, I had been experimenting as part of my famed the 20% time at Google with machine learning technologies. And it was actually a very specific course, Introduction to Machine Learning by Andrew Ng on Coursera, that had just come out. This is back in 2013, mind you. I think I was like the second cohort or something. And that class had completely changed my life. I taught myself how to program and just started tinkering around with machine learning on the side. Right. It was very interesting technology.

And your background was mechanical engineering and environmental systems type things.

That's absolutely right. My responsibility at the time was to one, help Google design and operate their very large data centers. And once these very large data centers, which consume enormous amounts of energy were built, we of course shifted our focus to operating these complex industrial systems in the most energy efficient way possible because they use billions of dollars in electricity. So that was kind of the background. I was already tinkering around with machine learning technologies on the side to analyze the enormous amounts of data that Google's data centers were generating. In 2016, AlphaGo came out and I was one of hundreds of millions of people around the world watching. It was like 3am in the Bay Area or something. And I found it absolutely captivating and to the point where I sent an email to Moose describing this idea that if DeepMind could beat the smartest, most intelligent people in the world at complex games like go, then surely we can train these same AI agents to play a very different game that I'm familiar with called let's Optimize the Peewee. The power usage effectiveness of Google's data centers. Right. So that was the context for that email. And I remember internally the way I pitched it to Google's leadership, so specifically Joe Cava, who leads Google's data centers and ors, was I showed a picture of a GO board on one side and a video game controller like an Xbox controller on the other. And I'm like, look, there are objective functions that we're trying to minimize or maximize. There are concrete knobs and levers, so actions that we can control, there are constraints that we have to stay within. And all of this happens within a very measurable environment. I think reinforcement learning and operating large, complex industrial systems are actually one and the same thing. Right. So that was the original kernel of insight, I guess that inspired it all.

And I know Sonia has accused me of going rogue with some of the questions we ask here. I'm going to go ahead and go rogue for a minute.

Minute already. It's been like one minute.

We're going to come back. I don't want to skip the story. This needs a brief diversion. Bear with me. The three things you mentioned that allowed you to see the parallel between reinforcement learning and control systems, or control theory, objective function, actions, constraints. Are those the three key ingredients for where reinforcement learning can be applied to real world systems?

Yes, absolutely. That is 100% how we think of it. Right. You know, the reinforcement learning systems, they need like KPIs to optimize for. They need to know how good or bad an action is. Right. They obviously need things to control and they need to know what are the constraints they have to stay within. So really what we're saying is as long as we can map, you know, the problem we're trying to solve into a reinforcement learning framework, which really from a mathematical perspective, what we're saying is we're solving a constraint optimization problem. Right. If you can map the constraint optimization problem, if you can define it and map it to the underlying data, then it should be able to be solved using reinforcement learning. So that's very much the lens through which we look at things at Phaedra as well. And to take it one step further, we often talk about how reinforcement learning and controls and optimization are two wildly different fields historically that have somehow independently converged to the same area. Right. Like they're two very similar concepts. Well, we've been calling them by different names this whole time. So you've had like almost these independent evolutions, Right. Different ways of tackling the same problem. And Phaedra is really kind of the intersection of both of these.

Okay, let's get you back onto the story. So you sent the email. Sorry for the diversion. So you sent the email to Mustafa and then what happened?

Yeah, so we sent the email to Mustafa. Two weeks later, Moose had actually flown out to Mountain View, right, where I was working at the time on Google campus with a team of DeepMind folks. And we actually started mapping out exactly like how reinforcement learning could be used to control and optimize Google's data center. So that actually kicked off the original partnership between Google and DeepMind around the application reinforcement learning for the data center work. It was very, very fascinating. But you know, most importantly, it's actually also how I met one of my two other co founders. Right. So Veda was one of the original engineers on the AlphaGo project. So he had gone to go to, you know, he went to Seoul, South Korea, right? And you know, he actually got to meet Lisa do and Larry Page and all, you know, all this fun stuff. And after AlphaGo he came back to the cities and you know, or rather to the uk and he was wondering, well, what is my next big thing going to be, right? And I managed to convince Veda like, hey, what if we applied self Learning frameworks like AlphaGo to control and optimize Google's data center? So that's actually how I started working with my co founder Veda.

Did people think he was going to work or was it like, this is a crazy moonshot, let's just try. But who knows?

I didn't even know if it was going to work. Conceptually it made sense in my mind, right? I'm like, hey, it's a. Operating a data center is just a different game to play, right? And there's all kinds of different games in the industrial world, right? Maybe the game is maximize energy efficiency. Maybe the game is minimize water consumption. Maybe the game is maximize the yield of a factory, right? But there's all these games that we're constantly playing, right? So in my mind it made sense. But to answer your question directly, no, I had no idea if it was going to work. I still vividly remember to this day when we turn on the AI system and we watch the energy just drop. And it was so surprising for two reasons. Number one, well, we had designed the system. I played a role in designing that very mechanical system that the AI was now controlling, optimizing. So in theory, I'm literally supposed to be the subject matter expert who knows everything about these systems. But the AI is teaching me things that I didn't know about the system I helped design in the first place. And two, the moves that the AI was making were just very counterintuitive, right? Like when we looked at the decisions that were coming out, you know, we looked the plan operators and I, we were sitting in a giant cornfield in Iowa, right? Where like Google likes to put its data centers. And we were looking at the decisions and we thought to ourselves, there's no way this is right. Like this AI sucks. I learned the wrong thing. But we're here anyway, so let's try what the AI is saying. And we tried it and it worked and we saw the energy plummet. So I think that was kind of when I became a believer in this technology, right? That fundamentally this technology is creative. It helps us discover new knowledge that didn't exist before.

From raw data, Was there a performance trade off or was this just straight up Pareto gain like performance held and.

That'S the same question. No, it was. It respected exactly the same constraints as the plant operators and engineers had already put in place. So this is pure gain, right? Respecting exactly the same temperature profiles, exactly the same constraints around how quickly you can turn on and off a chiller, minimum pump, VFD speeds, all that sort of stuff. So this is pure optimization, pure gain, which I think is one of those crazy things we don't really expect. Like usually when you think about energy efficiency, for example, right. Like in the world that I come from, people usually think about expensive capex, like oh, we gotta rip out the chillers, we gotta buy a bunch of new chillers from Johnson Controls and Trains or whatever and then we have to install them. So they're like hardware efficiency gains, Right. But you don't really think about like pure software, like data driven efficiency gains, right. And I think that's part of what was surprising for us.

Can you walk us through the before and after? Maybe before what you all implemented was this industrial control systems, was this manual plant operators turning knobs. How did this work before and then after?

Yeah, it's a great question. So let me set the stage for, you know, for, for folks who are not as familiar with like large industrial facilities, right? So they're very modern. Industrial facilities are very, very complex, right. There's all kinds of machines that people are operating and controlling, right? So you know, I often, you know, tell folks to do like a simple thought experiment. So imagine you have just 10 machines you're controlling. So say they're like pumps, right? And each one of those machines has 10 possible set point values. So 10 modes associated with it. So think something like 10% pump speed, 20% pump speed, 30% pump speed, et cetera, right? Then in this very simple toy example, you have 10 raised to the 10 or 10 billion different permutations for how you can operate your toy system, right? So then the question becomes, well, at any given point, what is the most optimal way of operating your toy system? And by the way, these are dynamic systems, right? So the IT load is changing, the weather is fluctuating, right? You know, the people operating these systems are changing, the pipes are corroding, the heat changes are fouling, right? So the point is these are very complex dynamic systems. Real world systems have a lot more than 10 machines and each machine has a lot more than 10 set points, right? So you can start seeing why technologies like AlphaGo, which manage to navigate MX complexity are helpful over here. It also helps explain why there's often so much room for optimization in the first place. Because there's so much complexity. Right. Like if you think about the total action space. Right. Like all the possible actions within a modern data center, for example. Right. Because of risk averseness, but also because of hard coded rules and heuristics. Right. We've only ever explored like 0.00001% of all the possible ways that you could operate that system. So then the question becomes what is in this 99.9999% of the action space we've never explored? Surely there are more optimal ways of operating the system than what we've done historically. So it's kind of an intuitive explanation hopefully of why there can be such large efficiency improvements in the first place that are undiscovered. And the way that we operate these facilities is constrained by a mixture of hard coded controls logic. Right? So don't get me wrong, these are automated systems today already, right. They're just not optimal, you know, automated intelligently, I would argue. Right. And you know, there, there is a healthy mixture of human intuition as well, right. Where we have people like myself or plan operators who are constantly monitoring the system, who are like nudging the system, adjusting things or setting, adjusting the rules for that system, the constraints that the system has to operate within. But fundamentally human intuition plus hard coded controls logic is still limited when you talk about this degree of complexity. Right? Yeah.

Can you talk to us about the key results? So you saw the energy levels drop immediately, but what results were you able to drive for Google initially?

So there's two types of results for Google. In particular, there was a results from the pilot. So in 2016 we, we released, we announced like the results of the pilot. Right. Now the pilot was done on a couple of data centers, but fundamentally it was not an autonomous control system. Right. So what I mean by this is it was the AI generating recommendations, right. Which for, for human experts like myself to manually review and implement. And of course, you know, we didn't want to jump straight to taking our hands off the steering wheel because it's a new novel technology. But also no one knew at the time is it even possible to use AI from the cloud to control big ass infrastructure. So step number one was do the pilot. The AI generated recommendations. That's where we saw really steep 40% energy savings. Now that experience taught us, hey, we think there's something real over here. We should actually just let the AI control things directly to get the value automatically. And also, quite frankly, the plan operators were getting tired of checking their email like every 15 minutes, waiting for the AI to tell them what to do or the manual implement things. They had better things to do. So we actually decided, and rather Ors and Joe decided like, hey, it's time to go to a fully automated system, right? This was total uncharted territory at that point. Forget about can AI control things? We didn't even know. Is it possible to control machines from the cloud? Like, huge industrial infrastructure on the cloud, because to our knowledge no one had done it before, right?

Is it fair to assume that a lot of the hardware, a lot of those machines are things that Google built from scratch, or does Google use a decent amount of commercially available data center stuff?

It's a mixture of both, right? So obviously, like Google, it does a lot of things in house, right? But it doesn't manufacture like chillers and that sort of hardware. So Google does buy like off the shelf hardware. But, you know, there's a lot of like, modifications on Google, specific things, you know, that, that we did, right? For example, like, you know, programming some of our own PLCs or, you know, making modifications to the building management system. So like the, the software control layer looked quite different, right? That was done in house. But, you know, I still remember, like, very vividly actually to this day, Veda and I, we were standing in a large, like 90 megawatt data center, right? So it was a fairly large data center. And Veda is like typing away in his MacBook, right? He submits the PR, he gets merged, and all of a sudden this huge, honking, huge chiller that is the size of a bus that we're standing right next to roars to life. And as it's coming to life, right, like the ground is shaking vigorously and we're like, oh my God, like, with a few keystrokes on its MacBook, like we just turned on this enormous chiller and that was like the very first data point to us. Like, yes, it is possible to control things from the cloud. So now the next question is, how do we control things intelligently from the cloud? Right? You know, where all the compute resources live.

What were your biggest takeaways from that experience? You mentioned the creativity of the machine. Any other big takeaways or learnings?

So the creativity is absolutely a big one. I think, just to elaborate on that briefly, a lot of times when we talk about AI, both in the Valley and elsewhere, I think there's a conflation Between AI and automation, AI can absolutely automate things, there's no doubt about that, especially routine things. But I think that honestly undersells the real promise of AI. I think the real promise of AI is what Dem is the CEO of, of DeepMind calls, you know, like, AI creativity, right? It's the ability to acquire knowledge that did not exist before, right? And I of course, experienced this firsthand. The reason why I'm such a true believer in the technology is because again, I was the expert who helped design the system. But this very AI agent that we created is telling me new things about the system that I didn't know about before, right? And that's a very, very powerful feeling. It's kind of like when, you know, if you think back to AlphaGo, right? Like Lee Sedol was the best in his field at go. He was the world champion for a decade, right? He was at the top and his ELO rating was just something outrageous. It was like 2800 or something. It was outrageously high and. But it had flat, you know, flatline, right? So for a full decade his ELO rating was the same and there was no one to challenge him because he was already at the top. So once he hit the top, he just kind of plateaued. And then after AlphaGo happened and he actually got to play against AlphaGo privately a few more times because DeepMind had let him continue interacting with the system. What happened? For the first time in a decade, his ELO rating started climbing. This is what I mean when I say that I think the real power of AI is helping us discover knowledge that we didn't necessarily know about before. And where you're going to see the most gain from that. It's not going to be in routine automation, things like call centers or whatever. It's going to be, I think, in very, very complex areas, areas where human intuition is insufficient because of immense complexity, but that is yet underpinned by data. So that's why you're seeing such things like protein folding, for example. That's fucking extraordinary, right? And you know, it's those areas of just like massive permutational complexity underpinned by data. That's where I think we're going to see some of the most interesting companies and products. So that was a rather long tangent. But so one, creativity is something that I learned. The other one lesson that my co founders and I learned is really around, you know, if you want real impact, you got to turn the technology to a product. And this is actually the core reason why we, we Decided to leave de mining train technolog to start Phaedra. Right. Like over and over again we were seeing the technologies that we were helping to develop at DeepMind were just extraordinary, right? I mean they were, they were achieving crazy things like with protein folding. But the problem is, you know, in order for the technology to make the most impact, you have to get into the real world. People have to actually use it. Right. And that fundamentally means we're talking about a product. Turning a technology into a product is like, I mean, you guys would know much better than myself, it's like a hundredfold, a thousandfold more work. And that for us led us to the conclusion that hey, it's time to leave. It's time to actually start a company that creates these intelligent virtual plan operators, these intelligent AI agents as a real product.

Let's talk more about that. For what you're building now, how much of what you Learned at Google DeepMind sort of translates directly into what you're doing now? How much is new? Because the environments are different, the customers are different. There's something different about it.

I think the most important thing that we learned from our Google DeepMind experience is that it's possible. Like this is not a crazy like. And that isn't to like, you know, like downplay like what we learned. Like, it's actually a huge thing. Right. We learned that it is in fact possible to use closed loop learning systems like reinforcement learning to drive very large improvements in complex industrial facilities. It hadn't been done before to our knowledge. Right. And that was a massive proof point. I think the problem though is that the real world is quite diverse. Every single customer is diverse. And especially when you talk about industrial facilities, every industrial facility is a snowflake. Right. So for us, the learnings have just been massive since we left Google and DeepMind. Right. Because every time we onboard a new customer, we're learning something new about how equipment are connected or some product gap that we didn't know about before that needs to be fixed or new ways that data can break. At this point, I can tell you 100 different ways that data associated with mission critical cooling systems can break. Probably not the most interesting party topic for most folks, but I personally find it quite interesting. But yeah, there's certainly been quite a lot of learnings in that regard.

Are the folks you're talking to, are they ready to let the technology take over the system and let the cooling system just start going?

Yeah, I mean, yes and no. Right. And that actually gets back to your earlier question, Pat. As well, right. About the specific learnings from Google, I mean when I look back, I think what we helped pioneer at Google and DeepMind could only have been done at a company like Google. The reason why I say that is because Google is a very forward leaning company. But also one of the things I've learned is that Google is absolutely an anomaly when it comes to how much data it has and the pristine quality of the data and the ease of access of the data, right? Like Google is fundamentally a data analytics company, right? And you know, assets should invest in all this like infrastructure in high quality, high availability data on which you can do things like real time intelligence applications like what we were doing and there are many other examples of this within Google and DeepMind. Having left the nest, one of our rude awakenings was, you know, Google is definitely an anomaly. And I mean, gosh, everyone is in various stages of their AI journey, right? Like Google. Certainly on one extreme we have customers who've encountered where like, you know, forget about real time intelligence. They're like, they're not capturing the data in the first place, right? Or you know, they may be sensorizing. In the industries we work with like pharmaceuticals and district cooling and especially data centers, almost always the customer is sensorized, right? Because these are billion dollar facilities, of course it makes sense to throw a million dollars worth of sensors on it. But that doesn't, just because you censored doesn't mean that you're storing the data, right? A lot of customers of ours aren't necessarily storing the data beyond like 90 days or six months or a year or whatever, right? And you know, and they might cite some reasons like, well, it's costly to store the data, right? Or like, well, we're not, the more commonly we're not using the data for anything, which is a true statement, right? A lot of our industrial customers, they aren't using the data, right? It's more like a forensics thing where if something goes wrong, then we go back and we look at the logs to see what happened, right?

Yeah.

And then, you know, so if we think about it like Maslow's hierarchy of data needs or something, right? You got your sensorization, you got your storage, then you have to invest in making sure that the data is cleaned, right. There's a lot of effort, as we all know here, you know, around making sure the data is actually cleaned and usable, right? And you know, that requires you to know what bad data looks like, what good data looks like and how to convert bad data into good data. So it's Actually useful. And then once you have clean data, you also need to make it accessible in a streaming and like batch historical manner. Right. So there's different gradients, I guess, is what I'm trying to say of AI readiness. The customers whom we work with are all over this, are all over the spectrum. But Phaedra, today is at the point where we are autonomously controlling data centers for our customers.

So I was going to ask you if the basic workflow or the basic loop is data goes in, which is a lot of what you just talked about, getting the data into the system. Step one, step two, decision is made. Step three, action is taken as a result of a decision that was made. Step four, action is evaluated against the objective function of the system.

Yeah.

And then the loop continues. So the front end of that process, which is data goes in. Sounds like there's a lot of work to get some real world data ready to go.

We call it the AI readiness journey. Right. So like if you think about our work with customers, like there is a chunk of upfront work where it's just like, hey, we're going to get your facility, we're going to get you and your facility AI ready.

How about on the action is taken piece of that. Are the systems ready to be controlled by some sort of autonomous system or is there work that needs to happen there too?

Yeah, it's a really good question. Yes and no. Right. And I'll elaborate on what I mean by that. Right. Control systems today were designed in the 1980s. So was I. Well, me too, for that matter.

There we go.

But you know what I mean by that is that was the 70s and 80s was the third industrial revolution. So with that, you know, was the shift from analog to digital and the advent of the first automation systems. Right. In order to automate, you fundamentally first have to sensorize. But these are simple automation systems. Right. The fourth Industrial revolution, right. You know, is, you know, we're biased, but Phaedra, you know, we think the fourth Industrial revolution means intelligent infrastructure, right? Infrastructure that can operate itself and fundamentally get better over time at doing so. Self improving infrastructure. Right. But right now we're shoehorning intelligence into systems from the third Industrial revolution, right. So they certainly weren't designed for this. But what we do instead is most importantly we ride on top of the existing control system. So there is a hard coded layer of rules and heuristics, so millions of lines of if then statements programmed into what we would typically call the bms, the Building Management System or a SCADA system. Right. That defines how the facility should operate. The problem with hard coded systems is that because they're hard coded, they operate the same way today as they did yesterday or a year ago or five years ago, more like 10 years ago. Because people don't very frequently go into the backend programming. Right. To update that controls logic. Now what Phaedra does is we insert a new cloud intelligence layer at the very top of the control stack. So we're not, we don't introduce any hardware, we don't introduce any new sensorization. Right. We actually ride on top of the existing control stack. That's really, really critical. Right. You can think of it as a general in the battlefield. The general. Right. Has a global view of everything that's happening across the system. Right. And it's issuing command signals right. To the troops on the ground for actual execution. So the AI is looking, in our case at 10,000 trans a minute in real time. Right. And it's issuing decisions like which pumps to turn on or what their pump speeds should be. Right. To the local BMS system and or the PLCs for automatic implementation and execution. So that's why I said it's a mixture of yes and no. Were they designed for this in the first place? No. Right. There is a lot of work that we have to do with our customers to be able to accept this type of external intelligence. There's a lot of work that we do in defining the safety nets and guardrails to ensure that the AI can't do bad things to the customer system. But fundamentally we are still riding on top of the existing controls architecture. And to be clear, we always want to do that. You don't want AI controlling things like how fast a valve opens and shuts. That's a terrible application of AI. Hard coded rows and heuristics will do great there. So if you were to look at the, you know, like the overall system, like 90% of it is fine with just, you know, hard coded rows and heuristics because it's like granular controls logic that doesn't need non deterministic crazy powered intelligence behind it. Right. But it's the higher level thinking and reasoning, that's where you want the AI. It's the global optimization.

Have you seen any of your customers at Phaedra kind of get the DeepMind order of magnitude results?

So I'm glad you asked. So the one of the things are really excited about is actually, just actually literally this week. Earlier this week, Merck Pharmaceuticals became our first public customer. So we're pretty proud about that. We've been actually been working with them for two years now. They've been using Phaedra for over two years, the full like autonomous AI system to control a massive 500 acre vaccine manufacturing facility in Pennsylvania. This is the definition of mission critical complex, right? They've got 62,000 tons of cooling. So they've got four very large shuttle plants interconnected with each other across 500 miles of manufacturing space. Hundreds of machines interacting with each other. This is where the AI really shines. And yeah, the results that we saw with them were quite strong, right? Like, you know, I think Merck actually just shared some data at a conference we were at, you know, which showed 16% energy savings when we first, you know, trialed the system at one of their chiller plants. So but I, you know, what I always tell our customers, right, is don't over index on the magnitude of the energy savings initially. Like we honestly have no idea what the energy savings are going to be or the reliability improvements are going to be ahead of time. Because these are non deterministic systems. And by definition, if I could tell you what things you're not doing in order to get energy savings, why do you need the AI in the first place? But what we do know is that the unique thing about this technology, about Phaedra and about reinforcement learning in particular is, is that it is a closed loop system. It is a self learning system. It can learn because it's able to take actions and it can measure the impact of its actions against its predictions, right? And that means it gets better over time. So maybe we start off at 1% energy savings, maybe we start off at 5%, maybe we start off at 10%, right? But fundamentally it will learn and it will get better over time, right? Not infinitely because there are so our laws of physics, right? But it will get better over time and once it reaches optimal, it will stay at optimal. That's super important because with hard coded rules and heuristics, when you tuned a system as you were commissioning it, so when you're turning it on for the first time, that system today no longer performs the same way that it did 10 years ago when you first commissioned that system because the pipes have corroded and the, the heat exchangers have fouled and your cooling towers have scaled, whatever, right? Or you ripped out equipment. So but the promise of a adaptive self learning system is that it will change with you, right? As your customers are for example now putting in a bunch of H100 and soon H200 GPUs. Right. Well, the system will learn and adapt on the fly with you. Right. So it can stay optimal.

I'd love to transition for a minute beyond industrial controls and get your opinion on. I mean, you were one of the first and maybe one of the only real world applications of reinforcement learning.

Yeah, we're definitely not the only.

Not the only. I'd love to get your thoughts on the not the only. So, I mean, what else are people doing with reinforcement learning in the wild today?

Yeah, absolutely. So unfortunately, my knowledge is very heavily indexed on the Google and DeepMind space because that's where we spent so much time. Right. But even within Google and DeepMind there were other very cool reinforcement learning applications. For example, the team that sat right next to us, they used RL systems to help prolong battery life, for example. So you may notice that your Android phone, the battery life has been increasing. And yes, there are hardware changes associated with that, but there are also intelligent software changes behind the scenes that proactively manage your battery life. There were reinforcement learning systems for, you know, for like YouTube video recommendations, for example. Right. And a whole host of other things. Right. So absolutely there are, you know, reinforcement learning applications in the wild. To your point though, I wouldn't say that there are a whole lot of them. Right. And I think that it's not a coincidence that you tend to see them at more of like the big tech companies where they've already invested in the data infrastructure. Right. So that the underlying infrastructure so that they can benefit from this technology. Right. Like outside of the big tech companies, there are very few applications of like real world reinforcement learning, like in production at least.

Yeah.

And do you think that's because of kind of low applicability? You know, you started this podcast by talking about the necessary ingredients for RL to be a good solution. Do you think it's just there's not that many applications where RL is a good solution or do you think it's just tech readiness?

I think the applications for reinforcement learning are freaking massive. And where Baron Phaedra is one of many examples, and we're just scratching the surface as an industry of what we can do with this technology. Right. Like fundamentally the power of the technology is that it is a self learning system. AlphaGo and its successor, AlphaZero, taught itself to become the best in the world at go chess and shogi. Three vastly different games, same learning framework. Right. And it taught itself. So I think there's a lot of very interesting application areas. Right. I think the data infrastructure is Missing in a lot of them. But just to list off a few, right. I mean, obviously we've already talked about the protein folding, right? But there's an entire untapped field around logistics, right. That is such a gnarly computational challenge, right. When you start looking at operations research. Operations research underlies trillions of dollars worth of industrial like activities, right? Not just industrial, but other source activities, right. Like shipping airplanes, FedEx, like driving routes like these are all applications of operations research. Grid balancing, right. I mean, I think grid balancing is probably the single most important way that AI can fight climate change. I generally believe that is where AI will have the most impact on climate.

If you had to, if you had to guess.

Yeah.

You know, you deploy, the first time you deployed this into a data center at Google, you saw 40% energy savings. If we had just killer AI doing load balancing on the, on the grid, what sort of energy savings do you think we could see?

Oh my gosh. I mean that would be wild. I think it's not so much about the magnitude of the energy savings per se, but rather about the potential cost savings. Because then you could start shifting your loads around, right. To when it's most cost effective to do compute. Or if you had CO2, two signals, you could start scheduling loads around when it's the least carbon intensive to do your non latency sensitive workloads, which I think Google has already been experimenting a bit with. But honestly I think it's really more around the global system level optimization. We have to keep in mind that data centers already are, but increasingly are just massive, massive load banks. Right? Like data centers like they, they were one and a half, 2% of U.S. energy consumption, that's about to increase to 4%, right. Like this year, I think. And then by the end of the decade it's projected to get up to like 9% of the US in, in Ireland, right. Right now Ireland is 22% of Ireland's national energy. Electricity consumption goes to data centers alone. The International Energy Agency predicts that that's going to increase to 37% by the end of the decade. Right. Like just mind boggling numbers. But the point, the reason why I mentioned this is because these are massive load banks on the grid, right? There is an actual opportunity if you could somehow coordinate the data centers together, right. To, to help balance the grid. Right. And that is such a gnarly, gnarly challenge and it is what is holding the energy transition back. Because you know, as more and more renewable energy starts coming onto the grid, right. The supply side becomes increasingly stochastic. Right. We used to have this perfectly deterministic system, at least on the supply side, where, you know, a good operator can call someone who operates a coal fired power plant and say, hey, ramp up or down your power production, right. It's deterministic.

But now ramp up or down the sun.

Yeah, totally. Right. So now as you get more and more renewable penetration, you know, coming onto the grid, right. You have a somewhat non deterministic demand side, right. It's somewhat predictable, but there are definitely spikes and a massively non deterministic supply side. Right. And what is the problem with that? The problem is that, you know, we, because we do not know how much energy we're going to generate, you now have all this wasted excess capacity in reserve. Right. So there is a concept of spinning reserves on the grid where, you know, like there are peaker plants, you know, like giant natural gas turbines, right. That as we speak are just sitting there idling just like your car idles at a stoplight, right. In case we need that power. Right. As a buffer against the uncertainty. And as renewable penetration increases, ironically, the amount of buffering you need also increases. If you look at Germany's failed energy transition, right. They decommissioned their nuclear baseload while ramping up their, their renewable energy penetration. Right. Good motivation on the surface. Although I personally think we need a lot of more nuclear on the grid, but that's a whole another topic. But it ended up backfiring, right. Because Germany actually ended up needing to build more fossil fueled power power plants to buffer against all the renewable energy that was coming onto the grid now. Right. So that's why I think AI for grid balancing, we need it and it's probably the single most impactful thing that AI can do to solve climate change.

Let's talk a bit about some of the limitations of reinforcement learning and also where you see it intersecting with transformers.

Yeah. So I should state that first of all, my co founder Veda is by far the expert on this topic. He knows way, way more than me. You know, I'm just a simple mechanical engineer, right, who, who happened to learn a bit about AI. I think, I think the intersection is really interesting, like very, very, very potentially complementary strengths and weaknesses is how I would describe it. Right. It's certainly not mutually exclusive. Right. Like what I mean by that is. And I was just talking with Veda about this earlier. Right. So Veda will tell you that like, you know, all intelligence systems have certain hallmarks, right. Of, of intelligence, so that we can say they are intelligence they need to deeply understand the world, the environment that you know that, that you know that they're modeling, right. There needs to be some element of memory, right? So like remembering things. And very importantly, there needs to be the ability to plan and reason. Right. Very. Interlinked transformers are clearly quite good at the first one, right? In the sense that they can take in huge amounts of structured and unstructured data to learn quite good models of the world. But it is limited in the sense that these models are primarily through correlation and not causation. That makes it challenging, at least for what Phaedra does. Because, because we work with real world systems, we have to have causality. We have to understand why is the AI doing certain things, why is it not doing other things, how do we force a certain behavior that we know has to exist in our system. So these are mission critical systems. What I'm trying to say, there has to be causality. So that's where the limitation is with reinforcement learning systems. The power of RL based systems is very, you know, much in the planning and reasoning part, right. Where you know, you're able to plan long trajectories of actions and learn really, you know, like intricate policies. Right. I think where it gets really interesting is the intersection, right, where potentially transformer architectures can learn models, right. You know, like value functions. Right. That the, that the, or models of the, of the world that the AI can learn policies against. But without that causality piece, it's going to be quite tricky to cut it over into at least industrial control applications like what Phaedra does.

Should we move into a rapid fire round? What are you most excited about in the world of AI in the next five or ten years?

So in the very near future, I'm excited about just the absolute explosion of AI applications. It feels kind of like Precambrian explosion of sorts where they're like, there's like a primordial soup and all these AI startups and services are all of a sudden springing up. So it's quite exciting. But when I look at where that activity is happening, where that research and that entrepreneurial activity is happening, is very clearly focused on around LLMs and even more specifically around natural language interactions, text based interactions. Right. And that certainly is a large part of the economy. It is very exciting. But in the five to ten year frame, to answer your question, I'm most excited about when we can start getting some of this technology into real world physical applications. It's the intersection of this technology with the real world infrastructure that we Live in big industrial systems, cars, homes, like, you know, physical things. I think that's where we're going to see some really interesting things in the future.

Who do you admire most in the field of AI?

Gosh, a tricky question. I admire a lot of people you.

Worked with, some of the greats. And so it's going to be hard.

Yeah. I mean, of course my mind jumps immediately to a lot of the people whom I've worked with. Right. You know, I admire very much the dmind researchers whom we worked very closely with. I often tell people working at DMind is kind of like being a kid in a candy shop. If you're a technologist like myself, it's like you get to see years in the future, all this cool technology on the forefront, and it just makes your head spin as to, like, all the possible applications of that technology. I admire Moose a lot, my old boss, who has of course, since moved over to Microsoft. I was saying earlier, one of the biggest lessons I learned and my co founders learned at DeepMind is that making a technology like what we did for Google's data centers versus making a product like what we're doing at Phaedra. Totally different things. Wildly, wildly different things. And there are few people as good in the world as Moose taking technologies and turning them into real products. Right. I remember my co founder, Katie and I, you know, we were sitting down, we were grabbing drinks with Moose at like, at some random dive bar in Seattle, right. He happened to be up there. And this is before, you know, OpenAI released ChatGPT 2 and just like ushered in, like a world of craziness, Right. And, you know, he was raving to myself and Katie about the applications of LLMs and like, you know, the, you know, and how powerful these systems are. And we were like, okay, Moose, but, you know, let's tell us, let's tell you about Phaedra. Like, we had no idea what he was talking about, right? But I mean, he was prescient. Like, he saw this ages in advance, Right. You know, of what? The technology. The technology that was being developed and the capabilities. Right. That it would usher in. And then of course, he went off and he started inflection. So I admire him a lot for the ability to turn technology into actual products.

All right, last question. You are building a very ambitious business, very hard business to build, and you've been at it for a while. In the context of the new wave of AI startups, what advice do you have for other founders or would be founders who are trying to build companies.

Here, I mean, I'm not sure I'm even qualified because one, I hope you ask again in one or two years when hopefully Phaedra is wildly successful. We certainly didn't choose the easy path by focusing on world infrastructure. Honestly, my mind gravitates towards more like would be founders, people like my co founders and I who were thinking about leaving to start something new, right? And my advice there is twofold. One, make sure you have co founders. Like, my God, it's so stressful. There's so many things that can go wrong and you're constantly on this emotional rollercoaster of up and downs, having co founders to lean on both for the workload, but also just for the emotional support and mental sanity. So important, right? Advice number two would be the risk is less than you think it is, right? I'm biased, but I think people should take the jump, right? A lot of times when I talk with my former colleagues and other people who are thinking about making the jump, they'll say things like, well, but I've got a nice job over here, they pay me well. I'm on a rising trajectory. But my point to these folks is always like, no matter, you know, like how valuable and successful, you know, you are today in the organization, like, you will only be more valuable and successful for that organization or other organizations or to society in general if you learn new skill sets, right? Like take the plunge, go out, start a company, learn what it's like to turn technologies into products, right? And if that fails, for whatever reason, hopefully it doesn't, right? But if you fail, then the Google's, the Microsoft's, whatever the world, they will only want to hire you back at an even higher premium. So why not take the plunge, right? It's the, the, you know, the biggest, best investment. And obviously much smarter people than me have said this for a really long time, but the best investment you can make in yourself, right, is you, right up leveling yourself, learning new skill sets. That's always the best thing you could do.

Thank you, Jim. This is a fascinating conversation.

Yeah, thank you very much for having me. You guys really enjoyed it. Thank you, Sam.