Material Progress: Developing AI's Scientific Intuition, with Orbital Materials' Jonathan Godwin & Tim Duignan

Finding efficient ways of keeping track of the important information and losing the unimportant information is just a central problem in a lot of physical modeling. And I think AI machine learning algorithms are just very good at doing that.

The thing that completely blew my mind was training on small inorganic crystals, like 20-atom systems, to then simulate a protein through out-of-the-box generalization. That's telling you that we're learning something really fundamental. At that small scale, which I don't think anyone had ever expected.

The hope is that if you can simulate a ton of things, you start to make important and new discoveries and find new things out just by looking at them, right? Which is very exciting.

By the time we get to making a decision about what to make, we've answered 90% of the questions that we need to in order to feel confident that we're going to have that sort of material.

The real key challenge there had been these potassium ion channels, which no one has really been able to fully understand, unfortunately, using experimental techniques or traditional computation. And in fact, we don't even know some of the most basic questions about it.

Hello and welcome back to The Cognitive Revolution.

Today, my guests are Jonathan Godwin, founder and CEO of Orbital Materials, which is pioneering the application of AI to materials science, and Tim Dignan, who was previously here to discuss his work on the simulation of electrolyte solutions. And who's since joined Orbital Materials as a researcher. Material science underpins virtually every aspect of modern life, from the semiconductors that power our devices to the batteries and solar panels driving the clean energy transition. Advances in materials have been at the heart of human progress for the last century, at least. The challenge has been that discovering and developing new materials has always been painstakingly slow, Traditionally relying on trial and error and scientists' hard-won intuitions developed over decades. And more recently, with the shift to computer simulation, still requiring huge computing power to simulate even small molecular systems for short time intervals. Orbital Materials aims to dramatically accelerate this process with, of course, AI. Their immediate focus is on developing novel materials for data centers, both to improve efficiency and to capture carbon emissions, but their technical breakthroughs could unlock advances across clean energy, electronics, medicine, and beyond. Their technical approach is really quite fascinating. Using an architecture called message-passing neural networks, which are trained on small crystal structures and which, because they don't use positional embeddings like large language models do, are capable of scaling up indefinitely with computing power, they can design new materials with specific target properties via a diffusion process, and also predict the forces between atoms orders of magnitude faster than numerical methods can, which allows them to simulate larger systems for longer. They recently demonstrated the power of this approach by simulating a potassium ion channel, a critical protein that controls electrical signaling in our cells by selectively allowing potassium ions to pass through cell membranes. Despite its importance in everything from heartbeats to brain functions, fundamental questions about how this channel works have remained unanswered for decades. And while Tim's recent work still needs to be experimentally confirmed by the broader research community, his simulations were able to show a level of detail in the mechanism that was never before seen, and which does help explain previously inexplicable data. The implications for biology and medicine are significant, but perhaps more important still, this work illustrates a critical phenomenon that we are seeing time and again as AI is applied to the different branches of science. Namely, that neural networks seem to have the ability to develop a sort of intuitive physics in virtually any problem space. Just as humans can catch a ball without explicitly calculating its trajectory, these AI systems are developing efficient shortcuts for predicting complex physical phenomena— whether that's material properties, protein folding and interactions, weather forecasts, single-cell transcriptomes, or even the evolution of human brain states. For me, this is the clearest reason to believe that superhuman intelligence is not just possible, but increasingly likely. For human scientists, meanwhile, this means a shift away from hypothesis generation and toward more validation and implementation work. And while that might mean lower job satisfaction, the potential to dramatically accelerate scientific progress and more effectively address critical global challenges, for me, makes it a worthy trade-off. And again reminds us that we might soon need to look beyond our work for meaning. As always, if you're finding meaning in the show, we'd appreciate it if you'd share it with friends. You can also write us a review on Apple or Spotify, and we love to read your comments on YouTube. We value your feedback and suggestions too, and encourage you to leave them either via our website, cognitiveevolution.ai, or you can always DM me on your favorite social network. For now, I hope you enjoy this look at how AI is transforming materials science, and by extension, how it might fundamentally reshape and accelerate scientific progress in general, with Jonathan Godwin and Tim Dagnan of Orbital Materials.

Jonathan Godwin, founder and CEO of Orbital Materials, and returning guest and now researcher at Orbital Materials, Tim Dagnan, welcome to The Cognitive Revolution.

Thanks for having me.

I'm excited for this, guys. I'm on quite a journey to try to figure out what is going on at the intersection of AI and science. And obviously there's a lot of different sciences and a lot of different intersections and a lot going on. I feel like life right now for me is one big crash course. You guys are advancing the frontier when it comes to material science and the use of AI in material science. And I'm excited to get into all that. I thought maybe just for starters, because Listeners to this feed are obviously plugged into the AI moment, but probably most don't have much background in material science. Maybe just give us the basic foundational pitch for materials. Like, why do they matter so much? Maybe a couple highlights from recent times where an advance in the materials directly led to a change in the way we live, just to get the juices flowing and get people motivated to understand all this stuff.

I mean, the most obvious one for people who follow AI is semiconductors. You have this wafer, the silicon wafer, and you etch a bunch of stuff, your circuits onto this silicon wafer for your chip. But of course, those etches, they're incredibly, incredibly thin. They're incredibly, incredibly small. And if a piece of dust or some debris gets stuck in that etch, Then your whole chip fails. So you've got to find materials that don't conduct electricity because conducting electricity across the etches in your wafer will screw up your chip. So it can't conduct electricity, but it's got to be small enough to fit in between all of the different bits in your wafer in order to protect your chip from any sort of debris or any sort of pollution that comes on. So these things are called low-K materials, and without significant advances, improvements in the performance of these materials year on year, you don't get more transistors on your chip and you don't get AI. So material science, it underpins literally everything in our lives. The entirety of the Silicon Valley revolution that's given rise to, you know, the tech industry in the West is based upon material science. You know, material science is the original tech startup. And we've just kind of forgotten that these companies built a lot of the industry in America, in the UK where I'm from, but all around the world. And I don't think the 21st century is going to be any different from the 20th century. The 20th century was the century kind of material science. 21st century is going to be even more so. We're going to have more chips. AI is even more important. We're going to go to Mars. We're going to have to have incredible spaceships built out of new materials to do that. We're going to have HoloLenses in virtual reality. We're going to have new optical materials to do all of that. So I think material science is the fundamental missing piece in achieving the science fiction future. And AI as our most powerful tool is certainly a thing that we want to be doing to achieve that. So that was just one example, I think, of all the things, but Tim, you've been working on materials for longer than me, so you must see many more.

No, I think that's an excellent one. And you don't think about it a lot, but the Ideas Factory about Bell Labs is fascinating about the birth of the transistor. And it's just all material science, right? Trying different elements. But the other big one obviously is climate change, right, where essentially all of the tools we need to fight climate change are materials science problems, broadly defined. And the most successful tools we've built so far have come out of materials science. So batteries and solar panels leading to massive reductions in CO2 both came from breakthroughs in materials science. It's so universal that I often like to play the inverse problem of name a problem. And then I can show you how it's related to material science in one way or another. It's just at the foundation of so many things.

I'm excited. So maybe a little bit of the status quo would also be helpful in terms of how material science advances today. We've done versions of this for biology, for example, and, you know, have contrasted like somebody sitting there with a pipette and what that throughput can look like. And obviously some of that is starting to get automated in the physical world. But then there's starting to move more and more of that into simulation and even into AI models. You guys are basically doing the same thing, as I understand it. I saw the motto on the website, engineer in silico, materialise in reality. But for folks who don't know, before we get to that engineering in silico, what is the sort of life of a material scientist look like today? Obviously, it's such a big field, there's no single answer to that, but Just a little flavor for sort of what it takes and what the experimental cycle looks like to make advances pre-AI.

Sure.

So for an experimental material science scientist, right, there's still a huge amount of experimental and trial and error and guesswork involved, right? So if you talk to an average experimentalist trying to make a new material, you know, they'll have hypotheses, they'll have ideas about what's going on and what they want to do. But there's a huge amount of luck involved, trial and error, and intuition as well, like built up over years of what's going to work and what's not. I like to make the analogy to bridge building 100 years ago where we didn't have highly accurate models of what bridges were going to be stable and what weren't. And so there was a lot of guesswork and intuition about how you would build these bridges and things built on experience. And they could build incredible things, obviously, but it wasn't nearly as efficient. They couldn't push things to the limit and they couldn't do it nearly as quickly as we can today. Where to design a modern bridge, you would plug it into some kind of computational modeling software and every kind of bolt and beam, you would know the exact forces and stresses and tensors. You would know the limits of that material and you could design everything really optimally. So if you look back at like Brooklyn Bridge, for instance, massively overdesigned versus what it needs to be because they needed to make it safe so it wouldn't fall over. And you did used to get new bridges falling down. There's a famous one where it hit a resonance because they hadn't tested it computationally and it collapsed after they built it, right? Whereas today, we model these bridges incredibly carefully, we know before we build them exactly what they're going to do. I think we want to transition materials science to that kind of world where we know in advance, if we can make this material, we're very confident it will have the properties we want, and it will do the things we want. We also want to be able to answer questions much more quickly that materials scientists have. So they might say, oh, this isn't working for some reason, or I'm not getting the yield I expected, or the performance is bad. We want to be able to go to the computer and say, Okay, understand this from the ground up and say, okay, here's exactly what's going wrong. And here's some suggestions on how to fix it. And here's what's actually happening at the molecular scale is a big problem often where we can't see what's going on.

So where are we on the path to that dream? And that's maybe a great opportunity to introduce the company and kind of tell a little bit of the founding story if you want and give the big vision.

Yeah, so Orbital came about, I was working at at DeepMind and leading a team building large-scale AI models for material science. And, you know, I thought that it was going to take us a decade to reach the level of performance we're able to reach in 2. The speed at which this field is changing is exactly like every other area of AI. It's just changing and developing at unbelievable pace. And new capabilities that we had dreamed about, now reality. Some of the things that Tim, I'm sure, is going to talk about in more detail later on, they just weren't possible on the world's best supercomputers just 10 years ago, and we're now doing them on our laptops. And so the goal of our company was to take and continue to develop those models and those techniques, but combine that with a wet lab, with a team of extraordinary experimental material scientists to translate those AI capabilities into real materials. And we felt that by creating a team that had both of those skills, we were going to be able to move faster than anybody else was and bring those materials to market and commercialize them better. And we're going to ground the AI that we develop in the real problems and bottlenecks associated with making breakthroughs in material science. And that team wasn't going to exist at DeepMind. It wasn't going to exist at Google. Those are AI companies. It wasn't going to come out of a traditional chemistry or material science company. They've been saying no to computational tools for the last 25 years. They're not going to suddenly turn around and be building frontier tech. So it had to be a new company if we wanted to achieve that goal. So I think we're so much further along than we were just a couple of years ago. The things that we're doing now are completely different. I think it's only going to be a matter of a couple more years where we're achieving this dream of being a computer to design materials. Just get a GUI in front of you with an LLM copilot and you'll get to a new battery in a couple of iterations just on your laptop. and have a real high confidence that when you go and make that, it's going to go and have those properties. And all the things that I spoke about right at the beginning, you know, spaceships, better chips, better semiconductors, more GPUs, more AGI, solving climate change, you know, all of that is going to happen through the technology that we're building.

One thing you said that caught my ear that I want to dig into for a minute is this idea of Simulation. So it seems like we have kind of 3 modalities for pushing things forward. The original is hands-on with actual physical materials, doing stuff, measuring properties, and ultimately everything still has to ground out to the ability to do that successfully to achieve the results we want to achieve. And then there's been this sort of middle time of simulation, and you said that Some of these things like we couldn't possibly simulate this even on the world's greatest supercomputer. And now we've got this third wave, which is the AI wave. And this was a big theme of the conversation I had with Tim last time when we were talking about the crystallization discovery, was basically that what distinguishes the AI era for this sort of work from the simulation era, maybe above all, is it runs orders of magnitude faster. So I've been really chewing on that quite a bit, and I'm coming to a worldview that basically amounts to a belief that today's AI architectures seem to be capable of learning what I might describe as an intuitive physics in basically any space that you might want to have them learn that. We obviously have an intuitive physics where we can play sports and react to things in sub-second time. Which, if we were actually using our feeble brains to do all the calculations and simulate step by step, we would never be able to catch the ball. But we're quickly sort of grokking the situation and not calculating in the way a Wolfram would suggest that we might have to. We seem to be taking a sort of intuitive shortcut.

Right.

And it seems like that is popping up in all of these critical domains, whether it's protein folding or material science or solving the wave equation for a bunch of atoms in a configuration or whatever. Is that how you see it too? Would you edit my picture of that? Because that seems like a pretty foundational update to my worldview that I kind of want to go shout from the rooftops, but I want to make sure I have it right before I repeat it too often.

I think that's right. And I don't think we fully understand why yet, or at least I don't. But it's very exciting, right, that this seems like a universal tool. So it's like revolutionising climate simulations as well, it seems to be, right? And weather modelling as well. It's across the board for physical simulation. My theory is it's something to do with dimensionality reduction, right? That often you don't need to treat every single part of a system. And so in our case, we're kind of— this is called coarse-graining in physics and statistical mechanics. There's great theories of it. It's all based on Langevin dynamics, actually. So it's intimately related to what a lot of the AI algorithms are doing. But the basic idea is that you get these universal things across different scales, and they have repeating physical properties. And the key one is that you don't need to treat every single piece of the system to predict what it's going to do. So in our case, we're ignoring the electrons. In a lot of biology, you might want to ignore the solvent, say. And then when you get to larger scales, obviously when you're looking at a ball moving through the air, it's just one object you're tracking and you're ignoring this vast amount of other information going on. And so finding efficient ways of keeping track of the important information and losing the unimportant information is just a central problem in a lot of physical modeling. And I think AI machine learning algorithms are just very good at doing that. But what do you think, Jonny?

Yeah, I think you're absolutely right. You know, what's been extraordinary to me has been not just the kind of universality of the sort of AI models that we use. So, you know, the stuff that we develop is very similar to the stuff that's used in weather forecasting, similar architectures, almost identical. But also just how much implicit physics knowledge can be contained in a very small amount of space. When you look at some of these world models, like some of the Genie 2, if you've seen that from DeepMind, or a couple of other startups have released something similar, they have these single models that simulate physics because they're simulating water dynamics in reflections of lights in puddles. They understand gravity. They're able to deal with very small, tiny effects that would, just a couple of years ago, taken really expensive computer rendering, CGI-type computer software in order to produce. But this only has, I mean, I don't know the precise number for these models, but these must only have a couple of, a few billion parameters, right? These are going to be diffusion models. Diffusion models are not parameter-heavy. They're very data-intensive, but they require far less compute during your inference pass or for learning these dynamics than a large language model does. So there's this implicit, as you say, this intuitive knowledge, right? Because we know that the bottoms-up calculation of all of these effects and the effects that we model when we think about atomic-scale quantum effects, which is the stuff that we really care about, Calculating from bottom up requires a huge amount of floating point operations. So we're taking a lot of jumps. Whatever's going on, you can call it intuitive, but it's shortcuts that preserve the fidelity of that simulation in the ways that we care about. It is just quite extraordinary that we're able to condense that information in a way that has that level of fidelity into something just so incredibly small. I mean, the They're big compared to neural networks for a couple of years ago, but they're tiny compared to our brain. They're tiny compared to the amount of compute that we're throwing at it with bottoms-up physics simulations. So it is quite incredible. And I think that intuitive physics explanation is a really nice way of thinking about what's going on because it's just in the same way that you get extraordinary sports people who can score incredible goals and do incredible feats, but not know any maths. That's exactly what these AIs are doing. when they're doing these simulations.

Hey, we'll continue our interview in a moment after a word from our sponsors. Before we get more into the practical details of how you've gone about all this, one other philosophical question kind of relates to— I don't want to be too literal about grokking, but there are these debates in language models where we get very confused. And we're like, you hear people say things like, well, sure, they might learn the concepts that we have learned and encoded in language, but they're not going to learn anything beyond that because how would they? They can only learn what's already encoded in language. And I sort of find discussions around that sort of topic can easily get circular. But then I move over to a different domain, And we've now seen multiple examples of this out of the intersection of AI and biology, where you can take interpretability techniques and apply them to, say, a protein language model and find out that, oh, I recently had a professor from Stanford, James Zhao, on, and he was like, they discovered a new motif by looking at applying interpretability techniques to a protein language model. And so it seems like there's something there where it's pretty hard to deny that this thing has learned a concept that we did not know, and it didn't get it from us, right? Because in that case, it's just training on sequence data. But I don't think I work that way. So this is where the analogy to intuitive physics maybe breaks down, right?

I don't think—

maybe I have, just can't articulate it. I really have no idea, I guess, what's going inside my brain. But it seems like there is some sort of conceptual grokking that is happening in, for example, the protein language models that are allowing them to do these shortcuts. It's these higher order concepts. Is that also what you think is going on in material science? And do we know what those concepts are? I have a sense for a motif and active site or whatever, but I don't even really know what those higher order concepts would be in material science that these models might be grokking out of the raw data that they're trained on?

Yeah, I mean, one thing is, I think that may be going on in scientists' brains, right? Where like a lot of science, I mean, people don't think of it this way, but it's really intuitive, right? And it's really gut instinct driven, right? Like, you rely on your gut all the time. And it's possible that even though we don't rationally use these ways of discovering things, that it could be that scientists are doing this intuitively, and definitely material science, I think that's very common, is based, you know, people call it intuition, right? Or, you know, chemical intuition is a very important concept, and people rely on it, and they can't often, they'll give you a story, but it's a very powerful tool. So it's very possible that the AIs are doing something similar.

So what does that look like, just to calibrate myself? Like, does a good material scientist just sort of put a finger to the wind and say, Yeah, let's throw in a little more acid.

I got a little bit of a story about this. I'm stealing someone else's story. This is exactly what happens in semiconductors. The people basically spend 30 years of their life on this very tiny thing. I was talking about etching before, right? It's like, what gases do you use to clean? You etch a silicon wafer and then you need to degas it, you need to clean it. with another gas. Which gases do you use? How do you change that slightly? How is it going to affect the silicon? And people just spend 30 years of their life in each one of these. I think there are about 10,000 steps or something, process steps, in making a chip. The packaging, the etching, the lithography. And it's just, it is like, these are craftspeople, artisanal. They've They've done so many experiments and gained such intuition that, and it's not written down anywhere. You can't go and learn this stuff. It's not easily deducible from first principles chemical knowledge. And it's passed down in this sort of master and apprentice type way. And it's one of the reasons why people just can't recreate TCMC. However much money the US goes and throws at new chip fabs, or China, how much espionage they go and do to try and build their own fabs in China. You can't beat TCMC because they've just spent all of that time and that intrinsic knowledge that sits within their chemists and material scientists. So it really is this kind of deep sense of intuition. This won't work. This will work. One of the goals of AI is to break that, really, to say, well, AI should be able to, without having to go through all— ultimately, it's still physics. So AI should be able to get it right if they can get physics correct, and it should be able to democratize that level of knowledge. But yeah, it's like a sage person who's done this 30 years pronounces their views and it's almost prophetically correct in many cases. You think, no, that can't be right, but it actually is. So my level of admiration for the human own practice and intuition, which is something that's kind of not really captured in something like a reasoning model, but this intuitive movement through the search space as scientists and chemists and material scientists, that has grown exponentially through my experience at Orbital. Because we've got some incredible people that just know what will and what won't work. But anyway, Tim, whether your experience with chemists and material scientists has been the same?

Yeah, I think so. And it's sad because it's a beautiful thing, right? Like this built-up body of amazing knowledge. And it's sad to kind of be trying to replace it. But I think there are fundamental limits to the human brain and we want to push beyond that basically, right? So we need ways of doing similar stuff computationally. And I think it's very exciting if we can start to get there. And then potentially as well, combining it with the human intuition, I think is very powerful, as Johnny was saying. because they've obviously reached it from very different ways, they're going to be able to correct problems that each other makes. So I think it'll be a powerful combination when we can get them working together really well.

Another thing that caught my ear was you said that the diffusion models are not parameter heavy, but they're data intensive. We talked last time with Tim about the training on molecular simulation data. So if I remember that accurately, it was like it costs a lot of compute, and I think it was 10 to the minus 15th second time intervals that were being simulated. Yeah, 10 to the minus 15th. And then train the model on that, and then you get a model that can do essentially the same thing, but orders of magnitude faster. AlphaFold, if I recall correctly, was only trained on maybe 5 figures, maybe into low 6 figures worth of protein structures that were known at that time, which I think was basically everything that was known at that time. What does the data universe look like in a materials science context? Because I can imagine it could be like a simulation thing, or it could be a big catalog of all materials that we've characterized to date, or maybe even the both of those.

So, yeah, this was a big issue in chemistry, right? For a long time, I was quite skeptical of AI for chemistry just for the problems I wanted. There's just vanishingly little data, right? So experimentally, like, I was interested in electrolyte solutions and there's like some databases out there, but there'll be like a few parameters per electrolyte. And so I really thought like, oh, this is going to be really tough to do anything. Now there's other areas of chemistry where there is large datasets, but in my particular area where I wanted to apply AI, I just couldn't see how to do it. Obviously, yeah, there's the PDB and biology of these huge datasets. But the thing that really clicked for me was, well, we can automatically generate these training datasets using first principles computation, right? And so it's kind of trivial. I realized I had huge datasets. If you take into account the massive simulations I've been running based on first principles physics, these are generating billions of data points pretty trivially. Because the data is generated computationally, you can do all sorts of nice tricks like active learning and automatically generating the data in loops where you need it and things like that. So. That's why I think first principles generated data is a big area of future growth in science.

Somehow you can think of this as like a, where does AI generalize well? Because the data that you can generate quickly is often pretty small data. Maybe I can give you like an example. You know, you are simulating what Tim's just simulated, you know, a protein complex interaction with a metal. Now, you're simulating that thing in isolation. It's a small part of something far bigger. And, you know, all of the water molecules that surround it, all of the rest of the cells, all the rest of the proteins affect the dynamics of that in some small way. Now, hopefully for the protein situation, it doesn't affect it so much that the protein stops functioning, but for something like a catalytic reaction, if you're in green chemistry and you want to convert CO2 into sustainable aviation fuel, In order to simulate that correctly, you need to simulate thousands, if not millions, of atoms, and you need to simulate that over very, very, very long timescales. And so the thing that I think is extraordinary about AI is you could take just a tiny piece of that simulation, maybe just 100 atoms, and you can run that simulation, which isn't going to tell you very much about chemistry, but it's going to give you rules and insights into physics. And you can learn your algorithm on lots and lots of examples of very small systems. And then that's going to generalize to very, very big systems when you then go to inference time. And I think that's the crucial observation that really accelerates and is the game changer, because what you're able to do is take something which is too simplistic to be useful in prospective design of materials. But you can learn the rules from that thing, and then you can then apply that into the large-scale systems that do tell you something useful that would be completely impossible to be able to be tractable with traditional approaches, but work with AI. So that generalization capacity of these new AI algorithms, I think, is what has been just so game-changing. And I think Tim has been really leading the way here. With some of the Org models, the thing that really just blew, completely blew my mind was training on small inorganic crystals, like 20-atom systems, to then simulate a protein through out-of-the-box generalization. That's telling you that we're learning something really fundamental at that small scale, which I don't think anyone had ever expected.

Yeah, that's a great point. There might be like kind of a finite amount of complexity that you have to capture. And then once you've captured that, you can kind of do a huge amount of other things. So there's only so much data you may need for some problems, I think.

Yeah. Hey, we'll continue our interview in a moment after a word from our sponsors. Okay, so you mentioned these orb models. I was doing my homework, of course, preparing for this. And I understand that there's sort of a constellation of models, But I need your help to understand the hierarchy of how these things relate to each other. So I gather from the website there's a model called Linus that's kind of the internal, most foundational foundation model, and then those are getting kind of fine-tuned for different purposes.

Yeah.

Maybe it would actually be most useful just to talk about some of the tasks. We've alluded to one, which is like simulate a system through time and You alluded earlier to same architecture as the weather simulation models, that those tasks seem to line up, right? Given some initial weather conditions, propagate that through time. Given some initial setup of atoms, propagate that through time. But I've also heard other task types described, for example, like, can you come up with a material that will have these properties? And that sounds like a very different sort of input-output. So Maybe just give us a sense for what task types we're trying to tackle here.

Yeah. So here's where Tim saying everything is Langevin dynamics becomes really opposite. We start off with a generative model of materials. So what we mean by that is, similar to a generative model of a protein or a generative model of an image, that's probably the right way to think about it rather than a large language model. And these are what are known as diffusion models or score-based models. And I was about to give you a description of the physical interpretation of these, but Tim will get the maths more correct than me. So maybe Tim, you can explain how these things are actually all just physics all along.

Yeah. So the interesting thing about the diffusion models is that if you look into the inspiration, it comes from this non-equilibrium statistical mechanics. It comes from these kind of generating these molecular simulation algorithms, Langevin dynamics, designed to generate trajectories of atoms moving through time. So there's kind of a deep connection between this task of generating new molecules and making these simulations of things through time, where the mapping is the score, which is this vector field that the atoms move on in a diffusion model that you generate the atoms in, is also basically analogous to the force field, which is the thing that pushes all the atoms in a molecular simulation. And you can actually show— there's a lovely paper from Cecilia Clementi showing that if you train a diffusion model on one of these simulations, right, this time evolution, the score is mathematically identical to the force field. So there's deep mathematical connections between these things, which is very exciting. And so the generative model, the way to think about it is it's doing essentially a molecular simulation. You can change the atom types as well, right? It's not just the positions of the atoms, which you allow to move, you can also change the atom types as well. So there's actually deep connections between the molecular simulations and the diffusion models.

Yeah, it's an extraordinary thing. And it allows us to do this pre-training, fine-tuning step. So the way I think about it is these diffusion models are learning simplified physics. The great thing about the diffusion model is that you get to sample infinitely many points, almost infinite training data in a diffusion model. Because the way you train a diffusion model, in the case of an image, you take an image and then you gradually corrupt it with noise, and then you predict— you then do the reverse step where you start with noise and then you predict how to denoise that image. And that denoising process is the generative process. But, you know, you can sample infinitely many noise corruptions, and so you've got infinite amount of training data That you can train on. So you've got really high data density, right? Like you've got infinite samples of your force field. And so that is a great way for doing pre-training. Pre-training, you need a lot of data. So the way I think about our generative model is that it does the generative side. So we get to generate materials according to functional properties of that material, but it's also a great way for us to learn a simplified physics. Like you've got infinite data for a simplified physics, and that is a great pre-training step for us to then learn the real physics in fine-tuning when we're learning the real forces computed through quantum calculations. So the Linus is the foundational, foundational model, learning a simplified physics, which we then adjust to give the correct physics that we can use in real simulations in that fine-tuning step. And that fine-tuning step gives us Orb. Which is basically calculating the interactions that happen at a quantum level between atoms. And once you've got that right, then you've basically got physics right. You can then move on to simulating whatever you like. You know, you can simulate a nuclear reaction, you can simulate what happens in the body, and you can simulate all the incredible materials that we use every day.

So I get the image one. Let me just stay on this for a minute because I want to understand this better than I do. I've given a very similar explanation to many people many times as you just gave about the image version of this, where it's like, okay, I've got an image, I can add noise to it. I have a pretty good intuition for what that means. And I also have a pretty good intuition for how if I just keep adding a bit of noise over and over again, eventually I get to something that for all intents and purposes is just noise. And then I can imagine Reversing that and training to predict. The way I always phrase the question is, what would this image look like if it were a little less noisy? And then you also have the steered version of that, which is, what would this look like if it were a little less noisy and a little bit more like this text prompt that was aligned to image space via CLIP or whatever? Now, in the context of proteins or peptides, I've learned, but not super well understood, that There are custom noising processes because it's just more complicated. I guess if you just add pure noise, you end up with something basically incoherent and it doesn't really work in the same way as you can just sort of add kind of pure noise to an image. And then there's also probably issues with discreteness where you're talking about atom type changing, and that seems like not something that is super differentiable. So And then there's also this aligning question too, right, of like, how do these sort of low-level detailed configurations of atoms translate to qualitative macroscopic properties that we care about? So if that was the 101, give me the 201 on what noising means in this context and how you're aligning low-level things to properties we care about.

So noise is super straightforward in materials. It's very similar to images. You don't have to treat it as specially. You take a crystal and then you just jiggle the atom positions around and you jiggle them more and more until it becomes totally incoherent. And then there are techniques that you can use for discrete diffusion as well. And they're kind of pioneered in images and they work pretty well when you transfer them over to materials. So it's actually more similar to images than it is to Proteins in some ways. The only thing that is super weird in a crystal structure that you have with a material is that if, here we've got a box and you've got your crystal inside that box, an atom moves across one face of that box, it appears in the opposite face again. And so noising means something slightly different because, you know, you don't have an opportunity to go all the way off into the distance. You're always going to be sitting inside one of these small boxes. And that adds a little bit of challenges to the maths, but in practice, you just treat it like you would an image. So it's pretty straightforward in that sense. When you think about guiding towards property, it actually ends up being very similar. In an image, you will train, I mean, there are a bunch of ways in which people do this, but the way you call it sort of guidance. So the way you do that is you create an image classifier basically, and that creates an energy function, right? Something that says, is this close to a cat? And as that number goes down, it becomes closer and closer and closer to a cat. We just do the same thing. So we'll take a crystal structure or a material and we'll train a simple supervised learning algorithm to predict a property. And we'll set that property to what we want it to be, and we'll try and just nudge it towards that property through that differentiable machine learning classifier. So your description is basically entirely the same as what we do in materials.

Yeah.

The thing that we have to do differently is we can't use the same machine learning architectures. We may need to make adaptations to our underlying machine learning architecture in order to get really accurate results within material science. And that's, I think, the big difference between materials and images. If you were to look at the maths, the maths looks pretty similar.

Sorry, could you say again what the big difference was?

So what's really important, you know, if you have 2 atoms and they get closer together, then the forces that are acting on those atoms increases. And then if atoms go really far apart, then you get no forces applying on them whatsoever. Um, but there might be a sort of Goldilocks zone where they're attracted to each other. Um, and so the distance between 2 atoms, or the vector if you're thinking of 3D space, is like a first-class citizen in representing an atomic system. It's not a first-class citizen in an image because all the distance, if you think of a pixel as an atom, all the distances are the same. So, you know, you don't need to, you don't need to represent it in your architecture, or, you know, all tokens are, you know, the same distance apart. So you don't need to represent distance in a transformer. But for 3D point clouds, you know, for atomic configurations, distances are first-class citizens. You don't get to predict a property unless you have a really good, high-fidelity representation of distances. And so that means that you have a different type of modality as an input to your machine learning model. You need to have distances, of which there are quadratic number, and you need to have the atoms themselves. So that gives you a kind of very, very different type of input modality. And so you need to have a different type of machine learning architecture called a message-passing neural network, which is basically just like a transformer, but a transformer that takes distances inputs as well. And that is what we use. And that's the major difference.

Okay.

If you were to kind of be training You know, look at the code for our models and you just look at the code for a vision transformer or a language model. That's going to be where the major differences are.

And I would love to have a little bit better intuition for how we make those leaps from like atom type to atom type.

Yeah. So you basically treat categorical prediction. So, you know, you're predicting atom species from 0 to 117, as a continuous function. So instead of having a cross-entropy loss on it, you treat it as a continuous target, and then you cast that back into a categorical one. So this is a technique, I think it's called Bino Analog Bits. It came out like 2022 as a way of doing discrete diffusion. You basically just cast it as something that has a gradient. in it as a kind of softening of that discrete distribution. And then it's a pretty elegant and pretty simple technique that works pretty well. Yeah, it's a Hinton paper, so you know it must be good, right?

Okay. So can we go back to tasks then and say, with all that in mind, when I do an image diffusion model, I might show up with just a text prompt and say, make me a picture of a scientist in a material science lab, or I might say, here is a picture and I want you to make it, you know, more like a dinosaur working in a material science lab. What do I show up to the different Orb models with, and what do I then get back?

Yeah, so for Linus, you'll show up with something like, you know, I want to have a band gap of x, give me 10 materials that have this band gap, and it will generate a material that has that property. Very similar to an image generative model. Start off with a description of the image and you'll generate an image that fits that description. So it's exactly the same. That's what Linus does, our foundation, foundation model. And then Orb, the Orb models that we've released, the ones that are publicly available, have a different task. And that's the simulation task. So that is, I've got a system, but I want to know what's going on. Like, I want to know if this, like, from a first principles perspective, how is this reaction going to be taking place?

So the main thing they're doing is predicting forces from coordinates. So you give them the positions of all of the atoms and the atom types, and they'll print out the forces and the energies. It is essentially all they're doing, but that actually unlocks a huge amount of other things you can do, right? Because people have been running these simulations for decades and developed all of this theory and code and know-how about how to use this ability to run simulations with these forces, that we can then leverage that to compute a whole host of other properties. So you can compute things like diffusivity, so how quickly atoms will move, their chemical equilibria, so like how strongly they stick to each other and stick to other atoms. Essentially all of the really fundamental stuff you want to know about materials, you can then extract from these molecular simulations. Which is a really exciting thing. So there's then kind of 2 ways you could go about generating new materials that have the properties you want. One is you could try and just directly simulate a ton of them and see if they have the properties you want. And then you can try and bypass that with a direct generative model. And you can combine them in different ways, right? So coming up with hypotheses with the generative model and then feeding them into the simulation to check that you do see what you want with these more exact simulations.

Yeah, I kind of just want to say, keep going there because the sort of outer loop is really interesting. I mean, we talked about that in the context of your work with the solutions and it was like calculate all the forces, apply a time step increment, everything moves a little bit, recalculate the forces. Next thing you know, you're seeing crystallization popping up out of nowhere. I'd love to, you know, yeah, kind of keep zooming out internal tasks and talk about the scaffolding. that you're building around them?

So in terms of molecular simulation, there's really 3 key things you can do with it, which is exciting. One is, I think of as a computational microscope, right? So the idea is you just directly see what's happening, right? So you can make a movie and you see what the atoms are doing. This is a very powerful and useful tool, right? Because in particular, it gives us access to a temporal and spatial scale we haven't had access to previously. And that's particularly the nanoscale, right? Like nanometers, nano seconds, kind of a billionth of a meter, billionth of a second. This is a really important scale because a ton of interesting stuff happens there, all sorts of chemistry, chemical bonds are formed and broke at that kind of timescale and spatial scale. And so we'd really like to be able to see what happens. Very hard to do it experimentally because these things are much smaller, you know, they're atom-sized, right? So you need something similar size to kind of observe them experimentally. But we can potentially get at it computationally, which is exciting. And Amelie, your previous guest, talked about this in the protein folding context, right, where— The statics problem is, you know, maybe been solved. You can now predict the static structure of all these proteins. What we're really interested in now is how do they move? What are their dynamics? And particularly, you know, these very short timescales, which we can't get to experimentally below, you know, a microsecond. And that's where these techniques are really useful. This idea of a microscope is really powerful, I think, because it's revolutionized science many times, right? When Leeuwenhoek built the first microscope, Everything he looked at, he made new discoveries, right? Like, just looked at some pond water, right? Found bacteria, looked at his blood, and he found red blood cells, just discovery after discovery. And so the hope is that if you can simulate a ton of things, you start to make important and new discoveries and find new things out just by looking at them, right? Which is very exciting. The second one you can do is, I think of as in silico experiments, right? So you don't now have to go and order the chemicals and synthesize the new molecule. and do all this work to try and then see it in action and then use microscopy to try and understand what's happening. You can just change the atom type in your input file and rerun the simulation, right? So you can do simulations much more quickly, hopefully, than you could do them experimentally. This is a while off, but eventually you could think about doing high-throughput things with this, right? So trying, you know, thousands of simulations and just trying to brute force find out which material works best. The third thing you could do And this is actually long-term, the most exciting one is actually use those simulations to extract data from them to then use for larger scale models. And you can think of this kind of as providing a bedrock for physical simulation, right? So for many, we simulate all sorts of physical systems, right? The climate, rockets, whatever, you simulate these things with all sorts of modeling approaches, but these all need to be parameterized, right? They need to have numbers plugged into them. And it's often very hard to work out what those numbers should be. You have to do experiments, very hard to do those often. Whereas if you could use lower-scale simulation to extract that information, you could plug it into these models, which would be very useful. And then the other thing you could do is you could generate training datasets this way, which you could then go and train your generative models on, which would be great as well. And then there's also this coarse-graining thing I've mentioned, which is what I was doing with the crystallization. And this is a statistically mechanically rigorous well-defined procedure where you can show that you can ignore certain aspects of the system and keep other ones. You could ignore the solvent, for instance. So one day, you could think about simulating how proteins fold with time. And in fact, people are already starting to do this using some of the approaches that Amelie was talking about, training on all-atom molecular dynamic simulations to then enable you to look at how proteins fold dynamically, which is very exciting. And that will come. And it's very similar in material science. You want to know— Both the static structures these things are forming, but also the dynamics, how the chemical reactions, what are the transition states, all things that's very, very hard to get at experimentally now.

Maybe we can say how all these things link together. Like, what's our loop? How do we go from idea to material using these technologies? Do you think that'd be helpful?

Yeah, please.

So these technologies are used in a bunch of different ways. Pay a lot of attention to try and make them really useful in the prospect of designing materials. So you often start off with, you know, a functional property that you care about. You'll decompose a chemistry problem into a subset of things that you care about in the material. So maybe you're saying something like, I want to create a metal-organic framework type of material. It's a big kind of sponge-like material. And you say, I want to have a very big pore size. You know, I want to have like— each bubble in my sponge needs to be really big because it needs to contain a lot of stuff. I'm going to absorb some things, so I need a large pore size. And I might want to attach little tendrils, like little ligands, to this metal-organic framework. And so it needs to have lots of sites, active sites, for me to attach these little ligands in, which are going to act like little tendrils that pick up the things that I want to absorb from my weird mixture that I'm going to pass this whole thing through. So you'll use a generative model to say, okay, well, generate me a material that has a large pore size and has lots of sites for me to attach ligands to. And that's what the generative model does. It will give you maybe 10, 15, 20 different candidate materials that have some of these properties. But, you know, making 20 of these materials is still going to take you quite a long time. And there might be a bunch of things that are really difficult to use as generative prompts, but are really important. We know we can access through simulation. Maybe we want to understand, you know, what are the kinetics going to be like? How fast is this absorption thing going to take place? Am I going to block my entire pore space with these little tendrils? And so I won't be able to get access to any things I want to catch. All these sorts of questions, which are really hard to put down in like a prompt. Really hard to condition against, but simulation can answer really, really, really well. So then we'll go through quite an in-depth process of using our AI-accelerated simulation tools to answer these questions at unprecedented levels of detail and accuracy. So by the time we get to making a decision about what to make, we've answered 90% of the questions that we need to in order to feel confident that we're going to have that sort of material. And so, you know, we're using the generative side to have that creativity, and then we're using the AI side for qualification and for giving great insight into what's going on. That stimulates more creativity a lot of the time because you see something unusual. Tim is always coming up with like, I hadn't expected this thing to happen. I saw this simulation and this was not at all what the literature had said, but I actually think this agrees better with the experimental data. You see these new things, and it stimulates creativity, stimulates new ideas for new types of materials. So you might go through that process 2 or 3 times. And then you'll end up with 2 or 3 things that you've really fully qualified, that are probably slightly outside of the scope of what you'd come up with by yourself. The creativity of the AI and the generative process is putting you in a different direction. And then you go into the lab. Your success rate when you go into the lab at that point is just way higher. You're pretty confident it's going to have those sorts of properties. You're doing something more novel than you would otherwise do. And that novelty point is just as important as the success rate because you're going into new areas of chemical space, and that's often where the really exciting materials and chemistries are. So that's the kind of process that we use, and we use these different tools and functionalities at different stages of the materials design R&D. function orbital.

Going back to the very beginning of that, when you say you want something with a large pore size and lots of active sites, is that something you're saying in natural language or is there sort of a well-defined property space that you are specifying that in?

Those are things that you could have a well-defined property space on. I think we've generally found that Language in some ways is not the best conditioner for a generative model of materials, because you're often looking for quite specific scalar values rather than a textual description. You know, text is great for describing an image because you don't have a scalar value with which to condition on, but, you know, if you want a specific number for your pore size, it's better to just give it as a number than it is to give it as a text Free text box. Yeah.

'Cause a lot of things you're trying to hit a sweet spot, right? So you don't want, you know, and it can be quite quantitative. You want like the right, but not too much Goldilocks, right?

That makes sense. Maybe one more just detailed question on the models themselves, and then we'll get into the application that is kind of the new headline. How big are these things and what is the sort of compute requirements that go into them? Are we talking Biden executive order qualifying 10 to the 23 if we wanted to call this more of a bio type of thing? Or where are we in terms of just resources into these models?

I think your mental model should be like an image model, like a CNN-based diffusion model, like these U-Net models, which I guess most Image generation models are still probably units. So I think you should think of it like that, similar levels of data, similar levels of compute. So a lot smaller than the large language model. That's one of the reasons why we can deliver so much, world-leading models and the lab and develop new materials, all less than $40 million raised. That's obviously still a huge amount of money, but We wouldn't be able to train a large language model for that much money.

Yeah. I mean, the exciting thing is to think that we are at that kind of GPT-1 stage, right? If you really scaled these, I mean, what you could do with them, particularly on the training data side, right? If you threw some of these huge clusters at generating really high-quality training datasets, there's probably a lot of room to improve these things.

Yeah.

Okay.

I remember we had 2 guys, Jonathan Frankel and Abhi from Mosaic on to talk about their, among other things, efficiency driving work. They took Stable Diffusion training under $50,000 last I heard. So I don't know if that's exactly where you're at, but it just as a general order of magnitude for the audience, those are certainly way less than your trillion-dollar data centres.

Yeah.

So with that, let's talk about this paper that you graciously shared a preview of with me, where you are modelling no less than the potassium ion channel. that sits in the membranes of our cells and lets potassium ions in and out in an ordered fashion. So, maybe again, we could start with a— I think people will probably be familiar with ion channels in cells, but why does this matter so much? It seems like it had been something for you that was a longtime dream to try to tackle a challenge like this. I think people would benefit from hearing why this was such a prominent mountain in your mind that you wanted to climb?

Yeah.

So there's always a problem that's fascinated me, as you say. And the basic idea is that the body is really an electrical device, right? We don't think of it this way, but it is. It's just like a computer uses electrons and metal to carry electrical currents around, but the body uses ions, right? Things like sodium chloride. Sodium is positive, chloride is negative. Potassium ions as well are critically important. And, the way it carries these signals is it creates the cells which have a high concentration of ions, potassium ions in this case. And then, when it wants to send an electrical signal, it will open up these things called potassium ion channels and the potassium ions can flow down those and that creates a change in voltage which then triggers follow-on processes. So, it's really operating kind of in a way that you can think of it as an electrical device. And, it's how just a huge number of the body's functions work. It's how your muscles contract. It's how your heart beats. All of the neurons firing in your brain carried by these electrical signals, carried by the flow of these potassium ions in and out of cells. So it's incredibly important. And when ORB, this model, first dropped, I wasn't working for Orbital and I saw it and I thought, wow, this looks really exciting. Really good accuracy. Very, very fast was the exciting thing. And I started to try it out for many different things and it was doing a fantastic job, always giving me nice stable results, simulated many electrolyte solutions. Which is what we talked about last time, which is what I was very interested in. But the fundamental reason I had always been interested in electrolyte solutions is, apart from the fact that they're very important for climate change, they're also central to biology, right, to this electrical signaling function. And the real key challenge there had been these potassium ion channels, which no one has really been able to fully understand, unfortunately, using experimental techniques or traditional computation. And in fact, we don't even know some of the most basic questions about it. So like down these channels, is it just potassium ions flowing or is there water going down it as well? Like that's an unresolved question. There's papers out coming out on both sides in top journals like every few months. And it's kind of a, what I think of as a keystone problem or a kind of problem that demonstrates a fundamental lack of, of some capability we're missing. And so if a new tool came along that could solve it, Then I think it would be really important. And so it's always been the dream to try and use the things I'm working on to address it. And because Orb was doing so well, I started to think, maybe I'm thinking about this wrong, right? Like maybe this is the universal force field, the thing that can simulate anything that computational chemists have been dreaming about for decades. And I thought maybe this thing is finally here or some form of it. And I thought, well, if it was, what would I do, right? This is the problem that I would go and try and solve. So I set up the simulation and At the time, I literally thought, like, this is such a silly idea. Like, this is just not going to work. It's going to crash straight away. The models I had been building myself previously, you know, quite unstable as soon as you got out of distribution. So I thought it's going to crash and do unphysical things. But in fact, it didn't. It started running stably. I started to see interesting things, and it was just so exciting. I literally started jumping up and down. It was very cool. And the main thing is that, like, The training data looks nothing like these things, right? It's small crystals, like highly periodic, repetitive structures. So it's never seen anything really like this. So the fact that it even just gave something reasonable and showing interesting things was really exciting.

Eureka moments.

Yeah.

Increasingly coming from AI systems. Multiple follow-ups there, I guess. One is like, to what do we attribute the generalisation from the small crystal training dataset to this quite wonky. And in the picture, there's papers and people can just Google a picture of the potassium ion channel to at least get a rough approximation of what the thing looks like. You've got a cell membrane and then there's this sort of wonky thing that has certain symmetry to it that kind of protrudes through both ends of the cell membrane. And through that, channel the ions can flow. That's way different from a— like, there's nothing crystalline about it, for starters. Is this like a divine benevolence sort of thing, or do we have an explanation for why it would generalise so well?

Yeah, I mean, Johnny might have a take. I don't know. As I said, I didn't expect it to work. So my general philosophy is I started seeing— I don't trust a lot of AI things at first because they can hallucinate and do things. But I've tried to change my philosophy because I think you can err too far in the wrong direction, right, of not trusting them enough. So I think sometimes it's worth trying and testing to see these things. And so it was definitely surprising. I wouldn't have predicted it. If I had to give an answer, I think I would say something about— I think it's what Johnny was saying earlier, right? Like there's only a fundamentally small, some finite amount of information about molecular interactions that you need to know. There's things like the dipole moment of each atom and how those interactions look, how they spatially decay. And those can change in different circumstances, but there's just a finite set of information that you need. And maybe the crystal structures are enough to get enough of that information out that you can then generalize to simulate like a much broader space of materials. I don't know. Do you think that's right, Timmy?

Yeah, I think that's right. I think the thing that's remarkable and continues to amaze me though is the effects of scale. We take scale, especially data scale, engineering excellence incredibly seriously at Orbital. It's part of our pedigree as an organization. I think it's, without casting too much shade on the academic community, I think we're kind of bringing that engineering, AI engineering into this discipline. And that has led to extraordinary results. And natural language processing, images. And to a certain extent, I think it's part of the result of that, that we're really bringing the best out of these very flexible networks now to train them. And just like you've seen these unbelievable results in so many other areas, applying those same sort of techniques and abilities to materials has given something truly extraordinary. But I would agree with Tim that All the information is there, right? Like, there's a limited amount. You just, you're learning interactions between different types of atoms. If you've got small, even if those interactions are the same, whether they happen in a protein or they happen in a small crystal, as long as they're there in your crystal structure training data, then if you've got a network that learns very well and very sample efficiently, then it should extrapolate really well. So I think it's extraordinary, but after the things I've seen in AI over the past few years, you kind of learn to expect these things. Yeah.

Expect the unexpected. Expect to be surprised. Expect to be amazed. Definitely unavoidable these days. What's at stake with this? Is this more like a sort of AlphaGo moment where it's like, holy shit, but it's not going to change the economy all that much in the immediate term, or are there things that get unlocked very practically as a result of a better understanding? And, I guess also, what did you learn about the potassium ion channel? How has our state of knowledge advanced due to this?

Yes. So, I mean, the first thing I need to caveat it with is that this hasn't been peer-reviewed yet. I've sent it to some potassium ion channel experts who say it all looks very interesting and plausible and consistent with what we know. experimentally, but, you know, ideally we'd like to see some direct experimental confirmation of the things that we saw. But the exciting things that I saw were that water molecules did enter the, what we call the selectivity filter, like the narrow part of this channel. And that isn't normally seen in classical simulations, in the classical approach to simulating these things, where they tend to think that the potassium is going through alone. But there's some, although that's mixed, experimental evidence that water is going through. So that was very exciting. It looks like something different there. And in particular, what we saw was that there was a particular hydrogen bond where this little hydroxyl group on the protein was reaching out and kind of grabbing the water molecules and pulling it into the channel, which was very cool. No one's seen that before, as far as I'm aware. And it just makes a lot of sense, right? Okay, there would be some new kind of qualitative phenomena which the other methods don't pick up that explains why they didn't see water going in. That we do. And we know if you delete that hydroxyl group, which they've done through mutation studies, the conductivity drops by almost an order of magnitude. So it really is doing something important experimentally. No one's really known what. This seems like a pretty likely candidate to me. So although it still needs to be confirmed, I'm very optimistic that this now explains this longstanding dispute in the field about what might be missing there. And it's something we can go out and experimentally try and validate now, which is exciting. In terms of why this matters economically and these kinds of things, well, this is very important actually. A lot of diseases are caused by mutations in the potassium ion channel, right? So all sorts of heart diseases, long QT syndrome, heart palpitations, things in my family, right? If your potassium ion channels aren't working properly, it will stop these electrical signals flowing in the right rhythm, and that'll impact the beating of your heart. And so if we want to understand, like, how do we fix those mutations, we may need to understand first, how does it work in the non-mutated form, and then we could simulate those mutations and see how they impair the function. And then even maybe one day start thinking about designing drugs to restore that function. And that sounds a little sci-fi, but this is actually already being done for chloride ion channels. Vertex have small drug molecules that can restore the function of chloride ion channels. So I think there is Definitely biological applications for these kind of things in terms of real-world impact.

Cool.

Just to make sure I understood the piece correctly about the hydroxyl group, previously observed was that if you make a cell that has a mutation such that that group is not there, then the channel works order of magnitude less.

Exactly. Yeah.

And what you saw that was new is presumably without feeding any of that kind of background information into it, you saw a mechanism that made you say, well, that's probably why that's happening.

Yeah. I mean, if I had to bet, I would think it definitely is. It seems very plausible. It was just doing something new that made a lot of sense that I haven't seen reported in the literature anywhere else. So at the very least, it's like a very good hypothesis. And if I had to bet, I think it'll be confirmed.

So how far do you think this goes in terms of—

and there could be a really obvious hard limit that relates to sort of the structure of the models that have been trained, and then maybe there's more like a conceptual answer. But we've got these very small crystals training models that, lo and behold, can simulate something as complicated as the potassium ion channel. It was interesting to read how you sort of fixed essentially a sort of superstructure. You've got this whole cell, right? This whole thing is in a cell and that's in a body and that's in a world, but you got to draw a box around somewhere to bound your simulation. So you basically said, beyond a certain box, we're going to kind of treat everything outside of that as fixed in place. And so if I understand correctly, there's like some more atoms that are represented in the simulation, but they're just not allowed to move in order to kind of focus the simulation in the zone of interest.

Yeah, exactly. So this, if you go back to early ways we did simulation of these things, they would use these kind of tricks. And so I thought, you know, I could use some of them here with this new technique. What you do now in a modern simulation, and we will do this with ORB now that, you know, if we could scale up the computer. At the time I was doing these, I only had one V100, right? So very restricted in my computational resources. And the basic idea is you want to simulate this, what's called the selectivity filter, this like narrow part where everything interesting happens, but you actually need the surrounding atoms because they help stabilize that central part, right? Where the interesting stuff happens. And then you have a problem because now you need the outer atoms around those. And so you have a kind of infinite regress. and you end up needing a huge system. The way we handle this normally is a thing called periodic boundary conditions, which is what Johnny was talking about earlier, where if one leaves the edge, it comes back on the other side. And so you run very big simulations is what people do now with the classical techniques. I just wanted to have a quick look to see, like, is there anything interesting going on with the selectivity filter? So I had to kind of cut that infinite regress off at some point. So I did it about 20 angstroms away. So it's, I don't know, 4 or 5 layers of atoms out. And then I just froze that outer layer to give it a kind of thing that would keep it nice and stable for long enough to kind of see the interesting dynamics in the selectivity filter. So it's a little bit of a hack. We'll fix it up eventually and do the full simulation of the entire system in the membrane is the other thing. So because this potassium ion channel sits in the cell membrane, which is like these fatty layers, you actually need a very big system to simulate the whole thing. And that should be possible, you know, With bigger GPUs and a bit more time.

Yeah, it's insane that this was done on one V100, you said?

Yeah.

Yeah, that's wild. So is it just a matter of scaling up your inference infrastructure? I mean, the model, I presume, has some sort of structural limits too, right? You couldn't take the ORB model and simulate a whole cell in it if you just had enough inference compute, right? What sort of constraints are we going to hit as we try to do bigger and bigger systems.

Yeah, it's very interesting. I don't know, we'll have to wait and see. I did have a go at simulating a larger one. I saw it, it got a little unstable, but it's very tricky setting these simulations up. Obviously, you need to get the cell membrane has to be the right thickness and all of these things. So there's a lot to do there to work out how to scale these up to larger systems. I mean, the dream is that you can just continually scale them, right? It's an open question as to whether the current architectures we're using can do that. It's possible you may need kind of longer range models that can treat these longer range, particularly electrostatics. In these systems, you get charges which have very long range interactions because they decay as 1/r, which is the same as gravity, right? So they can go to the moon. So you can have these very long range effects. So it's hard to say what will happen when we go to very large systems. But it'll be interesting to find out.

But there's not like an equivalent of a context window or some sort of hard limit built into the current models that—

Yeah, you would hit the memory limits of the GPU if you tried to scale it too much, although that's not really our bottleneck at the moment. We can do huge ones. The problem is it gets a bit slow when you do huge systems, right? So then you would have to start scaling to many GPUs. People have done this with a neural network, simulated a whole HIV capsid, and it looked stable, not with all, but with another force field. So you can simulate huge systems with it. People have done it. It looks pretty stable. It's hard to validate it because the timescales are quite short, right? Because it's so computationally expensive. So that's really the problem at the moment is how do you get these longer timescales? And I think there, this coarse-graining idea will be key. We will run all-atom simulations, and this is what I did for the salt crystallization. But then you say, we're going to ignore the solvent molecules and train a new model which just predicts what's called, what technically is a free energy now. And, but it does the same thing as an all-atom simulation, but now you're just simulating a subset of the system, just the pieces you care about. And that will get us to these very large scales. So I'm optimistic that we can start to think about doing these kind of simulating whole cell type stuff with this approach. It will just involve some hard work, but I think the theory is all there. The tools are all there. It's just a matter of really investing in it.

And there's nothing about the current Models, obviously large language models have just a finite number of tokens that they can handle at any given time. Beyond that, they just can't do it structurally. Is there, I guess there's not a similar hard limit here to a certain number of atoms or whatever?

So what happens in a language model is that when you have a finite size of your context window, that's because you have this positional encoding. That is a feature of your input. And so during training, because you only can train from a memory perspective at training time, a certain context window length, each position in that context window has a specific token, a specific feature associated with it. So at test time, you can't generalize much further than what you've trained on. That's stuff like Mamba or these state-space models, right? They don't have this explicit positional encoding. That means that you can have an infinite context window. And I think similarly, when you think about the sorts of models that we use, we don't have any explicit positional encoding. Then that means that we can really extend just as far as the compute resources can take us and basically linearly increase the size of your simulation by the number of chips that you can add on to your simulation when you think about atomistic model. And so really, the sky's the limit there. You can make as big as you possibly want, as long as you've got the chips. You don't have any fundamental model architecture challenges like you would do with a transformer-based language model.

And so we're exploiting the nature of molecular interactions that they tend to be local, right? Which is a nice property that they have, which enables you to kind of do this.

So I have to ask, when do we get our room temperature superconductor?

It'll happen. Give us at least 5 years, but it may happen. Before that, we're going to be solving carbon removal and scaling AGI energy efficiently and cheaply. So we've got some things we're solving first, but once we've done those, we'll think about the room temperature, room temperature superconductor. Yeah.

The problem with superconductivity is what we call a highly correlated electrons system. So it doesn't have this property of very local behaviors. And so some quite complex quantum mechanical things going on, which our, the current level of theory we're training on, density functional theory, is not really good enough to describe that. But it could be very useful for finding new crystal structures and, you know, working out how to make new superconductors once you know how to induce the kind of electronic properties you want.

Interesting. Okay, cool. Well, maybe talk a little bit more about that sort of roadmap, you know, between here and the room temperature Superconductor, you know, what else is on your to-do list?

Our focus at the moment is really on data center materials. We think that scaling out AGI is one of the most important things that we can do, and that new materials are going to be a really important part of making that both cost efficient and sustainable. As the rack density increases, as the power density increases of the chips that we're using, We need better and better materials for thermal management, better materials for all sorts of different things. You know, we can only imagine that we're going to need better and better chips in order to bring AGI to life. So in that sense, new materials for data centers, new materials for AGI is our focus. We also want it to be sustainable though. And so decarbonization is our other focus as well. And we'll see coming forward a pipeline of materials and products in this space. So watch this space. We've released information about one of those products. More will be coming over the course of this year.

You want to tell a little bit more about the one that's already out?

So that's our, that's our carbon removal material. This is a material that we worked on and we'll be piloting in partnership with AWS. Towards the end of this year. We have designed this material to operate and capture CO2 under the operating conditions of a data center. It allows us to utilize the waste heat and waste airflow of a data center to maximize the CO2 capture capacity and ultimately offset some of the Scope 2, Scope 3 emissions of data center operators. You know, it's perhaps the biggest, largest CapEx buildout in history, what we're currently seeing with the hyperscalers. and their data centers now. But those are also companies that are very serious about their climate goals and certainly have the resources to invest in reducing the carbon intensity of their operations. So we see that this is a great opportunity to be deploying really advanced and really effective carbon removal and decarbonization technologies.

Yeah, I think technologies like this are the only thing between the hyperscalers and a lot of broken Net neutral promises, I suspect. Because I don't think they're going to not build the AGI data centers. So something's going to have to give. Is that material, were you kind of describing it earlier when you described like a spongy, you know, high pore size with a bunch of active sites to grab onto things? Is that the sort of mental model I should have?

I think, I think that's like, we can't say too much about that because we're currently protecting some of that information, but It's certainly a sponge-like material. That's right. And with certainly reactive sites that should selectively interact with CO2 over all the other stuff that's going to be in the air. The things that we care about, like the absorption capacity at the temperatures that the data center is operating under, the composition of the air, things like that. And we kind of tailor our materials and develop new materials that are specifically designed and can take a unique advantage of that operating condition. That's a really hard thing to do because Developing a new material for CO2 capture takes a huge amount of time, years and years and years and years. We're doing that incredibly fast. And so therefore are really able to maximize the unique opportunities of that deployment situation. It's just one example of the impact of AI in some of our work.

Do you have to make, like you could imagine multiple paradigms for that sort of material. You can imagine a sponge that absorbs a bunch of stuff, and then you sort of bury the CO2 with the sponge. And you kind of keep bringing more and more absorbent material. And it's like, itself becomes sort of a waste product. Or you could imagine, you could like, clean it off somehow and reuse it.

So you're gonna do that, that's it. So you got to be, you know, absorbing and desorbing the CO2. And that's a really important part of the overall costs of doing the CO2 capture. If you can bring down the heat at which you are desorbing that CO2, or tailor that temperature so that you can utilize the waste heat of a data center, then that's going to really massively reduce the energy costs. But the longer that you can reuse that material, the better. I don't think we're ever going to get to a point where desorption of that CO2 is going to be more energy efficient Oh, sorry. It's going to be more energy costly than making another, you know, X tons of that material. Like that, that making process is super energy intense and it's a costly thing. And if that material can then last for 10 years, then that's one mechanism which you can radically reduce the cost of that CO2 capture.

Yeah.

Like the one-off capture is pretty solved. It's really that regeneration problem. That's the challenge.

Making stuff in general and just manufacturing a lot of stuff is obviously a quite a different business to be in versus coming up with the models that design the materials in the first place. I know you said earlier that you have traditional material scientists on the team and that helps you like close the loop. I assume you're like licensing the things, you're not like actually building factories and also like scaling up production, are you? But tell me.

I think that's, that's right. You know, exactly, you know, whether it be licensing or whether we have suppliers, you know, NVIDIA. don't have a fab, but they still sell chips. You know, we don't have to build up manufacturing of material in order to sell materials. Um, but I don't think that we're going to be, you know, standing up a massive plant somewhere in Jersey. That's true.

Do you need to have a lot of physical capital though, to even just do the validation experiments? I mean, it would seem like that in and of itself would be a kind of scale game, right?

Yeah, we need more physical capital than A software business does. I think that's right. But we don't, you know, we're not going to be raising money for CapEx. Certainly our lab is cheaper than a lot of the GPUs you'd need to run an AI company. It's not beyond the scope, I think, of what venture capital funded businesses should be able to do.

But I think it's so important to have that internally, right? In terms of being able to get feedback really quickly from experimentalists about what's working and what's not working. and what they need and stuff is such an important thing that it would just be so much harder to do it without.

Yeah, cool. This is amazing stuff. So I'm sure you've seen this paper that kind of made the rounds recently from Aidan Toner Rogers from MIT, where I don't think we know what models are being used or what company this research was done at, but basically a sort of behavioral industrial economics kind of study of the application of models in this general space anyway to the practice of material science. The headline results were significantly more materials discovered, 44% increase, resulting in 39% increase in patent filings and 17% increase in downstream product innovation. Interestingly, the AI's as characterised in this paper, are doing a lot of the idea generation, and then the scientists were kind of following after and more focused on the validation. So overall, more output from this system. But one of the kind of important caveats was the scientists, 82% of them reported basically lower satisfaction with their work because I guess they felt like they want to be the ideas guys, and now the AIs are the ideas guys, and they are sort of left to be the hands that validate the AI's ideas. I guess you could react to that in multiple ways. I'm interested in sort of how you think those numbers in terms of increased throughput relate to what you expect or aim for. And then also, is there anything— we got Tim jumping up and down here, so I guess not. You're in the 18% who are finding the joy in working with the AIs. I wonder if you have any outlook on like, and maybe it doesn't even really matter. I mean, I'm not one who gets too hung up on like, oh, my job changed. Hey, if it's gonna save, if it's gonna solve climate change, I'll accept a little lower job satisfaction here and there.

Yeah.

But I do wonder what you think this implies for kind of future of work, big picture sort of thing. Are we gonna adjust and find our joy in the work again? Or Is this like a sign of things to come? So yeah, just multiple reactions to that paper if you have any.

So on the enjoyment front, I mean, my take is I'm a physicist and mathematician by training, right? And I've been trying to get into chemistry because it's so important and I love it, but I don't have that kind of deep knowledge of it. In fact, I didn't actually study it in undergrad or anything. So from my point of view, these tools are incredibly empowering because they kind of give me access to that intuition. that we were talking about. Because like now I can just run a simulation and see these bonds forming. And so I just look at these simulations and I'm like, oh, okay, that's what's happening, right? And I don't need to know anything about potassium ion channels. I don't need to know the literature that that hydroxyl group is known to be important. I don't need to know like a deep intuition about what bonds with what. I just run the simulations and see it. So I have sympathy with the people who've spent their careers building up this intuition and now something else is coming in and can do it as well. So yeah, I do think that's a sad thing. But I think people, I think there will still be things people can be useful for. And I think the satisfaction of like getting materials out there in the world that are having an impact much quicker and much more faster will more than offset like that spark of like, oh, I solved the problem myself. You know, like you can do crosswords for that, whatever. But like, this is too important to like think about this, you know, the happiness of the scientists working on is my take on that.

It reminds me, I think the job satisfaction of a bunch of my friends who work in AI research has declined.

Yeah.

I mean, it's declined significantly because, you know, you used to do really creative stuff and now it's engineering and there are a limited number of experiments that you yourself can run. You can't go and do your own research project anymore. I've got a great idea. You can't really experiment with it because it's so costly. There just aren't the compute resources to test it fully. So people's jobs have radically changed in AI research over the past 2 and a half years. To, you know, for many of those researchers, they've declined despite the fact that their jobs have become more prestigious. They're sort of getting paid more. They've, you know, they're seeing AGI take place in front of them. I think that for a lot of people, that's going to be the case. We're going to get more productive, we'll get richer, but we will have to find other ways in which to feel value from our work. And that creativity, you know, these systems are really creative. Creativity is one of the things, first things that's going to go, you know, it's going to be manual tasks. Robotics will come quite a lot after the creative industries have been Fully accelerated through the use of AI. I'm not so optimistic that people are going to enjoy AI. I'm though fully convinced that the only way that people are going to, you know, the future is going to be a happy world is through AI because we need to produce more stuff for less money. And that means that we need a greater acceleration of scientific discovery. than we've ever had in the history of humanity. And the only way that we're going to achieve that form of acceleration is through AGI. So we've got to have it as fast as possible, deal with the pain over the short term to get to the, uh, glorious future, as Sam Altman would put it, of, you know, a large portion of economic activity being fully automated.

That's probably a good note to end on. Anything else you guys want to talk about that we haven't touched on?

One point was just on the combination of LLMs and neural network potentials, these universal force fields, is quite powerful. A point worth making is if you're worried about— so central challenge, if you're worried about artificial superintelligence and things, is how are these things going to get some kind of experimental validation? Then maybe they can design whatever new materials they want, but they need to be able to validate those things. This combination of LLMs with the force fields, which gives them the kind of ground truth, Perhaps enables them to do these things internally without needing to go out into the experiments and measure things experimentally. So it's gonna be very interesting. I think it's a powerful combination cuz you get that kind of grounding, the way to check hallucinations and correct hallucinations physically. And we're seeing this with these new O3, these models that can solve these incredible math benchmarks, but it needs to be problems where you can validate them somehow, right? And so I think these neural network potentials potentially provide that for science, a way of validating hypotheses.

Yeah.

Automatically in the computer rather than having to go out into a lab. So I think they're going to be a powerful combination moving forward. It'll be very interesting to see.

Do you think we end up with omnimodal models where the sort of AGI is something that with the same weights is talking to me and running molecular simulations?

Certainly you'll have a shared embedding space. I don't know, you know, because I think there's some of these image generative models, they're not— your decoder for the image is kind of trained perhaps like separately or feels like a different model in their structure, but they've got the same embedding space. I think we'll certainly have that. I think we'll have omnimodal embeddings. You know, that's where the concept and the thinking happens. I think that's almost inevitably going to happen. integration of the atomistic stuff into the training of a scientific AGI, I think would be a really cool thing to do. And I imagine someone's already working on that somewhere.

Yeah, I wouldn't doubt it. It's going to be hard to make sense of it. I do sort of worry that for multiple different reasons, obviously there's been deception discourse recently and deception discourse was quickly followed by reasoning in continuous latent space. I am among those who's like, geez, I kind of liked it when I could read the chain of thought. You're telling me now that the chain of thought is going to be all in latent space? That's going to be a little weird. And yet it does seem like it's a very powerful attractor for multiple reasons. I mean, including that the systems will be able to communicate with each other much more effectively if they don't have to condense everything down to language tokens to send messages back and forth.

forth.

So I don't know how we get around that, but it does seem like that vision takes us very quickly to a world where we are sort of just hoping the AIs are doing the right thing for us, right? I mean, is there an alternative vision aside from that?

Yeah, I mean, someone I saw recently said you just hope moral realism is true, right? That it's going to find the same, it's going to do the right thing.

All right.

Sobering notes to end on, but inspirational work. I mean, really cool stuff. And we'll post links, of course, in the show notes, but some of the images back to the crystallization and also with this potassium ion stuff are really impressive stuff. So I appreciate your guys' time today in educating me about this. And I'll give you the official sendoff. Jonathan Godwin and Tim Dagnan.

Thank you.

of Orbital Materials. Thank you both for being part of the Cognitive Revolution.

It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.