Founder Stories: Raphael Townshend, Atomic AI & Unlocking the RNA Universe

Hello everyone, welcome to episode 73 of the Stanford ML Seminar Series. Of course, this quarter we're partnered with CS324, Advances in Foundation Models. Today I'm joined by Michael and Ivonika, who will be fielding questions from the CS324 students. And our guest today is Raphael Townshend. So Raphael was until pretty recently a PhD student at Stanford. So somebody from our own home institution, but now he's the founder and CEO of Atomic AI. He's been on this like Forbes 30 Under 30. He's been featured on the COVID of Science. He's won a number of awards, published in Nature Cell and a number of other places. And today Rafael is gonna be talking a little bit about unlocking the RNA universe. So we're very excited to hear from him today. And Rafael, go ahead, take it away.

Hey, thanks for the, the intro there, Dan. Yeah, let me pull up the slides here. Just testing that you can see this very quickly.

Yeah, looks good.

Great. Fantastic. Yeah. So, yeah, as Dan was saying, happy to talk a little bit today about unlocking the RNA universe, and in the theme of this class, perhaps a little bit about how we might use some large language models in order to do so. Yeah, so a number of concepts here, you know, maybe it's a bit different fare than usual. It's also going to be a little bit of a science lesson. So let's have some fun here and dig in. So, you know, first, you know, before I dive into RNA, I want to like provide a little bit of context of this general area that I operate in. And that's this area known as structural biology specifically, which is sort of reasoning about the structures of molecules. And sort of, if there's one thing that I want people to take away from this is that really, you know, structural biology is emerging as this very interesting area where, you know, you have this fundamental machine learning data type, in this case atoms, that you can really, you know, collect in abundance to predict very interesting properties relating to these atomic systems. So very similarly to like the pixel in computer vision or the character and natural language processing, the atom is emerging as a fundamental machine learning data type for structural bio. And so obviously a couple examples down here, like my own background actually is originally from computer vision. That's what I started my PhD at Stanford in, in fact. But obviously you've got ChatGPT, but then also methods like these proteins here, like hemoglobin here, which is fundamentally composed of these atoms in 3-dimensional space. Okay, so I've said structural bio, it's got a lot of atoms. Maybe the natural question is, why do you care about that? So continuing the science lesson for just a second here, you know, really at the end of the day, there's this fundamental notion that structure determines function. It's kind of a central mantra in structural bio. The structure of a molecule is key to what it does. And in a way, this is almost too obvious, I suppose, in that, you know, the shape of objects are key to what they do, right? Like the shape of a bike in this case might be very important to the fact that it can transport you around. So if the wheels were in the wrong place, it wouldn't do a particularly good job of transporting you around. And so very similarly, the shape of a molecule is key to what it does. Now the other very interesting piece of structural biology is that really these structures are composed of recurring patterns arranged in these very elaborate hierarchies. So what I'm showing down here in this case is you start with just a single atom, which, for example, could be composed of amino acids, which then could be composed in these bigger units called secondary structure elements, which could then be composed into full domains, which can then be arranged in these whole complexes that form cells. And while these patterns and structures are fundamentally governed by physical laws, we only really partially understand them, especially at these higher levels. And on top of that, these higher-level structures are linked to many important properties that are really hard to derive principles, like molecular simulation, and that we therefore like to predict. And so to make this very concrete for a second, you can pull up a fairly recent example. In this case, for example, SARS-CoV-2. I'm showing you here what was on the COVID of the New York Times in October 2020. And in this case, we're looking at the structure of the virus particle, including these spike proteins. Shown here in white and blue. And in particular, what you can see is that if you can understand these structures, you can figure out how these things work, right? So in this case here, what happens is the spike protein can latch on using this little teal element to these other proteins sitting on the surface of a cell. And in that way, by latching on at that point, they can really start the process of infecting your cells. And so if you can really model that interaction very specifically, you can start designing molecules to block that interaction. And this really becomes this process known as rational design of really understanding what's going on and then designing just the right thing to fix the problem. Okay, so this tells you maybe a little bit about structural biology for a second, but let me tune it back now to talk a little bit about machine learning and some large language models in particular. By going down another case study, in this case AlphaFold, which some of you may have heard about in the last couple years and was actually considered the breakthrough of the year in 2021 in Science. And to justify this very quickly, the fundamental task that we were working on in this case is one known as structure prediction. In this case, you're going from a 1D sequence, in this case an amino acid sequence that forms a protein, and you're predicting a full 3-dimensional structure, okay, the full 3D shape this molecule adopts in your cells. And again, we care about that because the shape of something is important to what it does, and, you know, once you understand these shapes, you can better cure diseases. Now, this is a really important problem for a number of reasons, and the entire field of structural biology has been trying to solve it for many years. And the classic way of doing this are through these experimental wet lab techniques, For example, X-ray crystallography, which I'm showing one of the machines down here. This is actually the SLAC linear accelerator, which has been converted to one of these machines and is, you know, basically a billion-dollar machine to solve this mapping here in the top right. And so that's great when you can use them, but it's obviously quite expensive and quite slow. You know, a good friend of mine did his entire PhD solving a single one of these mappings for his entire PhD. And that was considered a huge success overall. And so really the dream of AlphaFold was, hey, can we learn this mapping instead with the right sort of algorithmic techniques? And fundamentally what AlphaFold is, is really it's a large language model at the end of the day. It's consuming basically this one-dimensional sequence, going through a number of different embeddings here, the details of which perhaps don't matter so much. And then you end up with this sort of trunk overall, which is really just a bunch of different attention layers, sort of learning more about these sequences. And at the end of the day, you can do both sort of masked language models, right? But you can also predict other very interesting quantities. For example, the pairwise distance between any two amino acids in this three-dimensional structure. But on top of that, you really end up with these embeddings that are usable in many different contexts beyond even simply predicting structures. And I'll talk about that a little bit more in a bit. But at the end of the day, you can end up with this nice model that essentially can fold proteins for you built on top of these large language models. So what I'm gonna show here in the bottom right is really a simulation that's driven by this large language model to fold a protein into its desired 3D shape, in this case, the one up here on the right. So you can sort of see it very quickly there, just folding it up into its 3-dimensional shape. And again, this is actually all guided by this AI-driven potential function. At the end of the day than with a gradient descent method trained on top of that. And really the end result of it is that basically this problem known as protein structure prediction has basically been considered solved by these kinds of techniques. This problem that people have been working on for decades, right, and where the performance measures— in this case I'm showing this accuracy measure GDT, the specifics of it don't matter so much, but really what you need to know is it goes from 0 to 100. If you had 100, you're doing perfectly. And in just the last few years, really, you've seen that these AlphaFold-type methods have really created this dramatic improvement, hitting these levels of accuracy that are basically within experimental noise at this point. And so this has led to this huge amount of interest in this field of structural bio, and really this recognition that these kinds of machine learning technologies in this space are really opening up some fairly fundamental and impactful problems. Okay, so I've just been talking a lot about structural biology, about the shapes of molecules.. And a lot of this so far has been focused on proteins. Now I want to, you know, pivot a little bit here and talk about RNA specifically, which is really a lot of the focus of my own PhD. So let's talk a little bit about the RNA world and about how these kinds of large language models can really unlock that space. Now I've been talking about proteins for a fair bit, and if you look at most drugs on the market today, Like 99% of them target proteins at the end of the day. And what's very interesting though is that there's this vast amount of RNA biology out there that we're only beginning to scratch the surface of. A crazy stat that you can list off is if you look at the human genome overall, you can see that about 1.5% of the genome becomes protein at one point or the other, but about 80% of it becomes RNA. And so there's this vast world of RNA we'd love to go after to cure a number of diseases. But we oftentimes do not have the capabilities to do so. And to list one very specific example to make this really concrete for a second, there's this protein called c-Myc. Okay, c-Myc doesn't— the name doesn't really matter in this case, but fundamentally there's too much of this protein in over 70% of human cancers, and people have been trying to drug this protein for decades. And so it's a protein, and if you remember your fundamental bio, right, of DNA goes to RNA goes to proteins, If you go one step earlier in the process, you're effectively increasing the attack surface and giving yourselves new ways of going after these diseases. Okay, so that's kind of the dream of RNA. It's like, we'd love to be able to go after RNA. And so very similarly, we'd love to be able to model the structures of these RNAs, build the AlphaFold for RNA, so to speak, to be able to cure these undruggable diseases. The problem that really emerges here is that there are a lot less known RNA structures. In protein land, we know hundreds of thousands, millions, depending on how you count. We only know a couple hundred of these RNA structures. But there really is starting to be evidence that there are orders of magnitude more of these kind of ripe for their discovery and for really enabling this RNA drug discovery field. And so this is where a lot of my work that initially started at Stanford really kicks off. And the first take that we did on this was essentially that of being clever, so to speak. You can really train these specialized machine learning models on molecular structure. In this case, a thing which we call an equivariant graph neural network that encodes the right rotational symmetries into these graph neural network layers to really be able to train it on a very limited amount data, in this case 18 RNA structures, to learn what makes a good versus a bad RNA structure. And fundamentally, the model just looks at an RNA and splits it out into its individual atoms in 3D space. So all you're encoding really is their 3D position and their element type, so carbon, nitrogen, oxygen. Then you do multiple layers of, you know, GNN layers of these equivariant layers essentially to learn an embedding for each atom. And then average over all of these atoms into a dense neural network to predict a final sort of measure of quality, essentially. And what you can find is that if you do this right and do the right sort of data augmentation and the right sort of model construction, you can actually do quite well. And in fact, this let us consistently win these international blind competitions with this method and really led to our work, which ended up being featured on the COVID of Science. Now, this is all well and good, but there's sort of a limit here as to how well you can do. In this case here, what I'm showing the top right is sort of a rough measure of accuracy over time, right? And, you know, just very simply, you know, the Science paper represented an initial inflection point. However, the problem was far from solved at this point, and what we realized is that really the key to making progress and further progress here was really in some ways the bitter lesson of more data, more compute, and these large language models at the end of the day. And so the second take was basically these LLMs with this custom-generated data. And so we've actually designed these custom in-house wet lab techniques to generate huge amounts of data at this point and are continuing to scale this. And really this is enabled at the end of the day by this dramatic decrease in the cost of DNA sequencing, which in fact is also RNA sequencing. And so what you can see is that the cost of DNA sequencing has been dramatically falling over the last couple of decades at a speed that's even faster than Moore's Law, which is kind of crazy once you start thinking about that. And so that really enables us sort of very uniquely to start generating these very large sort of sequence-level datasets around DNA and RNA. And in fact, Just in the last couple of months, we've generated a billion-plus token dataset and are scaling that up by another 1,000x over the next coming months as well. And so this really lets us train these models on the sequence-level datasets to create essentially these models that understand RNA, can be then fine-tuned to various downstream tasks such as structure prediction. And in this case, that has led to a dramatic increase in our accuracies overall. To the point that we essentially can very confidently measure RNA structures in a way that was previously not possible. And it's probably about 3 times more accurate than the original ARIES algorithm from the Science paper of my work from a couple of years ago. And the key to a lot of this, again, is really this tight integration of both the algorithmic development, of the development of these LLMs, et cetera, with the right sort of wet lab models, right? The right sort of data generation techniques. And because you can scale these to such an absurd degree in some ways, really creates this nice opportunity essentially for generating what we call this RNA structure engine, which is really just this integrated bio and AI sort of engine that's there to discover and drug RNA structures. And so on the one hand, we have the AI-enabled computation, right, that can guide sort of targeted experiments, right? But then on top of that, the experiments can further refine the AIs and really be taken in the areas of low confidence, essentially, in this sort of active learning type process. And so overall, this is built in this sort of tight virtuous cycle, as Andrew Ng would put it, perhaps, to continually improve the AI and enable its predictions of RNA. And this, again, is really achieved by not only the right algorithmic advances, but by putting the right sort of team of world-class AI experts in the same room as the number of RNA chemists and biologists and really figuring out how to best integrate these together. And so overall, we've built this, what we call this PARSE platform. Name, I suppose, doesn't matter so much, but it's the platform for AI-driven RNA structure exploration to discover and drug these RNAs. And at the end of the day, the applications of this platform are quite broad. There is many different areas. You can design typical small molecule drugs for cancers or infectious diseases or neurodegenerative diseases. Such as Parkinson's or Alzheimer's. You can help design the next generation of mRNA vaccines to increase their stability so you no longer need to freeze them or avoid these sort of immune-triggering structures, as well as many other things involving gene therapies, which have really been taking off lately, and all sorts of other kind of more esoteric, I suppose, RNA applications across the space. But from a machine learning standpoint, for a second here, there's really this notion that at the end of the day, you're building these models in sort of structural biology, and in our case specifically for RNA. And we really see this, pulling back on my computer vision roots for a second, as kind of an ImageNet moment for molecules at the end of the day, where being able to predict single structures is really only the beginning of what you can do with this kind of work. And that there are a lot of very deep sort of technical machine learning problems at the end of the day that really become enabled by this, including, for example, then predicting interactions between molecules. How do these different molecules interact with one another, right? Which is really key, right? They don't exist in a vacuum. Really how molecules work is they are all operating together to form the engine that is the cell. Or really the flexibility over time, which is then much more of a conformational flexibility. And people have been looking at normalizing flows and now diffusion models for this kind of work, where you're really— these molecules aren't static but are changing shape over time, or even getting at the design of completely novel molecules on this side of things to generate new RNAs that do completely new things. And by building these kind of foundation models that have learned the fundamentals of RNA structure, you can really fine-tune them to these tasks in nice ways. And because it's also related to the structure, you know, and it's fundamentally based on the laws of physics, you actually get some very nice generalization properties as well from this whole thing. And, you know, I wrote a little bit of what I call an opinion piece on this kind of work, uh, this Atom3D tasks on molecules in 3 dimensions, really laying out how there's this whole sort of wealth of different interesting problems in this sort of structural bio slash, you know, RNA space that you can tackle through these machine learning methods. And I guess it's really what I want to leave you with here. In a lot of ways, there's sort of this new field of structural biology that's emerging with these homogenous data types, atoms in 3 dimensions, with all sorts of interesting properties to predict off of them, fairly technically interesting from that standpoint. But at the same time, at the end of the day, if you get this right, and you're solving the right problems, you have the potential to cure diseases that are currently incurable. So going for that, you know, global social good impact at the end of the day. Great, and I think that's really it for the presentation here. Thank you so much, and yeah, I guess we're doing questions now, right, Dan? Is that correct?

Thanks so much, Rafael. That was a really, really great talk. I'm just going to stop your screen share real quick so that the, the audience can, can see our faces. Um, all right, cool. So, uh, I— we will start with the Q&A now. So if you're watching on YouTube, you have any questions, feel free to post them in the YouTube chat. I saw, I saw you got one thumbs up from YouTube chat already, so, uh, people are, people are liking this. I think they're, they're really enjoying the talk and the content. Um, I guess I'll start us off with just a couple questions that I had. Um, so one piece that you mentioned is that you mentioned something about we don't know all the RNA structures that are out there. And I think you mentioned that there are a lot more that are out there. When I think about training a foundation model or a large language model over this, are there concerns about generalizing to new structures that you haven't seen before in a way that, for example, on human language, we have so much text that you've seen all the different— there's only so many ways to say something in English. Are you concerned about that at all, generalizing to new structures that we haven't seen before experimentally in the lab or anything like that?

It's definitely a concern and it's a great question, first of all. I only alluded to this. I didn't want to get too much into the weeds of the specific experiments that we're running today or things like that. But effectively, you can generate data that's effectively structure-adjacent data, like ground truth data, if that makes sense. You can train effectively these semi-supervised algorithms on it, where you're using the structure-adjacent sequence-level data from your DNA sequencers, essentially. In learning that mapping from sequence to those readouts, your model ends up encoding a lot of information about structure at the end of the day. Then once you take your limited structural dataset, you can fine-tune to that relatively easily, even given the limited set of data. Out there. Now, that's all to say, we still end up testing a lot of these in the lab afterwards. Once you have a hypothesis for what something looks like, you can definitely check it experimentally as well. But that's a much easier process than trying to derive the whole thing from scratch. The last piece I'll say to this whole thing about generalization is, it is kind of funny, you mentioned there's only so many different ways you can say things in English. Well, fundamentally, these structures are based on the laws of physics, and the laws of physics do not change across different structures. So if you're learning the same concepts, there's actually this promise of being able to generalize that to new structures as well.

Gotcha. So there, in some sense, it may actually be, you know, a more tractable problem because it's just physics at the end of the day, whereas, you know, I can say anything. Yeah, exactly. Great. So another question I had, maybe that would be interesting for our audience. As you're training these models, I think you mentioned you use attention in various ways. Do you see any issues with modern deep learning architectures that get in the way of this? One that comes to mind for me is DNA sequences or RNA sequences are probably very long and attention scales quadratically. Are there things like that that the ML community should be aware of and should be trying to develop maybe new techniques or new architectures for that would be particularly useful for structural biology problems?

Yes. So actually, you sort of took away my first point for me in a way. But yeah, one of them is you need long-range attention. Some of these interactions are happening like, you know, 5,000 bases apart, you know, 10,000. Some of these sequences are 100,000 long. And you want to like predict the correct interactions across these very long ranges. And that's actually key in many ways. It might be actually the most interesting things to design novel drugs around. And so, you know, I know there's been a lot of progress in terms of these more efficient kind of attention layers and things like that. And there's definitely a need to import those into this space at the end of the day. Right. And, you know, it's actually kind of funny when you get down to it, like For all the science I was talking about there, at the end of the day, you've got these sequences, right? You either have a 4 for RNA character alphabet, ACGT, right? Or ACGU, I guess, I suppose. And then for proteins, it's a 20-character, the 20 amino acids. But it's the 1D sequence, and you're trying to learn a lot off of those, essentially. Let's see, to list a second sort of area as well, the other interesting piece of this is that the the problems you're predicting at the end of the day are actually fairly uniquely structured. Like, imagine— like, I only alluded to this briefly, but you've got a large language model and then you have these modules on top of it that's then outputting a full 3D structure. That's kind of a fairly specialized data representation. And how do you build a model that can map from your embeddings to this sort of weird output in some ways is also pretty key. And I think it's a pretty fascinating area of research in and of itself. Right. And that comes back a little bit to what I was talking about, about like atoms being its own kind of ML data type, right? They've got rotational symmetries, they've got translational symmetries, they've got permutation invariance, right? And like, how do you encode all of that correctly into one of these sort of, you know, large models?

Right. Yeah. It's a really interesting question because it's got, you know, structure in a way that I'm just thinking about language because we were talking about language models. There's kind of structure in a way that isn't necessarily the case, necessarily going to be the same in language. Like, you might have a very long essay, but the chances that you need to reference some particular word that happened 2 pages ago, and that is going to change things. I imagine those long-range dependencies that you mentioned are quite critical for structures, but You don't quite kind of see the same things in language necessarily.

Yeah. Yeah. And there ends up, you almost end up needing, like, you can use a lot of the tools from NLP or computer vision or what have you, but then you also find the need to start developing your own new stuff. Like a lot of this equivariant kind of work is specifically driven from the fact that you have rotations in 3D. And if you're rotating your whole molecule in that space, it's still the same molecule, but then like. Your forces rotate, and that's a vector, and you might want to predict that, and that's a whole different ballgame at the end of the day.

Right, cool. So we're getting some questions in from the YouTube chat, so I'm gonna go over to those for, for a little bit. I also want to just encourage people to keep sending those questions in there, they're very interesting. So one question from Pat Langley is, ML researchers have been working on predicting protein structures since the 1990s. What changed most since then? Were embeddings the key? Would they work with other induction methods, not just neural networks? Maybe some references like SimSys or something like that. So what do you think has changed the most since the '90s to make all this possible?

Yeah, I think some of it is in the embeddings. Some of it is definitely in the neural network architectures themselves as well. Like, I showed, for example, there is a plot where it's like alpha fold 1 and then alpha 2, right, or something like that, of the progress. And a lot of that progress was attributed— well, the first version was basically like ultra deep ResNets, essentially, and then the second one was really like Transformers is what pushed that forward on that side of things. And so these, right, sort of like high-capacity models at the end of the day ended up being very key there. Um, on the embedding side of things, there's also this realization that there's certain sources of data that were very information rich. I'm talking about specifically there's evolutionary data. There was, it turns out that you can like look at the same protein across different species and you can see certain positions that mutate, but that are correlated mutations. So if like one position mutates, then one really far away might also mutate in tandem. And what that tells you, this co-evolution, is that those two pieces are actually close together in 3D space. But it's not quite that simple, and you need the right sort of statistical algorithms or large language models, et cetera, to really extract what exactly that means at the end of the day.

Right. We have another question from the audience. So can these foundation models that have been trained on molecules, DNA, RNA, proteins, can they be applied to predict behavior of maybe inorganic chemical molecules, for example, in material science?

Yeah, it's, it's, it's, that's, you know, a fascinating sort of line of research that people are sort of actively pushing in. I think the short answer is currently no. However, I think it is possible to do so. And, you know, it's, you know, the thing about like RNA or proteins or things like that is that while they're fairly big molecules, they're fairly regular. They're called biopolymers. They're built out of the same thing repeatedly. And so they have the same building block. So if you look at the end of the day, it's not covering the space of every single possible atom type or things like that. But then other kinds of materials oftentimes, like, you know, like, you know, if you have like some silicon in there or something like that, right? That's not an atom that you're going to see in an RNA molecule, for example. So collecting the right sort of data to augment it and train on these other kinds atoms, basically, is— I see as the key moving forward there. But fundamentally, you know, it's all physics at the end of the day. So if you can collect that data, there should be, you know, capabilities to generalize nicely.

Exciting. Maybe, maybe the secret to our robot overlords.

Perhaps.

We have one more question from the audience. And so are you looking to— what use cases for generative models are you looking into? I think you mentioned diffusion in there. Are there other things? Does it make any sense to generate new sequences of RNA or DNA?

Yes, that's actually a huge— there's a huge amount of excitement in that direction specifically at this point. For example, you want to design a molecule that, I don't know, binds to a different molecule. And you just specify, I want you to be able to bind there. Now generate the thing that looks like— it's sort of like conditioned generation at the end of the day. You can also do completely unconditioned, just say generate me a bunch of different proteins, which is kind of interesting in and of its own sake. But you can also condition it on specific functions. Say, hey, design this thing that does that makes an interesting interaction or that has this function. And there's actually exciting work, like people are trying to design the next generation of flu vaccine using these kind of generative technologies. I'm talking about diffusion models. I mean, previously people were using GANs or normalizing flows or what have you. But at this point, look at the latest workshop on this stuff at NeurIPS, and it's all about diffusion models these days. That's really you know, what everyone's very excited about.

Right, cool, exciting stuff. So I know we also have a lot of questions from CS324 in the Discord. I'm going to point it over to Michael and let him start asking some of those questions.

Cool, yeah, of course. And Raphael as well for a really interesting talk and lots of interesting questions as well to ask you about from the class. Yeah, actually kind of related to that last bit about the diffusion. People are wondering sort of with respect to, let's say, actually doing, let's say, some sort of like targeted design. Let's say you want an RNA structure of sorts, right? Yeah, I've done some reading in this and there's this question of, okay, like if I want, let's say, a Stable Diffusion for RNA sequence or some something that will pull it up like an aptamer of sorts, right? Like, do you think that we should be conditioning them on text or is there perhaps like another modality, something else that you want to train them with? Or how would we impose, let's say, these structural properties or things of interest.

Yeah, there's a lot of different ways you could do it. It could be on text, like natural language at the end of the day. You can also condition it on a target molecule, like if you have another structure, for example, of like, hey, I want you to build something that can bind to this molecule specifically. So that's a very different kind of input data type. Another one is really, oh, you want to paint a scaffold and say, I want the molecule to roughly look like this, right? And, you know, almost like, you know, doing an MS Paint, right? And then letting it really fill in all the details on it at the end of the day. I think one thing I want to highlight about this is this field has been incredibly neglected from an AI standpoint for a long time. And so each one of these things I'm saying, like, if people are working on it at all, it's like barely started. So there is, I don't know, I think I could probably list like at least 20 different research directions that no one's looking at right now that people definitely should and would be very impactful. Actual map, right? So maybe that's all to say that this conditioned aspect, there's a lot of ways you could do it and there's been very little work into looking into exactly how to set that up.

So, actually wanted to jump in there because you said something really interesting there, which is there are all these, you know, really interesting questions that nobody is looking at right now. You can just list off 20. From a, maybe a structural perspective, from being in academia or industry or what have you. How do you think we can set up those incentives to kind of get people to work on these questions, to put people in contact with the right people so that the ML people know what to work on and the biologists and the chemists know what—

I can go on about this stuff for hours. There's a reason I started this company in the first place, let me tell you. I mean, I think a lot of it is a sociological problem at the end of the day. Like the way biology is taught in high school or in undergraduate is not like, you don't really realize this is even a thing. Like, you know, if you'd asked me in undergrad if I'd wanted to do bio, I'd have probably said, hell no. Like, this is like the, you know, I don't want to be working with test tubes all day. And so I think a lot of it has to do with getting the right sort of courses or exposure to people, setting up the right sort of things, like even at the undergrad level to show, hey, Like bio, like there's a whole thing known as biophysics, for example, right? That is like, you're really, you know, you're at the intersection of biology and physics at the end of the day. And that's a very sort of analytical, like quantitative space. And I think making that clear and exposing people to that is key. You know, given that, you know, that's not set up currently, I think the other way to do it is to like, you know, just have more venues for these kinds of conversations. You know, we started this workshop, you know, 2 years ago on machine learning and structural biology at NeurIPS. And that was literally just me emailing the 6 people in the world that I knew that worked on this stuff. So it was kind of an interesting time, to say the least. I think that there's definitely more excitement now, especially with AlphaFold having come out. That's definitely been a shove in the right direction. But it's still a small community in some ways. And I think there's a huge amount of very interesting, deep technical stuff to be done in that direction. Cool.

Well, I wish you luck with your workshops and your company, and hopefully, hopefully you will lead the charge in establishing this community that it sounds like we really need.

Hopefully, that's what I think, at least. Yeah. Everyone's worth it. Yeah.

But I mean, they've done a good job in terms of like, I guess, shepherding some interest. I mean, you did mention the AlphaFold stuff.

I guess isomorphic is doing some cool work too. Yes, to be clear.

But related on the sort of like the technical bit, a couple of the students were actually curious about, yeah, this connection between, let's say, making the jump from large language modeling to structural biology modeling. And so one thing was sort of this notion of like inductive bias where you mentioned before that you have like these covariant networks like, I don't know, I've said that to C3 and stuff like that. And so, like, it's clear that these, like, sort of inductive biases help with the modeling, but do you think that they are necessary, or do you think there are advantages from perhaps, like, doing something where we just, like, I don't know, don't have that constraint and just try to collect as much data as possible and, like, do this virtuous cycle of, uh, yeah, collect a lot of data, test it out, see what happens?

I think there's a balance. I mean, in the long run, sure, like, with enough data you can learn anything, I suppose. You just don't have to put any inductive biases in it, right? Bitter lesson in everything, as said before. But there's certain things that like are just fundamentally true about molecules that you probably could just model. And that would increase like your efficiency. Like the fact that if a molecule is rotated, it's still the same dang molecule. Like, seems like just a very easy thing to encode in there. And it's, it's worse than in computer vision, right? Because you actually have 3 axes of rotation. So it's not like you're not just doing data augmentation on a single axis, right? But you actually have 3. So that's like an obvious one. The SC3 stuff that we did is another obvious one, right? If you rotate your whole molecule, the forces at play are just going to rotate alongside it, right, at the end of the day, right? That's unless the laws of physics change at some point. I don't think you're losing anything by encoding that assumption into your model. So yeah, I think, and we're definitely not in the data regime yet where you can just throw all those rules out and do just as well is the other thing. Like even the AlphaFold models essentially needed sort have those equivariant kind of networks built into them to, you know, work properly. So it's, it's a balance, and perhaps it's a boring answer in some ways, but a little bit of both.

Yeah, I mean, it seems like useful, and it's like, yeah, like, I mean, you mentioned, right, like, if we're going with the laws of physics, might as well use what we know about it, right?

Uh, yeah.

Kind of like a slightly more pointed question, and just out of curiosity, was Iris submitted into the latest CASP competition? CASP-15, because CASP-15 does have RNA prediction stuff now, right?

Yeah. I think some other people used ARIES at this point. We did not actually submit personally at this point. I think we've been internally building systems essentially and making some exciting progress. But I think it's clear from the latest CASP competition that the broader community has not solved RNA structure prediction at this stage. It's like, I think the methods that won out were still very hand-designed, sort of manually designed approaches, right? I think internally we've seen some data that I think is very much compelling and pushing us to believe that, you know, it can be solved. But yeah, that's essentially where we're at. I see.

And are you allowed to say more about like what you see as the challenges or gaps between, let's say, maybe perhaps some of the more ML-driven or data-driven techniques? And you said sort of like, yeah, perhaps like these physics-based or hand-tuned things seem to be outperforming, at least in the RNA prediction.

Yeah, I mean, it's really about the amount of data out there to train these things. I think with enough data, the ML methods would win out pretty clear as day. It's just that right now the amount of publicly available data is minimal, right? And I mean, you know, not to get into the weeds too much about economic or anything like that. But, you know, at the end of the day, we realized, hey, you know, we could probably, for the right investment, actually build out these data generation capabilities that no one else has, right, in specific ways. And I mean, it's not cheap, we're still talking like millions of dollars at the end of the day, but that lets you really scale. Like, there's some really nice economies of scale essentially that start kicking in at that level. And so at least the decision that we came to was like, hey, we actually need to build out a lot of that data capability in a way that would be inaccessible, at least, to a starting faculty member at a university or something like that.

So speaking of that, I have a quick follow-up on that. So if you can talk about any of this, or if not, just blink twice and we'll know that your investors are holding you hostage. What is the— can you talk a little bit about kind of what the plan for Atomic is and what your vision with Atomic is and how you see— you mentioned this data aspect, kind of like what are the other pieces of that vision?

Right. So at the end of the day, we're building this sort of like RNA foundation model. And I listed out like 6 different applications there of that kind of technology. And so really, we built this sort our core platform. We're continuing to develop it and build all these additional capabilities, right, that I was also mentioning, like interactions and design. And we're going and partnering with pharmaceuticals, major biotechs, to sort of— in these sort of technology platform partnerships to supply that to them. So, you know, the model in some ways is sort of this co-discovery kind of model where we can work together with the pharmaceuticals to discover new drugs. Um, and, and this works very well because we Essentially what happens is we do all the R&D essentially of finding these things, but then we don't do like the clinical trials, the marketing, or the manufacturing or anything like that. That's sort of like the big pharmaceutical. So like the analogy I like to give is, you know, you have the BioNTech-Pfizer vaccine, right? Well, BioNTech did the early innovation, Pfizer did, you know, the manufacturing, clinical trials, and, you know, um, they split that 50/50 essentially.

And, uh, you know, so Atomic would be playing a very similar kind Is there a plan to kind of open source all that data that you'll be collecting and doing that so that you can kind of jumpstart the whole field and get people working that cycle?

The hope is very much more to communicate with the— to collaborate with the academic community, especially in the longer run. I think it's key, like, in some ways to increase the interest and the capabilities in this field. And frankly, I've been an evangelist for this space for, I don't know, 8 years at this point. And I have no intention of stopping that. And I think the more people are able to do this kind of research, the better. We're definitely not in a state where it's super crowded and a million people are working on the same things or anything like that. Cool.

All right, back to Michael for some more questions from guests.

And so I guess another question related to this aspect of data is, at least so currently, I guess, like, what is the limit of sort of like collecting sort of like these ground truth or gold standard structures, sort of just solving these structures in, I guess, real life? Like, do you just need a bunch of, let's say, cryo-EM machines or things like that to actually just find like the different structures and add that to your supervised dataset? Or can we like bootstrap techniques from, I don't know, like use I guess AlphaFold wouldn't necessarily be useful for RNA stuff, but could we eventually imagine doing some sort of bootstrapped—

Yeah, yeah. I mean, or you can integrate it, right? So the, like, cryo-EM, and partially the reason it's so expensive, right, is you have to collect like millions and millions of images of the same molecule before you get enough data to figure out what it is. But if you have like a machine learning potential, right, that gives you a nice prior for it, you could, you know, dramatically reduce those data collection needs at the end of the day. So you could combine it in that way, right, to make it much more efficient that way. Or you could redesign the cryo-EM experiments, right, because those are oftentimes primarily focused on a single molecule, right, but you're sort of marginal decrease in information content if you're collecting the same image of the same molecule over time. You can instead throw 1,000 molecules into the microscope at once and collect it that way, right? As far as I can tell, no one's done that before, but that seems like a very interesting direction as well. And so I think there definitely needs to be the right sort of methodological development, both on the AI and the structural bio sides of the equation to like push this forward at the end of the day. But then on top of that, it's also, you know, like really just, you know, collecting that data. But if we just use the techniques that exist today, right, and just naively tried to use them to collect this amount of data, like a single cryo-microscope can like cost like $10 million, like a good one essentially. So it's not— and then you spend like a month to get a single molecule on it, right? It's not like, naively speaking, you would be spending a heck of a lot of money essentially to get that done. And in the RNA space specifically, there's a lot of RNA out there and most of it is not structured. So finding the needle in the haystack in that way would probably not be the most productive use of your time. Um, got it.

Yeah, that's super interesting. Um, yeah, you can also sort of like tying these priors with ML to also create better datasets. Uh, Chris would like—

Yeah, yeah, you could also like— another sort of proposal that I wanted to work on, I never had the time to, speaking of another direction that I think would be really interesting, is putting in priors for X-ray crystallography, which is another sort of structure determination technique out there. Interesting. Very good.

I guess so another question going back to sort of the actual task of structure prediction. Can you speak to any sort of, let's say, interpretability metrics you might employ to understand sort of like what the model is learning? For example, visualizing 3D structures and mapping that perhaps to like embeddings, how this is—

Yeah. I think that's a fascinating question. And because in some ways there's really this potential of like these models have learned things that like humans are not able to do, which I think is fairly interesting and a bit different from like, I don't know, like NLP or computer vision or things like that, where it's like learn concepts that we would love to be able to extract back out, like learn like fundamental, like biological knowledge. And so I actually think interpretability in this space is super fascinating. And we've done some basic work like that in the science paper where like, hey, our RNA module model has spontaneously learned about base pairing or, you know, number of hydrogen bonds or things like that. Like you can pick out specific neurons that have like learned those concepts. But backing out novel things, still open research question. Like, you know, the ARIES model does better than any of our hand-designed things. And the question is why? And I don't think we have a very good answer to that still. To this day, right? And similarly for AlphaFold, like, you know, pulling out what it's doing and why is, I think, you know, I mean, if you can learn that, that actually could then help biologists worldwide and probably improve like what they're doing essentially. So yeah.

Yeah, that's pretty interesting. Slightly related as well to that is I've always been kind of curious about, and also students have been curious about this notion of, let's say, let's say the emergence, right, where— or I guess I should just call it emergence— where, for example, like you just take like that simple task of like next word prediction, maybe in this case it's next amino acid or next like base pair prediction, and then like the model is seemingly able to learn, let's say, like these more general structures or things like that. And so do you think like something as simple as like, I don't know, causal or masked language modeling over sort of like DNA or RNA sequence, like can take us really far? Like, yeah, like what's your stance of emergence of these structural biology models?

I mean, we're already seeing emergence like that from these models is the first thing. Like, you know, you can take like these mass language models or things trained for completely different purposes and like, for example, like I forget the details, I think there was a group like at a fair, they trained some sort of fairly standard sort of loss term on an LLM and they found that 3D structure just spontaneously emerged from it essentially. Like they're able to just train a simple like linear layer on top of the embeddings and directly predict 3D structure from it, which I thought was pretty cool on its own. And so— ESM stuff? Yeah, yeah, I think it's ESM, or maybe it's a follow-up to that, like Roshan Rao's kind of stuff. They're doing some cool stuff over there too. Yeah. Yeah. Okay, cool.

Yeah.

Yes, I see that there's another question from the YouTube chat, so, so let me take it back and ask that one real quick. So I'm just going to read this one out loud because there's a couple words that I don't quite understand, and hopefully you'll be able to interpret. So the question is, can you speak to MA-based drug design using synthetic biology and predict function?

And does it— N-A, like the letter N, like Nancy?

And like Massachusetts, I guess.

MA-based drug design. I'm unfortunately— actually, I'm not familiar with that term, if I'm being honest. What was the rest of the question? Actually, say it one more time and I'll see if I can infer.

Yeah, that's mass spectrometry.

Oh no, it says RNA.

Oh, okay, okay.

All right, thank you, YouTube chat. Classic, classic Classic, classic problem.

So I guess— so what was the question?

Okay, can you speak to RNA-based drug design using synthetic biology and predict function?

Yeah, um, yeah, the interesting thing is, and the thing that's got everyone really excited about these sort of AI structural biotechniques, is that you have this forward process of going from a sequence to a structure, right? And once you have that, if you can do that quickly now, like do that in a second, then you can start running the whole thing in reverse, right? And designing to a desired structure instead, be like, just run stimulated annealing or something on sequence space and find, you know, the one that folds into that structure, essentially. And that's, you know, people are pretty excited about that because if you can design these structures to specific forms, that's really getting at desired functions. Now, I don't want to say structure is the entire piece of function. There's still a gap there of like, even if you design for specific geometry, that doesn't guarantee it's going to do what you need it to do, and that's in fact a challenging problem of its own. But, you know, there's really— you're opening this possibility to do that kind of generative design of designing novel RNAs themselves. Like I mentioned, the mRNA vaccines, right? If you could design them to be more compact and stable so that they, you know, you don't need to freeze them anymore, as, you know, a very direct example that I think I mentioned before. And just overall, right, there's all sorts of these RNA technologies, including CRISPR, for example, that includes RNA in it as well, or like gene therapies and things like that. It's just— and right now we're still very much in this realm of almost like, um, a lot of medicine is done in this guess and check kind of way of like, let's try things, throw things at a wall and see what works. And the dream is really instead moving to the state where we understand what's going on, understand the physics, the structures of everything, and then we intelligently design or rationally design the molecule to fix that. And that's sort of at least part of the dream of synthetic biology, but also of a lot of this structural bio kind of space.

Great. Yeah, that's super exciting. I think we have a couple more questions in the class.

So let me just turn it back to you. Keep them coming. Love it.

Sounds good. Yeah, I think I might ask like 1 or 2 more. And I guess they're both sort of more forward-thinking. So one question was related to like this, maybe like a criticism that has floated around with respect to, let's say, structural prediction now is that, okay, like it's gotten so good that it solved the benchmarks. So it's like solve the task that we can, we sort of have posed. But like there's this question of like, well, I think you mentioned this as well earlier. Like, so biology is not static, right? We're not just bringing a single structure. We should be predicting, let's say, I don't know, things that are moving around kind of sort of motivating molecular dynamics.

And all of that, right?

Yeah. So like if you're, if you're able to share, if you have any thoughts on this, I'm just curious, like how do you think we should go about it from like this task of, let's say, going back to, let's say, I don't know, your image roots, like image generation to like the video generation of structural biology.

Right. Yeah. Yeah. No, there's actually some great work that's been coming out recently specifically about like, okay, like even if we believe we've solved static structure prediction, which, you know, has an asterisk, unclear whether that's really the case or if we've just gotten really dang close at this point. But even if you have solved that, that's really not the whole problem by any means, right? And so maybe this is a way of saying that I agree with what you're saying, is that, yeah, it's only one piece of the puzzle. It's an important piece, but really molecules and RNA in particular actually is quite flexible as a molecule. And so really being able to model its dynamics is actually key in many ways. Like, I don't know, if you took a single image of the world, right, you probably would learn a lot less than if you had a running video of the whole thing. In fact, to give a very concrete example, some RNAs are called riboswitches, and specifically they switch between two different shapes, and that's how they work. And sometimes they like chop themselves off, they cleave themselves essentially, and that's like a key mechanism. You can see how those two different shapes are key to what it does, right? It has to have like the open state and the closed state to like, you know, switch to these different things. And if you just modeled it as a single structure, you would miss all that intricacy and nuance at the end of the day. And like, for example, say you wanted to design a drug to lock it into one of those states. Well, if you modeled the wrong one as your prediction, then you're kind of out of luck, right? You want to model the one you want to jam it in essentially. And so Maybe that's a way of saying that you definitely need that flexibility. Like my own lab, Ron Rohr, who was my advisor at Stanford, fantastic advisor, right? Really, he made his name on molecular dynamics and investigating ensembles of structures. And you cannot fully understand these things without getting to that level.

Do you have any thoughts on how we could get there with machine learning?

Yeah, I mean, a lot of these, you know, these sort of generative models essentially, like people— there was paper like Boltzmann generators that I think was effectively a normalizing flow trained to sample different ensembles of conformations, but that was trained on one protein and then generalized to other states of that protein. I think you could, with the right data, generalize that kind of what and probably more modern sort of generative models as well, like these diffusion models. If you collected the right data, sort of essentially create these ensembles of shape for each molecule. Instead of predicting a static 3D structure, you're instead predicting an ensemble in that case, and you can then sample from that to get different states, essentially. I think that's a key piece, and for that you essentially need the right data as well collected underneath the hood. And, you know, that it's entirely possible that data is present, but this is again something that, you know, really people only starting to scratch the surface of. A good friend of mine, Ellen Zhang, who's now like faculty at Princeton, is actually working on this specifically of, you know, essentially extracting, you know, from cryo-EM data specifically ensembles of conformations out as opposed to sort of using that Oftentimes for cryo-EM, that's seen as a source of noise, the fact that it's flexible and you're trying to average that out. Obviously that increases your data requirements, but also you're leaving a lot of information on the table there. So if instead you can leverage that with the right sort of ML techniques, then you can actually get at this kind of ensemble data.

Awesome, thanks so much.

Thanks, Michael, and thanks everyone in the class and on YouTube for those great questions. So we're nearing the end of the hour, and Rafael, I just wanted to give you a little bit of time to talk about what is kind of exciting for you, what's coming up next, maybe speculate a little bit about where you hope this field will go, and how you— if you could wave a magic wand and change the field as you wish, how would you go about it? What would— what is kind of exciting for you?

Looking forward like that? Yeah, I mean, I'm excited moving forward of, you know, I mentioned the sociological aspect of this, of like more sort of ML people like realizing, you know, structural biology in the field of biology as a whole is, you know, in many ways a very quantitative field with like very, you know, very apt for machine learning, right? And at the same time, also getting the biologists to buy into that because there's certainly backlash on that side of, oh, this is, you know, the dark arts or something. And like, they try it once and it doesn't work. And then like, oh yeah, machine learning, it's just a load of crap essentially. But so, you know, I think people are slowly moving in that direction and I'm excited to see that continue moving forward. Because I think that will lead to the generation of new datasets that are sort of key to solving a lot of these critical issues on one hand, and the development of more models that are sort of both inspired from the existing machine learning network, but also very novel kinds to solve a lot of these key issues at the end of the day. And so, you know, if I could wave a magic wand, it would probably be, I don't know, somewhat similar to what I just said before. It'd be great to have some like undergraduate courses that like actually get into this a little bit more, like, you know, give you exposure in a way that doesn't present biology with such a, you know, classical view, or I mean, the classical view is fine as well, but adds to that picture as well. Um, so that's, uh, maybe, you know, that's at least the direction I would push this whole thing in.

Um, right, yeah, that, that's really interesting. Maybe, maybe you don't need to wave a magic wand, maybe just email the right professor, be like, hey, your biophysics should be—

yeah, maybe, maybe that's the easiest thing. At least that wasn't my experience. Maybe that's already being done in certain places, but I think it could be done more generally.

You know, we academics are pretty stubborn, so maybe you do. All right, so we're nearing 4:30, so I wanted to thank Rafael again for coming on and giving a really interesting talk, teaching everybody a little bit of biology. I'm sure it's good for us, and I'm sure we all learned something really interesting. And really excited just to learn about this field and hear everything exciting that's been going on.

Yeah, thanks so much for having me. Yeah.

On Wednesday, we have Rob Reich from Stanford, and he's going to be talking about some interesting things. So if you want to kind of see our schedule for the rest of the quarter, you can go to mlcis.stanford.edu or go to the CS324 website, which I don't remember the URL, but I'm sure you can get from Google. And so if you go to our website, you can find our schedule, more about the seminar and the class. We also have an email list that you can go on to get announcements about these talks. I think we send two 2 or 3 emails every week just announcing who's giving the talk. Yeah, so I want to just say thank you to Rafael again. It was a really, really great talk. And with that, we'll wave goodbye to YouTube and say thank you, everyone.

Bye.