Scientist Stories: Eric Nguyen, Using AI to Design from the molecular to genome scale

and he'll be around for Q&A at the end to present EVO, which I'm sure we've all read a lot about, seen a lot about. But to briefly introduce Eric, he's a PhD student at Stanford in the bioengineering department, advised by Steve Bacchus and Chris Ray and Brian He now. And he's a part of Hazy Research, so Chris Ray's lab, and also now the EVO Design Lab. So I think I'm sure a lot of people will have questions and things to talk about at the end, but Clarification questions, perfectly fine throughout. You can also leave them in the chat and I can read them out. But just save sort of like the more meaty discussion questions for the end. Yeah, with that, I'll give it to Eric.

Awesome. Hey, everyone. Thanks so much for showing up. Really excited today to talk about recent work of ours called Evo. And this project is all about treating DNA as a language. If we think of a person and their genetic code as a book, this book has 23 chapters of chromosomes and 3 billion characters of just 4 letters, this small vocabulary, which sounds simple but is responsible for everything needed to create life and create you as you are. I think this analogy is really appealing to think of for AI because We know that AI is quite powerful for natural language, things like ChatGPT, powered by things like the transformer, which have shown to be really successful when you throw a lot of compute and data at it. The cool thing is about for DNA, there's also an explosion of DNA data now emerging. And that's due in large part because the cost to sequence DNA has dropped quite a bit from the hundreds of thousands to just a few hundred dollars to sequence a whole genome. And we care about this because learning from DNA, because there's still a lot we don't understand or know about DNA. And so if we could model this better, perhaps I can unlock the ability to better understand disease, improve drug and target discovery, and also make possible potentially personalized medicine, right? Imagine if you could, for example, fit in context an entire human genome, feed that to ChatGPT, and then be able to query questions about what's in that genome for diseases, maybe predict drug reactions, or even guide treatment options based on your specific DNA. There's a long way to go. There's many things we just still don't understand in DNA and biology for sure. But in this work, we focus on one particular area, which is captured quite well, this, this quote that I love, which states that today we can, for all practical purposes, read, write, and edit any sequence of DNA, but we cannot compose it. And this was said by Francis Arnold, a Nobel Prize winner in chemistry. And what they were getting at was that there's still a big gap in being able to understand DNA to the point where we can design it, right? Designing DNA. And so a group of researchers at Hazy, myself included, teamed up with Brian He, who's joined here, who just recently joined Stanford as a new PI in chemical engineering and data science and an investigator at ARCC. We asked the question of, can we design an entire genome from scratch using AI? So this is sort of that pie-in-the-sky research question that kind of motivated myself in particular, perhaps naively, because we thought if we could do this, right, maybe we can potentially create organisms that could target cancer cells, make better biofuels, discover antibiotics. These are some of the pipe dreams. Now, to be clear, we didn't get to do those things, but I'll outline what we did get to do today, what we get to do with this project, and dive into some of the challenges of learning from DNA effectively, the innovation behind a new architecture that we call Hyena, and the results and new capabilities that we're really excited about that Evo helped enable and still have work to do to improve upon. Okay, so what, you know, I want to first describe or talk about DNA and why modeling it can be particularly challenging. And one of the things that I guess, is focused on in terms of AI and biology is usually modeling at the molecular level, a single molecule. So we're pretty good at modeling proteins, complexes, groups of molecules, even RNA molecules, predicting their structure or their functions. But when you want to go from just single molecules to an entire cell, yet alone entire organisms, It's just orders of magnitude more complex, right? The task is quite daunting to get to from what we can model to what we want to model. Fortunately, everything needed for life is encoded in our DNA, from all the single molecules to different complexes for the entire genome of a species. And so the trick is, or the idea is, that biology will encode this information but encode it in different languages, right? So we know this from the central dogma, right? DNA encodes information for RNA, which encodes information for proteins, which are, you know, sort of the building blocks for life. The challenges, right, that most people have so far in modeling these types of modalities or sequences is that there's the understanding or belief that we need to have domain-specific models or specialized models for each of these modalities. So a separate one for DNA, a separate one for RNA, a separate one for proteins to effectively learn because they're so complex, right? In this project, we try to see if we can learn all across all of these modalities just from DNA. So that's our approach. So the team put together a large DNA pre-training dataset that we called OpenGenome, which was led by Matt Durant, who's an expert on prokaryotic and phage genomes, or what I like to think of as microbial life. So we put together 300 billion tokens of raw DNA sequences, unannotated, representing about 2.7 million genomes and sequences. And the idea is that we're going to train a foundation model or a large language model on this DNA. So we're going to sample from OpenGenome a random sequence, and then we're going to feed it to a model, Evo model in this case, a sequence of DNA and have it predict what's the next token or the next nucleotide in that sequence. And we'll do this essentially billions of times. And the idea is to have our model learn the distribution of the data, like all the underlying statistics, all the interactions of DNA or RNA and proteins as well. And we could take this pre-trained model and apply it to downstream tasks that we actually care about, right, from predicting to generation as well. So that's the full pipeline that we are setting up this project as. And we'll talk about what some of these capabilities and downstream tasks are. I just want to preview that there. Okay, so I mentioned we're going to model across these multi— these modalities, right, of the central dogma. But we're also going to model across scale, right, in terms of, you know, starting off with molecules. This is what most models today focus on. We're also going to try to model and generate systems, right, so groups of molecules with some kind of function that work together to the whole genome scale. That's where our aim is at, across not just modalities but the scale as well. So we're not working in a vacuum, right? There's models that do try to model DNA as a language, and there's been great work in this space from models like DNA-BERT or GenSLM, and more recently the Nucleotide Transformer, which are really exciting. Projects, but they are constrained by a couple of key technical things that we try to overcome with new architectures. But these constraints are twofold. So one is context length. But typically, they model at the 512 to 4,000 token range. And for sense of scale, the human genome is very large, 3 billion tokens, as mentioned before. And so this is just a tiny fraction of the context that we might be actually interested down the road. There's this other concept called the tokenizer that language models or foundation models need to decide on, which controls the resolution that you model your data. So typically, a token— this is a single unit of a language model— will represent a group of DNA characters, or in other words, DNA words. And this allows the model to basically learn patterns simply, more simply. So they'll have these BPE or byte-pair encodings that group into like 6 nucleotides or so, or you'll have k-mers, or there's another popular way for folks in bioinformatics. Okay, so what's the big deal about context length and tokenization? So we care about context length because a lot of molecules can be captured by today's language models, single molecules, but if we want to get to whole genome scale, then these things just can't simply fit into the models that are currently being used. And I'll talk about why that is. So most models in language or even in DNA or proteins use the transformer, which is powered by the attention algorithm. So I'll talk about some qualitative properties of attention. So attention is really powerful in terms of expressivity, meaning it can model your data in all interactions very well. Because it has this global receptive field, which means that for a given position on a sequence, it can aggregate information from all its neighbors essentially simultaneously. And so it's really rich computation, but it comes at a cost. So this is a time complexity measure that computer scientists use to measure efficiency of algorithms, and it scales in sequence length of O(n²), which means if your sequence length doubles, your computation requirement quadruples. And so if your sequence length grows, you know, quite large, like for DNA sequences, it just becomes intractable to try to aggregate information along that sequence. I do want to contrast this with another type of operation, a core operation in deep learning, which is the convolution. It sort of has this opposite property, set of properties, in that its quality of expressivity is considered lower because the receptive field is usually around a local neighborhood of aggregating information. But the trade-off is that it's much faster. It has a near-linear time complexity. And so in terms of capabilities, ideally we want to get the best of both worlds, right? We want to be able to have expressive models that are fast. And so we'll talk about how we can do that a little bit later. Going back to this tokenization concept, it's really about resolution, right? So we want high resolution. Models. In particular, for DNA, it's really important, more so perhaps over language, because of things like SNPs, or single nucleotide polymorphism, right? We know that a single change or SNP could alter the meaning of a sequence or a particular region by changing its genes, its functions, or traits. It also can change the difference or mean the difference of having a disease or not, right? In the right position, you know, a single SNP could mean the difference of having a sickle cell anemia, for example. There's also this concept of frame mutations where, you know, if you shift the coding of a sequence by one nucleotide, by like, let's say, an insertion, how that's encoding proteins will completely change, right? And so ideally, we have a model that could not be so fixed or locked into that regime. But instead, we want to model at this high-resolution level where one nucleotide equals one token. That's the ideal case for all these reasons. That's great. But the problem is if you try to do that, there is no large-scale language model trained at this resolution. They just don't exist. They're unstable to train. They're hard to train. They're hard to converge. And so they've never been competitive with lower-res models or lower-res tokenization schemes. And so last year, this is when we started working on new architectures for DNA that focused on these two problems: to extend the context and to increase the resolution to single nucleotide level. This paper we put out was called HyenaDNA, based on a Hyena architecture that I'll describe. We put it out last year. It got some, you know, interest from folks There's this fun quote I saw from an AI minister of the UAE, which I thought I'd share for kicks. It said that from ChatGPT to Midjourney through Hyena DNA, it is now undeniable that AI is no longer a hypothetical technology, but one that warrants far more government time and resources across the globe. I thought this was flattering. Thanks for saying that. But to put us in the category of ChatGPT and as capable as that is a stretch. That being said, it may have gone to our head a little bit. We wore t-shirts and we got really into the marketing, which was fun. To balance things out, to bring us back down to earth, we also got opinions from domain experts who we also really cared about because they're the end users. We want to make sure that we're making capable models. There's this quote from a person that said, For now, there is no serious evidence that DNA language models provide any gains in any useful downstream application. It's early days, so folks should still work on these, but there is zero evidence that this is the future of genomic sequence modeling. So I took this very seriously, and we went back to the drawing board, and EVO is sort of that result. But just for some context in the environment or headspace, I thought I'd share that for Kicks.

We have a quick clarification question, Eric, in the chat. Someone asks, why would a tokenizer with K-mer reduce the resolution if we define 4^k words? I think all of the information is still contained.

Yeah, that's a good question. So maybe the information is contained, but the model's ability to extract the differences at the single nucleotide or that high-resolution level could be more challenging to pull out. So it's theoretically there if you had enough data, for example. But I think our hypothesis is that it's a bias that you're enforcing the model to have that actually works against what you're trying to teach it. And so this has kind of showed up in a few different areas, and some people notice this pattern in coding as well. Programming coding languages where the models seem to struggle with like single character mistakes as it generates code, for example, or in math as well. And so I think this is especially true in DNA as well, but it's still potentially an open question of like what is the right tokenizer to use. It's just that what we observe in empirical results, at least for us, that we got better results using single nucleotide tokenizers. But that's a good question. All right, I'll talk now about the Hyena architecture itself to give folks a sense of some of the benefits and trade-offs perhaps of these models. So the starting point to the Hyena operator, the Hyena model, is actually a transformer block. We use this as our starting point. We'll swap out the attention layer, keep everything else like the MLP, which is a multilayer perceptron. We'll swap out the attention layer for what we call a HYENA operator. Inside the HYENA operator, from a high level, it's made up of two key operations known as what we call the long convolution and the element-wise gate. Now, these are two operations that we essentially are trying to mimic what attention is doing, but do so with more efficient algorithm or operations. So long convolution is going to capture that global context, and this element-wise gate is going to describe or capture this data dependency, which I don't dive into too much detail here, but we talk more about it in the Hyena paper if you're interested in that. Okay, so zooming into the Hyena operator itself, we're going to talk about how data actually passes through this layer, So the first thing you do, you project the input into 3 projections using a dense and conv layer, short conv layer. And this is similar to attention. You have a query, key, and value where you pass it through to get projections in attention, but we use them differently here. You take 2 of those projections and you just literally multiply them together to get an element-wise gate. They're the same shape tensor, so you can just literally multiply element by element. And then we'll pass it through a long convolution. So for those familiar with convolutions, right, it's— you have this kernel that slides across your input. This is done in 1D for a long sequence. The difference here from your standard convolution is that the kernel that you stride across is global, so it's the same length as your input. So if you have a very long sequence, your kernel is also very long. You take that output and pass it through one more gate. And then voilà, you have a, you have a operator. There's some more complexities about the implicit, about the long convolution that we describe more in the paper, but I'll just preview here for folks interested. It's an implicitly defined convolution, meaning the kernel has a separate function that parameterizes it, and we use the FFT convolution or a fast Fourier transform, which basically the key takeaway is when we do a convolution through the Fourier or frequency domain, we can reduce the time complexity to make it more efficient from n squared to n log n, which is a near linear operation. So it's much faster. Okay, so more recently, more interestingly, you know, we moved from a Hyena architecture for the DNA model the last year, and this year, or this project, we're using the striped Hyena architecture. So this is a Hyena architecture, but we mix in a few layers of attention. So it's a hybrid model. In this case, it's a 7 billion parameter model, about 1,000 times bigger than the Hyena DNA model we did last year, made up of 29 HiNA layers and 3 attention layers. We train it in 2 stages, where the first stage is trained in a lower context length of 8,000 tokens. In the second stage, we train it at 131,000 tokens. So this final architecture design was chosen after doing a pretty extensive scaling analysis on DNA. And so I'll describe what scaling analysis means. So, so far, scaling laws, right? This is a type of analysis done mostly in natural language, where the idea is that you're going to analyze the behavior and performance of a model as you scale it, as you increase the model size, compute, and amount of data. And the hope is that if you can model its performance predictably at a small scale, then you can project what kind of performance you should get when you go very large, right? Because these things can be expensive, and so you want some kind of measurable reliability in trying to train these models, right? What's interesting about this work is that it's the first scaling law analysis done on DNA. So it's the first one done outside of language, broadly speaking. And we do it across a number of architectures. So not just Transformers, but also other prevailing exciting architectures like Mamba and Hyena and the hybrid StrikeHyena model as well. The key takeaway here is that as you scale across all models, you can predictably get better performance, but there notably is a pretty big gap in terms of Transformers and Hyena, where Hyena is actually quite strong at modeling DNA, which is interesting and perhaps surprising to a lot of folks because Transformers have been successful successful in a number of domains, right? And so to really sort of pick at this and try to understand, you know, why might this be the case, or like, are we really sure that that's the case, here's one experiment where we start with a full transformer and then we gradually swap out one layer with a hyena layer until we get to a full hyena layer. And then we measure the performance, like we fully train it at small scale Then we see that as we swap out one layer at a time, we can see this gradual improvement in perplexity. In general, perplexity, the lower the number, the better performance for pre-training, I should clarify. We see that this gradual performance as we use more Hiatt layers, which is again, still surprising but worth mentioning. Another natural question that falls from this is, Why is that the case? It's really hard to know actually why. We can really just probe at this sort of empirically about understanding in what circumstances a hyena outperforms a transformer or vice versa. And so we did have this paper that focused on trying to probe where the strengths are in transformers versus a hyena. And so we had these synthetic tasks that essentially will— you can think of it as like probing specific qualities or abilities of a model, where we had this synthetic task where we'll inject a bunch of noise into the data and see if the model can still do that task with a noisy environment. Or we'll have this task where we compress the sequence into like a single token and see if the model can decompress and reconstruct the token. And what's interesting here is that Hyena was particularly good at these two tasks, like filtering out noise and compressing, which we think is particularly important for DNA modeling, especially at the single nucleotide level. You need to be able to filter out all the noisy information in DNA. It needs to be able to compress tokens to meaningful units, like regions of sequences that are working together to do something. We need to have the model do that on its own, essentially. And so it appears that Yanny is quite strong at those two things.

Eric, there's a quick question in the chat if you can briefly comment on performance compared to Mamba.

Yeah, sure. Yeah, that's a good question. So Mamba's this really exciting architecture that is from Albert Gu, who's certainly brilliant, and he came out of our lab at Hazy Research as well. And so I think Mamba is really exciting because it's an alternative architecture just like Hyena. It's shown to be quite strong on language. I think there's quite still a lot of open questions about the trade-offs of the two. But I think the main takeaway that I'll mention about Mamba for DNA is that it's still quite strong. It certainly outperformed Transformers on DNA as well. And so I think it's a viable architecture to apply to DNA sequences. It's just that in our experiments and extensive set of scaling laws, there was a gap still with Hyena. And so at least for Striped Hyena and Hyena on DNA, it looked like there's a gain for these gated convolution style architectures over something like Mamba. All right. Okay, so I, you know, we have this plan about, you know, what we're trying to model, the data set we've curated, and we got this architecture that seems to scale well. Now I'll describe sort of the phases of the project. Through emojis. The first phase is what I call the pre-train and pray stage, and I call it that because even though we had, you know, these preliminary pre-training kind of runs set up, there's still a lot of questions, open questions, about if this was going to actually learn anything meaningful. Even toward, you know, deep into the project, we were still quite unsure if this was actually going to work. And so questions like, is is DNA too noisy of a distribution of data to learn, right? The perplexity we're seeing in our pre-training is actually quite high. Can it generate long sequences at this high-resolution level? That seems like quite a lot to ask, right? Especially because there's no large language model at this byte level or single nucleotide level for natural language, right? Is there a reason for that? Is it going to be the same for DNA? We started pre-training anyways. We see the loss go down. Great. We still are not sure if it's actually useful yet. Seeing loss go down is one thing. We did compare at this large scale, at the 7 billion parameter, with Transformers again. And again, we saw a pretty big gap actually here as well in terms of the loss. All those higher value ones are the Transformer ones. We get to the stage where we start to probe now after our initial pre-training. Start to see some, what I call, glimmer of hope that the model does appear to be learning something. The first set of tasks was run by Brian, where we do a deep mutational scan. And these are datasets from ProteinGym, a popular benchmark for evaluating protein language models. And the idea here is that we're going to randomly mutate a codon in a sequence pass it through the model, and get a likelihood score, right, like probability of seeing that sequence basically, and see if that score that the model produces is correlated with some kind of fitness measure of that protein sequence, right? This dataset, it has actual fitness values, so we can correlate them. And fitness here, we mean things like protein stability, solubility, binding ability. Those kind of properties. It's a clever way to model or probe a trained model to see if it's learned useful features, essentially. It's done in a zero-shot way, so you don't do any fine-tuning. It's just something you get for, quote unquote, free. The main takeaway here is that we started observing pretty competitive performance with domain-specific models trained on protein sequences. So EVO here in red, is close to or competitive with ESM and ProGen. These are protein-specific models trained on just coding sequences, which is notable because Evo is trained on raw DNA, right? It doesn't know which regions are coding proteins versus non-coding. It has to learn that on its own. And so it's perhaps pretty surprising for this to emerge, this capability.

Eric, is it surprising, given that the training data was extremely protein dense? And so is your DNA model really like a protein model, basically?

Yeah, that's a fair question. So yeah, prokaryotics do have a lot more coding regions compared to, let's say, eukaryotic sequences. It's surprising in the sense that virtually everybody told us It does not make sense to model DNA like this. And so I think it was surprising in the sense that, you know, it has to distinguish between the coding versus non-coding, even though that it's majority coding region. You have to pick up being able to model somewhat longer sequences because it's, you know, now your nucleotides versus codons, for example. And the fact that, you know, there's a key advantage in this protein model is that they only seek protein sequences. It's still interesting to see that it's at the same level, roughly.

Is it possible to isolate the DNA component? So for example, if you generate an amino acid and then condition on the amino acid, like what is the codon that you want to generate after that?

What would be the purpose of that?

For example, let's say I want to— Let's say someone gives me an amino acid sequence and they say, please codon optimize this and generate a really amazing DNA sequence for this amino acid sequence. Is one application of that. But the idea being you don't want to entangle the signal of like, well, what should the protein be versus if I'm constraining the protein, what should the DNA sequence be?

Yeah, that's a good question. I'm not sure. Yeah, I think at least for me, I'm not sure. I think it's still an open question for what's more ideal. But I do know that there's some folks who— I've got colleagues actually trying to work on codon optimization using a similar type of model. So yeah, I think there's still questions about, is that the best way to model proteins? We're not saying that this is the right way to model proteins. I think it's more of an observation that you can learn proteins also from DNA, which to some folks that might seem obvious, to other folks that might seem not obvious, I guess. But it's more just kind of reporting what we see and perhaps try to focus on generalization. So can a single model learn proteins? But also learn these other modalities as well. I'm sure there's other ways to improve performance or try to probe protein-specific things, but yeah, those are good questions. All right. Okay, so we did this for proteins, but we also did this for, as I mentioned before, we also did similar things for DNA and RNA as well. Where we mutate or perturb the sequence, pass it through the model, and get a likelihood score. For DNA, we're interested in trying to predict if this perturbation can be reliably used to predict mRNA expression or protein expression. And in RNA, we use it to predict non-coding RNA fitness as well. What's interesting here is that now that you have a single model that could perhaps generalize across each of these modalities, it's cool that one model can model across the central dogma of biology. There's still questions about other models. If you fine-tune them, can you surpass performance? Very possible. But I think at least for DNA and RNA here, it shows to be quite competitive with domain-specific models. And then for proteins, it's quite competitive as well.

Sorry, but as far as I know, for example, for protein expression prediction, just the GET-Seq content gives approximately 0.50 correlation or something like that. I have I've heard this from Anshul, and it was his post you mentioned. There he said that, for example, your claims that the expression is really good, that your model is really good at predicting expression, is not well justified, maybe.

Yeah, that's a good point. So I should clarify, yeah, there's other techniques that perhaps could do better. I think what we want to show here is that compared to other modality-specific language models, it's competitive. Now, I'm sure there's potentially other ways that you could try to model different features and it might do better. Yeah, I think you mentioned GC content. Yeah, I think that's a certain failure mode for this that is perhaps surprising for DNA-specific models. I think it's worth mentioning like maybe analogous things you see in like other other domains for things like transformers. People have observed similar things in language where they'll ask it really, really simple questions, but differently or out of distribution, and those models will just completely fail. There's these phenomenons that are occurring across language models. By no means are they perfect or there's no room for improvement. There's absolutely tons of room for improvement. And so yeah, those are good things to highlight and point out for sure. And along those lines, one of those things, right? So what we show here is that since our model is trained on prokaryotic and phage genomes, we wanted to see what happens if we fed it human protein sequences. And we see the performance is not very good, right? And, you know, some simple explanations. This is out of distribution, right? So you have protein language models that are trained on a bunch of proteins, including humans and microbial life. And so their performance is much better. That being said, we think if we can reduce the perplexity by training on eukaryotic sequences, this will likely improve, right? So it's, in this case, like an out-of-distribution type of failure mode. OK, so now I'll talk about moving from this stage of, oh, it looks like it's learning things, to parts where we started getting really excited about its capabilities, or at least the promise of some of the capabilities that start to emerge, which center around generation and longer-range abilities. So the first thing is we'll start describing the design capabilities of EVO. At the multimolecular level, so beyond single molecules. I'll just show here, actually, does it show up, this demo? Is that showing?

Yeah, we can see it.

Okay, so we put a demo up of the model in terms of an API, and someone made this really cool wrapper to essentially allow you to generate and control the generation by mixing what kind of prompts you want to feed and the different types of properties for inference. But it will pass it through AlphaFold and then allow you to quickly generate a structure, which I thought was pretty fun to have that iteration time closed. Now, specifically, the things that we focused on generating in this paper, one of them was a CRISPR-Cas system. This is work led by Brian King. And what we're trying to do here is try to create novel CRISPR-Cas systems. That's the goal. You might ask, what is a CRISPR-Cas system? It's a defense mechanism from viruses that bacteria have. And the idea is that these systems will allow them to keep a copy of virus DNA such that when they encounter that virus again, they will recognize it and essentially chop up that DNA and render it harmless. So for us as humans, right, we've hijacked CRISPR-Cas systems for gene editing tools. So you'll have a mix of RNA and protein coding sequences to create such a system. And the idea is that we're interested in novel CRISPR systems because they can give us new capabilities or accuracies or different modalities in which to provide gene editing tools. And so this is done or shown in work from last year from folks at MIT and Broad. Where they tried to discover new CRISPR systems from nature. So they've mined bacterial and archaea sequences and looked for nearby genes or systems in their genomes for possible candidates for CRISPR systems. So we want to do something similar, except we want to do it through training instead. So we'll take examples of these CRISPR-Cas systems, pass it through our model and train, and then be able to essentially prompt on a special token, right, that class of— that describes a class of CRISPR, say Cas9, Cas12, Cas13, for example, and then have the model generate or essentially autocomplete the components, which include the, you know, tracer RNA, protein sequence, or CRISPR RNA, for example, or— and CRISPR. And so to, you know, make sure that obviously these sequences are viable, there's going to be a number of stages to that to validate it, right? So the first set of stages we're going to try to do is the computational validation, right, where we're going to essentially compare these synthetically designed sequences with those found in nature, right? So we can look for things that indicate a real system, which are, you know, the presence of CRISPR RNAs, as I mentioned before, trace RNAs, the proteins, or these domains that, you know, a recognition lobe or a nucleus lobe, things that are required to be a functioning CRISPR system. We can also pass them things through things like AlphaFold to get a similarity score with known CRISPR systems, right, to see how their structure, if they fold similarly. We can check the sequence identity. There's a number of different metrics that we try to validate first to essentially shortlist the most viable candidates. Similarly, we can do this with another class of sequences or systems known as transposons. So these are multigene systems, and this is work headed by Matt Durant, which I'll preview here. We have much more detail in the paper, but what's interesting here is that these are mobile genetic elements that are essentially self-replicating systems that enable DNA to move from one part of your genome to another. Which can be helpful for enabling genetic diversity or allowing some kind of new function or even move evolution, essentially. All right, so I'll talk also next about some of the longer-range capabilities. So, so far, we've talked about the shorter-range, relatively shorter-range. The first task along this line is on gene essentiality. And this is a work, again, headed by Brian, where the idea is that we're going to take a whole genome and we're going to add stop codons prematurely to essentially deactivate one gene at a time. And so there's these experiments done in the wet lab, essentially, which use CRISPR to do the same process. They'll edit a particular codon to render the gene to turn off the gene and see if it was essential for the survival of that organism. So that's what we mean by gene essentiality. And so this can be pretty costly, right, because it requires wet lab experiments and some CRISPR experiments to do this. And instead, what we do is we're going to model this computationally from a dataset that was created experimentally. And the idea is the same as before, which is we're going to use the likelihood scores. We're going to pass sequences through the model that have been added with the premature stop codon. So we're going to mutate it. And then we're going to do a zero-shot, no fine-tuning test again to use that prediction score, the likelihood score, to correlate that to— sorry, to make that as a binary prediction, either essential or non-essential. And again, it was surprising to see that EVO was able to even learn a measure of this gene essentiality. So we compare from shorter context of just the gene, and then we gradually increase it to 8,000 into 66,000. And we see the performance maintained with longer sequences. And we do compare with the baseline DNA model as well, which we were able to surpass.

Eric, were you able to compare it also to like ESM, for example, and do essentiality prediction from that?

We did not. We didn't do that. That's an interesting question. Maybe Brian could speak to that. Oh, you're muted, Brian.

I mean, for the specific way we kind of mutated the sequence here, like ESM is not even well defined because it doesn't model stop codons. Could like truncate the model or something. We would have to use ESM in some kind of supervised framework. But as we showed here, actually, the model does use the context for gene essentiality. So if you only provide it the information from the gene, it like knows a little bit. But the context is actually the genomic context of the that the gene is in is quite helpful for this kind of zero-shot kind of probing of the model. So, and that's kind of— that's what I found pretty striking about this result too. It's using the genomic context to improve this kind of in-context function prediction. And the way we're actually mutating the sequence is very close to how you would do this. Like, if you're doing the actual experiment, you're actually inserting a stop codon. And a stop code is just outside of the vocabulary of ESM.

I do think that's a really nice result in that way. But I do just wonder if it would be a really way of isolating whether the context, like the DNA modeling specifically, is important. Because presumably, your models are both learning evolutionary signals across a lot of different genomes. And so you could imagine that ESM would have as strong of a— of an ability to predict essentiality as a potential outcome and may not even need the context. And it would be like your essentiality predictor would be primarily driven maybe by, or as much driven by, like EVO as protein model versus EVO as DNA model specifically.

Yeah, that's an interesting question. How would you extract this information in say zero-shot from ESM? Or the likelihoods.

I think maybe you'd have to find some prompting mechanism to destabilize the protein. And so proteins that are not as essential maybe are more— can be destabilized more readily than proteins that are essential, for example.

Yeah, it's an interesting thought. Yeah. Good baseline.

So have you compared your model with just gene occurrence score. What I mean is that usually that you trained your model on many, many genomes across for many bacterias. And so they know that there is a simple rule that more frequently occurring genes across different organisms, they tend to be more essential. And so maybe your model just predicts higher probability for essential, for more frequently seen genes. And those, then you mutate it by inserting stop codon. You just have a larger drop in probability.

Yeah, I think we actually do like have a, have an additional conservation baseline that we will include in future work. And this performs better. But if you think about it, I do think that's what the model is— one of the aspects the model is learning. But even the fact that the model is learning this kind of within its parameters is itself striking. If you think about what a model needs to do in order to kind of do this kind of analysis needs to know at least where the gene boundaries are. It needs to know what a start and stop codon are. It needs to be able to kind of look up in a very, in a soft kind of not literal way, like what this gene is across the, across the training dataset and taken into account these things about conservation and such. So, Yeah, I think the model needs to learn some signal to predict this. It's probably related to conservation, but the fact that the model can do this, I think, is showing us that we're onto something.

Yeah, thank you for your answer, but is it really so surprising? We know that ChatGPT or even GPT-2, they were able to learn a huge amount of text. They are able to reproduce it exactly. So it is not so hard for model to memorize genes it's seen.

Yeah, at least fair. Yeah, I think that's a fair assessment. Maybe we should preface surprising to us, maybe not surprising everyone.

I mean, I was surprised, so I don't know.

Thanks for the question, guys. For sake of time, I hope it's okay I continue forward and we can save the longer discussion toward the end, but really appreciate the questions. We're almost done, I promise. Last thing we wanted to describe is to really push the model. We wanted to see what would happen if we tried to generate a whole genome, right? It was one of our earlier questions, pie in the sky kind of questions. So what you're seeing here is predicting essentially a whole genome sequence up to 250,000 bases. 6— sorry, 650,000 tokens, which contained hundreds of synthetic proteins. And so we, you know, want to see how realistic are these whole genome sequences that we're trying to generate. And so we use some tools. One of them is CheckM, which essentially allows us to check or assess the quality of some of the sequences. Normally, these tools are used for checking metagenomic sequences, those in nature of bacterial life. But they can check for qualities around coding density or coding organization, right? How are the genes grouped along the sequence? What's the coding density, like coding versus non-coding? And they do have some similarities with actual genomes. But what's worth mentioning for sure is that we are quite far from having an actual whole genome, right? This is by no means an actually viable sequence, at least what it appears to look like. It has some similarities in terms of how the proteins fold to natural ones, but some of these confidence scores, for example, are lower, but they're missing key marker genes or rRNA, for example, things that are required to actually have a whole bacterial genome, a whole genome. And so it's worth mentioning that This is sort of what we describe as a blurry image of what a whole genome sequence looks like, and it looks like the model is kind of learning this high-level coarse-grained view of what a genome sequence is. And so that kind of begs the question of like, all right, so given what we've shown, what's next, right? Where do we go from here? Thinking of looking at some feedback, because I love pulling quotes from folks online. You know, some folks have said, great application of DNA language models to prokaryotic genomes, potentially impactful application, right? There's other folks who mentioned things like, what the heck is a foundation model? What does that term mean? And does that mean that if I type in, you know, the phrase design a dragon with blue tail, it's going to output a genome with such a characteristic. So there's a lot of questions about what these things are useful for, and I think they're all worth asking. We definitely want to talk about it. For us in the near term, we're going to focus on improving the generation quality and controllability of our generation for these models. We certainly want to incorporate biological feedback and pull in techniques from the language community. Which we've seen things like ChatGPT require things like reinforcement learning with human feedback to actually make quality sequences. What's notable here is that we don't do any of that. We just use a raw pre-trained model and we're able to get these type of sequences. But we absolutely do believe that we can improve the quality with these type of techniques. We're also interested in, for sure, how these things scale. What happens, what emerging capabilities will potentially emerge as we scale to larger models, as well as how do you model more complex genomes, right? That's a very open question because, as Yuri mentioned, prokaryotic sequences are coding rich, and so it's going to be potentially much more complex to do something similar for eukaryotic sequences. But whatever we do do, I think being at the intersection of both AI and biology is going to be really important. Some things I've heard from folks on, for example, the AI side, they'll mention things like all you need is scale, right? Just get a bigger model, get more DNA, and things will happen. And on the flip side here, folks from the bio who think all you need to do is find the right problem, like the models don't matter. You know, there's— you take existing transformer, and then you'll be fine. I strongly believe to drive this new capabilities and have innovation, it really needs to be at the intersection of both of these areas. To really drive this home, let's take an example of how OpenAI did this with text and images. They really were able to figure out how to do so in a somewhat simple task formulation, but merge text and images to give us things like DALL-E, or SORA, where in bio I sort of think there's an even bigger opportunity, right? I think of DNA as sort of the initial conditions perhaps of an organism, but you can have things that try to measure new information about the state of that organism, right? At the cellular level, at assays or the environment. And so there's, I think there's a huge opportunity to merge modalities and potentially even more impactful ways, right? You're not trying to create better chatbots. You're trying to push what we know about science, potentially improve human health. And so I think it's a really exciting space to work in. And so circling back to this original question, can we design an entire genome from scratch using AI? And short answer is no. But I think what's really cool about this work is that seeing what happens when you go after these big, you know, crazy ideas, what emerges and what you can learn along the way. And what kind of directions might be perhaps opened up. And so I encourage folks to go after those big ideas, which tend to be the riskier ones. And so if you are going after these big ideas, I'd like to hear from you. We'd like to hear from you. We're always on the lookout for crazy ideas and the potential to change how we think about modeling biology. And so I'll close with mentioning that this was part of a much larger effort with really awesome team members from biology to machine learning to folks in medicine who think about the ethics of this as well, which we definitely put a lot of thought into. And so, yeah, we have tons of resources online for the code, demos on Colab. The weights are public. And so you can access all these resources through our blog, which is a good entry point for the links for these, and my contacts as well for folks who want to have any follow-up questions. And so with that, I'll close and mention thanks so much for listening. And if we still have time, I am happy to take questions. So one quick question.

You guys, in terms of biological feedback, you didn't check if the transposons you made or the Cas9 you made are actually active, right? Or did you—

do you know if they're active? Yes, we are actively doing that right now. So this is the preprint, and we're actively in review. And so we hope to share results as soon as we can.

Great. Thank you.

One application I think would be amazing is a few years ago, Rotem Serek published this paper on discovering new antiphage defense systems. And they basically used heuristics to find structures in the genome that suggest a defense island and that kind of thing. And they discovered all these new proteins. And I feel like using a model like EVO would just supercharge that and be really amazing at discovering totally new defense functionalities that could be co-opted for new biotechnology. Would love to see that.

Yeah, that sounds awesome. I'm very interested in that. Thank you. Yeah, Jacob, go ahead. Hi, thanks.

Thanks for the talk. It's awesome. I was wondering, you showed in your scaling study that striped hyena and hyena, they perform pretty similarly. I was just wondering what ended up to tip the scale towards using striped hyena over just plain hyena.

Yeah, so the takeaway there is what we observed, especially in language, because we definitely did scaling laws in language as well, At the smaller scale, it looks like hyena, like a pure hyena could perhaps do better. But as you get really large, the trajectory of a striped hyena in particular seems to be better essentially. And so when we scaled to 7 billion, we also have differences with regular hyena as well. And so it made sense just to use a striped hyena There's also things like context extension that, you know, we don't describe too much of those nuances, but the idea there is that it potentially could work better, this context extension idea, with the striped versus the regular hyena.

Okay, but it's not like explicitly visible in the scaling study plots, right? But it's just something that you know from language, basically.

Yeah, we didn't showcase as much, but I think in the current study for the DNA scaling, you can already see the slope being stronger for the hyena, the striped hyena. And so this was at a small scale. So like they're all small scale. So like that gap will just grow until you get to 7 billion. And so we're essentially using that, that projection, like extrapolating, which, yeah, you can't see as much because it's at a small scale in the deep scaling, but just from our own intuition about what's likely going to happen with a much bigger worm, it was clear to us at least. Thanks.

Thanks.

Good question.

Is it okay to ask a small technical question?

Yeah, go ahead.

I was just wondering, we're trying to train larger HiNA models and What we sometimes had, and I guess you maybe also alluded to that problem with Mamba in the preprint, it seemed to be unstable and we would be getting NaNs pretty early in training for larger models, even though they would have no problems fitting for the same dataset when they were smaller. Do you have any ideas or tips of what could be causing that because apparently you can scale it up to large numbers of parameters. I was just wondering what could be the thing that we're doing wrong and we are using your code.

Yeah. So I'm guessing you're referring to the Hyena DNA specifically code. Is that right? Yeah. So there are some key differences from the Striped Hyena code actually that we've incorporated since then. To help stabilize and really focus on scaling large versions because the Hyena DNA, as cool as I thought it was, those models were very, very small. So 7 million parameters was the biggest one. And so yeah, it does take some learning to figure out how to scale these effectively that hopefully through time we'll be able to share some of those lessons. But we try to describe those as best we could in the paper. But yeah, I do realize that there's probably more nuanced things to do, which is a learning process for everyone, I think, for us too, as well as we even want to go bigger, right? So yeah, I think it's going to be some trial and error that will definitely be involved in trying to learn along the way. Yeah.

Thanks so much. You mentioned this one thing that you incorporated that you were willing to share. Could you repeat that?

I didn't— Yeah, so maybe for sake of— I'll go to the next question, but refer to the paper for the details. But thank you for the questions. Thank you so much. All right. I don't see any more questions, so feel free for the next person to jump in if they'd like or if anybody—

Yeah, there's a couple of questions in the chat, Eric, but I think maybe we can prioritize people that would rather just unmute and go for it if you have a couple extra minutes to answer. Otherwise, you can maybe just pick through the chat and see what you would like to answer.

Yeah, I'd say jump, go for it. Whoever's first come, first serve. Sorry about that.

Okay. Yeah. If anyone wants to read their question out of the chat as well, I think that would be really helpful.

Sure, I'll do it.

I was the one who asked the scale questions. Try playing versus Mamba thing. But basically, does being able to train and improve perplexity over a longer time frame help us improve expressivity? Can we capture biological nuances that we otherwise wouldn't capture in the earlier stages of training the model towards the lower ends in training that are permitted by the increase or the better performing architecture, let's say?

Yeah, so maybe I can— It's a good question. It's hard to know for sure, but what we observed in general is that better pre-training measured in perplexity generally leads to better downstream performance for the things you actually care about. So those downstream tasks. It's maybe not a linear relationship, but it seems to be in general, like what we observe is the model just does better in pre-training, therefore in downstream tasks., it also will improve is what the signs point to, I should say. That's pretty cool.

Is there anything that was— well, I guess this is hard to evaluate, right? But is there anything that was learned towards the, let's say, longer or towards the end of training the models that wasn't captured in earlier phases explicitly, anything of the sort that you guys found? Or is there one particular downstream task or tests, sorry, that really gains on the longer training time?

We actually didn't really focus on that kind of difference. So I think we are now trying to do some of that analysis to see like later checkpoints versus earlier checkpoints, what's some of the differences. I think like on some of the zero shots, actually Brian mentioned this before, one of the tasks that it was doing very poorly on for protein fitness prediction, in particular the ones that was an outlier, was doing very bad at, when we trained it for even longer and just used those checkpoints instead, the performance had a huge bump. So I think that particular example was less— what's that word— less represented sample. So it was doing poorly because it was not a huge part of the training set. But if we train longer and longer on it, then I suppose the model was seeing it enough so that it started to learn it better. It's one instance. Good question. Very insightful. Very cool. Thank you very much. For sure. Yeah. Hi. Hi. Yeah. So can I ask a question right now or? Yeah, go ahead. Yeah. So I have a question. So it's just my curiosity. Do you think this, your high-end architecture will be useful for diffusion model as well? Good question. So potentially. I mean, as a totally unbiased party in this question, I think Hyena would be great in many settings. But in general, I think the fusion models, there's a little bit of like a TBD on how well they work for language model settings. They work really well in continuous data representations, so like images and videos, they seem to work very well there, obviously. But for languages, it's a little bit unclear if it'll work well or as well or better, especially as you scale them. Like, that's a very open question. There's no large-scale discrete diffusion model for language. So I'm talking about the Yeah, so image diffusion model, actually. Oh, OK. Maybe. I haven't tried it before. But I tried the high number images, and it seemed to work well. But yeah, that's an even more open question. No one's tried that yet.

Awesome.

Thank you.

Sure.

So can I ask one more question? Sure. Thank you. So then you say that your model is able to generate new proteins. You use AlphaFold 2 predictions as they're simulated to known structures. As an evidence for that. But as far as I understand, AlphaFold tends to generate similar structures for similar sequences, even if you, for example, replace the whole alpha helix with a set of prolines. So is this evidence is really strong enough?

I mean, ultimately we want to validate the wet lab, right? That's the gold standard. So, so what we present are, are things that computationally that we could show as promising signs, but by no means are we saying that these are valid sequences in the real world. So that's, we want to make that clear. At the same time, I think there's a lot of literature that when they analyze protein sequences in existing papers or existing models, we try to borrow similar techniques. And so we try our best not to make our own benchmarks from scratch, but we try to follow similar lines of work. And so a lot of the protein models we observed also use AlphaFold to showcase similarities of new sequences, for example. But, but good question and very, very fair question, I'd say.

Thanks.

Yeah. Yeah. Cool. Okay. So I think maybe in out of, you know, in the interest of time, I think, we want to thank Eric and Brian both for being here and giving this presentation. Clearly, there's a lot of interest in the work. So if people have follow-up questions, I think reaching out to Eric would be a great way to explore those. And yeah, the recording will be posted soon, so keep an eye out for that if you want to go through the slides as well.

Awesome. Thanks so much, guys. I love the interest and questions and feedback. Such a really good engagement. That was really fun.

All right. Thanks, everyone, and see you all in 2 weeks.