Scientist Stories: Anshul Kundaje, Deep Learning for Genomic Discovery
In this interview‑style presentation, Anshul Kundaje, an assistant professor of genetics and computer science at Stanford, explains how deep learning is transforming the study of the non‑coding ("junk") genome. He begins by outlining the historical context: the Human Genome Project provided a reference sequence, but only ~2 % of the 3‑billion‑base human genome encodes proteins, leaving the vast majority as regulatory elements that control gene activity in a cell‑type‑specific manner. Large‑scale sequencing technologies and consortia such as ENCODE and Roadmap Epigenomics have generated massive, multi‑dimensional datasets that map biochemical “flags” (protein binding, chromatin marks, accessibility) across thousands of cell types, forming a high‑dimensional tensor of genome‑wide activity. Kundaje describes how these data are reframed as a supervised learning problem: short DNA windows (≈100 bp) are paired with quantitative signal profiles, and deep neural networks (e.g., BPNet) are trained to predict base‑resolution profiles from raw sequence. The models achieve near‑experimental accuracy, effectively learning the syntax of regulatory DNA – fuzzy “words” that transcription factors recognize, their spacing, orientation, and cooperative interactions. By back‑propagating contribution scores, the models pinpoint which bases drive each predicted profile, revealing combinatorial motifs and even helical‑periodicity effects (≈10.5 bp) that reflect DNA’s three‑dimensional structure. The talk then showcases how the trained models serve as in‑silico oracles. Synthetic perturbations (inserting or spacing motifs) and systematic mutagenesis of real genomic elements uncover higher‑order syntax rules and asymmetric cooperativity between factors such as OCT4 and NANOG. These insights guide CRISPR‑based experimental validation, confirming model predictions about motif importance and protein‑binding changes. A major application is interpreting disease‑associated genetic variants. By feeding single‑nucleotide variants (e.g., from Alzheimer’s GWAS) through cell‑type‑specific models, Kundaje’s team can predict variant effects on regulatory activity, prioritize causal variants within linkage blocks, and infer mechanistic explanations (e.g., creation of a FOS binding site in an oligodendrocyte enhancer). Similar pipelines have been applied to autism de‑novo mutations, congenital heart disease, and mouse disease models, demonstrating cross‑species transferability. Kundaje also discusses community resources: the KIPi model zoo provides open APIs for sharing and reusing trained models, encouraging collaborative model development. Finally, he acknowledges limitations – current models capture only local sequence context and rely on high‑quality, cell‑type‑matched data – and outlines future directions toward chromosome‑scale architectures and better handling of long‑range genomic interactions. Overall, the talk illustrates how deep learning can move beyond black‑box prediction to become a discovery engine that decodes the regulatory grammar of the genome, links non‑coding variation to disease mechanisms, and accelerates experimental design.
Chapters
Let's, let's get started. I'm Casper Mossman, Communications Director at QB3, and with us today is Anshul Khandaje, who is Assistant Professor of Genetics and Computer Science at Stanford. And Dr. Khandaje's primary research area is large-scale computational regulatory genomics. And I was earlier joking to him about how people call, call his field junk DNA. So I guess we're going to hear about exactly what that entails, junk DNA, the human genome containing about 3 billion or more base pairs, but only 2% of that is actual genes. So what is the rest? And in short, a lot of it is this regulatory mechanism to control gene activity. But it's unknown exactly how this happens in each cell type, what the elements are, what their sequences are, and especially for human health, how this is connected to disease and normal health functioning. So, Dr. Khandadji, it's great to have you. What can you tell us about deep learning and genomic discovery?
Yeah, absolutely. I'm, you know, I'm joined between the departments of genetics and computer science, and we have been exploring this interface for several years now. And biology has become a very, actually, a very data-driven area now. So there's huge amounts of data being generated by high-throughput assays. And so it's extremely ripe for large-scale machine learning approaches. But, you know, typically when you think of particularly deep learning, you think of it as this very powerful hammer for prediction, right? Like it's these black box prediction tasks. Here I'm going to turn this on its head and show you how these same models can act as amazing discovery engines. You can use them to actually discover entirely new insights about the genome. So, um, should I get started?
Yeah, please share your slides if you would.
Okay, so, um, yeah, thanks again so much for the kind invitation, and, uh, it's, it's really great to be here today. Um, hopefully in person sometime. Um, so my— the title of my talk, as I mentioned before, is, is Deep Learning for Genomic Discovery, and I will slowly walk you through the entire through the entire talk. First, sort of motivating the biology, and then trying to get to how we can use machine learning to decipher novel insights about the genome. So I'll start with, you know, what happened about 20 years ago. The Human Genome Project completed its first draft sequence in 2003. That was a decades-long project, and we obtained for the first time, some view of the sequence of the genome, which is basically we now know about 3 billion nucleotides or letters. It's a 4-letter alphabet, A, C, G, and D. What's amazing is that even though in 2003 the draft sequence was obtained, the first complete human genome, right, end to end, also known as telomere to telomere, actually happened last week. So there's a publication last week in Science, an entire series of publications, where for the first time we actually know the entire sequence. And it's amazing, they discovered 99 new genes which we didn't know existed before in repetitive regions that were very difficult to sequence previously. So we are constantly making new discoveries. By we, I mean the field is making new discoveries about the genome, right? So going back here, you know, in 2003, we had the draft sequence of the genome. This was like one genome sequence for all of humanity. But as we know, humans are very diverse. We have amazing diversity in various physical traits and so forth, and also disease susceptibility. Some of us are more susceptible or have higher risks for diseases than others. This is all encoded, a large part of this is encoded in our DNA. The genomes of no two individuals are identical, so except for maybe identical twins, if you look at any pair of individuals, any group of individuals, there are positions in the genome where the sequence is not identical. There are mismatches and insertions and deletions, and that really gives rise to this beautiful genetic diversity across individuals that manifests as various kinds of traits, and in some cases, unfortunately, also makes us— increases our risk for specific diseases. So it's very clear that cataloging genetic variation is an essential first step to understanding the genetic and molecular basis of disease and also understanding the genetic basis of just natural variation in various traits, human traits. So what has happened is since 2003, there has been a revolution in sequencing technology. The machines have gotten smaller. Like here you can see a USB-sized sequencing device. It literally is a USB device. You can stream, you know, DNA sequences right into a computer. But not only has the size dropped, but, you know, similar to the semiconductor revolution, there has been a dramatic drop in the cost of sequencing and hence a tremendous increase in throughput. And so now you don't have one sequence for all of humanity, you actually can sequence personal genomes, genomes of individuals, right? And so there have been large-scale population sequencing efforts that have sequenced, you know, hundreds and thousands to millions of individuals. Many of these are just healthy individuals, and the goal is to simply catalog natural variation in the genome. So here's an example of a location. You can see right here each of us has 2 copies of the genome, and this individual has an A at this position and a C. This other individual has 2 copies of A. Another individual might have 2 copies of C, right? And so there's a lot of this sort of natural variation, and these kinds of large-scale population sequencing efforts have found millions of genetic variants You have about 1 variant every 1,000 bases in the genome. These are common variants that are variants that are seen in at least 2%, 5% of the human population. And if you look at rare genetic variants, that is genetic, you know, changes that are found only in a few individuals or exclusively in one individual, that can often occur 1 every 4 bases. 4 letters in the genome. So the genome is, at the population level, is chock-full of genetic variation. That really gives rise to, you know, the incredible diversity we see. The same strategy can be used to study genetic basis of disease as well, because you not only sequence a bunch of healthy controls, right, but you can sequence a bunch of individuals that have a specific disease. And so you can actually conduct case-control studies. These are often called genome-wide association studies, and here the idea is again that you sequence a bunch of disease individuals, you sequence a bunch of well-matched controls, and then you can perform statistical tests that literally are simply at each genomic position where the variant lies. You can perform a statistical test which will simply, you know, correlate the number of individuals that have one specific combination of these letters against other combinations, and if you see a strong imbalance between the healthy individuals and the diseased individuals, that can be picked up as a strong statistical association, right? So that might indicate that individuals that have, you know, a specific version of this variant might be at higher risk for the disease. So that's called an association study. Okay, and this has been very successful and has been used to study thousands of diverse traits and diseases. Okay, so this is what one of these association studies looks like. On the x-axis is the whole genome, 3 billion bases. Each of these segments are different chromosomes. We have 22 pairs of autosomes and, you know, sex chromosomes. So This is the whole genome, and each point in this plot is a genetic variant. There are millions of them, and the y-axis is the statistical significance of association. The stronger the statistical significance, the stronger the association or the confidence in the association. You can see most genetic variants have low associations with, in this case, Alzheimer's disease. This is a case-control study of Alzheimer's disease.. But then you see these huge towers, right? These collections of variants that really pop out. This is referred to as a Manhattan plot because it looks like the skyline of Manhattan. And these huge, you know, these spikes that you see right here, these are all genetic variants that have strong associations with Alzheimer's disease. So these genetic variants might be predisposing individuals to this disease, okay? So this has been a very popular approach, and it's been allowing us to map locations and positions and variants in the genome that potentially are associated causally with disease risk. But this doesn't tell the whole story because like, okay, now I've identified this genetic variant right here, this letter A. If an individual has a C instead of an A, they might have an association with Alzheimer's. But that's just a catalog. I don't know what these letters, what these variants are actually doing. We need to be able to go in and actually understand the function of these variants. And we talk about variants, why not go to other kinds of subsequences? Like there's this word right here, CCAGAGGAA. What does it do? So we need a functional annotation of the human genome, and that's where we come in. We really focus on functional genomics and decoding genome function. We want to understand what every letter in the genome does or does not do. Every word in the genome, what it does or does not do in a molecular fashion. Now, what makes genome interpretation or functional interpretation of the genome complicated is that we are not unicellular organisms. We are multicellular organisms, and we have thousands of diverse cell types and tissues in our body. If you take an individual, that individual has the same genome in all the cell types. So it's the same instruction set, but it's giving rise to, again, this amazing diversity of cell types and functional morphology. How is that happening? So let's— to understand that, we need to analyze the genome in the context of different cell types, cell states, and cellular responses, because the same instruction set but giving rise to different, completely different phenotypes. And the reason is because, as you know, if you're given a collection of instructions, you can put them together in different combinations. And the different— if you have a universal instruction set, if you select different combinations of instructions, you can give rise to different programs. So each cell is like a different program, and it's using the same global instruction set written in the genome, but using different parts of it in different combinations. And so to understand how that works, I need to go into the genome and now at least give you an overview of the main functional components of the genome. So here's the cell, and here's the DNA packed inside the cell, and let's unwrap it. So I'm showing you a zoomed-in version of this. Um, one of the main functional units in the genome are genes, right? Everybody knows about genes. Genes are pieces of the DNA sequence that directly code for proteins. You can literally read them, the letters on the gene,. And they get translated exactly— they have a mapping from the ACGTs in the gene to the amino acids in the protein. So that's the classic genetic code. It literally is an instruction set. It's copied, and you can make proteins out of genes. So genes code for proteins. But then what makes cells different is there are about 26,000 genes in the genome. Different combinations of genes are expressing different combinations of proteins in different cell types, and it's the specific repertoire of proteins that are produced that define the functional state of each cell. That's what makes a neuron different from a muscle cell, is the repertoire of proteins that are expressed. So different genes are turned on and off, right, and that really defines cell identity. If you mess with this program that turns genes on and off, you're going to get weird cell states, which can essentially result in pretty serious disorders. But if you think about like you're turning genes on and off, you need control switches. You need on-off switches to turn genes on and off. And where is that encoded? Well, that is encoded in the rest of the genome. So the pieces of the genome that encode for proteins directly is just 2% of the 3 billion letters. The 98%, what is it actually doing? Well, most of it is not doing anything particularly useful, but it does contain the instruction set for the control elements that turn genes on and off. They happen to be these little pieces of DNA. I'll just call them control elements to avoid jargon. These are like on-off switches. They are again segments of DNA sequence, and their job is to encode information that allows genes to be turned on and off. So we know where most of the genes are now in the genome, but these control elements have been quite elusive because they're scattered all over the place in the remaining 98%, and they're pretty sparse. It's not like the 98% is full of these control elements. These are sparse elements. There are several millions of these controlling 26,000 genes distributed across 98% of the 3 billion bases. So how do we map these? How do we find out where they are? Well, thankfully, alongside the genome sequencing revolution, there has been a revolution of other kinds of sequencing-based assays, experimental assays, which allow you to profile different kinds of molecular activity on the genome. So for example, let's say this gene produces a protein, and this protein is a control protein, okay? So what it does is it goes and binds other locations in the genome, control elements in the genome,, and it turns genes on and off. Okay, so if you want to figure out where this protein binds, you can actually perform a genome-wide profiling experiment that gives you this beautiful track across the entire genome. It literally gives you a signal at every base in the genome, and when you see these big spikes in signal, they correspond to likely locations where the protein is bound. So you can perform one such experiment for any protein of interest in any specific cell context. And you can do this for this black triangular protein. You can do it for other kinds of biochemical modifications. I'm showing them here as colored flags. You can see right here, this colored flag is marking the start of this active gene, right? This modification, orange one, is marking the start of the active genes, but it's also marking these control elements that are pretty far away from the genes. Here's a yellow modification. By the way, this is real data. Even though this looks like a cartoon, this is an actual piece of the genome, and there are two genes sitting next to each other. This is a few megabases in length, so it's a zoomed-out view of a piece of the genome. Here's another modification, chemical modification. This seems to not mark the start of the gene, but it seems to mark these distal control elements that are far away from genes. Look at this green marker. The green marker exactly maps the body of the active gene. And here's an amazing other marker, this gray marker. These gray flags are marking the complement of the active genome. So this gene is being actively expressed, and there are these active control elements turning it on. This gene is being repressed, and you can see that it's kind of a dead zone. There isn't much biochemical activity, but it is being marked by this gray marker. So there is this amazing biochemical information laid on top of the genome And this is dynamic information. You can add and remove these flags. You can add and remove proteins. And so this is what allows the dynamic expression of the genome. The same genome can be manipulated and modulated to express different genes. And that memory, short-term and long-term, is encoded through these various kinds of chemical flags and proteins that do their jobs binding the genome and turning genes on and off. Okay. Now, as I mentioned, each of these experiments represents one biochemical flag across the genome in some cell context. And as I mentioned before, there are hundreds and thousands of cell types and tissues. So what the NIH did is after the Human Genome Project concluded in 2003, they funded two other large consortia, ENCODE and Roadmap Epigenomics. And the goal of these consortia was just as we obtained the first draft reference sequence of the human genome, the idea here was to try to get a reference molecular map of the human genome. And this is not a simple linear string, right, because this is the dynamic expression of the genome and how information is encoded dynamically on it through these molecular flags. And so this happens to be a cube of information. It's not a string, it's not a vector, it's actually a cube or a tensor. And one axis happens to be the genome, 3 billion letters, 3 billion positions. The other axis happens to be hundreds of different molecular readouts, these colored flags I showed you, different thousands of proteins that have various kinds of binding activity. Each of these molecular readouts is across the entire genome in thousands and thousands of cell types, tissues, individuals. So this axis is enormous, right? Humongous data cube of molecular information. And think of it as dynamic genome activity of various molecular markers across the genome in thousands of cell types and tissues. And so this is an amazing dataset. You can't make sense of this by just looking at one data point at a time. It's a tremendous source of information, and this is what feeds a lot of our machine learning models. And what is our primary goal? Our primary goal is to take this beautiful molecular information decorating the genome and trying to make sense of how their DNA sequence encodes all this information. Because in the end, all the information is in the genome, right? It's all there. The instruction sets are all there. How is that same genome sequence giving rise to this kind of dynamics and richness of molecular activity? So what I'm going to show you today in my talk is I'm going to take this kind of data. I'm going to show you a bunch of machine learning models that we have developed, and we're going to use these models to discover two things. We're going to try to decode the syntax of the non-coding genome, that is, the control elements. You know, what is the, what is the language of these control elements, and how do they encode dynamic genome activity? And secondly, I'll use these models to also show you how we can make sense of genetic variants and mutations. That might disrupt genome function through these control elements. All right. So why is this actually important is because people have performed these genome-wide association studies, as I mentioned, for hundreds and thousands of traits and diseases. And they have mapped thousands of loci and genetic variants associated with the disease. And so here's one example. For example, let's say there's a C at this position. And this is usually in healthy individuals. And when there's a G, there is an association with risk for some disease, let's say Alzheimer's. If you take the catalog of all genetic variants associated with diseases identified to date and you just see where they fall in the genome, only about 5% fall inside coding regions. So you can think of like, how will mutations or genetic variants disrupt cellular function? One is they could actually you know, affect the coding sequence of a gene, which means the protein that's translated out of it is messed up, right? So if you messed up a protein, wow, that's going to be really bad, right? So that's one way to kind of really mess up, you know, the genome and potentially cause disease. But the other way is more subtle. You keep the gene intact, right? The genetic variants are not really directly disrupting gene sequences or protein sequences. They're instead disrupting sequences of control elements., and those control elements might overactivate or repress the genes in the wrong context, and that can cause havoc, right, in developmental processes, or this could happen slowly over time and cause diseases that essentially happen at old age, right? So you can have different kinds of mutations and variants, and what's amazing is that 95%, this yellow block, of genetic variants associated diseases are primarily non-coding. They are likely, or they're enriched in these control elements. That's why studying the control elements and dissecting their syntax and grammar and try to understand how mutations in the sequence manifest in changes in the activity of these control elements is very, very important because it not only is a basic biology question of how the genome functions, but it will also give us insights into 95% of disease variants. All right, so part 1, let's jump into decoding genome syntax. And here we're going to focus on very basic biology of how these proteins that I mentioned to you, these control proteins, how do they regulate gene activity by binding these control elements and doing interesting things. So let's again open up this anatomy sketch of the genome. And now let's take a control element and look inside it. So what are these control elements doing, and how are they turning genes on and off? Well, what's actually happening is the sequence of these control elements have evolved to contain specific DNA words, as shown right here. They're fuzzy DNA words. As you can see, they're not exact. Like, you can see there's a— the height of the letters corresponds to kind of like the preference. And these DNA words act as landing pads for proteins. Okay, these are control proteins, regulatory proteins, and they recognize specific DNA words as not just random words placed, you know, in some sequence. There's actually beautiful syntax and grammar just like in a language, but it's fuzzy and it's a different kind of language. And evolution has played with this so that, you know, the information encoding these elements is these sequence recognition codes, these words that are recognized by proteins. And you can see how this can actually give rise to a very complex combinatorial control system, because you have, let's say, 6,000 proteins. Each of them recognizes, you know, fuzzy words with different affinity. And you can put together now complexes of proteins. You can take these 6,000 proteins, and you can take pairs and triplets and 4 of them and 5 of them and create complexes. And each of these complexes will recognize different sequences, right? You can encode incredible precision and specificity by coupling proteins and their recognition codes, and the genomes have evolved to have this really beautiful syntax. And that's how you can activate an element in one specific cell type in the body, because it has this combination of a specific set of proteins that only get expressed in that cell type, and the sequence acts as a landing pad for those specific proteins. And these proteins bind the DNA, and then these elements loop in 3 dimensions to the start of the gene, and then they turn the gene on and they transcribe it. All right, so deciphering this genome syntax is really coming down to identifying all of these words, the fundamental vocabulary, the lexicon of these DNA-binding proteins, and the complex combinatorial syntax that gives rise to the you know, the combinatorial specificity of these groups of proteins and how they bind the genome in different cell types and contexts and how they regulate different genes. Okay, so as I mentioned before, how do we dissect the syntax? Well, thankfully, as I mentioned, we have experimental approaches to actually measure or at least capture shadows of these proteins. So for example, this protein, let's say it binds this, this piece of DNA, How do we measure this? Well, we can perform a sequencing experiment that can actually almost take a snapshot of the shadow of this protein. So imagine this protein binds this location, right? It creates like a— like imagine you threw a pebble in a lake. It creates waves, right? So, you know, it kind of blocks the DNA at this stage, and you can perform a sequencing experiment where you can capture sort of a virtual shadow of the protein, okay? So here's an example of the shadow of this protein. The spike on this side of the DNA, the spike on the other side of the DNA. And you can imagine this doing this across the entire genome. So wherever you see these spikes, they represent the shadows of the proteins, and you can get a beautiful genome-wide readout of where the protein binds, right? So here's another kind of experiment that is a different kind of enzyme, and that also gives you a different kind of shadow. But think of it as using sequencing, genome sequencing, as almost like a molecular camera, and you're taking snapshots of the genome across the entire genome, you can look at these shadows of proteins. And the precise shapes of these shadows and the precise profiles of these shadows really capture exquisite information about the protein-DNA contacts, OK? And so we can use this data to now reverse engineer how the protein might be recognizing DNA, because it's not recognizing— it's not binding all over the genome. It's binding at specific positions in specific cell states. So here's one such example. This is data generated by the ENCODE consortium. Each row here corresponds to a specific protein, and each of them you can see these beautiful spiky signals. They're representing combinatorial binding of these repertoire of proteins in one cell type in a specific region of the genome. So you can see right here, this protein, this one, and this one bind together. At this location is this one, this one, this one, and you can see it's quantitative. It's not binary, right? You actually have qualitative signal. In some cases you bind really strongly, in other cases you bind weak, right? So it's a continuum of binding. So you have these beautiful protein-DNA binding experiments, and our goal is to now use this sort of information to try to decipher the underlying syntax. And by that I mean the words, the lexicon, as well as, you know, rules of preferential spacing, arrangement, combinations, and how all of that drives cooperative binding of these proteins. All right, so now let's move into the machine learning front of this. We now have data. How do we make sense of this data? So we can transform this data into a very classical machine learning task. So we're going to take this data, and we're going to use it to formulate a machine learning problem. And the formulation is as follows. I performed some experiment. And I got a readout across the genome. Every base in the genome is assigned a signal from the experiment telling you the likelihood that the protein is binding that location. It's a quantitative signal that has beautiful shapes and spans, as I showed you before. And what you can do is you can take the entire genome, 3 billion bases, you can chunk them into little bins, let's say 100 base pairs, right, 100 letters, contiguous bins across the genome. And now every— each of these bins has a sequence, and that sequence is now associated with a molecular readout. So what you can do is you can set up a classic machine learning problem, which is a supervised problem of having some inputs, in this case sequences. You're trying to learn a mathematical function that takes these sequences and learns to map them to some kind of quantitative readout from the experiment, right? So the experiment gives you labels across the entire genome, And then you have millions of bins of sequence and their corresponding labels from the experiment. These labels could be binary, right, like it's a bound, unbound, or it could be quantitative, which will tell you the actual signal and the shape and the strength of the signal. So it's a classical, you know, supervised machine learning problem going from sequence inputs to labels. You're trying to learn this function f(s). Classic ML, right? You can apply any machine learning model to this thing and you try to learn this mapping function. What we've done is we've used deep learning models to try to solve this problem. What deep neural networks do is— why they're amazing is not just because they're very powerful, but one of their unique superpowers is their ability to engineer complex features de novo from raw data. So when you're given a sequence, you don't know how to exactly represent it. Like, should I represent it as, as, you know, a bag of words? If I'm representing it as a bag of words, should it be words of size 3, 4, 5, 10? I don't know, right? So instead, a neural network can just take the raw DNA sequence, okay? And the neurons in the neural network are pattern detectors. Their goal is to try to learn elemental patterns in the sequence. And the lower layers learn simple patterns, and the higher layers of the network combine these simple patterns to make more complex patterns. And by stacking more and more layers of neurons, you can create arbitrarily complex models that can learn very, very complex patterns and syntax and grammars, and then act as transducers to take these sequences and transform them into these profiles coming out of the experiment. Think of it like a text-to-speech converter, right? The input is text, the output is speech. In this case, the text is the genome and the speech is this spiky signal coming out of a sequencing experiment. So it's very analogous, right? And the neural network is learning these features de novo from the genome and it's trying to translate or transduce this genome sequence into speech. Experimental profile. So you can train this, this model across the entire genome. As I mentioned before, there are 3 billion base pairs, so there's lots and lots of information from a single experiment. And these kinds of models, we've recently published these last year, BPNet. They are state-of-the-art models that can take DNA sequences and map them to single base pair resolution profiles from a variety of different kinds of experiments, okay. And now once you have these models, what can you do with them? Well, they act like abstractions of the entire experiment. You learn to take the experiment and you projected it onto the genome. You know how to convert the sequence now to a profile. If the model's able to do that very accurately, it must have learned biology, right? Otherwise, how is it able to do it? It must have learned the biology of the underlying information, the syntax encoded in the genome, that's the only possibility, or it could have overfitted or some other technical issue. But let's assume that the models are great and they've actually learned something useful. We might be able to use them to discover biology. The patterns that the neural network is learning could give us really cool insights into the underlying genome syntax. But neural networks are typically thought of as black box models. We cannot understand what's going on inside. They're just powerful predictors. Well, I'll show you that's not the case. Over the years, we've developed actually many tools to try to open up this black box. One of the tools is very simple. What it does is once you train a model, once you learn the parameters of this model and it has learned to accurately predict profiles from sequences, you can take the model and you can invert it on its head, and you can take any prediction from the model and you can backtrack through the neurons all the way down to individual bases in the sequence. And get a contribution score of every base in the sequence to the output profile. So it tells you like what is the contribution of every base in that sequence to the output profile. So the first thing I'll say is— I'll show you how we make use of this to understand syntax. So the first thing I'll say is that these models are incredibly accurate. The recent ones we've developed, they can make predictions at single base pair resolution that are in some cases indistinguishable from replicate experiments. So if you did the experiment twice and you saw how similar the experimental noise is, right, how— like, that's sort of the upper bound of how similar the profiles could be. That's what— our predictions are as good as the replicate experiment concordance. So here I'm showing you, these are 4 proteins. Don't worry about the names. These are very famous proteins, actually, for those of you who might know them. Used to reprogram cells. You can take, you know, somatic cells like skin cells and reprogram them to stem-like cells. And these 4 proteins, if you overexpress them in cells, they can reprogram DNA and push cells back into a stem-like state. Anyway, that's an unnecessary detail, but like, let's, let's take these 4 proteins and they bind the genome at many locations. We can train these kinds of models, and on a subset of the genome we can hold out entire chromosomes, and then we can take the sequence of chromosomes not used in training and then predict what the underlying binding would have looked like. And these, again, these squiggly profiles, those are the shadows of these proteins. And you can see in many cases the predictions are almost indistinguishable from the actual data, right? So if we evaluate this genome-wide as well, we actually see extremely high concordance. The blue curve corresponds to the similarity between replicate experiments, and the red curve corresponds to similarity between the predictions of our model and the actual experiment. You can see that in many cases they are right on top of each other or very close to each other. So we are kind of maxing out, you know, how much information we can pull out of the models, right? They're very accurately able to take sequences and map them to these profiles at single base resolution. So now what we can do is invert them and figure out the underlying features in the sequence that are driving these predictions. So here's an example of how we do that. Here's a control element that actually regulates the Oct4 gene itself, and Oct4, Sox2, Nanogain, KLF4, all four are binding this element. And you can see our models are making these nice predictions of these squiggly profiles corresponding to the binding shadows of these proteins. As I mentioned, we can take each of these predictions from the model, and we can reverse engineer which bases in the sequence are driving each of these profiles. There's one model for each protein, and we're taking the same sequence and interpreting that same sequence in the context of models for 4 different proteins. Now imagine doing this for 6,000 proteins and hundreds of cell types and tissues. You can take the sequence of the genome, any piece of it, and you get this beautiful dynamic interpretation of how the sequence is being utilized by different proteins in different contexts. And so here's an example you can see right here. You see this, these letters really pop up. So there's this DNA word which is very important for binding for all four proteins. This DNA word is specifically important for this protein. This DNA word right here is very important for this protein. And you can see the combinatorial logic starting to kick in, right? There's one recognition code for like this protein OCT4, that is important for all 4 proteins, but then there are specific other recognition codes for other proteins. And the combination of these words in a specific syntax arrangement gives rise to the beautiful binding profiles of these 4 proteins, right? And we can map these to DNA words, so we can get a very high-resolution, context-specific annotation of which bases in the genome are actually driving all of these binding profiles. So that's great. This is the way we can actually decipher words in the genome. Now I mentioned about syntax. Now what we can do is we can play games with the model. It's a very accurate model. We can use it like an in silico oracle, and we can do all kinds of perturbation experiments in the computer, right? So there are two types of perturbation experiments I will show you. One is using synthetic sequences. So we can take sequences, we can take random DNA, We can push it through the model. The model should predict nothing, right? Just a flat line. Nothing's happening. We can insert an interesting DNA word that we discovered in the genome. We can insert another DNA word. And we can, for example, change the spacing between them, right? And for each of these spacings, we can push that through the model, and the model will predict how the binding of different proteins would change as a function of spacing between these two words. This is a way to do it in a very marginalized synthetic setting where you're controlling for very specific properties. In this case, the spacing between two words. You can also do it in the genome. The genome is very complex. It has all these combinations. There's a lot of overloading of information. But you can do a mutation experiment where you discover words, as I showed you in the previous slide. You take an actual genomic control element, use a model to discover the words, and you systematically make mutations. You mutate this word, you mutate this one, you mutate this one, you mutate combinations, and you again use a model to predict what would happen if you made these mutations. So it's a way to dissect the higher-order architecture of these sequences. So I'm going to show you some really cool simulations and how that can be used to actually give really interesting insights into syntax. So here's one example of what we call in-silico reporters. This is an example of we construct a synthetic DNA sequence, and then we insert this DNA word at this position, and we insert this DNA word at this position, and then we make several versions of the sequence. They're identical except for this word moving towards this word. We're reducing the spacing between those two words. We have two neural network models, one trained to predict the binding of this protein, nanog, and the other trained to predict the binding of this protein, OCK4. And what you're seeing is the model making— it's a GIF that is showing you the model making predictions for these two proteins, the binding shadows, as these two words— you change the syntax, right— moving towards each other. Okay, and you can see some really cool, like, there's some dynamics, you know. You can see this nanog signal kind of popping up and down. There's some kind of interesting periodic behavior. As this ox-ox motif starts coming towards it. So we can actually graph this. On the x-axis is the distance between the two words. That's the syntax, right? The syntactic distance, smooth change in the syntactic distance. On the y-axis is the fold change or the change in the shadow of the protein. And I'm showing you two shadows. One is the shadow of the nanoprotein that's graphed in golden.. And then there is a shadow of the Octo protein graphed in red. And you see something awesome pop up, right? You see that as a function of distance, this Octo protein doesn't care where Nanog's word is. This word, it doesn't care about the spacing syntax at all. It stays exactly the same, right? But if you look at the other way around, the other side of the coin, if you look at the response of this Nanog protein as a function of distance, you see this, firstly, this amazing exponential rise. In binding. So there is some very strong cooperative effect. As this ox ox word comes towards a nanog word, it amplifies nanog's binding exponentially. And then as you get closer, you actually see this beautiful periodicity kick in. And this periodicity happens to be exactly 10.5 bases, which is the helical turn of DNA. So DNA is a double helix. And what this is telling us, the neural network has learned this exquisitely fine syntax asymmetric causal syntax where there are two proteins that cooperate with each other. One of them is the master, the red one. It binds and does its thing. The nanoprotein dances around the master as a function of distance syntax, and there's a preferential soft syntax. It's not a hard syntax. There's an amplification of binding up to 150 bases away. And then when you're really close, you prefer multiples of 10.5, which means the two proteins are binding on the same side of the helix. When they bind on the same side of the helix, they cooperate. When they bind on the opposite side, they do not. And this is a beautiful example of how a neural network that has been trained just on raw data, it's been given sequences and it's been asked to learn how those sequences map to these squiggly profiles. It has learned something very fundamental. It has learned about, an asymmetric relationship between two proteins as a function of sequence syntax, spacing syntax, and it's learned a soft constraint involving a very fundamental property of DNA, which is the helix, right? So this is the power of neural networks. They are not black boxes in the sense, yes, they are black boxes, but you can query them like oracles and you can pull out really beautiful information. And this has never been annotated before. This is a new discovery that has been made by a neural network, right? We can do this for thousands of proteins, and we've been doing that now. And what's also exciting is we can repeat the same experiment not just in synthetic DNA but in actual genomic sequences. So here's a control element that is bound by Oct4 and Nanog. As I showed you before, there's this DNA word that's recognized by Oct4, and this DNA word, this combination of DNA words, is recognized by Nanog. So we can mutate this DNA word And you can see the model predicts Oct4 binding dyes and Nanog binding dyes. So this word is very important for binding of both proteins. We can mutate this word, and you can see again, the Oct4 protein doesn't care at all. The Nanog protein gets hit pretty bad. Even though Nanog binds this specific word, even though we deleted its word, it can still be indirectly recruited by this other protein. And if we perform this kind of mutation experiment across the entire genome and we average it out, we again pick up this beautiful asymmetric cooperativity, where the red protein is the king. It does its thing. And this golden protein, the Nanog protein, depends on Oct4 in an asymmetric fashion. There is a distance-dependent syntax. And the neural network once again tells us that there is this beautiful helical periodicity. So this is really cool. I mean, we have been able to discover all kinds of really novel syntax through the lens of the neural network. And then we can use this syntax to design actual genome editing experiments, not just stop at predictions, but we can now design the next set of experiments to validate this biology. So we can go into the lab. Our collaborator Julia Zeitlinger has done these beautiful experiments where we actually go into real control elements in cells and we make mutations using CRISPR. So here's an example of what the model predicts would happen. Here's the DNA word that's bound by the SOX2 protein. Our model tells us this TT, these two Ts are very important. And if you mutate them to AG, there's going to be a pretty big effect. So we're going to do that. And you can see that the signal drops pretty strongly here. But there's also attenuation quite far away. We perform the actual experiment, and that's exactly what we see. We see complete loss of signal here, and we see attenuation here. So this is breaking this DNA word on the binding of SOX2. We can then break the same word but now measure the binding of the other protein, Nanog. And again, you see this amazing correspondence between the predictions of the model and the experiment itself. We can invert this and we can break another word nearby, the nanog word, and here again you see a local effect. And then we can break this nanog word and measure the effect on SOX2 binding, and you see no effect. This goes back to the asymmetric relationship that we had predicted before. So we can use the models now as oracles. We can make thousands of millions of predictions. We can highlight and identify specific ones that are really good candidates for testing in actual CRISPR experiments, and we can perform the experiments to decipher syntax. All right, so that's the first part of the talk, and in the next about 5 minutes, I'll try to conclude and give you a view of how we can use this to decipher genetic variation. So again, going back to the case control studies, as I told you before, we can identify millions of genetic variants, common and rare, across human populations. So how do we use our models to study what these variants are doing? Well, we can again use our models as oracles, and I'm going to show you how here the model makes predictions about single nucleotide mutations. So here's a piece of the genome. It's a control element in a specific cell type. Healthy individuals have a C at this position. So we take the model and we make predictions about what would happen if you saw a C. The model predicts this squiggly profile for a particular protein. Now what we do is we swap the C to a G. We keep the rest of the sequence the same. And we again push the sequence of the model, and the model predicts what would happen. And you can see the model predicts that if you swap the C to a G, you get this dramatic amplification of binding of this protein. So we can do this for thousands of variants. We can just push them through the model, and we can actually measure— we can predict what the effect of that mutation would be. We don't have to stop there. We can actually use a model as a lens to understand what is this mutation actually doing. So here's what it's doing. We can use our interpretation tool to try to derive the contribution scores of individual bases in the sequence for the sequence containing the C and the sequence containing the G. And you can see the neural network tells us an awesome story. It tells us that when the C is present in the sequence, There's this DNA word which is weakly driving binding of the protein. When you swap the C to a G, it creates this really strong DNA word. And these two DNA words together cooperate to dramatically amplify binding of this protein. So now we have a very specific molecular mechanism through which the mutation is working. So I'll show you how we do this for Alzheimer's disease. Remember I showed you this Manhattan plot? There are thousands of variants in the genome, hundreds of variants in the genome associated with Alzheimer's. What are these variants doing? Well, we can actually use our models. We can actually get molecular profiling data from the adult human brain. We can actually get this at single-cell resolution. We can actually take all the cells in the brain, we can segregate them, we can perform molecular profiling at single-cell resolution. Here you can see, for example, Each row here is a different cell from the brain, and the x-axis here is a piece of the genome. For each cell, you get a readout corresponding to whether that region is active or not, or is bound by a protein or not. We're getting this information at single-cell resolution. We can take all of this data across the entire genome, millions of cells or tens of thousands of cells, across the entire genome. What we can do is we can compute similarities between the cells in this activity space. Imagine you take each cell and you compute the similarity of the vectors of this genomic activity from the experiment. You can get cell-cell similarities and you can cluster the cells together into groups that have very similar molecular profiles. The amazing thing is we can discover all the cell types in the brain completely de novo computationally. So here's an example of a two-dimensional sort of embedding of all the cells. Each point here is a cell, and you can see cells cluster together. These are all microglia, these are different kinds of oligodendrocytes, these are inhibitory neurons, these are nigral neurons. So we can kind of de novo take an entire adult human brain, we can perform these experiments at single-cell resolution, we can discover the cell types,. And we can then obtain cell type specific molecular maps of genomic activity. So we get this, and then what we do is we train neural networks on each of these experiments, like I showed you before. The sequence mapping neural networks, they take each of these experiments and they try to figure out how the DNA sequence is predictive of these profiles. And then we can take all these mutations in the Alzheimer's disease experiment the case control experiment, we can propagate it through each of these models. And each of those models will tell us what this variant might be doing, which of them have big effects, which of them have small effects in different cell types, in all of the 40 to 50 different brain cell types. And we can interpret the models to understand what these mutations might be actually disrupting. So I'll show you one cool, quick case study. Here's a locus in the genome that is associated with Alzheimer's disease from the case-control study. There's a gene right here called PICAM, okay, and there are 165 variants right here that are all inherited in as a block. So it's very hard to figure out which variant is actually causal, okay, because when a, when a person inherits one of these variants, all the variants that are in that block get inherited together. So in an association study, you can't really tell which variant is causal. Now we can use our molecular maps to start to home in on the likely causal variants. So we just take all of these 165 variants and we overlap them with our molecular maps in all of these cell types. We can restrict ourselves to 24 variants that fall in any of these control elements, okay? Wherever you see the spikes, it's potentially a control element. You can see some of these spikes are very specific to specific cell types. This is the oligodendrocyte cell type, and this spike is a control element that is specifically active only in oligodendrocytes. This spike is a control element that is active in all the cell types, right? So again, this beautiful dynamic activity of these control elements, and if you just overlap them with the data, you still don't get to, you know, potentially which variant is causal. It could be any one of these 24, Now what we can do is take our models, we can push these 24 variants through each of these, you know, 50 models for each cell type, and the models will tell us which of these variants likely have a strong effect on activity of these control elements. And the amazing thing, our models home in on just one variant in this locus, and that variant happens to be in the control element that is active in this one cell type. It loops in 3 dimensions to regulate the activity of this PIKAM gene. And when we use a model to interpret what the variant is doing, the models tell us that when you swap the A letter to a G letter, it creates this DNA word, which is the binding site of this protein called FOS. So we can come up with a very specific prediction. This genetic variant RS123799 it disrupts the binding of the FOS protein in a control element of the PIKAM gene in oligodendrocyte cells in the brain. This is the level of detail that a neural network can allow us to dive into genetic variants and give rise to very specific hypotheses about function. I'll stop there. I think we're out of time. I just wanted to say quickly that we don't have to do this only for common diseases. We can also do it for rare diseases like We've taken de novo mutations in autism, and we've again performed these kinds of single-cell molecular profiling experiments in developing human fetal cortex. And we can take de novo mutations in these patients and actually try to decipher which one of them is functional. We can make beautiful predictions like I showed you before, creation of this DNA word when the mutation occurs. We can go to congenital heart disease,. And we can again look at molecular profiling data from human fetal hearts, very, very early development. We can fit our models to all this data, and then we can make exquisite predictions about which mutations in these children suffering from congenital heart disease might be causal. We can get to actual DNA words, proteins that bind them, likely target genes, and the cell type of action. And more recently, we've actually performed CRISPR-Cas9 experiments in vitro in cells. And we've validated many of these mutations. So it's really nice. And neural networks are acting like oracles, allowing us to hone in on very specific mechanisms and design sequences to get deeper insights into genomic syntax and disease variants. All right? So with that, I'll just conclude and say I hope I showed you that large-scale molecular profiling data allows you to decipher genome function. Neural networks can map DNA sequences to molecular profiles with unprecedented accuracy. The models are not uninterpretable black boxes. You can actually query them like oracles to decipher functional DNA letters, words, syntax. You can decipher effects of disease mutations and the underlying biology. And you can use the predictions to design downstream genome editing experiments for validation.. And these predictions can hence provide clues for potentially interesting therapeutic interventions. And so with that, I'll thank my lab members who are amazing. They've been building these frameworks for the last 7 years. It's been really quite a journey, and I'm very proud of what they've accomplished. And of course, none of this work would be possible without our collaborators. My lab is purely dry. We just do computational work. These are the fantastic scientists and their lab members who generate all the beautiful data I showed you. And we work very collaboratively in large consortia.
And of course, I want to thank my funding sources as well. Thank you so much, Anshul, for a fascinating talk. I'm just— it's going to take a while to fully appreciate everything you've spoken about. You'll see that people have entered some questions in the chat and also the Q&A. So let's see, it could be confusing to toggle back and forth.
I don't know how long you can hang around for.
Um, I have, I have 15-20 minutes, so I can, I can take as many questions as people. Okay, awesome. Well, we have— it looks like we've left the chat and the Q&A open, so, um, are you able to see those, or would you like me to? Yeah, I can see them.
I can just walk through each of them. Well, yeah, please do. So Will Connell asks, uh, in, in the, in the Q&A, where do you see these models fail? Excellent question. So the models do fail. They fail in two contexts. One is that the current models I showed you are models of local sequence. We're taking 1,000 base pairs of sequence and mapping them to profiles, right? But the genome is not a linear string. It is actually a very complex 3-dimensional molecule, and there are long-range interactions between different parts of the genome. And so what we're now trying to do is build chromosome-scale models that can take all the information in the entire chromosome and predict what happens at any single base in the genome, because there is long-range interactions, there's long-range transfer of information. So we need sort of models with larger context, sequence context, and those will improve predictions of gene expression and so forth. The other places where our models fail is If you have an amazing neural network and you train it on data that's from the wrong cell type, you're never going to get a right answer, right? So if you're studying Alzheimer's disease and we took data like molecular data from muscle, it's not going to help us figure out Alzheimer's disease. So like the data is actually very important, and so we actually spend a lot of effort trying to make sure we get the right cell states, we get the right kinds of data. And that's another point of failure, that is if we don't have a good match between the disease the cell state and the model, your models can make some random predictions which are going to be completely nonsensical and not going to work. OK, so that's sort of one mode of failure. I think the next question, which is from— I hope I get your name right— Onur Alt. It says, would be great to hear your thoughts on the best ways to crowdsource predictive models for genomics. Oh, thanks for asking that question. I skipped a slide. So we've actually tried to do this. We've tried to democratize this entire effort. So we've built one of the first model zoos for genomics. It's called Kipi, kipi.org. And we are basically openly sharing, not just us, but the entire community. It's a community effort. We are openly sharing all of these models so that people can— and we have beautiful APIs around these models that can allow you to do very quick things with them, mix and match them, make predictions, and so forth. So we hope that people will, you know, continue contributing models to this Model Zoo, develop new Model Zoos, and that might be a very nice way to build resources that can be used by many. Okay, Fu Long Yu says, how many single nucleotides will directly disrupt transcription factor binding? I'm not sure that question— I can answer a question exactly because there is potentially some context missing. If you're talking about like all the genetic variants in the human genome, how many of them disrupt transcription factor binding? Probably a very small number, less than 1%. There are like millions of genetic variants and most of them are not doing anything. They're just there. So, you know, maybe about 1%, less than 1% of them actually would be disrupting control elements. And while there are millions of control elements in the genome, they don't occupy 98% of the genome. They are in that space. They probably occupy about 10 to 15% of the genome. The rest of the genome probably doesn't do much. It's probably just neutral. Okay, next question. David Arnosti asks a question about the call quality of the datasets and their impacts on neural networks. We experiments are accustomed to ChIP or ATAC-seq data that show low reproducibility. Excellent question. So what Dave is asking is, how dependent are your models on the quality of the data? I don't have slides to show you today because I try to keep the talk slightly high level. But one of the advantages of modeling the data in a quantitative way at single base resolution is that you make the models quite immune to some biases in labeling when you're working with datasets of very different quality. So for example, people often used to take this data and binarize them— high signal, low signal. And what can happen there is if you have low quality data, then your sensitivity is low, and so you have much fewer so-called high signal regions, and you're going to mislabel several regions that actually have high signal as low signal just because your experiment has a low signal-to-noise ratio, right? That's going to mess up your neural network because the neural network is going to take the labels and think of them as ground truth. And if your experiment is sort of having label noise, that can be very problematic. So we directly model the data at single base resolution, the quantitative data that's coming out. And I don't have the slide here, but we've shown that you can take high-quality datasets and you can corrupt them you can train the models on the corrupted data and you can make predictions. And the predictions actually, because they're learning from sequence, they can fill in the gaps and they can impute missing information to the level of high-quality datasets. So we have nice benchmarks where we've done artificial corruption of the data, trained models, and seen that they can actually denoise and impute missing information. And that's how we've been able to train this on single-cell data, which is very sparse, very noisy. Still works out very well. Okay, so of course data quality matters, but these models are much less dependent on the artifacts of data processing and data quality than previous generation models. Okay, Xing Tang asks, have you tried to train a model with mouse data? Oh, that's such a good question. I don't have— I didn't have enough time and space to show you slides, but we have some really beautiful recent results where we've trained these models on single-cell data in mouse models. We've been trying to study coronary artery disease, and we would love to like look at coronary tissue, you know, in, in the presence of a high-fat diet. Now, of course, you cannot do this in a human. We're not going to feed humans high fat and then sacrifice them and get their aortas, but we can do this, do this for mice. So we actually have a mouse model that one of our collaborators has been using to feed them high-fat diets. We can extract coronary arteries for them. You can do these kinds of single-cell experiments. We've trained models in the mouse, and the beauty of these models is they just take sequence and they make predictions. So even though the model has been trained in mice, we've transferred them over to humans and we use them to score genetic variants associated with coronary artery disease, and we're seeing really amazing results. And in this case, we have human data and mouse data. And they agree very nicely with each other. So even cross-species transfer of models is feasible, not in all cases, but in many cases. We at least have evidence of that. Next question. Do you think the affinity of a TF for a certain nucleotide at a position is independent of the nucleotides it's binding in nearby positions? So I want to distinguish between affinity and occupancy. Affinity is really a property of the underlying sequence, and that is what it is. The transcription factor has affinity to the sequence, right? But you can modulate its occupancy based on other proteins nearby. So effectively, you can change its affinity to the overall sequence by having other sequences nearby. So the example I showed you with the Oct4 and Nanog is a great example where you have two motifs right next to each other. And as you change the spacing between them, it can change how the proteins bind, their occupancy and the effective affinity of the overall sequence. So this is pervasive. This is not like a one-off example I gave you. We see extensive examples of cooperativity driven by the underlying sequence syntax. So it's not that each base acts independently. There's lots of complex interaction effects between bases, between DNA words, as a function of soft spacing. And that's not just local, it can also occur long range. So you can have a control element like a megabase away from a gene, and it can affect the activity of a gene very far away. And you can have tens of these control elements acting additively or superadditively or multiplicatively, right? So there's very complex nonlinear effects and complex interactions happening. Okay, uh, next question. How do we normalize input data in chromatin-based measurements, say single-cell ATAC-seq? How do we take care of batch effects, GC content, etc.? Very good question. Um, how do we normalize the data? Uh, we do not normalize the data, and the reason is because, uh, let me finish my sentence, because it's very important to account for normalization biases. But we do not normalize the data. We literally fit to the read counts that you get from the sequencing experiment on the genome. And the reason is because the classical approach has been to first normalize the data, correct for batch effects, correct for various things, and then fit a machine learning model. The problem is when you do that, you can overcorrect or you can undercorrect. You don't know whether you're doing the right thing or not, right? Because it's kind of two independent steps. You first correct the data, then you fit a model. I didn't get to tell you this, but our models, we actually account when we're fitting the models to the data. We don't just fit to the data, we include a whole bunch of covariates. For example, if you're looking at single-cell ATAC-seq data, we include the TN5 enzyme bias. So we have a bias track, and the bias track is added as a covariate. We can add GC content as a covariate, we can add batches as a covariate, and we fit a model that tries to predict the raw data, uncorrected data, as a function of the DNA sequence and all of these other covariates. The model learns how to do it optimally, so you don't have to pre-correct it. We correct it on the fly, and we find that to be a much, much more effective approach. It's a more intricate approach, but really allows us to avoid overcorrection or undercorrection, and the machine learning models are powerful enough to be able to do this de novo. Okay, so that's our approach. Next question is by Xing Tang. Larger context need larger training data. Could you explain how chromosome-level context models are doable? Good question. Larger context doesn't necessarily need larger data. If you think about like, you know, a text-to-speech converter, or in this case a convolutional neural network that's going from sequence to profiles. Depending on the number of layers you add— this is a little bit technical— but depending on the number of layers you add, you can increase or decrease, or how you design the architecture of the model, you can increase or decrease what's called the receptive field. And by receptive field, what I mean is, you know, for any output position, how much of the sequences it effectively sees. Okay, so if you have very few layers, it might just see 100 base pairs. If you do various kinds of pooling operations, or you have dilated convolutions, or use a transformer model, or some such other architecture, you can increase the receptive field of the model. So it's still training on 3 billion observations across the genome. So it's a lot of data. And your architecture choices you make can allow you to increase or decrease the receptive field. It is true that at some point, depending on how sparse the data is, you may need more training examples. But we have really cool evidence that, for example, we've trained these models in the yeast genome. It's a very small genome, so not that much training data where you have a few hundred events, binding events, and the models still do extremely well at prediction. It appears that even though the genome syntax is very complex, By leveraging the data at single base resolution and the quantitative nature of it, these models can be made to operate and fit the data quite well, even in low data settings. Okay, Sandra Batista asks, can you give examples of using models when protein interaction is not known before? For example, interaction between NANOG and OCT4 is known, but how can models be used when such interactions are not known? Excellent, great question. We picked, in this particular case study, we picked Oct4, Sox2, Nanog, and Klf because, precisely because they are well-studied proteins, and we wanted to see whether we can learn existing biology and whether we can learn new biology, right? So we obviously captured many known things about Oct4 and Nanog, but this sort of, this spacing syntax, this 10.5 base pair periodic syntax, for example, that hasn't been documented before. But we've now applied this to literally 600 transcription factors in 7 or 8 different cell lines. And we are picking up all kinds of beautiful novel discoveries because we don't have to, you know, we can just pick any pair of proteins and we can do this kind of syntax analysis and see what the models tell us. So there's incredible repertoire of syntax that these models have learned, and we are working towards a resource hopefully by the end of the year as part of the ENCODE project, where we are trying to systematically document sort of the syntax of the human genome from the context of all the experiments we currently have our hands on.
Okay, so that's the questions in the Q&A, and let's see in the chat. I think the people from the chat came over to the Q&A. Oh, great. Okay, okay, excellent. Yeah, thanks so much, Anshul. This is, uh, fascinating and mind-expanding. I feel like, you know, it's a glimpse of a world that I sort of knew was there, but I didn't even have a handle on it before. And it was really, really an accessible presentation of an extremely complex topic.
I really appreciate it. I'm sure our audience does too. Thanks so much. Yeah, I mean, I just want to again encourage people to explore models in a more open-minded setting. Neural networks have been difficult to interpret in many domains, and they are. In genomics, thankfully, we have some inherent causality because we have sequence mapping to other things, and there is a directionality that is implicit. That can allow you to do a lot of exciting things with observational models like these neural networks trained on observational data. And I would encourage folks to think about how we can use these models as oracles and make novel discoveries, and also for experimental design.
So thanks so much again. Great, thank you, and have a great day. I will be posting your video in a few weeks, but I'll run it by you first.
Sure, thanks. Okay, bye-bye.
Bye-bye.
More from Joshua
Scientist Stories: Hans Waterham, The value of phenotypic assays in peroxisomal disorders
Scientist Stories: Joel Zlotogora, Genetic carrier screening programs for reproductive purposes