Yunha Hwang, a Harvard PhD student, presents a genomic language model (GLM) trained on the MAGNIFI metagenomic contig dataset, which contains 15‑30 protein‑coding genes per contig from diverse environments. The GLM treats each protein as a token, uses ESM2‑derived protein embeddings, and is trained with a masked language modeling objective to predict masked protein embeddings from surrounding context. By capturing higher‑order genomic context, the model produces contextualized embeddings that improve downstream functional predictions, such as enzyme commission (EC) number classification, and reveal regulatory syntax, notably operon structures, via specialized attention heads (≈77% average precision on E. coli). The talk discusses the limitations of current functional benchmarks, the importance of genomic context for protein function, and future directions: scaling the model and data, adding non‑coding DNA modalities, extracting residue‑level information, and developing functional probes (e.g., protein‑protein interaction graphs). Potential applications span bioengineering, synthetic biology, microbiome‑human health interactions, and pathogen evolution. Audience questions address extending the approach to eukaryotes, context window size, and positional/orientation encodings, with answers indicating ongoing work on larger windows, relative position embeddings, and orientation encoding.
Chapters
Hello everyone, my name is Yuna and I'm a PhD student at Harvard. And I'm really excited to share our work on genomic language modeling where we use machine learning to shed light on functional dark matter. So when we take a step back and look at the genetic diversity that's present in nature, one thing that still astonishes me is that there's a lot of diversity, and a lot of it is really found in microbes. So out of— so there was an estimation that was made a while ago that there's about 1 trillion species on Earth, and 99.999% is considered microbial. And this is bacteria, archaea, viruses, and single-cell eukaryotes. And this is largely because for a large chunk of Earth's history, life on Earth was mostly microbial. So about 3.7 billion years ago, life emerged, and 3.2 billion years was purely microbial. And that's why there's a lot of biochemical pathways that's unique to microbial genomes that's found in these large amounts of data that's we call metagenomics. So if our objective is to learn the language of biology, then we really need to leverage the diversity that's present in microbial genomes. So where do we get this data? So about 10 years ago, that was when metagenomics as a method really became popularized. And metagenomics is a method in which we take a sample. And without needing to culture it, we can just extract DNA and then sequence that DNA. So, and an important aspect that comes out of this method is that even though only less than 0.1% of microbes are cultivable, now we have access to this extremely diverse and unbiased data set that we call metagenomic dark matter. And it's dark matter because we know it exists, but we just haven't seen it in the lab or haven't characterized it fully. And I think this is a paper that came out very recently by folks at JGI, Joint Genome Institute. And what they did was they looked at all the protein sequences that were present in the dataset, and they looked— they clustered them using sequence, and they found that as more samples become available, cluster number or protein sequence cluster number increases linearly with reference genomes, but exponentially and without reaching a plateau for metagenomes. So even though we have a lot of data, we might have just scratched the surface of the diversity that we're looking at here. So there's been a lot of really cool progress that's been made in structure. And structure is very important, but far from everything to understand biology. Function cannot be directly inferred from structure. And proteins do not exist in a vacuum. A lot of times what protein does is not encoded in the protein itself, but how that protein interacts with other proteins physically or in a pathway, and how that protein functions in different environments. And proteins with similar structure can perform vastly different tasks, and proteins with different structures can perform identical functions. And when we take a step even back and look at the regulation of proteins, which is where the function is really pulled, regulation of proteins is highly poorly understood. Because the regulatory sequences are very poorly characterized. So if we can predict structure for the most part, what do we need to predict function? I think some key element that we've been missing is genomic context. These proteins occur in a genome, and where that genome— where it's found is is telling of its function too. So evolutionary processes constrain sequence structure space, and that makes protein structure prediction feasible and using ML methods. And evolution also acts upon gene content and genome organization. Horizontal gene transfers, gene duplications, loss, and inversions, as well as modular organization of genes is highly selected upon. So we started with this question of, can we use higher order genomic information to learn a gene's function? With that, we would like to introduce a genomic language model. And similar to natural language modeling and protein language modeling, what we start off with is this large amount of data. With that is metagenomic corpus. And using genomic language modeling and a simple pre-training task, our objective is to represent these genes that capture the semantics or the functions or syntax of genomic language. And then that can be then fine-tuned to learn novel function and discover new circuits and so on. The dataset we began with is also from MAGNIFI, which is an EMBL dataset that consists of metagenomic contigs—contigs just refers to contiguous stretch of DNA—that contains 15 to 30 protein-coding genes. And it's important to note that these contigs or sequences are sourced from all kinds of different environments, and this includes oceans, soil, volcanoes, and so on. And in that dataset, we have 31 million proteins that were clustered at 90% amino acid identity. So for GLM training, we also use masked language modeling objective. So what we have here is 30 genes in a contig, and we have abstracted away the DNA sequences that are in between. And we just currently focus on protein coding sequences. So we translate the DNA sequences of the protein coding sequences into protein sequences or amino acid sequences that can then be transformed using a PLM, in our case ESM2, to obtain protein embeddings of same feature length. And our goal here is to mask out random percentage or percentage of random proteins and then have GLM predict the mass protein embedding given the context. So if protein language modeling looks at each amino acid as a word, for us each token is the protein itself. And in inference time, what we do is we freeze GLM, and PLM is already frozen. And we take a sequence of interest or a contig of interest, and then we use the contextualized protein embeddings and attention patterns to learn the functional semantics and regulatory syntax. Probing function in biology is actually a really difficult task. Structure— there's been a lot of benchmarks for structure, and it is is 3-dimensional, whereas for function, it's a lot— there's a lot more dimensions to function. And when we also look at the swaths of biology and organisms that we're looking at in this study, there's not been a lot of effort in developing benchmarks that are functional benchmarks that apply to all different types of organisms and biology at large. So we tried different methods and report this in our preprint, and but for the interest of time, I would like to focus on two downstream tasks that characterize the function and syntax or regulation. So one functional aspect that we were curious about was enzyme function. So enzymes— not all proteins are enzymes, only about 15% of the proteins are enzymes.. And some of them have enzyme commission numbers attached to them, which is a hierarchical classification scheme for enzymes that is based on chemical reactions that they catalyze. So there's been some work in using protein embeddings to predict EC number directly. And so we call this PLM embedding or context-free embedding. That's— and in this case, we have this a protein of interest that has an EC number attached to it, and then we're trying to predict that EC number given the protein embedding. And with GLM, what we're trying to do is, can we compress the context information as well to get the contextualized embedding that we call GLM embedding, and then do linear probing to predict EC number directly? And we show that Even with a very simple downstream probing, linear probing, we see general increased accuracy with contextualization. And when we look at different categories or different EC categories, we see a lot more ECs that are statistically significantly more high-performing. With contextualization than those without contextualization. And I think these sort of ends of the distribution will be interesting areas to look at as we further define enzymatic function. Another aspect that we were curious about was whether we can retrieve syntactic information from GLM. So syntactic information has been probed in LLMs or natural language LLMs using attention patterns. So in this example, they are able to retrieve the syntactic graph from the attention patterns alone. And the simplest regulatory syntax that we were curious about was operons, that's functional units of multiple genes. That are co-regulated through co-transcription. So what happens is there's gene A and B and C, and they're co-transcribed. And what these proteins not only work together, but oftentimes form a complex that conducts the same function. So we looked at— this is our models, layers, and heads. And so we looked at about 190 heads. And we find that there is specialization in attention heads that correlate specifically to operon structure. And when we use all that information that's captured in attention and use logistic regression, we're able to capture operon structures of E. coli with 77% average precision. So the key takeaways of this work is that genomic language modeling is a really exciting approach to take a step back and look at the genome and understand proteins' function in organismic setting. And since then, since our paper has been out, we have been working on the next iteration of GLM where we not only scale the model and data, but also include a new modality of non-coding DNA that's in between protein coding sequences, and we hope that that's going to help us understand the regulatory syntax of these really poorly characterized organisms and sequences. We also are experimenting with different ways to extract residue-level information from PLMs. Currently, we are taking a mean-pooled average of all the residues from ESM to get a protein embedding. But we think that a lot of evolutionary and functional information will be captured in these residue-level interactions and point mutations. So we want to focus our efforts in getting that finer-grained information to GLM. And I think another important aspect that we are focusing on is developing benchmarks for functional probes. For instance, protein-protein interactions really are sort of fundamental to protein function, and we're asking the question of can we get this graph directly from GLM. So I think manifold has been mentioned multiple times in the past few talks. I think what we're trying to do here is to learn the manifold of genomic information so that we can now have an understanding that gives us a tool to systematically illuminate dark patches of biology in terms of the functional landscape. And once we understand the relationship between the protein and its context, that enables us to not only understand biology but also harness biology. So when it comes to bioengineering and synthetic biology, we can now do precise— just like the way we're designing proteins, we can design pathways and design multi-gene circuits given the organismic context. And these proteins that are in metagenomic databases are important novel templates for enzymes that can do biocatalysis that fuels bioeconomy. And also they provide templates for materials, chemicals, and pathways. And even though right now we're focused on microbial data, still important to note that still has consequences in human health as we try to understand disease states in in human genome and microbial interaction with microbiome in human, and also predicting evolution of pathogens that are largely microbial. So with that, I would like to acknowledge our collaborators. And yeah, I'm happy to take any questions. Thank you. Really nice talk. Thank you so much. I was wondering, so since you guys only did microbial data, how would it sort of apply to, let's say, eukaryotic organisms that have interactions between genes through multiple chromosomes, or even like, you know, different subversions of genes because of the the isoforms and everything, how would you adapt? Yeah, that's a great question. So this is something that we're still working on. We're trying to see if we can go from pre-training tasks that's largely microbial because that captures the sort of diversity that's present in nature and then fine-tune it on human genomic data to see if there is— if that knowledge can be transferred to human or other mammalian systems. And I think a lot of— there's quite a lot of redundancies in cellular mechanisms. And even though sequences might be totally different, the evolutionary pathway, if we can map it, then we can deduce novel interactions that wouldn't have been possible if you didn't look at the full picture. Very cool. Thank you so much.
Thank you. I was wondering, do you see any differences in the size of the effective context window, maybe in different microbial species? Because we know that, like, the size of the microbial genome, they varies by, like, 2 or 3 orders of magnitude, right?
Yeah, that's a great question. So we have to restrict our context size to 30. We're still experimenting with larger context sizes. So obviously microbes have more than 30 genes, and we wanted to capture multiple operons. And given that most operons are about 5 to 10 genes long, so we just went from that and then wanted to capture 2 to 3 operons. So that's why we restricted our context size to 30 genes, but we're hoping to increase that context length so that we can capture longer range interactions.
And this is still work that's More of a technical question, but are you using positional encodings or any other types of protein-specific encodings along with the position?
Yeah, so, um, uh, yes, so we're using, um, uh, relative, um, position embeddings, um, and, uh, we also encode the orientation of the protein. So, so there are two, if there is the protein is more facing relative to the others, then that is the orientation.
Okay, great. Let's thank you again. And yeah, and sorry. And our last talk before the break is from Ron House, who's— this is the first of our sponsored talks. He's representing Shape Therapeutics. Thanks. I appreciate it. I mean, you have slides. Yeah, we do. We're really happy to be sponsoring this conference with a lot of really clever scientists that are involved in the field of biological sequence design, how to generate it, interpret it, and apply new algorithms to be able to do that. Just a little bit of what we do at Shape. Yeah, you reckon? Yeah. Probably better if we don't wait. And try to talk at the microphone because I think otherwise it's hard for people online to hear, apparently. Oh, really? Okay. I tried to do it now. Oh, you don't trust Keynote? Looks like it's doing okay.
