In this talk, Yiqun Chen (with advisor James) presents GenePT, a framework that repurposes off‑the‑shelf large language model (LLM) embeddings of NCBI gene description texts to create rich gene representations, and then derives cell embeddings either by weighted averaging of gene vectors (GenePT‑W) or by encoding a rank‑ordered gene‑name sentence (GenePT‑S). The motivation is to avoid the massive computational and labor costs of training single‑cell foundation models on tens of millions of cells. Chen first reviews the current landscape of biomedical foundation models—protein language models, GeneFormers, SCGPT—and highlights their reliance on huge datasets and transformer architectures. He then details how GenePT extracts textual gene summaries, feeds them through generic LLMs (e.g., GPT‑3.5 AdaText, LLaMA‑2) to obtain high‑dimensional embeddings, and uses simple algebraic operations to obtain cell vectors. Extensive experiments show that gene embeddings cluster by functional class, outperform dense co‑expression baselines on gene‑function classification, and achieve state‑of‑the‑art link‑prediction performance on gene‑gene interaction (GO‑based) and protein‑protein interaction tasks. Cell embeddings are evaluated on cell‑type annotation, batch‑effect correction, and integration, where GenePT‑S often matches or exceeds specialized models such as SCGPT and GeneFormer. A Q&A reveals that batch‑effect mitigation likely stems from rank‑based gene ordering rather than absolute expression, and addresses concerns about data leakage when GO terms appear in the source text. The talk concludes with future directions: combining textual and co‑expression embeddings, making the approach context‑aware for large atlases, and extending the paradigm to other computational biology domains.
Questions this video answers
The hypothesis is that representing a cell as a rank‑ordered list of genes (or using rank‑based weighting) diminishes batch‑specific absolute expression differences. LLMs may attend differently to top‑ranked versus low‑ranked genes, which helps to suppress batch‑specific noise while preserving biological signal.
The GO‑based interaction task is treated as a sanity check because the NCBI summaries sometimes contain GO annotations, so high performance there is expected. To avoid circularity, the authors also evaluated on protein‑protein interaction datasets that do not rely on GO terms, demonstrating that the embeddings capture information beyond the text labels.
Chapters
Hey everyone. Thank you so much for the opportunity to talk here. And I'm really excited to kick off our today's discussion by using a lot of interesting ideas from natural language models to apply it to single-cell data. And this is joint work with my postdoc advisor, James. So to start with, really, the conference so far has provided us with a really clear landscape about what the current biomedical machine learning is like, I guess, these days, where we see that we really have these really huge models also trained on massive data as well. We obviously know about the massive ability of the natural language processing models, but we also learned a lot about the protein language models from yesterday's talk about their ability to take in an amino acid sequence and then really encode the underlying biology that encodes not only the function, but also the downstream folding properties and evolutionary information that doesn't seem obvious to even a human expert. And we also have seen much more recently a lot of these foundation models that are built on single-cell data. So two of the names that you might be familiar with are referenced over here, GeneFormers and SCGPT, where they leverage this massive amount of data and claim that they're able to provide a really good understanding about what these RNA sequences are going to do in the downstream biology as well. So in particular, if you look through the recent literature, some of the features are going to really emerge out of these so-called single-cell "foundation models," and I really categorize them broadly into 3 different features. Feature number 1 is these models are often, as you would expect, large. So they often consist of a BERT or transformer-like style where you pre-train on massive amount of data using multiple transformer blocks and attention layers, trying to capture the co-expression patterns of genetic information. And the second feature, probably also not surprisingly, is the fact that to achieve successful representation, you really need massive amount of data to pre-train on. So in this particular cases, both of the methods have trained on more than 30 million single-cell data that are often collected across multiple studies and spans multiple organs and tissues of the human body. And the final feature, which to some is the defining feature of the so-called, quote unquote, foundation model, is the fact that these models, the representation they learn, can really be adapted to a wide range of different tasks. So in particular, for instance, some of the tasks that the authors have have demonstrated include cell type annotation, gene functionality prediction, as well as batch integration and batch effect correction. So with this in mind, often we know that there's no free lunch in the computation land. So a natural question that arises is that in order to collect and just even process these 30 million cells is a huge amount of undertaking on its own. And we also know that trying to train these large models on massive amount of data is also very not only computationally intensive, but also involves humongous human labor just to make sure that everything is working and there's no bug in your code. So with this in mind, the motivating question of our study is really trying to understand whether we can leverage some of these off-the-shelf natural language models that's really impressive out there already and simply adapt them to gene and cell functionality prediction or for more biological tasks without having to retraining a model from scratch. And to really motivate our approach, we took a step back and think about what a gene is. So if you talk to a biologist in a Bio 101 class, they'll tell you that a gene is simply a list of nucleotides that are consisting of ATGC that can encode information of, for instance, particular regulatory elements or or proteins that can be passed to its offspring. So this is indeed the approach that a lot of the DNA language models are taking, where the point is that the input will be certain nucleotide sequences, and the hope is that the high-dimensional representation it learns will be able to capture the downstream biology. Of course, you can take a bit more direct approach of single-cell data, where often from a very abstract level, single-cell RNA-seq gives you these measurements about single-cell resolution data that are represented by a cell-by-gene matrix, where the value of the matrix entries simply represent the relative abundance of the cell at a given transcript. And this is indeed the approach some of these single-cell foundation models are taking, where the point is that we can use very complex and expressive deep learning architectures to learn the co-expression level implicitly from this data. What James and I did was essentially taking a very much orthogonal approach by saying that actually we do know a lot about these genes, but not only from the implicit co-expression level, but also through the fact that there's numerous experiments being done and there's a lot of human power going into understanding and curating what a gene means. So to give you a concrete example, here is an excerpt I took from GeneCart information about this particular gene that encodes protein CD24, where we see that in the first sentence, we already know that this gene encodes a protein. And later on, we also know that what kind of pathways and gene annotation is this protein involved in. So you can very much think of this as an expert annotation that really crystal— that really extracts the more fundamental information that people have sifted through from the first or the second view of this data. Now, motivated by how rich of this information can be, our approach is also super simple. To arrive at the GenePT gene embedding, we simply took the NCBI description of the genes. So an example is shown over here. We're then just going to pass them through your favorite large language models. So in our initial paper, we tried GPT embedding, which is the AdaText embedding, as some of you may know, that powers GPT-3.5. But often you can really pass it in through various different ones. You can choose your favorite biomedical-specific embeddings. You can also just even use Llama 2. It turns out that they all achieve really good performance as long as they encode the rich information about the text. And once you pass it through the large language models, obviously you can get some sort of text embedding about the text that describes what the gene does. And then you can subsequently You can use that for all kinds of information that you care about the gene. For instance, we tried to cluster these embeddings. We use the embeddings to demonstrate that they encode gene-gene interaction network. They could also predict protein-protein interaction for protein-coding genes. We also demonstrate that on a subset of tasks that really encodes specific biology functions of a particular subset of the genes using our embedding. Also achieves state-of-the-art performance for those specially curated genes. So just to have— now that we have these gene-level embeddings, a natural question for the single-cell data is that that's often not quite enough, right? We also want to make sure that we have a good representation on the cell level. And with this picture in mind, a very natural idea now that we're given a matrix of cell by genes is to simply say that we have a good representation for each individual column because now we know that for each gene, we have this rich representation that's directly encoded by its text. It's a very natural thing to do to simply say that now to get a representation for a cell, I'm just going to take a weighted average of the gene embeddings that we have with the weight being the normalized abundance level for that transcript we detect in the single-cell data. This will, by some algebra, amount to a simple matrix multiplication. So in here, if you have this expression data and you have the corresponding gene embeddings I just showed you from the last slide, you essentially just get a cell embedding for free by multiplying these two matrices. And we are naming this approach GenePTW embeddings, where W emphasizes the fact that we obtained these cell embeddings by taking a weighted average of the gene embeddings we obtained. And of course, now you have a representation for each gene you have. You can do all sorts of stuff that you might be interested in in other canonical computational biology tasks. You can try to annotate the cell type, do batch integration, do cell clustering, do disease state prediction, so on and so forth. Of course, this is only one way to get a weighted average out of the gene embeddings we have. You can also think more about like a linguist and think about how we can directly embed single cell as language or sentences by itself. So this motivates our second approach where now, again, coming back to the single cell expression matrix, if you just zoom at one row at a time, what we're going to do is to simply say that the cell can actually be represented uniquely by a sentence where the sentence is essentially the concatenated list of gene names where the order is determined by the normalized expression level from the highest to the lowest. So in particular, we're trying to use this string representation for each cell because now you can pass it again through your favorite large language models. But also, in addition, we think that this ranked approach really encodes a lot of information because where in the sentence will a gene approach is essentially what we're trying to do. For instance, in using the canonical differential expression test kind of technique when looking at the raw matrix data as well. So now if we pass them through your large language model, I don't have to really belabor the details, you again get the embedding. And these embeddings, we're calling them GenePT-S, where S emphasizes the fact that they're encoded from this sentence representation of single-cell data. Now, with all these approaches I've explained to you before, it's only natural to understand how well they do. So the first set of results we're going to show is that through a quick sanity check, these gene embeddings actually do encode a lot of biological information. So in particular, over here, what I'm doing is that I took the gene embeddings for more than 30,000 genes that we have in NCBI Summary, and then I did a UMAP visualization of the embeddings colored by gene functionality. So over here in this plot, each dot is going to be a gene, and its corresponding color is going to be essentially what gene functionality course class they're in. And you can see that different color dots are clustered and pretty separated. So that's pretty good news that our embeddings are actually encoding some of this information. And we also did a sanity check with a very simple classification task where we took the top 15 abundant classes of these genes and just put a logistic regression on top of the gene embeddings. And trying to classify the genes, it turns out it outperforms the previous baseline that essentially uses dense co-expression data by more than 10% in terms of the accuracy. So, so far so good. And then we took this approach and went on to understand whether we can predict gene-gene interaction network over here. So in particular, this dataset we took, they transform or cast the gene-gene interaction problem as essentially a binary link prediction, where the goal is for a given pair of genes, to predict whether they have a shared common gene annotation ontology that's not extremely common among the two. And the inactive examples are just taken from the ones that don't really share GO annotations. And in particular, just by doing a very simple featurization, so in this case, what we did is we simply took the gene embeddings through our approach for two different genes and then add them up to preserve the permutation invariance about the pair and then pass it through a very simple random forest classifier., and we get state-of-the-art performance compared to previous dense embedding as well. In particular, one other thing I want to call out is we have a diagonal line of red. I'm trained as a statistician, so I'm always afraid that when I'm having high-dimensional embeddings, I'm going to overfit. So we also did a negative control where essentially what I did with the red line here is that I sampled the same dimension of Gaussian random noise for each gene just to make sure that through this approach, we're not overfitting to the fact that we're simply having more dimensional data coming out of our large language model embeddings. With this in mind, I will quickly go through the next slide. The message is also pretty clear. In addition to Gingi network that was derived from Go annotations only, we also tested the predictions on protein-protein interaction data and demonstrated that on various data we've looked at, this language model consistently performs competitively, if not better, than previously densely derived embeddings in this case. Now, I have a giant table over here, but don't worry. Before you panic about how many numbers you need to read, I'm going to tell you a set of numbers you need to pay attention to. So over here, what we see is that I have some classification tasks represented in 4 of the different columns, and the numbers reported are the cross-validated AUC through 5 different folds. What we have on the top is a GINformer, which is a fine-tuned transformer model that particularly really looks at the single-cell data and then fine-tuned on these 4 different prediction tasks. What we have on the bottom are different GenPT variants. The top 2 are going to be encoded through GPT-3.5, and the bottom 2 are going to be encoded through LLaMA-7B. And we also have either a logistic regression and random forest, but they perform, as you can see, very robustly, very similar to each other. So the point of this slide is really to tell us that through these 5 numbers, GPT-G embedding achieves really competitive results compared to some of these more fine-tuned and even very, very specifically designed models on these biologically relevant tasks. I also have a second set of numbers just as a sanity check. So over here, what I have as before is also that I did random embeddings of the same size just to make sure that on these tasks which are biologically relevant but also much smaller in scale, We're not overfitting just because we happen to have a larger dimension. So through an ablation study, by removing the entire context about what we're embedding, we see that really by using these gene information, we're able to encode a lot of the biology that people do care about as represented in these prior work.
Now, moving on from genes to cells, We also have some data, which I'm also only displaying partially, that compares the performance of GenPT cell embeddings with some of the more state-of-the-art models like SCGPT and GenFormer.
And we can see that while we don't always do the best—so this is only showing one particular dataset, you can find the full analysis in our paper online—we still do really competitively, especially the sentence embedding. And one other thing that's pretty interesting, and it's an active ongoing source of work, is trying to understand whether we can combine these two sources of information together. So if you recall in a second before, some of these larger single-cell foundation models that's out there really takes advantage of we have a ton of cells and we're learning the implicit co-expression data between the genes. But we're taking a more direct approach, which can almost be assumed as orthogonal because we're only looking at the prior work that describes what these genes do without really looking at the observed data that we have. So a way of combining them together would be really interesting direction of future work. And what we're showing here is that if we ensemble the predictions we get from these different sources, we do seem to get better performance, at least on some of the datasets that we've looked at for cell type annotation tasks. Now, going on, it would be remiss if I don't show that these models can also do batch integration and batch effect correction. So over here, we can see that I have this dataset taken from heart failure patients. On the top, I'm showing the original single-cell RNA sequencing data. The left column is coloring the data with the phenotype that we observed for the patient. And on the right column, we're showing the cell type that's being annotated from the original data. And we can see that there is strong batch effect present in the original single-cell RNA-seq because different cell types seem to be segregated from each other. Even though they really are annotated to be the same. And on the bottom, we see that by transferring this information into a sentence of strings ranked by their expression value and then embedded them through any large language models, being GPT-3.5 in particular here, we're able to remove batch effects, but also retain a lot of the biology here because we do see that the cell types in the bottom are much more clustered compared to the top. Now, in conclusion, what have I told you today? I've told you that the large language model embedding in our study provides a simple but very effective way to predict the things we might care about cell and gene data, especially in the single-cell landscape. There's ongoing extensions., as I mentioned before, to make this approach more context-aware, but also trying to understand how it could be integrated into these large-scale atlas-like data to enable fast and efficient retrieval tasks. And in conclusion, I also want to say that even though we've only tried this for single-cell data so far, we do think that rich textual descriptions in general can provide a very useful and very high signal modality. And we're really excited to see whether this idea can be replicated in other computational biology fields as well. With that, thank you so much for attention, and I'm happy to take any questions you might have.
Thank you so much. This was a great talk. So you showed that the model does a great job in removing the batch effects, but you're not using any explicit encoding of batches or doing anything. It just does that as you train it, right? Right. That's correct. We didn't give it any batch information. So why do you think that is? So let's say you have the same gene and let's say these embeddings are multiplied or somehow normalized by the count that you have, there's no information of batch. Wouldn't you expect to still see batch information there? What happens there that removes these batch effects?
Yeah, it's a really good question. It's a question that we've been asking ourselves through and throughout. Again, I think our current hypothesis is that a lot of the batch effects can be removed by simply considering the rank of the genes rather than the absolute abundance, even after normalization of a gene, and that's like why we think it's able to detect that. So indeed, if you actually transform your single-cell expression data by total rank normalization, so instead of looking at the values, you compute the rank directly, you get very similar performance to what we have, but we still have an edge in terms of batch effect correction. And I think that's partly due to the fact that you do pay different attention to the first and top bottom. So if you think about a rank perturbation, If you perturb the rank of the genes at the very bottom and the very top of the gene expressions, they might have the same effect, but that's likely not the case in a large language model. But that's a great question.
Thank you.
We have time for one more question.
Thanks. Great talk. I'm curious about the issue of data leakage when you're evaluating your model. Like, for example, you showed one of the motivating examples was predicting gene-gene interactions based on which the edges are based on GO terms, but GO terms were part of those NCBI text summaries that went into the model. So you basically told it the text label when you generated the embeddings.
Yeah, that's a great question. Yes, just to clarify, some of the protein functionality classes, we do expect them to do very well. So I think of them more as a sanity check to make sure that we get useful information. Rather than trying to prove that this is something that biologists are going to use because, you know, like we already have these annotations, right? Like in practice, really, you're just going to look it up at the NCBI website rather than using a prediction model. But the gene-gene interaction is a great question as well. We started with this dataset trying to understand whether any information is encoded because in the NCBI summary, it's unclear whether all the GO annotations are consistently provided. So I think it gives us some comfort that we're able to predict something that's not always there. But this is indeed the motivating factor that drives us to test it on more, I guess, protein-protein interaction dataset that doesn't simply use the GO annotations. And I'm happy to take any questions offline.
Just feel free to find me afterwards.
Let's thank Yiqun again. Okay, so next up we have Rebecca Boyarsky from MIT who will tell us more about single-cell RNA-seq foundation models. Rebecca is joining us virtually, so we're going to play her a recording of her talk, and then she'll take a live— she'll do a live Q&A session.
