Scientist Stories: Jeff Ruffolo & Stephen Nayfach, Designing Genome Editors

Yeah, so you didn't see the previous slides, but yeah, so Cas9 can be utilized in several different genome editing platforms. Cas9 is a really complex protein with multiple domains that all require functional coordination. So Cas9 has two nuclease domains, the HNH and RuvC, which are both required in nicking either the target or non-target strand of DNA. Cas9 also has several recognition domains that are involved in sequence-specific binding of the guide RNA and detection of RNA-DNA mismatches, as well as a PAM-interacting domain, which is required for the initial step of DNA sequence recognition and double-strand DNA melting. And the function of Cas9 really requires coordination of all these different functions and all these different domains. In order to be utilized effectively. And protein language models offer the possibility of designing these Cas9 proteins from scratch.

[Speaker:JOSHUA_ELKINGTON] Yeah, so I'll go briefly into some background on protein language models. So with this class of models, the objective is really to learn a likelihood function over sequences of amino acids.. So for example, for this short alpha helix, you might have a sequence S, and we can call the residues S1 through Sn, and then we want to calculate P(S). A really simple way to do this might be just to assume that every amino acid is independent. This is a unigram model, and to calculate a likelihood under this model, we can simply tabulate the frequencies of each amino acid in our database, then multiply those together to calculate the likelihood of a sequence. Of course, this ignores what we know about the organization of proteins. We know that residues close together are probably interacting in some way, forming secondary structures. So we might make this model a bit more sophisticated and condition on the prior amino acid. So in this case, if we wanted the probability of S2 given S1, we would look in our database and count up all the times we observe that sequence of 2 amino acids. This might get us closer to picking up some notion of secondary structure, But of course, functional sites of proteins can be spread over many residues and aren't just limited to 2 adjacent residues. So we can extend this model to something called an n-gram, where we define some hyperparameter n and tabulate all possible configurations of amino acids with that context window. Of course, this starts to run into problems when you're trying to model protein functional arrangements that span many residues. For example, for the zinc finger motif shown here, the metal-coordinating residues are separated by 15 positions. So the specific configuration we're seeing in this natural protein is only one of 20 to the 15 possibilities. This means to build an n-gram model that would capture this specific arrangement, we would need to tabulate all possible stretches of 15 amino acids. And that's just for this one protein. Of course, there's other proteins where your functional motif might be separated over hundreds or more residues. So instead, we turn to a class of models called language models, as we mentioned a few slides ago, where here the goal is to predict the likelihood of each amino acid given all prior residues. So we're no longer restricted to a context window of N or some pre-specified number. We can really teach the model to look at everything and then predict what comes next. After training this model, we can then compute an exact likelihood by decomposing the probability of a sequence into a series of conditional probabilities as shown towards the bottom of the slide. One example of a protein language model is ProGen2, which was scaled up to about 6.5 billion parameters and trained on really diverse metagenomic and genomic sequence data. Importantly, we can also generate samples from protein language models. Simply by predicting the next residue in a sequence one amino acid at a time. And then we can keep doing that until we have a complete protein sequence. When we generate a bunch of these sequences and predict their structures with AlphaFold2 or ESMFold, we typically find that they're confidently predicted to fold into structures that resemble those found in the protein databank. So here's three examples from the ProGen2 manuscript where you can see the predicted structures for ProGen-generated sequences mostly in blue, and then their closest natural counterpart from the PDB in gray. And in all cases, we see similar folds, sometimes with very high TM scores, but the sequence identity is typically much lower. This has been found in several protein language modeling studies and really provides a useful tool when we want to design something novel but functional like a gene editor. So for the OpenCRISPR project, we'll start broadly by looking at language modeling applied to CRISPR-Cas systems overall. And then we'll sort of focus in on Type II systems, which contain Cas9. And then finally, we'll design some gene editors in a constrained manner that we test in the lab and ultimately release one of. [Speaker:JOSHUA_ELKINGTON] Great.

And so towards this effort, we really needed to curate a really large and diverse set of CRISPR-Cas operons from nature in order to model these. And so CRISPR-Cas systems naturally occur in bacteria as an adaptive immune system that protects the bacteria against invading mobile elements like viruses or plasmids. And just to briefly go through the system, upon a virus entering the cell, 2 proteins called Cas1 and Cas2 will extract a snippet of DNA from the virus that's called a protospacer. That protospacer will get integrated then into the bacterial chromosome at a locus called a CRISPR array. And then upon subsequent infections in the future, this bacteria now has a memory of this virus. And can express these CRISPR arrays that get processed into CRISPR RNAs that can complex with the Cas9 and cleave the virus in a sequence-specific manner. And these CRISPR-Cas systems are very diverse, but they can broadly be categorized into Class 1 and Class 2 systems based on whether the effector sequence is a single protein or a protein complex. So in the case of Cas9, or I should say in the case of class II systems, the protein that we're focusing on, or one of the proteins we're focusing on in this project is Cas9. So Cas9 is a single protein that's able to on its own recognize and cleave a target DNA sequence. Now within class I and class II systems, there's also a broad diversity of types, types I through VI, And then within each type, there's a broad diversity of subtypes. So this slide is mostly just to say that there's a— these CRISPR systems have evolved throughout evolution into very broadly diverse proteins and protein complexes. So towards curating a comprehensive database of these CRISPR-Cas systems, we turn to genome and metagenome mining. So we started with 4 publicly available repositories of assembled DNA sequences. That includes JGI's IMG database, NCBI ENA, and then a very large database of assembled metagenomes from the human microbiome. Altogether, these, this collection of data comprised over 26 terabases of nucleotide sequence. But to really efficiently mine this vast corpus of data, we implemented a few heuristics. So the first was to simply throw away sequences that were very short. The second was to only focus on sequences that contain a CRISPR array. And so these arrays consist of a repeating unit called a direct repeat that are interspersed with these unique sequences that are derived from mobile elements. And then from these sequences that are longer than 5 kb and contain a CRISPR array, we extract a flanking region plus or minus 20 kb, and then carefully annotate the protein coding regions and assign these to their functions. So in this example here, we have a CRISPR operon from a Type II system that contains Cas1 and Cas2, which are used to acquire spacers. A CRISPR array, a Cas9 protein, and something called a tracer RNA, which the Cas9 uses in order to guide it to the target site. We also predict something called the PAM for Cas9 specifically, which, and the PAM is an inherent DNA motif that's recognized by the protein sequence. And so for Cas9 specifically, how much data do we actually get? So altogether, there were about nearly 240,000 of these CRISPR-Cas operons from type 2 systems that contain Cas9. Clustering these 240,000 sequences, we revealed about 65,000 that were unique. So how does this stack up against what was already existing? In the public or in the, from the literature? Well, a study from 2022 published a database of Cas9s that amounted to about 27,000 sequences. So our database is a little more than 2.5 times the size of that. And that dataset actually isn't publicly available. The next largest dataset that is publicly available is a little over 9,000 Cas9 sequences from this study, Gasciunas et al. in 2020. So our efforts to mine, to do metagenome mining yielded a really massive increase in the number of sequences that we could use for modeling. Building a phylogenetic tree of these Cas9s, you can see they're extremely diverse and our resource is very well annotated. So the outer rings of this tree, we can see that we have both tracer RNA and PAM predictions for most of the sequences in the tree. There's a large diversity of protein sizes, and actually also quite a bit of novelty with respect to patent databases. And so now I'm going to hand it over to Jeff, who will discuss how we utilize this dataset to model CRISPR-Cas proteins generally.

Yeah, so we took the approach of taking the ProGen2 find ProGen2 model and fine-tuning across all of the CRISPR-Cas families to start. And our goal here was really to see how much we could expand the diversity compared to what we've observed from nature. So in the bar plot on the top of the figure, we have gray bars, which reflect the number of distinct 70% identity clusters from the CRISPR-Cas Atlas. So this is reducing just over 5 million CRISPR-Cas proteins down to their clusters, so we can measure the number of effectively unique sequences. And then going along the x-axis are the different CRISPR/Cas families, specifically the 45 with the most sequences that we could find. And you can see the green bars where we've taken this dataset and fine-tuned ProGen2 to generate these types of proteins, produce sequences that significantly expand the diversity over what we found from nature. Some proteins like Cas8 and Cas6 had a lot of natural examples. We can continue to expand those if we wanted more. And there's other examples like Cas12a or Cas13 where we had very few natural examples. You can barely even see the gray bars. And in these cases as well, we can also generate significantly more samples that might be functional proteins. Of course, we can only generate a finite number at any given time, but we can also sort of project if we were to keep generating sequences from our models or collect more from nature? Would we have saturated the possible sequence space, or could we keep finding new sequences? So these plots show for 4 families a sort of in silico experiment. We've tested that idea. So going along the x-axis is the number of sequences you might have sampled from nature or from a generative model. And you can see on the y, the number of 70% identity clusters that can be formed from just those sequences. Also note the log scale here. But you can see that the generated sequences in green are significantly outpacing the rate of novel cluster formation compared to naturals. And they don't appear to be saturating either, meaning if we sampled for longer, we could get even more proteins from these families. The next thing we did to convince ourselves that these looked functionally complete and viable is predict 2,000 structures using AlphaFold2. We took sequences just uniformly from this set of generated sequences from the prior slides, passed them through AlphaFold2 with MSA inputs. Shown here is the identity to the CRISPR-Cas atlas on the x-axis. So this is a measure of novelty of the sequence relative to the training data. And on the y-axis is AlphaFold's confidence or the PLDDT, where higher means AlphaFold is more confident in the structure that it's predicting. And you can see that for most sequences with greater than 40 to 50% identity to the CRISPR-Cas atlas, we get really confident predictions from AlphaFold. And only when we go below around 40% identity to the training data do we start to see the predictions degrade. So on the right here are a few examples from this plot from 4 different families. You can see something similar to when I first showed ProGen2, where we have these predicted structures that closely align to examples from the Protein Data Bank, but they do so with remarkably low sequence identity to those structures, meaning even for CRISPR-Cas families, we can generate new proteins that fold up the same, perhaps carry out the same function, but do so with pretty different sequences. Next, we'll go into how we narrowed our scope to specifically Type IIA effectors. So we start by generating diverse Cas9-like proteins, and then as Steven mentioned, we need this piece called the tracer RNA to form the complete editing unit. So we built another model to specifically generate those based on the protein of interest. In these phylogenetic trees, we're showing the result of applying a similar fine-tuning approach as we did for all CRISPR-Cas proteins to just Cas9s. On the left is the natural diversity, So you can see this branching path taken by evolution. And then when we generate about a million Cas9-like proteins and add them to this tree, you can see it's then dominated by this synthetic diversity, in fact, over 10-fold more diversity than we can find with just natural proteins. And if you look really closely at the tree on the left, you can see lots of little nicks in the tree. These are cases where generated proteins have created new branches that weren't explored by evolution. So not only can we sort of add more resolution to paths that evolution did explore, we can create entirely new paths and lineages that might have highly functional genome editors for specific applications. To convince ourselves that these proteins were realistic, not just novel, we did a few analyses. The first was to look at how different they are than the dataset. We're really interested in these low identity proteins, assuming that they have all the pieces needed to function. You can see the distribution of percent identities from the CRISPR-Cas atlas here, where we have this peak around 40 to 50% identity, as well as some sequences that are closer to the training data. When we take the generated proteins and the naturals and co-cluster them, we can also look at how well the length of those proteins correspond. So this can be maybe a coarse sign that the proteins look realistic. In fact, when we plot these against each other, you can see we have a really strong Pearson correlation of 0.97, meaning that when you co-cluster the generated natural proteins, they're generally the same length as each other. And then finally, we can turn again to structure prediction and predict structures for these generated Cas proteins and actually check if they have all the domains that a Cas9-like protein needs. So we did this, and we looked for HNH domains, RoC domains, PAM-interacting domains, and recognition lobes. And we found that the majority of generated proteins had all of these separate domains, which gives us confidence that these might be able to function as gene editors. One interesting protein that we found was this generated sequence shown here with the domains colored. And labeled. This protein was only 39.2% identity to the closest natural counterpart in the CRISPR-Cas Atlas, but was predicted by AlphaFold to fold into a structure that looked complete. So you can see all the domains of this Cas9 protein colored and labeled. And when we aligned this to the PDB, we found that it most closely resembled NME1 Cas9 with a TM score of 0.71, meaning there generally the same structure and only a sequence identity of 25.8%. Okay, so we can generate Cas9 proteins that look realistic. We can repeat this process to generate large libraries of these Cas9-like proteins that we can look within to find functional proteins that we might want to test in the lab. But of course, as we deviate further from nature, we run into problems with trying to test these because we need a guide RNA. So to address this problem, we trained a separate model where here the goal is to take a protein sequence as input and then generate the CRISPR RNA and tracer RNA components of the guide conditioned on that sequence. In practice, we do this just by concatenating the CRISPR and tracer RNAs from the CRISPR-Cas atlas and then generating them in a random order. Once we have this model, we can then sort of assemble the complete inference pipeline shown on the bottom, where we can pull diverse Cas9-like proteins out of the protein language model and then pass those to the second model to generate complete Type IIA effectors. Great.

Yeah, so to convince ourselves that these generated guide RNAs are realistic, we conducted several experiments. So the first was to conditionally generate guide RNAs using a set of 10 natural Cas9s that had been previously used in genome editing applications. When we clustered together the generated and natural guide RNAs from these 10 different Cas9 proteins, we find that they cluster together as expected. So for example, for SpCas9, the generated guide RNAs are at the sequence level quite similar to the natural guide RNA that's found with spCas9. This gives us confidence. This is the first thing that gave us confidence in the model. The second experiment we did is in which we reanalyzed a dataset published in 2014 by Fanfara et al. In this study, the authors measured the nuclease activity of Cas9s paired with their cognate guide RNAs, as well as the activity of Cas9s paired with guide RNAs from different orthologs. And what this study found was that the guide RNA is exchangeable between Cas9 orthologs that are close in evolution, but the exchange in the guide RNA is incompatible between Cas9s that are more divergent. And so what we wanted to know was, can the model actually predict whether or not these guide RNAs are compatible with the right Cas9s? And so for each Cas9-guide RNA pair, we use the model to score those combinations, which is what's shown on the right. So on the right, each row is a Cas9. Each column is a guide RNA. And then in blue, you can see the guide RNA model score, which can be thought of as a proxy for the compatibility of that guide RNA with that Cas9. And then overlaid on this matrix, you can see these pluses, either 3, 2 or 1+, which is the experimental measure of nuclease activity from the publication. And what you can see is, with just a few exceptions, the model is accurately predicting whether or not a guide RNA and a Cas9 combination will yield a functional Type I effector.

Great. And for the— The last focus of this talk, I'll now go into how we took these models that can generate diverse Cas9-like proteins, that can complete them with the cognate guides, and applied those to specifically develop a gene editor that could be used as perhaps a drop-in replacement for spCas9 in existing technologies. This involved constrained generation of Cas9-like proteins. Cas9 that could target the same PAM as spCas9. So we wanted it to bind the NGG PAM. Of course, we also wanted it to work with spCas9's guide RNA so that it could be used on the same targets that you might already have a guide RNA for. And then finally, we wanted it to have high activity so that it'd be a protein you would want to use. To generate these proteins, and satisfy these criteria, we came up with constrained generation strategy that I'll go through now. So first, we decided to generate the full-length Cas9-like proteins in two parts. We either initialize generation with the PID domain, this is the PAM-interacting domain of the protein, or all of the domains on the N-terminus of the protein, so just excluding the PID. By doing this, we thought we could enrich for sequences that are likely to maintain these compatibilities that I mentioned. So by initializing with the PID of spCas9, we could enrich for generated sequences that would likely be compatible with the guide RNA. And similarly, by initializing with the N-terminal domains, we hope to enrich for sequences that would be compatible with the NGG PAM. Of course, we're not asking the model specifically to do these things. We're just asking it to predict the next residues in the sequence. So we built a couple predictor models to help us filter out the sequences and really select for those that we thought would work in this editing system. One of those was simply a binary predictor that tries to tell us if a given full-length protein is going to be compatible with S.p. cas9's guide RNA or tracer RNA. And then another one is doing the same thing but for NGG PAMs. So we can pass these protein, get a prediction of whether they'll be— the protein will be compatible with S.p. cas9's components. And if so, we can let that sequence progress in the pipeline. Of course, we can repeat this process many times to produce many generated sequences, but ultimately we needed to select some to test in the lab. So to do this, we scored all of the generated sequences that made it so far in the pipeline and just selected by the log likelihood under both pre-trained and fine-tuned language models. Once we have these proteins, of which we selected about 200 The gene editing experiment looks something like this. So we have the protein and the guide RNA. We put those into separate plasmids and transfect those into human cells. We can then sequence the genome and look and see if editing has happened. So in this case, we're looking for on-target editing. So these are cases where the genomic site perfectly matches with the guide RNA spacer. And we expect to see editing at that site. When we sequenced the genome. These were the results of our first screen at 3 different sites in the genome. So for each of these plots, on the y-axis, we have the indel formation rate. So if this value is higher, that means we're seeing more gene editing. This specifically is normalized to spCas9. So values above 1 are corresponding to gene editors that are producing more indels than this exemplar natural system. The proteins are ordered on the x-axis by the indel rate. So you can see at the first target site, HEC3, we had 16 proteins that exceeded the activity of spCas9 and 120 total that showed some activity in our assay. For the other two target sites, we see something similar, though perhaps with more proteins that are exceeding spCas9, fewer that are active at these sites.

Cool. So the next experiment that we wanted to do was to not only look at on-target sites, but also look at off-targets. And so one type of off-target site that we looked at were sites in the human genome where the sequence is very similar but contains mismatches to the guide RNA. That was used to guide the Cas9. I see. And so yeah, I guess taking a step back, looking broadly at the sequences that we tested in the context of both naturals and Cas9, that's shown here in this phylogenetic tree. And so spCas9, you can see, is shown in the top of the tree and is surrounded by a group of other naturals shown in blue. And then extending outwards clockwise around this tree, you can see just this massive— these large number of new lineages that are only represented by these generated Cas9-like proteins. In certain places, you can see there are naturals. But overall, a lot of the new diversity in this tree is just represented by the generated proteins. In the outer rings of the tree, you can see some interesting and important information. The first ring is whether or not that Cas9 sequence was active or not in the previous assay that we just showed. And so you can see there's actually large regions of the tree that have many active sequences. Also, many of these sequences were very distinct from any sequence in a patent database, and many of the sequences were fully generated at both the N-terminus and PID domains. Overall, there were many, many of the generated sequences that were active. Even exceeded 400 mutations from spCas9 and hundreds of mutations from any natural sequence in the CRISPR-Cas atlas. And so these active proteins are very distinct from both the canonical gene editor spCas9 as well as anything that has been observed in nature. Yeah, so going back to the— the experimental overview that I, that I shared previously. This is a figure where in this figure, each bar is a different protein. The green bars are each of 48 proteins that showed some level of activity in the first experiment that we shared. Yes, the green bars are these generated proteins that we knew were going to be active. And then the blue bar is spCas9. The top of the figure is a measure of the on-target activity rate. So this is the activity at these on-target sites. And then the bottom part of this figure is the activity at off-target sites. And both the plot in the bottom and the top are both normalized to spCas9's activity rate. And what you can see on the top is that many of these sequences were indeed active, with a handful even exceeding the activity level of spCas9. But what was even more striking on the bottom is that the off-target rates of these generated proteins was far lower than that of spCas9. Our most active protein, which we are denoting OpenCRISPR1, is shown on the right-hand side of this plot. It was our most active generated protein, while at the same time having an off-target rate that was 95% less than that of SpCas9. So going back to the phylogenetic tree that we showed earlier, you can see OpenCRISPR1 is also one of the most distant proteins from SpCas9. So it's all the way on the other side of this phylogenetic tree, having over 400 mutations from SpCas9, and almost 200 mutations from anything natural. Overall, the structure of OpenCRISPR1 was fairly similar to that of spCas9. However, there were several notable differences that included an insertion in the Rec1 loop. This insertion is predicted to make sequence-specific contacts with both the target DNA and guide RNA. And then there was another small insertion in the HNH domain. So together, we speculate that these novel domain insertions could help to either stabilize the protein in complex with the target DNA or could be involved in increasing its activity based on this loop in the HNH domain. The other type of off-targets we looked at were sites in the human genome where the spacer sequence was matching, but there was a difference in the PAM. So as I said, as we mentioned previously, the PAM is the sequence just adjacent to the target site. And this motif is actually recognized by the PAM-interacting domain in the Cas9 protein. And so what we wanted to do is we wanted to measure activity levels at sites in the human genome where the PAM was NGG, which is the canonical PAM for spCas9, and also measure the activity level at sites where the PAM is slightly different than NGG, where there is a mutation in one of those PAM positions. And so overall, so this is showing the activity level of OpenCRISPR1 compared to spCas9 at 49 sites in the human genome that have the canonical NGG PAM. And so you can see that overall, the activity level at these 49 sites was comparable to that of spCas9. There was no statistical difference in the activity levels across these sites between proteins. However, when we look at sites in the human genome that have a mismatch in the PAM, so we're calling these non-canonical PAM sites, we tested the activity at 43 of these sites and saw a dramatic decrease in activity at these sites, at these potential off-target sites. Again, that's just shown in a different way in this scatter plot where we have open CRISPR-1s and del rate on the x-axis and spCas9's indel rate on the y-axis. The green points are these canonical— is representing the activity level at the canonical PAMs. And then the gray points are the activity level at these non-canonical PAMs. And what we can see, again, is that for several of these PAMs, for several of these non-canonical PAMs, for which we wouldn't expect editing, and really where we don't want there to be high editing, S.p. Cas9 has, in some cases, orders of magnitude higher editing. And so again, this analysis underscores the result that OpenCRISPR1 seems to have comparable activity to S.p. Cas9, but really markedly improved off-targets. And another experiment we did we generated guide RNAs for OpenCRISPR1, as well as 4 other generated proteins that showed some activity level. We tested these 5 proteins at a single genomic site, HEK3, using these synthetic guides, as well as spCas9's guide RNA. And in many cases, what you see is that our design guides that were generated using our machine learning model, these proteins show good compatibility with these synthetic guides, equaling or in some cases even exceeding that of spCas9. Sort of interestingly, for OpenCRISPR1, spCas9's guide RNA actually did display the best editing. And this underscores that OpenCRISPR1 works, can be used essentially as a drop-in replacement for existing genome editing systems that use spCas9's guide RNA. Another interesting result we found was that, yeah, so maybe I'll take a step back and walk you through this figure. So in the bottom part of this figure, we have spCas9's guide RNA. That's shown on the top and a multiple sequence alignment of generated guides on the bottom. On the top of the plot with asterisks, we've indicated positions in the generated guide RNAs that often differ at that position with respect to spCas9. And then on the top of the figure, we have the predicted secondary structure of spCas9's guide RNA. And then annotated in that structure, we have these positions in the generated guides where we see a difference with spCas9's guide. And what you can see is that many of the differences, many of the mutations that the model is introducing are at locations in the guide RNA that can essentially be tolerated functionally. So these are at loops or linkers. So the model has essentially learned some of the biophysics of the guide RNA and knows which regions of the sequence that it can introduce sequence variation into.

Great. So, so far, we've shown OpenCRISPR1 as sort of a standalone effector protein. But as we mentioned at the very beginning, Cas9 proteins can also be used to build more sophisticated gene editing tools. This includes things like base editors, prime editors, and activators and repressors of genes. So what we did next was we took a similar strategy and wanted to see if we could design base editing systems that were fully produced by language models. So similar to before with the CRISPR-Cas atlas, we went to large protein sequence databases and looked for TadA-like proteins because these have been used previously with Cas9 proteins to enable single nucleotide base editing, we fine-tuned another model on these deaminases or TADAs that we could then use to generate diverse TADA-like proteins. Interestingly, these are dimer proteins, so we actually generate one copy with the language model, but we need them to assemble into a dimer for base editing to happen. To build the base editing construct, we simply take these generated TAD-A-like proteins and fuse them with an N-terminal linker to a Cas9. So then you get something like the structure shown on the right here, where you have Cas9 as the gray protein that'll target your genomic site via its guide RNA. And then the base editor, by being tethered to the Cas9, is then positioned to edit an A to a G, for instance, for the base editors we built. So we ran an experiment where we took an existing base editor that came from directed evolution. This is ABE 8.20, as well as two language model designed base editors that we called PFDM 1 and 2, respectively. We then combined those with either spCas9, which is what ABE 8.20 was developed to pair with, and OpenCRISPR1. So we have these different combinations of deaminase domains and nucleases, that we can then screen against target sites that have A's that we might want to edit to G's. So we did this at 3 sites. In the bars on the right side of the slide, we have the different nuclease components indicated by the color of the bar. So gray means a base editor was using SB Cas9, and green means it was using Opacr1. And then along the x-axis for each set of bars, we have the deaminase domain that was paired with that nuclease. And in general, the biggest takeaway is that OpenCRISPR-1 works well in this context. So not only is it a good sort of general-purpose genome editor for creating indels, we can also start to build on this similar to how many groups and companies have done for spCas9 to build other editing systems. We have a bit more about these base editors in the supplement of the paper. They also have interesting properties like reduced bystanders. So if you have an A nucleotide that you want to edit and it's surrounded by other As, we find that the language model-designed base editors typically have less of those bystander edits compared to something like ABE 8.20. And then the final thing we looked at was whether OpenCRISPR-1 is likely to elicit immune responses in healthy donors. This has been found for SPKES9. For this figure, this is from a paper from Ferdosi et al., where they took a panel of 12 donors and found these existing T cell immunity epitopes against SPKES9. They picked out 4 that appeared across most donors, and those are labeled at the top of this with alpha through delta. So then we did a really quick in silico experiment where we simply checked if these same epitopes from SB Cas9 exist in OpenCRISPR1. What you can see here are a few different segments of OpenCRISPR1 and SB Cas9 aligned. And then the red boxes indicate those T cell epitopes. So what we want to see here, if the epitopes don't exist in OpenCRISPR1, is just mutations between the two sequences. And indeed, you can see the light blue in each of the red boxes indicates that we do see mutations here. At each of these sites, meaning that at least for these T-cell epitopes that many healthy donors had, OpenCRISPR1 shouldn't be activating the immune system. This is just the beginning of our analysis of OpenCRISPR1's immunogenicity. There's much more thorough tests that we can actually run in the wet lab to see if OpenCRISPR1 is likely to elicit immune responses in healthy patients. To wrap up, just to summarize what we've talked about, so we started by showing that language models can learn the blueprint of CRISPR CRISPR-Cas proteins. And we can use this blueprint to generate lots of novel CRISPR-Cas proteins across families and significantly expand over the diversity produced by nature. When we look specifically at Type II effector systems, we found that language models can really shine here. We can generate many diverse Cas9 proteins that create entirely new lineages not explored by evolution. And they can also be used in conjunction with guide RNA design models to complete the editing system, including proteins that we tested in human cells. And finally, OpenCRISPR-1 is really the fruit of this project. It's highly active in human cells. It works as a base editor. And we're really happy to open this up for the community to experiment with for ethical applications. For next steps, we're really excited to sort of pressure test OpenCRISPR1 both within ProFluent and hopefully across the community in more challenging editing environments. This is really just preliminary work in fairly friendly human cells. There's a lot more work to be done in more challenging therapeutically relevant systems. We're also really excited to kind of push the boundaries and deviate further from nature. For OpenCRISPR1, we showed that it's over 400 mutations away from spCas9. And about 200 away from the nearest natural protein in our training dataset. We think, given some of the results we found for the novel proteins that were functionally complete, we think we can push these numbers further. And then finally, we're really excited to use OpenCRISPR-1 as a platform to build customized and fit-for-purpose editing systems. So this involves things like optimizing editing in these more challenging environments, changing the protein so it's a better fit for certain applications like base editing or targeting different PAMs and the like. And finally, we just want to thank everybody at Profluent and our collaborators who contributed to this work. We're still a fairly small company, so we can put everybody's name on our acknowledgments slide, but particularly want to thank the folks whose names are italicized who contributed or are on the paper, as well as our co-first authors who are highlighted in green. The preprint is available on bioRxiv if you want to read more. We've also released OpenCRISPR-1 on GitHub, so you can check out the sequence there if you're interested. Thank you.

Great, thank you so much for that great talk. So feel free to ask some questions in the chat or unmute yourself if you prefer that as well. So the first question in the chat we have from Salem, who asks, what was the deaminase engineered to be active with DNA substrate over RNA substrate?

Yeah, so the way we did that was we took some of the learnings from the original ABE paper from BEAM and David Liu's lab. So there they focused on this TadA protein family that is actually naturally functional at deaminating, I believe, tRNAs. But they found this set of 1 to 2 mutations that confer specificity to single-strand DNA. So then when you tether out that to Cas9, which exposes this single-strand DNA, if there's an adenine in that exposed window, then you can get base editing. So we did the same trick with the generated proteins.

Thanks. Next question is from Jesse. What natural Cas9 is OpenCRISPR1 closest to? How does its structure, activity, and specificity compared to that Cas9? Was its guide RNA better with OpenCRISPR1 than spCas9s?

Yes, there's a few questions in there. Yeah, I forget the exact name of the natural. A lot of the naturals that we found in other parts of the tree were derived from metagenomic sequences. So a lot of those sequences are not from like an isolated organism that has a species name. But I think, yeah, I think that's a good question. And something for follow-up experiments is to look at, yeah, possibly testing OpenCRISPR1, not just with spCas9's guide RNA, but maybe with the guide RNAs from other naturals. The fact that the generated guide RNAs were a little bit less effective than spCas9's guide RNA suggests that maybe there's not a lot of room for improvement in terms of the guide RNAs. But I think that's a good question and something to follow up on in the future. All right, thanks.

Next question from Dylan. Did you look at off-target rates across your generated guide RNAs?

Yes, we did, and found that for OpenCRISPR1 at a set of 15 off-target sites, OpenCRISPR1 had a 95% reduction in off-target rates.

Next question from Saber, what was the base LLM that you fine-tuned?

Yeah, so for all of the generated sequences, we use the ProGen2 base models. This is about a 700 million parameter language model trained on just protein sequences. For guide RNA design, we also needed an encoder model for the protein so that we can condition for a guide RNA generation. And then for this, we used an ESM2 model. I believe it was the 8 million parameter model. And we found that bigger models didn't really help as much there.

Okay, next question from Jonathan. Maybe you have to look at the chat to see the full question. But the first part is, how important do you think the filtering process for selecting the Cas9 proteins is? And which, which part of that do you think is the most important for having a functional, in real life protein? And the second part is for fine-tuning, it seems like for generating Cas9-like proteins, fine-tuning on the smaller set of the Cas9-like sequences, what sort of range of sequences do you think might be necessary for successfully fine-tuning ProGen on a protein of interest? And related to the fine-tuning, the original ProGen2 paper mentions fine-tuning at most 2 epochs to avoid overfitting. But the current publication goes through 5 epochs. Do you have recommendations related to this?

You want to go to this to answer the first one? Sure. Yeah, so for the first question, with respect to what parts of the filtering process were most useful, we did this retrospective analysis to sort of see of all the scores we considered and the different novelty metrics. Which ones were the most indicative of whether a generated protein would show activity at this HEC3 target site. And in fact, what we found was it's really just the language models that give you strong predictive power. So these different panels are sorted by how well a given score could classify a protein as functional for true or nonfunctional for false at this site.. And you can see things like the edit distance from SBKS9 tend to be the least predictive. So those are the ones in the bottom row, as well as some of the second row. Some of those filters we used, like the NGG PAM predictor and the tracrRNA compatibility predictor, provide a bit more discriminative power with ROC AUCs of 0.26 for PAM prediction. And 0.66 for tracer prediction. But again, those are kind of dwarfed by the language model likelihoods, which have AUCs of 0.7 and above 0.8. And then the second question was with regards to fine-tuning the models. I'm just rereading that really quickly. Yeah, so generally fine-tuning is more of an art, we find. So the question mentioned the 2 epics used for the ProGen2 paper. For those protein families, I don't remember the exact number of sequences we had there, but for the Cas9 proteins, we found we could generate longer. It is important to hold out data so you can see that you're not overfitting if you don't have very many sequences. But in our case, we found that we could keep training for multiple epochs without seeing signs of overfitting.

Great, thanks. The next question is from Zoe. Any ideas on why the design proteins have lower off-target rates, since I assume it is not trained specifically to have a lower off-target rate?

It's a good question. That is a good question that I don't have an excellent answer for. Yeah, yeah, I'm sorry, I don't, I don't have an excellent answer. But it could be that the protein has a little bit lower affinity. It seems, seems that the protein has a bit less affinity. For sites where there is a mismatch either in the guide sequence or the PAM. So in those cases, spCas9 seems to be a little bit more robust in the sense that if there's not a perfect match to the genome or if there's a mismatch in the PAM, it doesn't care quite as much. And so, yeah, seems like spCas9 is a bit more robust, but more robust in these cases where we actually don't want editing. So in this case, it seems like OpenCRISPR-1 being a little bit less active in some cases is actually to our advantage.

Great, thanks. Next question from Cannon. For fine-tuning, did you further curate the sequences, delineate domains, et cetera?

No, we did not.

Next is from Mohammed. Is the CRISPR-Cas atlas database open to the public?

At this moment, no, it is not.

Okay, next from Zachary. Do you have any ideas why RUV-C was the least conserved domain in your discovered effectors?

There are some Yeah, maybe it's not, maybe it's not in our slide deck. Maybe it was just from the bioRxiv. Yeah, there are some, there's one paper that did this domain deletion assay, where they deleted different parts of the Cas9 and assessed for functionality in terms of DNA binding, and found that in terms of binding DNA, Cas9 can actually tolerate almost a complete deletion in the Rec2 domain. And there are some natural orthologs that are missing that domain as well. So it seems like based on evolution, that's one of the more flexible, non-conserved regions of the protein. Great.

Next question is, how did you generate deaminases that have extreme specificity and have zero editing at bystander base right next to it?

Yeah, this is a great question. We haven't fully worked out why these deaminases have such narrow editing windows. There are some analogs if you look at the evolution of the ABE editors from David Lu's lab, where some of them had narrow editing windows, and then those opened up as they pushed for higher activity. It's possible we've found a way, or the language model has found a way to stay in this narrow editing window regime. While maximizing activity. We're definitely interested in learning more about what kind of determines those bystander edits, though.

All right, next question is, you showed that mutations were depleted in the PAM-interacting domain in your generated Cas. This makes sense, and given the role of the PAM-interacting domain in self/non-self recognition in bacteria, could you adjust your model to be more lax within that domain to generate highly specific but with less strict PAM requirements?

Yeah, that's a good question. I think maybe a related point, I'm not sure if it's exactly the question, is around efforts to make PAM-less Cas9s. So this is, these would be Cas9s that are fully programmed by the guide RNA. With these proteins, you typically have a sort of specificity trade-off that you're, you're taking on because the PAM would restrict where the protein could edit. If you remove that, you're then opening up to more potential editing sites. It's possible that because OpenCRISPR1 had higher specificity with spacer matches, maybe that trade-off becomes a bit more favorable. If you were to remove the PAM and create something like a SPRY open CRISPR. But we haven't tried that yet. Would love for somebody to try porting over those mutations and see what happens.

Next question is, can the prediction model be utilized to generate more enhanced versions of Cas9?

Yeah, we think so. We think there's more room to improve the protein. Here. This was really kind of the first, first iteration of OpenCRISPR. We think that as we both collect more data, including the data presented here, and more natural Cas9 examples, we can start to navigate the landscape better and better, and hopefully find really functional proteins that are well tailored to specific gene editing tasks.

Okay, and final few questions. Do you think you could also decrease the size of Cas using large language model modeling? Have you also generated variants of RNA-targeting Cas versions? And do you have an explanation why it was so efficient to generate the design variants that have lower off-target rates, as you did not design for it? It seems almost like spCas9 was selected for the off-target activity.

I can take the first one. So with respect to smaller proteins, there are natural examples of much smaller Cas9s. They're typically not as frequently observed in the databases that we and others have access to. So one really nice application of LLMs like we showed here could be focusing on those and specifically seeing if we can expand the diversity of these protein families that could be really useful for gene editing, especially as therapeutics, and pull more out of those that might be functional. In fact, the one protein I showed that was only 39.2% identity to the CRISPR-Cas atlas was a fairly small protein. It's about 1,000 amino acids compared to 1,380 amino acids for OpenCRISPR1. So it's possible we could kind of push that further and pull out novel proteins that that are smaller.

Yeah, I would just add on to that first point. That would be like for expanding to other families with smaller size. I think there could be also an application for using these models to shrink the size of OpenCRISPR1, certainly for applications like if we just wanted like a dead Cas9 that was a good binder, or maybe even for like OpenCRISPR1 as a nickase. There could be opportunities to use these language models to screen like deletion variants that may function well at a smaller size for some— for certain genome editing applications. What were the second and third questions? Yeah, no, we have not focused on any RNA targeting Cas variants. And then the last question was, do we have an explanation for why it was so efficient to generate the design variants that have lower off-target rates? I think we— seems like spCas9 was selected for the off-target activity. Yeah, you're right that that first set of off-targets, the first off-target analysis that we presented, those sites were selected based on spCas9 having high editing at those off-target sites. So I agree that experiment is a little bit biased. However, our second experiment where we screened activity At a more unbiased set of sites where there's a mismatch in the PAM, we saw a very similar pattern. We were also considering doing an unbiased off-target assay. But I would be confident that the results of that would confirm what we saw already.

Great.

Thanks.

In the interest of time, let me just— last question from Ron. Do your deaminases incorporate David Liu's later mutations, such as in ABE710,8?

They do not. In fact, the deaminases that we used in this work, I think, were in the range of maybe 60% identity to any natural protein. So in general, we don't think the mutations are super portable between the two. The specific ones that enable editing of single-strand DNA were, But beyond that, it's harder to bring them over.

All right, great. Thank you so much for the great talk, and thanks everyone for the great questions. We have another seminar series next week as well. So thanks again, everyone, and hope to see you there.

Thank you.