Mamba-Palooza: 90 Days of Mamba-Inspired Research with Jason Meaux: Part 2

Hello and welcome back to The Cognitive Revolution. This episode is part 2 of my MambaPalooza with fellow AI scout Jason Moe. If you missed part 1, you might want to start there. And if you haven't heard my original Mamba episode from last December, I'd recommend starting with that one for important foundational context. In this episode, we'll be covering Mamba's application to computer vision, experiments in extending the effective context window, And a bit of biology as well. Once again, special thanks to Jason for putting in a ton of work to make this happen, and an open call to all of you to suggest additional topics for us to explore on the show, and especially to volunteer to go on one of these adventures with me. You do not need to have a PhD in machine learning to do this work. You just need extreme curiosity for the subject and a relentless drive to understand what's going on. As always, we appreciate it when listeners share the show online. A tweet is worth a lot, and a review on the major platforms is especially valuable. With that, here's part two with Mamba Scout Jason Mo, beginning with a discussion of a friendly bet that we made about just how many Mamba papers we each expected to see. Enjoy. Okay, cool. So we are back. First, let's talk about the bet. Yeah, we got an interlude.

The bet was between you and me. Maybe it was a few weeks ago. And so we decided, okay, in this 2-week period in February, how many Mamba papers will be published? I think I'm pleased with the results.

Yeah, this is definitely an object lesson in exponentials. Be crazy. In the process of setting this over-under, I was thinking, okay, where are we on this curve and how, how fast is it going to bend? I said 5.5. So you wisely took the over. It quickly became clear that the over was going to win. And then we revisited. Did I reset it or did you reset the over-under?

Yeah, you reset it. I thought, wow, 14, that's going to be a tough get for the reset. But we got fairly close, right? 13 papers in 2 weeks.

Yeah. So 13 was the final answer. My instinct is to say, what's the over-under for the next 90 days? It seems crazy to say that it would be more than 100 that would come out over the next 90 days. But then again, maybe not.

I'll happily set the line and then I'll let you take the over-under.

Okay, sure.

So yeah, 90 days from now, I'll set the line at 55.

And that's more 55 new ones over the next 90 days?

55 new ones. Yeah.

I think I have to take the over, right? Because that would be less than a doubling relative to the first 90 days. Is the second 90 days of Mamba going to have more or fewer papers than the first? I think I have to say more. We could check back on June 5th to resolve our wager. So let's get into vision. This is the next big deep dive section for us. And this one, I think, is interesting because, as you noted, it is the majority of the papers. It's been striking to see this much work on the vision modality, particularly because there was no aspect of vision or image processing in the original paper. I don't know why that is, but one theory would be that images are not sequential in the same way that all the other things are sequential, right? In language, it's token to token. In music, it's proceeding through time and you're predicting waveforms or whatever. In DNA, it's one base pair at a time. But in an image, you have this kind of different sort of challenge where it's not like there's a single order to the pixels, right? They're in at least usually a 2-dimensional configuration. We could have additional dimensions, whether that's going from image to video, adding a time dimension, or even going into 3D imaging. So all of those are represented in this Mamba literature so far. Now, transformers also have a little bit of a challenge with this, and the way that it's typically been handled is by moving things into patches and just chunking images down into small bits and then treating those bits as tokens. That's worked pretty well in transformers. But again, with the transformer, it's not recurrent, right? It's not like processing just one at a time. It's processing everything at once. Even though there is a sequence to it, everything can talk to everything. That patch approach has been basically fine and it's worked really well. And of course, we have lots of vision transformers. So how did folks handle this in the context of image understanding? Let's do a rundown.

Sure. So I guess if you look at the literature of Mamba, we already covered half of the papers. Over half of the papers deal with vision. By far the most common vision task that's tackled is biomedical image segmentation. This was kicked off early January by researchers at the University of Toronto, heavyweight for a lot of machine learning work, and their paper is called UMamba. They used a convolutional neural network block, CNN, that they then interleave with a Mamba block. Their motivation was thinking the convolutional layers could be useful for local feature extraction. So really focusing on what is at the moment at hand, and then using the Mamba SSM for tracking long-range dependency. In the paper, there's a really useful diagram of that. It has this U-shape of what's going on. And you can see there's a chart on the left where we first enter this sort of convolutional layer. It's flattened and it goes through a layer normalization. And then there's this block that includes the— Mamba SSM layer. Their combination of these 2 features helped them get positive benchmarks for organs, CT scans, MRIs. This architecture was able to perform right there at the state of the art, sometimes beating the state of the art, at least how they measured the competition. Very promising paper, and it kickstarted a flurry of image segmentation papers. That came later. I guess just quickly, I could name one. There was also Swin-UMamba, and they actually cited beating out UMamba. So we now have Mamba variants not just comparing themselves to legacy Transformer or other vision models, but now comparing themselves to other Mamba variants. And so each paper is slightly innovating on the next. Just one thing to say about that use case, because it's One that is somewhat new to me. So you can think of a doctor who's trying to monitor disease progression of a patient. They may see hundreds of grayscale images that creates a huge burden for the doctor to separate organs from tumors to different types of tissues. The better these models can be, the better that feedback cycle is for the biomedical field. If you can really quantify in very precise ways what's happening. in a certain organ, that can lead to better outcomes.

Yeah, it seems like in this section then, that is definitely worth emphasising that unlike the language tasks throughout this whole body of literature, which mostly seem to be attempting to compare whatever Mamba version they've cooked up against some sort of best transformer recipe we know, which is actually the language that the original Mamba paper uses, right? They talk about, is it Transformer Plus or something like that, that they call it in the original one. But basically they describe it as the best transformer recipe that we know, which is interesting in that, and I trust those guys because they're obviously leaders in the field. I trust them to know a pretty good transformer recipe, but there could be a better one out there. And obviously a lot of these things are now kept secret. So we do have a bit of a challenge in those language ones where it's like smaller datasets, not huge parameters, not really trying for state of the art, but instead trying to compare on a somewhat like-for-like basis. But you always do have to keep in mind, how hard did they really work to optimise the other one? Was it really an awesome transformer version that they used, or perhaps not the best? I think that is a lurking caveat in some of these papers for sure. Here in the image segmentation portion, it does seem like we are really talking actual state-of-the-art type performance. These datasets, first of all, are hard to create. It might even be worth taking just one more step back and just talking about what is the image segmentation task, at least as it's done in the U-Net. In a way resembles, it really is image generation. It's like image to image. Essentially, you put an image in, it processes it through this U-Net. And the reason it's called a U-Net is because it gradually works from more bits of input and fewer dimensions for that input to a different shape where there are fewer channels, you might say, but greater depth in each channel. And the whole point of that is to— and this is a fairly classic approach for a convolutional network structure, right? You're working your way up from the lowest level information to these higher level features through these blocks. And so it makes sense that you're considering more of the image and they're mixing together as you go through these layers and there's more information for each of these bits of the image that you're considering, but you're abstracting away from the very low-level pixel details up to some sort of higher order, still positionally related, but more kind of blurry portions of the image. And there's more parameters or more numbers to describe each of those positions within the image. This is like pretty classic U-Net type stuff, and it's analogous to how the transformer goes through layers and works up to these sort of high order concepts. This is the vision equivalent of that, where at the bottom of the U-Net you have the fewest locations to consider because you've blurred and zoomed out a little bit from the lowest level details. But you've got the greatest depth of meaning for each of those positions. And then the upside of the U-Net is working back the other way and now moving back toward a new image. So we start with an image working down the U-Net toward high order concepts, and then working up the other side of the U-Net toward a new image to output where all of that high order stuff can now be cached out into a new prediction. And there are also skip connections The way these are typically drawn, there's like the U and then there's lines across the U. And this is basically saying, okay, we're also going to at each stage give that earlier, that as we were working down the U-Net, working up concepts and down the U-Net, we'll also pass that over to the corresponding same size portion on the generation upswing so that we are grounded by that original most kind of concrete representation. And then what's output is basically a color overlay. that goes on top of the original image. I actually don't know if they're generating a pure overlay that then gets merged onto the image, or if they're generating the image with colour all kind of in one. What you end up seeing is this black and white grayscale image, as you said, and then a version of that where shapes have been drawn on in different colours, and the different colours correspond to different things. And this is useful, as you said, for all sorts of reasons. I think it is interesting to realise that the image segmentation is essentially an image generation task grounded in the original image. And to make the analogy between the U-Net, the bottom of the U-Net being akin to the middle layers of the transformer. And now you might also say, okay, with all that said, what's the Mamba?

Doing here.

And there, the traditional way to do this has been convolutions. And the challenge with convolutions is that they are inherently local. So you're going patch by patch and convolving and mixing the information in each of these local patches, working your way through the image. And that's how you progress from one block to the next, is doing all these local convolutions, passing them on. You could eventually have everything convolved into one, perhaps, but there's no way early on to start to understand the relationship between distant parts of the image. And that is where the Mamba portion of the architecture really shines. That is, now I can take a full pass through the entire thing, or perhaps bidirectional or multidirectional passes, and I can get that kind of comprehensive view at the same time. And then combining that with the convolution, is where this thing gets its very best results.

That was an excellent summary of how everything is working. I think your framing of it as an image generation task moves my values a little bit. That's maybe the one area I have underappreciated, having focused more on language. That's not just make a picture of a group of puppies. It's perform an image generation task that segments this image that could ultimately save lives. Yeah, your framing has somewhat moved my priors on the importance of image generation. So thank you.

Check out definitely the old episode that we did with Tanishq Matthew Abraham too. He did a project on virtual staining of tissue. A typical tissue stain that you might use to look for various pathologies in the tissue involves first of all, cutting the tissue out of your body, which is not to be taken lightly, and then slicing it on a glorified meat slicer, and then plating that on a piece of glass, and then dropping a chemical on it to stain it so it turns colours so that you can actually see it effectively. And then having a person look at that under a microscope. And these days you might also take a picture of that and you can have a classifier trained on those images. But to actually get your tissue into even an AI classifier, you have to go through that process of cutting it out, slicing it, staining it, and it takes a lot of time, hours, for somebody to work that thing up. So that means you can't do it while in surgery. His new technique was building in part on a probe, and this was not his particular work, but there's a new kind of probe that allows you to essentially get these kind of 2D images of tissue while the tissue is still in your body. So you don't have to cut it out to get a look at it, which is a great start. But then you also can't stain it because you don't actually have it, so you can't drop the chemicals on it. So now how do you actually bring this to visual accessibility? Because it's all kind of grayscale and hard to see. Virtual staining was the answer for that.

Hey, we'll continue our interview in a moment after a word from our sponsors.

And this now takes the time down to get a read on what is up with this particular bit of tissue from a carve out and a slice and whatever and however many hours that takes to finally get something back to using the probe to get the image, sending this into the virtual stain, and boom, you've got a stain on screen that whole round trip can be seconds. Now, it may still be years before they actually get this into the clinic at any scale, because that's a whole process, but that was definitely a really impressive piece of work. And one of these last papers, the Swin-U-Mamba, incorporated pre-training. They did a very similar thing to some of the other papers in terms of having a 4-way scan, but they boosted the performance even further just by doing pre-training on just like general images. Starting from scratch, because the data is so scarce in these medical settings, can be really tough. And that was something that Tanishq also had to overcome. In the type of— I think even with stains you would have more. But there was another thing that he did called reconstructing the mind's eye, where they basically take result of a brain scan and try to reconstruct what you were looking at the time based on the activity observed in the brain. That data, super hard to get. You have to be in an fMRI. How many people want to sit there and look at images and have their brain scanned just to create datasets? It's expensive. There's not a lot of data. But where he had a significant result was using large-scale foundation models for vision and then fine-tuning them to this particular context where there's not a lot of data, but there was enough to fine-tune. And I think this is a similar thing that they did with this Swin-U-Mamba paper where they basically said, hey, we only have so much of the various kinds of medical imagery in these off-the-shelf datasets, but maybe if we just train this thing on just general imagery first, it will already have a lot of the kind of at least low-level features, right? It maybe should already have edge detection. It should already have these kind of primitives, if you will, of vision. And then we can do that later training and adapt it to this particular situation. And that seemed to work. I don't think it was like a huge advance in terms of the number of percentage points improvement that they got. And it turns out there's a lot of medical image segmentation benchmarks out there. So from what I could tell also, they seem to be referring to somewhat different benchmarks for their evaluation. So it's hard to say exactly how much improvement they got out of that, but at least a few points. And that's just from pre-training on ImageNet. So you could imagine definitely a bigger, better version of that too, if you wanted to take something like this up to internet scale. You want to take Vision Mamba first?

Yeah, absolutely. We'll jump into Vision Mamba. This was a work that came out mid-January from some groups in China, Huazhong University of Science and Technology, an outfit called Horizon Robotics. They use bidirectionality to learn visual representations. So the model consists of a forward and backward pass. And the idea is when you allow information to flow in both directions, you can get a better representation of what's going on visually. They mentioned that VIM, short acronym for Vision Mamba, is 2.8 times faster than the transformer. That they compared to, it saved over 85% in GPU memory when they performed batch inference. Those images had resolutions of 1248 to 1248. And just to underscore the level of enthusiasm they have, I'll just quote their paper. So the results demonstrate that Vim is capable of overcoming prior computation and memory constraints of transformer-style understanding for high-resolution images. It has great potential to become the next generation backbone for vision foundation models.

Yeah, a couple of things jump out to me about this one. First of all, going back to the original monologue, one of the things I had speculated about is that we should start to see variations with multiple states, multiple state-space models working together. And this, I think, was the first one that actually hit the public. And just to clarify, when you say a forward pass and a backward pass, maybe we should use the terms forward scan and backward scan as so as to differentiate between— that's funny, differentiate's another overloaded word in this case. We want to distinguish between the forward pass of like proceeding through the model and the backward pass of backpropagation and actually updating the weights. Here what we have is within a Mamba layer, there is a fork where the image as it has been turned into a sequence, and you can take an image and just put all the pixels in a sequence or put patches of the image in a sequence. But then as we mentioned, you have the problem of the Mamba structure can only process that in order, and it can only decide how to handle certain information based on what it has already seen. What if there is a surprise later, a pattern that wasn't obviously important the first time you saw it, maybe later becomes important, but you didn't really remember it. So how do you handle that? What they do within a given Mamba block is just take that order, reverse it, and then run in parallel 2 different selective state space models. And then they do at the end of that one of my favourite things in all of machine learning, which is just add them together, just superimpose them on one another and proceed. And in that way, you now have basically 2 angles, 2 views of the same data, and they call it forward and backward. And as we'll see, there's going to be other variations on this as well. Where my mind went to was the Silicon Valley center-out compression. What about, why are we starting from one end and going to the other? What if we could do something more radial? Right. It seems like the important stuff is at the center of the image. Who knows if that will really be needed to have a further unlock. But again, this definitely feels like one of those things where it's early going and it's, let's try. We know that images fully be represented in a a single sweep through the image or a single scan through the image. So what if we try 2? We'll take one from either end and hope for the best. And sure enough, it seems to work pretty well. Another thing that definitely jumps out is the fact that there's a robotics institute. I don't know what Horizon Robotics is, actually, if that's a company or a research institute or whatever. But clearly, when you are working with robotics, you're interested in doing stuff on the edge. The Resources that you can bring to bear are quite different. And so when they compare the transformer to the Vision Mamba architecture, they show that you actually get to a memory requirement of greater than 80 gigabytes to process an image of 1,200 by 1,200 pixels. It's a lot to carry around on the edge. The Vision Mamba version drops that memory requirement down to 11 gigabytes of memory. And now that's like under what my several-year-old MacBook Air has. So you go from exceeding what like a pretty high-end GPU has to fitting onto a laptop level of memory. And that's obviously a big deal if you're trying to build robots and have them do stuff and have them be responsive to their environments.

Related to this, V Mamba came out. So if you want to tell us a little bit about V Mamba.

Yeah, it's going to be hard to keep up with all these names. So the first one was Vision Mamba. The second one is V Mamba. And going through this, I felt good about my intuition, or perhaps I should feel like my ideas are obvious because just as I was looking at the forward scan and the backward scan and thinking, geez, I bet you could do a lot of different scans. What would a center-out scan look like? The V Mamba introduces what they call the cross-scan, it is actually a 4-way scan where they start in each corner and they stripe the image 4 different ways. That creates 4 different representations, run those in parallel, and then just can superimpose them all right back into the same space again. From the diagram in this paper, it looks like there are 4 independent parallel computations through the state space mechanism. But that's not entirely clear from the paper. And there are some other examples that show it seemingly differently. My best guess right now would be that you can do it multiple ways and we may not know what is optimal, but they may all kind of work. But in this case, it looks like they are running 4 copies of the state space mechanism. Now, this is relatively small. The parameter range for this is like 20 million up through 70 million parameters. For the last one, similarly in the low tens of millions of parameters. So this is small to the point where you could run 4 in parallel and you probably don't have to worry about it that much because it's all going to be pretty fast. Definitely a question starts to arise if you think, Hey, what would it look like to try to do this at 7 billion or 70 billion? Then can you just run 4 state space processes in parallel and merge them? You could conceptually, but how much would that start to erode the hardware-aware design? My guess is it probably can be made to work, but it's just complicated. If the internal hidden state is big enough, that you can only fit one on the available SRAM, then presumably you have to actually start parallelising this across different hardware, which is certainly done. But those are the kinds of things that a lot of these early papers seem to be skirting around and not really worrying about, just because the datasets and the models that they're working with are so small that they just don't really have to worry about it.

Yeah, I'm with you on that. I think You and I have talked about SRAM constraints. There's a clear signal being sent to the market that there are architectures, including transformers, that benefit from increasing the size of SRAM. There are some companies that have already released some chips that do that. And yeah, if you run things on a distributed compute cluster, how does it affect the hardware-aware algorithms? A lot of that we're yet to really answer.

Already we have 2 related works. It's not always super obvious in going through this literature how much is directly based on earlier stuff and how much people are coming to the same conclusions independently. But one adaptation of this is called VM-UNet. That's Vision Mamba U-Net for medical image segmentation. And they take very direct inspiration from the 4-way scan and essentially apply that exactly as developed in the Vision Mamba paper, but they're applying it for medical image segmentation. But I also wanted to highlight this other one called MambaND, and this one came out a little bit later. This is 8th of February. They again do this 4-way scan, and they also generalize a little bit past this to consider video, modality, and that involves adding a temporal dimension as well. So they describe the image as 2D, and they talk about basically the height and the width dimension, and then they add on a temporal dimension. They also do some interesting things. They like look at weather data as well. So a lot going on here. But what I thought was really interesting about this paper is they take this multi-way scan and then they look at this question of how should we wire up these different state space processings of these different scans. And they've got quite some elaborate wiring diagrams in the paper that basically show, okay, here's what the basic one looks like. Just as a reminder, here's what the bidirectional one looks like, where you've got 2 running in parallel. That's what we referred to a minute ago. Then there is the 4 running in parallel for 4-way scan. And that is, again, I think what the Vision Mamba paper does. But then they also start to look at doing things partially in series. What if we put all 4 or all 6 of our things in a series? What if we put 4 of them in parallel and then recombine and then have 2 more in a series? So obviously the kind of permutations on this are endless. What they report in this Mamba ND paper is that the sequence actually was the highest performing. So they literally just put one state space, selective state space mechanism after another, each handling a different version of the scan as input, but happening one at a time and just passing their results onto the next one. So you've got the skip connection where it's, here's what came out of the last state space processing. And here is your particular version of the scan. So what would I rather do? Would I rather take a— again, this is modelising myself, right? Would I rather clone myself 4 ways and have each version of myself look at something from a particular perspective and then come back and combine our— put our heads together, merge our thoughts, so to speak, and figure out what we can say about it? Or Would I rather, which is really the only thing I actually can do in practice, would I rather just run one scan, learn what I can from that, and then scan a different way, learn what I can from that on top of what I already learned, and then scan a different way? And that sequential processing of the different scans, taking into account what was already learned from the earlier scans, is what they report as giving the best results in this MambaND paper.

Hey, We'll continue our interview in a moment after a word from our sponsors.

I guess I want to say my intuition is that as I talk through it, that does make sense because now I'm not just scanning independently in isolation and then merging later. Now I actually can take into account the results of the first scan as I do my second scan. I already looked at it one way, now I'm looking at it another way, but I can keep that first view in mind, so to speak. I'm mindful that I might just be post hoc rationalizing that, but makes more sense now as I talk through it. What do you think? Do you feel like I'm talking myself into something here or does that resonate?

No, I think it resonates. Doing things in parallel is great and efficiency and distributing workloads. But now I think your comment is correct. The whole point of the selection mechanism is these features are input dependent. So whatever bits of information you picked up on the first scan, the only way you could incorporate that extra input dependence would be doing it in sequence. It's interesting to think about though, because there may be ways around this.

I think that wraps it up. Toward the conclusion, they have a nice little figure that shows what they call the effective receptive field of various designs. They do show a unidirectional, just a single scan versus a bidirectional, both directions versus the multidirectional or the 4-way or however many way scan. And you can see that the receptive field, that is like the way that an individual bit of processing can be informed by the rest of the image, is definitely clearly maxed by the multidirectional.

approach.

But it's fairly intuitive really to say just, again, everything has to be a sequence, right? This is one of these, emphasize these key points for intuition. This thing can only handle sequences. Everything has to be a sequence. So you have to turn whatever it is you're working with into a sequence. And then you also have to think, if I do turn it into a sequence in that way, what might I miss? Or how could I maybe represented as multiple sequences in order to get a robust view of this particular data type. So, yeah, that's definitely a big theme. And I think that basically is the core. That's my biggest takeaway of this entire vision section. So we can maybe move a little bit faster through some of the subsequent papers with that in mind and just see how it's going to play out in a number of different ways. Let's run down the, I guess it's 4 or 5 more still that we have in the vision section here. I think we've got the core themes established. So really at this point, it's just noting that this same technique is working in all these different contexts with subtly different modalities. Next one is called SegMamba, and it goes now to 3-dimensional data. And it definitely also shows the importance of being able to handle long sequences because they're looking at data that is in a 64 by 64 grid. And you're like, that doesn't sound that big. That's 260,000 plus minicubes within the 64 by 64 by 64 cube. So that's a 260,000 length sequence if you're just going one cube at a time. It's also amazing, by the way, I don't know exactly what this one, but some of these things are actually quite crude measures. The one that Tanishq was working with reconstructing the mind's eye was blood flow to the region. And it was a region of, I think, 0.2 millimetres cubed, something like that. A small region, but you can fit a lot of cells in there. And they were literally just measuring blood flow. We're not talking measuring neuron spikes here, nothing that precise, literally just blood flow to little— I remember it being like almost a grain of rice kind of size. Really not big. Big in looking at such a crude measure as blood flow. Something probably fairly similar happening here if you're breaking any substantial tissue into 64, 64, 64. You're getting little chunks. But again, that's already a long sequence. What they do in this paper is a 3-way scan. They do slices. They do a forward and a backward on each slice, and then they do one that cuts through the slices. The whole thing was run on just 4 A100s.

Yeah, that's pretty promising. You can only imagine how much compute you could get yourself into when you start talking about 3D medical image segmentation. That's, yeah, 4 A100s is almost nothing. So that's a positive sign. Anything else on vision we want to cover?

Oh, I think this next one, VVim, this one now moves to video. So instead of having a 3-dimensional object in terms of space. It's now a 2-dimensional object that changes through time. Again, it's another 3-way scan. So here they are scanning forward in time and backward in time, and then the 3rd dimension is within single frames from the video. So generalizing to video seems again to work pretty well. And then 2 more that I lumped into this vision section, one called Mamba Morph. I thought this one was maybe most interesting just because of the nature of the task. I hadn't really seen this before. Here they have 2 different kinds of scans, a magnetic resonance scan, typically an MRI, and a CT scan, a computed tomography scan. Now, you take these 2 different scans, it's 2 different machines, right? It's the same body of the person, but you're going to have disagreement between these 2 scans in terms of the shape that they are seeing. What do you do about that? This task is called registration. You have each of the scans and then the output is, and this is another kind of segmentation-like sort of task, but instead of generating a new image, the output is a deformation field which describes how the one image needs to be deformed to fit with the other image. Because your body was in different positions or whatever as the scans were taken, they just don't align. And so it becomes very hard to read them together. This deformation field that this thing can create can then be applied to one of the original scans, snapping them essentially together in alignment so that you can see the results all in one view. But just another interesting use case where, you know, imagine how hard that would be to do, right? You basically just can't do that stuff manually. There's actually 2 GraphMambas. You want to take GraphMamba?

Yeah, GraphMamba. These are 2 papers I probably are a little bit light on. The first one, it's titled Towards Long-Range Graph Sequence Modeling with Selective State Models. What they observed is that attention mechanisms really have been the state of the art here for tracking long-range dependencies among nodes and wanted to see if state-space models such as Mamba, could it enhance the long-range context modeling in a more efficient way. And they go up to 1,400 nodes. The average GPU memory usage for GraphMamba is very low, sub-200 megabytes. And the transformer-based model they used previously basically errors out at almost half that, 800 nodes. And you can just see memory kind of top out there. That was probably the biggest thing that jumped out at me at this paper. Anything else, Nathan, on this one?

Yeah, I can't say I'm an expert in graphs either. The sort of problem seems to be like predicting what amino acid might appear in a given position within a protein sequence, for example. And that's a linear sequence in some sense, because the DNA sequence is linear, but it's also like a 3-dimensional thing, and there's different residues interact with each other, and it just gets extremely complicated. They end up being represented as graphs. I think the key thing to know here is that graphs don't necessarily correspond to a single sequence, right? They have— and again, I'm not an expert in the math of graphs, but you can just imagine a graph. And let's say you have A, and then you have B points to A, and C also points to A. What's the order of that? There isn't really an obvious order, you have 2 things that both point to the same thing. Whether B or C should come first, or maybe even A should come first, that is not obvious. And so they essentially have to develop heuristics in, and there are 2 of these graph papers, to flatten the graph structure into some sequence. And that's a deterministic process that uses the best judgment that they have to figure out how to cluster these and what should come before what. And then one of the papers also does a reversal again. So you have whatever their best heuristic guess was and the reverse of that. But also, as you noted, especially the performance, trying to handle all these interactions is extremely memory intensive. But with these scans, it's just way, way easier to handle it at a reasonable amount of memory. The flattening, to me, again, the art, we're in the seeing like a state era. I think of this as seeing like a Mamba. It's about what is the sequence representation that I can bring to all of these different things. And I may need deep expertise to figure out how to flatten this graph structure into a sequence. And then for good measure, I might need to just go ahead and reverse that as well. But if I can do that, then I can cast all these different kinds of problems as something that the new architecture can handle.

Yeah, that's a good— you need to trademark that, seeing like a Mamba. That's going to be in a paper title coming up. I'm sure.

That's right.

Watch out for it.

Calling our shot now. So that brings us to the end of vision. And I think we've probably harped on the key points enough. Everything has to be a sequence. Figuring out how to represent different things as sequences seems important. And it seems like a lot of ways to do that. A lot of different just multi-scan approaches and seemingly a lot of flexibility in terms of exactly how you wire that up. Do you put each scan in parallel and then combine them at the end? Do you put them in sequence? We've got at least one paper that did a systematic study of that and found that the sequence works. Most of them seem to be doing it more in parallel. So I think the final word is definitely not yet written on the best practices there, but my current read would be it seems like if you just kind of do a multi-scan approach and wire it however you want, it'll probably work.

That was a great summary of vision. I would love to jump into Long Mamba. Yeah.

So now we're on to, I think this is our final deep dive section, right? And this is about actually using the super long context. And again, this is one of the things that got us really excited about this in the first place, that you could process, in theory, an arbitrarily high number of tokens and develop some high-dimensional internal representation of that full history, not with full fidelity necessarily, but ideally with all the important parts represented that you'd really need to be successful. That's again why I think multiple states seems inevitable, because probably can't have one approach that will do everything the right way, but maybe across a few you could really get there. But the fact that this is linear in sequence length because it is constant time per step just is so exciting. It's, man, what if we had a million? What if we had 100 million? There's, in theory, no limit to how many tokens you could process and gradually build up this super high context state. But now, okay, cool. We saw in the original paper DNA was running long. They didn't have super long text. So this first paper is taking a step in the direction of, okay, let's actually use some longer text episodes for training.

Yeah, absolutely. It was exciting when this code repo came out. LongBamba, they used a subset of the same dataset this model was trained on, and they simply cut it off at a little over 16,000 tokens. And so after training on 16,000 tokens, the perplexity curves look good. Perplexity itself generalizes well past 16,000 tokens. You don't see increase in perplexity till past 40,000 tokens. So that, that was an indication that something's happening. Something's generalizing past the training window. And then they ran a test that's become popular called Needle in a Haystack. It was popularized by someone on Twitter, G. Kamrod, who ran these evaluations on Claude and GPT-4. Essentially, What that eval is meant to do is to hide some information inside varying positions in the context window. So you can imagine a 10,000-token window. What's the sensitivity in that window for the model to find information if it's near the beginning, in the middle, or at the end? They used slight variant that was in the academic literature from a paper last year. I'll just read the actual text of it just to give an idea, because it's funny that we're evaling models this way. But it says, there's important info hidden inside a lot of irrelevant text. Find it and memorize them. I will quiz you about the important information there. And then there's this simple repeating of sort of noise. So the grass is green, the sky is blue, the sun is yellow kind of repeats. And then somewhere within the context, There is a passkey. For example, the passkey is 12362. Remember it, 12362 is the passkey. So imagine running that eval now hundreds and hundreds of times. And they came up with a needle in the haystack result that actually looks pretty good at 16K context. It's nearly perfect. There's a couple spots where it looks like there may have been some imprecision. At max context length of 16K, it was actually in the initial position near the front where the model failed to make the retrieval, which is sort of interesting. There's been this tendency in transformers, some evaluation transformers where this lost in the middle idea in the SSM, it actually seems to be more starting lost at the beginning, lost at the top. When you try to push the model past its training window for this evaluation. It does generalize a little bit, so there is a bit of an ability to go past 16,000 tokens and retrieve the passkey, but it quickly diminishes. Anything beyond 20,000 tokens, it's going to struggle. I guess one other thing is the author was only able to train on 16,000 tokens of context. I believe there was a memory constraint up to that point. He extends a conversation that was on GitHub between TreeDAO and Zhang, which is this idea of applying this Transformer XL style modification. If you want to train on something like long context length, perhaps you don't have to do that all in one go. You can, in the case of Mamba, save the SSM hidden states after a batch. And then load those SSM hidden states back into the model and then continue training from there. So rather than stopping training Mamba at 16K, you can just keep training in batches of 16K on the same hardware. So that's an interesting thing. There's some code in this repo that tries to implement that. There's nothing inherently about the architecture that would prevent this. You should be able to do this Transformer XL style training. You should be able to do this saving of the hidden state. So that's LongMamba. Any thoughts on that?

Yeah, I guess the big bottom line takeaway is it's a pretty strong piece of evidence that, and we already had pretty strong evidence in the form of performance on long DNA sequences from the original paper, But we didn't have examples of language going beyond a couple thousand tokens. And even in the chat fine-tuned version that I tested in December, I found that it basically unraveled after a 3,000 or so token level. And then I looked into the data that it was trained on, and sure enough, it was trained on pretty short chats. So not crazy that it was just not going to go any further. I would love to understand. I don't know if you have the understanding. It's interesting. It seems like it works. There's 2 interventions in this paper, right? If I understand correctly, one is maybe 3, but at least 2 is one is changing this one hyperparameter that somehow the delta parameter, I don't quite get what it's doing, but somehow just changing that allows it to immediately work at longer context, although not as well as just actually training it on longer context. And then it's also interesting to see that when you do train it on the 16,000, it seems like it works well for much longer than that, with aggregate scores continuing to improve up through around 40,000, or a multiple of the 16,000 training length. And then it does start to get a little worse as you get out past that. It's still not too bad, but the needle in a haystack is gone at that point. It suggests that it's like a kind of fuzzy memory of things that came before. Still, maybe the fact that perplexity is going down, would seem to suggest that it's still getting some additional value from those earlier tokens that it's seen, but it no longer has the precision to do the needle in a haystack. It seems like it's kind of, again, I'm modelizing myself here, but I always say, I can't remember what I had for breakfast today or yesterday, but I can remember what I generally have had for breakfast. And it seems like there's a little bit of maybe that happening where the representation that's building up in the state is more useful than not having it. Training on longer sequences would probably overcome that challenge. It seems unlikely to be a coincidence that this was trained on 16,000 token sequences and has needle in a haystack high scores up to exactly that level. But yeah, and then you can also— another thing that I think about here, one of the One of the more interesting things that somebody has said to me recently was Robin Hanson saying that we will likely continue to struggle with rot in AI systems in the same way that we struggle with rot in biological systems and in traditional software systems. And I'm still not 100% sure what I think about that at the level of foundation models, but it does seem to be an issue with states. If I was to interpret the reversal of Perplexity, right, where we've trained on 16,000 tokens, we're seeing improvements up through about 40,000, and then it starts to get worse again, but not too bad, but it is getting worse again. That kind of seems like a rot thing where it's now you're just gumming up the works. It hasn't really been trained to handle something this long. They're not catastrophically failing, but the works are getting gummed up and it's getting marginally worse. You might wish that you could just throw more and more at them, and maybe it doesn't get better. Hopefully, at least it doesn't get worse. That would seem to be another conceptual advance that would be needed to figure out how to prune the state, to let go of unneeded things. And that would maybe again go back to one of our earlier topics around some sort of optimisation targeting the state itself. In fact, here's a prediction: in the next 90 days. We will see some sort of regularization of the state. You might have talked about weight decay and state decay in the past, something where the old stuff is gradually let go of, some sort of strategy to clear out the gumming up of the works and keep things hopefully running at least constantly as you go through time. But you have to do that by letting go of some stuff. It's interesting that we definitely have that in our brains, right? So there's some sort of pruning of memories. We're not keeping everything. We're consolidating.

And when you say pruning, you're thinking pruning of the state, not the weights?

Yeah, I'm thinking— yes, I am thinking about the state. Pruning may not be the right word exactly. I don't have an intuition right now for exactly how you would do it, but we've seen a lot of things in general over the last few years where it's, oh, here's a clever penalty that I'm adding to the loss function. A great example of this was Seeing Is Believing, which was out of Max Tegmark's group, and Zhiming Liu was the lead author on this paper. They trained very simple kind of toy networks, but with a loss function that had a penalty for just the magnitude of the weights. It was optimised to solve the problem, but then also subtract I think it was just the sum of all weights. So any weight that wasn't directly contributing to solving the problem at every step was just getting turned down. And this created these sort of crystallising effects where you'd start off with a very messy, dense network, and over time you would get down to the most sparse network that could solve whatever problem they were solving for. There's a couple of parts maybe missing here because we're not seeing anything yet that is optimising on the nature of the state. So figuring that part out alone, and that may be a little tricky depending on how, again, getting things on and off of SRAM could be an interesting challenge associated with that. If you need to do that at every step, that could be tough. But one kind of crude version of it would be however big the state is, penalise the sum of all the values in the state and therefore just try to filter out information that isn't carrying its weight so that only the actual needed information over time would be retained.

Yeah, that's a great prediction. What you just said reminded me of an idea that you had riffed on, which I agree with, is this idea of memory decay, state decay, in the same way that you could think of almost weight decay in a model. And I think that is very interesting. Because there are all kinds of ways you could design that decay function. You could do it over time, you could do it over tasks, you could do it in all sorts of ways. And it's just intuitive that could add a lot of value and efficiency in the same way that human memory is both frail because it has a recency bias, but it's also incredibly efficient because we have a recency bias. And it's somehow equilibrium's all out. where we tend to have the right balance of that.

Thank you.

I don't know that I have anything much more to offer at this point in time, but yeah, I guess to summarise my understanding of this right now, it doesn't seem like there's any obvious barrier to super long context natural language tasks, but it does seem like we will need datasets to actually do that. The fact that you can scale up to 16 1,000 tokens on 8 A100s is definitely pretty encouraging and seems like it probably is pretty scalable. But if you don't specifically really reward high attention to detail of the needle in a haystack variety at a given length, then it seems like beyond a certain point, memory is allowed to get fuzzy. And beyond an even further out point, The state starts to get gummed up and essentially, as Robin Hanson would put it, rot. And at some point it catastrophically fails. So we obviously can't train infinite length states. So if we want to have something that can run infinitely, we're going to have some sort of cleanup mechanism that would prevent that process from eventually ruining things. Now, Robin believes that will not be easy to achieve. He notes that evolution could have created organisms that live forever and just regenerate themselves, but in practice didn't. And instead we have generations. He expects the same thing to happen. And we also do see that in software, right? We do not incrementally upgrade software. Instead, it's, it's time for a new system. Partly you could also argue that's because the technology foundations have advanced and maybe if they were held constant, maybe you'd more incrementally update things. Even I think a lot of software developers would be screaming for the rewrite. In this case, can we come up with that mechanism that effectively clears out the old accumulated gunk that's no longer needed in the state? Yeah, throwing down the gauntlet of that challenge to the community. If we want to see guys that live alongside us as long as we do, that's a lot of tokens. They're going to have some version of that. I'd be also very interested to go look at more detail, and I'm certainly not schooled in this, but what do we know about how that works for us? We have certainly a better mechanism than the state space models do at this point. Our brains do ultimately decay and fail, but they last a lot longer. They process a lot more tokens. So there is something that we're doing that kind of addresses that problem. And it would be interesting to see what biological inspiration we might be able to draw from. And that perhaps is a good transition to the next in this sort of super long context section, which is this project called Evo that came out of the Arc Institute. This is something that has super big deal written all over it. Honestly, I would say the Arc Institute, if you haven't heard of that, it's well described in the recent Patrick Collison and Dwarkesh podcast episode, where basically they observe that a lot of people would like to do science, but they don't necessarily want to be a lab leader as you have to be if you want to do science in today's academic environment, right? You have to become a professor, you have to write all the grants, you have to recruit the grad students, you have to manage the grad students, you have to go acquire your own machinery in many cases and maintain it within your own lab. And there's just this whole managerial and financial and machine maintenance aspect to the enterprise of doing science that a lot of people are either not that excited about or perhaps not that well suited for. And then, of course, many people worry, geez, what if the best scientists are just selected out before they ever even get there? Arc Institute basically tries to tackle that problem. They essentially try to get the best scientists to just come there and just be scientists, and they'll pay them a salary, and they don't have to do the grant stuff, and they have all the machines. Hey, you're good at science. Do science. That's what you do. You're a scientist. So it actually sounds pretty awesome. And they've had a couple hits already in a pretty short organizational history. This one, Evo, is relevant to us and may be relevant to everyone. This might be the most important thing that we talk about, depending on exactly how things go from here. But they use the Stripe Taena model, which is another hybrid. This is the architecture originally put out by Together AI, proved by Together to be effective and fast language model and now adapted to training on DNA sequences of up to 131,000 tokens. In this case, each token is a base pair within the DNA, working primarily on prokaryotic genomes. So basically, I think mostly bacterial genomes so far. And they're observing all these kind of advanced behaviours out of this model where it can now generate long sequences. And because it has this long context awareness, distant parts of these generated genomes seem to relate to each other in interesting ways. And I would even say somewhat speculatively, but seems pretty real to me, if you believe that a language model, is developing a world model, because developing a world model is the best way to get good at next token prediction, then it seems like this is maybe the first thing that we've seen that is doing something similar for biology in order to best understand and predict the genome, especially with these super long range dependencies. And DNA is like a terrible case for that, way worse than language in language, something that's hundreds of thousands of tokens away, could be related. But by and large, you don't need that much context to know what's going on in the text that you're currently reading. Whereas in a cell, the different portions of the genome, they may be close to each other, they may be far from each other, but the way that they interact is to a significant degree independent of just how close they are on the genome. A biologist probably could come in and correct me and say, actually, it's more complicated than that. But definitely fair to say you could have things on polar opposites of the genome each express a protein, and if one of those is broken, then the whole cell might die. You have these long-range dependencies which can be super important. And it seems that in order to make these long-range predictions, this Evo model has started to develop what you might think of as a cell model. Again, if you look at the language models and you're like, this thing seems to have a pretty good intuitive sense of what's going on, obviously not perfect, but it's amazing how much of a world model does seem to develop just through next token prediction. Here we seem to have something similar where a broader multilevel understanding of biology is starting to emerge from just training on the DNA because the long-range dependencies can be effectively handled. So I thought this one was probably worthy of an episode all to itself. And by the way, if anyone's listening still this far into this episode, I am actively looking for the Jason of the intersection of AI and biology, and I know there are some of you out there. But I would definitely love to team up with somebody a little bit to do some deeper dives into the intersection of foundation models and biology. This is an area where I'm aware that there are super important things happening. Arguably, it could transcend what we're seeing in language in terms of importance. And I think that the language stuff is going to be very important, but the biology stuff, if it works, it's going to be every bit pound for pound as important. And I struggle to read the papers because at this point I can handle the machine learning side pretty well, but the biology is far more complicated and just has far more facts. As Patrick Collison said, we don't really have record of somebody in their early 20s revolutionising biology. And the reason for that seems to be that it's just super complicated and there's really no way to get to the point where you can lead the field without slogging through years of all these details and facts and interactions and whatever. It's not, I mean, as much as people may think machine learning is complicated, biology is way more complicated, way more messy. Everything is not designed and it's totally haphazardly interacting. Consider this my call. If you're out there and you're interested and you're working at the intersection of machine learning and biology, specifically foundation models for biology. I would love to do a similar lit review survey episode and would welcome any outreach or even any nominations. If you have somebody in mind, let me know and I'll be willing to go knock on their door if that's what it takes. But anyway, Evo, 131,000 tokens trained on. Again, you see that kind of ratio where it seems to work up to— they say it works up to 650,000 tokens. Perhaps that's just a coincidence, but it definitely caught my attention that the last one was trained on 16 and worked pretty well up to about 60s. That's a similar ratio. Maybe that's nothing, but long contexts and seemingly profound understanding coming out of this new architecture as applied to biology.

Yeah, I have a follow-up question for you. They have piece in the paper, is DNA all you need? You think you made a good case of why it might be. Do you see it as a natural extension of some of the work we've discussed or some of the toy problems where the model can understand a simple linear regression or can understand a board state of a game? Do you see that as just scaling up that sort of emergent understanding of something or something fundamentally different happening? How would you think about that?

I think of all the papers we've talked about, it seems like the graph one is maybe most analogous in some ways. The DNA is a sequence, but the interactions at the RNA level and the protein level, those are gnarly graphs, basically, right? You've got things that promote other things and that inhibit other things and that block other things from going to the target that they need to go to. And it's just, man, there is a very complicated causal web or causal graph that seems to play into any actual phenotype that we care about.

Yeah.

In that sense, you could maybe make an analogy between those graphs and the, and their flattened versions and understand the DNA as a flattened version of that graph. What we really care about is all the interactions that are happening within a cell and whether it's working appropriately or not happens at that graph level. The DNA, while it is a sequence, is a naturally occurring flattened version of that graph that actually is the cell. Maybe I squint at it and see it that way. It's weird because the DNA is an actual physical thing that encodes what happens at the higher level and is not a synthetically flattened thing. But I'm not sure how much that matters for the purposes of the machine learning challenge with these techniques anyway. If you did have the graph representation of all the different parts of the causal machinations that go on in a cell, and you said, okay, here's my big gnarly graph representation of everything that determines what's happening in a cell, you might just be like, okay, I'm going to have to make a couple of cuts here in these cycles. Now I'm going to flatten. Now I'll reverse. Let's see how we do. And given that, if that's the best strategy that we have, then it's also not crazy to think, hey, maybe in a sense nature did that for us with the DNA encoding of everything that's going on in a cell. And yeah, maybe we can learn from that.

Yeah, it's fascinating. This seems like a model that can be legitimately useful to researchers. It's what you might call the most production-ready version of almost everything we've described today. You think this is somewhat of an inflection point?

Yeah, I don't know. Arguably, maybe the inflection point has already happened. I think certainly things like AlphaFold are a huge deal. They have a whole division now within Alphabet that is dedicated to taking AlphaFold, extending it, commercialising it. They have drugs in or very near clinical trials that were originally discovered with these techniques. So I might even say the kind of critical technology threshold maybe even already passed even before this. When is it worth the wet work? That's a good memorable phrase. When is it worth the wet work? There's an infinite amount of experiments that we could do in actual chemical space, but they can be slow, they can be expensive. Somebody has to measure out the stuff. You need all the reagents, blah, blah, blah. Obviously, the advantage of computer systems in general is that they run faster than our ability to do those sorts of things. These systems are always going to have, or at least for the foreseeable future on the current paradigm, they're going to have errors. They're going to have false positives. They're going to have false negatives. They're going to be definite imperfect models of the cell or whatever it is that they're modelling. But if they're good enough, then they can really narrow the search space. I think that's how people are broadly using them right now, is to say, okay, let me just run a zillion generations and we'll take the top whatever percent and actually look at those in actual tissue culture or whatever next. And then eventually we can work our way into actual organisms and then into people. And so it's just the space is so vast, even if these things have some false positives, false negatives, whatever, If we can just narrow the search space by a couple orders of magnitude, or another way to think about that would be increase the hit rate at the experimental level by a couple orders of magnitude, then it's a total game changer for the ROI of doing the science. If it costs $1 billion to bring a drug to market, a lot of that is clinical trial and probably doesn't change. But there's definitely a part upfront that is, I need to just throw a tonne of stuff at a tonne of diseased cells and try to find a hit. Think of it as like playing the old kids' game Battleship. You just very largely historically have been shooting in the dark. You don't necessarily know the shape of the proteins up until pretty recently. You don't have much of a sense of how things interact, and you're just throwing stuff at cells and seeing if anything kills the cancer. You're just hoping to get hits. That's been, I think, a lot of it is pretty blind groping search. And the hit rate has been pretty low. There's certainly a lot of techniques that people use that are, again, finding things in nature. Hey, look, this frog doesn't get cancer. What's going on there? Maybe we can find something in that frog. There's definitely other things besides purely blind search, but to be able to just crank up a supercomputer and start scoring everything and then just narrow your search to the top 1% or whatever and hopefully get something that might be 100x higher hit rate on the other end of that, It is definitely a regime change in science and also would seem to increase just like how much of that work we would want to do. I don't think we're going to deplete the hits in the near future. If you could find them at 100 times the speed, one would assume that you might imagine doing 100 times as much of it. Jeez, this has just got 100 times more valuable. Let's go do a lot more. That's my general high-level understanding of how this stuff might really come to change the world. And I think there certainly could be a lot more nuances to it than that. If you can get that hit rate up from what it has been, it definitely could be a real game changer. I do think in terms of your comment that the state-space models are influencing attention, I think going back to the original episode, or maybe it was in Twitter discussion afterward, somebody had asked me, what are the odds that you think this actually happens? state-space models are part of frontier systems in the next 2 years. And obviously we could debate exactly what the percentages are there. I think they're pretty significant, if only because the world is definitely hungry for compute, and this takes a little bit of the edge off of that hunger. That alone is enough reason, I think, to expect that it's going to make its way into the mainstream. But I also said, And still believe that if it doesn't happen, why wouldn't it happen? And the best answer I could come up with there is just that attention really is all we need, and it just keeps working, and there's just enough optimizations. And hey, we already have line of sight to 10 million tokens or whatever with ring attention, and we're seeing that come online in some form with Gemini 1.5. Obviously, we don't know what's under the hood with Gemini 1.5, but my guess is, as we talked about, I think there probably is some full attention there. still. But hey, maybe it just works so well that we don't need to send as many people or optimize the state space side as much. If the first 90 days are any indication in terms of the literature that we've seen, nobody needs to incentivize people. They're just excited to go do it. So it doesn't seem like there's going to be any critical shortage of people eager to play around with this and figure out how to make it work. But it is definitely notable that the progress continues to advance on the attention side as well. And you can imagine memory tokens or other kind of things like this. Did we talk about memory tokens or no?

No, we did not.

Okay. This would be the other way in which I could imagine something like this happening within the attention paradigm. We have these different kinds of tokens, the pause token from Think Before You Speak, the backspace token, And we have seen one paper now that has something that I think of as a memory token, which is basically just compressing a certain part of the context into embedding space in a similar way that images get ultimately put into this. It's a joint text-image embedding space, but it's largely text-based, right? These models are ultimately going to output text, right? GPT-4V, Claude 3, Gemini Ultra, Gemini 1.5, they all output all text. So they take in these other things, but they're ultimately working those things through a text latent space and ultimately outputting text. If you wanted to extend the window and you know that you can represent these— hey, a picture's worth 1,000 words, but a picture only costs 85 tokens at the low-res setting on OpenAI GPT-4V. It seems like there is an inherent efficiency there where individual tokens, the one-hot embedding, space that the text itself gets moved into is not the most efficient way that you could represent that text in embedding space. So perhaps you can imagine a 1,000 to 1 ratio, right, where you'd say, okay, I'm going to take this whole body of text and compress it down into a single token. Does that give you a 10 to 1, 100 to 1, 1,000 to 1, some ratio like that? And then by compressing those full text sequences down into these memory tokens, you could potentially still have a sort of full normal attention mechanism downstream, except that in this deep history, you're maybe working at like 1,000 to 1 sort of ratio. Now, obviously, the compression has to be effective. The attention has to learn how to take advantage of that compressed representation. And you might even imagine the librarian kind of setup where it's also, if I get a hit, on a certain token and I need more, maybe I can even do an inbuilt RAG type thing where I actually go pull the original context that compressed token represented, pull the whole thing into context if I really want to be extra careful about making sure I'm recalling it effectively. But these are the kinds of things that I could see substituting for a state space revolution would be the just relentless march of progress on the transformer and the attention mechanism and finding these sort of compression hierarchical native RAG formats that might just be so good that it takes the air out of other lines of research. Certainly, I would say transformers have done that in general, right? We've seen everybody has left whatever their pet architecture was over the last few years to largely go do stuff with transformers. Now, here's something that's like pulling some of that energy away from transformers again. Still a very small percentage, of course, but it is pulling some away. But will the transformer have the last laugh by just continuing to work so well that nobody needs to go a different direction after a while? I certainly couldn't rule that out. It feels like it could very much go either way, where the state space models are definitely getting momentum, as we've seen. Incredible flurry of activity in just 90 days. But don't count the transformer out either when it comes to just figuring out perhaps more local modifications that could achieve similar things.

Yeah, you mentioned the Gemini 1.5 Pro data point. I think that's important because this model came out. It seems that they've achieved the ability for very long context windows. And as far as I can tell, they're indicating that this is transformer-based work. It's also interesting to Just put the data point out there that there's another company, Magic.dev, that recently announced, I think, an additional $100 million raise from very legitimate people in the space, Nat Friedman, Daniel Gross. And they clearly state on their website, quote, transformers aren't the final architecture. We have something with a multimillion token context window. So you can only begin to speculate what they might be cooking up. At the same time, we see Transformers flexing long context windows. There is this interesting data point with magic.dev.

We may soon have an episode with somebody from magic.dev to probably not spill all the secrets, but maybe at least tell us more about what it is able to accomplish. I'm definitely really intrigued to see that. I've just dropping a full codebase into Gemini 1.5 has done a lot for people and it wasn't even really productized for that. That's just just dropping stuff into it and hoping for the best. Yeah, it sounds like what the Magic team has got is going to be a big deal. It sounds like that's pretty likely at this point. Mindful of time here, we have been at it for a while. Do you have any concluding thoughts, or should we adjourn and resolve to come back and take stock of this again in either 90 days or perhaps even sooner when our personal context windows are hitting the point of overflowing. Should we leave it there for now?

Yeah, let's leave it there for now, Nathan.

Thank you for all your hard work in putting this together. The website where people can check out the link index at least is statespace.info, and we'll share some more stuff online as we publish this episode as well. But for now, it's my pleasure to say, Jason Moe, fellow AI scout, Mamba Scout in particular, thank you for being part of the Cognitive Revolution.

Thanks, Nathan.

It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.

Turpentine is a network of podcasts, newsletters, and more covering tech, business, and culture, all from the perspective of industry insiders and experts. We're the network behind the show you're listening to right now. At Turpentine, we're building the first media outlet for tech people by tech people. We have a slate of hit shows across a range of topics and industries, from AI with Cognitive Revolution to Econ 102 with Noah Smith. Our other shows drive the conversation in tech with the most interesting thinkers, founders, and investors like Moment of Zen and my show Upstream. We're looking for industry-leading hosts and shows along with sponsors. If you think that might be you or your company, Email me at erik@turpentine.co. That's erik@turpentine.co.