AI On An Exponential? Data, Mamba, and More

I hope everyone watching had an excellent 2023 and is looking to get 2024 off to a rambunctious start. This video though has been made to show that we are on the steep part of the exponential and will be for a while yet. I'm going to give you four clear reasons why, though I could have easily given eight or 16 depending on how you categorize them and how much time you've got. We will look at how data quality will change everything according to the famed of Mamba and mix trial and then how models will start to think before answering and according to this fascinating new paper, just how much low hanging fruit there is out there in AI that doesn't even require more compute. And even setting all of that aside will end with the explosion in multimodal progress that is occurring around us. That by the way, will include listening to a version of my voice that you might find hard to distinguish from my real one. I'll also finish with some predictions for the year ahead, but I'm going to start with Mamba, but not yet the new architecture that is causing shockwaves. I'll cover that in a minute. I'm going to start with one of the co authors tried out and despite all the buzz about his new architecture, here's what he said about data quality.

All the architecture stuff is fun. Making that hardware efficient is fun, but I think ultimately it's about data. If you look at the scaling scaling log curve like the model architectures, different model architectures would generally have the same slope, they're just different offset. Seems like the only thing that changes the slope is the data quality.

Yes, we're going to cover Mamba in a minute, but for language modeling it does perform better than the transformer. Basically the best version of a transformer. But according to this graph with five or ten times as much compute, you could replicate the performance of Mamba with a transformer. And all of this remember is for a small 3 billion parameter model. We don't know what it will be like at bigger sizes. So for language modeling, potentially a big breakthrough, but data quality still means more. We are not even close to maximizing the quality of data fed into our models. Here's Arthur Mensch, co founder of Mistral, the creators of some of the most cutting edge open source AI models. To increase data efficiency, you do need to work on coming up with very high quality data filtering things. Many new techniques that needs to be invented still, but that's really where the lock is. Data is the one important thing and the Ability of the model to decide how much compute it wants to allocate to certain problems is definitely on the frontier as well. So these are things that we're actively looking at. I'll speak more about letting models think for longer and inference time compute later on in this video. But if you still weren't convinced about the importance of data quality, here's Sebastian Buback who might be able to persuade you in an interview for AI Insiders on Patreon. That last slide that you did on for your channel, amazing channel, where it was like a thousand x increase in effective compute from parameters and data, it seems massive to me and I don't think enough people are appreciating that point.

Yeah, I think it's pretty massive, to be honest. You know, before working on this line of work, I was thinking about improving the optimization algorithm, I was thinking about improving the architecture and I worked on this for a few years and we could get, you know, 2%, 3% improvement, like this small type of, you know, around the edges still it's, it's nice but it's, it's tiny. But then suddenly when we started to focus on the data and really trying to craft data in a way that is more digestible by the LLM at training time, suddenly we saw this incredible like, you know, thousand x gains. So yes, I think, I think it is massive and it's really pointing to where the, where the gold is and the gold is in the data.

Now at this point, I know many of you will be saying it, Philip, data quality is important, but tell us about the new architectures that might be processing that data. In other words, tell us about mamba. That's a new architecture that has been generating a lot of buzz in AI circles. If not for the general public. I've been hoping to speak to one of the only two authors on this paper. In the meantime, I want to try to translate its significance to the layperson. I'm going to try to convey what Mamba is and what it means in just two or three minutes. Before we touch on contender, let's talk about the King Transformers. That's the architecture behind everything from Dall E3 to GPT4. And transformers are famous in part because they're great at paying attention. In this diagram from the famous illustrated transformer, we see that as we encode or process the word it, the transformer architecture pays attention to all of the previous words or tokens. Some of course, like animal more than others. And it will do this for all of the words. It's going to encode. And the truth is that paying attention is great, but the kind of attention in Transformers where every element must attend to every other element is complex at big scales. In fact, it's called quadratic complexity. And whenever you hear that word quadratic, think square. You may remember the word quadratic from school and the y x squared graph. And hopefully that makes sense, because as you double the number of elements, you far more than double the number of pairwise connections. In a very rough analogy, imagine everyone shaking hands in a room. And if you triple the number of people in that large room, you are roughly 9xing 3 squared the number of handshakes. But now imagine sequences of 1 million interconnected tokens. No one has that much attention to give. But there are other known ways to process a sequence of inputs. How about a state of fixed size being updated by inputs step by step? It seems simpler, right? Although it's a lot less parallelizable. That is a hard word to say. Now, there have been attempts for quite a while to get the best of both. Here's Albert Gu, one of the lead authors of Mamba, and his paper from 2021 was called Structured State Spaces for Sequence Modeling. Indeed, that 4S's name had so much S sound sibilance, it inspired the Mamba snake name. But now let's get to the Mamba paper itself and the key diagram therein. There's that hidden state on the left, merrily processing its way across to the right, updated in turn by the inputs coming in from here. This long lasting state needs to be a rich but compressed expression of all of the data seen so far. But if this approach can be made to work, it would mean far fewer connections to compute and therefore faster inference. Indeed, the paper claims 5x faster inference on Nvidia's A100. And without that quadratic complexity, it would mean that even extremely long sequences, think vast code bases, DNA sequences, and even the video input From a long YouTube explainer could now be handled without a mental breakdown. But remember, that state needs to compress all the data it's seen, including therefore ignoring certain inputs. And that is where the selection mechanism comes in. It decides what inputs to ignore and which to concentrate on. The trouble is that this rich and expressive hidden state with the distilled selected inputs is slow to process and computationally demanding. So how did Triedao keep this hidden state rich and expressive? With the distilled wisdom of previous inputs, how did they expand that state without bringing everything to a standstill? Well, by painting it orange. No but seriously, what orange means in the diagram is that it's processed in the GPU sram. Think of that as the super fast part of the GPU's memory with the shortest commute to the processing chip. In contrast, all the model parameters in green which change and the inputs can be handled by the slower high bandwidth memory. All of this is where we get the term hardware aware state expansion. It's an architecture built with an awareness of the kind of GPUs it's going to run on. Let's try to make this more tangible with an example of what we can achieve with all of this freed up complexity. Take this induction head comparison. First take the word explained, which is made up of two tokens, expl and aimed. An induction head is a circuit that might be attending to the token explor and its function is to scan the existing sequence for previous examples of that token that it's attending to. Then it needs to find the token that in that survey came after the token, which in our case will be end, and then forecast that that will happen once more to give us explained. Obviously you need to be great at recall to do this. Especially if the sequence involves thousands, hundreds of thousands or even millions of tokens. Other architectures fall apart as the sequence length gets longer than that found in tr, but not Mamba. Look at the top line accuracy, staying at one even up to a million tokens. And remember that architectures like Mamba need not necessarily hunt alone. Take this announcement of striped hyena from together AI. They showed that we need not necessarily choose with a hybrid architecture involving attention performing well. But there's one more thing before we put the mamba snake back into the basket on the left you can see its superior performance at great apes DNA classification. In other words, deciding whether a sequence of DNA was human, chimpanzee, gorilla, orangutan or bonobo. Notice that it's on the longest sequence lengths that its performance starts to really shine. This task, by the way, they made artificially hard for themselves because it was originally about distinguishing between a human lemur, mouse, pig and hippo. But you could easily imagine other use cases like healthcare. You could rapidly analyze a patient's genetic data and come up with personalized medical treatment for them. More speculatively, you could imagine a chatbot that remember a conversation you had months or years ago. And then of course I'll leave it to you to think of all the other long sequences out there, like stock market data or weather data, and as I mentioned before, video data from those long Video explainers that annoy everyone. But wait, is this video becoming one of those? I really hope not. So I'm going to move on to the next reason that AI is not slowing down anytime soon. That is inference, time compute, or the ability of the model, as Arthur Mensch said, to decide how much compute to allocate to certain problems. Now, I touched on this in my Q Star video. But here's Lucas Kaiser, one of the authors of the transformer architecture and indeed a senior figure for OpenAI for reasoning.

You also need these chains of thought that like you need to give the model the ability to think longer than it has layers. But it can be combined with multimodal. Especially when you have a nowadays models, they can generate. You say, okay, how does it look when a boy kicks a ball? And you can generate a few frames of the video. And now more and more there are models that will generate the whole video. And then there start to be models of the world that say, well, if you drive a car like this, how will the street look? How will the people look? What will happen? Will you crash into something? So in the future the models will have this knowledge of the world and this generation, which we call chain of thought and text. But multimodality, this means just it's a chain of frames of what's going to happen to the world, which is basically how we sometimes think. So it will be multimodality and this ability to generate sequences of things before you give an answer that will resemble much more what we call reasoning.

For more on that, check out my Q Star video. But here's another taster. This is from Gnome Brown, also of OpenAI. He admitted that letting models think for longer would occasionally have drawbacks. Inference at times may be a thousand x slower and more costly. But he said, what inference cost would we pay for a new cancer drug or for a proof of the Riemann hypothesis? AlphaCode 2, which I've also done a video on, gives us a foretaste of the kind of results that we might expect. And remember, this is separate from data quality or dynamic new architectures. But we can't mention AlphaCode 2 and inference time compute without also mentioning let's verify step by step, AKA process based verification. But I know what you're wondering. What is this graph and where does it come from? Well, it comes from a fascinating new paper entitled AI Capabilities can be significantly improved without Expensive retraining. In a way, this paper sums up the message of this video. We are not even close to being done with the exponential gains in AI and the way that this paper measured it was fascinating. It basically asked how much extra computing power would we have to provide to get the equivalent gain that these methods provide. As you can see, the methods are quite diverse and almost all of them have been covered before on this channel. Things like prompting and scaffolding, a bit like SMART GPT tool use or indeed data quality as we saw with ORCA and self consistency. If you're not familiar with either ORCA or self consistency and majority voting, check out the videos that I've done before linked in the description. The X axis, by the way, is the one time compute cost that these methods entail. Yes, some of them go up to 1% or even 10% as a fraction of the training compute used to create the models. But look at the returns on the Y axis. Having a verifier check the steps of a model does cost 0.001% as a fraction of the training compute, but the compute equivalent gain is around 9. In other words, you would have had to use 9 times as much compute on the base model to achieve similar results. And yes, many of these methods can indeed be combined. Indeed, that's what SmartGPT was all about. It was combining chain of thought, few shotting and majority voting. In 2024 we may see the Phi 2 data quality approach combined with say the MAMBA architecture. And all of this is before we scale the models up. As the paper says, gains from enhancements and gains from scaling might interact and compound in non trivial ways. And they give the example that chain of thought prompting didn't even work on smaller models. I remember saying toward the end of my SmartGPT video in August that we need to find out the ceiling of the models, not just the floor. And the authors concur saying researchers could also study whether there is a ceiling to the total improvement you can get from post training enhancements that will enable the AGI labs to better predict how much more capable their model might become in the future. Now I know what some veteran researchers will be thinking while looking at these charts. Surely it depends on the task and the benchmark and the models. And all of these numbers must be very approximate. And of course you are right. But I have one more critique in addition to that and that is the third or is it fourth? To be honest, I've lost count reason why AI is going to continue to improve dramatically and that is prompt optimization. I spoke to Tim Rocktaschel of Google DeepMind about this for Patreon, but this is what I mean in A nutshell Language models can optimize their own prompts. There are many techniques for doing this, but the manual methods we're coming up with, the heuristics like this is important for my career and my grandma wants me to do this are our manual feeble approaches. Once we deploy LLMs to help us optimize the prompts going into LLMs, we might see dramatically better performance. Indeed we already have for things like high school mathematics and movie recommendations. Anyway, this is the simplified version. Do check out that video. But the point is this. Even if we weren't going for dramatically better data quality, dynamic new architectures and getting models to reason and think for longer, then prompt optimization would allow us to squeeze out significantly better results even from existing models. And notice I've got through the whole video without yet mentioning of course scaling the models up to 10 trillion parameters or indeed 100 trillion parameters as promised by etched AI. I've got much more information about them coming soon. But let's say you're someone who doesn't care about stats, benchmarks or indeed AGI countdowns. Maybe you're someone who is skeptical about tweets like this from an OpenAI employee who said brace yourselves, AGI is coming. Well, even for you, 2024 looks set to be a dramatic year. These outputs from the Walt team at BE Google may be low resolution, but they're quite high consistency. I use Pika labs and Runway ML2 and the progress season on season is quite something to watch. And that would be a prediction I'd make for before the end of 2024. I think we will get a 3 to 5 second photorealistic text to video output that could fool most humans. Now you might say, oh I'd never be fooled, but let's test you on that with text to image. Which of these then is the real Roman arch? One of them is From Midjourney version 6 upscaled with magnific and the other is a real arch. The answer is that the one on the left is the real Roman arch. Now in my Christmas video, while misspelling some prompts, I did show off the style raw input. This is for anyone using Midjourney version 6. But since then from Reddit I found an even better tip. Tip. Use the phrase phone photo. You can get as you can see strikingly realistic images. And of course you can further upscale any of these too. You can definitely get some quite interesting results this way. And at this point I want to show you this quite fascinating prediction made a hundred years ago. This is what the cartoonist Harold Tucker Webster thought the world of drawing would be like in 2023. Now, yes, we're a day into 2024, but I still think this is interesting. Notice the drawing is being done autom and the caption is in the year 2023 when all our work is done by electricity. He called it the Cartoon Dynamo. We call it Midjourney version 6. But tomato tomato. It's time now to draw the video to an end. But I'm going to end where I started. And no, I don't just mean this tweet from Jim Fan, I also mean the words I used in the introduction and here via 11Labs is me in quotes AI explained Philip, who is a real person and not GPT 5 or 6 as some assume. But here's the AI version of my voice delivering the intro. I hope everyone watching had an excellent 2023 and is looking to get 2024 off to a rambunctious start. This video, though, has been made to show that we are on the steep part of the exponential and will be for a while yet. Just for fun for all of my legendary supporters on Patreon, I'm going to process this entire video so you can hear how good AI is getting at imitating the human voice. Soon it may be impossible to tell who's human and who' using audio alone, and then soon thereafter even video. But I know you guys know that I'm real. So thank you so much for watching all the way to the end. And as always, have a wonderful day and a wonderful 2024.