DeepSeek, China, OpenAI, NVIDIA, xAI, TSMC, Stargate, and AI Megaclusters | Lex Fridman Podcast #459

The following is a conversation with Dylan Patel and Nathan Lampert. Dylan runs Semianalysis, a well respected research and analysis company that specializes in semiconductors, GPUs, CPUs, and AI hardware in general. Nathan is a research scientist at the Allen Institute for AI and is the author of the amazing blog on AI called Interconnects. They are both highly respected, read and listened to by the experts, researchers and engineers in the field of AI. And personally I'm just a fan of the two of them. So I used the Deep Seq moment that shook the AI world a bit as an opportunity to sit down with them and lay it all out. From Deep Seq, OpenAI, Google, XAI, Metaanthropic, to Nvidia and TSMC and to US China, Taiwan relations and everything else that is happening at the cutting edge of AI, this conversation is a deep dive into many critical aspects of the AI industry. While it does get super technical, we tried to make sure that it's still accessible to folks outside of the AI field by defining terms, stating important concepts, explicitly spelling out acronyms, and and in general, always moving across the several layers of abstraction and levels of detail. There is a lot of hype in the media about what AI is and isn't. The purpose of this podcast in part is to cut through the hype, through the bullshit and the low resolution analysis, and to discuss in detail how stuff works and what the implications are. Let me also, if I may comment on the new OpenAI O3 mini reasoning model, the release of which we were anticipating during the conversation, and it did indeed come out right after its capabilities and cost are on par with our expectations. As we stated, OpenAI03 mini is indeed a great model, but it should be stated that Deepseek R1 has similar performance on benchmarks, is still cheaper, and it reveals its chain of thought reasoning which O3 mini does not. It only shows a summary of the reasoning. Plus R1 is open wait and O3 mini is not. By the way, I got a chance to play with O3 mini and anecdotal vibe. Check wise I felt that O3 mini, specifically O3 mini high is better than R1. Still, for me personally I find that Claude Sonnet 3.5 is the best model for programming except for tricky cases where I will use 01 Pro to brainstorm. Either way, many more better AI models will come, including reasoning models both from American and Chinese companies. They will continue to shift the cost curve, but the deep seek moment is indeed real. I think it will still be remembered five years from now as a pivotal event in tech history, due in part to the geopolitical implications, but for other reasons too. As we discuss in detail from many perspectives in this conversation, this is the Lex Friedman podcast. To support it, please check out our sponsors in the description. And now, dear friends, here's Dylan Patel and Nathan Lambert. A lot of people are curious to understand China's Deep Sea Ki models. So let's lay it out. Nathan, can you describe what Deepseak V3 and Deepseek R1 are, how they work, how they're trained? Let's look at the big picture and then we'll zoom in on the details.

Yeah, so Deepseek v3 is a new mixture of experts Transformer language model from DeepSeek who is based in China. They have some new specifics in the model that we'll get into. Largely, this is a open weight model and it's a instruction model like what you would use in ChatGPT. They also released what is called the base model, which is before these techniques of post training. Most people use instruction models today and those are what's served in all sorts of applications. This was released on, I believe, December 26th or that week. And then weeks later, on January 20th, Deep Seq released Deep Seq R1, which is a reasoning model, which really accelerated a lot of this discussion. This reasoning model has a lot of overlapping training steps to deep seq v v3 and it's confusing that you have a base model called V3 that you do something to to get a chat model and then you do some different things to get a reasoning model. I think a lot of the AI industry is going through this challenge of communications right now where OpenAI makes fun of their own naming schemes. They have GPT4O, they have OpenAI01 and there's a lot of types of models. So we're going to break down what each of them are. There's a lot of technical specifics on training and go from high level to specific and kind of go through each of them.

There's so many places we can go here, but maybe let's go to open weights first. What does it mean for a model to be open weights and what are the different flavors of open source in general?

Yeah, so this discussion has been going on for a long time in AI. It became more important since ChatGPT or more focal since ChatGPT at the end of 2022. Open weights is the accepted term for when model weights of a language model are available on the Internet for people to download those Weights can have different licenses, which is effectively the terms by which you can use the model. There are licenses that come from history and open source software. There are licenses that are designed by companies, specifically all of Llama, Deepseek, Quen, Mistral. These popular names in open weight models have some of their own licenses. It's complicated because not all the same models have the same terms. The big debate is on what makes a model open weight. Why are we saying this term? It's kind of a mouthful. It sounds close to open source, but it's not the same. There's still a lot of debate on the definition and soul of open source AI. Open source software has a rich history on freedom to modify, freedom to take on your own freedom from any restrictions on how you would use the software. And what that means for AI is still being defined. So for what I do, I work at the Allen Institute for AI. We're a nonprofit. We want to make AI open for everybody. And we try to lead on what we think is truly open source. There's not full agreement in the community, but for us that means releasing the training data, releasing the training code, and then also having open weights like this. And we'll get into the details of the models. And again and again, as we try to get deeper into how the models were trained, we will say things like the data processing, data filtering, data quality is the number one determinant of the model quality. And then a lot of the training code is the determinant on how long it takes to train and how fast your experimentation is. So without fully open source models, where you have access to this data, it is hard to know or it's harder to replicate. So we'll get into cost numbers for deep seq v3 on mostly GPU hours and how much you could pay to rent those yourselves. But without the data, the replication cost is going to be far, far higher. And same goes for the code.

We should also say that this is probably one of the more open models out of the frontier models. So like in this full spectrum where probably the fullest open source, like you said, open code, open data, open weights, this is not open code, this is probably not open data, and this is open weights and the licensing is MIT license or it's. I mean, there's some nuance in the different models, but it's towards the free in terms of the open source movement. These are the kind of the good guys.

Yeah, Deep Seq is doing fantastic work for disseminating understanding of AI. Their papers are extremely detailed in what they do. And for other teams around the world, they're very actionable in terms of improving your own training techniques. And we'll talk about licenses more. The Deep Seq R1 model has a very permissive license. It's called the MIT license. That effectively means there's no downstream restrictions on commercial use, there's no use case restrictions. You can use the outputs from the models to create synthetic data. And this is all fantastic. I think the closest peer is something like Llama, where you have the weights and you have a technical report. And the technical report is very good for Llama. One of the most read PDFs of the year last year is the Llama 3 paper. But in some ways it's slightly less actionable. It has less details on the training specifics, less plots and so on. And the Llama 3 license is more restrictive than MIT. And then between the Deep Sea Custom license and the LLAMA license, we can get into this whole rabbit hole. I think we'll make sure we want to go down the license rabbit hole before we do specifics.

Yeah.

And I mean, so it should be stated that one of the implications of Deep Secret puts pressure on LLAMA and Everybody else on OpenAI to push towards open source. And that's the other side of open source that you mentioned is how much is published in detail about it. So how open are you with the sort of the insights behind the code? So like how good is the technical reports? Are they hand wavy or is there actual details in there? And that's one of the things that Deep SEQ did well, as they publish a lot of the details.

Yeah, especially in the deep seq v3, which is their pre training paper, they were very clear that they are doing interventions on the technical stack that go at many different levels. For example, to get highly efficient training, they're making modifications at or below the CUDA layer for Nvidia chips. I have never worked there myself and there are a few people in the world that do that very well and some of them are at Deepseek and these types of people are at Deep Sea and leading American Frontier Labs. But there are not many places to.

Help people understand the other implication of open weights. Just, you know, there's a topic we'll return to often here. So there's a fear that China, the nation might have interest in stealing American data, violating privacy of American citizens. What can we say about open weights to help us understand what the weights are able to do in terms of stealing people's data?

Yeah, so these weights that you can download from hugging face or other platforms are very big matrices of numbers. You can download them to a computer in your own house that has no Internet and you can run this model and you're totally in control of your data. That is something that is different than how a lot of language model usage is actually done today, which is mostly through APIs where you send your prompt to GPUs run by certain companies. And these companies will have different distributions and policies on how your data is stored, if it is used to train future models, where it is stored, if it is encrypted, and so on. So the open weights are you have your fate of data in your own hands and that is something that is deeply connected to the soul of open source.

So it's not the model that steals your data, it's Clover's hosting the model, which could be China if you're using the Deep Sea G app. Or it could be perplexity. You know, you're trusting them with your data or OpenAI, you're trusting them with your data. And some of these are American companies, some of these are Chinese companies. But the model itself is not doing the stealing, it's the host. All right, so back to the basics. What's the difference between deep seq v3 and deep seq r1? Can we try to like lay out the confusion potential?

Yes. So for one, I'm very understanding of many people being confused by these two model names. So I would say the best way to think about this is that when training a language model you have what is called pre training, which is when you're predicting the large amounts of mostly Internet text, you're trying to predict the next token. And what to know about these new Deep SEQ models is that they do this Internet large scale pre training once to get what is called deep seq v3 base. This is a base model. It's just going to finish your sentences for you. It's going to be harder to work with than ChatGPT. And then what DeepSeek did is they've done two different post training regimes to make the models have specific desirable behaviors. So what is the more normal model in terms of the last few years of AI? An instruct model, a chat model, an aligned model, a helpful model. There are many ways to describe this is more standard post training. So this is things like instruction, tuning, reinforcement, learning from human feedback. We'll get into some of these words and this is what they did to create the deep seq v3 model. This was the first model to be released and it is very high performant, it's competitive with GPT4, llama 405B, so on. And then when this release was happening, we don't know their exact timeline or soon after they were finishing the training of a different training process from the same next token prediction base model that I talked about, which is when this new reasoning training that people have heard about comes in in order to create the model that is called deep seq R1. The R through this conversation is good for grounding for reasoning. And the name is also similar to OpenAI's O1, which is the other reasoning model that people have heard about. And we'll have to break down the training for R1 in more detail because for one we have a paper detailing it, but also it is a far newer set of techniques for the AI community, so it's a much more rapidly evolving area of research.

Maybe we should also say the big two categories of training of pre training and post training, these umbrella terms that people use. So what is pre training and what is post training and what are the different flavors of things underneath? Post training umbrella?

Yeah, so pre training, I'm using some of the same words to really get the message across is you're doing what is called autoregressive prediction to predict the next token in a series of documents. This is done over. Standard practice is trillions of tokens. So this is a ton of data that is mostly scraped from the web. In some of deepseek's earlier papers they talk about their training data being distilled for math. I shouldn't use this word yet, but taken from Common Crawl and that's a public access that anyone listening to this could go download data from the Common Crawl website. This is a crawler that is maintained publicly. Yes, Other tech companies eventually shift to their own crawler and deepsea likely has done this as well as most frontier labs do. But this sort of data is something that people can get started with and you're just predicting text in a series of documents. This can be scaled to be very efficient. And there's a lot of numbers that are thrown around in AI training, like how many floating point operations or flops are used. And then you can also look at how many hours of these GPUs that are used. And it's largely one loss function taken to a very large amount of compute usage. You just set up really efficient systems and then at the end of that you have this base model. And pre training is where there is a lot more of complexity in terms of how the process is emerging or evolving and the different types of training losses that you will use. I think this is a lot of techniques grounded in the natural language processing literature. The oldest technique which is still used today is something called instruction tuning or also known as supervised fine tuning. These acronyms will be IFT or sft. People really go back and forth throughout them and I will probably do the same. Which is where you add this formatting to the model where it knows to take a question that is like explain the history of the Roman Empire to me or a sort of question you'll see on Reddit or Stack Overflow and then the model will respond in an information dense but presentable manner. The core of that formatting is in this instruction tuning phase. And then there's two other categories of loss functions that are being used today. One I will classify as preference fine tuning. Preference fine tuning is a generalized term for what came out of reinforcement learning from human feedback, which is RLHF. This reinforcement learning from human feedback is credited as the technique that helped ChatGPT break through. It is a technique to make the responses that are nicely formatted like these Reddit answers, more in tune with what a human would like to read. This is done by collecting pairwise preferences for from actual humans out in the world to start. And now AIs are also labeling this data and we'll get into those trade offs and you have this kind of contrastive loss function between a good answer and a bad answer. And the model learns to pick up these trends. There's different implementation ways. You have things called reward models. You could have direct alignment algorithms. There's a lot of really specific things you can do. But all of this is about fine tuning to human preferences. And the final stage is much newer and we'll link to what is done in R1. And these reasoning models is I think OpenAI's name for this. They had this new API in the fall which they called the Reinforcement Fine Tuning API. This is the idea that you use the techniques of reinforcement learning, which is a whole framework of AI. There's a deep literature here to summarize. It's often known as trial and error learning where. Or the subfield of AI where you're trying to make sequential decisions in a certain potentially noisy environment. There's a lot of ways we could go down that. But fine tuning language models where they can generate an answer and then you check to see if the answer matches the true solution for math or code. You have an exactly correct answer for Math, you can have unit tests for code. And what we're doing is we are checking the language model's work and we're giving it multiple opportunities on the same questions to see if it is right. And if you keep doing this, the models can learn to improve in verifiable domains to a great extent. It works really well. It's a newer technique in the academic literature. It's been used at Frontier Labs in the US that don't share every detail for multiple years. So this is the idea of using reinforcement learning with language models. And it has been taking off, especially in this Deep SEQ moment.

And we should say that there's a lot of exciting stuff going on on the again across the stack. But the post training, probably this year there's going to be a lot of interesting developments in the post training. We'll, we'll talk about it. I almost forgot to talk about the the difference between deep seq v3 and R1 on the user experience side. So forget the technical stuff, forget all that. Just people that don't know anything about AI, they show up like what's the actual experience? What's the use case for each one when they actually like type and talk to it? What is each good at? That kind of thing.

So let's start with deep seqv3 again. It's what more people would have tried. Something like it, you ask it a question, it'll start generating tokens very fast. And those tokens will look like a very human legible answer. It'll be some sort of markdown list. It might have formatting to help you draw to the core details in the answer. And it'll generate tens to hundreds of tokens. A token is normally a word for common words or a subword part in a longer word. And it'll look like a very high quality Reddit or stack overflow answer. These models are really getting good at doing these across a wide variety of domains. Even things that if you're an expert, things that are close to the fringe of knowledge, they will still be fairly good at. I think cutting edge AI topics that I do research on. These models are capable for study aid and they're regularly updated. Where this changes is with the deep seq r1. What is called these reasoning models is when you see tokens coming from these models to start it will be a large chain of thought process. We'll get back to chain of thought in a second, which looks like a lot of tokens where the model is explaining the problem. The model will often Break down the problem and be like, okay, they asked me for this, let's break down the problem. I'm going to need to do this. And you'll see all of this generating from the model. It'll come very fast. In most user experiences, these APIs are very fast. So you'll see a lot of tokens, a lot of words show up really fast. It'll keep flowing on the screen and this is all the reasoning process. And then eventually the model will change its tone in R1 and it'll write the answer where it summarizes its reasoning process and writes a similar answer to the first types of model. But in deepseek's case, which is part of why this was so popular even outside the AI community, is that you can see how the language model is breaking down problems and then you get this answer. On a technical side, they train the model to do this specifically where they have a section which is reasoning and then it generates a special token which is probably hidden from the user most of the time, which says, okay, I'm starting the answer. So the model is trained to do this two stage process on its own. If you use a similar model in say, OpenAI, OpenAI's user interface is trying to summarize this process for you nicely by kind of showing the sections that the model is doing and it'll kind of click through, it'll say breaking down the problem, making X calculation, cleaning the result and then the answer will come for something like OpenAI, maybe it's useful.

Here to go through like an example of a deep seq R1 reasoning.

Yeah, so if you're looking at the screen here, what you'll see is a screenshot of the Deep SEQ chat app. And at the top is thought for 1517 seconds with the drop down arrow underneath that. If we were in an app that we were running, the drop down arrow would have the reasoning.

So in this case the specific question, which, you know, I'm philosophically pothead inclined, so this is asking DeepSeekar1 for one truly novel insight about humans and it reveals the reasoning. And basically the truly novel aspect is what's pushing the reasoning to constantly sort of the model asking itself, is this truly novel? So it's actually challenging itself to be more novel, more counterintuitive, more. Less cringe, I suppose. So some of the reasoning says this is just snapshots. Alternatively, humans have a unique meta emotion where they feel emotions about their own emotions. For example, feeling guilty about being angry. This recursive Emotional layering creates complex motivational drives that don't exist in other animals. The insight is that human emotions are nested. So it's like it's reasoning through how humans feel emotions, it's reasoning about meta emotions.

Gonna have pages and pages of this. It's almost too much to actually read, but it's nice to skim as it's coming.

It's a stream of, it's a James Joyce like stream of consciousness. And then it goes, wait. The user wants something that's not seen anywhere else. Let me dig deeper and consider the human ability to hold contradictory beliefs simultaneously. Cognitive dissonance is known, but perhaps the function to allow flexible adaptation, so on and so forth. I mean that really captures the public imagination that holy shit, this isn't intelligent. Almost like an inkling of sentience because you're thinking through, you're self reflecting, you're deliberating. And the final result of that after 157 seconds is humans instinctively convert selfish desires into cooperative systems by collectively pretending abstract rules, rules, money, laws, rights are real. These shared hallucinations act as quote games where competition is secretly redirected to benefit the group, turning conflict into society's fuel. Pretty profound. I mean, you know, this is a.

Potential digression, but a lot of people have found that these reasoning models can sometimes produce much more eloquent text. That is a at least interesting example. I think depending on how open minded you are, you find language models interesting or not. And there's a spectrum from there.

Well, I mean it's some of the, we'll talk about different benchmarks and so on, but some is just a vibe like that in itself is a, let's say, quote, fire tweet. Yeah, if I trying to produce something, something where people are like oh shit, okay, so that's chain of thought. We'll probably return to it more. How were they able to achieve such low cost on the training and the inference? Maybe you could talk the training first.

Yeah, so there's two main techniques that they implemented that are probably the majority of their efficiency and then there's a lot of implementation details that maybe we'll gloss over or get into later that sort of contribute to it. But those two main things are. One is they went to a mixture of experts model which we'll define in a second. And then the other thing is that they invented this new technique called MLA latent attention. Both of these are big deals. Mixture of experts is something that's been in the literature for a handful of years and OpenAI with GPT4 was the first one to productize a mixture of experts model. And what this means is when you look at the common models around that most people have been able to interact with that are open, right? Think llama. Llama is a dense model. That is every single parameter or neuron is activated as you're going through the model for every single token you generate.

Right?

Now, with a mixture of experts model, you don't do that, right? How does a human actually work? Right? Is like, oh, well, my visual cortex is active when I'm thinking about vision tasks or other things, right? My amygdala is when I'm scared, right? These different aspects of your brain are focused on different things. A mixture of experts models attempts to approximate this to some extent. It's nowhere close to what a brain architecture is. But different portions of the model activate, right? You'll have a set number of experts in the model and a set number that are activated each time. And this dramatically reduces both your training and inference costs. Because now if you think about the parameter count as the sort of total embedding space for all of this knowledge that you're compressing down during training, when you're embedding this data in, instead of having to activate every single parameter every single time you're training or running inference, now you can just activate a subset of and the model will learn which expert to route to for different tasks. And so this is a humongous innovation in terms of, hey, I can continue to grow the total embedding space of parameters. And so deep SEQ's model is 600 something billion parameters, right? Relative to llama 405B, it's 405 billion parameters, right? Lama relative to llama 70B, it's 70 billion parameters, right? So this model technically has more embedding space for information, right? To compress all of the world's knowledge that's on the Internet down. But at the same time, it is only activating around 37 billion of the parameters. So only 37 billion of these parameters actually need to be computed every single time you're training data or inferencing data out of it. And so versus again, the llama model, 70 billion parameters must be activated or 405 billion parameters must be activated. So you've dramatically reduced your compute cost when you're doing training and inference with this mixture of experts architecture.

So we break down where it actually applies and go into the transformer. Is that useful?

Let's go, let's go into the transformer.

The transformer is a thing that is talked about a lot and we will not cover every detail. Essentially the transformer is built on repeated blocks of this attention mechanism and then a traditional dense, fully connected multilayer perceptron, whatever word you want to use for your normal neural network. And you alternate these blocks. There's other details and where mixture of experts is applied is that this dense model, the dense model holds most of the weights if you count them in a transformer model. So you can get really big gains from this mixture of experts on parameter efficiency at training and inference because you get this efficiency by not activating all of these parameters.

We should also say that a transformer is a giant neural network. And then there's for 15 years now this what's called the deep learning revolution. Networks gotten larger and larger and at a certain point the scaling laws appeared where people realized this is a scaling law shirt by the way, representing scaling laws. Where it became more and more formalized that bigger is better across multiple dimensions of what bigger means. But these are all sort of neural networks we're talking about. And we're talking about different architectures of how construct to construct these neural networks such that the training and the inference on them is super efficient.

Yeah, every different type of model has a different scaling law for it which is effectively for how much compute you put in. The architecture will get to different levels of performance at test tasks. A mixture of experts is one of the ones at training time. Even if you don't consider the inference benefits which are also big at training time. Your efficiency with your GPUs is dramatically improved by using this architecture if it is well implemented. So you can get effectively the same performance model and evaluation scores with numbers like 30% less compute. I think there's going to be a wide variation depending on your implementation details and stuff. But it is just important to realize that this type of technical innovation is something that gives huge gains. And I expect most companies that are serving their models to move to this mixture of experts implementation. Historically, the reason why not everyone might do it is because it's an implementation complexity, especially when doing these big models. So this is one of the things that Deepseek gets credit for is they do this extremely well. They do a mixture of experts extremely well. This architecture for what is called deep SEQ MOE MOE is the shortened version of mixture of experts is multiple papers old. This part of their training infrastructure is not new to these models alone. And same goes for what Dylan mentioned with multi head latent attention. This is all about reducing memory Usage during inference and same things during training by using some fancy low rank approximation math. If you get into the details with this latent attention, it's one of those things I look at and it's like, okay, they're doing really complex implementations because there's other parts of the language model, such as embeddings that are used to extend the context length. The common one that Deep SEQ used is rotary positional embeddings, which is called rope. And if you want to use rope with a normal moe, it's kind of a sequential thing. You take two of the attention matrices and you rotate them by a complex value rotation, which is a matrix multiplication. With Deep seq's mla, with this new attention architecture, they need to do some clever things because they're not set up the same and it just makes the implementation complexity much higher. So they're managing all of these things and these are probably the sort of things that OpenAI, these closed labs are doing. We don't know if they're doing the exact same techniques, but they actually shared them with the world, which is really nice to feel like. This is the cutting edge of efficient language model training.

And some of this requires low level engineering. Just it's a giant mess and trickery. So as I understand, they went below CUDA, so they go super low. Programming of GPUs effectively.

Nvidia builds this library called Nickel, right? In which when you're training a model, you have all these communications between every single layer of the model and you may have over 100 layers.

What does nickel stand for?

It's NCCL Nvidia Communications collectives Library.

Nice.

When you're training a model, you're going to have all these all reduces and all gathers between each layer, between the multi layer perceptron or feed forward network and the attention mechanism, you'll have. You'll have basically the model synchronize, right? Or you'll have allreducer and all gather. And this is a communication between all the GPUs in the network, whether it's in training or inference. So Nvidia has a standard library. This is one of the reasons why it's really difficult to use anyone else's hardware for training is because no one's really built a standard communications library. And Nvidia has done this at a sort of a higher level, right, deep seq, because they have certain limitations around the GPUs that they have access to. The interconnects are limited to some extent by the restrictions of the GPUs that were shipped into China legally, not the ones that are smuggled, but legally shipped in that they used to train this model. They had to figure out how to get efficiencies right. And one of those things is that instead of just calling the Nvidia library Nickel, right, they instead created their, they scheduled their own communications which, which the lab, some of the labs do, right. Emeda Talked about in Llama 3 how they made their own custom version of Nickel. They didn't talk about the implementation details. This is some of what they did. Probably not as well as, maybe not as well as Deepseek, because Deepseek, necessity is the mother of innovation and they had to do this. Whereas in the case OpenAI has people that do this sort of stuff, anthropic, et cetera. But Deepseek certainly did it publicly and they may have done it even better because they were gimped on a certain aspect of the chips that, that they have access to. And so they scheduled communications, you know, by scheduling specific sms. SMS you could think of as like the core on a gpu, right? So there's hundreds of cores or there's, you know, a bit over 100 cores SMS on a GPU. And they were specifically scheduling, hey, which ones are running the model, which ones are doing all reduce, which ones are doing all gather, right? And they would flip back and forth between them. And this requires extremely low level programming.

This is what Nickel does automatically. Or other Nvidia libraries handle this automatically usually.

Yeah, exactly. And so technically they're using, you know, ptx, which is like, sort of like you could think of it as like an assembly type language. It's not exactly that or instruction set, right? Like coding directly to assembler instruction set. It's not exactly that, but that's still part of technically cuda. But it's like, do I want to write in Python, you know, Pytorch equivalent and call Nvidia libraries? Do I want to go down to the C level, Encode even lower level, or do I want to go all the way down to the assembly or ISO level? And there are cases where you go all the way down there at the very big labs, but most companies just do not do that because it's a waste of time and the efficiency gains you get are not worth it. But Deep seq's implementation is so complex, right? Especially with their mixture of experts. People have done mixture of experts, but they're generally 8, 16 experts. Right. And they activate too. So one of the words used like to use as like sparsity factor, right? Or usage. Right. So, so you might have four, you know, one fourth of your model activate, right? And, and, and that's what Mistral's Mixtral model, right. Their, their model that really catapulted them to like, oh my God, they're really, really good. OpenAI has also had models that are MOE and, and, and, and so have all the other labs that are major closed. But what deepseek did that maybe only the leading labs have only just started recently doing is have such a high sparsity factor, right. It's not 1/4 of the model, right? Two out of eight experts activating every time you go through the model, it's eight out of 256.

And there's different implementations for mixture of experts where you can have some of these experts that are always activated, which this just looks like a small neural network and then all the tokens go through that and then they also go through some that are selected by this routing mechanism. And one of the innovations in Deep seq's architecture is that they change the routing mechanism. In mixture of expert models, there's something called an auxiliary loss, which effectively means during training you want to make sure that all of these experts are used across the tasks that the model sees why there can be failures. And mixture of experts is that when you're doing this training, the one objective is token prediction accuracy. And if you just let training go with a mixture of expert model on your own, it can be that the model learns to only use a subset of the experts. And in the MOE literature there's something called the auxiliary loss which helps balance them. But if you think about the loss functions of deep learning, this even connects to the bitter lesson is that you want to have the minimum inductive bias in your model to let the model learn maximally. And this auxiliary loss, this balancing across experts could be seen as in tension with the prediction accuracy of the tokens. So we don't know the exact extent that the deep SEQMOE change, which is instead of doing an auxiliary loss, they have an extra parameter in their routing which after the batches they update this parameter to make sure that the next batches all have a similar use of experts. And this type of change can be big, it can be small, but they add up over time. And this is the sort of thing that just points to them innovating. And I'm sure all the labs that are training big moes are looking at this sort of things, which is getting away from the auxiliary loss. Some of them might already use it, but you keep accumulating gains. And we'll talk about the philosophy of training and how you organize these organizations. And a lot of it is just compounding small improvements over time in your data, in your architecture and your post training and how they integrate with each other. DeepSeq does the same thing and some of them are shared or a lot. We have to take them on face value that they share their most important details. I mean the architecture and the weights are out there. So we're seeing what they're doing and it adds up.

Going back to sort of the efficiency and complexity point, right? It's 32 versus 4. Right. For like mixed draw and other MOE models that have been publicly released. So this ratio is extremely high. And sort of what Nathan was getting at there was when you have such a different level of sparsity, you can't just have every GPU have the entire model, right? The model's too big, there's too much complexity there. So you have to split up the model with different types of parallelism. And so you might have different experts on different GPU nodes. But now what happens when this set of data that you get, hey, all of it looks like this one way and all of it should route to one part of my model. So when all of it routes to one part of the model, then you can have this overloading of a certain set of the GPU resources or a certain set of the GPUs and then the rest of the the training network sits idle because all of the tokens are just routing to that. So this is the biggest complexity, One of the big complexities with running a very, you know, sparse mixture of experts model, I. E. This 32 ratio versus this 4 ratio is that you end up with so many of the experts just sitting there idle. So how do I load balance between them, how do I schedule the communications between them? This is a lot of the like extremely low level detailed work that they figured out in the public first and potentially like second or third in the world and maybe even first in some cases.

What lesson do you in the direction of the bitter lesson do you take from all of this? Where is this going to be the direction where a lot of the gain is going to be? Which is this kind of low level optimization? Or is this a short term thing where the biggest gains will be more on the algorithmic high level side of like post training? Is this like a short term leap because they've figured out like a hack? Because constraints necessity is the Mother of invention or is there still a lot of gains?

I think we should summarize what the bitter lesson actually is about. Is that the bitter lesson essentially if you paraphrase it, is that the types of training that will win out in deep learning as we go are those methods that which are scalable in learning. And search is what it calls out. And this scale word gets a lot of attention in this. The interpretation that I use is effectively to avoid adding human priors to your learning process. And if you read the original essay, this is what it talks about is how researchers will try to come up with clever solutions to their specific problem that might get them small gains in the short term while simply enabling these deep learning systems to work efficiently and for these bigger problems in the long term might be more likely to scale and continue to drive success. And therefore we were talking about relatively small implementation changes to the mixture of experts model. And therefore it's like okay, we will need a few more years to know if one of these were actually really crucial to the bitter lesson. But the bitter lesson is really this long term arc of how simplicity can often win. And there's a lot of sayings in the industry like the models just want to learn. You have to give them the simple loss landscape where you put compute through the model and they will learn. And getting barriers out of the way.

That's where the power of something like nickel comes in. Where standardized code that could be used by a lot of people to create sort of simple innovations that can scale. Which is why the hacks. I imagine the code base for DeepSeek is probably a giant mess.

I'm sure they have DeepSeq definitely has code bases that are extremely messy where they're testing these new ideas. Multi head latent attention probably could start in something like a jupyter notebook where somebody tries something on a few GPUs and that is really messy. But the stuff that trains DeepSeq v3 and DeepSeq R1, those libraries, if you were to present them to us, I would guess our extremely high quality code.

High quality readable code.

I think there is one aspect to note though, right Is that there is the general ability for that to transfer across different types of runs, right? You may make really, really high quality code for one specific model architecture at one size and then that is not transferable to. Hey, when I make this architecture tweak, everything's broken again. Right? Like that's, that's something that could be, you know, with their, with their specific low level coding of like Scheduling SMS is specific to this model architecture and size, right? And whereas like Nvidia's Collective's library is more like, hey, it'll work for anything, right? You want to do an all reduce, great. I don't care what your model architecture is, it'll work. And you're giving up a lot of performance when you do that in many cases. But it's worthwhile for them to do the specific optimization for the specific run given the constraints that they have regarding compute.

I wonder how stressful it is to these frontier models. Initiate training to have the code to push the button that you're now spending a large amount of money and time to train this. There must be a lot of innovation on the debugging stage of making sure there's no issues, that you're monitoring and visualizing every aspect of the training, all that kind of stuff.

When people are training, they have all these various dashboards, but like the most simple one is your loss, right? And it continues to go down. But in reality, especially with more complicated stuff like MOE, the biggest problem with it, or FP8 training, which is another innovation, going to a lower precision number format that is less accurate, is that you end up with loss spikes, right? And no one knows why the lost spike happened.

Some of them you do, some of them you do, some of them are bad data. Can I give Aitoo's example of what blew up? Our earlier models is a subreddit called Microwave Gang. We love to shout this out. It's a real thing you can pull up Microwave Gang. Essentially it's a subreddit where everybody makes posts that are just the letter M. So it's like M. So there's extremely long sequences of the letter mm.