Meta’s Joe Spisak on Llama 3.1 405B and the Democratization of Frontier Models | Training Data

You know, if I was a founder right now, I would absolutely adopt open source. It forces me though, to look at the engineering complexion of my org, right, and think like, I'm going to need people doing LLM ops and things like data fine tuning and how to build rag and things and APIs. There's plenty of APIs that allow you to do this. But like, ultimately you want control. Like your moat is your data, your moat is your interaction with users.

Hi everyone. Welcome to Training Data. Today we're excited to welcome Joe Spisak, Director of PM for Generative AI at Meta, where he leads llama and third party ecosystem efforts. Joe spent the last decade in AI, leading products at Pytorch and working on initiatives that span protein folding and AI math, many of which have spun out from Meta into their own startups. We're speaking to Joe just two days after the Llama 3.1405 B launch and we're excited to get his view on questions like where is the open source ecosystem headed? Will models commoditize even at the frontier? Is model development becoming more like software development and what's next in agents and reasoning, small models, data and more? Joe, thank you so much for being here today. We're so excited to have you just two days after the Llama 3.1405B launch. It's an incredible gift to the ecosystem. We'd love to learn a little bit more about how you, what specific capabilities you think the 405B is particularly unique at, especially in comparison to the other state of the art models.

Oh, thanks so much for having me. This is so much fun. I haven't done a podcast like this since something pre Covid, so it's like fun to be in the same room and just like, you know, chatting about this cool stuff. Yeah, I mean we're like beyond excited at Meta. This was something that I think a lot of us have been working on for such a long time, months and months and months. And we kind of put out that nice little appetizer, I'll call it in April, of like Mama 3. And I was actually like, people really going to be that excited about these models? And the response was through the roof. Oh my God. Everyone's excited, but they really don't know what's really coming. And so, yeah, kind of hold that back for a while and kind of keep it to ourselves and then kind of build up for this launch. And the 405B is a monster. It's a great model and I think the biggest Thing we've Learned about the 405B is just a great, it's like a massive teacher for other models. And we kind of had that plan all along because when you have a big model you can use it for improving small models or just distillation. And that's how the 8 and 70s became the great models that they are in terms of capabilities. We listen to the community, we listen obviously to our own product teams because we got to build products for Meta. And I mean long context was one of the biggest things people wanted and we have much longer context internally even than when we were released. But we saw just the use cases start to build up multilingual. I mean we're a global company so we released more languages, many, many more to come. Because obviously like Meta has billions of people on the platform and hundreds of countries. And so I think that was like to me those are like table stakes things but they're really done well on the models. I think we spent a lot of time in post training on our different languages and improving them in safety. Just they're really, really high quality. So we don't just like pre train on like a ton of data and say look at us, we're multilingual. You know, we actually did a lot of work in our SFT phase and supervised fine tuning and a lot of safety work. I think one of the coolest things that I'm excited about. Well there's a couple things I'm excited about, but one is tool use. Like I think models. Oh my God, zero shot tool use. This is going to be crazy for the community. I mean we show a few examples like we can show like calling Wolfram or we can brave search or Google Search and it works really great. But zeros tool use is going to be a game changer. The ability to kind of call a code interpreter and actually like run code or kind of build your own kind of plugin for things like rag and other things like and have that really be state of the art. I think it's going to be a really big game changer. And I think just the fact that we released the 405 itself and we changed our license so you can actually use our data. Like that was a big deal. Like that was a big discussion. We had many meetings with Mark on that and we and ultimately like landing on a place where, you know, this was like this pain point for the community for so long. They're like these closed models, like I can't use the outputs or maybe I can use them but maybe I'm using them slightly unscrupulously or whatever. Like we actually are encouraging people to do it.

I'm sure that was a tough decision to make. Walk us through the things that you had to consider in actually making that leap to open up licensing in that way.

Yeah, licensing. Oh, licensing is like a huge topic in itself. Obviously. We could probably spend the whole podcast talking about it. I don't want to, but we could. I think we wanted, number one, just to unlock new things. Like, I think we wanted to have the 405 and our Llama 3.1 models differentiate, give people new capabilities. Like just like, we just looked at like what people were really like excited about in the community, not only in, like in kind of enterprise and products, but also in the research community because we obviously have a research team and you know, we work with academia and we talk to folks. I mean, you know, Percy Lang at Stanford texts me all the time saying, you know, when are you going to release it? When are you going to release it? Can I use it? Can I use it? And Percy, like, you know, stay patient. But I think we heard them and we knew kind of what they wanted. And I think ultimately we wanted llama everywhere. We wanted just adoption, maximal adoption, really. The world using it and building on it. And I think Mark even used in his letter, he put out the new standard or standardized. So I think to do that you kind of have to enable stuff like that. You kind of have to unblock all these different use cases and really look at what the community wants to do and make sure that you don't have these kind of artificial barriers. And that's what the discussion really was. And so actually even beyond that, we started working with partners like Nvidia and AWS and they started building distillation recipes and even synthetic data generation services, which is pretty, pretty cool. I mean, you can start to use those and actually create specialized models from it. And the data that, I mean, we know how good the data is because we used it in our smaller models. It's really good and improves our models significantly.

So I want to pull on the open source thread a little bit more and I've read Zuck's manifesto, it was great. But I'm still, I'm trying to wrap my head around like what's in it for meta? This is a massive investment. So open source, in some ways you're leaving a lot of money on the table because you now have a state of the art model that you're offering to everybody for free. And so I guess. Is this an offensive move? Is this a defensive move? Like what's in it for Meta?

I mean, we've so. Well, first of all, our business model doesn't depend on this model to make us money directly. So we're, we're not, you know, selling a cloud service. We've never been a cloud company. We've always worked, I would say, with a partner ecosystem all the way back to the five years I was helping to lead Pytorch and the ecosystem the community built around that. We never built a service. We probably could have in some way, but it would have been weird. We saw basically going back to Pytorch, we saw it as this lingua franca bridge to this area of high entropy, this kind of weird way to say it, but there's all this innovation happening. How do we kind of build a bridge to it and actually be able to harness all that innovation? And the way to do that is to be open and is to kind of get the world building on your stuff. And I think that ethos is kind of carried over into Llama. And if you look at Pytorch, that was a huge way for us to kind of pull in at the time when we really started working on Pytorch in earnest, computer vision and CNNs and all that, if you remember that old times now. But we actually would see these architectures come like constantly. The people would, and they write code and they'd publish it in Pytorch and we'd take it internally. We evaluated people at open source models and put them out on model zoos and we'd evaluate them and we'd see just how quickly the community was improving things and we'd actually leverage that, especially for integrity applications where we release hateful memes and some of these other data sets. We just saw the improvements like week over week, month over month. And it was built on something that we were using internally. So it was very easy for us to just take it inside. So I think Llama is definitely similar in that regard. Where academia and when companies start to red team these models or try and jailbreak them, we want people to do that to our models and so we can improve. And I think that's a big reason. And it's like, be careful what you wish for, right? Of course. But it's the same with Linux, right? The Linux is open source and the kernel is open source and people will. It's much more secure when things are transparent and bugs can be pushed faster. And so that helps us a lot. I think it's, there's also the angle of we don't want this to turn into kind of a, a completely closed environment. I think just like today, if you look at Linux and Windows and in my opinion there's room for both. Right? There's room for closed, room for open and people use depending on what they need and the applications. I think that there's gotta be a world of open models and I think there's going to be a world of closed models and I think that's totally fine.

What was the primary argument against open sourcing? Was there one?

I mean there was definitely like competitive concerns. We talked through, you know, do you want to give your technology, you know, put it out there and, and that. And I think we're like less concerned about that because we're moving, we're moving really fast. Yeah, like if you look back, I mean I've been, you know, back, I've been a meta like close to what, six or seven years now and like in the last, you know, year year. So we've done, you know, we had a connect launch, we released Purple Llama last December. We released Llama 3.3.1. Before that we released Llama 2 in July. Llama 1 was like in February. So like just if you think about the like the pace, incredible, the pace of innovation that's like coming out of our team and our company is like just at a crazy pace right now. So I'm not too worried about it. I don't think we're that worried about it.

So I'd love to kind of move into your personal views on the broader ecosystem. I think a lot of the questions that people have center around what happens to the value of all these models, especially as meta open sources more of them at the state of the art level. With Llama 3.1, with OpenAI launching GPT 4.0 Mini, what is your view on do models commoditize even at the state of the art frontier?

Oh, this is a great question. I mean I think if you look at just even the last two weeks, I mean 400 mini is a really, really good model. You know, input, I think input per million tokens is something like $0.15 $0.60 out. So it's incredibly cheap to run, but it's also an excellent model. Like it's just like they, they've done an incredible job in distilling and getting to something that's like really, really performant yet really, really cheap. So I think Sam is definitely pushing on that. And then if you look at what we've done in the last week and pushing out, I would say pretty compelling state of the art models across the spectrum. I do think it's rapidly getting to a place where the model is going to be kind of a commodity. I think there's this frontier of data where we can certainly gather data from the Internet, we can license data, but at some point there is kind of some frontier of limitations I think that we're all going to have. And this goes back to our conversation, we gotten kind of the bitter lesson of data and scale and compute. Is that enough? It's probably not quite enough, but it's like compute and data becomes kind of, if you have enough of both, you can kind of get a first order approximation of the state of the art without anything else is kind of what we've seen. So I do think the models are commoditizing. I think the value is elsewhere. And I look at Meta and I look at our products and I look what we're building. Like that's honestly where the value is for us. It's Meta, AI, it's our agent, it's all the technology that we're going to put into Instagram and WhatsApp and all of our end products where we actually are going to monetize, where we're actually going to add real value. The model itself, I think definitely we'll keep innovating new modalities, new languages, new capabilities. That's what research is. Right? It's pushing the frontier in emerging capabilities and then we can leverage those in products. But the models are definitely pushing in that direction.

Yeah. If that's the case and all these existing companies that have massive distribution and wonderful applications that are already out in the wild can just adopt these state of the art models, what advice would you give to the whole wave of new startups that are trying to make it out there? Either building their own models using other state of the art models and then trying to build applications on top.

Yeah, I mean there's definitely like some model companies or companies that are building, you know, they're training pre training foundation models and it's expensive. It's like I think we're, you know, I can't say how much llama 3 costs but it was very expensive and you know llama 4 is going to be even more expensive. And so I, to me, given kind of the state of play and things it, to me it doesn't make that much sense. If I was a startup to try and go and do a pre training, I think Llama models are absolutely incredible as foundations to build on. And so I do think there is. If I was a founder right now, I would absolutely adopt open source. It forces me though to look at the engineering complexion of my org and think I'm going to need people doing LLM ops and things like data fine tuning and how to build rag and things and APIs. There's plenty of APIs that allow you to do this. But ultimately you want control. Like your moat is your data, your moat is your interaction with users. And you may want to deploy these things onto a device at some point and have kind of a mixed interaction or something. You might want to have simpler queries running on your device and have very low latency interactions with your users. You might want to split, you know, and have like a more cloud based approach for more complex queries, more complex interactions. And I think like the open source approach gives you that flexibility, it gives you the ability to modify the models directly. You own the weights, you can run the weights, you can distill them yourself. There's going to be distillation services that allow you to take your weights, distill them down to something smaller. I think that's pretty awesome. We're like just now seeing the beginnings of that. So I think like in my mind, like control matters a lot and ownership of the weights. There are a lot of API services where you'll do fine tuning your model, so you're bringing your own data, you're fine tuning and they use something called low rank adaptation or lora. And unfortunately you don't actually have access to those lower weights at the end of it. You're kind of like forced to use their inference. So you're like, let's see, I'm kind of like held hostage here. I've given my data, I don't have access to the actual IP that was generated from that data. And now I have to enforce to use their inference service. That's not a good deal. So I think open source kind of like brings inherent freedom that I think that approach doesn't.

What do you think of Mistral Large was announced I think maybe a day after llama 3.3.1. What do you think of them and I guess more broadly for everybody at the frontier, is everyone kind of pursuing the same recipes, the same techniques, the same kind of compute scale data, et cetera. And so, you know, everyone's kind of going to be roughly similar at the frontier, or do you think you guys are doing something very different?

So first of all, I'm Nistral. I mean amazing team. It was one of my old teams in fair. They were working on through improving and AI and mathematics. So Guillaume and Tim and the team are Marianne. They're, they're incredible.

Joe was just talking about fun banter last night.

So I mean this was like one of the scrappiest teams that I've ever worked with. I mean the team I don't think ever slept. So it was like basically by day they're doing like pushing probably even less now. Probably even less now. I mean they would push the state of the art in like AI and theorem proving and you know, during the day and we published some work on that, you know, I think a couple years ago now. Geez. But by night, you know, they were basically like scrappily like grabbing compute to train Llama one. And so we were building large language models several years ago in Fair and that team basically just, they were just really ambitious and they were kind of working by night and that's really where llama1 came from. So the team is great. I mean I think they're doing really good work. I think they're definitely challenged in that they're trying to also open source models but also make money. And models like 4.0 mini are not helping them because like, and this is I think why they changed their license for example to like, you know, research only license. Which kind of makes sense because they were you know, open sourcing models and they immediately like their own ecosystem is like competing with them in a lot of ways because they'll release a model, they'll host it, like use this model. But then they have you know, together and Fireworks and Lepton and all these companies that you know, provide like sometimes a lower cost cost, you know, per million token, like, like offering. So it's really tough business right now. In terms of like Large two, I think it's a really good model. I mean I, we just on paper I haven't evaluated it, we haven't looked at it internally yet. I think if you look at like artificial analysis, I think they added up in kind of like the. It was a little, it was under I think like the 70B model in terms of like quality. But you know that's like a blended, they blend a bunch of benchmarks to make that distinction. But on paper it looks really good. We're going to evaluate it. I think for me anyway the more, the merrier, the more models are out there, the more companies are doing this, the better. It's not like we're not going to be the only one. And I think that's good that we're not the only one. And I think more generally the Gen AI space, you wake up every single day and you kind of expect something like this, Right. You expect a model to be released or something groundbreaking to happen. And that's kind of like the fun of being in it, so.

Totally, totally. Do you think everyone at the Frontier is comparable though? Like, are you all pursuing comparable strategies?

Yeah, this, this is actually a good question because you know, if you read the Llama 3 paper, which was I think 96 pages, you ended up at. Right. Lots of citations, obviously, lots of sharing, lots of sharing, lots of like, you know, contributors and core contributors and that. So like it was, it was a, it was a detailed paper and Lawrence and Angela on the team spearheaded writing that. And I think that was like one of the hardest things. Like developing the model was like relatively easy compared to writing the paper. It was a lot of work putting that paper together. I think if you look at Llama three, there was a lot of, I would say, innovation that happened, but also we didn't like, we also didn't, I would say take on a lot of research risk either. Yeah. So I would say the primary things we really did with llama, with the 405B especially was really pushing scale. I mean it was still. We used group query attention, for example, so gqa and that improves inference time and kind of helps solve the kind of quadratic attention computational challenge. We trained on over 15 trillion tokens in post training. We used synthetic data which improved the models, the smaller models quite a bit. We trained on over 16,000 GPUs on our training runs, which is something we hadn't done before. It's really, really hard to do that because GPUs fail and you know, it's not the table. Yeah, I mean it's everyone's like, oh, I'm just going to train at 100,000 GPUs, like good luck. Right. You better have a really, really great infra team, a really great MLSYS team. You better be like ready to innovate at that level. Because this is non trivial. Everyone says it's easy or says you can do it, it's nontrivial. So I think like I almost look at llama 3 as very similar to like the GPT3 paper. So if you, if you ever talk to like Tom, he was a lead author, Tom Brown now at Anthropic, and there's a reason why Tom was the first author on that paper is because like a lot of the innovation was really scale. It was really like how do I take something that's, you know, like an architecture and like push it as hard as we can push it. And that involves a lot at the mlsys kind of layer and infra layer and how do I scale the algorithm? I think that was really the mentality we had with llama 3.3 and llama 3.1. And I mean internally, obviously we have great research team, we have fair, we have research in our org and we're looking at lots of different architectures and MOE and other things. And so I think who knows what level four will be. We have a lot of candidate architectures and we're looking at it, but it's kind of a trade off. It's a trade off between how much risk you take on for research and potentially how much reward or the ceiling and the potential improvements versus just taking something that's relatively known and pushing scale and getting that to improve even more. So ultimately this becomes a trade off.

I think this is such an interesting point. I actually also think it makes Llama and Meta quite unique in the strategy. It's taking the words that I like you used yesterday were, is model development becoming more like software development? I'm curious to hear if you think unlike what many of the other labs have been doing on pushing more of the research, you guys have been focused on just executing on strategies that, you know, work. Do you see that representative of the continuous strategy, you think as you extend llama out 4, 5, 6, 7, 8. And then also how do you think the other research labs and maybe some of the other startups in the ecosystem will react? Will they kind of switch and veer a little bit more to the strategy that you've been taking.

I mean, it's a really great question. We don't have all the answers for sure, I think, but there's definitely somewhere in the middle. Right now is kind of where I see things landing, where we'll continue to push on execution, we'll continue to push models out because we want our products to iteratively improve as well. So we want meta AI improving constantly. So there's definitely a software engineering analog here that's happening where you can imagine something like a Llama train and new features, new capabilities get on that train and we have a model release. You know, it's actually, it's much easier when you start to componentize the capabilities too. Like we're doing that with safety right now. And you saw in the release we released Prompt guard and a new LUM guard and you can iterate on those components externally and it's great. Obviously, the core model is much more difficult. I do think, you know, we'll start to include or start to kind of push on the research side because, as well, because I think architecture is going to evolve. I mean, you've seen what AI2, for example, has done with their Jamba and their Mamba and everyone kind of thinks Mamba has got a new architecture that could have promise. I think what's interesting though is to truly understand the capabilities of the architecture, you kind of have to push the scale. And I think that's what's missing right now in the ecosystem is if you look at academia and academia is a lot of absolutely brilliant people there, but they don't have a lot of access to computer. And that's a problem because they have these great ideas, but they have no way to truly execute them at the level that's needed to really understand will this actually scale? Because the Java paper and model was really interesting and the benchmarks look great, but they didn't scale it beyond I think under 10 billion parameters. So you're like, okay, what happens when we train this in 100? Does it actually, do you still see those improvements or not? And no one really, at least outside of these labs knows the answer yet. So I think that's like one challenge. So I think, like to me we're going to get into this hybrid space of, you know, we are going to push definitely on architecture. We have a very, very smart and well accomplished research team, but we also are going to be like, you know, we are going to be executing and I think that's when we start to get like a recipe. You know, we're going to push it to the limits and you know, we are going to start to release more models on it, but in parallel to that we have to push on architecture. And I think it just makes sense because the next breakthrough, at some point you're going to reach a theoretical limit and you need to evolve the architecture. So I see kind of a little bit of an in between. And obviously we're really good at execution. I think we're pretty good at execution, but we're also good in research and we just need to marry those two. So it makes sense because research and products are very different, right? Like one should be pretty deterministic the product side and one is inherently non deterministic. Right. It's like, is this going to work? I don't know. It's a really Big bet. If it fails, it's research. It should have a non zero chance of completely blowing up in our face. We just need to go in another direction. But that's what research is.

I'm curious about one branch of where a lot of I think model research is happening right now. Agentic reasoning. And I think you all have announced really great results in reasoning. I'm curious, maybe at a very basic level, how do you define reasoning? And then are you all seeing reasoning fall out of kind of scale during pre training? Is it post training and is there a lot of work left to do on the reasoning side?

Yeah, reasoning is a bit of a loaded area. I mean you could argue it's things like multi step and I think the best, unfortunately the best examples we have are kind of like the sort of semi gimmicky Bob is driving the bus and he picks those kind of things. Right. If you troll local llama, you'll see a billion of those. Right. But those actually force the model to take multiple steps to respond to you and think through and logically kind of respond. I think coding is actually really when you look at pre training, to answer your question directly, reasoning improvements come in both post training and pre training. So what we've learned, which is now everyone's like oh of course this is the case, but definitely the last year or so everyone's kind of learned that code. Having a lot of code in your kind of pre training corpus really improves reasoning. But that's when you think about it like of course, duh, it's step by step, it's very logical. Code is very is just logical by nature and kind of step by step. And if you incorporate a lot of that in your pre training your model will reason better. And then we of course look at examples in post training and SFT to improve as well. So we look at the pre trained model and it kind of depends on how you balance things as well because you can balance how well your model reasons with how well it responds in different languages. Ultimately in post training everything's a little bit of a trade off. Like you can super optimize things for coding if you want to. And we did that with code llama. It was really great. But of course the model will suffer like in other areas. And so ultimately it becomes kind of like Pareto frontier of capabilities we want to bring out if it's a general model. And I think like yeah, I mean ultimately it's a trade off. So anyone can kind of pick a benchmark or some capability and say I'M going to super optimize for it and say, by the way, I'm better than GPT4. Well, great, anyone can do that. But is your model as generally capable as GPT4 or LLAMA 3.1 or whatever? That I think is a different story.

What do you think are the future levers to unlock reasoning for anyone going forward?

The obvious answer is data. The more code and supervised data that you can get, I think is a natural answer. I mean, I think we need to find applications as well for how we define it and that would help us. Like once you've kind of start finding like those kind of killer applications, then you can like, then you kind of know where to kind of focus in terms of your other, your gated, exactly what you're solving for. Like, and this goes back to like evals and like what is, what is your eval? Because we're starting to saturate evals and so we tend to as a community, we define a benchmark and a metric and we just optimize the living hell out of it. And it's great. But then you actually look at the model in an actual environment and you're like, oh well that model has a better MMOU score, great. But how does it actually respond? Well, it doesn't respond as well, but it has a better MMOU score. And so I think we need better evals and better benchmarks that allow us to, you know, I would say like find clearer line of sight to actual interactions. And I think like, you know, the live, what is it called? The abacus benchmark. The live bench, I think it's called, I can't remember the name of it is pretty good. I was looking at that. And of course like lmsis and Chatbot arena, like these are more natural even though, you know, still not perfect, but it's like moving in the right direction of things that are more human, like interactions versus a static data set or static prompt set that is not that helpful. So I think once we start to find these other what reasoning use cases make sense, we're going to start to generate more data and you're going to start to improve the model there. And hopefully that has again line of sight to a benchmark or an eval that actually feels like it improves an end product. And a lot of this actually depends on the end product, of course. What is my application?

Out of curiosity. Within large research labs, coding and math have always been two primary categories in trying to unlock reasoning in the startup ecosystem. Now we're seeing more folks who really want to go it from the math angle. Do you have a perspective on whether or not that has led to interesting unlocks?

I mean the answer is yeah. I mean I think if you look at our data or at least our models we've like coding and math have been I would say the primary levers. So I mean it's, it's. Yeah, I mean I think that's like having more obviously is better because obviously math is also very logical and very like step wise. So obviously you can see the pattern here. The more data you have like that kind of follows that sort of pattern, the more your model is going to be able to reason and you can see that in how actually models respond. Like if they start and you ask them to respond and step me through your thinking process and it'll actually do that and some models do better than others. So anything like that? I think scientific papers also we had some projects out of FAIR that trained on you know like archive papers and you can see like not only is like code and math like pure mathematics, but also like scientific paper which is like science scientists are very logical in how they write things and how they stepwise and how they create images of their like charts and stuff and like that also I think we've seen like just general scientific information like helps as well. Sorry. Galactica was our project. Yeah. So Robin Ross from the People team led that still in my opinion like one of the coolest projects ever. It got a lot of bad press but wow. They were ahead of their time in my opinion.

I'd love to talk a little bit about small models given the scale of capital and the compute that many startups have. The 8B and 70B models are an incredible gift to the ecosystem. And it's funny that you called them appetizers at the start because I think they're super powerful for that for that set. But they're also really powerful for a number of different applications where you want smaller models. And so I'm curious to hear what do you hope to see developers use the 8B and 7.8B models for given that they are best in class for their size of model.

So it's interesting though, when we released yeah we released April Llama 3, we released an 8 and a 70 the appetizers as we call them. The 8B was actually better than the Llama 2 70B by leaps. So we were, you know, I had to look at the chart and, and I was like is this right? Yeah, like is that really the case? And we're like yeah, like it really is. It was that much better.

What's the intuition for how that happens?

I mean, it was more data. We, we had what, 7x more data? Yeah, obviously, like we, we put a bunch more compute at it as well. So going back to compute and data, we're pushing on those. So I think we saw, it's almost like every generation, which is again, the generations are accelerating, you start to see the benchmarks for a large model basically get pushed down into the smaller size regime. And so 70 becomes an 8. And we have, internally, we have models where the 8 is much even smaller than 8. Actually, we're starting to see really nice benchmarks on even smaller models. So you continue to see the models improve at smaller scale. And that I think is just. We're pushing the architecture, we're pushing scale and we're starting. We haven't quite saturated it yet. And I think that's really interesting. So for me, one of the biggest reasons that I think a small architecture is useful is obviously on device. Everyone loves to talk about on device and Apple's talking about that and Google has Gemma models and Gemini running in Android devices. So I think on device makes sense. I think safety is kind of interesting because one of the things, we have our own internal versions of Llama Guard, which we used, that are orchestrated for our applications internally at the company, at Meta. And today they're built on an 8B model, which is kind of expensive to run. If you think about a safety model that's kind of like the secondary model. And so I do think internally we've been experimenting with much smaller models in that regard. And it creates efficiency, it lowers latency. Because really those models are really just classifiers. They're not really autoregressive chat like interfaces. They really just classify the input prompt of does that violate this whatever category in the taxonomy, in the output the model, when it generates, does it violate that kind of stuff? So you can actually push those even further. I think that there's also really interesting cases though for on device where you almost have, when you think about privacy and you think about data, you want to have like your data stay on device. You can think about like a rag like architecture on device. So you have data, you have even your chat history that's like on say WhatsApp or other things, you can imagine that model having access to data, aggregating it, and then running some type of almost like a mini vector database, right, where you're using rag and doing your kind of fuzzy search or fuzzy matching and with your like small model and that becomes its kind of own system in itself. And you can basically do things like local summarization. I don't know, I get so many text messages like hey, Summarize my last 15 messages please. Because I've been in meetings and I haven't looked at my phone and that's super useful. And then I don't have to send data up to the cloud or anywhere else. So there's those kind of use cases I think where small models actually are going to be really compelling. And then for super complex queries and things, obviously you have a big model in the cloud that can always service those. But for many things, I think on device or even in the edge and on prem, these small models actually can do pretty good.

You talked about scaling up compute and data as the two fundamental vectors to improve performance. I guess there's been a lot of chatter about how we are going to hit a wall, or maybe we're not going to hit a wall on data and maybe synthetic data is the answer, et cetera. I'm curious your perspective on that. Is there an impending wall that we're going to hit most likely of cheap accessible data?

What do you think?

How do we scale beyond that?

I mean I think we've shown with this release that synthetic data does help a lot. I think we've in pre training we train on 15 trillion tokens or give or take and in post training we generated a ton of millions of annotated synthetic data. A lot of it generated by the 405B. We obviously paid for annotations as well. I do think synthetic data is a potential path forward. I think it's going to like we know now and the kind of proof is in the models. Right. It's great to talk about it in that I do think data is going to be a challenge at some point for us. And this is why I think companies are licensing a lot of data these days to get access. Open ad is licensing data, we're licensing certainly data. I think having access to services that generate data to improve models is important. So I think that inherently is an advantage for a lot of companies. I mean Google has YouTube, right. I'm sure is a value to them, which kind of implies that bigger companies have an advantage which is not something that's anything new. Right. We've been talking about this for a long time in terms of a data wall. I don't know, I mean, I think we're not there yet. I would say like let's talk another. Let's schedule this for like a year and let's see where we are next year. I'll save my calendar for one year exactly from now and meta AI, but let's talk in a year and see where we are. But we haven't hit it yet and we're still scaling and we're still gathering a lot of data and we're generating data and our models are still like continue to improve.

So yeah, let's close it out with some rapid fire questions.

Sure, sounds great.

And what year do you think will surpass the 50% threshold on swebench?

Good question. If I've learned anything, it'll be faster than whatever answer I give you because I think any benchmark will. As soon as we zero in on it, people are going to go and figure it out. So I don't have an answer. It'll be fast, I'm sure.

You know, one of the questions we have been asking people is in what year will an open source model surpass the other companies on the front, the other models on the frontier? And we have to take out that question now. Thanks to you all this.

I mean it's, it's true. We're almost there. I mean I think 405B is incredible. It's definitely, it's definitely in that class.

Yeah, absolutely.

Which is incredible.

Will meta always open source Llama?

I mean, I think Mark's pretty committed. You saw his letter. I mean we've, we've open sourced for years and years now. Back to Pytorch to fair to Llama models. This isn't something that's a flash in the pan for the company. The company's been committed to open source for a long time. So I wouldn't never say never. But the company and Mark are really committed.

Amazing. Joe, thank you so much for being here today and also for all the work that you're giving to the entire ecosystem. I think the entire AI community is, is very much grateful for all the work that you've done with pushing out Llama and the advancements to come.

It's a huge team. Check out the paper. Look at all the acknowledgments.

I mean we spent all of yesterday reading it.

We need like the Star wars, like scrolling text of all the contributors because it was an incredibly big team.

I was thinking about that same thing.

Yeah. So my hat's off to the team. This was a total, I mean this absolutely took a village to get Llama out there and so proud and excited to represent the team here. So thank you.

Thank you.

Sam.