#229 Inside Meta's Biggest and Best Open-Source AI Model Yet with Thomas Scialom, Co-Creator of Llama3

Hi, I'm Maggie, VP of Curriculum at datacamp, and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey on datacamp.com.

We are reaching a level of intelligence where this kind of agentic behavior will start to increase more and more. It's always the case, and for that you need a model that is able to format a structured output to reason, to write a plan to evaluate step by step refine its own output.

Welcome to Data Framed. This is Ritchie introducing an episode on behalf of Adele since he's on vacation. Meta's Llama 3.1 family of large language models has just been released and it's pretty exciting. It's the biggest open source LLM so far and we're eagerly awaiting to see how well it performs in real world usage against current performance leaders like GPT4O, Claude and Gemini. The developments in large language models over the last few years have been astounding, and both Adele and myself have been keen to hear a developer's perspective. Just what do you need to do to create these high performance foundation AI models? And looking to the future, There are some wildly differing opinions, even amongst experts, about whether the dramatic increases in performance can continue over the next few years, or whether we'll hit a plateau again. I'm curious to hear the inside scoop from from an AI developer. Today's guest is Thomas Sialom, a senior AI research scientist at Meta. His research is focused on generative AI, as you might imagine, and more specifically developing the llama series of LLMs and its surrounding ecosystem of tools. That is, he's helped create much of the technology we're talking about today. Previously, Thomas was a partner at Recital, an AI research lab, and he switched working in AI from being an investment banker trading in exotic commodity derivatives. So all in all, he's ideally positioned to tell us all about LLAMA and the developments in generative AI. So I'm as excited as you are to hear his conversation with Adele.

Thomas Yalom, it's great to have you on the show.

Thanks for having me.

You are a senior staff research scientist on LLMs at Meta's AI organization. You've been integral to the training of the LLAMA models, arguably one of the most widely used and best open source LLMs today. Maybe to set the stage and start off our chat, how has it been like being at the center of such an important moment in AI today?

Right, look, it's weird, but maybe one quick anecdote I can tell you is I'm based in Paris where it's very high but less than in San Francisco. And when I went with my wife in San Francisco like a few months ago, she was like, we met people in the street or like in her. Oh, you're the guy from Lama. And she realized it's something.

Yeah.

Like here all my friends are, oh, you're working on AI. Cool. Like, do you do that? I also, yeah.

And yeah, in San Francisco, people were stopping you on the street. So yeah, I think, yeah. I recently saw a video of some Altman with paparazzi behind him. It's a pretty interesting moment in AI today. So you just unveiled llama 405B which is an incredible milestone in open source AI. Maybe walk us through llama 405B in a bit more depth, which I think is the biggest open source LLM out there. What makes it unique to other open source LLMs?

Well, the size of course, but beyond that and that just one scaling dimension is a scale we train it on. Again like 15 trillion tokens. With respect to the preview released we did at the time, we have done much more post training in lhf. So the alignment is also much better and so the resulting performance follows. Let's see the results on the RNALIDOBOTNs and other benchmarks. But I'm hopeful that it will be the best performing model out for open source. It has long context. It has. Although we may not release it like multimodal and other capabilities. So it's really filling the gap with the frontier models.

And you know, we're definitely going to deep dive into the training of LLAMA 405B and also all of these models. Maybe when we're speaking about training of these models. This is quite a technical challenge. When thinking about the scale of llama 405b before we talk about llama 405b, just general training, these large language models. What are some of the biggest challenges in developing these models and how do you navigate these challenges?

The biggest challenge to me is how to balance exploration and exploitation when you do some experiments. You cannot do that at 4.05b on 15 trend tokens and do like 20 ablations, that's just not possible. That's the kind of run even for smaller size, to be honest, like 70B that you put a lot of resources of this one. And so you need to be confident with the EP parameters, with the data, the training mix everything. And so how to choose that you need to do smaller scale experiments and to then infer the right parameters at a larger scale, you need to infer the right dynamics. Maybe like a simple example, what is the impact of multiple epoch? So seeing repeating the same data multiple times at a larger scale, there's more weight, so the model memorize more. And so you could have some new phenomenon that you didn't observe at a smaller scale. And how to trade off this unoptimal large scale run because we just couldn't grid search everything. There's just the exploration you want to do at a smaller scale is something tricky. And so how even to balance the resources between exploration and exploitation is very challenging.

And maybe walk us through some of the best practices that you guys have developed while having that balancing act between exploitation exploration. What would be your advice for the small handful of other teams looking to train large language models?

I think in Genan the best advice is to follow first principle like garbage in, garbage out, for instance for the data and data is probably one of the most important thing. So making sure that the data you collect is high quality, reviewing it manually with some processes, with quality analysis, some classifiers, we use Lama 2 to do that and collect the tokens or pre training and making sure you validate your ideas and your intuition with a run at a smaller scale and you see the improvement. It's never guaranteed, but it's likely to lead to some gains at a larger scale. And I will say at every stage of the pipeline being very, very rigorous and meticulous to get the best effort to have the best class pipeline.

Okay, that's great. And you mentioned here the data and making sure that the data is high quality and doing qualitative analysis. I'd love to learn a bit more about that massive amount of data is going into training these models. How do you make sure that this data is set up in a way that you're able to ingest it into these models and have confidence in the output. So I'd love to see kind of what are the processes that you adopt to make sure that the data is high quality, because that's a big aspect of making sure that the model is successful in the long run.

For pre training in particular, you just collect Basically web tokens. Let's say you have like famous data sets like Common Pro or the PI. But the truth is that the web is garbage. It's just random. If you look at a random paragraph or document from these data sets, it's random noise. And clearly there's something wrong to train your model on 50% random noise. Like it's a waste of compute. So the first basic thing to do is can you train a classifier using some heuristic or machine learning models to select what is the right. Then can you do some topic classification? Maybe you can rebalance your data mixture so that it covers not like 0.5% some domains, but it's more like balance. You don't have a hole in the knowledge. Can you add additional sources? What is your pipeline to clean the data? For instance, we developed NUGA last year that performs a scientific ocr. Just putting a lot of effort in this model so that you end with a high quality text from a PDF on scientific text leads to improvement on reasoning for instance.

Okay, that's really great. And this here we're talking about the pre training aspect of it. But you mentioned as well there's a lot of work that happens on post training and reinforcement learning from human feedback. So I'd love to learn as well about that aspect of training a model and what that looks like in practice.

Overall the general stage for people is you do supervised fine tuning so that you start from your pre trained model which is just the next token prediction and you align it to follow instructions in the chat model style through for supervised fine tuning. In general, it's like what humans will have written as the expected answer from the chat model and then you do reinforcement learning with human feedback. An interesting thing to note here is we stated that in llama 2 generation from the model itself with truth reinforcement learning feedback, synthetic data are higher quality than human written outputs for a lot of. So if you follow this principle on what should we initiate LLAMA 3 model with supervised fine tuning data, we could ask humans or take what we did for llama 2 and take the supervised fine tuning data we used. But this is actually worse than what llama2 generated. So we use llama2 to generate the first pass the first round of supervised fine tuning data. So from the round one and we had multiple rounds of LHF, we used llama 2 to initiate with synthetic data. And once you have that, you have llama 3 supervisedly fine tuned on llama 2 outputs, front end outputs and then you start your ILHEF rounds you collect new preferences with this model, you train your sample with your llama 3, you do reinforcement learning and so on.

That's fascinating. And essentially you reach a point where you have kind of a self sustaining loop of producing data because you're improving the model which creates supervised fine tuning data, which improves the model. At what point do you reach a limit in improvement in that phase?

We have so many open research questions on lhs and this is one of them. We haven't seen plateau really yet, but we are also in a process where we try to nudge annotators to always ask harder and harder prompts at the edge of the model capability in a way that, well, write a poem about language model is something that now the model does pretty well. So we will ask annotators to ask less about that and more about something more complex, but not too complex because again, the process of annotation here is that you ask a prompt, you sample two answers from your model and you ask the annotator which one he prefers. And so if the two answers are bad, you don't have any signal basically. So you need to always be at the edge. One difference with La Methe 2 I can emphasize is the T shirt mode we had, which is basically if the two inserts are bad, we can sometimes not always enter in what we call the teacher mode, where the annotator can basically write a critic prompt saying oh, this answer, even if I prefer it to the other one, it's pretty bad for this and this reason, and then edit the answer to make it better.

And that creates another way to retrain the model with a human and to.

Move away to have a shortcut when the model is just bad.

Okay, that's very fascinating. And so that's one of the open questions is that that has yet to plateau. And I think this is a great segue to discuss the broader generative AI landscape as well, because there's quite a few open questions on LLMs today and what the next generation of LLMs will look like. The leap in intelligence from models seems to be getting smaller with each release. Or for example, the jump from GPT2 to GPT3, the GPT3 to GPT 3.5, 3.5 to 40 point for seems to be getting slightly smaller with each iteration. So one, I'd love to see if you think that is correct. And then second, do you anticipate another significant leap in intelligence soon? If so, what does that look like and what's driving it?

If we think in terms of order of Magnitudes, Yes. I anticipate much bigger jump than what we had. I think it's easy to measure the gap of intelligence between or no age, not intelligence, let me say between a 5 years old kid and an 18 year old kid rather than an 18 year old kid and a 30 year old.

Yeah, that's true.

I think we are seeing kind of the same thing with those models memorizing massive amount of knowledge. And at some point they know almost everything. And so yeah, maybe the gap starts to be smaller and smaller in terms of knowledge, maybe beyond the long tail, but it's harder to measure. And so the perception from people is that the gap is smaller. I would argue that in terms of intelligence reasoning it might not be true. There's like all those tasks on for instance on Mazda Benchmark we had seen immense progress from GPT2 to GPT3 4 fair. And now like Gemini Cloud 3.5 and almost 90. I think with one of our recent panel that indicates some progress. I will also argue that at some point what matters is not the knowledge. It's fine to not know something if you know how to find it by browsing or by looking at Internet. And so I think we are reaching a level of intelligence where this kind of agentic behavior will start to increase more and more. It's already the case. And for that you need a model that is able to format a structural output to reason, to write a plan, to evaluate step by step refine its own output. Ultimately I think it's one of the key feature self critique itself. Right now on the LHF stage everything is synthetic, yet we have a human at some point that says this output is better than this output. And we may see the early sign that the model is able to judge. In some contexts it's still limited. And while we will not see this kind of skills improve on some main benchmarks which are very knowledge intensive anyway, maybe the gap of improvement on these kind of new skills and abilities will increase, which to me let me think that the gap is just not narrowing at each generation. The second thing is when you achieve this level of intelligence you can compound with some additional order of magnitude improvement like tool use. We did Toolformer one year ago and now with instructed model in a Zillow shot you can give a tool and the documentation in natural language to an API and then they will learn to leverage that. And what I was saying before, learn to search online information for instance, if it doesn't know a calculator code execution and all those tools, multiple steps and self refinement and say oh, now it's a dead end. Can I maybe explore another way to solve that? And all that will compound to orders of magnitude more intelligent behaviors and capabilities that may not be reflected with the pure LLM performance on some benchmarks, but with new generation enabling that.

That's a really great insight. And yeah, I couldn't agree more. I think the agentic path as well will definitely drive a lot more usefulness. Even if you know on the LLM benchmark you may not see that come out very directly. But maybe to expand on that, how much of that agentic workflow is enabled by software engineering with having LLMs with a layer of tools access and making sure that there's kind of a recursive nature to like using the tools and all of that engineering work versus the training of the model. So how do you balance those to create a really strong agent? Like you need both. But where are we now in intelligence essentially to enable these set of agent use cases?

So we created a benchmark last year called General AI Assistant Benchmark GAIA. It was interesting because GPT4 had like 5% GPT 3.50%. It corresponds to tasks that will require a lot of different steps. So complex tasks for which models are failing, for which the answer is not present on the Internet. An example I can give you is for instance, oh, you know, like the NASA published a photo at some point on Twitter about this astronaut and they published a similar photo but with another one, same condition a few months later. Can you tell me the number of days between the two? And you know that this information is not spent on Internet arguably, so there's no memorization possible. So the model has to do the steps. This is a kind of question that doesn't require any expert skills. We didn't want to go to three agents or things like that. That's what any human will solve easily given enough time. But that models are totally failing. And with the last year what we have seen is some agentic framework like Autogen, OS copilot, hugging face agent that did what you mentioned, connecting tools and some like self refinement loop and things like that. And, and this was critical. It moved performance from like 10% with GPT4 to something like 30%. It's yet to be solved, but the progress is immense. It's trendy. That being said, if you power Those agents with GPT4, that's the performance you get. If you power them with GPT 3.5, the performance is zero. That said, the gap between the two is crazy in terms of capabilities, editing capabilities.

Okay. And yeah, you can imagine that, you know, GPT5 or you know, Llama 405E sorry, will drive a lot of the progress there.

No, that's true. And we put a lot of effort. We emphasize that I think in the paper about the tool used for post training. I think Lama3 is very good at zero shot function calling, which is a key feature that so many people ask about. So I recommend to test it for that.

Definitely the community is going to play a lot with Llama 405B and the Llama 3 family. So maybe one additional thing on the state of generative AI. Do you believe that scaling laws will continue to enhance performance of models as we increase size of data and the training runs and the compute with this famous Twitter image of small circle GPT4, massive circle GPT5 and it will have 1000x performance at this point became a meme. Do you see that scaling laws continuing on as we see today or are we reaching a plateau?

Yes and no. I think it's a bit more nuanced. I think that absolutely scaling will lead to improvement. Yes. That being said, what do you want to scale? And by that there's so many improvements in terms of algorithms, in terms of training weight that leads to a better scaling slope, that the 2del model at 7b are often better than what we had like 2 years ago, or than GPT 3.5 which was much bigger or something like that. So I think There is a 7B model that is performing Minerva, which was something based on, if I'm correct 300B, I forget parameters, so 50 times bigger. So there's so many improvements that comes from algorithmic optimization as well from data quality, from the training method and from additional external things like tool use, things like that that could end in a better loop with synthetic data, which are augmented than expert iteration method. But yes, scaling is a way but not scaling anything. Like if we had to scale lstm, we will be nowhere today. So yes, scaling will keep playing the biggest factor role. But if we scale by one to three other managers, we can get maybe 10 or 20 orders of magnitude scaling by improving all the other aspects.

That's great. So quality of data, algorithmic optimization are also big levers outside of just additional data and compute. One thing that we saw as well this year, especially with GPT 4.0, was multimodal capabilities. Definitely multimodal capabilities improve a model's usefulness. But how crucial is it to train models on these different types of data to boost raw capabilities. Right. Maybe intelligence is not the best word, but their ability to perform tasks. So yeah, walk me through the importance of having different types of data to have more performant models.

First thing is you want your model to be able to process ideally any input to any output. And that's just much more smoother scalable than a very complex orchestration. That being said, with respect to how it will improve a multimodal transfer to let's say text is a bit unclear. My intuition is that either we didn't crack the problem yet or another hypothesis is that text is so dense in information signal from what it contains, it's the log of the thoughts somehow that you need so many more images to basically map the signal of information you will have in the text. Maybe in multilinguality there's an argument that a speech end to end model may with the accent in donations like that leads to improvement. There's a hypothesis there which we haven't explored yet on our side.

And then maybe one last question on the kind of the state of generative AI and the technical landscape that we're living in today. You mentioned something, if you scaled LSTMs, we wouldn't be where we are today. Right. Gary Marcus famously repeats this notion that deep learning is hitting a wall and that LLMs and generative AI will plateau as time goes by. Granted, he's been saying this for the past at least five years and we've been seeing improvements all throughout. Do you think deep learning and the transformer paradigms are reaching their limits?

No. Anyone that said that in the past has been proven wrong in a very short term.

Yeah.

No, what I can tell you is look, what I often say is one we know that we have, I was saying before exploration with respect to exploitation, I think if we had like zero more progress, zero more breakthrough in the next decade we will have still like five to 10 years to just explore, understand all these breaks, breakthroughs and revolution we just had, which will lead to incremental significant progress, but incremental only for the next 10 years. Integrating all the modalities together, managing other things, agentic behaviors, well done. Inference, quantization, better post training, we are still like at the historical age. All those things together will lead to actually probably shoulders or better models. And yet there's so many resources put in there. Not just money, but people that are working on that and more and more every day. It's likely that what the last decade of deep learning told us that a breakthrough will come again. What it will be, I don't know. We had seen post training LHF method, we had seen Transformers, we had seen self supervision and all those things progressively it's likely that something else will happen and will again give us few orders of magnitude more better deep learning stuff. So I think it will be very crazy to bet against that.

I personally agree with you. We should have Gary Marcus on the show, see his thoughts. Maybe as we shift a bit to discuss the broader LLM landscape as well and other providers, the open source community, the private foundation models with the announcement of GPT 4.0 by OpenAI OpenAI Claude 3.5 by Anthropic First I'd love to see from your perspective, how do you view these models, right, like from a technical perspective, what is impressive about these models and then what are your thoughts on their capabilities? And then I'd love to see kind of how you view the ecosystems of open source AI versus closed source or private foundation models emerging or evolving over the next year.

I think all those models are pretty good. I guess we're at a stage where we are more or less at the same level of capabilities. Besides maybe GPT 4.0 being like end to end multimodal, which is something kind of breakthrough compared to the others. That being said, we keep going at least incrementally and then for the next generation of compute and scale, I'm really keen to see what will happen. I would say we have filled the gap between open source and closed source. Think about one year ago we were releasing llama 2. I think it was a few days about one year ago before that you had zero open source chat model. We have come from a long way now there's more and more open source models that are better and better. Hopefully with this one we'll fill the gap even closer. Now the question is how to have open source being number one and not just number two, which we're working hard on that. So trend is pretty good. I guess the question, which is an open question to me is in a world where there's more and more companies still leading the space, but with some secret source like Gemini, Anthropic, OpenAI, will it mean for them that they cannot compete with the open source community? Because every time there's a new thing that they will work on, the open source community will fill the gap and will be able to get better and better models and closed source is not the way for open science and will just limit progress. They will be able to make some breakthrough internally that will be very hard to produce externally, which leading to some very important advance for some of them and not others. There's an open question there. We'll do our best so that open source leads the game.

And this really segues really well to my next question because I think meta is the exception here in a lot of ways. In a few other organizations such as Mistral, maybe. But do you believe private models have an inherent advantage over open source counterparts because of their compute the amount of money they have, their ability to hire researchers, the data that they're procuring? If so, how can you mitigate this as a community, as an ecosystem rather than as meta?

Yeah, that's a question I often ask myself. I don't know to what extent. I mean, for instance, we don't train our model on user data. We don't have anything special about that besides working hard on high quality data. So we don't have an a competitive advantage to be at meta right now for that kind of models. Do they have at Google or at OpenAI? Maybe, I don't know, I'm not sure entirely. It may be helpful, but to what extent is not totally clear to me yet. So I'll say the main thing is OpenAI started that game and betting on scaling language model a long time ago. We had our first language models one year and a half ago. Like I was working on Galactica December, it was one year and a half. Then my friends in Paris followed with Llama 1 pretrain model and we are very, very early in the game on our side. So we move very fast I think to fill the gap. And the train is good, I think it's the main reason.

And the open source communities is super vibrant. Right. LLAMA has an incredible open source community and there's a lot of players today in the open source community. Which LLM providers do you think will survive this wave? Or maybe like what will be the defining feature of an LLM provider that will survive this? I don't want to name names here but. And why do some models gain popularity while others don't? This is such an interesting aspect of the ecosystem.

Yeah, that's interesting. I mean the performance of the models is something that works. There's of course a bit of hype and marketing around that. Most of the time I tend to think that also the credibility of the organization that reads the models, you can release everything with putting all the numbers and overfitting benchmarks. If people know that and discover that once, they will have much more doubts than next release you will do. For instance, we have proven in the past to be very lead in our evaluation with Lemma. So people trust us and others have done the same and that's good. So I would say then what I'm seeing right now, but that might change, like things are changing very fast is a lot of small models that are either specialized or general good, generalized. And then you want to take this one for code, this one for science, this one for math, this one for general stuff. And then for the bigger size there's less and less competitors though for the O5 like at GPT4 performance we are the only one the age there's probably a lot of platform actually that we take advantage of that and serve all the models together. I think that's good, that's good for the community, that's good for transparency. It's good to not have a monopoly there. So overall I'm in favor of having more and more open source companies.

Couldn't agree more. And then because something that you hinted at, I'd love to also unpack a bit more. There's going to be kind of a generation of really small models for really specific use cases. Right. Like Code Stroll is a good example of that. For, for example. But there are also massive models that you can use across a variety of use cases. How do you see that aspect of the LLM ecosystem evolving? When should you use a massive model versus a small specialized model? I'd love to gain your thoughts on how you see the ecosystem evolving there.

I think it's always a question of price to pay and time to influence for your product needed and capability in the setup you need for your client. If you need something very very good and you cannot accept any hallucination and the 405e model is giving you enough performance so that your product is useful for your client, but a smaller model is not reaching the threshold, well you don't have choice then it's a balance with how much you are willing to Pay. For the 4B model which costs more than a smaller model. It will take also a bit longer to infer versus being better. So it's always case by case, a product question rather than anything else. Now there's smart strategies that I'm seeing more and more leveraging good models at the beginning to create synthetic data that you maybe distillate in smaller models and all that. So there are also all this stack that people are building.

Interesting. So a lot of it is depending on inference time and costs for serving these models and the use case which does definitely agree with you and I think this segues as well to my next question which is you're in a research environment that's actually shipping models that are being used in products today, which I think is quite not necessarily common for all types of research environments. A lot of the research that you produce is being used in production almost immediately as soon as it's shipped. So what's it like working in a research environment that's also delivering real world value as soon as the research is done? And, and how do you succeed in this type of environment?

Yeah, it's very new and actually it's reflected with even our organization at Meta. We had, as you may know, split between. We were all working at fair. Exactly. The foundation AI model research. And we split it in a small group at the beginning to a new org called Genai which is also responsible not just to create foundational models but and Llama two, llama three but also products, Meta, AI and all those things that are now shipped in our products. I think it's the right move and it's very interesting because we are at a level of capability through those models where it's directly useful for some products and so driving the research with the project feedback. What is useful or not? What do you need where the model is bad? Just that evaluation of those models which are general by nature is extremely hard. Having some people actually using your model for direct products, having billions of users telling you your model is bad at that and that and then telling you yeah, I saw the progress, now it's good at that. This is extremely useful.

You mentioned something like serving a model to billions of users. That's I think a challenge that only a handful of teams on the planet have to deal with. Yeah. How do you deal with such level of scale when serving like what are some of the best practices tricks you've learned along the way to be able to serve so many users with inference.

So I think one thing I can answer you technically is having super efficient small models. Which is why we kind of from the scaling laws went way beyond what is optimal compute at training time to actually be on the scaling the chinchilla trap. Basically we train it on so many tokens that it makes small models very efficient. That's the first thing. You have two dimensions, basically the size of the model, the weight or the training tokens, the number of steps and thing is at inference time, one of the two dimensions drops and not the other. Your model is big, it will remain big at inference time. But training number of tokens of steps is not a factor that impacts inference time. You can train on one beyond token or 100 beam tokens. You will have the same inference time. And so we train them on a lot of tokens. That's one way first to tackle efficient models for inference serving a lot of users now the second thing I want to add is there's huge teams at meta and that's in our DNA to be able to scale. There's super good teams here. I saw they're able to sell that to start to work on that. And while we haven't saved yet on billions of user, we are starting to see the on meta AI. I don't know if you tested it, but it's pretty good. We've already integrated generation of Reshimu and televised. That's pretty cool.

That is definitely pretty cool. And then maybe as we cap off our chat, Thomas, what current research in the field excites you the most at the moment?

Agentic barrier I think pre training has still a lot to get but it's getting more major. There's yet a lot of different things to get post training we are now starting to understand in the beginning but there's a lot of research to improve it incrementally. But agents is to me what will lead to the next many orders of magnitude more capabilities and it's the far west for now. So when I did Transformer that's when I realized we need language models aligned to follow instruction to move to the next stage. My view is now that we have GPT4 level capabilities with llama 3. That's level of capability where you start to see the preliminary self refinement capabilities. There's this nice paper Eureka from Jim Fanhood Nvidia for instance. And from now on when we will keep increasing and improving on post training and pre training, this is where you can start to do agency behaviors and.

Then maybe as we wrap up doma, what are some major trends that you expect this year to happen in the LLM space and the generative AI space? And when do you think that next breakthrough will happen? Like what's your time range for the next breakthrough if you were to put money on it?

Right. So about trends, I would say end to end multimodal integration that starting to we had Moshi the first model release. Actually GPT 4.0 is not ready released in mixed model. So the startup QT did the first release there and I expect to see more and more we will go there as well. Of course that's 12 agents. Of course we are seeing that more and more compute, optimal inference, a lot of strategy to improve compute. And actually the fact that we release for the first time such a big good model, I expect the community to play with it and to find a lot of crazy ways to make it more inferent. Funny for, I would say robotics. It's something very, very at the age, at the beginning, but I expect to see more and more progress, which is the same. It's very correlated to adjointing behaviors. But I'm also seeing the cost of robots decrease, which will make these models grounded in real physical work. Basically. To answer your question about when is the next breakthrough, I don't come from the future. I don't know. I would say something big will happen in the next five years or I'll be surprised.

Okay, so that's the only prediction I was asking for. Next five years, something big will happen. We'll see what it will be. But I think we don't. Yeah, we don't know what it is. And yeah. Thank you so much, Tomo. I think that's great. To cap off today's episode, thank you so much, Tomo, for coming on Data Frame. I really, really appreciate the chat and thank you so much for coming and sharing all things, all the great things the Llama team is working.