How To Read AI Research Papers Effectively

Sam.

Hey everyone, My name is Diana Chan Morgan and I run all things community here@deeplearning AI. Today we have a very special event and topic on how to read AI's research papers effectively. This session will be recorded and the slides will be sent out afterwards. For all the links to the papers discussed in the session, you can find them in the video description and for any questions for the speakers. We have dropped the slido link in the chat for you to vote on which questions you want answered the most for today's session. According to a recent survey, over 2/3 of developers and machine learning teams are planning production deployments of LLM apps in the next 12 months or as fast as possible and 14% are already in production. Given the rapid rate of progress and constant drumbeat of new foundational models, orchestration frameworks and open source libraries, it can be difficult to find the time to digest and read the dizzying array of cutting edge AI research papers hitting arxiva. That task has never been critical. More critical however, as the time between academic discovery and industry application moves from years to weeks. How can teams discover and read AI research papers quickly without losing nuance with an eye towards pragmatic application while balancing real world challenges? In this session we will digest multiple academic papers in real time. And like I said, all the links can be found in the video description for our event partner Today we have Arisen. Arise AI is an AI observability and LLM evaluation platform. The company's LLM observability tools, including its popular task based LLM evaluation libraries and tools for troubleshooting LLM traces and spans, rag and prompt iteration are counted on every day by top enterprises. Now I'd like to introduce our first speaker, Aparna Denakaran. Hey Aparna. Happy to have you here.

Hey, thanks so much for having us. We're really excited to be here. Hi everyone, My name is Aparna, one of the founders here at Arise and I run product here myself. I come from a technical background so was an engineer ML engineer for many years where I built and deployed models and felt the pain myself of having to actually manage them in the real world. Which is what got us to build Arise. It's never been observability and valuation has never been more important, especially in the age of LMS and generatives. So excited and super excited to be working in this space.

Absolutely happy to have you here. And now I'd love to introduce Amber. Welcome.

Hi, thanks for having us Diana. My name is Amber. I'm An ML growth lead at Arise and I primarily work on bringing the almost like developer advocacy angle, making sure folks can understand how to implement observability into their own models. And and I come from an academic background. I was an astronomer and I kind of transitioned into this new role of data science, which was a new role in like 2017. And from there going into like machine learning and really wanting to know like why, why do teams retrain like every two weeks or every six months? Like why this arbitrary number? And I wanted to fully like understand the kind of, of responsible AI angle and how to observe their models in production so that they could retrain only when is necessary or when things start breaking in production.

Absolutely. Well, we're super excited to have you here today. So many of our community are here with us to learn more about how can they get caught up, how can they learn what's hot, what's happening, how to put it into production. So I'll let you guys take it away and we'll dive into it.

Perfect. Thank you. I will share my screen. Had to take it out of presentation mode to share. We all know tech is hard, right? Yep, Classic.

I can see it now. Amber. Perfect.

All right.

Awesome. Well, let me jump in and give you guys a little bit of an overview of what we're going to cover today. And hopefully you guys get a lot out of this today. So first off we'll do a quick overview of why you should read research papers. Not just the data scientists on your team, not just the CTO or the leaders. Why should you specifically read research papers? We'll go into a quick overview of how do you find the right papers to read. There's just so many coming out. So we'll point you to some quick resources that we use, people we follow, great places to find good content, give you a quick overview of different types of papers. Not all papers are the same, so some of them are going to be slightly give you an overview versus some of the more interesting ones you're seeing People deep dive on Twitter are more of the breakthrough paper. So we'll walk through a couple examples of survey papers, data sensitive benchmarks and then the so called break breakthrough papers and then we'll actually do a little live paper reading so you guys can see how we break down a paper, what's important to take away what's in background knowledge. You might how do you go dig into that background knowledge and then lastly jump into you yourself can contribute to the growing space of research. Awesome. Let's go to the next Slide. So why should you read research papers? I mean, in the last year so much has happened in this space, in the space of AI. I mean, literally for those of us who are even in the field, the pace is insane. From, you know, everything from GPT4 launching to now, you're seeing really, really viable rivals in the market. With Claude, Opus 3, Google, Gemini, you're seeing not just all these happening in the world of text, but also you're starting to see GPT v the Vision come out. You're starting to see. Many of you guys have probably seen the demo of Sora, which was really, really cool to see everything from mid journey. I mean the pace is moving really fast. Not just on the research side, but we're also seeing the pace move on how people are starting to build applications on top of these. And with just how fast the pace is moving, the best way we've found to stay on top of the field, to really know which foundational models to be using, which ones, what the boundaries of, you know, what should you be doing around things like instruction, fine tuning. You really want to stay on top of the field, read the research papers directly from the facts itself. And I think being able to do that in a really, really efficient, quick way is going to help you stay on top of the field. So, and this is just another, you know, why do it? I mean really, the market size is just going up. This is your typical, you know, right, access, everything is going up type of side. But we're seeing a lot, not just on the research side, a lot of investment, obviously from the foundational model teams, but also from organizations starting to deploy and build these applications. Maybe last year you were seeing a lot more Twitter demos. We're now actually starting to see real applications and products being built on top of this. Everything from applications like chat to purchase, customer support, chatbots, to really complex use cases that are powering revenue for organizations. So it's happening, it's happening also on the research side. So this is just a little, little view of the number of papers actually AI papers and arXiv has been actually growing exponentially over the last especially 24 months. We've been feeling it just keeping on top of all the papers in this space. And so there's a lot happening in the field right now, which just means that you yourself will probably, you know, as people are consuming it, there's a lot to then have to communicate back and keep everyone else updated. And so having that skill to do it yourself is going to be immensely valuable. And to you Amber. Great.

Perfect. So, and just a few resources on like how you can get started. I know Diana is putting in the papers that we'll be talking about and the deep dive paper, so those will be available. But in terms of like how we find those papers, we have a lot on socials, so we're constantly looking at LinkedIn, X Slack, Discord, YouTube following, AI newsletters, attending paper reading events. Paper reading events have gotten bigger arise. Like we do bi weekly paper readings as well and then meetups. So going to meetups like discussing these papers and then different websites. I would say things like Forbes, AI, VentureBeat are really good for if you're in industry and you want to know like what other companies are leveraging. And someone on our team said that they actually just asked GPT4 for papers on a given topic. So you can try that approach and you will be having access to these slides. So don't worry about like screenshotting or taking these all down now. But just some people that we follow on Twitter who are on kind of the cutting edge of the research that is coming out in the field.

All right, one maybe note to add on that. Just, just going back, I think we've been seeing. You probably agree, Amber, but Twitter is where people are discussing this right now. It's. We're seeing people drop paper summaries or the latest papers that they're reading and kind of dissect it and having that conversation online. So definitely, I'd say even if you're not a fair person, it's been a great place to learn. And then like Amber mentioned, there's a lot of awesome paper reading resources. So I think Amber linked one of them, which is the YouTuber Yannick. He's awesome. There's a lot of great deep dives on papers, but there's also, like I mentioned, we do bi weekly paper readings. Latent Space is another one that does really awesome paper readings. So find a paper reading community because even if it's one every two weeks, that's one more than you're probably doing yourself.

Awesome. Thanks Aparna. And Aparna and I wanted to add in those resources because we had already gotten questions leading up to this event about where do you even get started on finding these papers besides just kind of entering in keywords in Arxiv. So those are some great options. And a lot of the time we're not just finding those papers from Arxiv ourselves, but we follow people who fully commit and look at Arxiv every day looking for those breakthrough papers. Or the diamond and the rough type papers. And when you're looking through all these papers, what you might start noticing is they fall into certain categories. So Part and I have divided the papers that we find into three categories, which are surveys, benchmarks, and data sets, and then breakthrough papers. Note there's probably more categories, there's definitely more subcategories. But as a general sense, when you find a paper, it's likely to fit in to these categories if it's being talked about and if it has a lot of promising, promising results. So surveys give a detailed overview of a certain topic, they provide a summary spread of where the field is right now in a specified area, and normally they say survey on them. These papers tend to be very long, very detailed. And in terms of what your goal is for reading these types of papers, it's to understand kind of just what's happening in the field and to look for trends and patterns. If you're in the the area of AI research or you want to learn more about AI research, looking for trends and patterns from these surveys is key because you'll know where most of the research is focused and what people are spending the most time, effort and money on in terms of research categories. But also you might be able to understand if there's an area that's not very well explored. And then you can start like kind of diving deeper into those opportunities and then to learn a broad overview of the space and how it's being evaluated. So if you're looking for like, so you can look for benchmark and data sets, those papers are great. But if you want kind of an overview of all the benchmarks and evaluations that are being released, then it's great to have a survey paper because they're going to try to get their hands on every model and every evaluation metric that they can in order to compare these different models. What you're not going to try to do is get a technical deep dive into a specific model. Those are more commonly from technical reviews. And you're not expecting to be introduced to any like, new or novel idea. Those tend to come from through, like the breakthrough papers that we'll get into later. If you're in the area of AI, you've probably heard of RAG or retrieval Augmented Generation. We're not going to be going too in depth into this paper, but if you saw this survey and you are implementing RAG or you're considering a RAG implementation, essentially implementing RAG gives context to, provides context to your LLM. So, so that when maybe a user interacts with your LLM application system, they get very specific responses. And so if you're looking at a survey that's focused on retrieval, augmented generation, you want to be looking at the different techniques being used and where it fits into this space. You're not going to use a survey to look at different techniques or like advanced RAG methods or, or advanced evaluation methods or RAG failure points. Those are going to more likely come from benchmarking papers. But understanding where something like RAG would fit into the space when you already have other optimization methods like prompt engineering, fine tuning. Surveys are great for understanding where a certain field is going and where subfields kind of fit in overall to a larger area, like where RAG fits in overall to generative AI. One of the most popular surveys that came out, this came out September 2023, the survey for Large Language Models. If you checked out this paper, tell us what you thought about it in the chat. Again, I'm not going in depth into this paper because it is 90 pages. And you do tend to want to go into something like a survey of large language models for specific reasons. These surveys are great, incredibly comparison. But again, like, even once these surveys are out, there's this, this field has just been moving so fast, as Aparna mentioned earlier. Like, the pace is incredible. So there's a lot of models that are now being used and tested that weren't available in September 2023. And even though that's just like a bit over six months, it's huge and puts a lot of these models that have been looked at kind of on the back burner of a lot of people's minds. So for the survey. Oh, yeah, yeah.

No, just one thing maybe to add if you're, you know, I'm just noticing some of the folks in the slido asking questions like, what are good papers to read? How do you know what are credible? Survey papers actually are a great spot to, you know, they kind of pick out the, the key ones that spurred on other great areas of research. They give you that bit of lineage that you're seeing in this image right now of like, okay, the discoveries made here were then later brought on over here and later brought on over here. And so they do give you this sense of, you know, what were the key milestones and breakthroughs to get to where we are right now. They're not going to be on the cutting, cutting edge, you know, have the latest breakthrough that happened a month ago. It might, if it came out a month ago. But I think, I think that's the trade off is you get caught up really and we'll do this. Actually when we go to the Mistral paper, sometimes there'll be terms that you don't know or those peoples were actually refer back to survey papers to say, hey, we're not going to go into this, but if you want to get an overview, go look at this. So just for those folks of you who are asking in the slido, these, these are great places just to start because it's kind of the papers that quote unquote made it.

Yeah, absolutely. Thanks for that, Aparna. Yeah, so like as Aparna saying, like this is a great place for anyone to start and kind of why we're talking about survey papers first. Even though kind of the breakthroughs come out first and then the benchmarking and then they end up in a survey. The surveys give you just an overall layout and whether you're, it doesn't matter if you're an AI engineer or just someone interested in this space, it gives you a great overview of kind of everything that happened. And I think it covers a year and a half of models and everything that's kind of advanced in that field and really makes you see just how far we've come. Like just reading this paper and I'm like, oh, this paper from a few years ago, like no, it's, it's just what's been happening and what can happen in six months in a really fast evolving field. And so just kind of like takeaways like you would read this paper a lot of times because if you're using one of these LLMs or you're thinking about using one of these LLMs, being able to look at the comparison and a lot of different comparisons is great. Like if you're a team that's debating open and closed source capabilities or you're debating like how, you know, how should you maybe train up your own language model or how are some of these models pre trained? Because they're going to give you the most information they can. Obviously. Like they're not going to know everything about the pre training because that information is not out there for the closed source models, but comparing pre training, data curation and prompting methods and then even though these don't go in depth into the architectures, they do mention like which one, like which models are like encoder, decoder, decoder only in the size and the different parameters. So you can see that side by side for those comparisons. And then who I see reading survey papers that don't often read like the in depth research Papers are kind of teams that are managing budgets and looking at comparative cost compute the hardware components. Those are really important and they're mentioned because kind of breakthrough papers and benchmarking, you're trying to get the best performance as possible. But still the surveys tend to look at performance as well as all the other factors that go into enterprise large language models, then being able to review the comparative capacities and evaluations. What's great is all the benchmarking and data sets you could want are available in that survey. They're going to try and do every benchmarking method that they can, put that in the paper and then you can actually replicate those tasks and those results yourself on any model that you choose. And you do that by leveraging these benchmarking and data set papers. Aparna, anything else you wanted to add before we go into benchmarking and data sets?

That's awesome. Thanks. Thanks for covering the. Yeah. And feel free folks. If there's any questions on the survey papers to keep dropping them in the slido. We'll. We'll definitely go through all of them at the end.

Yeah. So yeah, we'll have them up and we do have a good amount of time for questions I think after we get through the breakout, the breakthrough paper. So keep the. Keep that in mind and we can go through papers retroactively. So benchmarking papers are used to introduce a data set for testing or to introduce a new evaluation approach on a data set. So some really popular benchmarks that we've seen are the MMLU. So this is how well LLMs can multitask. Hello swag. So kind of testing common sense truthful AI essentially checks against spreading of misinformation. And then there's image ones as well. So chart QA is a benchmark for Q and A, but with charts and visuals and breaking down benchmarking papers. Like the goals of a benchmarking paper is that you want to see like, can I use this, Can I use this to evaluate a new data set? Or can I see the capabilities of this model I'm using? Like, would I have better insight into this model and can I learn the limitations of the model based on what I'm evaluating? So this is really comparing the like once you've selected a model or if you're in the process of selecting, seeing what the outputs are. So considering how to even expand on certain benchmark capabilities, I put this one in because that's sometimes a goal when I'm reading benchmark papers just to see if like the evaluation metric is one I would use. And if it's not like, is it, is it slightly like more similar to something I could use. Like if I changed it out just a little bit or added a bit more data or modified the data, would it be a better metric for my own use case? And then non goals again, this isn't introducing new capabilities. They're not like introducing a new model or new architecture. This is benchmarking that architecture. Architecture and non goals detailed model breakdown. So you're not going to get like, you'll see when we go through the mixed rule paper, you're not going to get that kind of breakdown.

And if you go back actually one slide Amber the one more actually. So this is actually a screenshot from the Gemini technical report and what you'll see on that one is basically they did a number of comparisons of themselves. I think we just left out that part of the screenshot but themselves and then some of the other model providers on these benchmark benchmarks. So mmlu, you see hello swag in there. Each one of these benchmarks that you see in these technical reports have their own paper. It's like the equivalent you can think about it is like, I don't know the, in the, you know, if you guys are all familiar with, with the ImageNet data set that, that's really popular in the computer vision space. It introduces. Some of them will introduce a data set, most of them will introduce a data set and then they'll introduce kind of a way to, you know, what's the, what's the way to evaluate across that data set and what does that test actually mean? And so especially in the LLM space, there's, there's kind of a lot of, I'd say right now people starting to feel like, you know, there's definitely these benchmark data sets, but they're kind of public and they're known. So are the LLMs themselves trained on this data set? Is it part of their training data? I mean there's, is it truly running these tests blind or is this actually being part of the training? Just. That is something that I think a lot of us in the space wonder right now just given that the data sets are open.

Exactly. And being able to test these is why so many leaderboards are coming out like the open LLM and chatbot arena becoming increasingly popular where you can test different large language models. And specifically looking at open source large language models to be able to test against. Aparna mentioned, you know, this happened first in the image world, like looking at like the imagenet competitions, which really benefited deep learning, research and computer vision. But yeah, benchmarks are essentially that next step you take after you get a breakthrough paper. When you hear big claims and it might be a technical report, it might be published by the team that built. Might not be, but if you're getting a breakthrough kind of publishing, you want to be able to see like, is it actually living up to the hype? Is there a way to kind of demystify some of that hype a bit? And so we're not.

You guys are probably also. Sorry, just one more thing to add on that last slide. So if you guys have all you guys have probably seen this, but this is actually from hugging face, the OpenLM leaderboard. You guys can all go check it out today. We'll drop the link, but it's literally a leaderboard that has the latest open source foundational models that are coming out and then it has the metrics that you guys just saw us go through. Mmlu, the Hella swag. Yes, that's a real metric. Truthful qa. All of these are actually metrics and you'll see how these foundational models are actually stacking up against those metrics. And the point of this is just, this is when you have all these foundational models, each one of them can just report their own results. Oh, it did great on xyz. But what a data set paper. It's just so important because it's, it's, it's kind of the only standardized way that we can look at the performance of these models and see how good it is. So even though, yes, it might have its flaws of we don't know if they trained on that data set or we don't know how truly blind it was. It's kind of the best approach that we have right now to know, like, is Llama 7B better than Nistral or better than GPT4? So it's been super helpful in that way.

Awesome. Yeah, try out the LLM leaderboards for yourself. Great way to also stay involved in the community. Like if you're working with these models and just kind of showing like what you or your team have been creating. So we're not going to get into each one of these papers. But yeah, as a Brian said, like, each one of those metrics has its own papers because it's the data set and goes into the full discussion of how that data set was curated. But hello, SWAG is essentially trying, it's trying to test your common sense. So trying to complete the Sentence. Because these, it's like 70,000 questions and only one right answer. So multiple choice, only one right answer. And the other three answers are adversarially generated. So they don't make any sense. So when this hello, SWAG benchmark came out in 2023, it's testing against a lot of the different large language models because the human ability is about 95%. So 70,000 questions, human ability, like 95% because maybe they misread a question or like didn't, didn't understand one of the questions. And that's great, right? You're able to kind of test against this. But what happens when, you know, GPT4 is now pretty much at the human capability for these metrics. And this is pretty much critical thinking around benchmarking because the finish line always moves, especially when these models are getting more and more performant and just getting better and better. So even though, you know, some of these, you know, results don't make any sense for like, for a question of like, a woman is outside with a bucket and a dog, the dog is running around trying to avoid the bath. She, she gets the dog wet and then it runs away again. Or does she jump into the bathtub with the dog? So being able to get to human level completion isn't always going to be enough. And so that's why these benchmarks keep coming out and they keep getting more and more advanced. So we've pretty much discussed these for, like, why reading a benchmark paper, you want to use that, you want to use that benchmark in your own work, you want to compare it, you want to look at cost, performance and potentially modify that benchmark better for your use case. For example, hello, Slag. Now that you know GPT4 is getting 95% same as the human capabilities, you might need to make those questions a bit more difficult in order to accurately test that, especially when these other foundational models are getting closer and closer to that level of success. What to look out for. So we talk about reading these papers, like when you see a benchmark, like the things to be a little bit critical of, like, you know, thinking about bias in the data set when working with these text models and working with something that could be a little bit subjective, checking to see if there's one definitive answer. MMLU has been getting a bit of feedback that sometimes there's not a single definitive answer to one of their questions. And so it like, not always like the best option or sometimes there's not enough context provided with the question. And so it becomes really difficult to answer. Like, if you say. I think, I think one of the examples was asking about like a mortality rate. If you don't give enough background context on the demographic, the age, natural causes of death, then the LLM might hallucinate. And so that. That's just kind of a bit of information about it. And then if you're seeing benchmarks around kind of the mathematical explanations and equations, if they got the answer right, but the process is wrong, like they got the answer right, but they got it the wrong way, like, does it count? And this is kind of the show your work, show your explanation aspect because you don't want these language models essentially memorizing the answer. So it does demystify these LLMs a bit. But just keep in mind, like the capabilities of benchmarks and sometimes these data sets, they're not always perfect, but they are an amazing way to start understanding the capabilities of LLMs. And again, you can replicate these evaluation tasks on any model that you choose. Parna, anything on this until we go into the paper. The breakthrough papers.

Yeah, I mean, I think, you know, and just to note by replicate, you probably won't rerun the actual, you know, entire data set itself. There are already published results. But I think it's more. What we're just trying to get at is, you know, these data sets and these evaluation metrics might give you a general sense, but you might want to modify or pick a smaller subtask that's more important to the application you're building. And so you might. These are slightly. All more model evals, which are slightly different than task evals, which is more of what we see application builders doing in the LLM space. So model evals just help you pick the LLM, use the existing results they probably provided to do that. The task evals, I think, give you slightly more. Is it actually going to work on your task that you have? And you can, you know, I think learn from some of the stuff that the benchmark papers do to build your own.

Awesome.

Awesome.

All right, Aparna, take us through breakthrough papers.

Cool. All right, so for those of you who've been waiting for this one, this is, this is probably when you guys are all seeing the goes mentioning the papers on Twitter and the discussions that are happening in the space right now. A lot of the paper readings are probably on these type of breakthrough papers. These are the ones that is getting all the hype right now. And there's a lot of. And the reason for it is there's a lot of breakthrough papers coming Out. So some examples of one, we'll drop these again, the sides and the, and the links to all of them. But there's been a lot of breakthrough papers of course on the foundational models, but then also just some on the architectures itself. So we'll go through a bit of those today and we'll just do an example paper reading and maybe you can go to the next slide. Amber Yep. So I think just, you know, some high level key points on this is the thing you're trying to do when you read a breakthrough paper. And for some folks have been asking this question in the slido, how do you distinguish between a good paper and a bad paper? Which ones to spend time on? I think just what are you trying to read? If you're trying to learn about a novel area or novel idea, just first confirm that it actually is a breakthrough paper, it's not a benchmark or survey paper. And then if it is a breakthrough paper, what's the novel idea that's really being introduced? This is the key thing that we're trying to take away when we're reading a paper and we'll go through this in the Mistral Mistral paper and the most important thing is what's the novel idea? And then what does that mean? So what, so they discovered something or they found some new approach for how you can get xyz. It's faster, it's better performance. What does that actually imply? Is there more applications that can be built on top of it? I think that's kind of the what and the why does it matter is kind of the two big things that we're trying to take away. When we read this paper, you probably in these papers it will not give you the full, you know, if there's a term you don't know or there's, you know, it relies on some previous breakthrough paper, it's not going to give you a whole deep dive or an overview of the, you know, like if you read today we'll go through the Mistral, the mixture of experts paper and you'll see that they just link out to other papers. So they're not going to give you the full explained version of what it is. And it's also might not give you all the details on the exact test that they ran. We're seeing that as a trend lately. They don't talk about the data. We'll go through that in the mixture of paper. But it really is the what and the why does it matter that really try to come through in these papers. So with that I'm going to actually go ahead and share my screen here. Do that. Let's see if I can share this screen over here. Okay, can you all see this?

Yes, the front end, we can see it.

Okay. Oh, it's not the, the whole screen or just some part of it?

It's part of it. But I think if you zoom in like. Yeah, we could see that.

That make it go wider, I guess.

Yeah, yeah, it's, it's wider there.

Okay, let me just confirm on the. Okay, it's, it's wider now. Okay, let's go through it. And then if you guys have any issues with the, the sizing of it, I'm happy to zoom in more on, on the paper itself. Okay, so this is the mixture of experts paper you guys know, came out by at the French startup and they are, I'd say probably building the most open of the open LMS foundational models. So this one's actually under. It's even more than, I'd say the stability AI when this one's under the Apache license. And so this one's been getting a lot of stir actually in the ecosystem because its performance for being a much smaller model is really, really impressive. So first off, what we do is we're going through a paper is I always just start with the abstract and the intro itself to first figure out what the heck this paper is even talking about. So first off, you'll see they're introducing, they're going to mention this here. They're introducing Mixtral 7B. It's a sparse mixture of experts model. You'll notice he said it has the same architecture as Mistral 7B, which is another paper actually. And they briefly mention kind of some of the architecture which we'll get into, which is the novel part of what they're actually adding. And then they confirm that basically with the changes that they made, it vastly outperforms llama 2.70B on math code generation and multilingual benchmarks. And so clearly what they're getting at is, hey, we've changed some core things in the architecture of this foundational model and because of this it's led to some better performance on XYZ areas. They've also released, this is the other key part, a model that's fine tuned specifically to follow instructions. So this is the Mixtral 870 instruct and they mentioned that this one actually surpasses even GPT5.3.5 Turbo Claude Pro Llama 270B on, on, on following instructions. And so this is kind of, you know, as I'm. We were talking about what are the key, key takeaways when you're reading this paper, they're going to go through, they're going to talk about the architecture of this model and then they're going to probably talk about the results that this model had on specific data sets and compare kind of those results to these other top contending kind of foundational models. So that's kind of their. The goal of this paper as when I read this in the fact that they spent a big chunk of it talking about, okay, here's what it's doing. The difference of each layer being eight feet, four layers. As I'm looking through this, I'm like, okay, they're actually going to talk about the architecture of their model. And so to me, I know immediately this isn't a survey paper, this isn't a benchmark paper. They're going to get in the depth of this and then it's probably a quote unquote. Some of you guys asked on slido, a very relevant paper to me because I'm in the space of, you know, looking and comparing all these different foundational models. We see all of our customers using them. So understanding kind of where it does better, where it doesn't. What they did in the architecture is really important to, to me. So that's kind of the reasoning. I would, I would jump in. Let me keep going here. So just confirming Ember, you can see I'm scrolling down to the introduction section.

Absolutely.

Okay, awesome. I just don't have the streamyard kind of on, on me right now.

So based on some folks questions just how. Because like we, we chose this paper, we knew this was a good one. Like, how did we know, like, how did you know that this was going to be a paper that basically we wanted to talk about, like in the breakthrough session, like, how did you find it and what were the key things that helped you understand that it was in that category?

Yeah, good, good point. I mean, to be honest, for all of y', all, the, there's, there's a lot of hype around this paper. So first I'd say is like external social attention being given to certain papers. And if you follow the, the Twitter folks that will, you know, we shared earlier on the call, you'll notice all of them have like talked about or reshared this paper or had a discussion about it. I'd say as you read more and more papers, there's things in the papers that you'll start to be Able to weed out which ones are really going to stick around and which ones. Maybe the breadth of that discovery wasn't. Or the breadth of the depth of that discovery wasn't that important. I think it's just a. You, the more and more you keep reading these types of papers, you'll start to notice, oh well, this one really all it did was apply it to a new application or you know, it took this and it, you know, it only took it XYZ further. I think in this specific case, I mean clearly it's making big claims. It's beating and GPT 3.5 on a much, much smaller model itself. So that's really, really. It doesn't even use all of the tokens or sorry, all of the actual feed forward blocks. Yeah, exactly. And so it's really only using two experts to process the entire token. And so even though the token has access to all of the 47 billion parameters, it's really only using 13 billion. I mean it's really. That, that's kind of one big takeaway, I guess, as, as you're reading the abstract of it. And then I, to be honest, I'd say the best answer to that question is probably the people you follow will kind of probably start highlighting them. And so that might be a good place to start as you guys are first trying to build your own foundation of papers. Also, the paper readings will typically highlight the ones that everyone's talking about. So yeah, it's, you know, the reason I say that is sometimes there are. And women. We'll go through it as we go through this paper reading. There are papers that were like really awesome when they released, but they didn't get that kind of attention. And then it wasn't until later where it's like, oh, actually this paper made that claim so far back, but you just, you really probably are a researcher in this space if you're, if you're going and reading at that depth.

Yeah. And that's just like one question there, there are a hundred. So I'll just let you keep going. But yeah, we're also, we're not saying that and mix rule isn't saying like Mistral is not saying like they invented this. This was already invented before.

Yeah, yeah, yeah, and we'll go into that. So, so first off, what, you know the, the introduction of the paper basically goes through and we'll read some of this together. You'll see they presented 8.7 B. It outperforms. It only uses. This is a subset of the parameters for every token it allows for faster inference speed at low batch sizes and higher throughput at larger batch sizes. This is the key part of the, you know, overview of the architecture where it says the decoder only feed forward block picks up from a set of eight distinct groups of parameters at each layer. For every token the network chooses two of these groups, the experts and combines their output. And so here you know, the what in this paper is kind of like the TLDR you're going to find in the intro section. They've kind of already told you the TLDR is basically a mixture of experts, which is what the mixture paper is. It's got a faster inference. It's already the, the pre training of this, if I go down I'll mention this, the pre training of this was actually faster. You can mention that it demonstrates superior capabilities. This is what they're going to go through. And then they also talk about Instruct which is going to be the specific fine tuned model that does better on actually following instructions. So we already kind of heard the TLDR of what this paper is going to focus on. The next thing we care about is the architectural detail. So this is the actual depth of the paper itself. And for those of you, you'll notice here, this is where they'll sometimes do it says we'll present a brief overview of the mixture of experts layer for more in depth review C12 and they actually started actually leaving the architectural details with it's based on transformer architecture. They link you out to a paper and uses the same modifications as link you to another paper. And this is all kind of what they assume is pre knowledge itself. So they're not, they're not even going to go into what a mixture of experts actually is. This is actually where as a reader you might actually want to decide. Okay, I don't actually. If you don't have that background knowledge that's in the first paragraph, this actually means you should probably go get that background knowledge if you really want to understand what was the level jump that this paper did. And so with that I'm going to actually jump over to another screen. Let me share this one over here.

People are asking like if you use any additional tools when going through these papers. I will say for some folks that arise like using like plugging, you can even plug things into like GPT models and say like explain this to me. You can use GitHub like copilot, you can rerun the source code like a lot of times looking at this, well depending on what it is. But Sometimes you can they give you an option to like kind of run a version of it if you have access to like, like an API key from a certain model. But those are really helpful being able to like use external tools to just verify sometimes the math, sometimes like the explanation and having them as a secondary tool.

Yeah, yeah, let me try to keep going through this because I know we only have like another 10 minutes or so and then we'll definitely jump into some of the questions here. I'm going to just hop over to the, to sharing this from Google here just to hop between the papers easier. So this is the architecture section you notice that they mentioned. Okay, well it's based on transformer architecture, uses the same modifications. If you don't know what a mixture of experts is, well the first place that you might want to start, which is super, super helpful. And they actually link this in this paper. Actually mixture of experts started with this paper back in 1991. This is where they first, first introduced what is a mixture of experts. This is actually a really great diagram of it. And what it is you'll see high level diagram here is that basically for every input there's some kind of gating network that essentially decides which expert to basically pass it to. And that is essentially what decides which one will be generating the output itself. This is the very first paper that actually introduced this. This is probably what you would call the OG breakthrough paper itself. And then post that there was actually a lot of other work again there's probably even a whole survey. This is a survey paper actually on sparse expert models in deep learning. You can see this one's actually from the Google team. But in this one they actually go through and they really explain the differences. It's again difference between a dense model, difference between a sparse model and they actually go through, for example, probably not gonna have time to go through all this. But the actual, for every input itself, how does it decide which expert to actually get passed through? And so these are actually all linked within the mixture paper itself. And so if you're reading a paper and you get stuck on a certain section or you don't understand it, the links that they have are actually super, super helpful. And sometimes finding your own survey paper or finding your own where did you know where did this actually start? Is helpful just so you have the base knowledge to understand what this paper specifically is adding or contributing to that research. Yeah, and so in this case. Go ahead.

I was just going to say like, you know, before even, you know, doing the deep dive, did you look like, you know, if you just like look through kind of this, this paper for like things that jump out at you because. Yeah, well, there's so many questions. Yeah, maybe we just talk about like what we notice. Because for me, I honestly go to graphs first. Like after I read the title, after I read the abstracts, I probably look at conclusions. But a lot of times the graphs are the first thing and I thought can you go to the first. Like they have like one benchmarking graph in here Because I think we all also like read papers a little bit differently. I'm definitely a visual learner. And so like first you're seeing like, okay, things are looking pretty good. I know that. I think the reasoning they use, hello Swag and hello SWAG can get 95% human and GPT4 is getting like 95%. So seeing things a little bit lower and that's just kind of like what I first take away. Can you go a little bit lower there, Parna?

Yeah, yeah. And yeah, Amber, maybe I'll do another. Just two minutes of the paper and then, yeah, I'll probably jump through the questions. I agree there's, there's quite a lot coming in. The. Maybe the big takeaway for all of you all is sometimes the paper reading will require going to previous papers to actually understand it. I guess we won't have time to get through the mixture of experts. But the key, key takeaway, this is actually from a previous paper is that instead of an actual, you know, in, in something like a. Let me get a good image actually, this one's fine. This is actually a good, good view into it. There's a gating network basically decides which experts to pass it through before, you know. So for every token it gets converted into an embedding. The embedding then gets passed forward through the attention mechanism. And then instead of the feed forward blocks, there's actually the mixture of experts layer. And the mixture of experts is essentially instead of each token going through its own feed forward block, it essentially decides to go through one or two experts. The key thing is you have to figure out what are the right experts to pass it through. This is all just kind of pre knowledge that existed and this team just borrowed from. So all of this is just the pre knowledge of how actually the mixture of experts and all that work in their key paper. The main thing that they're doing is essentially the. And they don't mention all of this actually is they are definitely using sparse mixture of experts. So not all of the actual, you know, the, the Gating vector itself is very, very sparse. So this is actually what's determining which expert to pass it too. So this is a key part that's important in this paper. And then the distinction between the total parameter count versus the sparse parameter count is kind of a key distinction they make between what did the model actually use versus what was actually provided to it. The rest of this is all just kind of, you know, how did they actually get a. To run at scale? They used expert parallelism. This is another technique. And the rest of the paper, you'll notice, go straight into results. If you are just not, you know, aware of all the, you know, or this is a lot of background knowledge, I'd say go start. You know, this is a great way to add papers to your reading list is to go back through each of the links that they provide, add them, understand them. You could also, if you're just trying to get the gist of a paper, just jump to the result section. So in this case, this is kind of what, what Amber was saying. Sometimes she'll just jump straight here. And in this case what they did was they got straight to the results. They compared Mixtral to LLAMA against the popular benchmark tests that Amber was just going through. And you can actually see the performance of LLAMA versus Mixtral across all of these different benchmarks itself. And so you can go through, you can read what are the actual results itself. So here you can see Llama GPT 3.5, Mixtral 8.7B. And then there's a deep analysis of basically the results. This is actually the bulk of the paper. They don't spend a lot of time in the architecture section at all. The arc, this is actually really, really sparse compared to a lot of papers, you know, in the past, just like half a page to a page on the architecture. But the majority of this is actually spent on the result, discussing where it's better. You'll notice that they have a section on multilingual benchmarks. So, and this is probably the only bit where they actually mentioned the training data at all. In this, they said they significantly up sampled the portion of multilingual data during pre training. This is why Mixture performs well on multilingual benchmarks compared to, you know, the llama 270B model itself. And so you can, you know, they're not giving away a lot at all actually on, on the actual architecture of the paper. They're spending most of it on the results of it in, in here.

And I think, and I kind of laughed at the one Go up Aparna the. Yeah, like this image here. So first, yeah, they don't tell us anything about the data. They give us, like it's this type of data, so no information about the data. It's great that it's, you know, it, it's publicly available, great license. But for this, like, this is essentially a benchmark, like what we were talking about earlier. And it's all green. So it's, you know, like, what does this really tell you? If it's all a hundred percent, like at the. For me, this graph doesn't really give me anything. Like, if something is all green, like you want to be able to see kind of differentiations in it. So if it's kind of like not giving you that, you need to probably change your benchmarks. The perplexity option, like, it's great. Like it's great. Like, oh yeah, we can add 30,000 tokens and the perplexity isn't going up at the end. Essentially we're not losing information, but we do see that the best information, once you get past 10,000, like a context window, it kind of falls flat. And then maybe we just do kind of takeaways for. What does this mean for future models in terms of like speed, parameter count. Parameter count, throughput.

Yeah, I mean, I think the, the biggest takeaways for most people is, well, a mixture of experts, sparse mixture of experts is performance wise, doing pretty well, which is, you know, they're showing the performance metrics across here. So that's kind of one big takeaway. The inference is faster. They have a bunch of tests on that. And then, you know, we didn't totally get to it, but they do do a decent amount on instruction fine tuning. And they talk about where it's not as, you know, where it's not as good, which is, which is really important. And so as you're reading and going through this, I know we, we had to kind of rush through this one, but in a lot of the paper readings, what people spend time on mostly is we'll discuss the architecture, talk about the big takeaway. But then really the, the important part at the end of this is where is it, you know, for something like this, where they do a deep analysis of the routing analysis, which is which of those experts to actually pass it through, that's a really, really, you know, this is probably a next.