#300 End to End AI Application Development with Maxime Labonne, Head of Post-training at Liquid AI & Paul-Emil Iusztin, Founder at Decoding ML

Hi, I'm Maggie, VP of Curriculum at datacamp, and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI.

But staying ahead doesn't have to be hard.

That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey on datacamp.com.

How personally I see the future is that will be mostly focused on thinking and planning and solving solutions or other creative processes. We'll automate a lot of these boring mundane tasks and it will let us actually be more human, which is not that intuitive.

A lot of people try to apply fine tuning to problems where it might not be the best solution, but in a lot of situations you can get away with like few shots prompting by providing a few examples to the model everything related to a rag pipeline to retrieve the context and include it in the prompt.

Welcome to Dataframed, this is Richie. AI Engineer is perhaps the hottest role right now, but just like the data scientist role, there's more to it than you can learn in a lifetime. That means it's really several related jobs covered with a single umbrella title. Today I'm chatting to two AI engineers with very different jobs between them. They cover the whole AI workflow from conception right through to production. That means we have a rare opportunity to see the whole AI development life cycle. Maxime Le Bon is a senior staff machine learning scientist at foundation model startup Liquid AI where he focuses on post training of LLMs. He's created many popular LLMs including Alpha, Molec and Neuralbeagle as well as developer tools for automating LLM pipelines. Paul Eustin is a senior machine Learning engineer at a stealth AI startup and the founder of Decoding ML. His work focuses on pipelines, evaluation and deployment of AI. The two of them joined forces to write the LLM Engineers Handbook on which this is based and I've cherry picked the most interesting chapters to talk about today. So let's find out how to be an AI engineer. Hi Paul and Maxim, welcome to the show.

Hello Richie, Excited to be here.

Hi Richie, thanks for the invitation.

Wonderful. So I'm curious as to what you think people misunderstand about AI application development. What do people get wrong about it?

So I think that in my field, a common misconception is about fine tuning and when you can apply it and its effectiveness in general. I think that a lot of people try to apply fine tuning to problems where it might not be the best solution. And it's understandable that you want to customize your LLM because you have your specific use cases, you have your specific data. But in a lot of situations you can get away with like few shots prompting by providing a few examples to the model, everything related to a RAG pipeline to retrieve the context and include it in the prompt. All these techniques tend to be like really good for a lot of stuff that people want to do. But a lot of them try to directly go to fine tuning, which is never really the best option unless you really have to go through fine tuning. So yeah, to me this is the main misconception on my side that feels.

Like a very good news story because there are some very simple things you do. So you mentioned the idea of few shot prompting. It's like it's a relatively cheap technique, so you don't need to go all the way to fine tuning in order to get customized model behavior, I guess. And Paul, what do people get wrong in your side of things?

Well, on my side of things, I guess the most common misconception is about using frameworks to build this more on the IFR side of applications. So we all know about LangChain, Lema Index and all of that. And I seen that mostly everyone start with them. And I think that's usually like what it takes to build the application. But in most cases usually you hit some threshold where you cannot move forward. Because from what I seen, these frameworks are mostly like a very low code solution to getting into RAG and agents and all of this. And it's like similar to local solutions, they're quite rigid and you need, when you need custom stuff, you usually hit some ceiling that you cannot pass. And what I recommend is always like when you need to interact with the database. I basically ingest data and vector embedded data into your database and retrieve it, just start writing it from scratch from day zero instead of starting with these frameworks. And I believe that these frameworks are only good for very quick concepts to see that it's worth it to start writing it from scratch and the idea and the data is manageable and workable.

Okay, so at the moment the frameworks that are only good for very simple projects and you want to go a bit lower level with your code if you need to write something a bit more sophisticated.

Yeah, exactly. Because most of the time you need like very custom filters, very custom ways to store your data. I don't know custom ways to pre process, post process your data. And you will realize that if to customize LangChain or Lamadex or similar frameworks only takes longer in the long run than just writing it from scratch. But I want to highlight that there's like the other dimension to this, like more agentic workflow related frameworks like Langgraph and LEM Index workflows which realize these issues because they're made like from the same people and they're mostly used to orchestrate your logic, your steps to orchestrate them, to monitor them, to deploy them. And that's a way to go because they don't limit you in how you process your data and how you manage your data, how you integrate your workflows or agents with your current infrastructure. So these are two sides of the coin. So things like Langgraph, yes. Things like LangChain, no.

Okay, so maybe we can get into the tech stack into more detail, but it sounds like you'll really think about which pieces of technology you're using when you're building stuff. So I'd love to talk a bit more about like what your roles are because you're both kind of normally AI engineers, but it feels like your jobs are very different. So I'd love to hear about like what, what exactly it is you do. Maybe. Paul, do you want to go first this time?

So my background is more from a software engineering ML engineering side of things. So I'm like used to trading models but it's not like my bread and butter part of the job let's say. So my side of AI engineering, because it's a very vague term, is mostly related to like taking models from the researching from the trading theme and integrated into like bigger systems which can be things like more analogy like putting them on the right infrastructure to scale well to hit requirements like costs, latency, throughput. So this is one aspect of it and another aspect of it is like integrated into the code itself because most of the time you need to pre process your data between putting it into the model and the reality is that how the data is pre processed in the research part of things most of the time is not like production ready for integrating into the code itself. So most of the time you need to rewrite it to again think about requirements. And also we have to think about like a training certain skill how to manage your pipelines to actually not introduce differences because only small parts of your data are like pre processed different from how the models were straight and how it's deployed things most of the time will not go well. That's why like I think the MLOps world with the more engineer world merge really well together because both have to solve these issues. How do you deploy your model, how you design architecture, your architecture to make your features reproducible, to make your features shareable, to make your feature accessible, Add training and service time and I like this part of my job and on the other side another aspect of it is more related to like workflow and agents development. So I think that more on the like this combination of engineering, MLOps, software engineering side of things you'll be get into frameworks like as I said, Niagraph where you need to orchestrate all sorts of agents to group them together to implement right pipelines and these aspects of engineering. So to conclude I don't do that much fine tuning I guess.

But it's interesting that a lot of the problems you mentioned, they're very familiar. I think like coming from my data science background where it's like you think you're going to be spending a lot of time messing about with models but actually like a lot of the problems are just dealing with data quality issues trying to figure out how to get from prototype to putting stuff into production. So very familiar issues there that you're dealing with.

Exactly. But I will also add that because of the scale that AI engineering and LLMs introduced because usually here you need to work with bigger models and bigger scales you also add the dimension of data engineering and software engineering. So I think this role is a little bit tricky because you need to really know a little or more of everything.

And maxim, tell us about your role.

Yeah, so I have a quite unique role as working in post training for an LLM provider. So in my job what I do is like I fine tune base models to create general purpose models like a bit like ChatGPT, you can ask it pretty much anything. Right. That's not really what people do in companies. In my previous role I was more machine learning scientist where the goal was to work with someone like Paul. I would train the models and I would give the baby to Paul so he could deploy it and improve the code. Also in general like as Paul said, the code that we tend to do to pre process the data is not production ready. That's, that's Paul being nice to say It's a jupyter notebook, I can't reuse it. Thank you Paul. I think that in terms of tech roles you have these two very strategic Positions of having machine learning scientists that are responsible for the model. And the goal is really to get the data, make a good model, evaluate it, make sure that everything works. And then, and then you have the machine learning engineer who's also responsible for a lot of breadth actually. They have to cover a lot of things from data processing to deployment to inference optimization. And usually it goes really well together in my experience. And I think that with large language models we see exactly the same kind of dynamics. It's just that the knowledge, the tools, they changed a bit. The scale also changed because now you don't have to deploy hundreds of machine learning models every week, but you have to deploy one or two. But they're very big and you need to make sure that they really work. So it switched a bit like some kind of focus in what's important in these jobs. But fundamentally we still have the same dynamics.

So that's interesting that your job feels like pretty closely related to the more traditional, say traditional, but like for the last decade or so, machine learning scientist role. So it's just like make sure the model is sort of high quality and then hand it off to someone in engineering to go and put it in production.

Okay.

I think between you you have the sort of full flow of creating AI applications just to make sure the audience understands like what are the different sort of steps in order, like at a high level for creating an AI application, like where do you start and what's the end point?

I think that the main part is understanding the problem really well. Like with any project you need to really understand the outcome. You need to understand the inputs, the outputs. What we are trying do really here because sometimes people will tell you, okay, I want to fine tune this model on this data set. But actually this will not solve their problem. So you can do it, but then they're going to tell you no actually. So understanding the problem and scoping it well is probably the most important step. And then we have like different step of data collection, data generation. If needed to create a data set, it doesn't have to be a fine tuning data set. It can also be a data set for rag. It can be also a data set for evaluation. It's just like broadly speaking, dataset creation is probably the first step. If needed, you have a step of LLM training where you going to fine tune the model on a specific dataset. After that you probably want to have an evaluation framework. It's often better to start with the evaluation framework even before you generate the data to do the fine tuning. Because you know what you optimize for. You're probably going to iterate a lot of times over the evaluation data set. So it's good to start early. Once you've evaluated the model, I think I can give it to Paul, but then you are going into like rag deployment, monitoring and testing. Paul, do you want to continue?

Yeah, I will not continue like directly, but on the same page, let's say. So what I like to do is like again, more on the engineering side is to think carefully about the requirements. So one part of the requirement is the data, which Matsim already said. But other aspects of any software application in the end are the requirements of like latency, throughput costs and infrastructure. So I usually like to start laying down an overview of the infrastructure and thinking about the flow of the data, how it will run through your infrastructure, where it will be stored, other components will be connected to each other. So people can really understand how this will work in production. Because one thing is to work like on your internal clusters where everything is stitch together with your own internal scripts and all of that. And another thing is to have automated workflows where everything is basically working automatically with DevOps Pipeline or whatever other pipelines you implement. So this is another important aspect of it. And one thing that I usually preach is to start really small in the beginning. So to have an end to end workflow that works like with your minimum features, basically a mvp, let's call it. Without trying to be smart, without trying to do complicated things and all of that. But you see that everything is working end to end. And then you start using the evaluation data sets to evaluate how your application works on your benchmark. Basically we take the data set that Maxim told us about and test the whole application on it. Sometimes this works, sometimes the dataset needs some adaptation because it's one thing to evaluate your LLM in isolation and another thing to evaluate your whole application. But yeah, the idea is that you want an end to end workflow that works. You want your evaluation to be really in place and based on the evaluation you have a feedback loop. And based on that you can start exploring new features, adding more complexity and all that. You can think about it like integration or unit test. In the end, like you, you keep adding stuff on top of it and on every new feature you test it. And if the test pass, it's okay, let's push that feature. If not, then we keep need to work on it.

That's very cool. It just seemed like a common theme between the two of you is that you need to like plan what you're doing beforehand. I always find this very disappointing. I want to like dive in and start writing code and things. But actually, yeah, working out what the problem is you're trying to solve beforehand, a very useful idea. Okay, so all this sort of workflow, I have to say you've got it really well in your LLM Engineers handbook. I'm amazed how you got like the full workflow in a relatively concise book. I'd love to go into like some of the chapters in more detail because there are a few parts of this where a bit of the workflow I wasn't particularly familiar with. So you have a whole chapter on something called the RAG features pipeline. So tell me, what is a RAG feature pipeline?

Basically, at a very high level overview, you can split your machine learning system in four big types of pipelines. The feature pipeline, the training pipeline, the inference pipeline and the observability pipeline. I know for me personally the names are quite self explanatory, but I will dive deeper into the feature pipeline. So this one basically keeps raw data, which usually comes out of your data pipelines, which can be implemented by your data engineering team or in smaller projects implemented still by the AI ML engineering team. But the idea that what comes out of these data pipelines is usually clean standardized data to some extent. And the feature pipeline takes this information and transforms it into features. Right, because it's called a feature pipeline and because you are implementing a RAG feature pipeline, these features are collections that are used for ram. Right? So what we do here is basically we further clean the data if we think it's necessary for our specific RAP use case. And we also chunk it embedding or do other advanced RAM related things before indexing and loading it into a vector db. Usually, but this is like a standard use case. We can of course dig more into this because RAG is not only about vector DBs. In the end, RAG is about storing your data in a proper database that can help you in your problem and retrieve it as smooth as possible so you have access to the right context to when you want to generate answers.

So I'm curious as to what a feature might look like. If you've got a text document, then what are the features you're pulling out of this? Can you give us a concrete example?

Like a very basic example is that you have a text document and within this text document you usually have multiple entities. And the idea is that during generation, so when a user hits a question into the chat and wait for an answer, you don't want to pass to the LLM the whole document because most LLMs are like biased. When you hit context that is not relevant, for example, it can get confused and start giving answers on context that is not, as I said, relevant to your question. And then in this use case it's usually hallucination happens. So we can dig that into this later on. But on how a feature looks like. So you have a big document which can be set from like one page document to hundreds of pages document. So the core idea is to split this document into entities that are like make sense on their own. And usually you want to be as small as possible, but still relevant. So like when you take this chunk as we call it, it makes sense on its own because in the end we will pass this chunk to the LLM to generate an answer. So it's like if it's fragmented information, then it will be useless or the answer will be like not complete. So the trickiest part actually in rug is to take these documents and split them to create these packets of information that are really relevant on their own. And after we take this, we split the documents into these packets, these chunks, we embed them and store them into a vector db. And why we need to embed them is because the most popular technique for react is semantic search, where you basically do the cosine similarity between two vectors, which intuitively find semantic similarities between two vectors. So basically using these embeddings, you embed your data in the vector database which is indexed on these vectors and you embed your query. So instead of doing like classic searches into a vector database, you use this embedded query to index the query, the vector index, and find the most similar vectors relative to the semantics. And you will retrieve those packets of information and pass them to the LLM. So this is like, you know, more high level overview how RAD works behind the scenes.

Okay, yeah, so I like that idea. It's just basically about retrieving relevant bits of information so your model doesn't hallucinate. It's pulling from real data.

Yeah, exactly. You summarize it.

Nice. Yeah. So I guess it sounds like the tricky part then is still just about how do you divide your document up into smaller chunks that are meaningful in some sense, in order that you can pull out like the right facts rather than just entire paragraphs or just like fragments of sentences or things You've got to get at the right level.

Yeah, exactly. And it's a lot more harder than it looks because you usually work with unstructured data. And every document is unique, every image is unique. And that's where most of the AI problems start to fail because I try to generalize how you do chunking and most of the time that that doesn't work at scale.

And so another one of the chapters in your book is around fine tuning. So I think Maxim, this is your specialty here. So talk me through one of the chapters on supervised fine tuning. So what does supervised fine tuning involve and why do you want to do it?

Supervised fine tuning allows you to train the model on a data set that you created. And that's really good if you want to instill some knowledge, for example, in the model. Let's take an example. You work in a company and you would like the model to have some basic facts about the company. If you chat with it, it can be interesting. For a chatbot, for example, spurious fine tuning can help you instill this knowledge in the model. It can help you if you also want to change the formatting of the answer. If you want to have like a specific structure, it can be also a good solution. In general, it can be pretty good if you want to. If you have a bigger model and you want to distill it into a smaller model, can use the bigger model to create a data set and retrain the smaller model on that. This is done all the time to create general purpose fine tune models.

Okay, so it's really just about customizing what the output's going to be. I'm curious as to how it relates to. I know there's a lot of other different sort of post training techniques in order to change how the outputs are generated or what the quality of the outputs is like. Where does supervised fine tuning fit in compared to all these other techniques?

Yeah, so usually you would start with the base model. The base model has been pre trained on a lot of data, but it's only able to autocomplete the text because it's been trained to predict the the next word or subword in the sequence. So that's not really useful if you want to ask questions because it's just going to complete your question instead of providing the answer. So this is why you want supervised functioning in the first place. Because in supervised functioning we have a specific structure where you say this is the question, this is the answer, this is the question, this is the answer. So yeah, more broadly speaking, supervised functioning is pretty much always used at least to transform this base model into a useful assistant. And then you have other techniques that go on top of fine tuning. They're called preference alignment techniques in general. Another name is reinforcement learning from human feedback. But not all of them actually use reinforcement learning. So I like to call them preference alignment. So in this family of techniques, the goal will be more to tune the outputs from the model so they sound better, they sound closer to what you want. This is useful if you, for example, you have an instruct model and you see that some answers, you don't like them and you are able to write better answers. In this case you have a preference data set because all the answers that you don't like, you want to reject them and all the answers that you can rewrite to improve them, they're your chosen answers. So you can train a model on that. So rejecting the rejected answers and choosing the chosen answers, that's kind of the goal of DPO Direct preference optimization, which is probably the most popular preference alignment algorithm.

So this is like after you've done the fine tuning, you're then saying, well okay, these are some possible responses, which one do you like best? And you say in I guess you're doing either reinforcement learning or some other technique to say, okay, we want more output like these. Good answers. I know Deepseek made a bit of a buzz in recent months because they'd introduced a new algorithm for doing this preference alignment. Can you talk me through what the innovation is there?

This algorithm is called grpo and the idea is that I'm going to take a high level overview of it, but the idea is that you're going to compare different versions of an answer and you're going to take the advantage of this comparison to have your reward signal for the model to be trained. So in this case you do not need to have chosen answers and rejected answers. You only need some prompts and then the filtering will happen because you can specify some custom functions to make sure that you have the proper formatting. For example, to give an example of what I'm trying to say right now, Deep SEQ used two different filtering reward functions. One of them was to teach the model to output thinking tokens. So when you talk to deep seqr1, it will always think a lot, like really a lot sometimes, and then give you the answer. Well, that behavior was created, designed during this preference alignment process by punishing the model when it did not output the thinking tokens and rewarding it when it did it. Another thing that they did, and this one is probably more interesting, is that they extracted the answer when you ask a math question, for example, they extracted the final answer from the model and they compared it to the ground truth answer and they used that to train the model. And that's why it's so good at math, it's so good at these scientific questions generally.

That does seem pretty useful for. Well, yeah, I don't know whether it works in other fields, but certainly for mathematics science where you've got like a real concrete answer that you can test, did it get the right answer or not? That seems, yeah, very good to train against. I suppose. More generally though, if you don't have this concrete answer, how do you know whether an LLM is giving good answers or not? It feels like it's a bit more fuzzy trying to evaluate whether it works or not.

Yeah, the usual answer to this question is asking another LLM if it agrees or not because, yeah, this is pretty much the best thing that we have. This is used in other preference alignment algorithms, actually. You can have a function, one of these functions can just be calling another LLM and say, hey, is it correct or not? And get this reward signal in your GRPO training. But what GRPO gives you compared to DPO is that you do not need a preference dataset and in a lot of cases it's difficult to create the preference dataset. So that can be really helpful. And then the question is, do you have multiple reward signals? As I said, you can specify different functions, so that can be really good if you have different ways to reward models. And finally about computational resources, this is also more efficient than ppo which is the third and really heavy preference alignment algorithm. So this is also good. If you don't have the resources of OpenAI to train models, it can be handy.

Paul, did you have any thoughts on how you make sure an LLM is any good or not?

Actually I have one question for Maxim which I seen like different answers to this and I'm really curious what he thinks about it. So you have the base model and after you do this fine tuning step to basically create the instruct model that answer your questions properly. But if you want to only inject task specific or domain specific knowledge into the LLM, is it correct to take instruct model as your base model and further fine tune it on other instructions that only inject, as I said, more narrow specific or domain specific knowledge or you need to start from your base knowledge that doesn't know how to answer questions at all?

No. Yeah, you're right. You can do it from an instruct model that would work too. But it's just that sometimes it doesn't work. So it's a Bit difficult to recommend it. People often ask me like, oh, do I have to take a base model? And the short answer is no, you don't have to. And in most scenarios actually it's going to be pretty much the same, so you don't have to. But I wouldn't necessarily take the risk or at least I would try both approaches and evaluate them to see which base model works better. But I don't fully trust these instruct models to be retrained. Like sometimes it goes really wrong and.

Based on your data samples. So you need like the same data sample size, the same data sizes for both scenarios. If you move from instruct model or.

From a base model, in theory you don't need to. Right, because the instruct model already knows the structure, it already has more knowledge. So in theory you don't need to in practice if you want to have the same performance in your downstream task. Because here we're talking about fine tuning a model for a specific use case. If you do that, it's probably better to have as many samples as you can. Of course, if you have millions of samples, it's too much, you don't need that many. Right. But usually I would say no. Like I would retrain with the entire data set if I can. Because like creating data sets is often the problem, it's often the main problem in the entire training pipeline. Training models is easy. Evaluating them is becoming harder, generating the data, it's, it's really hard.

Yeah, well that's where my question was rooted because like intuitively for me, you need less data to further fine tune an already instruction model than to take it from the base model. And that's why I was curious because most of the time you don't have like thousands or tens of thousands of high quality samples and I was hoping that maybe you can hijack it with a few hundred samples.

Yeah, I'm not sure that it will work as well as for later said, but yeah, it can happen.

I have to say I like this idea of guests asking other guests questions. It makes my life much easier. I can sort of sit back and have a drink.

No, I was, I was really curious about this one because you cannot find a lot of information of this or if you can find it, it's very polarized and you tell the.

I'll make a LinkedIn post about it before you do, Paul.

Yeah, for sure. It will blow.

Wonderful. So yeah, it sounds like the challenge there is about getting high quality data and can you sort of use synthetic data from another LLM Just in place of real data set.

Cool.

All right, so relates to this, it seems like deployment is very much a sticking point for a lot of organizations. So Paul, you've just received some jupyter notebooks from Maxim. What's the next step to getting this into something in production?

Yeah, I would first ask him if we can do this with Claude or OpenAI models so he doesn't have to deploy anything at all. Jokes aside. So. So I think the hardest part is to find compute when it comes to deployment because to deploy these big models it's very costly. Right. So there are I think two main ways to deploy them is either to like quantize them really heavily and try to deploy them on a cpu. To be honest, I don't have a lot of experience with that. Only when I deployed them on my local machine by playing around with things. What I did like to deploy them more for production grade was still using quantization plus GPU models plus GPU machines. And in that way most of the time you need as I said to find a compute provider and to find like a Serdi engine which would basically serve your model. Some deserving engines are tools like V Vllm which is very popular TGI for hunting Face. These two are open source and I tested them and really like them. But you can also opt for more vendor locked options like SageMaker which we showed in the book how to do which I think it's a more more easy to do because AWS provides everything that you need like in one place. The computes, the, the tools, the framework, everything. So that's why I think that so many people adopt solutions like AWS and SageMaker because they're I think the easiest way to get started. Other options that I've seen like more community based that are really really cool is I recently tested Huggy Face's dedicated endpoints where under the hood they implement TGI because it's made by them. So TJ is made by agiface and it's just insanely simple to be honest to deploy a model you just do a few clicks like choose your compute, choose your quantization methods, choose your model, choose your scaling strategies and deploy them. But I think that this is nice and sweet for like more toy projects but when you want to deploy them more at scale you need to be really careful like into thinking what your cost should be. As I said before like do the math really well behind choose the auto scaling methods properly choose your as I said framework, properly configure your framework for requirements Again, in the end it's quite simple because these requirements most of the time boil to four aspects that I kept saying is podcast cost, latency, throughput and your data type. And another thing that I realized recently is it will make your life so much easier if you pick like battle testing tools. So instead of trying to deploy your models on the fanciest latest serving LLM tool that's out there and it's raging on LinkedIn or whatever, just sticking to the AWS ecosystem would make your life so much easier. Or your cloud vendor like aws, Azure, GCP and so on and so forth. And for example, I'm a big fan of aws. You had so many options to deploy your model. Like you can start with Bedrock for the easiest option where you don't have a lot of control but it's good enough to start with. Then you have SageMaker where it gives you more control, but it's quite costly. But I believe that is a really good way to quickly ramp up your training and inference pipelines. And then you can go more low level and actually deploy your model on ECs and EKs which are like Kubernetes or Kubernetes Live clusters where basically have full control over whatever you want to do and you can start playing around with whatever you have in mind.

I like the idea of keeping your technology stack simple. You mentioned that cost control could be a problem. I can certainly see like with the rise of these reasoning models, with the rise of agents which consume a lot of tokens that get very expensive, that costs are going to be a problem. Is it possible to predict how much you're likely to need to spend in advance? Because it feels like this is something you need to worry about very early on in your project.

Yes and no.

There's good news and bad news there.

In the end you can make speculations and predictions similar to the stock market. So you can calculate how much compute you need for your models and you can know how big your model will be and then you can say on what type of machines your model can fit, then you know how much those machines will cost, then you can assume how much traffic you will have. But again, when it comes to model sizes and on what machine you can put the models, these are more predictable calculations. But when it comes to pure traffic, that can be yes or no and they're more like low hanging fruits. For example, what if your users start to input very high documents like 3,000 pages books, because also your costs are dependent. So despite the infrastructure that your model will Sit on. You also have like auto scaling mechanism. So basically you will need only one machine to serve a few clients or more machines to serve more clients with more bigger inputs. So for example, let's assume that one client inputs an outlier, a book with 3,000 pages. Then a machine will be locked to process that, that book for a long period of time and it will start to ramp up other machines that will start to eat your money. So basically you had a few variables that are more fixed which you can predict. But basically your users traffic and the type of inputs that they will input, they're more hard to predict. You can only make assumptions.

Okay, so it sounds like users are kind of the problem. They do things that are unexpected. So I suppose, I guess the solution to that is you start small and work out see like what are your users using your product for? And then you're going to get some data and that's going to give you a bit more information on like what you need to do as you, as you scale things up.

Yeah, yeah, exactly. And I like to say that these are the type of problems that you want to have because if you have users, it means that your product and what you're doing is good, but you need some future steps to further improve it. So as you said, it's better just to start small and do some like I know some protection layers on top of it so you don't wake up with a cloud bill of tens of thousands of dollars. So the easiest way to do this is just to specify a maximum number of replicas that you want to scale. So when you do this, the worst thing that will happen is it will hinder people's experience. So if there are no more replicas to scale up, it means that some people will not be able to access the models in real time or how they were designed. And the next protection layer is that that's why you need strong monitoring and observability pipelines because that's how you gather all this information. You can gather actual real insights or on average how much it costs to run your model by good average how many models you need to have up at a single point in time. What's the whole cost of your whole infrastructure? So better start early and monitor and get the statistics the right way. And based on that, take data driven methods to quickly adapt than just assuming how much it will cost.

That's actually quite reassuring that like a lot of the secret to this is doing data analysis, like collect data on what your user's doing, how much things are costing, do some analysis and then that's going to help keep things under control. So Maxim, I'd like to talk about something I saw you were posting about recently on your LinkedIn. You were talking about automatic obliteration. So tell me what is obliteration and why would you want to automate it?

So obliteration is a technique that comes from mechanistic interpretability research in LLMs. Well, specifically it was a blog post on less wrong called Refusal in LLMs is mediated by a single direction by RDT et al. And what they found is that when the LLM refuses a prompt, so all the safety layer is actually very, very brittle. It's actually very easy to modify the weights of an LLM. So it cannot say no know anymore, it cannot refuse any request. This technique has been refined by the open source community, especially by someone named Svelspy. And this is now what we mean when we talk about arbiteration. It's about removing what's called the refusal direction in the model weights themselves. To do that we take prompts that the models refuse. We, we take prompts that the model accepts. We take the difference between both in the intermediate calculations within the model and we use this vector, like this difference, we apply it in the weights like absolute barbarians and it just works. So it's a really, really funny technique because it should not work really, but it does, it does work so, so well. So this is my interest in it. It's like, I don't know, like this is the kind of stuff that should not be allowed, but we can still do it. And recently I applied it to Gemma3 released by Google and it was very interesting to see that their model seems to be a lot more resilient to this type of attack than others. So yeah, I tried to improve the technique a little bit by introducing different parameters that you can tweak and see what works, what doesn't work. So it was a fun little challenge with this new architecture, with these new models.

Okay, this is fascinating because I think a lot of corporate use cases for LLMs, you really want to have quite tight control over what outputs are being produced and having like moderated outputs. Certainly if you've got like a customer chatbot, like you don't want it to start going off the rails and not talking about things relevant to the job. So is obliteration then a problem to having these guardrails or is this more of a sort of theoretical attack on LLMs?

I think that there's a big community, open source community that doesn't really like these guardrails in general that are baked in the models and they want to do stuff where it can also confront it doesn't really work very well. And they might get refusals when it's not really meant to be refused. So they really like these models in general. So it's more like the open source community. And then this technique can be used to do the opposite. You can create a model that refuses everything that you say. You just want, I don't know, like, what's the capital of France? And it's going to say, I'm sorry, but I cannot answer that. So it can be used to actually upgrade the security level if you want. And it can be used in other creative ways. For example, Fail Spy made a model that is absolutely depressed using this technique. Everything that you say to this model, like it will answer with a very sad tone, you know, saying that it is very sad and the universe is bleak. So you can do a lot of creative little work. What's very interesting to me with this technique is that you can change the behavior of the model. You can customize it without training the model. So it's a lot cheaper to use because there's no training involved. And despite that, you can get pretty good results for some use cases.

Ah, interesting. So does that mean it might be a cheaper alternative to fine tuning then?

Not yet. I have to say, like, I'm working on it a bit on the side and for example, little project I made was to use it for classification. I thought, okay, if I try to classify some text using this technique and I do the same thing where I have like wrong classifications, correct classifications, I take the difference. It kind of works. Like I go from 0% accuracy to 23% accuracy, but 23% accuracy is not very high. So I wouldn't say it's super useful, just kind of works, maybe with more work by being a bit more surgical in the layers and the parameters that we use to apply this technique, the that could raise the performance. But I'm not sure we'll ever use it as a real contender competitor to fine tuning. I'm not sure this will ever be the case, unfortunately.

All right, so it sounds like it's more about moderation control then, rather than customization of output content.

For now, yes.

Okay.

But not even for traverse alignment, because you said like, you can introduce specific tones to the model.

It's true, it's true. I think that for this kind of changing the tone of the model, it can be really nice. But we would need to See a comparison between a model that has been properly preference aligned and a model that has been curated to see okay, what do we lose? Because arbiteration is quite a barbaric process where you really destroy the weights of your models and sometimes what you see is that the end model is a bit dumber, like it's not going to perform as well on major benchmarks like mmlu. You lose a bit of intelligence because you were not subtle enough.

Okay, makes sense. And I actually have another curiosity and is this tactic similar to merging? Because I know that there's also like for similar architectures you directly do computation between the weights of your model without training.

No, you're right, like it, model merging is also a way to customize models without training. So it's related. It's just the way that you modify the weights is different because with model merging you're just going to use the weights of other models and kind of apply them, combine them with what you have. And here with arbiteration you're going to calculate this difference between two things and then you're going to apply this difference to the weights of your model. But I agree like this is very much related.

Nice. And actually Paul, I was sniffing on your LinkedIn as well and I noticed that you're posting that people have been predicting the death of RAG for four years now, but it's not quite happened. So I'm curious as to why people think RAG is done for and whether or not it's going to happen.

Well, people think that the whole purpose of RAG is because the context limit of the LLM is 2 limited. So in the beginning when ChatGPT went out and the same for similar models, the context window was like quite limited. I think 32,000 captains or so. Maybe I'm wrong but anyway it was quite small and that's where RAG was born because we wanted like to throw in I know companies data which is usually like terabytes, petabytes even more in size and we want to make the model give the model access to this data. But years later on Research Advanced and hence also the context window got larger and I think now Gemini has like a 2 million token or so. Maybe I'm wrong about the actual number but like this is the, the scale of the context window that it can support. So 2 million tokens is huge. Like I think you can throw in many books in there just with a single call and call it a day. So if you see it like this, it doesn't make sense to do RAG anymore because you Just complicate yourself with chunking, embedding, finding these entities, storing, retrieving and all that hassle. But the issue is actually that usually it doesn't work well, because as I said, if you throw everything into a model, it will be really hard for it to focus on the right data. Then you will actually have a retrieval problem inside your crop. Because when you input a specific question, then you need to optimize your model to be able to retrieve the information it needs for question from the context itself, which usually doesn't perform that good or is not that controllable. Let's say it's more non deterministic. Maybe you do find it, maybe it will not. But from what I heard so far, usually it's more it will not or it will not find everything you need. Because usually the model are more biased at the beginning and mostly at the end of the contest. So most of their answers are based on the end of the context. I guess similar to humans usually remember the latest sentences from a conversation. So if what's relevant to your question is at the beginning or in the middle, most probably it will hallucinate or something like that. So this is one issue, you don't have the control over the flow of information. And the second issue is again requirements costs. If you every time you input millions or ten of thousands of tokens, the cost will skyrocket and also latency because the model needs to compute all these tokens. So you had huge latency issues over there. And again now we hit like compute requirements. You need not your machines need more VRAM to call this context in memory or not. There are other engineering issues at hand that you need to somehow store or cache that context. So there are other issues. And actually RAG is still the simplest, the simpler methods where you just throw in all the data into a vector database and retrieve only small chunks of data that you actually need to. And like this, you put into the context only what you need for the specific question. And the model will focus only on what's useful. And the prompt itself, it's smaller, so more cost effective, more latency effective. So it's actually more predictable, a more predictable way to give the right answers.

Something I would like to add is that models are also not that good actually with long context. We can see I worked a lot on that evaluating the effective context length and models that pose that they can process more than 32k tokens usually like quite bad at more than that. I think that it's not that true with frontier models. Frontier models can process like pretty long context, but still like 2 million.

I don't know.

I wouldn't trust it. Especially if the answer is somewhere in the middle. It's probably a very bad idea, but it's true. Even with like 64k tokens, for example, I think that you quickly the accuracy really drops quite fast. So it's another reason why even though it's advertised as being able to process M tokens, it's probably not that effective in practice.

Yeah, you said it's right to advertise. It's similar like to phones or machines. Hey, my phone has another camera, but just to make people buy it. But that doesn't mean it will actually be more useful or not. And another aspect of it that I would like to add is that when you want to further optimize your system, we actually go away from these like two, three big models to more smaller models that are more specialized on specific tasks. And again, usually these smaller models have smaller context window where you need rep basically.

All right, so it sounds like these sort of promises of like giant context windows in LLMs that are a bit overhyped and actually rag is still a lot more useful. Like rather than stuffing absolutely everything into the prompt, you would just want to retrieve the useful bits and then augment your prompt with those bits. Okay, so I'm curious just to finish, like what are you most excited about in the world of AI?

Yeah, I think that there's a lot of focus on everything related to AGI and that it's supposed to solve all the problems in the world with just AGI. You know, like we don't really know why, but AGI is magic word and it's going to solve everything. I think like this is really cool. Of course, like I love seeing LLMs acing math problems that I don't even understand. I think it's really beautiful when that happens. And this year I'm pretty sure that it's over for humans. Like we're going to get superhuman results in every international math competition. But what I'm mostly interested in is how we can embed this technology everywhere in a way that is not forced. Because we can see companies trying to push users to use LLMs even though sometimes it's not what you want. Right. But I think that having the possibility of getting this offline assistance pretty much everywhere can be really game changing in a lot of applications and a lot of use case it can be super useful for people to learn new skills. I know that I would have loved having this technology when I was younger and I was trying to learn German through Google Translate, which was not the best way of learning German. But I'm mostly interested in that, like how can we embed this technology in cars, in phones, in tablets, pretty much everywhere where it can be useful and not just pushing for it blindly.

And Paul, what are you excited about?

Well, I'm more grounded a little and not that I love productivity tools and productivity stuff. So I would say I'm a pro productivity geek in that sort of sense. And most of my work, as you can see, is around more or less around this. And I personally see like AI and mostly this like LLM. Even if they are like the current state of LLMs or more like physics or general purpose LLMs which we can slowly see this new trend. I think this will be like the next industrial revolution. So I think that it will completely change how people interact with machines and how we will work and how our life would be. Because how personally I see the future is that will be mostly focused on thinking and planning and I know solving solutions or other creative processes. Like okay, I want to talk about a specific topic, I want to write about specific topic to create a specific piece of art. So I think that will automate a lot of these boring mundane tasks and it will let us actually be more, more human, which is not that intuitive. But because we don't have to like feel in peculiar all the day or lose so much time answering to emails or even like writing blog posts or making types of videos. We can just focus on what we want to express, on what we want to send to the world, on what we want to create and it would boost a lot of our human parts.

Brilliant. So last question. I always want some follow recommendations. So tell me, whose work are you most excited about at the moment? Who should I be following?

Personally, I really like the work made by the people at Hugging Face. Everything. Everyone there. I think it's really cool to have as a kind of center of excellence for all the open source efforts. They've been brilliant. They started with this Transformers library and they slowly expanded and now they're everywhere. Like you cannot. They standardize everything so you're doomed to use one of their stuff. But they also do like really cool scientific work. They create really cool data sets. I think that yeah, we're really lucky to have them. And yeah, if you don't follow them already, this is a really cool place to learn about the latest news, the latest algorithm, they implement them, they talk about them and they're really good at it.

Wonderful.

Yeah. A lot of great people at Hugging Face, so definitely worth looking into their work.

And Paul, let me think. So usually, for example, when it comes to more like, educational part of it and when it comes to like, teaching people, for example, I really respect Chips Ruyan's work. Like, I guess like as a engineer and writer or creator or however you want to call yourselves, like, he's. She's a role model in the sense of things. But at the same time, like, also respect a lot what Hugging Face is doing in the open space, open source space. But again, in the MLOps part of things, I respect a lot what people are doing for companies such as ZenML or Databricks. So there are so many people that are doing so much great work, so it's hard to, like, pick all you want.

Absolutely. A lot of great people are speaking on all these topics. All right, wonderful. Thank you so much for your time. Maxime, thank you so much for your time. Paul, it's been great having you on the show.

Thank you.

Thank you, Richie. It was great being here.