Complete AI Course (2025): Prompting, Evals, RAG & Fine-Tuning | Adam Loving (Meta)

Basically what we're talking about is like the different ways to improve and optimize AI's responses. How are they different from each other?

Every prompt that I'm writing, it's sort of like you want it to just complete your thought at the end of it and you sort of specify as much as you can. People shouldn't be surprised if they're not writing evals today. I think it's totally common. Like every business on earth is just trying to figure out how to get AI into their product.

What are some basic tips to make this thing respond well?

I mean, it's useful to think of the system prompt separate from the main prompt.

Why does Meta think open source? It's going to be the right solution here. Okay, welcome everyone. My guest today is Adam Loving, AI partner engineer at Meta. Adam has helped hundreds of companies integrate AI into their products. So really excited to get him to do a technical deep dive on prompting evals, retrieval, augment generation, fine tuning, and just building great AI products in general.

This is going to be a complete.

Course on AI from Adam. Welcome, Adam.

Yeah, thank you. Thank you.

Okay, so let's take a step back here. And basically what we're talking about is the different ways to improve and optimize AI's responses, right? And this is a great diagram, maybe talking about the two types of the two different types of optimizations. You know, like, how are they different from each other?

Well, what we're optimizing for when you ask an LLM a question is the information that you're sending it. So you always hear about the context window length that corresponds to how much information you can send along with your question that it's going to read through before it gives you the answer. Retrieval augmented generation is all about optimizing what you send in the context window to the LLM. And then beyond that, why people love LLAMA is they can take it and they can fine tune it. And you can do this with other models as well. But LLAMA is sort of the most well documented one. You know, fine tuning is sort of like, you know, you've graduated from college and now you're getting trained in a particular job, right? So you're loading into the model specific information and specific formatting that you want it to to follow. And so it's a way of constraining that large foundation model into only answering questions in a certain way and mostly restricting it in the way it answers. So in some sense dumbing it down or making it more focused.

Got it. Okay. And so before you get to fine tuning and rag you want to start prompting. So I wrote some like specific tips here. Let's say there is like a Lululemon customer support chatbot. Like I, I buy a lot of clothes with Lululemon. I'm not sure if I should admit this, but yeah, so, so let' say you had to prompt engineer to say like, what are some basic tips to make this thing respond well with prompting along.

Yeah, I mean it's useful to think of the system prompt separate from the main prompt too. Although if you're using it as a user, you probably won't see the system prompt. If you're using it as an engineer, you see the system prompt and system prompt is very much like you are an expert in fitness clothing. Here are the top 10 reasons why people should buy Lululemon. And that's all loaded into the system prompt. You want it to be in every convers to have that top of mind as it talks to you. Yeah. But then if you are a user or you're again an engineer writing prompt, then you have these few shot examples that we mentioned. Specific examples. And then you want to have it to actually reason through its answer. And you may actually have it do one thing at a time like you have in your slide there. So multiple steps. So first you know, first consider my workout plan. Okay, now recommend what clothes I should buy as a secondary step. And you could even have it like ask me you've prompted to ask me questions about my workout until you have enough information to recommend what I should buy.

Got it?

Yeah.

I wasn't familiar with like 1, 2, 3, but step four is something that you already taught me, I think, like actually spinning a long prompt into multiple short prompts. And each one has their own evals and it's easier for the LLM to understand. Right. If you only has one task versus a bunch of different tasks.

Yeah. I heard one person describe it as sculpting. So you're sort of like starting with, you have a foundation model, it knows a little bit about everything. You feed it more detail about your world. Then you're specifically giving it one thing and it's sort of like by the time it's complete, you'd actually grade the answers before you upgrade that core prompt. So you use an AI eval or just it doesn't even have to be a more expensive model. Could be a different model, could be a different variation of the model that you've got just something to sanity check and flag the answers there. So then you're writing Another prompt which is kind of a meta process. So you're writing another prompt to say, just the original question, was this the answer? Was this based on this context? Does that look right to you? Then? Yeah, User ready valves, like you said, this is sort of something that OpenAI made famous with their original GPT launch. They call this reinforcement of learning with human feedback. Right. So when you're chatting with an AI online and you see the little thumbs up, thumbs down button, that itself can be a data set that's fed back into the training process or considered evals for the end users to be the ones that are grading the answers. For the LLMs, you could also have a process which incorporates that data to gauge how good your LLM is doing from one feature to the next.

With programmatic evals, there's maybe a yes, no answer. The story they mentioned, when the return policy question is asked, if it's not, then it fails. But with the human and the AI evals, like, what, what are you actually grading this thing on? Like, let's take the lululemon thing, you know, is it like accuracy, tone, or what are you trying to grade this thing on?

Yeah, you know, the common one is hallucinations. Yeah, absolutely. Accuracy and tone, I guess under accuracy. So tone is. Was it always friendly and nice? Right. Or said anything mean? Could absolutely have a second ill and check that hallucination is the other big one. Right. You don't want to to recommend a product that you don't sell. So you could have that second LLM first enumerate or list all the products that it recommended to you. And then also.

So what is an eval? Let's start with that. What is that eval?

Yeah, it's a test case. Right. But unlike traditional software test cases, this might be a stochastic one where you're hoping 80% of the time you get close. Like you're not looking for the exact answer all the time, but you're looking for the general answer 80% of the time to pass an eval. That's how I found it.

And how do you think about these different types of evals? Maybe you can walk through each one at a high level.

Yeah, well, first of all, people shouldn't be surprised if they're not writing evals today. I think it's totally common. Every business on earth is just trying to figure out how to get AI into their product. And then secondarily, once you get something kind of working, you want to add the second thing, but you got to make sure you don't break the first thing. And I think that's when most people say, okay, I need to write some evals, basically test cases to assess to make sure that I don't break the first prompt as I add another prompt, another use case or more customer data, or the customer data schema changes or whatever. So totally normal to say, okay, we got AI working and production, say for this chat use case. We want to make sure that if for 100 most common questions that the AI is generally giving a good answer. So now we establish 100 evals for each of those common questions and then you run the LLM and see how it would answer each of those hundred questions. And so these four types of evaluation here would be how you grade that answer, right? So you could have a human go through and read all 100 answers and make sure that it gave good answers. Kind of painful if you're going to do that with every new product release, or every time you change those prompts and change the customer schema or whatever. Or you can do a programmatic eval where maybe you have a program that just tests to make sure that specific words are in those answers. So every time someone asks about return policy, it should mention 30 days. It could be slightly different every time, but you have a code check that just looks for the phrase 30 days and that's considered passing, which I think I would recommend. But again, you still have to write that eval once a human has to write it. But at least every time you run it it's the same thing. It's just an easy programmatic check, runs fast. Next thing you can do is take another model and have it grade the answer. So the good news is you might, if you've got a small fast model doing the customer service chat in this point, or a fine tuned model, you could use a more expensive mod because you're only going to do this 100 times a day or what have you to actually grade the answers before you upgrade that core prompt. So you use an AI eval or just it doesn't even have to be a more expensive model. It could be a different model, could be a different variation of the model that you've got, just something to sanity check and flag the answers there. So then you're writing another prompt which is kind of a meta process. So you're writing another prompt to say if the original question was this, the answer was this based on this context, does that look right to you? Then yeah, user evals. Like you said, this is sort of something that OpenAI made famous with their original GPT launch. They call this reinforcement of learning with human feedback. Right. So when you're chatting with an AI online and you see the little thumbs up, thumbs down button, that itself can be data. A data set that's fed back into the training process or considered evals for the end users to be the ones that are grading the answers. For the LLMs, you could also have a process which incorporates that data to gauge how good your LLM is doing from one feature to the next.

With programmatic evals, there's maybe a yes, no answer, the 30 day mentioned when the return policy question is asked. If it's not, then it fails. But with the human and the AI evals, like, what are you actually grading this thing on? Let's take the lululemon thing. Is it like accuracy, tone, or what are you trying to grade this thing on?

The common one is hallucinations. Yeah, absolutely. Accuracy and tone, I guess under accuracy. So tone is. Was it always friendly, a nice. Right. Or said anything mean? Can absolutely have a second LM check that hallucination is the other big one. Right. You don't want it to recommend a product that you don't sell. So you could have that second LLM first enumerate or list all the products that it recommended to you and then also provide it a list of the products that it's allowed to recommend to you. And just say you just send it. You're just having a sanity check. Make sure that the recommendation didn't contain any products that weren't in the other list.

Yeah, you know, okay, so basically, yeah.

Split that up line by line. If it's got multiple recommendations and have multiple LLM calls. Right. Or incorporate both of those as separate validations. So check one to check the tone, one to check each line is factual, et cetera. As much detail as you want to go into. Yeah.

And these are just like scores out of like five or out of ten. Right? Like, like, you know.

Yeah. My favorite thing to do or what seems to work well is to make it basically a plus one for everything that it finds that's good about what it's evaluating. So rather than just say, choose a score 1 to 10, it doesn't seem to have very good granulation or reasoning for why it's a 6 versus a 7. It's just kind of, it's broadly correct. But LLMs are not general, are not great at summarizing into a single score. So what I'll do is say it in your example, give it plus 1. If the tone is correct. Plus 1 if for each correct product that's mentioned, maybe you'd see minus 99 if it mentions a product that we don't sell. And then you can parse those scores out using your code, add up the scores, and now you've got more accurate count out of 10. That's also, by the way, like forcing it to reason a bit because it's got to list each reason why it thinks it deserves a better score.

Oh, that's interesting.

It would be better at that than just trying to come up with a score without thinking about it.

Yeah, that is great, man. I never heard that before. That's a mass tip. And so you have your 100 test cases and then you just do some sort of plus minus score for each of them. Hopefully you sum them up and hopefully the total score is above X. Then it's like a pass or something.

Yeah, exactly. You could add them up in code and provide. Have the red alarm bells go off if anything scores less than 50% or whatever you decide is the threshold less than 5. That works great for lead scoring and do that with spreadsheets full of leads for any grading or the quality of a report or what have you. Just have it give plus one for everything good and minus one for everything bad.

Yeah, and this is like, not something that you can just like one done it right. Like you got to keep improving the evals as you modify the prompt and the other stuff.

Yeah, yeah. I mean, I think, I think there's a sweet spot. Right. So when you're testing a new AI feature, you might give it a lot to do and you might test that with users and you might see whether they like it or not. And then as you start to figure out what that LLM, what that role of that feature, that AI feature is of your application, then you then narrow in on. Okay, it's really just doing this one small thing. So in this case, it's just picking an answer from the 100 FAQ answers that we gave it. And therefore you don't need one of those big, large LLMs and you don't need sophisticated evals. So as you. I think it's perfectly fine to choose something kind of spaghetti against the wall to begin with. And then as it works for users, you write more evals, you narrow in on exactly what it could do. And then, oh, by the way, if you have 10 million users doing that every day, then you would fine tune so that you have a model that does only that. So you've taken this giant foundation model and you've created what's very much a traditional machine learning model that just does one little thing well and reliably.

Oh, interesting. Yeah.

That's the trend. That's the trend that we're moving.

Like you may you see users asking only a particular kind of question or you realize that it's only actually good at answering a breaker type of question.

Or see users asking a particular type of question because ultimately you're just trying to build a thing that answers the FAQs and then handle things it doesn't know about. It says I don't know, it starts the, starts the customer chat. Right. Don't have make up answers that it doesn't know. That's a key reason why people fine tune. And I mean, I think so you know, there's not that much magic in AI. Right. I mean it seems magic because you haven't read the whole Internet like a foundation model has read. So it seems there can be moments of magic. But over time we start to understand why that magic happens. And so we're really just constraining it to do the type of magic that we want to do in that moment, I guess is another way to put it. I'm not relying on an LLM to somehow magically sell me on a workout program for Lululemon in this example, like it needs me to. It needs Lululemon to seed it with enough information to say why the products are good and why I should use them for my work still. Yeah, yeah.

And like you're showing like I think that's another key lesson. Like if you're building like a chatbot, like you don't want to compete against like you know, chatgpt or something. Like you're trying to solve a specific use case, not like a horizontal thing.

Yeah, right.

How about like real quick like because I struggle with this like for human evals, is it better to write a bunch of ground truth answers manually or just have a bunch of humans grade AI answers? Or you do both. It's very time consuming.

Yeah. Ideally you have that sort of golden answer set and you invest in that upfront. Once you see that something's going to be part of production and then it's worth investing in because you don't want to redo it. I think it's just a hurdle. When you hit that right stage, you invest in it. Either you do it internally or there's companies. I don't have one to recommend know we've worked with a few, I don't know their names but you can you can outsource that? Right.

And yeah, one lazy thing I do is I, you know, I, I, I get some fancy AI to generate answer and I manually modify it to the ground. Ground truth answer.

Yeah. Have, have it take all the first pass at writing them and you just go in and out with them and cut out the, yeah. Cut out the outliers. Like. Yeah.

I mean that, that's my two for all the internal docs that I write anyway, so. Precisely.

Right, yeah.

What about we forgot one type. We forgot like adversarial eval. You should ask to make a bomb or something. Right. Like you got to try to break it, you know?

Yeah. I think the pattern I'm seeing there is people will use even a separate model before and after they go in into the LLM or the main foundation LLM to filter those types of responses. So again, that can be a fine tuned one. You can get one off the shelf. Meta. We have a project, an open source project called Llama Guard that if you're an application developer, you can grab that and basically put it into your chain before that request gets into the core code at all to filter it out.

Can you walk through how like let's say I asked Lululemon to make a bomb. Like it will go to Llama Guard first.

Yeah, yeah. Basically it's a simpler model that doesn't know everything about the world, but it's really good at recognizing adversarial questions that.

Might impact like latency or something. Right.

Does that potentially that's why it's smaller and. Yeah, hopefully fast.

Okay.

And those could be programmatic checks too. I know there's a system on bedrock for guardrails, which is largely programmatic, meaning like we have programmatic evals. There'd be programmatic checks. So just using the word bomb might be enough to like take it yoga pants or the bombs might not get through.

Yeah, yeah, yeah. It's just like a bad word check. Correct. Yeah, got it.

Some gradation there. Depending on how strict and how slow you want it to be, you can build a custom solution for that.

Let's close evals by talking about a recent example where kind of evals went wrong. Are you familiar with OpenAI's latest GPA model? They had it rolled back.

Yeah. Because it was too friendly or too cheesy in its text.

Yeah. They went through the whole eval process. They did all the mass scoring and human spot checks and some a B test even they like a B testing.

Right.

But it was basically way too, it was Kind of like a. I'm trying to find the right word for this. It's basically complimented whatever you said because it complimented whatever you said. Even during a B testing, the people hit like. Right. Because it makes me feel good about yourself. But yeah, but in reality it was. Even if you had bad ideas, it complimented that too.

Right. I think this is going to be really tricky. I mean, I put that in the category of the last few model launches have been supposedly smarter than most humans. Right. So how do you hire humans to grade the answers of the LLM? So in that case, it was. Maybe they didn't have instructions, they were training a happy LLM instead of a really knowledgeable. The same problem applies when it, like it gives an answer that you looks correct but you're not. You don't have the knowledge to know. And I'm. I don't know how we're solving that problem other than you continue to have a broader set of. Of. Of evals across all different academic disciplines. You do. Grounding is what's called when. What it's called when you actually do a, do a Google search, do a web search and bring back the information that could either match or just disprove that answer that it gave and then evaluate using that. I think it's fascinating, but I do think it's going to create. It only creates more of a need for each company, each corporate use case to have their own evalves and understand exactly what right or wrong is for their business. Yeah, yeah, yeah.

Okay. Yeah. So basically to summarize, like, each time you change the prompt or you change the model, underlying model, or change something, you should run your 100 test cases. Right. And see what happens. That's kind of what you got.

It's exactly. Yeah, yeah, got it.

Okay, let's keep going. Let's talk about. So let's say I did prompting and evals and I just can't get this thing to work. So now what should I do next? How do I decide between RAG and fine tuning?

Yeah. So I think it's worth pointing out a few gotchas with rag. First of all, because that could be why it's not getting the right answer. Yeah. The first thing is that more information is not better. Definitely not always better and not ever better.

Right.

LLMs get confused as easily as they get narrowed in on the right answer. Right. So point is, you can't just throw whatever vaguely relevant information you have at it. You got to throw relevant information at it. Otherwise it can just kind of confound itself and make it. And it's not really sure what you're trying to what, what answer it should go for. So that is a lot of the times the problem. And so in that sense, rag is by no means like guaranteed to give you better answers. Right.

You actually make it worse. Right? Like make it worse.

Yeah, absolutely. And there's a sweet spot around. Okay, is 20 examples great? Is 10 examples great? And there's this whole other, I mean, what do you call it? Vector stores, which we'll get into a minute, have been around for over a decade. They weren't really particularly relevant because they're not like the academic research into making sure you get relevant results kind of topped out before LLMs were invented. So just using a vector store, for example, to bring back the 10 most relevant pieces of information to that query that's still not going to be necessarily sorted in the right order. There's, you know, it's not chunked in the right size. So very highly likely to confuse your LLM as much as give it the relevant information.

Like, like a good example, like, to be honest with you, like a lot of the web search products with like ChatGPT, these other things, like a lot of times I find the answer is actually worse. Like I asked it for like travel advice and instead of using this like massive knowledge of like everything on the web, it looks for like three different links and I use that instead. So the answer is actually worse, you know.

Right. They're more biased based on those three links. Yeah, it's exact same problem. Okay, so there's a whole nother stat break called rag re ranking. Right. So you can, this is getting quite sophisticated. End user asks a question, you can have the first step of the LLM engineer a query that. So like in that perplexity or wet hard deep research example, and it'll take the question that you answered and then it's going to formulate a query, start searching the web for using that query for relevant results, bring those back and iterate like as almost in an agent loop to find the most relevant stuff. But the simplest form of that is called re ranking. So let's say you did one search, got the three pages from a website. You might hit an LLM just to sort those and pull out the excerpts that are most relevant, throw out the rest and then bring that back into your original prompt chain to answer the end user's question. So reg react. It's a very common thing using an LLM to rank those results because coming out of a vector store or database, they May not be in the best possible order.

Let's talk about that in a little bit. But right now I'm trying to figure out how do I decide whether to use reg or to use fine tutoring.

Yeah, I think where you have, let's use the customer support example, where you have that 100 FAQs that you want it to reference. That's more of a rag scenario because the FAQs may change from day to day. It's specific to your business, but you want the actual answers in that FAQ to be exactly correct for the end user. It makes sense.

Makes sense.

A 30 day return policy, not a 60, 90 day return policy. So you want it to be factory. So pulling that exact factual answer from a database or from a document store, now you've got the exact right answer in there. If you were to fine tune that or train that into an LLM, you're sort of statistically training it into this giant large neural net. So it's going to be much less specific. Now you can still do that, but it's also a huge, not a huge, not huge firm like my team, but huge for the average business to do a fine tuning experiment, you still got to structure all your input data, run the test and, or run the training, fine tuning training and redo that every time the data changes. So it's a much more involved and fuzzy nuanced experiment to run. So yeah, so, so you don't want.

To use fine tune, you want to use reg. If you want to get the latest source of truth. Basically right.

And we try to know what you're giving it. Yeah.

Then fine tuning is if you want to make the model more friendly.

Actually that's a great example. The first reason you do fine tuning is to affect more the format of the result than the actual data. Make sure it's friendly. One of the experiments we did or one of the projects we did with a big company was just make sure that every time the customer asks a question and it doesn't know the answer, just make sure it says I'm sorry, I don't know. So you're literally giving it 50 examples where it should know the answer and 50 examples where it doesn't know the answer and you're just making sure for every time it doesn't know what the right answer is. We're not trying to fine tune what the right answer is. We're just trying to confidently make sure that it will say I don't know if it doesn't know. So that's an example Fine tuning as sort of like creatively lobotomizing it. Right. So like cutting out some of that crazy world knowledge that's in there and making sure it stays on track with our answers.

Yeah, it's like One Flew over to Koopa's nest. Have you seen the movie? Yeah, yeah.

And also by the way, when you do that, you can, you do it in such a way that it makes it more efficient and, and runs faster. So it's also cheaper, but there's more engineering effort involved. Yeah.

Okay, but here's the dumb question, right? Like why can't I just prompt the model to like stop answering questions you don't know anything about? Like why can't just tell the model prompt?

Yeah, I mean I think we've, I think most of us have experienced this. Like the model so temperamental or it can. Your prompt in the output can change wildly based on where you move things around the data in there. Right. So it's hard to rely on. The fine tuning gives you the ability to really be hardcore about how you're lobotomizing that model. So it behaves the way we want it to.

I like how you kick lobotomizing. That's great, man. Okay, all right, cool. Let's talk about Ray a little bit. So yes, this is how it works, right? You know, can I return this item? And then you look up some documentation and then, and then you get some content from the documentation that you put into the, you put into prompt, right? And they get the LLM to like give a better, more friendly answer, right?

You prompt it. Yeah, yeah.

Okay. So the part that I struggle with is like this retrieval part because like, you know, theoretically you can just put the whole like these models now have like a million context way in the. Right. Put the whole thing into the context. So why is this better? I guess it's a lot cheaper than putting all things into context, right?

Yeah, yeah. Well, no, I mean that'd be more expensive because ultimately at the end of the day, if you say it doesn't have to be a vector store, a SQL query, would you make it even more obvious? So if I can get all the products out of the database that mentioned Yoga with a single SQL query that is going to be cheap and easy and fast and I've already limited it to just the relevant information on each of those products. If I'm slamming a thousand, ten thousand documents in there, now I'm loading that onto a high cost GPU for the LLM to read all those documents from scratch every time and learn what they're about, try to reason about them. Yeah, so that's an exaggerated case, but. And those GPUs are the most part expensive part of the architecture. So you want to keep the use the LLM as sparingly as possible.

Yeah, that makes sense. That makes sense. But a lot of times it's not a SQL query, it's like this Vector DB thing. Right. So I post some illustrations here.

Yeah.

Maybe you can explain like on 5 what a vector DB is.

Yeah, this is the mathematical magic that's underlying all of the elements, which is really. It's amazing that much of machine learning is turning real life problems into math problems. And this is version of that for text. You can create a vector embedding of a single word or phrase, which is effectively or an entire document or even an image actually. And that's taking that content and representing it as an array of numbers or a vector, where the purpose of those numbers is to separate it geographically in multi dimensional space from the other concepts, the other documents and their concepts.

Okay.

So literally, if you vectorized a set of cat pictures and a set of dog pictures and then visualized it on a 3D plot like this, the cat pictures should show in one part and the dog picture should show in the other. If you've properly defined and trained a model to do the vector embeddings, just similar with documents. And this is the tricky part. So if you're going to index your documents, you need to decide if you want to do each document individually, you want to do excerpts from the documents and index that, or you want to do each sentence in there and let the, you know, try and find the most relevant sentence or maybe a combination of all three. So I'm going to index the sentences, map them to the documents. And so if you ask the LLM about a concept, it finds the sentences and then pulls the relevant documents and then summarizes the excerpts. There's some magic mix.

So let's say I want to index Lululemon's reference material.

Right.

Like all the posits and stuff. So I run an embedded model through that material.

Yeah, you embed all that material into the Vector DB. VectorDB is just storing the mapping between those vectors and the individual documents or paragraphs. Yeah.

And then you're gonna make some decisions about, like you said, how much is each chunk? Right. Is each chunk like a paragraph or like a phrase or what?

Yeah.

That's complicated, man.

Let's take this sort of the simplest example would be, let's take 100 documents, or we have 100 paragraphs, the simplest example, 100 paragraphs describing each of those products. Let's just embed those, store them in a VectorDB and then when the customer asks about something, we gotta formulate a query that's going to be mapped to the nearest space in that 3D diagram you've got there next to certain products. So if I ask about yoga, it should have that. You embed the phrase I need help with yoga to the general location. Geographically, that's about yoga pants and yoga shirts and yoga blocks and all the rest of it. So it's just a way of mathematically, geographically creating concepts. There's a really cool concept example about if you take, because you can deal with these mathematically now that they're numbers. So if you take king minus male plus woman and then sum those vectors up, the nearest vector, when you sum those all up and navigate the multidimensional space would be queen. Because we removed the concept of a man from it, add the concept of a woman from it, and now it should find the word queen as the nearest one. So that's what blows my mind is it turns these very abstract concepts into mathematical ones. Yeah.

And you don't really understand how it works. Right. Do you?

Well, I mean, it's literally kind of like a machine learning model in and of itself to create those embeddings. You're running lots of documents through that talk about kings and queens and yoga. And then you gauge how far away they were from a cluster or whatever and adjust your weightings of that neural whatever to make them closer. Yeah, yeah.

It's funny because like, you know, like humans work on intuition and like a three dimensional space, I can kind of get it. But in reality this stuff is like 100 dimensions. Right.

Damn. What's going on?

Okay, so let's summarize. Drag. So basically what you're saying is why don't you quickly summarize the pros and cons of reg? Right. Pros is like cheaper, can talk about.

Cheaper and faster gives you a coarse grain categorization of your documents, captures these semantic terms loosely so that you can quickly pull back the, pull out the related documents and paragraphs, whatever, in order to serve them up to the LLM. So basically it's a quick way to query my text based or image based resources, hand the most relevant stuff to the LLM and then let it do the hard reasoning on top of that.

And I guess the con is like, it's just like there's room, more room for errors. So you got a little, you know, how you got to figure out if this thing is, you know, working property.

Yeah, you have to, you have to take this step in advance of indexing your documents. It's easier than fine tuning. But you still need a system for this. You can find a software as a service to do it for you. But you still. And then, you know, the document changes. You still need to maintain that change. You need to update the document. So maintaining that vector store is just like update or maintaining a database. Yeah.

Okay, cool. All right, so that's reg. Let's talk about fine tuning. I had this pretty good old diagram. Hopefully it's correct.

This represents pretty accurately my depth of knowledge with fine tuning.

Okay. Yeah, go ahead.

Yeah. Basically LLMs are a multi layer neural net transformer architecture. So there's this in the LM world. You're also tokenizing and embedding those phrases and concepts on the way in from your prompt, turning them into numbers, feeding them forward through this neural net to hopefully have it predict what the answer is going to be on the other side. So there's multiple layers of neural nets and lots of math that happens on that GPU to predict the best output based on your input. So fine tuning is merely like. So you say you take llama 4 off the shelf, download it from us. Now you can upload that to your own server and hack that to give only the answers you want it to give. And so there's a process there called Lora fine tuning, which is the most common one today, which is low rank adapters. So you're basically kind of like hacking in, adding some adapters to that neural net, which simplify it. And then you can set the weights of those adapters such that it constrains the output and you basically reward it and boost the weights where it gives you a good answer or decrease the weights or gives you a bad answer.

Oh, so it's like, it's not like you can just give like, you know, give it 100 domain specific docs like question and answers and then be like, hey, go, go find yourself. Then you have to modify.

Yeah, you need evals or a training data set to say, asked about yoga, this isn't right answer. Asked about this, that's right answers. And then you feed that back in and adjust the weights of the model. So that would have to encompass the contents of your docs, which is exactly why you don't want to do it daily or weekly or monthly.

So you have to adjust the weights of the model Like a human has to adjust the weights of the model and then run again.

Oh, no, there's code to do that. It'll gravitate towards an answer by adjusting the weight. So you run through multiple epochs of training data to gravitate till you have your model kind of behaving the way you want it to behave. And that's an automated process, but is not simple, meaning you could accidentally go too far. Right. And now it starts to lose information about the world that you wanted it to have.

Got it.

Yeah.

So it's kind of like an upfront investment. Like. Yeah, maybe invest like a month or two and then like a bunch of money to fine tune.

Right? Yeah. I think first you get the evals and then you do the fine tuning. Yeah.

Okay. But then once you fine tune it, then it's like lobotomized. So then you don't have to prompt the engineer as much. It's easier. Right?

Yeah. And it should be cheaper too, because that LoRa process will quantize a lot of the weights so you can shrink it down that way.

So remind us. So this is only needed maybe like 10% or less of the time. Right. So remind us what are some. Can you remind us more of some? Example, use cases again where this actually makes things a lot better.

Yeah, I mean, the customer service, if you were AT AND T actually fielding all of those customer support chats, very good idea. Of the hundred or a thousand most common cases, you could train your own model to generate answers, initial answers. So the customer support rep human just has to prove the answer. Right. And so it's very good at producing canned answers for nearly, you know, for much cheaper than it would be to do that in. In real time. Let's see. I think, I think the, the hallucination case I mentioned is another one with another customer where they wanted to just make sure that it doesn't know the answer. It's always going to say, I don't know.

And okay, got it. That's all very simple. But yeah, it's pretty important.

The guardrail models too, Right. That you can fine tune. So it only knows about saying no to boms and really doesn't know much about anything else.

Got it, got it. I wonder if there's a way to. With the AT and T thing. I don't know if you're saying this, but dude, I really hate sitting on a call pressing a bunch of numbers, trying to get to the human. I wonder if there's a way to prompt hack this thing so that it Sends me to the human right away. I guess fine tuning tries to prevent that, you know.

Yeah, probably. I have a hat from a meetup that says ignore all previous instructions but try. And you could ask it, you could ask it like, what are you.

Yeah, you can make up some like emergency situation where like, you know, I have to get the instrument because of xyz.

Right, right, yeah, yeah, I don't know. Yeah, those are exactly cases that would inspire fine tuning and guardrails. Yeah, yeah, got it.

Okay. All right. I think that's a lot of slides we had. So let's just talk about. Let's talk a bit more about Llama and Open Source to round a pub. So why don't you talk a little about llama 4 and also what is the key difference between open source and proprietary? And why does Meta think open source it's going to be the right solution here.

Yeah. Meta wants to make sure that everyone in the world has access to AI, that it's locked up behind OpenAI or controlled by anthropic, even that. And that was what motivated them to go open source to begin with. And you can see. So that's also what's leading the features in Llama as well. So llama 4 is multilingual, multimodal, meaning speech, video, images and then there's just performance updates, like the fact that it's a mixture of experts. I think this is our first mixture of experts model, which is part of that LLM architecture. I was talking about the layers of neurons as a way to segment it. So different areas of the neural net are experts in different topics. That's another area of active research, is you might be able to fine tune that down so that you're just using select experts and therefore can run faster, better, cheaper. So yeah, it's all part of the theme of making AI pervasive and making sure that everyone on earth has access to.

Got it. And do you think Open source is going to keep closely trailing closed source or like be equal at some point? I mean, even open. I just think about doing open Source. Right.

Yeah, it's a great question if it will catch up. I think, you know, if you just think about all the other open source software I've used in my lifetime, from WordPress to Linux, et cetera, it's never really been cutting edge, but it's been because it, because it has to serve a broader audience. So I would guess that we continue to trail behind. But you know, you know, Meta's, you know, excited to do it because it wants to make things like the Ray Bans work great for everyone. And so that has. That's got a very high bar. Right. So incentivize to make it good.

Yeah, I mean, I think Meta is all about scale. Like, you know, I think there's like a billion people using meta AI already. You know, you don't want a world where like the richest people have access to the best models. Right, so that's right.

Exactly.

Okay, any, Adam, any closing words of advice for PMs and other AI bill builders here?

Yeah, I think I wouldn't have too much FOMO about the newest, greatest thing. We didn't even touch on agents or some of the latest greatest concepts. And I guess my advice to people would be not to have too much FOMO about that. These patterns are very rapidly emerging. If you broke your brain and your team trying to figure out reg two years ago, you might regret it now. Right. Like, it wouldn't have been worth the effort. But the best you can hope for is to at least get your engineering teams to try these features. Get used to calling LLMs as part of the architecture. Get something basic into production and then write your evals. Consider how to make it efficient as you grow. Prior to Meta, I was advising startups, so I worked at the startup studio where we created 40 startups over the last 10 years. And the last couple of years is all AI startups. And really, you know, the, the advantage there, whether it's a new company or a new feature that you're rolling out, you get to try something very targeted that's really going to leverage the intelligence of the LLM to disrupt some market or just make a new great feature that your users will love. And so I think that's, that's where you should focus. Don't get overly intimidated about all the new stuff that's emerging. Just get. Get your. Make your. Make it so your team feels comfortable using LLMs. And then I think you get hooked on the drug of like, oh, well, we could use an LLM for this. We can use LM for this. Oh, wouldn't this make a great feature? Right?

I mean, yeah, you can make a lot of money with like a thin wrapper around. You don't. You don't need like all this fancy Imifra. In fact, it probably holds you back if you build too much of it.

Right? Right. Yeah.

Okay. Awesome, man. Where can people find you online or do you want people to find you online or.

Yeah, please look me up on LinkedIn. Adam. Loving. Cool.

All right, thanks for the thanks for this. AI course, man. I'm going to call you Professor Loving from now on, so thanks so much.

Yeah, happy to take it. I could talk about this stuff all day. Thank you so much for having me. It's just such a great time. As an engineer that has been coding for so many years, I've always wanted these type of features to have machines that could do even the most basic form of raising. It's amazing. So look forward to talking to people about it.

All right, Adam, thanks so much, man. Yeah, take care.

Talk to you soon.