Retrieval Augmented Generation with LlamaIndex

Sam. Foreign.

And thank you for joining today's code along. My name is Rhys and I'll be your moderator today. We're going to kick off today's session in a couple of minutes. We're just waiting so everyone has a chance to join. In the meanwhile though, we'd love to hear from you. So let us know where you're joining from using the chat or the comments depending on what platform you're watching on the and yet tell us something that you'd like to get out of the session today if you haven't already, then please register for the event. You can do that by scanning the QR code on screen. I'm also going to be posting a link for you to do that in the chat very shortly and you can also head over to datacamp.comwebinars where you can also see all of the future sessions that we've got in the lead up to the holidays as well. We are going to be using datacamp Workspace today, so if you don't have a Data Camp account yet, make sure you register for your free account. And I'll be posting a link very shortly when we start the session so that you can code along with us. Brilliant. I think that's everything from me. I will be back to repeat these messages for any new joiners very shortly, but until then, enjoy the background music.

Sam.

Hello everyone and thank you for joining today's codelong. My name is Rhys and I'll be your moderator today. We're going to kick off today's session in about a minute or so. We're just waiting so everyone has chance to join. Let us know where you're joining from using the chat or the comments depending on what platform you're watching on. And yeah, tell us something that you'd like to get out of the session today. We are going to be using datacamp Workspace today, so if you don't have an account already, then please sign up for a datacamp account and I'll be posting a link so you can go along with us. And remember if you miss it, if you have to log off at any point during the session today, please make sure you register for the event and then we can send you recording as well as any other resources that that we share throughout today. So if you haven't registered already, scan the QR code on screen, head to datacomp.comwebinars and also find the link that I've put in the chat as well. So yeah, make sure you get registered. And yeah, we'll send you the recording as well as the resources that you need to code along to this in your own time. Aside from that, we are going to be having audience Q and A during the session today. So if you have any questions at any point, let us know in the chat, in the comments and we will get those answered for you.

Brilliant.

I think that's everything from me. So now I'll hand you over to your host for today's session. Richie. Richie, please take it away.

Hi there, Data Scamps and Data Champs, this is Richie. Now, hopefully the fact that you're here means that you've noticed that large language models are kind of a big deal right now. They do, however, have have a problem that is sometimes the text they generate isn't true, so it's sometimes called hallucinating. And one solution to the problem is a technique known as Retrieval Augmented generation. And that's what we're exploring today. So the idea is that you take your existing prompt and then you augment it with some extra text that contains relevant information for the response and then the LLM has a fighting chance of giving you you a factually correct answer. The other thing you'll be exploring today is Llama Index, which is a framework for building AI applications for search and for retrieval augmented generation. And teaching you today is a bit of a legend is Dan Becker. He's a longtime Data Camp collaborator, having created the Introduction to Deep Learning in Python course and beyond that, Dan is one of the top thought leaders on machine learning and and AI. He's the chief generative AI architect at stra, where he builds AI solutions for content technology. He also runs the Build Great AI consultancy and he was previously VP of Machine Learning Development Tools at DataRobot. Dan is also a successful KAGGLE competitor and the author of Automated Machine Learning for Business. So without further ado, over to you, Dan.

Hey everyone, glad you're all here. So like Richie said, I'm going to be talking about Retrieval Augmented generation with Llama Index. That's quite a mouthful. So we're going to start calling it rag, which many people do. And the primary reason that I am focusing on this topic today is that if you go back a year ago, there was a lot of excitement about building AI based apps. I've done consulting and what I found is that a disproportionate number of the use cases that we see in business require rag. That's because they're for companies to have their own private data and they want that to inform the model responses. And RAG is Really the most appropriate way to do that in not all cases. And actually I'll, I'll talk about RAG versus fine tuning quite soon. But RAG is something that I see popping up so, so frequently that I'm excited to share this with you. So today we're going to go through a case study. The data that we're going to do use for this is from the News API website. With that website you can get data about almost any topic you want. Because we are excited about large language models, we're going to take news from the last month about Llama 2, which is one of the most popular open source large language models. And our goal is just to create this working example of RAG or retrieval augmented generation using Llama Index. You'll see that there's not an immense amount of code here and it is going to be relatively easy for you to take the RAG example that we build and then migrate that to whatever other purpose you have. I think this is going to be a really nice resource, not only for you to understand, but also to come away with this code. And then also we're going to look at how to interrogate and debug these RAG systems so that not only can you build it, but if it doesn't behave exactly the way you want, you can figure out why is that happening and fix it. I think of RAG as the most reliable way to use an LLM with a proprietary or internal or specific data set that might where that data set might change over time. And in a minute I'll talk more about why that's important. But before I do that, I just want to give you an overall context about what RAG means or how a RAG system works. So I'm going to zoom in. This is a. And when I share a notebook, you'll see you also have access to this image in your notebook. But we are going to think about an app. This could be a user facing website as an example and the user comes in and they have a question. So we are here and we've got this user, they're going to ask that question of our system. So they might ask how can I serve Llama two models with aws? That's going to come into our centralized RAG system or RAG framework. There are a number of tools for this. So we're going to go through Llama Index, which I think is my favorite. But you can also use LangChain to build as the central hub of this RAG index or this RAG framework rather. And to be honest, it's not so wildly complex that you couldn't also build your own framework custom to your use case. And certainly people do that. So the RAG framework is going to get the question and let's scroll up. It will then say to a vector database, here's my question. So that is, how do I serve a Llama 2 model on AWS? And the Vector database will find of all of the documents that we've stored in this set of this repository of documents, what are the documents that are most relevant to this request or this question or this query? So the vector database, an example, there are quite a few of these. So Pinecone and weaviate and Chroma is one of my favorites. There are all these different databases that can store primarily documents, but also images. And find the document that is most relevant to a given question and it will then pass back a section from that document to the RAG framework. The RAG framework will then pass that to a large language model and it will say here's the question I've received. So in this case the question was how can I serve llama 2 models with AWS? But in case the large language model, let's say, was trained 12 months ago and as a result the answer might have changed or maybe it didn't know in the first place. We're also going to take that the snippets of text and pass those to the large language model as well as the question. And so it will receive the text saying here's the question followed by the question, and then here's information that may help answer this question. And we'll embed those paragraphs. The large language model at this point has typically a pretty easy job, which is we've got a question and then we've also given it the answer and then it's just synthesizing the answer into something that is nicely written. You can use ChatGPT for this. We'll give an example of that. You can use llama2. You can use really any large language model for this. It creates the final answer and the answer could either be in this case Amazon Bedrock or. Or depending on the exact large language model, it might be a little bit more verbose and it might say the most common tool for deploying a Llama 2 model on AWS is Amazon Bedrock. But it's just creating that answer and then that answer is now passed back to either directly to the user or if we have a web based app, it's passed to the front end of the app that then shows it to the user. So this is an overview of this system. A common decision that you'll need to make before building a RAG system is I've got a set of documents or some set of knowledge that I want to have my AI have access to. And you'll need to decide do I want to build a radio RAG based system or do I want to fine tune that information into my model. So first of all, building a RAG based system is much more reliable or easier. It takes less, less fiddling with hyperparameters, but either of these can use any data. One of the things that in my experience, and I wouldn't have anticipated this ahead of time, but one of the things in my experience is really, really important is the ability to understand why am I getting the answer that I'm getting and can I change that if I'm not happy with the answer. So that's this. I've called it debugability. That is much easier with rag. If you're getting a new set of documents all the time. So I can have a persistent database, it's stored on an ongoing basis. And then I have new documents coming in on a daily or weekly or monthly basis, it's easy to update that with rag. The training cost for RAG is effectively zero. There are some advantages to fine tuning, so that would be if you want to change the tone or the style of your response, fine tuning can allow that. If you want your model to be able to learn to reason in a new domain, fine tuning can do that to varying degrees and I won't even get into that. And then inference cost is a little bit cheaper with fine tuning because you're not passing in these whole documents as you're doing inference. So these are some of the trade offs. You can use these two together as well. But our experience has been that we use RAG far more frequently. And so I'm a big have become a big fan of RAG and a big fan of Llama Index for using rag. So with all that, why don't we jump into the code? So like I said, I think it's useful to have a mental model of, of how RAG operates and as a result I've in this notebook and you'll have ongoing access to the notebook if you register. I've got this diagram at the top and as well as a link to where that came from, there are a few packages that we need to install. So we're going to get data from this news API service. You'll need a token for that. And for now I'm just going to use my token or this particular notebook has the data preloaded. So if you don't want to get a token, you don't want to go through their website. You can just use the data that I've already stored. Dakota along we'll be focusing on llamaindex. So I've installed that and then later on we're going to use open source models from Hugging Face. And so we'll install that. All right, so I want to get into looking at the data in a moment, but before I do that, here's just a quick view of the code and it'll be more interactive in a moment, but here is a quick view of the code to download the articles. So like I said, we're going to have an. I've got an API key that I've registered to download articles from News API. I create the client. We're going to save the articles in this savearticles JSON I'll briefly open. So you can see there are a few files here. One is as we look at Hugging Face or some of the other services, if you want to set those up, we have some guidance on how to set those up to take away. If you come back in the future and you are working through this and you can't remember a solution. Here's a notebook with a solution. Here's the notebook that I'll be working through. And then we have the saved articles. So we've got the saved articles and we can either retrieve those articles. I've got a little flag here. So whether we do that in real time or if you don't have a key and you just want to load them, then weigh in. Saved articles is false. We'll just load this. I will run this cell now. And we have retrieved 100 articles about llama 2 that were all published in November. So let's look at a single item. As data scientists, it sounds like something you would do an immense amount of math, but one of the things we spend the most time doing is just looking at the raw data and understanding how it is set up. So I'm going to inspect a single item. If I looked at all of the articles, it'd be really a little bit overwhelming. So why don't I look at one of them? I can look at zero. Which one you look at doesn't matter. If you're coding along. I would suggest looking at a different one than me so that we can just see a different example. So here is our raw data. We see that this was an article from a website I'm not familiar with called BizTalk it is. Here's the title. It's quite a long title. We have a description and then finally we have the content of the article. The content of the article is further validating how brittle the security of generative AI models and their platforms are. Some security company helped hugging face dodge a devastating attack by blah blah, blah. Because I'm using a free account, we are not going to have the full source of the articles or the full content. But you'll see that even the first sentences of an article is enough for this demo. And then if you go and get a paid account, you'll get full articles. So we're going to have this for a bunch of different articles. Look at. I'm going to just pick another one and we can see another example. So Here is someone's Q3 earnings call transcript. And this is going to be something that is related to the Llama 2 models at some tech company. We can see yet another one operationalized LLM evaluation at scale using some AWS service. So we've got a bunch of different articles and from this we can see the format of them. So looking at one is always very valuable. I'd also like to just get a sense for the breadth of articles, whether there are duplicates, just get a general sense of them. To do that I'm going to look at all of the titles. I will create all titles. You can see the format of this is each of these articles has something called. Has a field called the title. And so I'm going to just use a list comprehension. So I've said we're going to take each of the articles, each of the individual articles in our alter articles articles. I'm using this word quite a bit. I'm going to call that an article. And then we're going to take the title for that. And why don't we look at the top 20 of those? All right, so we see. This is nice. One of the things that I do like about the news API is it's got articles from a breadth of sources. So this is something for the hacker news website. So someone wrote a post there, it's showing up. We've got something else that is an earnings transcript. Here's a service that lets you serve LLM models with a PHP service. So we've got an overview of the titles here. We could look for whether there are duplicates and we could just write a line of code. I'll even do that. One way of doing that is see if there are any duplicate titles here. So we've got 100 articles and from that we've got 96 unique titles. So it looks like we do have some duplicates here. If you're building a production system, you'd probably do a little bit of de duplication, but I think for our purposes we don't need to worry about that. We've got a sense of the data. Let's now start doing the work that we need to build our system, our RAG system. So the RAG system is going to look at the text of the articles and it's going to put that into a database. I'm going to do something similar to what I did higher up. I'm going to say article text for article in all articles. Then again, I just want to make sure that I'm going to print that out. Maybe I'll just look at the top five just to make sure that the result we're getting matches our expectations. So let's go back up here and figure out. So I called it text, but the correct word is content. So I'm going to. You can see just looking at the raw data. Again, this is the reason that it's useful to look at your raw data. So here I've got the text of all the articles and here's the some results. Yann Lecun is Meta's chief scientist. It's got an article from him. Makes sense since Meta is also the creator of Llama 2. And now we're getting into the code to build a reg system. So we're going to go through this a few different times. We're going to build a RAG system. The first one is just using OpenAI, but then later on we'll do it with hugging face. The there's four lines of code here. These four lines of code. It's a little bit different from like a list comprehension where if you've used Python for a while, you might just know those lines of code. These are somewhat bespoke or specific to Llama Index. And so now you've got a resource and you can see see these lines of code, but it's really very few lines of code to get started. Llamaindex does a really nice job of what developers call progressive disclosure of complexity, which is to say that you can get started and it doesn't seem very complex at all. So this is four lines and then if you want to really fine tune it and change a lot of different knobs to change the behavior of it, you can get very, very deep and you could set up a system that has Hundreds of lines of code. But this actually will work pretty well. So let me walk you through these four lines of code. So the first is we're going to need to do two things with large language models. One is take the document and get a vector representation or an embedding of the meaning of each of the documents so that we can put that into the vector database. So the vector database, when we say what is a relevant document to a given question, it has a way of measuring relevance. And so we're going to use a large language model for that. And then we're going to use a large language model to create the final answer. And again, you can see that in this image at the top of the notebook. And if you're following along and have registered and have this notebook open, you can always scroll up there to see that context. Excuse me. The next is we're going to create a set of documents to insert in the vector database. The next thing we're going to do is insert those documents into a vector database. Llamaindex by default has its own in memory database that's getting created with this particular line here. But if you wanted to have any of the mainstream vector databases, whether that's Chroma or Weaviate or Pinecone or there's probably 10 mainstream vector databases, it is really just one extra line of code to set up your vector database and then use that instead of this default. But this is creating the vector store that has pairs of documents and vector representations or embeddings for those. And then the last is we're going to create the query engine. So this is really. The query engine is really the piece that we are excited about. It is the end to end pipeline that uses all of this. So we've got the index. We just say create a query engine from it. There's one argument that is particularly useful, which is when we ask a question, we're going to get back a set of documents and pass those to the large language model that gives the final answer. How many do we want in this case? I said two, but you could change that to be five or one. And you'll get different quality results depending on that number. If you were just. I think it's always nice when you are just getting started in a new area to find one or two parameters to mess around with it and see how it changes your results. This is a nice one that's easy to change and see how it changes your results. I'm going to run this line of code. Okay. And now we are going to use this query engine. So the way that you do that, a query engine has a method called query, and we're just going to give it a question. So I'll just start. I used this question earlier in the slides. What AWS call it AWS or Amazon Web Services, because it's a large language model. It really knows that those are the same thing. It has, in some sense, a bit of common sense. But what AWS service would I use to deploy a llama 2 model, llama 3 model? I need to have something that can see into the future. So I will run this and you'll see that I got back a response. The very first item in it is the answer and it's the correct answer. So that's a good sign. The other real takeaway I'd have is this is like horribly complex. This is like, there's so much in here. We're going to spend a moment looking at it, but then we're going to try and write a function that simplifies this for us so that we can run this on different queries and see results, but not have to parse through all of this mess to do it. Ok, so what is the content of this response? So we've got the response itself. The next item is called source nodes. Source nodes is a reference back to the documents that the model used to formulate this response. So source nodes is going to be a list of nodes. Because I said I want two documents, There are going to be two items in this Python list. Each item is going to be a node with score. Score is just a measure of how similar this document is or how relevant it is to the query. Then there's all this mess in here, there's an ID which we won't go into metadata. This is actually another topic which, if you really get into this, metadata, lets you have a way of filtering and say, I want only data, that is only documents from the last week or from a certain publication. And metadata filtering lets you do that. So if you get really into this field, metadata and metadata filtering is an interesting area, but the piece of this which we likely would most want to look at is the text. So we've got two source nodes, each of them has some text. And if I were just trying to generally debug, I would typically, when I run a query, want to see the actual response as well as the text of each of the nodes that we rely on. So I'm going to create a convenience function called search, which, which runs this query. So the first thing we're going to do is let me. Run this query and then we're just going to pull out the relevant information so that in the future I can run this search and get a simplified answer rather than something that is quite complex. So we're going to say I'll call it output equals. And we saw that that was this response item. So query result, response. And then. Someone's getting Query engine is not defined. I think they probably. There's this was in the chat. They probably didn't run this item up here that builds the query engine. So if you get an error about create engine not being defined, just rerun the cell above it. Yeah.

Just one thing to add to that. There are some instructions. If you go to the file tool on the left and go to the account setup notebook. If you need to create an OpenAI account, then that'll show you how to do that.

Thanks for t. The data sources that we rely on. I'll actually call it data source texts. And we're going to pull out the nodewith score. We're going to take output dot.

Let me.

Source nodes want to make sure that I get the capitalization rate. And then for each of these we want to pull out the text. So. And we're starting to get a bunch of questions. I'll tell you, I'm going to just batch those. I'm going to save those for a bit and then come back to those after we've gone through this notebook or I think we've got. We'll probably spend about 15 more minutes going through this and then I'll answer these questions because we're getting good questions in the chat. So let me return output and data sources with text. There's a chance that I didn't get this exactly right, but we'll test it. So we'll test it with the same question we used earlier. Yep. Same problem as the one we had earlier. Let me. I didn't close this bracket. And. All right, so we've. Let's debug this for a moment. We've got a string has no source nodes. So that means that. This. So this right here is a string. And I said output source nodes, but it really should be query result. And we'll rerun this node with score should be txt. So I got it. There's always a little bit of debugging involved. But here we got something that is a utility function and in the future, if you want, you can just copy this function. What is the best AWS service to deploy a llama 2 model and instead of this whole big mess that we got higher up, we've got a response which has two parts and which is, I think, very interpretable. One is the raw answer, that's this first piece. And then the second piece that we got back is the text that the model relied on when forming that answer. And then when you're debugging these systems, you're going to use this quite a bit to figure out. I like to break up if you're getting an answer you're unhappy with, I like to break that into two pieces. One is, is the LLM getting data or documents that have the right answer and if not, then we're going to debug our query engine or our query retrieval. And then if we are getting the right answer and that's feeding it to the LLM and it's still getting the wrong answer, then we would debug what's happening in that second stage. So this is a quite useful function. I think you'll use it on a, on a regular basis. Let's now switch to building a version of this ourselves to change. So so far we've done this with OpenAI APIs. Now let's switch and use it with open source LLMs. Those are going to be coming from Hugging Face, which is I think, the most popular place to find and serve large language models. In order to not be using the word llama2 in too many different places, too many meanings I've used, I'm going to use this Zephyr model, which also is, I think, a newer model and probably my favorite model right now. So we're going to use the Zephyr model, which is stored on the Hugging face. We're going to need to do the equivalent. If we go back to our lines of code that created our previous RAG system, the first one just created the object that can access the model, we're going to need to do that. This is the thing that you wouldn't off the top of your head, have any chance of knowing what to do. I'm going to do a Google search just to show you that it's not so, so hard to find this. So we are using Llama Index. We want to use the Hugging Face inference API because that's going to be serving our models. And if you do that search some of the top examples, really a few of these would have what you're looking for would have examples. I quite like to just work from examples. And this is a notebook that shows explicitly how to set up large Language models with Llama Index. I'm not going to go through this particular notebook just in the interest of time, but we could find that there are a few examples here of how to do it. So we've got. If I wanted to run the model locally, here's a line of. With an example of how to do it, I'm going to copy these two lines of code over. So one of these is if you don't have a hugging face token, which is you just haven't registered through hugging face, don't have an account, and there's a notebook showing how to do this in our workspace. How to get that hugging face token, then you would. If you don't have that token, then you would use this. If you do have a token, then you will use this. Bottom line, the advantages of having a token is that they are less likely to rate limit you or think you're abusing their service and as a result block your access. So let's. I'm going to copy that all down here. I do have a token, so I'm going to uncomment this. If you are running this on your own, you probably don't have a token or don't have it stored here. And so you might not use this HF underscore token. Instead you would comment this out. You would comment this bottom one out that uses hugging face token and instead just use this remotely run. But I'm going to use my token and. Let's see, go back and they've got the model name. Let's see if we can pass in the model name as well. Oops. Later on I'm going to call this HF model. So we'll do. And again, I've just created model name right here. I want to make sure that I'm specifying that and that's run. Okay the one change that. As you move away from defaults, a lot of the information that you use is stored in what's called a service context. So as you do more customization, you're going to add more arguments to the service context. For now we're just going to add the LLM we're using is this hugging face model. Now we're going to. Recreate some of these lines that we have above. So the documents are the same but we're going to recreate the index and query engine. So you'll see we just needed two index equals question mark queryengine. Query engine is again not changed at all. The only difference is that we need to pass in our service Context. Here I've saved something called service context which just says what are the additional arguments or the customizations we're going to use? I'm going to pass that in and the rest is really just the same. So let's test it. Let me. I'll say query engine. I just need to pass it a string. I'll be consistent. We'll use the same thing that we have used so far. So we'll do what AWS service is a good Choice to deploy Llama 2 models. This is the same. This is an object we need to actually call the query method on it. And here is our result. You'll see like this is one of those idiosyncrasies of different models when we use the. When we first ran this up top, we were using OpenAI. It gave us a terse answer. Here we're just using a different LLM, still getting the same documents. It gives llama 2 is more long winded. So it says Amazon Bedrock is a good Choice to deploy Llama 2 models and so on. Yeah, it has a little bit more information and we could, yeah, we could run search again to get a simpler version that doesn't have all of this extended information around it. Be easier to process. But I think that you've come away here with enough that you don't in 45 minutes know everything about building a rag system. But between this code and a general understanding, you can run some experiments and then have something to build off of. This is something that I've done an immense amount of. That's all been driven by the needs of projects where people have their own data. They want a large language model that works off that data. And this is the framework that I use. So I know we've gotten a bunch of questions. I don't know if Reece or Richie have queued those up, but happy to answer accumulated questions or new ones.

Brilliant. Thank you so much, Dan. I have to say I thought it was very cool way. It's almost exactly the same code whether you're using the OpenAI model or whether using Llama 2. So it just makes it easy if you say, okay, we want to change the model, see if it flows better. Just pretty simple swapping code there. All right, we're going to go to audience questions in a moment. Before we get to that, let me just stop sharing your screen. I have a question for you. You mentioned that you've been building quite a lot of these things, so can you just talk through some of the most common use cases for this sort of system.

Yeah, there's one that I'm working on most actively at the moment and that I've heard. So I've done this twice and I've heard of it happening a lot. You've got a customer service function today that's done with a mix of, to a lesser extent, chatbots that are just not very good, or historically chatbots that were just not very good. And many of us have had an experience of typing something into a chatbot, like, I just want to talk to a human, or humans. And the reason for that is that humans actually know the company's policies. So something that we're doing is building a chatbot where someone goes in, they ask a question, and now the chatbot can go and read and basically have access to the whole database of company policies. It looks up the relevant policy and then it crafts an answer that is. Yeah. And that both says what the company policy is and is responsive to the user's question. One of the really nice things about this use case is that you can imagine, hypothetically, a few years ago, we could have done a search, but with search, if you didn't use the exact words that showed up in the company policy, it didn't know synonyms. And one of the great things about large language models is they convert everything into an embedding and people can can ask a question that doesn't have any overlap in words with the exact policy. And maybe the policy also was like written with a bunch of jargon and the large language model can simplify it and give really human answers. That's something that we've done. There's something which also is to give another example of a use case, we've built an internal tool for a Fortune 500 company. They're HR professionals, just want to be able to identify. So HR contracts with employees tend to be very long, and for startups they tend to be very consistent. But for enterprise companies that have accumulated a bunch of companies that they've purchased over time, there's a lot of idiosyncrasies between them. And so we built a project for internal HR employees and they want to know, do we ever have any clauses in any of our contracts with our employees that say exactly X? And we built a rag system for them that can answer that. And that saves them an immense amount of time through versus trying to just browse through on their own.

I didn't really like those. I mean, I know there's a few people going, oh yeah, artificial general intelligence is coming. It's going to Save the world. But actually, you know, I'll settle for like a chat bot that I don't want to strangle through the computer screen. So these simple things are just a little quality of life improvements that definitely appreciate. Okay, so we've got a few questions from the audience. So the first one is coming from Mark. So Mark's asking about how are embeddings created and how are these vectors created. So maybe can you just give a quick overview of like how embeddings work in terms of converting text to numbers and how the search works to like how do you find relevant text?

Okay, so I'm going to answer this in three parts. The first, which is not responsive to Mark's question, but it's just, I want to say what a cool thing that's coming out in the sort of like the frontier in the last few weeks is some of the vector databases are starting to and Llama Index is now starting to support images. So that instead of just text documents, if you have a bunch of images that could be, let's say, diagrams about how to assemble something, those can now also be stored in these databases in vector databases. And those can also be fed into a large language model. And because now a lot of the large language models are multimodal, both documents and images can be stored that way. But that was actually a side I got excited about. Let's talk about how documents can be converted into vectors. That would be, I said there are three parts. One is a cool thing. Then the second two parts are answering Mark's question, how are they converted into vectors? And then the third is how does the vector database identify what documents are relevant? So when you some of the deep learning, the first part of that, which is how do you create the vector or the so called embedding. This is pretty technical. And so I'm going to give the best answer in one minute that I can give. But you could actually take deep learning courses on Data Camp or anywhere else and get a deeper dive into this. Large language models operate in a few modes or when they're trained, they have a few different goals they can pursue. The one that people talk about the most and some people mistakenly think is the only one, is take a bunch of text and try and predict the next word. So the way that we would, let's say go through Wikipedia, we say here's the first seven words of an article, predict the eighth, then take words two through eight and predict the ninth and so on. And the way that they do that is with a bunch of Operations. The first is to take embeddings of individual words and then aggregate those up into embeddings of sentences. When you first start training the model, you just literally randomly assign random numbers to the vectors for each of the words. That might be a poor way to do it, it might lead to poor results. But, but I think conceptually like the idea of random number generation, we can probably wrap our head around. And then in the training of the model, you continually figure out, is there a way that to predict the next word, which is at this point a bunch of operations on vectors. Is there a way that I can change the embedding for a given word in order to increase the likelihood that the model would predict the correct next word? And then there's a layer beyond that for how do we aggregate those words. This is something called transformers. And these recurrent, these are sort of like the next level of how the model works, but a large language model, even like the ones that you're familiar with. You know, if you go into ChatGPT, it was pre trained on exactly this task. And then when you run it, let's say you're using ChatGPT, the last thing it does is there's a little bit of a caveat to how ChatGPT works where it's using this thing called RLHF. But it is, approximately speaking, predicting the next word in what would be a good answer? The step right before it converts that into a word, it actually has a vector representation of the meaning of, of everything that has been written so far. So when we have a document, we run it through a large language model and instead of predicting the next word or predicting the word, that would be a good response. We stop just short of that and we say we're going to take the vector representation or the vector representing the document we have so far. We're just going to store that in the vector database. So this is like I said, super, super complex. I apologize if I couldn't quite get to the idea of it in so briefly. But the third part is less complicated. So that last part of how do we determine what's relevant? This part I can say briefly. So if you have two vectors, there are a lot of ways to measure the distance between them. And this is covered like in, if you go along in math for long enough, you'll probably at some point learned compare distances between two vectors. So the most popular is Euclidean distance. That's like if I have two points, how do I just measure the distance between them? And then so there are other ways of measuring distances, but we just for all of the documents in our database, calculate the distance between the vector for that document and the vector that represents the. The query and then we find the two that have the shortest distances and that's our measure of similarity. Long winded answer. But yeah, that's.

I thought you've done very well for just like explaining sort of many years of deep learning research in a few minutes. So I guess my sort of very simple like interpretation of that is words are hard to deal with but we know how to do linear algebra so we convert words into numbers and then that's right, just some simple, not quite well, university level math all happens in the background. And the magic transition bit, that's the embedding model.

Yep. I want to actually quickly I'm looking at some of the questions. Mark just popped in a really good one. I want to.

Okay, let me have a look at Mark. This one.

Sorry, there's a newer one. Let me.

Hold on.

The newest one.

Yeah, okay, another one from Mark.

Yeah, so this actually I want to address this because I've seen this done wrong actually. Llamaindex makes it hard to do wrong. But can there be a mismatch between the chosen embedding and the model? Because it seems that we start with OpenAI and then use Zephyr. So large language models get used in two ways. One is to create the embeddings then the other is to create the answers. As long as we are consistent in how we create the embeddings. Consistency there is when we create the embeddings for the documents that go into vector database we need to use the same way of creating the embedding of the query so that when we look at their similarity there they're comparable. But you can take any model you want from for the embedding that leads to the measure of which documents are similar and then when we synthesize an answer, all that's getting is text. That's getting the question and the text of the document. So you can use any model you want for answering the final question. Okay, so you're going to use the.

Same one in both cases and you should be fine.

Can use the same one or you could use different ones in either way as long as you're consistent with one for embeddings and then whatever. Yeah, as long as you're consistent with embeddings you have a lot of flexibility.

All right, super. Okay, so next question. So relation. This one comes from Jeremy. So I think one use case you talked about was like retrieving news articles. And so quite often you want more recent documents to be considered more relevant than older articles.

Perfect. Yeah.

How do you deal with that time component when you're retrieving information?

First of all, training models is not so messy. We've gotten better, we as well have gotten better at it. But it takes a lot of compute, takes from hours to. It takes a lot of time, a lot of computers. Fine tuning a model is a meaningful endeavor. On the other hand, inserting things into databases is relatively straightforward. I can take my vector database and on a daily basis just have a little job that runs, that takes any new documents, inserts them into the database. It'll typically take seconds, or if you have a lot of documents, it might take a couple minutes. But even that would be like a lot of documents. And then the retrieval augmented generation system, whenever it gets a new query, it's going to go consult the database. And the database in real time figures out what are the most relevant documents. And so you can just set up a little recurring job to keep the database up to date. And that means as a result, your system will always be based on up to date data.

So when you're doing that, find the most relevant documents. Normally, you said before it's using like dot product or some kind of linear algebra sort of thing to say these are relevant documents. Is there a way to include a time component for that? Say this document is newer, therefore.

Well, so there's essentially there's three ways to think about this and two of them work. One of those three ways is I'm going to ignore timing. The second way is to say I only want articles from the last X number of days. This would use something called metadata filtering. Metadata filtering is a way for you to, when you make the request, say I want to run this filter before I even look at document similarity. And so I only want articles from the last 30 days or from only want articles from a certain time period, or only want articles from a certain publication. And so metadata filtering lets you do that. The third way, which can't be done, is to say I want some sort of like measure of similarity, the conventional way of just document similarity, but then like weekly favor newer over older in some sort of fuzzy way. I suppose that could be programmed, but there's no simple way to do that really. It's if you care about recency or some other metadata, you would filter on that and then find the new, the nearest or most similar documents that meet that criteria.

Okay, all right. That's useful to know. All right, this sounds like a slight technical question from Ben saying, what's your point about using cross encoder re ranking? We get a lot of hallucinations using plain context generation based on the embeddings. So yeah, basically. Can you tell us about cross encoder re ranking?

You know, Ben's got a, an angle on this that I'm not familiar with. So I, I don't, I'm not, I'm not even familiar with crossing code or re ranking.

Okay, Ben, you're teaching the next one of these. Sounds like you're. You. You're pretty van in what you're asking. Okay, right. Next question comes from Saket saying, so you talked about like one of the use cases being a policy bot, so maybe like looking at contracts, something like that. Do you have any recommendations, recommendations around how you might go about doing that?

Yeah, I mean one of the things that for all. For many of us before we worked in large language models, we worked in just conventional predictive AI and predictive machine learning. One of the recurring themes in predictive machine learning is to first create a way of quickly and reliably measuring the quantity quality of your results. For me that can be. I've got a bank of 10 or 20 different queries that I want to make sure I can answer. Well, ideally eventually it gets up to 100. I can. I will then build something very, very simple and that could be as simple as the code that you've seen today, not fiddling with any changes. And think of that as my baseline. Run that against the set of documents, figure out are those good enough for me to trust. If not, then I am going to have to iterate so that I have something that is sufficiently high quality. Then once I've got, and then once I've got that, the next is just sort of the infrastructure piece of it. So we typically used fastapi as the entrance into the backend. Fastapi has some code that. Well, so typically we've got the vector database that lives as a long running service so that you don't need to reload that database every time we get a new request in. But then we've got a FastAPI backend, it will get a query and then basically run this queryengine query type code, hitting the vector database, getting the response and then sending that back to the front end. Then that fast API backend that can live either on a service like EC2 or on AWS Lambda or if you're on some other cloud, they're serverless, these things tend to be lightweight enough and they have sufficient little state that you can run it either way. And I would say more than half the time we're just running it on some serverless service like AWS Lambda or the equivalent on other clouds.

Okay, so it sounds like beyond this sort of messing about with just the AI side of things is quite a lot of backend infrastructure and messing about with cloud services that you need to deal with.

Yeah, I mean, I think that's probably typical for productionizing almost anything. Yeah, there is some infrastructure that we always need to set up, but yeah, just in some ways a typical amount for deploying ML systems into production.

Okay, super. We're coming to time. There are one or two more great questions from the audience I want to get to. But before we go to that, we've got four more webinars before the end of the year. So for those of you interested in coming back tomorrow, we've got a session, a session on the secret to successful Data literacy program. So if you're interested in learning or having your colleagues learn more about AI, then please do come along to that. On Friday we've got another code along on AI, so this time it's on fine tuning GPT using the OpenAI API. And then next we've got two sessions, so there's a beginner session on exploratory data analysis in Python. And then Wednesday we'll have a session on using Snowflake, the data warehouse for sales and marketing analytics. So please do go to datacamp.comwebinars and register for those. I will see you for those later this week and next week. So let's do two more questions. All right, so this one comes from Marcel saying, if you got very different amounts of text in each entry in your vector database, how do you go about chunking them? How do you go about dealing with overlaps and things like that?

Great. This depends a lot on the large language model that you're using. There's a trade off between roughly three things. One is when you retrieve. For those who don't know what we mean by chunking, let's say you've got a large document, could be 100 pages. When you get a request in, you're not going to return all 100 pages to your large language model. And so you're going to break each document into chunks, which can be typically like 100 to 500 tokens, which is probably 60 or 70 to 300 words. The question is like, how do you pick the length of how long each of the chunks are. And then also a lot of times you'll want the chunks to overlap a little bit. So that was a question. The trade off is the longer your chunks are, the fewer you can pass in because there is a larger cost if you pass a lot of text into your large language model. And then also at some point your large language model has a finite context window, these context windows, which is basically how much text can it take at once. Those have expanded meaningfully recently. But yeah, there's some trade off between how many you return, how long the chunks are, and then cost. And this is probably an unsatisfying answer, but going back to a comment I made earlier, we'd like to have some repository of test cases. And once I've got that repository of test cases, I can say I'm going to use a chunk length of 205 chunks or 5.5 chunks, or I'm going to return 2 chunks of 400 or 1 chunk of 600. Then I'm just going to see which of those empirically gets the best results. It becomes much like hyperparameter tuning in conventional machine learning. It just becomes an empirical search experience. The hardest part of that is setting up the evaluation so that you actually, because changing chunk length is a quite easy thing to do, you just change an argument in Llamaindex. So now it's just doing that search. The hardest thing is setting up evaluation that you trust so that you can do that iteration.

Okay, so it sounds like if you've got really small chunks, then the amount of information you're passing in is going to be kind of useless. It's going to be like a fragment of a sentence or something. That's nonsense. So bigger chunks is probably better, but also more expensive is that.

Well, and then the other piece of it is if you, if you think the answer is. If you think the ideal answer to many of your questions is distributed over many pages, you wouldn't want to return one chunk of a thousand tokens. You'd like to return three shorter chunks. And that way you could have one have an. The large language model. Get a piece from page seven and a piece from page 50 and a piece from page 120. And so one of the trade offs is chunk size versus the number of chunks. And that's just. Yeah, it's just an experimental decision.

Okay. All right. Play around and try and optimize it. Okay. Just to finish up, there are a few people asking about persistence. So. Yeah. How long do you. Does your text stay in the vector database. Just talk us through it.

If you run the code that we ran, we didn't set up an external vector database. If you run this code and don't do anything to store it, then when you restart your session, it's not persistent at all. You can. In llamaindex, there's a command to serialize the vector store index that we created and then reload it, save it to JSON. It ends up being quite big because the embeddings are typically big and there's embedding for each chunk. But there's a command I didn't show you, but we could look it up for saving it and reloading it. And then if you use an external database, which, when you're running this at a larger scale, you'll typically do, then much like other databases like Postgres or MySQL or whatever, or Oracle, whatever databases you run, those are typically set up in a way that they. They're almost always set up in a way that they are persistent and then they last until they last basically forever.

Super. So from basically nothing to forever. That's a. You got a good range there. Excellent. All right, thank you once again, Dan. That was really informative. Super exciting stuff. Thank you to Reese for moderating. Thank you to everyone who asked a question. Thank you to everyone who showed up today. Please do go to datacamp. Com webinars and register for future sessions. I hope to see you all again soon.