BEST OPEN Alternative to OPENAI's EMBEDDINGs for Retrieval QA: LangChain

If you're looking for an open source alternative to OpenAI/language models and embeddings, this video is for you. In this video we are going to be comparing local embeddings against OpenAI's embeddings. And the quality of response generated by these embeddings will surprise you. The embeddings I'm about to show you are the best and state of the art open source embeddings available. To create a chatbot for information retrieval, you need two things. The first is embedding for doing semantic search, and the other one is a large language model for generating natural language responses based on the documents retrieved using the embeddings. In this video we are going to be looking at an open source solution for the first part, but in the subsequent video we are going to even replace the large language model with a powerful open source model. Here's the architecture diagram. So I'm going to quickly go over this. If you're new here, I have a detailed video on this topic, so I'm going to put a link to that video. So in this case we're going to have folder which are going to contain multiple PDF files. Then we will divide each of these PDF files into smaller documents because the large language model that we're going to be using, it has a finite token limit size, so we don't want to exceed that. Next, we were going to compute embeddings for all of these documents. So in my previous videos we used OpenAI's API for computing the embeddings. In this case, we're going to be using an open source solution as well as OpenAI's. The open source solution that we are going to be looking at is called instruct embeddings. It's based on this work. One embedded any task instruction fine tune text embeddings. So these are fine tune already existing embeddings that can generate text embeddings tailored for any tasks. It's a very simple to use mechanism. I might create another video specifically focused on this embeddings, but just keep in mind that it's one of the most advanced state of the art embeddings and they showed that it achieves state of the art results on worse, 70 different tasks. Okay, so once we compute embeddings, then we put those embeddings in a semantic search vector store, so that becomes a knowledge base. Right, and this is in this case we are using F as a vector store, but you can look at some other options for Example, there's Pine Cone. Chroma DB is another good option right now in terms of information retrieval. So when a user asks a question, then we will use the same embedding computational pipeline to compute the embeddings. So we'll get embeddings for a question, then we'll do a semantic search. So the way it's going to work is you take the embedding, you compare it with the embeddings of the documents, and find the most similar documents based on the embedding space. Right. You get the results. So essentially you return the original documents, then we feed those documents along with the question. Right. To get a response. So the documents that we retrieve, they become the context for the large language model. And based on the prompt, using that context, the large language model will generate a response. In this case, we are going to be using OpenAI's DaVinci model for generating the response. But as I said in the subsequent window, we are going to be looking at an open source solution. You want to replace this part, right? And that's how the user gets the response. So this is a very quick overview of the overall architecture. I have a couple of more detailed videos on all aspects of this architecture. So I would recommend you guys to watch those if you are new here. Now, in this video, as I said, we're going to be doing a comparison between the ability of open source embeddings to retrieve these documents. The way we will judge the quality of the retrieval is we'll compare the responses directly with OpenAI's embeddings. Right. The large language model is exactly the same. So if both of them return the same documents as information, we would expect very similar similar response generated by the DaVinci model. So it's going to be an interesting video, so keep watching. Okay, let's look at the code and see what's going on here. So first we are downloading and installing all the required packages. Next, you need to have your open AI's API key so you can get the API key from your OpenAI's account. Right. Next we are importing the all the different packages that we need. So. So I don't think we need a Chroma db. I'm not using it here, so I'm going to delete this. And even for installation, I don't think I'm installing it. Rather I'm using the Vice Vector store. Next, we are importing the packages related to our instructor embeddings. So the the package is called Instructor and hugging face Instructor embeddings that's being loaded here, right? Since we are doing a comparison between OpenAI's embeddings as well as these instructor embeddings. So we are also importing OpenAI's embeddings. Now, the goal in this case is very simple. The embeddings are going to be used for document retrieval to figure out which documents are the closest ones. So we are basically talking about this step, right? So based on the embeddings, we try to find out the closest documents or the most similar documents, and then we're going to be using a large language model to create a response based on those documents and they will be treated as context, Right? So in both cases we are going to be using the OpenAI's DaVinci model to generate the responses in natural language. And the goal was to see whether using open source embeddings can give us a similar quality response. Because like, if the information sources are exactly the same or the context is exactly the same in both cases, then the DaVinci model should be able to generate a very similar response. Okay, next we are simply interacting with our Google Drive. So we will mount a Google Drive in this case and then we are using the directory loader to retrieve documents from a folder that I have a folder called Document in my Google Drive and it can contain different file types. So it could have text files, PDF files, anything that you want. Right, but I'm specifically looking for only PDF files. If you want to retrieve text files, you will simply replace this PDF by text file extension. Right? So that would become Txt. Now this will create a loader and load all the PDF PDF files from my directory and then they will be put in this document object. Now let me quickly show you what type of files we are dealing with. And this is going to be a very tricky task to complete. So in this case, what I did was I went ahead and downloaded the technical report related to GPT4All. So that's the first version of technical Report that they released. Then I actually downloaded the GPT for all J model Technical report. This model was the, this was the, the one that's commercially available, so you can use it for commercial application. Now the tricky part is going to be that in both cases the authors are the same. The contents of the technical reports are pretty much the same. There are no much differences because it's talking about exactly the same thing as the first technical report. So I wanted to see that if you have a very similar documents and they contain very Similar information. So where this is going to retrieve the information from. Okay, next we take those documents, right? So in this case that we will have just two PDF documents and divide them into smaller documents or smaller chunks. So the chunk size that we're choosing is around thousand characters tokens, and then there's going to be an overlap of 200 tokens. Now, why we do this? So the large language model that we're going to be using, they have a finite context window or the token limit that they can interact with, right? So you want it to be lower than that. I have actually explained this in much more detail in my previous video, so I'm going to put a link to that if you're interested. Check that out. Right. So basically we divided into thousand token chunks, right? And here is an example of how it might look like after this division process, instead of two documents, we end up with 22 documents. If you are a visual learner. So this is what's going to end up happening. You have a PDF and now you are going to create multiple documents based off the that PDF. Okay, the next step is to get embeddings for each of our documents. Right? So as I said, we're going to be using two embeddings. One is going to be the instructor embeddings, which is open source, and the other one is coming from OpenAI. Okay, now I wrote two different helper functions that you can use in your own code. So what I wanted to do was to store my embeddings, the embeddings from the documents, right? So once you run those to the embeddings and whatever the result is, I am storing that vector store and then another small helper function to retrieve those, because we don't want to be computing them again and again. So first look, let's look at how you can download or get the instructor embeddings, right? So we're using LangChain for this. The function that we are going to be using is called hugging face instruct embeddings. Right? So this is different from the normal hug hugging face embeddings. And this seems to be state of the art when it comes to open source embeddings, right? So you need to pass on two objects or two variables. So the first one is the type of embedding. So instructor Excel, that's the embedding that we're going to be using. And the second one is whether you want to run it on a GPU or a cpu. So in this case I'm passing A cuda. So it's running on a gpu, but if you want, you can actually run this on cpu, although it's going to be much slower. Right. When we run the code, it will download, I think around 5 gigabyte file, but it will do that only once and then it will be stored on your local machine. As you're saying, since seeing here I'm running this on a Google Drive, but you can run the code locally and then you will have the corresponding instructor embedding stored locally. The maximum sequence length that it can accept is around 512. So here are two function calls that I just mentioned. You can actually uncomment this and run it. So it will both execute the code, store the embeddings locally and then retrieve them. But here I just wanted to show you a stepwise process. So we are using files as a vector store, right? And in this case you pass on the documents that we created, the corresponding embedding object for the instructor embedding, right? And you will get the corresponding vector store. Now you can use this vector store as an information retrieval object. Retriever is a special type of object within LangChain that you can use for information retriever. And so we take the vector store, we use it as a retriever, then we say, all right, get me the three similar documents, right? The type of retriever is similarity, so we want to look at the most similar ones. And the number of documents is three in this case. Now, in terms of the architecture diagram, so this is the stage where we are at, right? So you have your original vector store, you get embeddings for your query, right? And the retrieval will simply pick three most similar documents out of your knowledge base. Okay, one thing I wanted to highlight, if you're looking for simple information retrieval and you want to find out where the information is coming from, you can actually stop at this step. Because let's say if you run this, you will get the relevant documents, right? So in this case, my query is who are, who are the authors of GPT for all report, right? If I say get relevant documents and run this on the receiver object, I'll get a list of three different documents. And here is the most similar first document, right? So it will have text extracted from your source document. Okay, now the second step is going to be you will use this as a context for your large language model to create a size answer in a more natural language. So, but if you're simply looking for sources, you can actually stop it here. And if you want to actually continue and get natural responses. So we will have to create a chain. The chain that we're going to be using is retrieval, question, answer chain. Right? There are multiple parameters that you need to pass. So the first one is the large language model which will convert the responses into natural language. So in this case we are using OpenAI's DaVinci model, but you can use any model that you want. In the next video I'm going to show you how you can actually use an open source model even as a large language model. Right? So keep an eye out for that. The second parameter is the retriever that we just created for our instructor embeddings. Right. We want this chain to return the source document. So it will actually return a document source like this. Right. And the type of chain is stuff. Now we did the embedding part for the open source embedding instructor embeddings. Now we need to do the same thing for OpenAI's embed. I'm doing this because I want to do a comparison, but you can actually just use that as it is. Right. So first and foremost I simply create an object called embeddings using the open AI's embeddings. Right? And then pass that embeddings along with the test text documents to create a vector store. Okay. And then we do the same process. We will create embedded corresponding retriever object based on the vector store that we created using the OpenAI's embeddings. And again we will create another chain for this. So this chain is called question answer chain and that's the OpenAI version. Next we simply write a small a couple of helper functions and the goal is to simply change the responses into well formatted responses. This is something that I have actually copied. I'm going to put a source of this in the description of the video. So we need to test them. Alright, so here is my first query. Who are the authors of GPT for all technical report? Now, as I said, it's very tricky. Both of the technical reports contain almost identical information, right? So I'm mostly looking for the information that it returns, whether this is that is correct or not, not really looking for the sources. So if we use the instructor embeddings, the responses the authors of the GPT for all technical reports are and then you have their names which are correct. Okay. And the sources it cites. So it all three of the sources that it picked are from the second technical report. Right. But specifically I was asking for GPT for all model, not GPT for all. J model. Right. But still I think in both cases the authors are the same, so we can say that it was correct. Now if you use OpenAI's embeddings, the response the DaVinci model has created simply returns the name of the authors, not any other details. So I think still good because the response is correct and then the sources that it cites are actually the second report as well. Maybe in the next video I'm going to look at exact same documents that it is returning. Second query, how was the GPT4 All J model trained? So we are talking about this information that is contained in the second report in this case I think the responses for both of them are good. However they give very different responses. So seems like the documents that both embedding founds to be similar are very different. So for example, in the first one it says the model was trained over a massive curated corpus of assistant interaction including word problems, multi turn dialogue, code poems, songs and stores stories. Right. So it looks at the type of data set that it was used. Right. In the second case it says that it was trained using LoRa, that is the algorithm that was used for training it. And then it gives us further information regarding the number of example that was used and what were the number of epics. As I said before, it seems to have picked different documents and that's why it generated two different responses. Both of them are correct in a sense, but I think the OpenAI embeddings would did much better in this specific case. This is actually an interesting one. So I asked it what was the cost of training GPT4 all model and I was expecting that it will get information from the first technical report. So in terms of sources, the instructor embeddings actually found a document or a segment from the first technical report as a part of the top three. Right. But the response is the cost of training the GPT for all model was about $800 in OpenAI's credit and $100 for the Lambda Labs GPUs that they used. Now in contrast we answer from OpenAI's embeddings just $200. I actually went back and looked at it. So it seems like the first one which is the open source embeddings, it picked information from here. So it says between GPT for all and GPT J models we have spent about $800 in OpenAI credit so far to generate all training samples that we openly released to the community. Right. So this information seems to be correct if you look at data curation part now the second response that we got from the OpenAI's embeddings, it's coming from here. So the information, the relevant information is a released model can be trained in about eight hours on a paper space in the GPU unit for a total cost of $800. So it's not the cost of of actually training the model, but rather if you want to retrain it again. But I'm not sure where this hundred dollars is coming from. Okay, a couple of other questions. So what license is GPT for all J using? So this is the second model or the updated version. Right. And it says Apache 2, which is correct. And it cites the second technical report, the first one, and then the second as a source. Right. It might be confusing. GPT for all versus GPT all for J. Right. GPT for all J. Right. It's very confusing even for me. Right. And in this case the OpenAI embeddings return the same answer, although the source of information or the source documents are all in the second technical report. Okay, the next question I asked was what was the size of the training data set used for training GPT for all? So this is again a tricky one. In the first case it came up with roughly 1 million prompt response pairs. That's from the first technical report, but I think it's talking about the original Llama model, not the training of GPT for all. The second answer seems to be correct. Now I asked the same question regarding GPT for all model and both of the model actually came up with or both of the embeddings came up with the correct answer. Now I asked it something which is actually not present in either of the technical reports. So I said which MPT7B model is the best? This is related to the Mosaic ML models. Right? And there is no information in either of the technical reports, but it's says according to the evaluation data from the self instruct paper, the best openly available Alpaca Lora model, which is actually a 7 billion parameter model. And it picked that as an answer. Right. And the second case, again it picked Alpaca Lora 7B is the best MPT model according to elevation data set. So this is definitely wrong. Right? Because it made up that, okay, this information related to MPT7V model which is not present. So you can actually take care of this by prompt engineering. There are ways in which you can restrict hallucination of the large language model and we are going to be talking about those in our new series of videos. So this was a lot. My final thoughts so it seems like even using open source embeddings, especially the instructor embeddings, they actually generate very similar quality responses. There are definitely like some cases in which the responses deviate a little bit, but I think overall these are as good as OpenAI's embeddings. And that is why I'm actually thinking of moving away from OpenAI's embeddings altogether, because there are a couple of reasons. First, if these are as good as OpenAI's, you don't really have to pay for it. The cost is zero. The second is if you are worried about sending your information so at least you are restricting the amount of information you are sending. But if you can replace the large language model with some open source model Instead of using OpenAI's DaVinci model, then we can even take care of all the privacy issues. And in the next video I'm going to show you how can you use an open source model as a replacement OpenAI's DaVinci model. I hope this video was useful. We covered a lot in this video. If you have any questions, please put them in the comment section below. I'll try my best to answer them. If you have any business problems or you want to implement a similar system for your own businesses, websites or other domains and you need technical expertise, you can actually reach out to me. I have started consulting folks on solving their technical issues, so I'll be happy to help out. If you're new here and you haven't subscribed, consider subscribing to the channel. Please push that bell notification button so so you never miss a new, exciting video. Thanks for watching. See you in the next one.