Chat with Documents is Now Crazy Fast thanks to Groq API and Streamlit
Build a RAG chatbot that answers questions about documents in 0.07 seconds using Groq's Mixtral model, LangChain, Ollama embeddings, and Streamlit. The tutorial walks through loading a Paul Graham essay, chunking it into 1,000-character pieces, and wiring retrieval to Groq's API for near-instant responses.
Questions this video answers
1000 characters chunk size with 200 characters overlap.
"How to Do Great Work" by Paul Graham (about 26–27 pages).
Mixtral MOE and a 270-billion-parameter LLaMA model (llama 270B).
Approximately 0.07–0.1 seconds (examples shown: 0.07 s and 0.1 s).
"ollama run <model>" (demo used: "ollama run llama 27 billion model").
Chapters
So this whole thing took 1/10 of a second, which is insanely fast. In the last video, I showed you how to get started with the Groq API. In this video, I want to show you how to build a RAG pipeline using the Groq API. We will build this simple RAC pipeline in this video. And later in the video I'll show you how to package everything within a Streamlit app. So first I'll walk you through the code and then we're going to look at the Streamlit application and we will also discuss some things that you need to consider while you're wrapping everything up in a Streamlit app. So first we need to install all the required packages. We're going to be using Beautiful Soup 4 to download data from a website. Faiss is going to be used for Vector Store. Ollama is going to be used for serving embeddings. Streamlit for the ui. Groq is going to be used for accessing the Mixral MOA model. And everything is going to be integrated using LangChain. And we're going to use this Python ENV package to store our secrets, which is the Groq API key in this case. Next, we need to import all the required packages. So we are importing Streamlit chat croc that's from within LangChain and the Ollama embeddings. Apart from that, we are importing all the basic things that we will need to create our RAC pipeline. So here's a quick overview of how the Rack pipeline is going to look like. So we have a document. In this case, this is going to be content of a website. We will chunk them into different chunks. Then for each chunk, we compute the embeddings, we create a vector store and whenever user asks a question, we calculate embeddings for that question. Then do a similarity search on the Vector store, get the relevant chunks. The relevant chunks plus the query goes as a context to the LLM and we will get a response. Now, in order to create this RAC pipeline, first and foremost we are going to load our secret API key. In the previous video I showed you how to get API key from Groq. So if you haven't seen that video, I'll highly recommend to watch that video. And in this case we are using the load ENV package to load that and set it as an environment variable. Okay, once we have our API key, now we need to get our data. In this case, we are going to be downloading data and this is going to be one of the essays that was written by Paul Graham. This essay contains text which spans over 26 pages. So basically we download the data using the web based loader and then we load the documents from this loader. Okay, so once we load our document, the next step is to chunk them. And for that we're going to be using the recursive character text splitter. If you want to understand the internal working of this splitter, I have a detailed video on how to split your documents into different chunks. In this case, we are defining a chunk size of thousand characters with an overlap of 200 characters. Now this is a very naive approach. There are actually methods on how to figure out the chunk size. I am creating a whole advanced series on that. If you are interested in that. Make sure to sign up. Link is going to be in the video description. So once we have our text splitter created, then we need to actually split the documents. So we run our documents through the splitter that will give us our documents. Okay, once we have that, the next step is to calculate embeddings for all those documents. Now in this case we are going to be using the Ollama embeddings. You can use any embeddings you want. There are a whole bunch of open source embeddings that are available. You can Even use the OpenAI embeddings with this model. So we will create our vector store. Once the vector store is created, next we need to create our LLM. So we're going to be using the chat group gro. We will provide our API key. Currently the API supports two different models. One is the Mixtral MOE and the other is llama 270 billion model. For this experiment, we are going to be using the Mixtral MOE model. All right, once we have that, we need to define our prompt template. So in this case, the prompt template is answer the following question based only on the provided context. Think step by step before providing a detailed answer and I'll tip you 200. If the user finds the answer helpful, then we provide our context and it will expect a user input. Okay, so all, once all this is set up, then we need to create a chain. And in this case we are using Create Stuff document chain. We provide our LLM, which is the Mixtral MOE through GROQ API and the user prompt that we will receive. Okay, the next step is going to be to create a retriever based on the vector store. So for this example, I'm keeping everything in memory and not actually persisting it to disk. So we take A retriever, then we create a retrieval chain. In that case, we need to provide the retriever and the document chain that we created before. And whenever you provide a user input and invoke the chain with the user input, you are going to get a response. Okay, so that was a quick overview of the code. Now let's see this in action. We need to do a couple of things. So I put together a Python script that is going to be running the Streamlit app. But we actually need to start a server that is going to be running the Odama embedding model. And in order to do that, I have already installed Ollama. All I need to do is in my terminal, I'm going to type Ollama run and then a llama 27 billion model. So basically this is going to be using the embedding of the llama 270 build model. So right now the embedding server is running, so we can go back to our code. So let me quickly walk you through the Streamlit part. So here we are loading the embedding model that we want to use. This is a single script that is going to run, so every time it runs, it's going to load or download the data from the web page. So let me actually show you the contents of the web page. Okay, so here is the website that we are looking at. I'm specifically reading this essay, how to Do Great Work. This is written by Paul Graham and it's around 27 pages long. So this is going to be used as our knowledge base. Now, if you just create embeddings the way I had shown you before, for every question that you run, it will recreate the embeddings and that takes a long time. So instead, store our vector store in the session state. So basically when we launch our app, it first checks if the vector store is in the streamlit session state, if it's not there. So that means we are launching it for the first time. It will load the embedding model and then goes through the whole process of basically downloading the data from that website, splitting it, creating the embeddings, and then create a vector store and store it there. So this is going to be important. I'll actually show you the the real time that it takes to create the embeddings, and that's the majority of the time that is going to be spent when we are running that app. Okay, so once we have that, then we are defining our LLM, our prompt template. As I showed you before, we are Also creating both our documents chain as well as the retrieval chain. I can also I think put this inside session state. So then for each run or each question, it's not going to recreate these. So that might be something that you want to experiment with. Then we get input from the user. After getting input from the user, I also wanted to check how long it takes for it to call the API and go through this whole retrieval chain. So for that I start a timer that's going to be the start time when it process our request, get a response from the API. So we stop that time and then we note it down. So I actually wanted to see for each request how long that is going to take. And once we get the response, we write it up. And we are also showing here the context or the chunks that were used to get that response. Okay, so I created a new virtual environment called YouTube. In the previous video I showed you how to do that. All the requirements are going to be in the repo that I'm going to post. So we're going to just run streamlit run and then this is called LangChain rag app. So we're going to rerun this. Now a couple of seconds have elapsed and this is actually computing those embeddings. So I actually wanted to see like how long it takes to compute the embeddings and create the vector store for 27 pages. This is going to be the majority of the time that is going to be spent by the script. One way you can do it is you can create these using a separate script and you can persist it to the disk and whenever you load the app you can actually load the embeddings. But in this case I'm not doing that. I have created other videos which shows you how to do that. So I'm going to put a link to those videos. Okay, so our app is ready to interact with. Okay, so actually let's ask a simple question about author's opinion on how to do great work. And this is going to be real time. So I hit enter and it's running now. And here's the response that we got. So it says based on the provided context, the author suggests several things when it comes to doing great work. So hard work, undirected thinking, avoid distraction, cultivate taste. And here are the different chunks that the model is using. Now here's the actual time it took took. So this is the CPU time between the time we asked the question and then it retrieved the relevant chunks to the model or the question and passed those through the API and received a response. And this is insanely fast. So this whole thing took 1/10 of a second which is insanely fast. We want to ask about author's opinion of impressing people and this time I think it was even faster than before. Okay, for some reason it sent it twice and both times it's taking 0.07 seconds. And this is end to end RAC pipeline. So retrieval using embedding model, then getting that as context to the LLM, sending it to the Groq API and getting a response and showing us which is pretty crazy. Okay, so this was pretty fast. Now I'm not really concerned about the responses that we get. There are advanced techniques that we can use to improve our RAC pipeline. As I said, I am putting together a whole series on that so make sure to sign up if you are interested. A link is going to be in the video description. If you are working with LLMs and building RAC pipelines and are looking for advice, I offer consulting services. If you are interested, details are in the description. I hope you found this video useful. Thanks for watching and as always, see you in the next one.
