No Chunks, No Embeddings: OpenAI’s Index‑Free Long RAG

One of the burning question everybody asks me about retrieval augmented generation system is what chunking strategy do we use? What is the best embedding model and what is the best vector store? OpenAI has an answer for you. They recently introduced a new multi agent retrieval augmented generation system which mimics how humans read information. And this one is indexing free. So you don't have to worry about your chunking strategy and selection of embedding models anymore. All this is possible due to long context models such as GPT 4.1 that has 1 million tokens context window. So they basically figured out a way how to combine multi agent system with long context window to get really great retrieval results. But this system is not for everyone. There are very specific use cases in which you're going to get some awesome results. We're going to talk about both the pros and cons of this new system and when to use it. Now this new technique was introduced in this blog post called Practical Guides for Model Selection for Real World Use Cases. And in this they show how you can use a multi agent system which are powered by different LLMs to build robust systems that are going to work in real world. One of the use case that they discuss is a complex question and answer on long legal documents using long context retrieval augmented generation system. So in this video I'm going to show you how this system works. We're going to look at specific code examples and also I'll walk you through a practical example of how you can start using this in your own workflows. Here is how the approach works at a very high level. So you take your document, then we have the first pass which is skimming through the document. So the idea is we want to look at the structure of the document divided into manageable sub section. Let's say chapters identify which chapters are relevant to our user query and which are not. So you're going to be able to simply discard a subset of the chunks or chapters just by looking or skimming through the document. Then for the chapters or sections that are identified as relevant, we can repeat that same process again. We can divide each section into subsection. Then the system looks at each of the subsection and figures out is it relevant or not at a high level. If it's not relevant, we discard it and we repeat this process at multiple depths. That is a hyper parameter that is going to be controlled by the user. At the end we're going to end up with subsections. These can be paragraphs which are going to be used by the system to generate answer. And then there's a final verification step which makes sures that the answers we are generating are actually grounded in the text that we selected. So this ensures that the system is not hallucinating. Now each of this step can be treated as a sub agent and for each one of them you need an LLM with different capabilities. And that's exactly what the OpenAI team is proposing here. Now here's a detailed breakdown of the architecture in terms of implementation. So the first step is figuring out how many chapters do you want or how many chunks do you want? At a first phase of high level breakdown they Recommend to use 20 chunks of equal size. The only condition is that each of the chunk is supposed to end with a valid sentence. So you take your whole document divided into 20 different parts and then we use a long context LLM such as GPT 4.1 mini, or you could potentially use a Gemini Flash model. But the goal is going to be to use a long context but very inexpensive model which looks at each of the chunk along with the question and say that okay, at a high level, can I answer this user question based on this chunk? Does this specific chunk contains enough information that could be used to answer the user question? Now we identify those sub documents or sub chunks. They use a concept of Scratchpad which gives the non reasoning models the ability to reason through this evaluation process. We're going to look at that in more detail because it's a very interesting idea, but you're going to end up with a subset of documents or subset of chunks that can potentially be used to answer user question. Now for each of those sub chunks we break them down further. So this is a multi step process. The idea or the goal is to identify specific paragraphs that we can use to provide accurate answers. Once we identify those paragraphs then we use a more powerful model such as GPT 4.1 that is going to generate structured output, but that is not the final output. You also want to have a verification step using a reasoning model such as O4 mini that is going to act as a judge to validate or verify the answer that the system generated. Now as you can see, it's not using any pre created index, but it's processing all the text for each and every query. So even though the system is going to be very accurate, you can't use this system for every application. Let's say for a simple chart with your document it's going to be a lot more expensive and very slow. But if you are working on a report generation and you need factual data where latency is not an issue. This could be a very accurate implementation of a retrieval system that is going to give you really good answers. And one such example is looking at legal documents. So here's our trademark Trial and Appeal Board Manual of Procedure which is about 1200 pages document. Now you can use this as a reference in the Patents Office. One idea would be you just put this whole document in the context of a long context model and you are going to pray that the model is able to retrieve information, which is a really hard thing to do. Long context models are usually really great at needling the haystack test. But if there is some sort of reasoning involved, that's where these models actually struggle a lot. Especially if you are trying to figure out information that is contained in multiple places and the model has to reason through it in order to generate the response. Okay, so we're going to look at that example. Now here's the general workflow. We're going to load the document divided into 20 chunks, then ask the LLM which chunk contains the relevant information. We're going to do this process multiple times. So there are going to be a depth parameter that we'll have to define. And once we get to a certain depth, we'll use the relevant information to generate answer. And then there's going to be a verification step and each of these stages we're going to use different models. What I have seen is a lot of time when people are even building these Agentix systems, they will use state of the art models such as Gemini 2.5 Pro or even O3 series or GPT 4.1 series. But the thing is that you don't need the most powerful model at every step. There's this concept of workhorse models which can process a lot of tokens at a relatively low cost and relatively low latency. So for the tasks that does not involve a lot of reasoning, these smaller models are really capable and you should be using them. But for the task which involve reasoning capabilities or deeper analysis, that's where you want to use the bigger models or the reasoning models. So you have to figure out a balance between the performance, accuracy and latency. Okay, so let me quickly walk you through how this whole system looks in code. So here I put together a Google Colab notebook based on their implementation. So the first step is to download the document. Now we are limiting it to 920 pages. Essentially this is going to load the document. If you look at this document, it has about 1200 pages, almost 600,000 words, which translate into almost 900,000 tokens. Now there could be documents which are longer than the context of the model. So let's say if you have longer than 1 million tokens in a document, then you want to divide them into sub documents, run the same process in parallel through each one of them, and generate the final responses and show those to the user. Okay, so the next step is to split the documents into chunks of equal size. Here we tokenize the text, then we select individual sentences and then iteratively add those sentences and try to figure out can we make chunks of equal size. If you look at this distribution, you're going to see there are chunks which are about 33,000 tokens. This is probably the smallest chunk, and the biggest chunk is all the way up to 62,000 tokens. Right? It's in the same ballpark, depending on how the document is structured, what are the paragraphs. But the goal is going to be you want to divide it into relatively equal sized documents. Now here is one of the chunks. You can actually see that it's not preserving the structure of the document or the structure of the text you could preserve. Also, divide your document into markdown and then do this chunking process on top of a markdown. Okay, so here's how the flow looks like. We process the document into 20 chunks. Now next we want to look at each of the chunk that we identified and figure out whether we can use that chunk to answer questions that the user is asking. So this is where this step of content routing comes into play. So here's the main function. We get all the individual chunks along with the user question. And again, we'll have to repeat this process for every user question. So this is going to be a relatively expensive process. Now here's a system prompt. You are an expert document navigator. Your task is to identify which text chunk might contain information to answer the user question. Record your reasoning in a scratch pad for later reference. Choose chunks that are most likely relevant. Be selective but thorough. Choose as many chunks as you need to answer the question. And first think carefully about what information would help answer the question. Then evaluate each chunk. Now they are introducing this idea of a scratch pad. This was introduced in this paper. Show your work. Scratch pads for intermediate computation with language models. Anthropic was the first foundation LLM company which really popularized the whole idea. And the idea is that even if you're not using a reasoning model, a non reasoning model can still be instructed to check the work that it's doing. Now, in this case, the scratchpad is just for the internal thinking. So we get all the chunks. Then we, we use this scratchpad as a tool. It's basically a text string where the LLM is going to add its reasoning and why it thinks a specific chunk is relevant to answering a given question. And it also has the ability to update that. So we provide a Scratchpad as a tool to the LLM and the output is going to follow this very specific JSON schema. Here it's going to provide the the ID of chunk that it thinks is relevant along with the reasoning based on the Scratchpad tool that is provided to the LLM. And we make sure that it's going to be using Scratch Pad as a tool for every chunk that it is analyzing. So it's going to make a couple of calls to the model. First, it's going to analyze all the chunks individually, put the relevant information in a scratch pad. Then we use that as a second pass to select specific chunks that are returned through the initial pass. We actually need to do this process recursively. So the way it works is we're going to initially provide the actual text of the document. There's a minimum token size of 500 tokens. It will split this into chunks. Then we recursively call the same function to further divide it into smaller and smaller subsections until we get to a specific depth that we are going to control as a user and the final output is going to be just a list of paragraphs. Now to put everything together, here's a question. What format should a motion to comply discovery be filed in? How should signatures be handled? So first we call that recursive decomposition function. We get the results and you can actually see how this recursive division or divide and conquer looks like. So initially it splits the document into those 20 chunks. Then it reasons through each of those chunks and here it selected these five chunks which it thinks are the most relevant chunks. Here's the reasoning behind it. This is coming from the scratch pad. Then it takes one of the sub chunks divided into further smaller chunks. Again, 20 chunks, right? Here's another one. Here's another one. Right? And this process repeats, right? So this is the second depth of reasoning. We could continue doing this. So here's another one. So you can now see that Instead of those 40,000 tokens, we are dividing or subdividing up our sub chunks into much smaller tokens. There is also a limit of how small these final chunks can be. So it actually has to look at that when it's trying to figure out when to stop. So then for each of those sub chunks actually has reasoning within Scratchpad, it really validates and checks each and every paragraph to make sure that it actually is able to provide relevant information. Now, in this case, we start from the go global context, right? And we are basically breaking it down into smaller pieces which are most relevant to our question. Again, this is completely index free, so we're not creating an embedding based index in the beginning. And that's why this system has the ability to actually preserve global context. Even though we are selecting smaller paragraphs. We also don't limit the number of paragraphs because we are using a long context model. It's very dynamic in nature. We repeat this process, then we need to generate our final answer. So here's a pedantic implementation of how that is going to look like. The system prompt is you are a legal research assistant answering questions about the Trademark Trials and Appeal Board Manual of Procedure. Now, something to highlight here. It's very important that you provide enough context to your generation model. If you just tell it answer the question based on the provided context, it doesn't actually know what the user is looking for. But if you assign it a role and you tell it what the language is supposed to look like, then you're going to get much better answers, right? So here we are telling it exactly where the context is coming from and then what exactly we want the answer to look like. So answer questions based only on the provided paragraphs. Don't rely on any information or foundational knowledge or external information phrases of the paragraphs that are relevant to the answer. Now, since we are breaking it down to single paragraphs, we can actually be able to reference those. Okay, so we pass this system prompt along with the paragraphs that we have retrieved so far and the system will generate an answer. And you can see here, it started generating the final answer for us along with specific citations to very specific paragraphs. Now, the last step is answer verifications. I'm going to just cover it here because this is the same implementation. So we basically look at individual paragraph and then use LLM as a judge. In this case, we have another system prompt which will control the behavior of our LLM as a judge. So we are telling it, you are a fact checker for the legal information. So here we are again providing enough context of what exactly it's supposed to validate. Right? So your job is to verify if the provided answer is factually accurate according to the source paragraphs. Use citations correctly, be critical, and look for any factual errors or unsupported claims. Assign a confidence level based on how directly the paragraph answers the question. So we want it to categorize each paragraph into. We use a reasoning model because we want to have a powerful LLM that is going to act as a judge. The system is going to look at the answer along with the source contents. Those are the actual paragraphs that are provided and give it a confidence score. In this case, the confidence score is pretty high. Okay, so that was a quick overview of how this system works. Now as you can see, since you are doing multiple calls to the LLM, you have the verification step. So the answers that you get are going to be reasonably accurate. However, it's also going to add a lot of cost. So they have the cost division here. Now they say estimated fixed cost versus variable cost. If you're using a traditional rack system you're going to have about $0.40 for embedding and metadata generation. I think you also need to add storage cost as well. For the agentic system you're going to have zero pre processing cost because you're not creating an index. However, for each query you are going to make multiple calls to multiple different LLMs. So for example initial routing, that's $0.10. Two recursive levels, that's $0.20 more, right? In general, if you add up everything together for this hypothetical query, you are looking at about 36 cents which is extremely expensive compared to a standard Rack system. Now this is where I think it's very important to understand different applications. So for example, if you're looking at an application in a legal domain or any place where you want very accurate answers, you are going to be looking at higher cost. Again, you also need to consider whether your system needs to be low latency implementation. If that's the case, an Agentix solution with a verifier is probably not the way to go. So depending on your need, there are definitely use cases for a very expensive but highly accurate system. They have listed some of the benefits. First is zero ingest latency because you're not doing any pre processing. Dynamic navigation. This mimics human reading patterns. Focus on promising sections. Then they talk about cross section reasoning. Right? Trade offs are higher cost per query that we looked at increased latency and in the current form this is going to be limited when it comes to scalability. However, there are certain things that you can do. One of them is you can actually use caching. I think most of the LLM providers now also provides the ability to cache your content, so that could potentially improve both the latency and the cost aspect. Another idea that they are proposing is to generate knowledge graphs. That is going to be a one time process and then a GPT 4.1 like model will be able to traverse that knowledge graph. This goes back to creating an index, but if you want to figure out what different entities are present and what are their relationships, knowledge graphs are really good for preserving those relationships. You could improve some scratchpad. So right now the way it works is we're just adding information in there and it keeps track of what chunks are relevant to the user query. You could also have the ability to remove information from that or edit it and then adjust the depth. Now you can have higher depth, but that will increase the latency and cost. In certain cases, like legal documents, you probably want to have more depth. For example, if you're looking at sentence level citations, right now we're just doing paragraph level citations. Now all of this is possible due to these huge context windows. I think we are going to start seeing a hybrid implementation between RAG in a traditional sense which is going to use some sort of indexing and then once a document is retrieved you pass it on to a long context LLM such as GPT 4.1 or the Gemini models. Do let me know what you think about this approach. I do see a lot of promise in this. Let me know if there are any specific implementations you have in mind or specific improvements that you can think of. Let's discuss them in the comment section below. I hope you found this video useful. Thanks for watching and as always, see you in the next one.