Jerry Liu, Co-Founder, LlamaIndex: Building an Advanced Knowledge Assistant

Cool. Hi, everyone. I'm Jerry Liu, co-founder, CEO of Llama and Dux. And, you know, good afternoon today. I'm excited to talk about how do you actually build an advanced knowledge assistant. So let's get started. First, you know, a lot of LLM use cases revolve around this idea of building a knowledge assistant. Chat over your data, like doing some research over your unstructured, like PDFs, and then generating a report or something. That all falls under this overall category. The overall goal is, you know, within a lot of enterprises, how do you actually build an interface that takes in any high-level task as input and gives you back an output? So the input forms could be simple questions, complex questions, research tasks. The output forms could be, you know, short answers, structured outputs, research reports. So if you think about RAG, RAG is one instantiation of this idea of building a knowledge assistant. We strongly believe in enterprise developer teams. We think they're the best equipped to build this new category of LLM applications. And we think that's a better strategy than trying to just purchase your way towards AI innovation. But we also recognize that there's a lot of productionization challenges in actually getting these things up and running very well. And oftentimes, they might diminish the value add of GenAI. So a lot of our focus as a company is really educating developers on how do you actually build something that's production quality so that you can actually deliver value to your end users. RAG was just the beginning. If you followed our company at all, our socials, our LinkedIn, Twitter, we talked a lot about RAG in the past year or so. But the issue with building a knowledge assistant with basic RAG is that it ends up being a pretty limited kind of setup. Set of capabilities. Typically what this means is you follow a YouTube tutorial online. You set something up in 10 minutes. And you show it to your boss. And it doesn't really work after some initial demo use cases. You're using a naive data processing algorithm. You're using PyPDF doing some basic text chunking. And then you're setting your top k to 5 or something. And then you set some basic QA prompt using one of these frameworks out there. And then the LLM really isn't used for anything beyond just synthesizing the response. So you're not using it for reasoning. You're not using it for tool use. And it's also stateless. You're not maintaining state over time. This tends to not work very well. And this common pattern of being easy to prototype but then hard to productionize, we've noticed as a common pattern for enterprise app development for LLMs. So some of the top categories of pain points we run into as you scale up your data and scale up your quality bar include accuracy issues, fixing your retrieval, your response quality, knowing how to improve your system, actually securing your overall system with data privacy access control, and then scaling it up to more than just one use case. You don't want every new RAG use case to take 4 to 5 months to build. Ideally, it takes days or even hours. So can we do more? There's a lot of issues with just the basic RAG stack towards this use case of building knowledge assistant. Hallucinations, limited time savings for your end users. So what can we actually do to help you build a production-ready knowledge assistant? So the 4 parts of a better knowledge assistant, we think, are the following. The first piece is a high-quality data and retrieval layer. The second is agentic reasoning over complex inputs. So layering on some elements of actually sophisticated query understanding to break down even complex research tasks into simpler ones. This involves some of the core components of agents like tool use, memory, query planning, reflection. I'll talk about that more in just a bit. Three is generalizing some of the outputs to go beyond just chatbot responses into agentic decision-making and output generation. So instead of just getting back a chatbot response where you as a human are responsible for figuring out what to do with it, we're seeing more and more use cases where you're just generating like a PowerPoint or PDF doc. You're filling out a tax form, or you're actually taking actions like scheduling an invite, sending an email. And then lastly is the surrounding infrastructure you need to actually go to production. Instead of just coding stuff up in a Jupyter Notebook, what are some additional tools to actually scale this as a production service? Service. First, on high-quality data and retrieval. We think any LLM application is only as good as your data. Garbage in, garbage out. Good data quality is one of the core components of, like, core necessary ingredients of ensuring good quality within any LLM application that you're building. Oftentimes, we're noticing for anyone building this chat over your data use case, a key culprit of, you know, bad response quality is because the data wasn't actually processed the right way. And this is because data can be very complex, right? If you have a PDF, you might have, you know, tables, charts, and images within it. If you have a slide deck, it's even more complex. You might have, like, different types of visual elements within it. You might have structured data. You might have Excel sheets. And all this is a little bit complicated to deal with. So we think there is basically this need for this ETL stack, this new type of ETL stack for, you know, AI engineering. And some of the components of this ETL stack include parsing, PDF parsing, and having a good parser, figuring out how to chunk your data, how to extract out the right structured annotations from that data, and of course, how to index it into a storage system, vector store, graph store, doc store, SQL store, pick your favorite. We think a lot of this can actually be automated as well. AI engineers don't want to spend half their time on figuring out the right chunk size. It's kind of like a dumb thing to do. And so we want to help provide, you know, what are the best algorithms to really tackle different types of complex data to help solve your use case. A lot of documents can be classified as complex. You know, if you're just indexing a simple text blob, that usually is kind of trivial. But a lot of documents are formatted for human readability, which means they have like two-column layouts. They have a bunch of like weird nested tables. You have charts, graphs, images. And users want to understand that data. If you have anything from like a 10-K report, which is actually pretty simple, to like a full slide deck, to your Excel sheet, which has like a million rows, how do you ask arbitrary questions over this complex data and get back arbitrary responses? So the first piece is we'll talk about document parsing. We've talked about this for a bit. But like most document parsing is actually totally inadequate for LM use cases. If you're using an off-the-shelf PDF parser like PyPDF and even a lot of the other kind of like state-of-the-art or not state-of-the-art, just like general tools available for document parsing, you'll notice they do a terrible job at dealing with tables and also charts and images and basically anything within a document that's a little bit nonstandard. The issue with this is that if your parsing is off, that is like going to be one of the core reasons why your data quality is bad and it's going to lead to hallucinations down the road. It's going to format your numbers and text in a really weird manner such that the LLMs can't actually understand it. It. So we spent a lot of time building this ideal GenAI-native parser. How do we build a good document parser for LLMs? And some of the core requirements we thought about is, one, it has to be really good at parsing tables into text and semi-structured representations. Two, it has to actually parse text into semantically coherent chunks, which let you chunk it well later on. It has to extract out visual elements into structured formats and help you build multimodal RAG over it. And also has to allow you to basically do automated metadata extraction. The main differences from like a traditional PDF parser is I think a lot of traditional document parsing deals with like bounding boxes and stuff. And we didn't really think about that. Like in the end, if you're just stuffing this into an LLM, it doesn't really matter if you also input the coordinates of the bounding box. But it does need to be formatted in a way that the LLMs can understand. So Llama Parse is the document parser that we've launched earlier this year. It's an advanced document parser specifically for reducing LLM hallucinations. It's part of our overall enterprise platform called Llama Cloud, which helps solve end-to-end RAG with fast time to value and also high performance over your multimodal complex unstructured data. There's a lot of users on it. And just to go without going in depth with too much details, but kind of just reiterating this idea that parsing is important. We took different types of data like the Caltrain schedule, financial reports, and we compared it against a baseline RAG approach where you basically just use a standard parser. And even if you don't have fancy chunking, if you have good parsing, that's actually a core ingredient towards reducing L on hallucinations. Of course, you can also add advanced indexing into it. So you can combine parsing with more advanced indexing retrieval algorithms to model all this data within a document. So you don't just do paragraph-level chunking and embedding, but actually do some hierarchical indexing and retrieval, which tends to give better quality results. And a general good RAG architecture that we've come across is you decouple the embedded elements a little bit from the thing that you're actually indexing and retrieving. So, you know, let's say you extract out tables and charts into top-level summaries. You index those summaries into a vector store, but then you dereference them into the underlying entity like the image, text blob, or the table. And so the slightly more advanced, you know, indexing retrieval algorithm is generally pretty good at helping users get back the results that they want over this like, you know, complex unstructured data. So that's part 1, high-quality data and retrieval. And we've spent a lot of efforts on that. But at the same time, that's not the entire story. A lot of people these days are interested in agents. I'm sure many of you are probably building out different types of use cases using agents, quote unquote, in production. And so let's talk about what are some of the base ingredients of agents and how that can help us build a better knowledge assistant. One of the issues with Naive RAG is that it works well for pointed questions, but it fails on more complex tasks. If you're trying to ask like a summarization question, a comparison question, a multi-part question, or a general research task, you're going to notice that just retrieving the top K most relevant chunks and leaving it at that oftentimes is not going to cut it. So we think that different people have different definitions of an agent. At a base level, it's basically just a Python or whatever program with LLM calls in the middle. But we think of it as there's different core ingredients of an agent, and there's a progression towards full agentic systems. And they all have trade-offs in both cost, latency, but then also performance. Right? And so if you just include some of the core agentic ingredients in your application, this tends to be much cheaper, but the application is just maybe a little bit less flexible. If you build like a full AutoGPT application that's fully autonomous, it's going to cost you a lot more, probably not going to be super reliable, but can hypothetically do a lot more tasks. And we coined this term agentic RAG earlier on this year where, you know, Really, if you think about RAG as like this overall pipeline and you just think about an agentic reasoning layer on top so that it's able to leverage some of the core agentic ingredients to really do some query understanding and processing before feeding it into any sort of retrieval, whether it's a vector database or a SQL database or a graph store or web search, then it's a nice way of thinking about, you know, how do I build something that's a little bit more advanced than the basic RAG pipeline? It will use agentic reasoning loops to break down, let's say, a multipart question through chain of thoughts into simpler questions that you can then launch against a vector database or any other data source to again answer the question at hand. And the end result here is that you're able to build these slightly more sophisticated query, QA interfaces over your data that can handle complex questions. And a common pattern we see with anyone building these types of chatbots is you have some agent reasoning loop. You have a set of tools. It's like a vector database, SQL database, so on and so forth. And it's able to leverage some of the core components like chain of thought, memory, reflection to, again, generate an answer. The end form factor is still a chatbot, but it's a slightly better chatbot. To that point, we see a difference between unconstrained versus constrained flows. And this starts getting into the definition of an agent. I'm sure many of you are probably using different types of agentic frameworks, some of which are higher level, some of which are lower level. And an agent could be just defined as a program, like a Python program, where you explicitly define most of the control flow, but then interleave little bits of LLM calls in the middle. That's what we call like a more constrained flow. You can integrate like different agent ingredients like routing. You could import like or input like a simple prompt for reflection. And we think this is a pretty valid way of going about things because there's a lot of distrust within kind of different enterprises around letting agents just run amok. And this is actually the most common form factor we see where you're still actually controlling most of the logic, but you're just defining like a sequence of steps to integrate LLMs into. As LLMs get better, we see more like people be comfortable with kind of having the agent actually dynamically figure out the execution path. And so, you know, one agent given 10, 20, 30 tools right now doesn't maybe quite work. In the future, it might, right? And so as long as you're able to prompt the LLM with the right set of tools, the right descriptions, the right few-shot examples, it should be able to dynamically take in a task and actually just figure out not like a fixed sequence of steps that's predefined, but actually, you know, what is the best set of actions to take to solve this? And that really just reduces down to, you just think about like a set of an agent and then a set of tools. These tools could be other agents. They could be, you know, just deterministic like services that you access. But it's very similar to how you'd imagine like a human behaving, right? So we've invested in this a decent amount. And basically, for the rest of this year, we're going all in on agents, agentic orchestration. We think an agent orchestration framework should have the following properties. It should be event-driven, so model every step as listening to input events and emitting output events. Allow you the low-level abstractions to basically define whatever you want, basically. So instead of being super high-level and being not easy to customize, actually just letting you write kind of different types of Python code. Speaking of Python code, we don't think it should be super unintuitive to do. I think there's different types of paradigms for building these things these days, and some of it just feels a little bit awkward. And so ideally, we just want you to write a Python class with some Python code and then have it run as an Adrentic system. And of course, the requisite tools around debuggability, observability, and evals. I'm going to skip the next two slides just for the sake of time. And then now that we have an agentic reasoning loop, the next two sections are how do we not just improve the sophistication of the input processing, but actually improve agentic decision-making and output generation. So again, going back to this idea of a chatbot, the thing you're getting back as an end user is you're just getting back a text response, maybe with inline citations and sources. But let's say your overall goal was to produce this like slide deck, right? Like I need it by tomorrow and I need to basically— let's say I'm using an AI tool for that. I'm going to go in, copy and paste the pieces of text and actually input it into PowerPoint or Google Slides and then, you know, format the pictures, make them all pretty, and then, you know, give that to my boss the next day. But we think like, you know, the time savings of that is not very high because you're still actually responsible for basically doing most of the heavy lifting. And you're basically just using RAG as a search and retrieval tool. Ideally, agents actually have the capability to help you produce these outputs. And in that mode, that means that as a user or as an end user, you become increasingly responsible for the supervision of these outputs and the correction, as opposed to being directly responsible for producing the final output. And some initial form factors, we talked to a lot of different companies that we're seeing include these categories of report generation, data analysis, and action taking. An example of this, which we have as an example notebook, but it's basically how do we actually generate a multimodal report, like a research paper that has both text and images. And we have an example here where we process an entire slide deck or multiple slide decks from different companies and then generate a condensed summary that has Markdown text, and then also interleaving images. And it's not actually that complicated to do. It's just a little bit of a mindset shift from a standard chatbot. So for instance, here we have a simple researcher-writer multi-agent paradigm, which I'm sure you've seen in other patterns too, where a researcher basically retrieves relevant documents or chunks or resources, puts them into some data cache. And then the writer just uses a prompt. And this structured outputs is a key ingredient. To generate the final report. We have another repo that I think one of our head of DevRel just launched, Laurie, like this week on basically generating PowerPoints automatically, right? And being able to generate like an entire slide deck from a set of resources. Another top external-facing use case that we see is probably customer support. I mean, I'm sure many of you probably realize this too, but It's by far basically one of the use cases that actually has like a clear ROI, like quantifiable ROI. And a lot of the flows are some mix between like deterministic flows and also using agentic reasoning in the middle, right? You want to do some basic like intent detection, but then you also want to have some like predefined flows according to your guidelines. And then there's different actions that you can take at the end to basically help, like, you know, deflect or basically resolve the customer issue. Oftentimes, the customer support bot is internal-facing instead of directly end-user-facing due to, you know, the skepticism around having this, again, directly be used by the end user. And that's okay. But basically, you know, the goal of this is to eventually automate more and more of these pieces to basically increase the efficiency of these support agents. An example, personal finance support bot that we basically created as a reference repo is maybe first, given a user input, you want to first authenticate them. Two, look up their finances. Make sure that they can check their account balance. Think about all the options you can do on your phone. This is all predefined. But then also given natural language inputs, decide to perform certain actions like transferring money, calling, talking to a support agent. And all these steps basically require some form of human in the loop. I mentioned this, but I think action taking and output generation potentially leads to a much higher ROI in terms of time savings and capability improvement. But the limitations here is that we see enterprises automating more and more steps at a time. Right now, it's usually in a trap bot, just one step at a time because you're still responsible for taking the outputs and doing something with it. But as humans become more comfortable with the agent doing things on its own, then again, it takes on the role of supervision and correction as opposed to directly being responsible for the final output. The last bit I want to talk about is just so far, you can basically write all this as Python code and run it in your notebook, run it locally on your laptop. As a developer, how do you actually use the right architecture and infrastructure components to serve these agent workflows in production? And so some of the core goals is basically to have this production service that can scale up your agents very easily, let different developers reuse these agent services, call them through a client, and enable still human in the loop for the end user. And of course, all the requisite features like streaming and and sources, citations, and debugging. We recently, well, renamed this repo. It used to be called Llama Agents. But we have this open-source tool that's fully open source that allows you to basically deploy your AdronTech workflows as microservices. So as first, like a server that runs, you can access it via a client. And you can also directly deploy it on Kubernetes. For instance. You can also swap out different implementation— implementations of this message queue. And the message queue could be like RabbitMQ, it could be Kafka. But the overall architecture that we see is that you have like agent 1, 2, 3, and more running as different services that communicate with each other through a centralized message queue. And the programming is pretty intuitive for this. Again, you just model everything as like an event-driven service. And then if you can just deploy this, then you basically model your entire agentic system as microservices that's, again, observable and also scalable. So it's very easy to add more agents over time. Check it out. I mean, I think the repo is linked below. But we're also very open to feedback on improvement with this. Some included components here, we're launching— this is a sneak peek. It's not out yet. But it's an open-source tool on an agent debugger where you can program a LlamaDex workflow and then basically debug every single step of that agent, inject counterfactuals if you decide to input a new input or something, and see what the resulting output would be. And it gives you a trace, basically. But it's really like an interactive trace. You can go in and, again, kind of edit things and see how it plays out. And then lastly, and I didn't really talk about this too much during this talk, but this could probably be worth an entire different session, is how do you actually have the right UX for the agent? Again, if it's just running as a server or as a notebook, it's not very interesting. How do you actually serve this as a UX to the end user so that they can not just have a chatbot interface, but actually some interface where they're able to collaborate with the agent inject feedback, modify the plans, view the different UXs, and see what the— what actions the agent is taking. And that effort we call Create Llama. In one CLI command, you can basically set up a full-stack application. Again, it's all open source and free. So I think that's basically it. Thank you.