Revolutionizing AI Data Management with Jerry Liu, CEO of LlamaIndex

You're listening to Gradient Descent, a show about making machine learning work in the real world. And I'm your host, Lukas Biewald. Jerry Liu is co-founder and CEO of LlamaIndex. He started his career in machine learning and autonomous vehicles, and since that has built one of the most popular frameworks for building text segmentation and text retrieval applications. This is a super practical conversation and I hope you really enjoy it. All right, Jerry Liu, good to talk.

Yeah, thanks for having me on, Lukas.

Yeah, thanks for joining us. I feel like where this needs to start is what is LlamaIndex and how could I use it?

Okay, yeah, getting straight to it. So LlamaIndex, for those of you who don't know, is a data framework for building LLM applications. Our goal, our company mission statement, is to connect the power of LLMs to your company's data, whether it's like personal data or organizational data. This includes everything from PDFs to CSVs to PowerPoints, Excel sheets, to APIs, databases, pretty much anything. And we kind of realized this all the way back before the ChatGPT hype, when people were just starting to hack around on GPT-3. And I think as people were developing with it, they realized that it had a lot of potential, but it had a lot of potential not just for memorization, but also for reasoning. And so as people tried to actually feed it their own data and try to stuff the prompts, there were certain patterns that emerged in terms of the best practices for how do you actually orchestrate data for use within LLM. And so we kind of found ourselves at the center of that by launching a project very related to this, uh, around November of last year. So it's been a crazy ride. It's been about a year since we started the project, and now we're basically a very mature framework to help users build LLM apps over their data for a wide variety of different types of use cases.

So you're a framework for helping build LLM apps, and, you know, there's like a lot of, a lot of projects that might call themselves that. So like, how do you fit into the ecosystem? Like, what's your like really sweet spot?

Yeah, so there's a few frameworks for helping build LLM apps. Besides LlamaIndex, there's also LangChain, there's Haystack, there's a few others that are focused on certain niches or areas. I think there's certainly overlaps with a few of these, but our focus has always been on connecting LLMs with data. And so if you actually take a look at the development of the framework, there's certain things that we haven't really done. So for instance, things that we haven't really done include a deep focus on very strong prompt abstractions or a deep focus on, for instance, agent simulations. A lot of our abstractions have been tailored around stuff like RAG, Retrieval Augmented Generation, agents in the use case of data analysis, search and retrieval, and being able to act upon information, conversational memory, those types of things.

Yeah, I guess these days when I think about data applications with LLMs, I'm thinking RAG, like how, um, is that like 100% of your use case or most of your use cases? Like how do you think about that?

I'd say it's the majority of our use case. I think a lot of users are using the framework for stuff like structured data extraction or adding like agent capabilities to RAG. It also kind of depends how you define RAG because I think RAG in its very simple form is just stuffing stuff into a vector database and doing some sort of like top-k retrieval and search. So you can certainly do that. You can index like a large document corpus. I would say probably like over 75 to 80% of our users are focused on the RAG use case with LlamaIndex. There are also users using LlamaIndex for structured data extraction, agentic stuff, summarization, and basically kind of using some of the modules in more of an ad hoc manner.

So maybe we should break down what RAG is for folks who might not know. Like, can you, can you kind of walk me through how RAG works? Wortz?

Of course. I think if you've been on like the Twitter or AI space or listened to podcasts recently, I'm sure the term RAG has been thrown around a few times. And I'd say it's actually grown in popularity probably since around like middle or earlier this year. RAG stands for Retrieval Augmented Generation. And it's kind of a term that's evolved over time. It was a term that I think came out of a paper a few years ago. The high-level idea is actually pretty simple. It's basically you don't train the model anymore, you fix the model. And then all you do is figure out how do you import data from wherever your data storage system is, whether it's a SQL database or a vector database, and you figure out how to stuff it into the prompt, right? Because once you actually stuff stuff into the context window of an LLM, then you can ask questions over the stuff you, you put in. Um, and it's basically a, a very common trick people figured out very early on with LLMs, especially with GPT-3. A 4,000-context window. You can just put in relevant context and then given the relevant context, all of a sudden the LLM can reason over new information. So the trick is how do you actually do that if you have, for instance, gigabytes or terabytes of data? You need to do some sort of retrieval mechanism first so that you fetch just the most salient pieces of context and then given the most salient piece of context, you feed it to the LLM to answer the question.

And I guess a key thing here is actually finding the most relevant bits of information in your database? Like when I think of that, I'm always thinking vector database, but you said SQL is also possible. Like how does that work?

Yeah, I think that goes into the part where maybe different people have different definitions of RAG. And so for a lot of people, SQL probably doesn't actually fall into that definition. Typically that lives in a separate stack, like a text-to-SQL type stack. I think for For us, our general interest is kind of imagine this like hypothetical scenario, right? Maybe taking a step back just in terms of the end user use case. You have a ChatGPT-like interface and then you have like just a mess of data sources and somehow just magically you connect all these data sources to ChatGPT regardless of the diversity or size of this data. And then you can ask any question you want and get back the answer that you want. And so there's a lot of complexity involved in actually setting up a system that can do that. But that to us is kind of this holy grail of RAG that we aim to achieve.

And I guess when you set this up, how does it work? Are you opinionated on which vector database to use? Do you think it matters?

That's a good question. I don't think we're opinionated on the vector database themselves. We have a rich ecosystem of vector database providers. We have 35+ vector databases. Integrations. And so pretty much any of the ones out there you can use with LlamaIndex. We are starting to develop opinions on the best algorithms of depending on different use cases for ingestion as well as retrieval. Like for instance, like there's a lot of techniques floating out there, each with different costs in terms of, or trade-offs in terms of costs, latency, and performance. So for instance, do you do like top-k hybrid search on top of your data? Do you do proper, like what type of chunking do you do for like code files, for SEC documents, for like legal filings, those types of things. And so that's something we're still actively working on, but we haven't fully formed the opinions yet. I think my typical, my idea here, especially as an open source project is that it's nice when you start from the bottoms up because at least you give users the tools to kind of like hack around on things themselves, like the developers to build whatever custom solution they need. Once you've become a bit more opinionated up the stack, you consolidate some of the practices into these abstractions that you can provide for users.

Okay, so maybe let's get specific. At Weights Biases, we have a whole bunch of different LLM use cases, but one that I think really fits your model is something called WandBot, which is actually in our Discord server where it searches our documentation and answers questions about how to use Weights Biases. If you're me and you're just starting out or you want to maybe improve this WandBot, what would be the kinds of things that you'd think about? What would be maybe your first recommendation of a vector database? What would be things to try? What would be the right way to chunk the stuff so that we're answering questions well?

Yeah, that's a good question. Also, if I'm not mistaken, I think WandBot actually uses LlamaIndex under the hood.

It does.

We did promote it. It's a great project. I think there's a few things to keep in mind. Let's say you're a developer, you followed the Quickstart guide of LlamaIndex or watched a few YouTube video tutorials and you've set up a RAG pipeline over your system. And so you've imported some data and now you want to, you've asked some questions over the data and then you realize that for some questions it's not working. So now what do you do? The first step that I typically recommend is to define some sort of eval benchmark. Basically collect some dataset and it could be like 50 data points, 30, 40 is probably the minimum, but just some curated collection of data points. And the reason for this is whenever you're trying or tuning something, you want to get a number out of it and make sure that whatever you're trying has a higher number than the baseline. And so this is a good way to benchmark different strategies.

This is kind of tricky because For Weights and Biases, there's lots of possible answers that could be okay. How would I set up an evaluation dataset that I could reuse with all the different strategies that I might try?

Good question. I think to be fair, evaluation is something that I don't think everybody does. This is more just a principled setting, the ideal. Of course, there's technical challenges, how to do it. Honestly, the simplest is probably just get a human to do it. Just get a human, go through 50 questions that you ask this bot and just write down the questions and answers. There are synthetic ways of generating this too, and we have some tooling in LlamaIndex to actually synthetically generate an eval dataset for you.

Okay, but before we get to synthetic, just say I've labeled some questions and answers. Are you saying look at answers and mark if they're they're good or bad or literally pose a question and be like, this is my ideal answer here?

Yeah, so I guess for just a very basic dataset for evaluating RAG could just be you have an input question and then the output answer. And so basically you come up with the input and then you also give what the ground truth output should look like.

But of course it's not going to exactly match the output.

Of course. That's why LLM-based evals is kind of— there's a lot of companies that I'm sure you know as well and also efforts going on in Weights Biases, I'm sure, towards analyzing this. A typical way that you can basically compare or do evaluation of a predicted response from the LLM and compare it against a human ground truth label is just use another LLM as a judge. There's a lot of papers out there as well as works on fine-tuning this stuff to basically use, for instance, GPT-4 to measure the quality of your response given a predicted one. This is still a work in progress. There are definitely imperfections, there's noise, but as these models get better, they typically exhibit a bit less stochasticity and more consistency in being able to compare different responses.

Okay, so now say I did it and I have done it, And our, this is a true story. Our first accuracy measured is around 60%. Doesn't really feel good enough to put in front of customers. Like where do we look to improve our flow? What's like the typical lowest hanging fruit?

Yeah, so I've given actually talks on this in a variety of different settings. And I think to be honest, like our own sense of best practices is constantly evolving. And so like, you know, it's very possible that a month from now after the time of this reporting, like I might have slightly different opinions on, you know, what are the best things you should try first. But I think typically the first things you try, and I have this like diagram, but that basically outlines, there's kind of easy things like table stakes things you should try. And then there's harder things that you should try. Basically easy things will be more incremental and maybe nudge up performance by a few percentage points. Harder things will basically just unlock new capabilities for your entire system. Easy things to try include tuning your top-k, your chunk size, your parsing strategy. Typically those are probably some of the main culprits. If you have, for instance, your total retrieved chunks is too small, you're not going to have enough context to actually answer a lot of questions. If your total context is too big, you start running into speed cost issues as well as loss in the middle problems, so low precision. And the thing is this typically depends on the use case as well. And the types of questions you want to ask. The other thing, the other kind of way of looking at this actually, and I have like a slide that I haven't really shared publicly, but maybe just like a mental map of how to think about this is basically like a table where like the left column is the set of pain points that you're running into. And the right column is like the set of solutions that solve some of these pain points. So instead of looking at it in terms of like, oh, easy stuff to like hard stuff, Like a pain point, for instance, could be very specific. It could be like, oh, this retrieval is actually not giving me back the right results. So if the pain point really is around like retrieval, then what you want to do is you want to invest in better retrieval algorithms, right? You want to use something to actually, given a question, give you back the most relevant context to answer this question. If the pain point is the LLM is— the retrieval is fine, but it's the LLM is hallucinating, then you want to invest in potentially more advanced prompt engineering and also the model itself. The other—

Can we—

Oh, sorry, go ahead. The last thing I'll just say is there's also just maybe one way of looking at this is there's a huge long tail of questions that are very complex that a user might want to ask. Just think about the spectrum, the distribution of all types of questions a user might want to ask. The simple RAG pipeline that you set up might answer 20 or 30% of them, but there's just what you can invest in, for instance, agentic strategies or things that basically make the entire capabilities of this pipeline more advanced to answer some of the more advanced stuff.

Okay, so let's talk about, let's get into all those. So first of all, this chunking. What do you think for our For our use case, we're trying to answer questions about documentation, just questions about how to use Weights Biases. We're looking through docs. Where would we start with chunk size and how would we do it? Would we slide a window across our documents and take every token and put in our database, or should we go in steps or blocks? What do you recommend there?

Let's talk about documentation, actually. I think documentation is quite interesting because we've done a few of these case studies as well as tutorials as well. What are some properties of documentation? Documentation, there tends to be one, a lot of different files. If you look at the octree of like an entire, or sorry, like the table of contents of like an entire project's documentation, it tends to be very large. Like for instance, we have over 300 guides in Llama Index. Each page itself tends to be not super long. It's usually not like a book. It tends to be like relatively concise. And also the format of each page tends to be in Markdown. And so given this, one way of thinking about documentation specifically is that when you ask a question, and this reflects human behavior too, you first want to find whether or not you're actually heading the right pages. And then once you actually hit the right page, you want to make sure you're heading the relevant context around the page that actually answers this question. For instance, if you have a question about the arguments passed into this specific class, you probably want to fetch the API reference of the project. If you have a question about the Quickstart tutorial or any tutorial, you want to fetch that page. And so basically, based on the data that you have, and in this case documentation, it's large corpus, small documents, you can kind of basically create intuitions around the types of retrieval strategies to use to actually query this document corpus. One general strategy you can do when, for instance, you have a large multi-document setting and each document tends to be somewhat diverse, is you can basically do what we call hierarchical retrieval. So you can index every document by a summary of that document. So that when you first ask a question, you first retrieve the relevant summaries corresponding to the question. Each summary then maps to the underlying document source. So the final context that's passed to the LLM could be basically the entire document corresponding to that summary. Or you could also do top-k within that document as well. In the case of documentation, what we typically found is it's short enough that sometimes you can actually just stuff the entire thing into the context window. And so really the key issue is finding the right pages to begin with. But that's just an example of given different settings, you can craft your intuitions to develop the right retrieval strategies here.

So let me just make sure I'm laying this out. You summarize each document using some kind of LLM, maybe probably GPT-4 or something like that. It could even be an expensive one because you do it once. Then you compute the embedding of each summarization. Then you match the embedding of the question to the summarizations. And then you look within the document that has the best summary match for the kind of top case snippets that match the query.

Yeah, so that last step is optional, but everything you said up until that point is spot on. So you summarize documents, you retrieve first based on summaries. And then once you actually fetch a document, you could just return all that entire document as a continuous chunk, right? You don't really need to chunk up the document. But if you really want to, if it's a long document, you could go down and retrieve more from it.

I see. I see. And now I feel like sometimes people probably feel overwhelmed by the number of choices of LLMs, the number of choices of vector databases. You said you integrate with 35 Vector databases, there's probably more LLMs out there these days and it gets more confusing every day. Where do you start? Is there a basic setup that's best practice here in November 2023?

Yeah, I think vector databases, I'm playing Switzerland right now. And so I don't know. I think the storage layer is also evolving pretty rapidly as well as every vector database adds new features. At the beginning of the year, there wasn't really good metadata filtering. Now I think every vector database has some structured information storage along with unstructured information. I think really for vector databases, it comes down to some aspect of your existing deployment environment. If you're already using existing systems, stuff like cost, latency, it's really not performance related. I think what I typically spend most of my time thinking about is performance related stuff as well as how that trades off with cost and speed.

And when you say performance, you mean the quality of the matching? Accuracy of the matching.

Accuracy of the matching.

Yeah. Gotcha.

This includes accuracy of the final retrieval results and then the accuracy of the final generated output. Typically alternating the vector DB isn't going to affect the quality of retrieval unless maybe one uses a slightly different nearest neighbors algorithm for approximate nearest neighbor search. But It's all roughly the same. And I feel like for most data volumes, typically people deal with small data anyways, so you don't really need to worry about approximate nearest neighbors. The LLM though does impact the quality. And so we actually have this LLM full list of different LLMs from paid to open source, as well as how well they do on what we call easy to hard tasks in LlamaIndex. An easy task is naive like RAG, so the easiest RAG setup possible in LlamaIndex. A hard task is agents. So basically, how do you get an LLM to actually do tool use, interact with different services, and actually do stuff like chain-of-thought reasoning? What we found is that currently, for paid LLMs, OpenAI GPT-4 is still the best at agent reasoning. That's a pretty common sentiment. I think most people use GPT-4 when they're building agents. For RAG, most LLMs can do RAG these days, like open-source Llama 2, GPT-7B, all the way to the Anthropic Cohere and OpenAI models. One thing I'm very interested in seeing for the LLM side is the open source LLM capabilities evolving into the more advanced stuff. So agentic reasoning, structured data outputs, and stuff like Texas-EQL. We're actually seeing a recent model like Zephyr 7B by Hugging Face, which was fine-tuned on top of Mistral, is actually exhibiting some of those properties. As these open-source models get better, then they can actually start doing some stuff that previously only proprietary GPT-4-like models could do.

Is there ever any reason to use a different model for the embedding versus the actual Q&A?

If the question is kind of like— because right now there's embedding models and there's LLMs. For the embedding model, it's typically Ada-02 by OpenAI, like a Hugging Face embedding model. And then the LLM is typically a generation-based model. But I think maybe kind of a broad question is, is there any reason to just have a mix of different LLMs within your app? I think that's certainly possible. I think for instance, one example architecture that I think is pretty interesting is for a lot of use cases actually, and I've talked to a few users, Some of them want routing. For instance, given a question or a task, route this task to an underlying choice. It's just a choice selection problem. If you use GPT-4 for that, yeah, it can do it. I mean, it'll be a little expensive and slow, but you could also just fine-tune a very fast small model that's very capable of doing this. That's something that feasibly could happen, especially if You want this LLM to do a specific task very well. My take is I think to start with most people just use the same model just because it's easier. And then as they start thinking about costs and trying to optimize that, then once they hit the performance threshold, they try to optimize the rest of the stuff and maybe then they bring in different models.

And so you're saying you want to play Switzerland for the vector databases But could you just say what some popular choices might be?

Yeah, sure. I mean, I can list off 5-ish, 5 to 10. And by the way, if you're a vector database and you're not on this list, please don't be offended. There's Pinecone, there's Weaviate, there's Chroma, there's Milvus, there's Quadra, there's ActiveLoop, there is Redis, MongoDB, and pgvector. And I think there's way more. Yeah, I just listed off a few.

Nice. Thank you.

Thank you.

And do any of them stand out with different functionality? I guess some of them would also function as your database.

I think again, it's less performance and more just system-related stuff. For instance, Pinecone is SaaS, or I think, I don't know if they deploy on cloud now, but it's not self-hosted. If you want to self-host something, you use something that, some people use Chroma, some people use Quadrant, some people use pgvector. Some are other databases that added on vector database capabilities. So Mongo as well as Postgres pgvector. But roughly speaking, I think it's more just a function of your kind of deployment setting and system requirements as opposed to any sort of performance gain.

Got it. Got it. And now what do you do? And this happened to us. I think you alluded to it. If the model starts hallucinating, we were pretty concerned about the model just giving totally wrong answers and documentation to our users. How do you work on that? How do you fix that?

Yeah, it's a good question. I think there's a few off the top of my head. And again, going from the initial things you should try to stuff that's maybe a bit more involved. The initial things you should try is probably just use a better model if you can afford it. I think typically, honestly, some models just hallucinate more than others and unless you fine-tune the model, it's just hard to prompt engineer your way out of it. The second is prompt engineering. I think sometimes if you make a very stern statement that you must return this, if it doesn't actually know, then it'll do it. But sometimes, again, this is a property of model. The third is some sort of, it's the product UX actually that you design matters quite a bit. For instance, I think people have moved on beyond the basic expectation of what you see is what you get in terms of the response and more like you can actually click into the response to look at different sources. There's novel UXs you can define too. Maybe there's some arrow drawing from different parts of the response to a highlight in the source document. That's basically what we did for some of the full-stack apps that we've built, like SCC Insights. We have a full PDF viewer, so you can actually just, you know, for every response, you can go and click the source and actually just look at the highlights. So it's a very direct way to kind of see if they correspond to each other. And then in terms of the model capability side, I think one thing that I think pretty much everyone that's maybe a bit more advanced and far ahead in the LLM space is doing is fine-tuning the LLM to varying degrees of success, like basically trying to fine-tune it to be better at not hallucinating. And by that, I mean, like, somehow you get it to bypass, you know, its original instruction tuning, given enough data and human feedback to basically tell it, you know, for this, like, kind of given that you have this question in this context, you clearly don't know the right answer. So stop making stuff up, like return like you don't know or something. And I have talked to a few users doing that. But I think that typically happens a bit later, you know, once, once you really need to.

It does seem like fine-tuning gets easier and easier every day. Do you feel like it's best practice at this point or at what point do you fine-tune? How much data do you need to really get a meaningful improvement?

I think right now there hasn't been, I think it's been a gradual change but not a step change in the capabilities of fine-tuning. I think fine-tuning typically is still seen as optimization. Tuning something that exists and making it better. It just happens that it tends to be more accessible now to everybody, whether you're fine-tuning OpenAI or fine-tuning Llama 2. A lot of companies popping out that allow you to just fine-tune Llama 2 out of the box. Here's some main issues why fine-tuning is not as popular as just building RAG these days, just taking a pre-trained model and stitching together some stuff. One is the conceptual understanding. It tends to take more to actually understand how to do fine-tuning. Are you going to How do you format the data? If you're not using an API, you have to write like PyTorch code, which most people honestly don't know how to do. And then the second is the time to value. Even the best fine-tuning solutions these days take at least like 10 to 20 minutes over a very small dataset to give you back anything. Whereas RAG will take you like a minute if you know what you're doing. Like it'll just give you results very fast. And so, you know, even for a very experienced developer, they'll probably want to see something quick first, just the easy thing to do first before they actually try fine-tuning. What I'm generally interested in, and this is maybe a little bit tangential to your original question, is a step change in fine-tuning. I think that would probably force us to rethink our own abstractions. But imagine there's some new capability that came out through fine-tuning that made people realize, wow, I always want to be fine-tuning on top of any data because this is a way to memorize information, ingest new information about the world, and basically get it to do better. And so far I just haven't seen that bottle up yet.

Interesting. So I guess you think fine-tuning is probably not worth the effort generally?

No, I think it's like the funnel. I think it's definitely not worth at the top of the funnel. It's probably worth middle to bottom of the funnel. I guess the funnel of AI development as you think about productionization and think about stuff like costs increasing performance in specific settings. For instance, that use case of all else fails and you have a cost constraint and you don't want to use GPT-4, try fine-tuning a model to something like Polysailors.

You mentioned text extraction, which in my more than 15 years of experience is one of those things that's so much more popular. In real-world applications than in research. I just feel like there's always this big mismatch. I'm sure everyone in the real world who's listening to this has some kind of text extraction use case they can think of. Can you walk me through how that might work? A real-world example?

Actually, I'm kind of curious, Lukas, do you have a real-world example from your side on text extraction?

Ooh, at Weights Biases?

Yet.

Good question. We haven't done text extraction yet. I think one, one place we might look is pulling out information from transcripts of sales calls. So for example, like a question that we might ask is like, what fraction of our customers are using AWS versus versus GCP versus Azure. And, you know, maybe we don't actually mark that reliably in our Salesforce. We probably should, even when we, you know, ask customers these questions. So I could imagine a situation where we would want to pull out, okay, like what adjacent technologies, what cloud providers are our customers using? We should actually definitely do this. As I start to say this, this would be incredibly useful for us right now. We have to, you know, we're relying on our customers to support people to market. Somewhere. And I don't think that happens all the time. So how would that, how could we, how could we do that?

Yeah, it's a good question. I think these days, just very concretely, I think the way most people do it is there is increased support for structured data formatting for some of these models now. OpenAI has supported function calling for a few months and then they just released JSON mode on their new models. Both of which at a very high level are designed to help the output be a formatted semi-structured piece of text as opposed to just freeform. And so this is very relevant for text extraction because imagine you're analyzing some piece of content and you want the output to specifically be, hey, I want to see if, let's say it's whether it conforms to certain guidelines, you want to output some Pydantic object like a schema. With a few Boolean fields and some numbers saying, "Here is my rating, 1 to 5. Here is whether it adheres to something, true or false." And so it's possible to do that, especially with these types of models. Certain models are better at it than others. Actually, structured data extraction is something that a lot of open source models tend to struggle with. And so if you're going to use some sort of just general text extraction model, the better models still tend to be better. The main thing— Interesting. Yeah. The main system's challenges with structured data extraction is cost and latency. I think with longer context windows, maybe kind of depends on your use case, but like you can imagine latency might go down a bit. But if you're running GPT-4 on say like a gigabyte of data, just raw text data, that is probably going to cost you a lot of money. And so like you kind of have to be a little bit careful and selective about like how do you actually do extraction from a giant corpus of documents. One other pattern I've seen, which is very interesting, is you could do something like, especially if it's a giant corpus of documents, first do retrieval, then do structured data outputs. It's actually very similar to just a RAG pipeline, but you just add in some structured data extraction at the end of it. Given some giant corpus of data that's much bigger than what the Context Window can handle, retrieve the relevant chunks and then just make sure the output is some sort of structured field. And so therefore, whenever you ask a question, you can basically get back some sort of structured result from it over a large corpus of data.

Cool. Can you say more about how you think about agents? I feel like I've seen so many evocative examples of agents, but I know they're notoriously hard to get to work and we actually haven't yet I think really gotten any agents really working for our use cases. I mean, how do you think about that? How do you support it? And what's the state of the art of agent-based systems right now?

Yeah, I would actually be curious if you were able to get it working because that'd be a pretty interesting case study. I would love to chat about that because I think your sentiment echoes some of the other enterprises that we've talked to. I think a lot of people are building RAG. I think RAG tends to be very popular because it's relatively simple as a concept, but also easier to implement. Agents are very powerful. I do think agents are probably the future, just its like current state is going to be a little hard to fully realize that vision. I think maybe taking a step back, people have different definitions of what an agent is. Some people think of an agent specifically as like a loop, like execution loop with tool use. If you look at the Assistants API that OpenAI released a few weeks ago, it's basically this, like you give it some tasks, It'll go ahead and solve it, plus maybe do some function calls on tools that you provide. Similar to a React loop, like the React paper that came out last year, you do some sort of chain-of-thought processing, tool use, and interlink that in a prompt-based fashion. So I think that is one definition of it. I think that tends to be pretty popular. That's basically for all intents and purposes the way we implement agents in LlamaIndex. And so it's some sort of execution loop with tool use. But, you know, honestly, like the idea of an agent is really just like reasoning, like anything that's more agent-like is just things that do more reasoning. And so even things like basic routing, where you get a question, and then you have an LLM pick and choose what is the right, like answer to route this to, that's basically agentic, because you're using some sort of automated decision making. Even like structured data extraction is kind of like agentic. I think the stuff that makes it more agentic is if you start basically chaining some of these reasoning calls together so that it does more stuff at once instead of just a single step. If it actually is able to take actions that affect services, so it can do writes and not just reads. And then the third is some vague notion of memory or just being able to retain information over time. But that's more of a subset. I think that probably matters for some agentic use cases but less others. The people that think about that are more thinking about personalization where you can create customized chatbots that are your personalized AI companion, like the Character.ai and Inflection.ai of the world. But there's a few definitions floating around. I think right now, basically with the current model capabilities, even if you had an infinite budget, I think it's hard to get reliable agents fully working in a general unconstrained setting. It is very possible to get it working in a constrained setting. So we're seeing, you know, if you have like 2 or 3 tools at most and you give that to an agent, that agent can generally figure out what to do with these tools. Actually, if you take a look at a lot of our apps that we put out for RAG, that's basically an agent using on top of some tools that do like vector search and retrieval. And so we're actually extending a little bit beyond just like the top-k RAG concept by adding in agents as well. It's a very powerful concept, but the moment you extend it too much, right, and you saw this with the initial release of like AutoGPT as well as all the other agent projects, you let it do a lot of infinite things and yeah, it can do some of it, but it's not going to be able to do all of it very well.

Interesting. So what else, what other tool would you give a RAG agent besides Lookup?

Yeah, it depends how creative you want to be with it actually. I think for instance, a basic RAG use case would be give it access as a tool to a vector database and also give it access as a tool to a SQL database. Additional capabilities you could augment it is web. So you could, if it doesn't actually find the information, do stuff to look up stuff from the web. And then add that information to the database. Some other examples here include being able to use this data to then automate your workflows. And so this definitely moves a little bit beyond the RAG use case, but imagine drafting up an email, sending it, and also sending a calendar invite, those types of things.

My co-founder made a code-generating agent that seems to work pretty well. It doesn't always work, but it's really actually very impressive.

Yeah, that's really cool. So code, actually, I haven't really talked about code at all. Code is very interesting because it's basically when you write code, it's the language that writes its programs, it writes itself. And so it's the tool that an agent can use to construct other things that can execute things. And so that is very interesting. You start getting into Basically, I think an interesting concept for agents is building things that can build themselves and then do other things. And you can see this in various flavors. You can see this in terms of an LLM being able to generate code that can then execute. You also see this with GPTs that OpenAI launched. It's the whole idea of you use natural language to program something that can then do a certain task. And so that honestly, if you can do that, maybe that is one criteria of AGI because just generate things that self-replicate. Totally.

Totally. What other surprising use cases are you seeing out there?

In terms of the surprising use cases, it's a good question.

Or maybe actually, what are the use cases that besides this kind of RAG workflow What else is working? What are companies out there doing with LlamaIndex where they're finding value?

I think the main use cases is around search and retrieval, summarization, and structured data extraction. Those have continued to probably be the most popular use cases that we've seen. Anything that's very workflow heavy, we haven't seen as much. And one thing that we have seen, but that we personally have not really done as much is using LLMs to basically index code. And so being able to look up stuff that's very technical or code-like in nature. So for instance, helping you do refactors, helping you do Copilot-like stuff. We just haven't investigated as much into that.

Got it.

Another thing that we have seen kind of, but haven't really invested too much in is just web automation stuff. For instance, you go on a page and you can automate stuff like checkout experiences, those types of things. It's just we haven't really invested that much. Honestly, the area, this vertical, if you will, of search and retrieval over data is still very complex. And I think no one's actually solved it that much. And so that's why we're still investing a lot of effort into this.

Can you tell me the story of Llama Index? What were you thinking when you started it? How did it become a company?

Yeah, so I think I've told variants of this story a few times and maybe I'll just drop in a few details just to see if there's anything new that I can share. I mean, I started this project, the first commit was on November 1st, 2022. I posted this on Twitter November 8th, 2022. I started hacking around with LLMs basically late October of 2022. This was around last year. I had started playing around with LLMs, GPT-3. Honestly, I got into it pretty late. I wish I got into it earlier. I've been following generative models for most of my working career because I did research on generative models. I did AI research for 2.5 years at Uber, have worked in ML for most of my career. But basically, I was hacking around with it and it was one of the first realizations. What I said at the beginning was how do you actually make this thing useful over your own data. And that got me down this rabbit hole. I created this kind of exploratory design project called the GPT Tree Index, where I organized data in some sort of tree. Launched on Twitter, woke up in the morning, got 300 likes, got really excited because I was like, wow, this is amazing. And then that motivated me to basically lean into this developer kind of this open source flywheel, basically, if you will. When you're building out in the open, there's a lot of feedback constantly coming in. You're seeing what other people are building. And so it just got me really excited about the space. And it was at the time when a lot of people were discovering applications of LLMs and ChatGPT. So it was a pretty opportune time.

And when did you turn this into a company?

It was officially incorporated April. And then Did the fundraising end of April, announced it in June, and then basically had talked to my now co-founder, but he was a good friend of mine and starting in January.

Cool. Did you want to talk about your Data Agents release? What is that and how would somebody use it?

Yeah, so going back onto the thread about agents, Data Agents is basically our implementation of agents and a lot of the way we design these abstractions as well as how it hooks into framework is centered actually around like RAG use cases with, with additional tool use. So it consists of a few different parts. What is a data agent? We call it a data agent. It's basically just for all intents and purposes an agent that we like to think is a bit more focused on data, like data search, retrieval, usage, those things. And consists of a few different components. One is the execution loop. So you have your choice of React, like the general chain-of-thought reasoning loop, OpenAI function calling, like a while loop over OpenAI. We also wrap their assistance API so you can just directly call it if you want. The second component is tool use. So you can plug in a bunch of tools. We make it very easy to basically define any tool that you define, or any kind of like RAG system in LlamaIndex as a tool to plug into an agent. You can actually plug in agents as tools to other agents too. And so it's very easy to create hierarchical agents that can do, again, focus on search and retrieval use cases, but that's kind of what we're focused on. And third aspect is some aspect of conversational memory, more for the chat use case. And so those are some of the core components. A lot of the use cases we've highlighted is how adding Agentic, this agent execution loop on top of your RAG pipeline, can enable more sophisticated behaviors. You can ask multipart questions like, oh, you know, can you tell me the answer to this? And given this answer, tell me the answer to B. Or can you, you know, do summarization of this document for me and then compare it against like this other document, like a resume, for instance. Enables more capabilities on top of your existing RAG pipeline. The other piece is we have like 40 to 50+ tools on Llama Hub which is our open source hub for community-driven contributions, which consists of a bunch of tools for users to just use to plug into their agent. So if you want to do beyond more stuff than just RAG, you want to actually send an email, send a calendar invite, do Google search, try and generate images even, you can basically plug those in as tools too. So if you're interested in contributing a tool to Llama Hub, basically to listeners, please feel free to. We're always looking for more contributions.

Is there any missing functionality that you'd especially like to see?

There's always new tools that we could add. Imagine if you can plug in, especially nowadays with all the multimodal stuff, you plug in text-to-speech, plug in text-to-image as a tool like DALL-E 3, and then you hook that into your other tools that the agent can use and then you just have some automated system that can generate images along with reasoning over your information.

Totally. Do you worry at all that folks like OpenAI will replicate your functionality? Do you try to stay away from stuff that they might do?

I mean, to some extent there is maybe a little bit of overlap with retrieval, the retrieval API they launched. I think, yeah, people have asked me this. I think my overall response is, I think, yeah, LLM providers are going to move into this space. OpenAI has a retrieval API now, It's part of their assistance API. I think Anthropic has some RAG thing going on. So does Cohere. And then I think AI21 as well. So I think my general take is I think that's fine. It is what it is. There are certain advantages to designing vendor-agnostic retrieval because this enables users just to have way more options in terms of you can pick and choose whatever LLM you want. And then we really just focus on making the retrieval part work well, but it'll plug in well into your model. And I think the bet is basically as long as there's a healthy competitive space of LLMs, then there will still be a need for third-party orchestration services that easily integrate with existing models that you use.

When do you think about LLMs, evaluating LLMs, do you have any best practices there? I think it's often surprising to people who first hear about this that that's a good strategy at all. How effective do you find it and are there any best practices you could share?

Yeah, I think we are actually working on it right now. So I'll probably have better insights in a month or two because we're doing some research on what's Yeah, how exactly do you do this? And there's a lot of open source projects that do it better than we do. And we don't want to really own this space. We're just more than happy to just have general abstractions that plug into other libraries. Some overall concepts for how do you do LLM eval. If you have the ground truth response, you can basically prompt GPT-4 to output a score between 1 to 5 on how much your predicted response, the generated response, matches the ground truth. Response. So given this, if you make the number range too wide, GPT-4 gets confused. But if it's small enough, then like 1 to 5, it typically can, can reason over stuff. You can also do evaluation in the label-free setting. So for instance, like if you actually don't have ground truth, that's actually fine. There's some general things you can do to make sure that it's not hallucinating. So like given the response and the context, can you make sure that some of the context supports the response, right? So it's not just making up facts. You can determine whether the response is adhering to guidelines, relevant to the question, and those types of things. So far, I think most of our evals use some variant of GPT-4. 3.5 tends to be a bit noisier, so the variance of results tends to be higher. But we anticipate that as models get better, they'll get better at being a judge. I think that the main issues I'm personally running into for evals is cost and latency. I think GPT-4, the way we implement it, is kind of inefficient and it's running up big bills on my OpenAI account. If we can somehow even fine-tune embeddings or smaller models to do eval as good as bigger models, that'd be amazing. I think we just haven't really done that yet.

Quite a proliferation of open source projects in your space, but it does seem like LlamaIndex has really emerged as a leader. Why do you think that happened? Do you think it was timing or do you think it was some architectural choices you made or the way you manage the community or listen to users? What do you credit the growth to?

Yeah, dumbly, a lot of it was timing. I think if we started at November of this year, it absolutely would have taken off. I think it was just we I just happened to think about these things at the right place at the right time. Honestly, I think that's also why I said, I feel like I started a little bit late. People were already getting excited about LLMs in September, August of 2022. And I think if I started doing stuff then, I would have been able to do stuff sooner. I think the other piece is, yeah, I mean, some aspects of execution and then, oh, when you're designing a framework, you got to make sure— this is something I learned over time. Honestly, I wish I did this from the beginning. Building a framework is a little bit different than just being a backend engineer at a company and trying to build a service for internal use. You got to make sure basically everything you build is going to be used by other people and it's got to have good documentation and it's got to have good abstractions. So you're not really building a system, you're building a toolkit. So you need modularity, composability, just core engineering principles. And so honestly, I think we added a lot of stuff to LlamaIndex in the early days, but we ended up having to refactor, rewrite a bunch of stuff in April just because it just wasn't at the point where we could actually help easily add new abstractions and make it a useful framework. And so honestly, the refactor, just rewriting the entire thing that my co-founder Simon led really helped encourage the future growth of this framework. 'Cause if we didn't, we'd probably be stuck a little bit in a place where like, yeah, it was useful for a specific purpose, but it would be hard to actually add new abstractions and things to, to advance this. So part of it is like the engineering.

It must have been a scary moment to do a refactor when you have kind of a popular project. Like how did you weigh that?

Yeah, I think at a certain point we just realized that it kind of needed to be done. The layers of tech debt was, was becoming a little bit hard to actually adapt to new things we wanted to do. Agents, different retrieval algorithms, parsing, ingestion, basically decoupling it, making it more modular, making it more useful for users. That was a conscious decision we made.

Got it. And what fraction of your code is written by you versus community contributions?

Well, me, my co-founder Simon, our founding team, and then community contributions. These days I haven't written as much. I don't really write that much code. I mostly just maybe put out some tutorials and guides. These days it's really most of the team working on both open source as well as kind of like this enterprise stuff. I think in the early days it was mostly me.

But now is it, it's mostly your team that contributes or do you get like useful third party pull requests?

Oh, I see. Yeah, yeah. We've gotten a pretty decent rate of community contributions. I think we're almost at like 500 contributions on open source and then like almost 200 on the hub. But it always could be more. I think, oh, that's the other thing, which is why if you want to encourage open source contributions, and this is basically advice for anyone running an open source project, you need to have a good set of tools and instructions. Because if you don't set that up, then people, they won't, even if people are excited, they're not going to know what to contribute to. And so if you have the right base classes set up, basically a lot of people add the integrations.

And how do you see interacting with big text corporuses evolving over the coming years? Do you think as context length gets bigger and bigger, the way we do it will change? I mean, you could imagine even putting the entire corpus into a context. Is that possible?

Yeah, so it's an interesting question. I think I'm not discounting the possibility. I mean, I can't make a negative statement against that. I can only say what the current state is today. As of today, Anthropic 1.0, was it 2.1, just came out and Anthropic has a 200K context window. So the GPT-4 Turbo has 128K, Claude 2 has 100K. And I was just reading some analysis on the attention, the ability of the LLM to attend to different areas of this context when you stuff the entire context window. There's still a lot of red in the middle. Meaning that LLM can't actually find things well when it's in the middle of the context. And so I think there's still real risks to not only cost and latency, but actually performance when you try to stuff like an entire LLM full of information right now, right? And that will get better over time. But I think it's just like, for a lot of users, it probably still makes sense to do some sort of retrieval from like a large knowledge corpus. Instead of just trying to always fill in as much information into the LLM as possible. I think we're constantly keeping an eye on this though. Context windows will get bigger, the cost will come down, we'll probably need to adapt some of the paradigms. I do think retrieval will probably be necessary for basically anybody that has datasets greater than a gigabyte just because even if you think about network transfer costs, If you need to send a gigabyte of data to the cloud for every inference call, it's going to take a long time for you to actually do that. Having more efficient ways to structure the interactions between LLMs and data would make sense.

We always end with two questions and I'd love to get your take on these. What is an underrated aspect of machine learning or LLMs that you think people should pay more attention to?

Yeah, I mean, I don't know if this is the answer you're looking for, but I rode in a Waymo for the first time 2 days ago and it was awesome. I used to work in self-driving, I think, and I took a bit of a 2 or 3-year hiatus. Rode in a Waymo. It's fantastic. I think if everybody rode in a Waymo, they'd get excited about robotics again.

And what do you think is the biggest challenge of getting LLMs to work in the real world as of today?

Yeah, the biggest challenge is probably a good chunk of the conversation we had at the beginning of this, which is stuff around evaluation and making sure you have the right strategies for actually tuning your entire system and not just the prompt. I think basically there's a big education and knowledge challenge in terms of how do you apply LLMs for different use cases. Hoping to help figure that out for a lot of users. But basically, like, that includes instilling the best practices as well as, like, understanding the best techniques to really maximize the value that you can get. This includes RAG, but also, like, agents, like, pretty much most use cases that people are building today.

Awesome. Well, thank you so much. We'll put some links to the stuff you mentioned in the, in the show notes. But a lot of fun to talk and congrats on such a successful project.

Awesome. Thank you, Lukas.

Cool. Take care. Thanks so much for listening to this episode of Gradient Descent. Please stay tuned for future episodes.