Data augmentation with LlamaIndex

Foreign.

Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI related technologies are changing the world, this is the show for you. Thank you to our partners at Fastly for shipping all of our pods super fast to wherever you listen. Check them out@fastly.com and to our friends at Fly. Deploy your app servers and database close to your users. No ops required. Learn more@fly IO.

Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist and founder of a company called Prediction Guard. And I'm joined as always by my co host Chris Benson, who is a tech strategist at Lockheed Martin. How you doing, Chris?

Doing very well. Enjoying this fine springtime weather of LLMs.

Yes, the spring LLM bloom.

That's right. That's right.

Well, I don't even think we can use the word bloom because that's loaded now.

Yeah, I was going to say that has a whole different meaning.

There's no word that's not loaded with some sort of AI meaning at this point.

Yeah, we should just go straight to our guest.

Yeah.

Including llamas, which we're excited today to have with us today Jerry Liu, who is co founder and creator of Llama Index. Welcome, Jerry.

Yeah, thanks Daniel and Chris for having me. Super excited to be here.

Yeah, I'm really excited because we've had a few conversations in the past and I've used Llama Index in some of my own work and also kind of tried some integration stuff with various data sources. So I'm really excited to hear a little bit more of the story and kind of the vision behind the project. If I'm just reading from the docs, Lambda Index is about connecting LLMs or large language models with external data. So maybe a first question, kind of a general question, not specific to Llama Index necessarily, is like, why would one want to connect large language models with external data?

Yeah, it's a good question. And so for those of you who are already in the space of LL application development, this might sound obvious to you, but for those of you who might be still somewhat unfamiliar, large language models have a lot of different sorts of capabilities that are really good at answering questions, doing tasks, being able to summarize stuff, basically anything you throw at it, like try to write a short story, write a poem, it can do. And the default mode of interacting with a language model like ChatGPT is that you would write stuff to it in a chat like interface. This query would hit the model and you'd get Back some output. I think one of the next questions that people will get into, especially as they're trying to explore building applications on top of large language models, is how can this language model understand my own private data? Whether you're kind of like a single person or you're an entire organization, and these days there's a lot of different ways for actually trying to incorporate new knowledge into a language model. The models themselves are trained on just like a giant corpus of data. And so if you're like an ML researcher, your default mode is just how can I train this model on more data so that I can try to memorize this knowledge? And the algorithm there is basically through some sort of like gradient descent through the weights or RLHF or any sort of like fancy ML algorithm that actually includes the knowledge and the weights of the model itself. I think one interesting thing about large language models these days is that instead of like training the model, you can actually take the model as is and just like figure out how to have it reason over new information. And so for instance, like use that input prompt as like the cache space to feed in new information, tell it to reason over that data and, and to answer questions over that data. And I think that's very interesting because you can take the model itself, which has been trained on a variety of data, but doesn't necessarily have inherent knowledge about you as a person or your organization data, but then you can tell it, hey, here's some new data that I have now. Given this data, how can I answer the following questions? And this is part of the stack that a lot of people are discovering these days where you can actually just use the language model itself as a pre trained service and then wrap that in this overall software system to incorporate your data with the language model. Cool.

Yeah. And your project is called Llama Index. Now when I, before the past like few months or six months or whatever, when I was thinking about like indices or an index, one of the things that first came to my mind was like, oh, I have a database maybe, and there's an index that I use to query over that database. And some of that is a little bit like fuzzy magic to me in terms of how that actually works at the lower level in a database. But like, what is this idea of an index or indexing in the context of Llama index or in the context of data augmentation for large language models?

It's kind of funny, I think when we first started the name, it was a bit more of like a casual naming convention. You Know, it used to be called GPT Index, and I kind of made up the name because it sounded roughly relevant to what I was building at the time. But I think over time, especially as it's morphed into more of a project that people are actually using, this concept of an index has become a bit more concrete. And so I can articulate that a bit better. The idea of Llama Index is, you know, just to step back and talk about. The overall purpose of the project is to make it really easy and powerful and fast and cheap to connect your language models with your own private data. And we have a few constructs to do so within Llama Index. And, and so part of the way you can think about Llama Index is how can we build some sort of stateful service around your private data, around something that at the moment is somewhat stateless? Like the language model call is a stateless service, right, because you feed in some input and you get back some output. So how can we wrap that in a stateful service around your own data sources so that if you want to ask a question or tell the LLM to do something, it can reference that state that you have stored? And so if you think about any sort of data system, there is the raw data that's stored somewhere in some storage system. There might be indexes or views similar to a database analogy, where you can look at the data in different ways and I can talk a little bit about how that works. And then there's usually some sort of query interface that you can actually query and retrieve the data. So if you look at a SQL database, you have the raw data stored and some sort of tables. You can define different indexes over different columns. And then the query interface is like a SQL interface. You run SQL and then it'll be able to execute the query against your database. And there's a lot of, like, kind of roughly similar concepts that apply to thinking about the Llama index itself as this toolset. Because if we're going to build this, like, stateful service right on top that can integrate with large language models. By the way, to clarify, we're not like, really solving the storage part, right, where we integrate with like, a ton of different vector storage providers, we integrate with, like, other databases too. But if you even think about us as like some sort of data interface or orchestration, you know, there's the raw data which needs to be stored somewhere. And so if you have like a bunch of text documents, you need to store that in like a vector database or MongoDB or S3, all those types of things. And then you can define these different indexes on top of this data. And the way we think about indexes is how do we structure your data in the right way so that you can retrieve it later for use without lens. And so then I can talk a little bit how this works. But the set of indexes that you can define is actually pretty interesting. Basically the set of data structures that offers a view of your data in different ways. And then you wrap that in this overall query interface that can use these indexes on top of your data to do retrieval and LLM synthesis and give you back a final answer. And so I would look at this in terms of the components of the overall system. There's just like, if you're building this, like stateful service, there is these three components. How do you address and store the raw data, index it, and then query it.

So I want to actually pull you back for just a moment as we're kind of learning this. And if you're an app developer and you're interested in creating a stateful service and you've started kind of going down the path about like, well, there's kind of the old school way of going and doing a SQL query and all that, and now we're using LLM models and adding our data to it. So I know that we've kind of gone beyond that just a little bit. But if you can back up and talk a little bit about what are you getting? If you're the app developer and you're listening to this and you're trying to understand, like, why would I go down that path? You know, I sense that there's value there, but we haven't talked about it versus a robust set of SQL queries on your own data. Why would you bring in that large language model in the beginning? What is it bringing to bear that's worth all of that effort? Could you talk a little bit about that baseline value add to it?

Yeah, that's a really good question. And I think I might have jumped the gun a little bit. So I appreciate you bringing me back.

No worries. It's because you're excited, as are we, but I also want to make sure that people listening have a chance to truly understand it at the same way that you are.

Definitely. I think one thing about language models that's very powerful is their ability to just comprehend unstructured text and also natural language. And so this matters in both ways in terms of how you can store the data as well as query the data because now, let's say you're the end user. You can just type in a NAFL language English question, ideally, into this interface and get back a response. And so the setup is way easier than having to learn SQL over some source of data, or having to even code up this very complex pipeline to try to parse the data in different ways, because you could treat the language model itself as a black box, feed us something, get something up, right? And so I think that by itself is a very, very powerful tool. And I think these days, people are trying to figure out what you can do with that tool. So another kind of, like, illustrative example of, like, the power of language models using this as like intelligent natural language interface is you actually don't have to do a ton of data parsing when you actually feed in a data. So for instance, let's say you have a PDF document, or any sort of Microsoft Word document, or even an HTML web page. Just copy and paste that entire thing, just extract the text from it, dump it into the input prompt, and then just tell the LLM, hey, here's just this giant blob of text I copied over. Now, given this text, can you please answer this following question? And the crazy thing is the language model can actually do that, assuming it fits within the prompt space. And that's also very powerful because this kind of affects the way you do ETL and data pipelining in the traditional sense. If you had a bunch of this unstructured text, you'd have to spend either manual effort or write a complicated program to pull out the relevant bits from this text, parse it into some table, store it, and then run SQL or some other query over this text. Whereas here, with the power of language models, you can store this text in a bit more of a messy, unstructured format as raw natural language, and then still figure out a way to pull out this unstructured text, just dump it into the input prompt and ask a question over.

Is it conceivable with what you were saying? If I'm thinking as an app developer about diving into this, that I'm hearing you say you're going to do this, which is an additional thing to learn and be able to go, you know, it's an additional skill set that you're adding on, but I also hear you talking about other things that I used to have to do that maybe I don't have to do anymore. And, and to some degree, is it realistic to say, from an effort standpoint, it becomes a wash once you have the skills A little bit, or maybe even you're gaining more power and doing less work along the way to do it so that it's kind of like, of course you would do it going forward. Is that a fair way of thinking about it?

Yeah. So it's an interesting way of thinking about it because I think the high level question is just like, you know, what parts have become easier and what parts have gotten harder once you have this language model technology. Because on one hand things have gotten a bit easier and powerful to build these expressive question answering systems with less effort. You take in this giant blob of unstructured text, figure out how to store it, you feed it into the language model, and then all of a sudden you can ask these questions over these files that you couldn't really do before with more traditional AI technologies or just manual programming. That said, I think this new paradigm kind of involves its own set of challenges that I'm happy to talk about. I think there's a lot of stacks emerging about how to make the best use of language models on top of your data. And there's some very basic stuff that's happening these days, but there's also kind of like more advanced stuff that we're working on. And I do think it's very interesting to think about what are the technological challenges that are preventing us from unlocking the full capabilities of language models. Because again, with a very basic stack, and again, like you can see this if you just like play around ChatGPT, you can already get a ton of value from your data by just like doing some very basic processing on top of it. And you can start asking questions that you couldn't really ask before, but you know, with some more advanced capabilities and some, once you're solving some more interesting technical problems, what are kind of like the additional queries that can ask on top of your data that you also.

Couldn'T do before, before we jump into. So I want to kind of dive into the weeds about like the two things that you talked about, like how do I index my data, how do I query my data? All the goodness around that in index before we do that maybe just as also like to set the stage for some people that are coming into this and maybe parsing some of the jargon that's thrown around. So one of the other things that people are really kind of diving into is thinking about like how do I engineer my prompts, how do I chain prompts together and all of that sort of thing. And could you highlight because my, at least the way I would phrase it like those two things are complementary with the things that you're doing with Llama Index. But could you kind of help people understand, like, how do those pieces fit together in terms of like architecting one of these systems?

I guess in the end, LLM application development, to put it in a very oversimplified view, is just some fancy form of prompt engineering and prompt training, right? It's actually not super different with how we're thinking about building this interface with data. And so just as a very basic example, you know, if you're kind of coming into this space afresh, like a very basic prompt that you could put into a language model is something like the following. Here is my question. And then you put the question here, and then you put in here's some context. And then in this context variable, you just dump all the context that could be relevant to the question. You copy and paste a blog post, you copy and paste API documentation, just copy and paste it into the input prompt space. And then now at the bottom, say, given this context, give me the answer to this question and you send it to a language model and then you get back an answer. So that's like the most basic question answer prompt that you could use to kind of perform some sort of question answering over it really is just prompting, right? Because you're putting stuff into the prompts. You have this overall prompt template and you have variables that you want to fill in. I think one interesting kind of like challenge that arises is how can you feed in context that exceeds the prompt window? Because for GPT3, it's 4,000 tokens. For Anthropic, I guess it's like 100,000 tokens. But if you look at like an Uber Sec 10K filing, it's like 160,000 tokens, right? So if you want to ask a question like what's the summary of this entire document? Or like, what are the risk factors in this very specific section of the document? How do you feed that entire thing in so that you can basically answer the following question? And I think that's where things get a little bit more interesting because you can basically do one or more of the following things. One is you could have some external model, like something separate from the language model prompt that's actually doing retrieval over your data to figure out what exactly is the best context to actually fit within this prompt space. Two is you can do some sort of synthesis strategies to synthesize an answer over long context, even if that context doesn't actually fit into the prompt. For instance, you could chain repeated LLM calls over sequential chunks of data and then combine the answers together to actually give you back a final answer. That's one example. In the end, all this architecture is just kind of designed around being able to feed in some input to the LLM and get back some output. And the core of that really is a prompting. And so part of this is just like developing an overall system around the prompting.

Well, Jerry, you had mentioned kind of these three levels of integrating external data into your LLM applications or sort of data ingestion. And there's like indexing and query. I'm assuming data ingestion has to do with like, oh, I'm going to connect to the Google Docs API and like pull the data over and then indexing and query build on top of that. But before we dive into those second two phases, which is where I think a lot of the cool stuff that you're doing is found, what should we know about sort of data in the data ingestion layer? In terms of relevance to how Llama Index builds on that and other things.

The data ingestion side is just like the entry point to building a language model application on top of your own data. I think LLMs are cool. I want to use it on top of some existing services. What are those services and how can I load in data from those services? One component of Llama Index is this kind of like community driven hub of data loaders called llamahub, where we just offer a variety of different data connectors to a lot of different services. I think we have over like 90 something different data connectors now. And these include like connections file formats. So for instance, like PDF files, HTML files, like PowerPoints, images even. They can include connectors to APIs like Notion, Slack, Discord, Salesforce. Actually, sorry, we actually don't have Salesforce yet. That's something that we want, but if we haven't. Yeah, it'd be very useful if you're interested in contributing a Salesforce loader, please. I would love that. And then the next part is just like being able to connect to kind of like different sorts of multimodal formats, like audio images, which I think I've already mentioned. So the idea here is you have all this data, it's stored in some format, it's unstructured, it could be text, or it could even be like images or some other format. How do you just like load in this data in a pretty simple manner and just wrap it with some overall document abstraction? So there's Not a ton of tech going on here. And the reason, it's more just like a convenience utility for developers to just like easily load in a bunch of data. And again, going back to the earlier point, the reason there's not too much tech is LLMs are very good at reasoning over unstructured information. So you actually don't need to do like a ton of parsing on top of this data that you load to basically get some decent results from the language model. And so once you actually load in this data in a lightweight container, you can then use it for some of the downstream tasks like indexing and querying.

Awesome. Yeah. And I see, like, this is, I think, where things get super interesting. Like I mentioned. So in llamandex, I'm in the docs right now. Like you mentioned list and table entry and vector store and structured store and knowledge graph and empty indices. Could you describe, like, generally how to think about an index within Llama index, and then like, why are there multiple of these and what categories generally do they fit in?

One way of thinking about this is just taking a step back at a high level. What exactly does the data pipeline look like if you're building a LLM application? So we started with data ingestion, where you load in a document from some data source like a PDF document or API, and now you have this unstructured document. The next step typically is you want to chunk up the text into text chunks. So let's say, naively, let's say you have just a giant blob of text from a PDF. You can split it every 4,000 words or so or every 500 words into some set of text chunks. This just allows you to store this text in units that are easier to feed into the language model. And, and a lot of this is a function of the fact that the language model itself has limited prompt space, right? So you want to be able to chunk up a longer document into a set of smaller chunks. Now you have these chunks that are stored somewhere. They could be stored, for instance within a vector database, for instance, like a pinecone weaviate chroma. They could also be stored, for instance, like a document store like MongoDB. Or you could store it in like file system on your level disk. Now they're stored. The next part is how do you actually want to define some sort of structure over this data? A basic way of defining some sort of structure over this data, and this is where we got into indices, is just like adding and embedding to each chunk. And so if you have a set of texts, how do you define an embedding for each set of texts? And that in itself could be treated as like an index, right? An index is just like a lightweight view over your data. A vector index is just adding and embedding to each piece of text. There's other sorts of indexes that you could define to define this view over your data. There's a keyword table that we have where you just have a mapping from keywords to the underlying text. You could have a flat list where you just basically store a subset of Node IDs as its own index. Before I get into the technicals of the indexes and what they actually do, one thing to maybe think about is just what are the end questions that you want to ask? And what are some of the use cases that you'd want to solve?

Before you dive into that, I was going to ask you really quick. Could you define what an embedding is for those people who are learning large language models at this point, just so they'll understand what it is when you say you're defining that as the index?

Embeddings is a part of kind of a very common stack immersion these days around this LLM data system that's emerging. And so an embedding is just a vector of numbers, usually like floating point numbers. You could have like a hundred of them, a thousand of them. It depends on the specific embedding model. And the way an embedding works is just think about this list of numbers as a condensed representation of the piece of content that you have. You know, if you can somehow, in a very abstract manner, take in some piece of context. Let's say this paragraph is about the biography of like, a famous singer, right? And then you get an embedding from that. It's a string of numbers. The embedding has certain properties such that this string of numbers is closer to other numbers that are semantically about, like, similar content and farther away from other strings and numbers representing texts that are farther away in terms of semantic content. So, for instance, if you look at the biography of a Singer, it's going to be pretty close to a biography of another Singer versus if it's about, I don't know, like the American Revolution or something like that, embedding will probably be a little bit farther away. And so it's a way of condensing a piece of text into some vector of numbers that has some mathematical properties where you can measure similarity between, you know, different pieces of content.

Maybe this is another Point of distinction. And I, I get all these questions very often, so I think it's useful to discuss them on the show. Like last week at odsc, I got a lot of these sorts of questions. So, like, we're talking about bringing in data, creating an index to access that data. That index might involve like a vector store or embeddings, but Llama Index is sort of not a, a vector store. Like, it's cool to be a vector database company right now, but Llama Index is something different. And like, again, these are two things that are complementary, I think. Could you draw out that distinction a little bit just to help people kind of formulate those compartments in their mind?

I think these days there's a lot of vector storage providers and they handle a lot of the underlying storage components. And so if you look at like a pinecone or weaviate, they're actually dealing with the storage of these unstructured documents. One thing that we want to do is leverage these existing kind of like, storage systems and expose query interfaces that I guess a broader range of query interfaces beyond the ones that are just directly offered by a vector store. And so, for instance, a vector store will offer a query interface where you can typically query the set of documents with an embedding, plus like a set of metadata filters, plus maybe some additional parameters. But we're really trying to build this, like, broader set of abstractions and tools through our indices, our query interfaces, plus other abstractions that we have under the hood to basically perform more interesting and advanced operations and manage the interaction between your language model and your data, and almost be a data orchestrator on top of your existing storage solutions. And so we do see ourselves as separate because we're not trying to build the underlying storage solutions. We're more trying to provide a lot of this advanced query interface capability to the end user using the power of language models on top of your data.

I think we got a little bit off track a little bit, but I think it was good. So kind of circling back to the, the indices that are available in Llama Index and you've talked about like this pipeline of processing and potentially like one index being like a vector store and maybe listeners are a little bit more familiar with kind of like vector search or semantic search or that sort of thing with everything that's going on. But you have much more than that, like these other patterns and these other indices that enable other patterns. Could you describe like some of those alternatives or additions to like Vector Store Index and when and how they might come into play.

Yeah, that's a good question. And maybe just to kind of like frame this with a bit of context, I think it's useful to think about certain use cases for each index. So the thing about a vector index or being able to use a vector store is that they're typically well suited for applications where you want to ask kind of like fact based questions. And so if you want to ask a question about like specific facts in your knowledge corpus, using a vector store tends to be pretty effective. For instance, let's say your question is, let's say your knowledge corpus is about like American history or something, right? And your question is, hey, like, what happened in the year of 1780? That type of question tends to lend well to using a vector sort. Because the way the overall system works is you would take this query, you would generate an embedding for the query, you would first do retrieval from this vector sort in order to fetch back the most relevant chunks to the query, and then you would put this into the input prompt of the language model. And so the set of retrieve items that you would get would be those that are most semantically similar to your query through embedding distance, right? So again, like going back to embed, the closer different embeddings are between your query and your context, the more relevant that context is and the farther apart it is, then the less relevant. And so you get back the most relevant context to your query, feed it to a language model, get back an answer. There are other settings where standard top K embedding base lookup, and I can dive into this in as much technical depth that you guys would want to, but there's a settings where like standard kind of like top K embedding base retrieval doesn't work well, right? And one example where it doesn't typically work well, and this is a very basic example, is if you just want to get a summary of like an entire document or an entire set of documents, let's say, you know, instead of like asking a question about a specific fact, like what happened in like, you know, 1776? Maybe you just want to ask the language model, can you just give me an entire like summary of American history in like the 1800s? That type of question tends to not lend well to embedding based lookup because you typically fix like a top k value when you do embedding based lookup and you would get back very specific context. But sometimes you really want the language model to go through all the different contexts within your data. So a vector index, storing it with embeddings would create a query interface where you can only fetch like the k most relevant nodes. If you store it, for instance, with like a list index, you could store the items in a way such that it's just like a flat list, right? So when you query this list index, you actually get back all the relevant items within this list and then you'd feed it to our synthesis module to synthesize the final answer. So the way you do retrieval over different indices actually depends on the nature of these indices. Another, just like a very basic example is that we also have a keyword table index where you can kind of look up specific items by keywords instead of through embedding based distance. Keywords for instance, are typically good for stuff that requires high precision and a little bit like lower recall. So you really want to fetch like specific items that match exactly to the keywords. This has the advantage of actually allowing you to retrieve a bit more precise context than something that like vector based embedding lookup doesn't. The way I think about this is like a lot of what llamaindex wants to provide is this overall query interface over your data. Given any class of queries that you might want to ask, whether it's like a fact based question, whether it's a summary question, whether it's or whether it's some more interesting questions, we want to provide the tool set so that you can answer those questions and indices. Defining the right structure of your data is just one step of this overall process and helping us achieve this vision of a very generalizable query interface over your data. Some examples of different types of queries that we support, there's the fact based question lookup, which is semantic search using vector embeddings, you can ask summarization questions. Through using our list index, you could actually run structured queries. So if you have a SQL database, you could actually run structured analytics over your database and do text to SQL. You can do compare contrast type queries where you can actually look at different documents within your collection and then look at the differences between them. You could even look at temporal queries where you can reason about time and then go forwards and backwards and basically kind of like say, hey, this event actually happened after this event. Here is the right answer to this question that you're asking about. And so a lot of what llamaindex does provide is a set of like tools, the indices, the data ingestors, a query interface to solve like any of these queries that you might want to answer.

So Jerry, you really got me thinking a lot about this. The possibilities of the query schemes is pretty darn cool. You know, we kind of started with ingest and moved into kind of indexing and now we're talking about queries. Could you kind of give me an example and with the tool at a little bit more of a practical level, because you kind of hit the concepts about like what the possibilities are. But as someone who hasn't used the tool myself, I'm trying to get a sense of what that workflow is like, pick what would probably be like a really common query scheme that you're doing and dive into that just a little bit to give us a sense of a hands on, practical, you know, fingers on keyboard sense of it. Because I'm trying to get a sense of where I'm going to go for playing after we get done with the episode.

So I want to try it 100%. I think one thing that has popped up pretty extensively after talking to a variety of different users is actually financial analysis. I think looking at Sec 10Ks tends to be a pretty popular example. If you look at the anthropic cloud example, they also use Sec 10Ks. And my guess, the reason it's popular is one, there's just like a ton of text and so it's just very hard to parse if you read it as a human. Two, it's like a useful thing for people in like financial institutions like consulting because you want to compare and contrast the performance of different businesses and look at the performance across years.

Believe it or not, I actually read 10Ks a lot and that would be a really useful example for me. Believe it or not. I'm not kidding you.

As a result, we've actually been playing around with it a decent amount too. Some of the cool things that we're showing that Llamaindex can do on top of your 10Ks is for instance, let's say you have two companies, let's say Uber and Lyft, right, for the year 2021. You can actually ask a question like can you actually compare and contrast the risk factors for Uber and Lyft or their quarterly earnings across like these two documents and one is the Uber 10K. One is a Lyft 10K. This is actually an example where if you do like just top K embedding based lookup, the query fails because if you ask the question, you know, compare and contrast Uber and Lyft and don't do anything to it. And let's say, you know, your Uber and Lyft documents are just in some one vector index. You don't really have a guarantee you're going to fetch the relevant context to this question to be able to answer this thoroughly. And then the model might hallucinate, you'll get back the wrong answer, and then it's just not a good experience. I think what you typically want to do is have some sort of nicer abstraction layer on top of this query that can actually kind of map that query to some plan. That would roughly be how a human would think about executing or answering this question. Let's say you want to compare and contrast the financial performance of Uber and Lyft in the year 2021. Well, first, okay, what was the financial performance of Uber in 2021? What was the financial performance of Lyft? You break it down into those two questions and then for each question, use it to kind of like look over your respective index. Let's say you have an index corresponding to Uber and an index corresponding to Lyft. Get back the answer, right? Get back the actual kind of like revenue, for instance, for Uber and Lyft, and then like synthesize both of them at the top level again, be able to like pull in the individual components you extracted from each document and then synthesize a final response that's able to compare the two. So that's like an example of something that we can actually do pretty well with llamaindex and we have like a variety of toolsets for allowing to do that. And that's an example of a query that's kind of more advanced because it requires comparisons beyond just like kind of asking stuff over a single document. Another example, just to kind of like take the 10k analogy further, is let's say you have the yearly reports of the same company across different years. Let's say from like 2018 to 2022, you can ask a question like, did revenue decline go up or down in the last, like three years? And then you could actually do a very similar process where given the query interface that we provide, break this question down into some questions over each year, pull out the revenue, and then basically at the end do a comparison step to see whether or not it increased or declined.

Just as an aside to any listeners out there wondering why on earth somebody would read 10Ks, especially considering that our audience is focused on on data and such as, like, if you want to learn about another technology company and really understand what it does and be able to compare it, this is an example where you can gain tremendous intelligence on another company. With publicly available information and by comparing multi year 10ks like you just said, you'll learn way more about that company than its own employees know about it. So anyway, just thought I'd mention that as an aside.

Yeah, I look forward to hearing your success with speeding up your workflows around reading the Tim Case Chris with Llama Index.

I'm excited about this. This is going to save me a lot of time.

Jerry, one of the things that we talked a little bit about in one of our previous conversations, which I know you've also thought very deeply about and even have like a portion of the the docs and functionality in Llama index devoted to is evaluation. Like query response evaluation. Like how do I know my large language model like barfed up an answer, right? Based on some query and I pulled in some external data and I inserted some context and maybe I strung a few things together like how am I to evaluate the output of that? Could you give us maybe a high level from your perspective how you think about this evaluation problem and then maybe go into a little bit of some of the things that you're exploring in that space?

Yeah, totally. Just to preface this, like, we are super interested in evaluation or more tailored towards this interface of like your data with LLMs. I can dive into that a bit more and we have some initial evaluation capabilities, but we're super like community oriented. Like we'd love to just like, you know, kind of like chat with. There's a lot of like different tool sets out there that allow you to do like different types of evals of your data and building like nice interfaces for doing so. And so I think this is an area of active exploration and interest for us as well. And so just kind of thinking about this a little bit more deeply though. Evaluation is very interesting because there is the evaluation of each language model call itself and then there is the evaluation of the overall system. And so diving into this bit more like at a very basic level, if you have a language model, you have an input and then you get back some output, you can try to validate whether or not that output is correct. Right? Given a single language model call, did the model actually give you the correct answer? Given the input, did it spit out garbage? Did it like hallucinate that type of thing? The interesting thing about a lot of systems that are emerging these days is that they're really like systems around like a repeated sequence of language model calls. And this applies whether or not you're dealing with like a more agent based framework which you know, you ask A question, it can like just repeatedly kind of like do react, chain of thought, prompting or be able to pick a tool, but the end result is it's able to give you back a response. Another example of this is like Auto GPT where you just let it run for like five minutes and just like keeps on doing stuff like over and over again until it gives you back something. Or you know, even in the case of retrieval, augmented generation is just like a fancy name for roughly like what we're doing with llamaindex, which is like a query interface over your data. Like even within our system there could be a sequence of repeated LLM calls, but the end result is that you send in some input into the system and you get back some output. You know, given this high level system, how do you evaluate the input and output properly? I think in traditional machine learning, typically what you want to have is you want to have like ground truth labels for every input that you send in. So if you like, for instance, ask a question, you want to know the ground truth answer, and you want to compare the predicted answer to the ground truth answer and see how well the predicted answer matches up. This is still something that people are exploring these days. Even in the space of generative AI and LLMs. You have ground truth like text and then you have predicted text and you want some way of scoring how close this predicted text is to ground truth text. I think the core set of eval modules that we have within llamaindex actually are ground truth free or label free. And that part in itself is actually very interesting because you have this input, you ask a question, you get back this predicted response, you also get back the retrieve like sources like the documents themselves. What we found is that you can actually make another LLM call to just like compare the sources against the response and then also compare the query against the sources and the response to see how well all two or three of these components match up. And this doesn't require you to actually specify what the ground truth answer is. You just look at the predicted answer, see if it matches up to the query or the context in a separate LLM call. And it's interesting because one, it makes use of LLM based evaluation, which is kind of like an interesting way to think about it, basically using the language model to evaluate itself. Right. I'm sure there's like downsides which we can get into, but you know, a lot of people are doing it these days too. And then the second part is it doesn't require any ground truth because you're using the language model to evaluate the capabilities of its own answer plus context. You don't actually need to as a human feed in the actual answer. And the benefit of this is that it just saves a bunch of time and cost. You don't actually need to like label your entire dataset to run evals. I still think this overall space is probably relatively early. I think there's still some big questions around latency and cost if you're trying to really do LLM based evals more fully. Using the LLM to evaluate a large dataset takes a lot of time and costs a lot of money. And so this is generally kind of like an area that we're still kind of like actively thinking about.

Yeah, that's awesome. As we kind of like get near to the end here, I know things are like progressing so quickly. I can't keep up with all of your tweet threads about like awesome new stuff that's happening in Llama Index. But I know there's a lot as you look to kind of the next year and like where your mind is at, what you really want to dive into, and also what, what's really exciting to you about the field. There's a lot of people excited about a lot of different things. But from your perspective, having been in the trenches building large language model applications, interacting with users of Llama Index, what is it that really excites you moving into this next year in terms of the possibilities, the real practical possibilities on the horizon, and how kind of our development workflows will be changing in the near future?

Yeah, totally. I think there's a few related components to this I'm both excited for as well as like, you know, the challenges that we're going to solve. Probably the first component is just being able to build this automated query interface over your data. You know, if you're looking at all the query use cases that we solve, one of the key questions that we keep going back to is here's a new use case on top of this data and here's a new question that you'd want to ask. How do we like figure out how to best like fulfill that query request? Right. And I think especially as your data sources become more complicated, then it's just like, how do you think about, like how to index and structure the data properly? How do you think about interesting automated interactions that happen at the query layer between the language model and your data? How do we basically just make sure we solve this request? And then the second component to this is we want to make sure that we build this interface that can handle any type of query you throw at it. Now how do we do this in a way that is cheap, fast and easy to use? For a lot of users, once they move on beyond the initial prototyping phase, they think about starting to minimize cost and latency. And then think about how do you pick the best models for the job. There's the OpenAI API, which works well generally, you know, they're probably the best model out there, but it can also be quite slow. And then there's also these like open source LLMs popping up, probably like a few new ones every week. And then how do users make the best decisions whether to use that over their data? And then I think the next part to this is, you know, a lot of LLM development is moving in this overall trend of like automated reasoning, right? If you look at like agents and tools, it's an auto GPT and all this stuff, it's just like, how do you make like automated decisions over your data? And then I think as a consequence there's always going to be this trade off between how few constraints can we give it and how many like, like should we give it more constraints or fewer constraints? Right, because the fewer constraints you give to something like this, it has more flexibility to potentially do way more things, but then also just like error, right? Like it'll just like make mistakes and then there's no, really no way to correct it easily and then you can't really trust the decisions. Whereas if you kind of like constrain the outputs of these like automated decision makers or agents, then you can potentially get more interpretable outputs, maybe at the cost of like a little bit of flexibility in terms of functionality. And I think we've been thinking a lot about that with respect to like the data retrieval and synthesis space too, right? Like how can we give you back results that are expressive but also, you know, like perform well and are going to make mistakes like a ton of time?

Awesome. Yeah. Well, I'm really happy that we had a chance to talk through all the great Llama Index things and make sure if you're not following Jerry on, on Twitter, find him there. He posts a lot of great stuff that they're working on. And of course you can find llamaindex if you just search for Llama Index, there's like a great set of docs and all those things. We'll make sure we include those links in our show notes for people to find out about that and get linked to their docs and their blog and all the good things. So check out Llama Index. And thank you so much for joining us, Jerry. It's been awesome.

Thank you for listening to Practical AI. Your next step is to subscribe now if you haven't already, and if you're a longtime listener of the show. Help us reach more people by sharing Practical AI with your friends and colleagues. Thanks once again to Fastly and Fly for partnering with us to bring you all Changelog podcasts. Check out what they're up to@fastly.com and fly IO. And to our Beat Freakin residents, Breakmaster Cylinder, for continuously cranking out the best beats in the biz. That's all for now. We'll talk to you again next time.