Jerry Liu, LlamaIndex CEO - Interview

Hi, my name is Jerry Liu. I'm the co-founder and CEO of LlamaIndex, and I'm super excited to be here.

Fantastic. And tell us a little bit about what LlamaIndex does.

Yeah, so LlamaIndex is a data framework for building LLM applications. The key mission is to enable you to unlock the full capabilities of LLMs on top of your own private data, whether you're an individual or an enterprise with large amounts of different types of data from structured, structured. And so we're really thinking about how do we provide the toolkit to allow you to manage every stage of the data lifecycle as it's going into your LLM application. And so this includes a few different components. So this includes ingestion, indexing or storage, and then also the query interface and the interactions between LLMs on top of your data to fully give you back the best results.

Fantastic. And so when did you found the company or the project?

Yeah, so I started the project about 7 months ago. Back then, for those of you who might know, it was called GPT Index. And so started it back in November. At the time, it was mostly a toolkit to help get around some context window limitations because I think anybody building applications on top of the GPT-3 API was discovering that there were certain restrictions on being able to ingest a ton of data into GPT-3. And so I wanted to build some initial project. That project got a decent amount of interest, which in turn inspired me to work a little bit more on it and it kind of just took off from there and now it's evolved into this overall toolkit that's solving this critical need around data.

And so it was originally just kind of a personal project?

Yeah, it was a personal GitHub project. It was an open source thing. I thought it was fun. I've always been interested in startups and entrepreneurship and AI and at the intersection of all these things. I didn't know for sure at the time I was going to create an LLM tooling-based company, but that part was a little bit spontaneous. But I think a lot of the experiences I've had from ML to research to working on ML engineering to working in a startup experience has roughly led me to this path. And so in the end, I would say in hindsight it makes sense.

Fantastic. And so back end of last year, Q4, did you have the problem yourself or were you talking to people who had the problem? How did you identify the fact that there were these limitations in GPT-3 and that there was a problem to be solved?

Yeah, definitely. So I was actually playing around with the GPT-3 API and Around late October, I was trying to build some sort of sales bot on top of my previous company's customer conversations. Because I was in customer conversations, I would forget what the agenda was for each subsequent meeting because they were spaced out every 2 weeks. So I wanted it to ingest customer conversation data and synthesize a to-do list for me before the next meeting. And so as I was playing around with actually doing this, that's where you actually run into some of the context window limitations where you GPT-3 had 4,000 tokens. Some customer conversations per account had way more data than what could fit within the 4,000-token window. So what are strategies for both storing this data and also retrieving this data so you could use it with your query?

Okay. And talk us a little bit about the project that you were presenting today.

Yeah, so I guess from that project, which was back in November, it's grown into this overall toolkit. And so there's basically 3 main components to LlamaIndex as it stands today. There's ingestion, there's indexing, and there's the query interface. So ingestion is how do you get data from its original format into some format and into a storage system that you can use later with LLM applications. These days there's some initial stacks around vector databases. So you take some documents, you chunk it up, parse it, add some metadata, store it in a vector database. I did talk a little bit about storage abstractions during the talk itself. And so we provide integrations with vector databases and also just generic key-value stores and allow you to basically flexibly store different types of data in different ways. From there, there's basically ways that you can define both indexing and structures on top of the data, as well as some sort of query interface over the data. A very basic one that's emerging these days is that once you actually store document chunks in a vector database, You can do top-k embedding-based retrieval and then put that into the prompt for synthesis. I think what we're thinking about is a generalization of this overall use case and really thinking about what are the overall set of queries that a user might want to run and what are some not just simple but also advanced retrieval strategies on top of your data? What state do you need to define beforehand to make that happen? And then how do you basically use the LLM combined with traditional machine learning information retrieval methods to give you back the most relevant information and synthesize the best results.

Fantastic. And so the kind of integration with MindsDB, how do you see that benefiting obviously the customers and the users of LlamaIndex and also vice versa for our customers and users?

Yeah, that's a great question. I could totally see that benefiting an existing user segment of data analysts that are roughly familiar with SQL. So basically SQL is still less accessible to some people, but for some people they're very familiar with, it's a very familiar interface. And as a kind of very powerful declarative interface, the fact that right now the integration is that you can basically use the SQL interface to reference a Llama index structure as a query engine and then basically feed in Llama index query engine as some sort of compute layer under the hood for certain classes of SQL expressions that you write. So for instance, you could select star from this model and feed in a set of questions in SQL style. And then basically join this question table with the model, which is powered by Llama Index, and use the power of Llama Index to fulfill the answers to these questions and give you back the right outputs. And so I think for a lot of data analysis use cases, I think that makes a ton of sense. And you basically get access to the advanced capabilities that we offer and can retrieve on top of your data in addition to the simple capabilities. The next layer I think is that In addition to catering to data analysts that know SQL, you can also cater to even like a level up, like end business users that don't quite know SQL but just want to query with natural language. Because once you wrap a layer where you can do some sort of like natural language to SQL, and then that SQL is basically powered by kind of your very powerful declarative interface, and then which can in turn call the LLM or other predictor models, then you basically have this like full analytics engine set up. And it's like at every level of the stack you can cater to a different audience, but in the end it's just a very powerful thing.

Fantastic. And so what are some of the unexpected ways that you've seen your users and community use LlamaIndex?

Yeah, I think the most unexpected way, so the core toolkit by itself, if you use as a standalone thing, is really meant for data analysis tasks. If you do question answering over a set of documents or do analyze financial statements or analyze structured data, or do some combination of both. That's what we want people to use. At the same time, or sorry, what we think is the framework really helps to solve. That said, we have seen some pretty interesting projects where people are actually using LlamaIndex constructs to build their own agents that can interact in the world. Even though we don't inherently have any opinionated agent frameworks, there are some basic conveniences that we provide. For instance, we abstract away the need to worry about context limitations. We abstract away the need to really think about how do you chunk up your data and structure it and link it. And there's a certain set of primitives that we provide that are actually helpful to build just a variety of different types of LLM experiences. And so we're starting to see some basic knowledge agent-like structures emerge using our toolkit too.

Fantastic. And what are some of the things that you're working on now? Some of the things that maybe new features or things that keep you up at night? What's—

100%. I think there's just a lot of things. And so I think it's more of a matter of prioritization as opposed to just too little things to do, if that makes sense. These days, during the presentation, I gave a sense of basically some use cases that we could solve for different types of abstractions that we provide can better solve different use cases, for instance, like document comparisons, like time-based queries and the like. There's just an entire class of queries that we want to continue pushing the performance boundary on. And just really thinking for this class of use cases, maybe for this SEC filings or for medical patient records, what exactly is kind of like the engineering and the right interaction and the interface in order to really get you back the best results for this domain. And so I think we're probably out of this spectrum of the entire set of queries a user might want to run. We're probably 50% there. And so there's just a long tail of other things that we want to solve. A lot of work. Beyond that, there's just a ton of partnerships, integrations, and keeping up to date with the latest research integrations with all the storage providers. Also starting to think about on the technological side, what sort of agent-like behavior, which I alluded to at the talk, really makes sense for data retrieval. And synthesis because again, that's something that we're really passionate about.

And how big is your team right now?

Right now it's 2 co-founders and 1 full-time person. And we have a few contractors, but we're actually hiring as well. So if you're interested, should be in your—

Check out your website. Fantastic. And so now kind of creating a real business, right? Started as a project, some point between the end of last year and now, transitioned into a real business. How was that transition? Obviously you didn't expect it when you first started this, but how have you found that transition to be? There's many different challenges to running a business than just having a really successful GitHub project.

Oh, 100%. I think, yeah, it's kind of funny. I think basically the way it worked was it started off as a side project. And then I think if you somehow were able to plot a graph of just the number of hours I spent on this and x-axis is the month and the year and then y-axis is hours committed, you could probably just see a straight line going upwards after a certain point. And maybe plateauing because at a certain point I need some sleep per day. Yeah, I think just on a very personal level, I think I've always wanted to start a company. I think this is something that has always excited me, just the ability to create a really cool project, have users and really try to build some core piece of tech that transforms the industry. It is pretty busy as you could imagine. And so I do think if you do want to start a company, you do have to be all in and really be dedicated to it. And I've rewired my brain at this point to just really think and just be passionate about work. And you do have to manage your own timeline so that you don't burn out and that you can manage stress. But I think at the same time, it is also a very exciting experience and I think I'm kind of at a very fortunate position where I just kind of built this project during a time when a lot of people were interested in generative AI. And I think that's something that is pretty rare because a lot of times you can have very smart people building really cool things and maybe the market's not the right timing. So I do think it's some combination of just a lot of it is luck and then some of it is just executing and trying to build the things that people would want to see. Yeah, absolutely.

And how important, obviously, you know, only 3 people, obviously expanding, but how important is kind of community in your ability to kind of keep up?

Yeah, that's probably one of the most important things, if not the most important thing. I think the community is how we basically got here. And then growing the community is one of our top priorities for the end of this year. I think just the reason is you're starting to see some basic stacks emerging. On how you use LLMs with other services. And we think that our toolkit can provide a lot of value to different types of users, whether you're kind of like a beginner who just trying to run this on top of your collection of notes or you're an enterprise user. And I think there's just been a lot of passion and interest from the community in using our toolkit for various types of data tasks, and we want to continue supporting them. And I think that's one of our key goals. And I think Once we feel like we've kind of figured that out, then in parallel, in addition to the stuff we're doing, we'll start thinking about how to scale this up to enterprise. But still building with the community is one of the key goals that we have.

Fantastic. Is there anything else you'd like to add about the project or about the collaboration or anything like that?

Well, I mean, general notes, I'm super excited about future collaborations, partnerships, integration opportunities with MindsDB. I really enjoyed this conference here today and I think there's just a lot of exciting features coming out for LlamaIndex, both on the agent-like behavior. Also, we're starting to think a lot more about ingestion indexing, being able to index different types of data from semi-structured data to image data, especially as these models become more multimodal. And so yeah, I think the last note is if you're interested in potentially joining the Llama Index team or even just hacking on some projects, just shoot me an email. We're super open to conversation.

Fantastic. And one last question. So obviously right now there is a huge conversation around the world about regulation of large language models, generative AI. Europe have started to propose some limitations around licensing. What are your thoughts on that? Just kind of high level generally.

Yeah, I think this is a very broad discussion and maybe I can kind of paint a picture of our focus area and so parts that could be relevant and parts that we're not necessarily completely thinking about at the moment. Right now we're an inference type tool. So you take a model that's already trained on something and then you can apply our tool on top of this model. So we don't own the model or the training data that the model was originally trained on. And I think there's a lot of interesting regulations and licensing issues around the training dataset itself, the behavior of the model. I think the model itself is a very interesting topic because these days there's a lot of open-source models coming out and there's a lot of concern from enterprises of depending too heavily on a third-party provider. So OpenAI, for instance, has the best models probably in class right now with GPT-4. There are going to be some businesses that are a little bit wary of using this model. I think what we can provide is probably some sort of cost analysis layer. As a toolkit that can operate as a layer on top of these existing models that can basically help them decide the trade-offs because we can say, hey, we think this model performs like this, but you're also going to have these trade-offs of this data will be hitting a third-party API and we can help them make those decisions as an application toolkit.

Fantastic. Excellent. Thank you very much.

Thank you.

Appreciate it.