#2 Jerry Liu - What is LlamaIndex, Agents & Advice for AI Engineers

Good morning, everyone. How's it going today? Welcome back to the podcast. In today's podcast, I'm going to be talking with Jerry Liu. Jerry is the CEO and co-founder of LlamaIndex. We're going to be talking about LlamaIndex, what it is, and what you can do with it. In case you don't know, LlamaIndex is a framework that allows you to create large language model-based applications. Although we're going to be introducing it with a little bit more detail in a moment. We're also going to be covering LlamaParse, their API for parsing unstructured documents. And finally, we're going to be talking about LlamaCloud, which is their enterprise solution, to see what they have to offer and how LlamaIndex and LlamaCloud come into that. In this process, we're going to be covering some technical topics and explanations as well, about how the systems work, including topics such as advanced RAG and data processing. So it might get a little bit technical at times. In addition, by the end, you will have some advice from Jerry Liu on becoming an AI engineer and starting a career or a startup in this industry. If you're watching the video version of this episode, you have subtitles here, and I hope that I hope these subtitles will help you to understand the more technical topics. In case you don't know what a concept is, you can always look at the subtitle and look up the concept. I also added some notes within the subtitles to help you find the concepts more easily. So yeah, I invite you to consider this episode as study material. It is not always that we have Jerry Liu himself explaining these topics to us. And during the conversation, we try to make them as approachable as possible. So without any further ado, let's get started. I bring you Jerry Liu.

Jerry, thank you for joining.

Hi, Andrew. Thank you for having me.

So let's talk a little bit about Llama Index and just To start, can you please tell me what is LlamaIndex?

Great. So LlamaIndex is a data framework and platform for helping developers build LLM applications over their data. And so we have two main product offerings. We have an open source component, which is a very broad orchestration toolkit enabling any developer to compose different models together to, for instance, like index data, put it into a vector store, and also build different types of LLM applications on top of that data. This includes, uh, some very popular concepts popping up these days, uh, including, for instance, Retrieval Augmented Generation, which is the core of building, like, a chatbot over your data. And this is also— this also includes use cases like agents, which includes more autonomous, um, pieces of software that can go and automate workflows, perform actions for you, like, send an email, schedule a calendar meeting, write code, and do other things as well.

I see, I see many people talking about agents recently. That's really taking some traction, isn't it?

Definitely. I think there's been a lot of interest in a variety of different use cases. I'd say the top most popular ones are probably RAG and also agents. And there's actually, you know, we talked about this in some of our talks, progression between them. 'cause we see this as like an overall spectrum as opposed to like there's like a fence between every different use case. RAG is really just a very simple mechanism of doing search over your unstructured data. And there's ways that you can basically add a bunch of agent ingredients to make it more agentic.

Amazing. How does it feel to be part of the driving force of this, this industry?

Yeah, definitely. I mean, as a founder, it was unexpected. I'm also very grateful for the opportunity. I'd say a lot of The reason the company got started was because we were at the right place at the right time. But also, you know, it's one of those things where we saw it and we really tried to take full advantage of this to basically create something that was really powerful. And this includes both the open source side, which, you know, includes the technology, but also the community. So basically, you know, a lot of the Llama Index, why people like the brand and the community is because of our kind of education resources and ways of showing others how to build different types of things. And not just in like a hacky way, but also building it well. And so that's something that is core to the company and something that we want to continue to invest in. Because part of this like ecosystem, right? Part of this like GenAI hype that is fueled by actually developers knowing how to build stuff. And if we can continuously be on top of whatever LLM advances are coming up, and basically show them how to take advantage of these LLM advances by showing them, you know, what are some of the new use cases that are unlocked? What are some changes in like paradigms or abstractions that you should think about? That's basically what we want to focus on.

Amazing. And just to, to be sure that people are like our listeners are understanding precisely what your company does. You said that you allow developers to create applications that allow you to chat with your data and create agents. That's, that's correct. Is that like a very high-level description of LlamaIndex?

Yeah. So let me, let me give a little bit more color to that. Um, so thanks for asking that question. Um, basically LlamaIndex, um, the overall mission, right, taking a step back, is to just— it's pretty broad— is to enable any developer, um, to build just, uh, LLM-powered applications. And it turns out a lot of these LLM-powered applications depend on a user's own private sources of data. Now, um, the source of data, these sources of data can include, uh, a variety of different things. Um, so it could include like unstructured documents like PDFs, uh, PowerPoints, uh,.docx files. Um, it could include like APIs that you have access to. Um, so, uh, commonly used like services like Slack, uh, or Discord or Notion. Um, it could also include, for instance, structured databases. Like if you're a company, you have a data warehouse of like very structured data. That's, for instance, the type of data you might want to get the LLM to analyze and give you insights over. And so there's a lot of data. A lot of times, especially if you're a bigger organization, you have a lot of it and it's also in different types of silos. And it turns out you actually, you know, maybe have some of it in use by your business analysts, but also some of it is just like sitting in a bunch of files and you really don't have time to like sift through all of it. One of the promises of LLMs at the same time is its ability to just rapidly process any type of information and give you back responses. To really illustrate this, imagine you're in like the ChatGPT interface and you're able to like, you just copy and paste like some piece of text from like a webpage, from like a book, just into that little text box in ChatGPT or Claude or any of these UIs that are popping up. and you realize the LLMs do a remarkable job of being able to really understand the text that's in this box. Um, even, you know, if it's not in like some structured, uh, like field, uh, a structured manner and, uh, actually give you back responses over it. And so you take that insight that LLMs are just really good at just reasoning over any type of data. And then you combine it with, okay, well now if they're really good, I want to combine that with like, um, you know, on top of all the data that I have access to in the organization. And you think about the challenges to basically make that happen. And so what we— how we started as a company is basically building developer tools to enable that bridge, enabling any developer that was using an LLM API to easily figure out how to make use of all these types of data and what are the different patterns and paradigms that emerge by being able to, you know, to, to like to load this data into the LLM prompt window. And you have to deal with different types of challenges, for instance, like figuring out how to fit context into that prompt window, figuring out how to basically get the LLM to interact with different types of data interfaces, unstructured data, structured databases, perform actions, those types of things. And so that's kind of like how we started. And that's where a lot of these like indexing, like RAG abstractions came into, like kind of arose as well. Because a lot of people were thinking about similar problems and they basically came up with this like general purpose technique, uh, like retrieval augmented generation, um, to figure— to basically, uh, show— to basically, um, uh, load it as a like basically set a standard for loading in different types of data, putting into a vector database and doing retrieval on top of that data to fetch only the most relevant bits and putting that into the outline context window. Um, So as we've started, we started off focusing a lot on that use case because it was also a pretty big use case for the enterprise as well. Because what, what, uh, like what enterprises wanted is one, they wanted to actually have their developers like build these types of applications. And for us as a company, we wanted to give these developers like the developer tools to build these applications. And so the open source project, uh, going back to that is a pretty important piece of it. Because open source is something that most developers really enjoy using. It's really just like a set of libraries that you can just import. It's very flexible and customizable. And so the trick is you want to make sure that you have a library that both like beginner to advanced developers can actually utilize to, you know, be that bridge, connect your different types of data and build these different types of use cases. And so one quick note on the different types of use cases, the orchestration framework is, uh, pretty broad. It supports a lot of the emerging patterns that people are building these days. And so one such pattern is, again, like RAG, which is basically question answering over your data. And then, as I mentioned in the beginning, there's also other types of patterns that are emerging, like the autonomous, like, like software engineer or sales assistant or research assistant, like basically things that can operate more in a continuous loop with a lower back and forth human interaction. That's something that we're focusing on as well. The last bit, which I'm happy to talk about in subsequent questions, is we have the open source project, but then we also have an enterprise platform, which is Llama Cloud. The goal of Llama Cloud is that it actually complements the open source project by being a managed data platform, data ingestion service, to basically help really solve that problem of connecting your data, uh, with any sort of storage system and be able to define that workflow, have it run very, uh, be very accurate, have it run reliably, and then also remove the need for you as an AI engineer to really maintain those pipelines. Um, this is an emerging system need that we see, um, as part of this like overall data stack that forms like Allen software. and that we realized, uh, for a lot of like enterprise use cases, it makes sense to basically build an overall managed service around it to really solve that data quality issue for any sort of LLM applications. And of course you can still use the open source to build like different types of, uh, applications on top of that, on top of that data.

Amazing. And to be sure that I understood that last part correctly, in this orchestration framework that is LlamaIndex, you have RAG. Which is Retrieval Augmented Generation. You have autonomous agents, and then you have the enterprise solution, which is Llama Cloud, which allows developers to more easily deal with the workflow of ingesting and managing data in these kinds of ecosystems, right? Is that broadly correct?

Yeah. So the way to visualize Llama Cloud is basically imagine like a general knowledge management interface. And so like this, the goal of this platform is to just expose all the different types of data that you might want and make sure that data is cleaned and ready to go. So that like, you know, your unstructured PDFs will be able to process it, make it into a format that your LLMs can consume. You know, in the future, for instance, like structured data, we're or any sort of like user data, like conversation history, we'll also be able to process it and store it in a format that your LLM can consume as well. And so basically, assuming like, you know, if we take the RAG pattern, what does this concretely mean? You have a set of like PDFs, you want to, for instance, like parse it in the right representation, you want to chunk it, make sure that it's actually split up into more bite-sized pieces so that LLMs can really understand them. and then you want to put it into a vector store with the right retrieval strategies, uh, to make sure that given kind of like any sort of question that the LLM or a human sends to the system, um, that's able to surface the relevant context. And so that, um, whole stack is kind of like the indexing retrieval piece of RAG. Um, and if we can provide that as a really nice data interface, then you basically have this like set of managed APIs. That anyone within the organization can consume, run, and basically have this like guarantee of quality.

And that ingestion is usually one of the most complicated parts of developing this kind of system, right?

Yeah, for sure. And so you can, just talking about that type of spending a little bit of time on talking about why it's complicated. We've mentioned this in some of our kind of like talks as well. It's basically you have this new type of ETL because you have like unstructured data and you want to process it into something that is a little bit more structured. And but it's a new pattern. So for instance, for RAG, you know, you're taking again like an unstructured document, like a PDF, and then you're parsing it and then chunking it and then putting it into a vector DB. And this actually looks a little bit different than the previous stacks. Like if you think about traditional ETL, It's typically just like a set of operations to move structured data from one place to another. Um, like part of the reason like Snowflake or, you know, some of these other, uh, data warehouses exist is you like load in a lot of kind of like messy data and then you typically use a human to basically write a lot of operations to massage this data into something that's very clean and structured. Um, but again, I think just like the overall stack has shifted because a lot of time, like the, the set of operations you need to do on this unstructured data is different for RAG. And the second piece is that, um, the second, uh, reason why it's challenging is that all these parameters actually affect the final performance of your LLM application. So your parsing strategy, uh, your chunk size, um, it's actually, there's like a lot of these knobs that you have to figure out how to tune. Um, also adding like metadata annotations on top of your documents and, um, all of these affect your performance. And so if you're not careful with some of these parameters, you end up getting a system that is just not very performant, and it's also a little bit hard to improve. And so the data decisions directly affect your accuracy. And this isn't really something that's occurred kind of in previous types of software.

Right.

Yeah.

I want to unpack a little bit more about the LlamaIndex and the LlamaCloud part. But before that, I would like to talk a little bit more about the history as well of LlamaIndex, because I think that our listeners will be very interested in getting the inspiration from your personal experience. Can you tell me how the company Llama Index started?

I think basically it started back in November of 2022. This is around the time that, or it was October, that a lot of people were basically trying to hack on language models because people were getting excited about GPT-3 again. There were a few startups popping up. And I was also getting pretty interested in just like, you know, trying it out, seeing what happens as a developer, just to like hack around on the API and see what are some of the learnings and what are things that are hard to do? What are some of the things that are easier to do? And so I basically opened up an OpenAI developer account and then, you know, like started playing around with text-dimetry-003. And then as I was playing around with it, you know, one of the emerging patterns that I wanted to build, and also I was seeing like some of the other people build, as well as like, you know, how do I figure out how to use this to basically, you know, um, uh, answer questions about like, uh, you know, all the sales data that I have in the company, right? Um, or just like have this, have knowledge of basically everything that exists, uh, within the, um, the, the organization. Um, because what I wanted to do was I wanted to feed it all the sales conversations, uh, because I was in a few of these like customer calls and I wanted to really like synthesize insights and help me prepare for the next meeting. Because otherwise that was actually taking a decent amount of time on my plate. And I was on the engineering team, but I was just sitting in a bunch of these customer conversations. And I just, because I was balancing a few different things, oftentimes I would just forget or just have to explicitly allocate time to listen to, to review these customer transcripts to basically prepare the right materials. And so, It was a, it was like a slight pain point for me that I wanted to solve with LLMs. But then I was like, I was trying to build and prototype these applications with, um, initial like LLM powered application. And I realized that the context window is like 4,000 tokens, right? So I can't just like dump, um, like the entire set of documents into the context window and call it a day. Um, and what I needed to figure out is like some more clever indexing strategy so that like I could index all the documents somehow. But then given that I had some sort of question, the LLM could somehow figure out how to traverse this overall index of information to basically find the right piece of information and give that back to you. This was pre this idea of RAG. Actually, RAG as a research paper had existed before I started this project. I independently just came up with some technique that, to be totally honest, was a lot worse than just this idea of RAG. I wasn't using any embeddings. I wasn't using any text embeddings. I was just trying to use the LLM itself to just think about like hierarchically organizing information into like a tree. And then just like having that LLM like traverse that tree. To be honest, it didn't really work the first few times around, especially because like DaVinci 003 was not super good at reasoning. So it wasn't great at being able to figure out like make a decision given a set of choices. That said, it was interesting. And I think, you know, the people— like, I put on Twitter, and I think there was a decent receptive response to it. Like, there were a lot of people interested in this overall approach. And I think a lot of people were getting just excited about LLMs at the time. And so it provided a nice starting point to basically grow some initial— have some initial traction, which then motivated me to keep working on this. And in this virtuous feedback loop, eventually realized a month later I should probably quit my current job and then also think about starting a company around it. But it really started off as just this developer-led, like, I want to try to solve this pain point and this is the tool I want to build to solve that pain point.

Right. So about that history of RAG, modern RAG was actually kind of born in 2022 then, right?

I forgot the exact date of the original retrieval augmented generation paper, which is not by me, obviously it was by others. This is either in 2021 or 2022 or even 2020. It basically proposed this overall idea of, oh, you take in some set of documents and then you want to embed them, put them through an embedding model and then put it into some storage system that's able to serve like the relevant documents through retrieval, right? And so that's why it's called like retrieval augmented generation. Like you want to do like a retrieval pass over some storage system before you actually put it into the LLM, um, prompt. So that kind of resurfaced basically, um, as more and more people start building with LLMs, people like kind of started discovering that, oh, hey, this thing is like a cool idea. Um, and my initial version was not doing that. It wasn't using embeddings. Um, because at the time I don't know why. I think it might've just been due to a design. My goal at the time when I first started was not necessarily to make this useful. It was to just do something cool. And so to me, doing something cool was like, oh, what if we just didn't have embeddings? Or I thought about it briefly, but I was like, what I really want to do is just have the LLM figure it out completely on its own. And I still think that would be quite an interesting concept. Instead of just relying on a separate model, just have a language model completely similar to like a human, just completely figure out how to reason, organize things, and then also traverse them via text. Yeah.

And that kind of reflects in the current state of LlamaIndex, right? Because I mean, you, it's kind of a central, I mean, as far as I can see, it's kind of a central part of LlamaIndex that you use language models as well in, during the ingestion process, not only in the generation process.

Yeah, so the default RAG paradigm really only uses the LLM at the very end. So you have ingestion. Ingestion doesn't need LLMs. You just take in some data, parse it, and then you just chunk it using an algorithm. And of course, you use an embedding model to put it into some vector store. And then the retrieval process doesn't use an LLM because at its simplest, it's just top-k embedding lookup. So You look up stuff by embedding similarity. And so in a standard RAG pipeline, the place where LLMs actually come in is at the very end. And it's only responsible for synthesizing an answer from a piece of unstructured text. And to be totally honest, even at the start when we were just implementing this, I thought it was a little basic. And it didn't really use LLMs to its full potential. LLMs are not just for generation and simple reasoning. They can actually help you make decisions. They can actually help you add a greater layer of understanding and decision-making. If you really wanted to make these systems more interesting, you could use LLMs at the beginning. For instance, during the data ingestion phase, or, you know, during query time, um, instead of just using it at the very end for generation, use it for like query understanding, um, use it for, uh, like processing, like evaluating, like the quality of your retrieved context. And then for instance, like not only just retrieving from a vector store, actually using a variety of different tools. Um, and so on the ingestion side, um, the places that you can use LLMs. And so this, this, this overall concept is pretty interesting, which is Um, the whole point of like ingestion is to process data for your outline map. Um, and so that's kind of like, uh, ETL for LLMs, right? But you can also use LLMs for ETL, um, because, you know, LLMs have an inherent capability of understanding unstructured data and transforming it. And that part I think is interesting. Like, so for instance, um, let's say, you know, for each unstructured document, you wanted to extract like a summary, the table of contents, um, like, you know, extract like a set of like topics or tags for each page. Basically, you can figure out a clever way to prompt the LLM, um, by feeding it in a bunch of data, uh, from the document to basically first extract out a set of like structured annotations or tags. And this represents like a data transformation basically, because you're trying to like feed in some input unstructured data and transform that into structured data. And then you can basically attach those tags on top of the unstructured data as well. And so these, like, this is just an example of like metadata extraction that's also powered by LLMs. And this is something that, you know, um, uses LLMs, but is also independent, like, you know, useful for just like the, any sort of downstream application you want to build. Because if you're trying to build a RAG system over this, having metadata tags is oftentimes very useful. It gives you like better retrieval results, better generational quality, and, and all those types of, types of things. And so I think that interplay between LLMs, um, and kind of like data transformation is very interesting because you can use it for like in the middle, but also it helps for any sort of like applications you want to build later on.

Yeah, that's pretty some advanced RAC techniques over there. And it kind of brings, goes back to a little bit to your, I mean, it's a little bit adjacent to your original idea of creating this kind of systems, right?

Yeah, I think the, I thought about this in the beginning of the project, but the project was not really close to realizing that vision at the time. But if you think about the overall picture of where I think LLM-powered software will evolve, it's basically like there's a new type of data stack that's emerging. A new set of operations within that data stack to basically power like, like AI software and to really like, kind of like we want to basically provide the right tooling to help developers build that data stack. And so this is helping them figure out how do you like, you know, move data from one place to another, specifically for LLMs to use. And that could include like LLMs in the middle as well. And this also includes the orchestration piece on top of that data. How do you figure out how to get LLMs to interact with the data through these different types of interfaces?

Awesome. I want to talk a little bit about your background, your personal background as well. What did you do before you started Llama Index?

Yeah, so I was at a Series B company called Robust Intelligence, which was doing ML Ops, testing, monitoring, evaluation. It had a pretty talented team. You had Harrison Trace on your podcast, and so we were all coworkers basically at the same company. And then yeah, it was fun. I mean, basically before that I had spent some time in both bigger companies, right? Like kind of, so Quora was like 200 people. Uber was like Uber, Uber is very big. But I'd spent some time at both like ML engineering and then, and then research. So let me actually just take you through the whole background. So, so basically like Quora was my first job out of college and I was doing like recommendation systems and doing like very practical hands-on ML engineering work. So that was a great first job. I learned a ton. And I also learned a lot about just like how to not only like train these models, but also how to apply them and basically make them reliable. And then I wanted to like really dive a little bit deeper into the research side and strengthen my ML understanding of fundamentals. And because I was very interested in training neural nets. And so I, you know, did like the Uber AI like residency program, which transitioned into like a full-time kind of like research offer type thing. But basically, I was working on like deep learning for self-driving systems. And this included like different types of image understanding, sensor understanding, and also planning. And some of these concepts, as you dive into it, they're starting to become some similarities between planning in the traditional deep learning sense and then also LLM planning. And there's some patterns that are pretty interesting to explore there. But basically, I was very interested. I worked a lot on sensor compression. So how do you compress images and sensors? Using like the relevant algorithms, but also using neural nets. And then I worked a bit on like robotic planning as well. So being able to figure out how do you like make the right decisions, like if a pedestrian is crossing an intersection, you obviously want to make sure you brake. How do you predict their behavior and make the right decisions there? And then after that, I had worked at bigger companies and I really wanted to see what like an early stage MLOps company was like. And so I joined Robust Intelligence. And Robust Intelligence was like, I think I joined as like the 9th or 10th employee. So pretty early on and, you know, like I wanted to really understand what it meant to one, build something in the ML space that actually could make money because as part of a business and then two is like really understand the growth journey of an early-stage startup. And I think I actually got a bit of both. Like, I mean, Like I learned how to really aggressively prioritize things, especially in a startup environment when you have like limited time and resources. And it's a little bit different than like the academic environment where kind of you do a PhD, for instance, you're supposed to, you know, like spend time, like really reflect on kind of like what, what to do, like build something that's deep and meaningful. In a startup, you don't have a lot of time and you really have to just make sure you like aggressively like 80/20, like the very first thing you want to build. And then you want to make sure that like, you're able to kind of like build initial versions of something before like diving into something more deep and complicated. So I learned a bit of like, you know, that type of fundamental trade-off, the ability to like lead a team and interact with product, a little bit of like being on these customer conversations as well. I think it actually gave a pretty solid base for just like kind of starting this company later on. I think, you know, in general, you know, if you're like a listener and you really want to like start a company, I think actually a good like training ground is to just work at like a, like a, like early stage company with very talented people, people that you respect and just like really just try to learn as much as possible, like inhale as much information as possible in the first year or two.

Yeah, that's some amazing advice, I think. So would you say that As far as I understand, pretty much all of your experience is in machine learning. Had you ever considered going to the software development side or not really? You were always completely on the machine learning side.

Yeah, so I think as a, I was a machine learning like engineer basically at both Quora and not at Uber, but at Robust. And for both of these jobs, I think it, does involve a decent amount of software engineering. I was a little more on kind of like the, yeah, but obviously it's a little bit different than say like pure infra engineering. Like I didn't, I never spent like a significant amount of time working on like distributed systems, for instance, or kind of like doing deployments and, you know, like networking, like a lot of like different types of concepts and like traditional software engineering. I did internships like, you know, throughout the college years. So I built up like some foundation, but In general, I think the intersection between machine learning engineering and product was where my primary interest was the most.

What did you study to start taking up these roles actually?

I studied computer science. I graduated from Princeton back in 2017, did computer science. Actually, I got into ML pretty late. Maybe that was one part that I didn't drop in the initial introduction to the background is I started doing machine learning or really thinking about it the latter half of my third year in college or junior year. And so a lot of people who are interested in this, especially nowadays, AI is so big. If you're a high school student or something or middle school, you start start like looking into AI and you start like trying to understand what it's about. But like, um, at the time I mostly got interested in AI because it was like junior year of college. I saw like one of these example new models that just come out, came out that had like really good kind of like image generation capabilities for its time. Um, it could generate like a little icon, um, like 64 by 64. Uh, and it actually looked like somewhat realistic. I was like, oh, that's cool. I want to learn more about that. And that kind of kicked off like my academic interest in the space and made me realize like this is kind of like an overall area of just like, I guess, computer science or computers that I wanted to focus on and really understand a little bit more.

Awesome. And that, that, when was that, by the way?

Like in like, this is like year 2016, 2017, right? Yeah. I mean, I think it was, like I graduated in 2017 and I think ImageNet had come out like a few years before that. I mean, I think there was like some initial excitement in deep learning. And actually, like, to be honest, I think if you were in the like ML research world, there was like that hype had basically started around like 2014, 2015 and basically continued. And then for basically everybody else, for anyone who didn't have exposure to like deep machine learning, I think that hype kind of took off with like ChatGPT and all these. Yes. But among like the research community, a lot of the advances and kind of like the understanding of the progression of like AI and results had been going on for a few years. Right.

So you were pretty lucky to be like there before actually everyone else was there.

I mean, I think one of my biggest regrets is honestly, I wish I had just gone into ML even like a little bit earlier. If I had like known about like really what it was or thought about it like freshman year, it honestly was like pretty interesting. I probably just, one, didn't have the right technical fundamentals to fully understand it yet. And two, like, I just, it was like, I just like discovered it too late. I think I'd always been interested in startups. Like I'd always been interested in like building something, but that technical interest in machine learning, just, I just didn't really discover it until later on.

Awesome. And well, and now you have built LlamaIndex and it's gone super far and now you're implementing new products to the public. So let's talk a little bit more about LlamaParse and LlamaCloud. And yeah, just tell me, what is LlamaParse?

Yeah, so kind of taking a step back, you know, we have this like orchestration framework. We have had LlamaIndex, which has been around for about like a year or a year and a half. It's fully open source. It's going to continue to be a core part of our strategy. We want to make sure that, you know, we build the right tooling to enable any developer to build these applications. both when they're prototyping and also when they're productionizing. Um, I think like the overall motivation is that as we started talking to a lot of these like enterprise developers, we realized that, um, they were running into just like general pain points, right? Um, to, in, in building something that was production quality, um, the response quality wasn't high enough. They were having a hard time trying to improve the application. Um, they were running into like hallucinations. They, they were having a hard time like trying to connect to more data sources. And so fundamentally, when we thought about kind of like a managed service to build, we wanted to build something that actually solves some of these pain points, especially in an enterprise and production setting. So the overall like summary of like the Llama Cloud platform is it's meant to be overall like data platform that is going to process and clean your data to the best quality for LLMs. And so we think that like, you know, Good data is essential for good, like, downstream performance with any sort of software that you want to build. Um, if you don't have good data quality, that typically leads to all these issues I just mentioned. And so we wanted to build some overall service that can basically parse, index, um, and clean, and also let you retrieve over this data to give you back the best interface for your LLMs to work with, to access any context that you have within the organization. Um, so that's the high-level picture of Llama Cloud. And it consists of a few different, uh, core components. The first is a set of like data connectors, uh, to basically load from any sort of, um, data source that you have. So like unstructured data, um, like any sort of documents, um, and then, you know, we're building these integrations for like semi-structured data and structured data. We actually already have a ton of data connectors on the, on Llama Hub, um, which is our open source, like purely community-driven set of like data connectors. And we're actually working to make all those available within the Llama Cloud platform as well. And so Llama, like Llama Hub, like the open source integration ecosystem has been very powerful because there's actually a lot of really good stuff out there. And a lot of that has been like community contributed. But anyway, so that's step 1. So like connect and load your data. 2 is actually having proper ways of parsing your data. and this is where specifically LlamaParse comes in. Um, and so parsing your data, um, in this setting really means like you have a PDF. How do I actually like load and extract the information from this PDF that basically like maintains, uh, kind of like the original structure of the document and also maintains all the content? And so this includes like, you know, oftentimes when you load in these documents, it's, um, a lot more complex than just like a pure wall of text. You can have like tables, you can have charts, you can have like, uh, complex like spatial layouts, you can have embedded images as well. Um, and so I think about like PowerPoint presentations as well, where it's not really just the text that matters, it's like the set of like objects you put in it and also the spatial positioning of everything. Um, and so if you tried like traditional parsing techniques, they tended to do very poorly on, um, on these, on these types of documents. Like if you just took an open source out of the box solution. So LlamaParse, what we had an insight into is, you know, like just roughly speaking, it's powered by like Trend AI, powered by LLMs. But we have like an insight that we can basically kind of like build a differentiated platform that can really like parse and extract information from these documents in a way that's specifically geared towards getting other LLMs to understand this data. And so LlamaParse itself exists as part of Llama Cloud, but also as a standalone API that you can go ahead and use. Um, and so, uh, after you parse your document, then you want to, you know, like process and, and embed it and do a bunch of transformations before putting it into your, um, your, your, uh, storage system. And that storage system basically represents like an interface that your LLM can interact with. So the, um, storage system is, you know, a vector database and the processing embedding include at a very basic level, like all the chunking strategies that you want to think about when you like split a piece of like process data. And then embedding, of course, is actually generating like a vector representation for each chunk. And so there's just all these different steps, kind of similar to what we talked about at the very beginning, toward making this like type of ETL possible. And we want to make sure that we provide, you know, enterprise developers with the right interface to basically perform this type of data cleaning transformations and retrieval so that it basically saves the developer time in building something that's production quality and can basically abstract away a lot of the data processing complexity. And as a result, the AI engineer within the enterprise can focus on the orchestration logic on top of this data.

All right. Just to be sure that I didn't get lost there, you told me there are 4 steps where Llama Cloud helps developers. You talk about data connectors that live in Llama Hub. You talk about parsing. What were the other two?

Yeah, so the 3 or 4 steps roughly is data loading, data parsing. So data parsing is through Llama Parse, data processing. So being able to transform that data. And then I guess like data storage, which is like putting that data into a vector store. So Llama Cloud, we're not like building our own vector store, but rather we're leveraging the fact that, you know, we connect to 40+ vector stores on the open source. And so we're able to basically process and transform and put that data into a storage system of your choice.

So developers can use pretty much any vector store they want with Llama Cloud. Does that mean that the data is going to be stored within your servers or not?

Um, that's a good question. So from like a systems architecture standpoint, um, we like, it exists, you know, kind of like in a, uh, like an initial state as, um, like a SaaS service, but, uh, we don't store that's like that data at rest, especially like the source data. Uh, we might store some like metadata or annotations, but like we move data from your data source and put it into like the storage system. And, and usually there's like API interfaces to connect to your storage system. For a lot of like larger enterprises, we're offering like kind of BYOC options so that we're actually able to deploy that in your cloud VPC. So then it's co-located with like this, the both like the data sources and also like storage systems that you're using.

So they can have Llama Cloud within their own premises as well. Amazing. Talking about the first step of this process that you mentioned, data connectors. You talk about Llama Hub, which if I understand correctly, it is a community-driven hub for data loaders, right? But yeah, what would you say are your favorite data connectors? If there are so many, are there some that were kind of unexpected that you would say?

Yeah, well, so we have a bunch of fun ones. I think we have one that like, you know, just reads from different types of websites. I think there was one that reads like, I think it was either haiku or something. Anyways, so it was just like some, some like, uh, fun, like fun. Oh, there was, there was like a dad jokes reader. I remember, you know, like reads dad jokes from this website. Um, anyways, uh, so, so that's obviously, you know, less on the useful side and more on just like, oh, there's like some fun community contributed ones. The most useful ones are the, um, document loaders. This is, uh, Microsoft SharePoint. Um, this is like, uh, S3. This is like, uh, like GCS. Um, the, probably the predominant way that people are using LLMs on data is through, um, processing, like, uh, indexing documents. Um, and so usually their documents are stored in some sort of bucket somewhere. Um, right, so this also includes like Google Drive, um, Azure Blob Store, uh, like basically all these different types of, uh, document store solutions. The next most popular ones are probably like the common software systems that, that we use, um, and this is probably more common for individuals or startups, um, but like Notion, um, Slack, uh, uh, Discord, those, those are, those are super popular. Um, basically I, and I think the reason for this is just like, you know, if you think about just where is information stored in like an employee or individual's like day-to-day use, um, it's typically in the form of like one, files, two, in conversations, and three, in just like any productivity tool that you use. right? And so that's going to be the most popular data sources.

Awesome. And then the second step that you mentioned is parsing, for which you have LlamaParse. Just to make sure, you mentioned that you can use LlamaParse as an API, so you do not need to be using Llama Cloud to use LlamaParse, right?

Yeah. So this, yeah, so LlamaParse itself exists as a standalone API. It's also natively integrated as part of this overall Llama Cloud platform if you wanted to find your indexing and ingestion workflows. So Llama Parse itself, um, as an API, um, you can upload your document like a PDF and you can get back like a parsed Markdown representation. Um, and what this will give you is it's very good at, yeah, like extracting out like, uh, like tables, um, images, embedded objects from this document. You can actually input— here's a special feature of Llama Parse. You can actually input parsing instructions. So, and like, this is not really the, like, you don't really see this in other parsers, but you can basically input a prompt. You can, like, you can say, you know, please extract out all paragraphs as bullet points, or please extract out, or like, for every piece of text, like, tag it with a page number. You can do that. And I think it will probably work if you try that. And so this is the property of the fact that, you know, LlamaParse is powered by just like generative AI at once.

All right. So, I mean, to be clear, LlamaParse allows you to ingest pretty much any kind of document, be it PDFs, be it slides from a PowerPoint presentation or something like that, and get a structured or a text representation of all the information within your document, right?

Exactly. Yeah, that's a good way of putting it. So it gives you back like a structured representation. And it handles most common types of documents. I think the one it doesn't handle, which I'm pretty open about, is like Excel sheets, just because Excel sheets are like its own beast. And we're still trying to figure out the best way to process it.

Right. And how is that different or better than just performing OCR in your document?

So OCR is like a part of it. But if you think about it, like there's like, um, there's OCR on top of like a scanned kind of like, uh, receipts or image and that type of thing. Um, that's actually part of LlamaParse, but there's other parsing complexities too. So for instance, let's say you have text in like a two-column layout, um, and you also have like headers and footers. Um, basically like we not only perform like kind of basic OCR, we also, um, do some general like algorithms, uh, to basically handle like document like reconstruction and processing, um, to make sure that like the spatial layouts are preserved and that we're able to tag it with like the right sections. Um, the other thing that we do is like, um, like for instance, for like tables, right, which is still represented as like text, um, there's like some special care that needs to be taken to basically make sure that the tables are kind of aligned so that like, let's say you have a bunch of rows and you have like a set of columns. You want to make sure that all the numbers in a column are preserved. Because oftentimes what you see in some of these parsers is the numbers get shifted and they get mapped to a different column or something. And that leads to obviously hallucinations because then the LLM doesn't understand what's going on. And what we found, I think an interesting thing is if you can somehow solve this problem and you can somehow basically get the text to appear in a nice representation, it helps LLM understanding and actually reduces hallucinations too.

Awesome. And that's, so that is LlamaParse. Basically, it allows you to perform OCR plus a, like a real understanding of the contents of the document, included images and tables, which are surprisingly difficult to parse when you're not using a system like this.

100%. And just a brief note on like, why are we doing this? The goal wasn't just to create like a nice PDF parser. I mean, it's pretty good at PDF parsing, like to be totally clear, it's actually very, very good. And that's why I think a lot of people are interested. Um, the overall goal is to make sure that we can tie this into this overall story of giving you like proper like RAG or like context, uh, without hallucinations, um, for, with your LLM. So like our goal is not to build in like all the features that a traditional PDF parser might have. Our goal is to build in specifically the right techniques so that we can basically represent your data in the right way so that LLMs can understand it and are accurate in synthesizing information over it.

Awesome. And yeah, it goes back to what we mentioned at the beginning, using LLMs also during the parsing process, which is kind of what you were trying to do from the beginning. Yeah.

So that, that, That's a good point. So actually you can have LLMs as part of this parsing process. You can have LLMs as part of the transformation process, like the chunking, the metadata extraction, all these things. Like extracting out structured outputs. And then all this is in the spirit of actually feeding this into some more LLMs that go and do RAG over your data and perform like agentic workflows over your data too. So yeah, you can have LLMs throughout this entire process. And it's very interesting to see where they can be applied and how they can all work together. Awesome.

All right. And is— okay. And so Llama, just to be clear, LlamaParse is already open to the public and people start using it now or not yet?

Yeah. So LlamaParse is available, like basically to the public. You can actually sign up for like, there's like a free plan as well as like a paid plan. We have a pretty generous allocation. You got like 1,000 pages a day. just for free. And then there's some, like, usage-based pricing. Um, and then Llama Cloud is, uh, kind of in, like, a private preview mode. Um, we're hoping to open it up a little bit more, um, very soon. But in the meantime, if you're, like, an enterprise that's interested in building, like, performant RAG in a team-based way, and you want to, you know, like, save engineering time on, on connecting to your data and processing that, come talk to us. Um, and yeah, uh, very open to these types of conversations.

Awesome. And okay, so we covered the first two steps that you mentioned, the processing part. Can you shed a little bit more light on that? How do you use, how is the processing part smart or smarter than something that you could code yourself?

So I think for processing, I think the interesting thing about processing, I mean, it's pretty related to, some of it's actually related to parsing. Um, it's also related to some of the stuff I mentioned before, which is like, how do you just extract out structured information from unstructured data? Um, and I think like metadata extraction, um, I mean, I guess that's an instance of the structured extraction, but roughly speaking, if you're able to do that, then you're able to attach or augment the existing unstructured data with the right tags, which then allow you to better, um, search and query and filter for this type of information. Because instead of just doing semantic search by embeddings, you can actually filter by like a SQL statement, or, you know, you're able to filter by the specific tag or topic. And so I think the way you can use LLMs to do like structured data extraction is you can basically, at a very basic level, feed in text into the input prompt window of the LLM and ask it to generate like the set of like structured information that you want. And so if you're able to do this scalably and reliably, over like a large corpus of unstructured documents, then you're able to almost like generate structure from these documents. Um, and so the different types of structure include, uh, like I mentioned, like kind of simple annotations. They could include summaries. They could also include relationships between documents. Um, this is where you start getting into like knowledge graph territory, where you actually not only look at kind of like the information within a document, we also look at how different concepts, um, or, or I guess, like, um, yeah, how different con— how the same concept across, uh, um, uh, different documents and how you, like, tie different documents under the same concept. And, um, if you're able to do that, then you're basically able to kind of construct this overall, like, structure or ontology over your, uh, document set. Um, and of course you can and probably should use LLMs to do that, um, after a certain point, because to do this, uh, purely by hand is, like, super time-consuming. And then, and then once you have that structure, you can leverage like advanced retrieval index, like retrieval capabilities over that.

And in your testing, I mean, I suppose that generating this smart metadata and associating it to your notes or to your documents in your vector store, that improves RAG by a lot, right? I suppose.

Yeah. So, this is on a very like This is a little bit technical, but there's a few ways it can actually improve. Generating metadata directly plugs into metadata filters in a lot of— that's implemented in a lot of vector DBs. And so what metadata filters allow you to do is basically filter by like structured tags in addition to just like top-k semantic search. This gives you more precise retrieval results. For instance, if you specifically know that, you know, you want to search within a set by this topic or by this date, or, you know, like, uh, lower than this value, um, you can basically implement, like, do that. And then you're guaranteed to only have, uh, results that satisfy those constraints that you set up. Um, so, um, this leads to more precise retrieval. The, the other two pieces is, um, having tags actually helps your embedding representation a little bit. So for instance, let's say you feed in a piece of text to an embedding model. You could feed in just the text itself. If you add in some additional context from like, you know, the overall summary of the document or like surrounding annotations, like the, the name of the file or the page number or those types of things, um, that can actually help your embedding representation. And really good embedding models can take that information to so that, um, when you ask a question relevant to the stock, it's more likely to be retrieved if you had this metadata than if you didn't. Via semantic search.

And then finally— sorry, yeah, go on.

Oh, sorry, no, I just mean, finally, it's a similar thing. This also helps with adding the right context when you feed this into an LLM. Because an LLM, let's say you didn't have a summary of this document, would just have less context on what's actually going on. And so generally speaking, the more context, or you can basically ground the LLM in with more information. So that it's more likely to generate a result for you that's what you want and doesn't hallucinate.

Awesome. Is it, does it not explode the costs of ingesting your data? How do you deal with that if you have a lot of data in a company, for example?

Yeah, I mean, I think you bring up a good point. I think, well, so metadata itself can be both like LLM and human generated. And so if you have like too much metadata, that's also a problem because then like every piece of text, every document, just has like a lot of metadata, and, and that can actually degrade performance in many ways. I mean, both accuracy and also cost and latency. Um, so it is a trade-off that people have to consider. Um, the other piece is like, yeah, if you use LLMs, um, always to basically just like kind of extract out like structured metadata from all your documents, that is going to cost a bit of money, especially if you have a lot of documents and you're using very powerful models. I think my hope is that one, the model, that, well, as context windows get longer and as the token costs come down and as models get faster, hopefully this problem does, like, get alleviated a little bit over time. And then the second piece is that you can do some basic optimizations so that you don't have to fit, like, the entire text in the context window. If you're trying to do structured data extraction. You can still do like vector search retrieval to basically do structured data extraction. And the trick is you just got to make sure you find the right bits of text. You know, basically you like pre-index the text and then retrieve just the right bits. This is very— it was basically a RAG pipeline, right? To just like extract out like structured information as part of that indexing or the part of that like processing.

Awesome. It really sounds like an amazing service, the entire Llama Cloud suite that you have. By the way, I also had a question about the embedding models that you're using over there. Are the developers capable of choosing the embedding models that they're going to be using in Llama Cloud, or do you have your own embedding models that would make it more costly to migrate later on?

Oh, that's a good question. I think right now we integrate with roughly off-the-shelf embedding models. I think the idea of having custom or fine-tuned embedding models is a very interesting one. I think, yeah, I don't know about training one from scratch. That one's interesting to consider. I think the thing that I'm interested in is if you have an eval data store, have human feedback over time, you could potentially have a fine-tuning layer on top of your embedding model. Awesome.

I don't think we talk about it enough, but I mean, the part of passing instructions to your parsing model, that's just amazing. I suppose you're the only ones who do that, right?

I think we're the only ones that do it right now, though, you know, like others might do it. It's a cool, it's a cool nifty feature. I think if you're a listener, I definitely encourage you to check that out and yeah, give us feedback. You know, we're always kind of hammering out like all the long tail, like kind of nits and bugs and those types of things in the parser. And so we're very receptive to just like helping to sharing your feedback and improving this even more.

That's amazing. Amazing. Also, I mean, talking about that as well, do you have any more services or products that you're thinking of bringing, or is that your main focus for now?

That's a good question. I think that's the main focus for now. I think the two main goals for this year is basically One, expand the open source, and I can talk a little bit about that. And then two, like expand Llama Cloud. And so 50% on each one. And, you know, we basically have our team kind of like the most— a lot of the team works on both. But roughly speaking, you know, they work like the overall allocation is like 50-50. In terms of the open source, it's basically about continuing to make everything production ready, robust. And basically evolve with any new use cases that are emerging. And so being able to stay on top of like all the latest LLM advances, integrating with these models, building out these use cases, and then from an enterprise side is basically kind of— it adds quite a bit more layers of complexity, like adding in really the product maturity to make this thing a scalable, reliable service that's like enterprise grade.

Awesome. And about the open source part, open source projects sometimes tend to be monopolized by super big companies, right? Do you see, I mean, how do you see the future of this open source, sorry, not models, this open source frameworks? Because do you see Google or Meta jumping in into the race with this? Like they did, for example, with React, or how do you see the future of this industry?

That's a good question. I actually don't believe as strongly in that. And part of the reason, I mean, it's also always possible they might, like, for instance, Microsoft already has a few frameworks. They have Semantic Kernel, they have AutoGen. And, you know, like some of them are actually doing pretty well. Like I think AutoGen has a decent number of downloads. I think the main thing is like, first of all, all these companies are creating their own models. And part of the value prop of being a framework is that you're completely agnostic to the model. And I think there's just always going to be a little bit of an adverse incentive so that they're less community-friendly than literally a startup that doesn't care what model you use. And we actually just want to integrate with everybody. And so I think we take that perspective for basically all the models, all the vector stores, kind of like the set of integrations, like all the data sources, for instance. We want to make sure that you have access to all the data that you want. And so it's just, I think, a little bit harder for a larger company to be able to kind of like build that type of ecosystem when they specifically have like one type of model or vector store or something that they want to push.

Yeah, because they're going to be biased towards their own products. Yeah. Right. Yeah. I mean, that's some pretty interesting insight about that. Let's talk a little bit as well about what you see in the future of the industry and some advice that you could give, some other advice that you can give to beginners in this space. Let's say, for example, that you were starting off today, like you're not a machine learning specialist, you're a software engineer and you want to start working in this industry. What would you do first?

Yeah, so I mean, not to like hyperdoc— like our documentation has like a nice sequence where basically it's I would start off with the basics, just like do the prototype first, do the thing that takes like 5 lines of code, just so that you at least see that this is like the overall use case that you're building towards and eventually see why it's interesting. And then once you start with the basics, then you start going on like a little bit more of a guided tour through some of the components, let's say to build a RAG system. I would try to like As you start at the surface layer, start to go a little bit deeper into each of the different components, really understand a little bit more about how they work. And then to build in some like development principles around like iteration and evaluation. I think the, maybe the difference between AI engineering and software engineering is that I think in software engineering, you know, when you want to like test a piece of code, you're very used to like writing unit tests or integration tests. And that generally speaking, you know, like the func— the, the, the types of logic you're writing are not super complex. And so it's pretty easy to reason about like the overall behavior, uh, or at least like what the outputs, uh, inputs, outputs should be. I think the difference with ML is that everything requires some sort of dataset because the function space is like super complex. Like if you put something into an LLM, you really don't know what the output is gonna be. Yeah. And so you need to make sure you have the proper benchmarks and ways to like test these stochastic systems. and so like, having like some sort of eval approach, like we have eval modules, you know, in LlamaIndex, we integrate with like a ton of different eval providers that'll provide like really nice services around like tracking these metrics and experiments. Um, having that is pretty important. Um, and then like from there, basically work your way up towards like scaling to more data sources. Like if you want to scale to more like, uh, documents, for instance, like you can use like Llama Parser, you can use like anything that you want really. And basically, like, start to add in more layers of complexity with, like, that kind of eval benchmark fixed so that you can make sure you're not, like, introducing any regressions in performance. And then as you start with the basics, I think, I think RAG is probably the thing you should probably learn about first, just because I think it's, it's easier to learn than agents. Agents can get really complicated. And, and, like, even I'm, like, confused sometimes about, like, What exactly am I supposed to be doing in this specific module? And there's also high-level agents, there's low-level agents, there's multi-agents. I would start off with RAG. RAG is a lot more simple to digest. And it's not simple either. You can go in and go into a rabbit hole, but at least it's easier. Yeah, but you learn that piece first and then you start adding on some ingredients or layers like memory, the ability to do multi-prompt pipelines, like chains, all these types of things to basically reason over things. And then you start getting a feel for how to build more complex systems.

Right. And about agents in particular, right? So that's, I've heard a lot more people talk about agents recently than let's say when GPT-4 came out or when GPT-3 came out, right? How do you see the future of agentic behavior in this sphere? Do you like, what is the future that you see with agents even though it's pretty early on in the sphere?

Yeah, I think, I mean, it's super exciting. I mean, I think like Andrew Yang, for instance, been tweeting about it. I think like people have been building it and it started to work. I think people started like building stuff last year with agents, but at the time it just didn't really work super well. And I think right now it still doesn't quite work super well, but at least like it's starting to kind of work. So examples like all the autonomous coding assistants, like, you know, GPT Pilot, Devin, SWE-Agent, I'm forgetting some of the names, like OpenDevin. Like some of the projects are popping up to basically be like a worker for you that can operate independently and autonomously as opposed to just like you chatting with it back and forth. And I think like people like things that are cool, that are really cool. And this is an example of something that's really cool. And then once they've afterwards they want to learn how to build it. I think there's a revived interest in how do you actually build scalable, reliable agents. I'm really excited about it. I think we, well, LlamaIndex on the open source side, right? I know we talked a bit about LlamaCloud. We're making a big push towards agents, just better agent education, abstractions, materials, resources. I think there's going to be like two ways that people build agents. One is if you're a developer that doesn't really care as much, let's say you're just like less technical or you don't like really need to understand the fundamentals, there's going to be high-level interfaces that you're going to be able to create agents. And I think like, for instance, this isn't like quite what I'm like, like, like an example of this is like AutoGenu Crew AI, actually, like if you take a look at the interface, I think it's like, You can just define the goals and kind of like the overall task you want to solve, create the network of agents that you want, and then it's roughly equivalent to pressing a button. You just press go and then let them figure it out themselves. And to some extent, if you go even higher level than that, it feels even less than programming because it's kind of like you can do this through a low-code UI, basically, right? Just define the set of things you want to do. And then just have them figure it out. And then like on the very like kind of other end of the spectrum, you have people basically building agents from scratch or kind of like writing in all the logic, like all the kind of like custom like prompt flows, like all these like decision points. And I think that's typically for people that are more advanced and really want to like have like, have control over like what this thing can do. But of course, it takes more time. But on the flip side, you have a lot more control over the logic. And I think like there's gonna be a funnel of people going from like building these like high-level things that can just like go off and do stuff to like kind of more advanced users that can really customize it to their use case.

That'll probably make it easier to implement them in production, right? 'Cause I'm not sure if we're going to get to a moment where we can deploy to production things like these orchestration systems for agents where pretty much they just figure things out by themselves. It's, I'm not sure how to, I mean, yeah, how easy it is to get to production with those.

I mean, I think that's an open question mark. I think, um, I think everyone's like scared about it right now. Um, because, you know, obviously there's obvious concerns like, okay, well, obviously you have like no proper guardrails. You don't really have a way to inject control, like those types of things. But assuming you were able to bake in some of those interfaces, almost like out of the box or by default, I could imagine like if you just have a very opinionated abstraction that somehow guarantees some degree of reliability, then people would be willing to use it. Awesome.

Yeah.

And that, have you seen some, that kind of systems being deployed in production already, like agent swarms or not yet?

Not really. Um, however, I think a lot of people are interested in it. Yeah. I think, I think, you know, there's all these like startups popping up. But yeah.

Yeah. And of course you can build agents with LlamaIndex.

Yeah. And we're making that part. So, um, yeah, we're making that part a lot more, uh, comprehensive and robust. And specifically, I think the thing that I think oftentimes people don't realize, you can build like your own custom agents in LlamaIndex. You can basically actually do whatever you want. You can use out of the box, uh, abstractions that we have. We have like function calling, like some cool new agent implementations, and of course React. The one thing we don't quite have right now, which is coming out hopefully relatively soon, is like multi-agent stuff. So being able to have like good ways of setting up like multi-agent communication, like being able to design like a, as you said, like a swarm of different things. And I think that's something we're working on. Hopefully, hopefully we have like an exciting release, you know. Awesome. Awesome.

Amazing. And to finish, I also wanted to ask you, is there any way or opportunities that people can contribute with LlamaIndex to make it grow and to maybe give you feedback? How does that work?

Yeah, I appreciate you bringing that up. I think we love contributions. We love the community. I think we're always open to more types of contributions. I think there's two different types of contributions. I think the most, uh, a very, um, classic one is actually Llama Hub. So Llama Hub is not just like data loaders, it's our entire ecosystem of any type of integration. And so this includes LLMs, vector stores, embeddings, agents, uh, like different templates that you want to contribute. We're always looking for new types of like packs or templates or new types of integrations. Um, so this is like a pretty easy contribution interface. Everything is version tracked, and so if you, uh, contribute something you can pin the dependencies to make sure it doesn't break later on. And so we spent a lot of work trying to make that really good. And so Llama Hub would be a great place for contributions. And then the other is, I think as people go through the project, they submit like kind of like bug fixes, like, and that's like documentation fixes. All these things are obviously like very welcome as well.

And one last question. If you were starting off in the industry right now and you wanted to create a company, What kind of companies would you think of creating?

Yeah, I, um, probably, I mean, probably LLM related, I think. Yeah, no, it's, it's interesting. I, I, um, I'm trying to think like, to be realistic, I think there are going to be certain, you can start any company you want. It's just like certain companies I think are going to be a little bit harder to start than others right now. Um, I think like, uh, if you start a foundation model company, I think that's going to be a little tricky. Unless you have a differentiated view on foundation models. Maybe it's super domain-specific, for instance. I think infrastructure, I think there's certainly opportunities there. I think certain parts are maybe a little bit more saturated than others. I think evals, for instance, and observability is a little bit more saturated to be totally honest, because there's a lot of these types of companies. I think like if you were to build something there, you'd probably need to build something more domain specific. And then actually, I think this is probably the general trend. And then I think you go into the application layer, which is inherently like a little bit more maybe like, kind of like domain specific. And you can, you know, like, you can like try to figure out how to build like general like, like applications that really solve end use cases very well. I think if I were to build a company today, I probably, and, and, um, okay, actually it's not necessarily something I would build. It's just like an idea. Um, I probably think about building actually an agent that actually works really reliably that can solve like a critical business workflow really well. Um, I think like, I think similar to how Devin, I think had like a really cool demo for like, you know, software engineering, which to be honest, not really hard problem. If you just built like kind of like an agent that can go ahead and actually like perform like a sales task really well. Or for instance, like perform like kind of ops, like assistant or even legal work really well. I think that's something that I'd be really excited about.

Yeah, that would, I mean, so many things that are probably going to revolutionize every single industry on earth right now. All right. Well, I don't know if you have any other comment that you would like to make, something that we forgot to mention.

No, I mean, Alejandro, thanks so much for giving me the opportunity to and asking all these questions. I think we've covered both the open source as well as some of the enterprise aspects. So LlamaIndex, the open source, broad orchestration framework, always improving, welcome to contributions. And then LlamaCloud is the enterprise data platform for data processing and ingestion. And then, oh, last bit is, I mean, if you're interested in AI engineering and those types of things, we do have like a careers forum. So feel free to just put your name down.

Awesome.

So, uh, just a quick call out there, but no, thanks for, thanks for having me on. I think this is a fun chat and for me, it's been amazing.

It's been amazing. And thank you for, for all your detailed explanations of everything that I, that I asked you about. Sometimes I was a little redundant probably at some questions, but to, to make it as clear as possible for, for our listeners and for me.

100%. Yeah, no, I appreciate you asking, doing that, because I think sometimes I just get in this mode of assuming that people already have context into things, and I think it's good to take a step back and really go dive into building stuff from the ground up.

Amazing. Well, thank you very much, Jerry Liu. It has been an absolute pleasure talking to you.

Pleasure talking to you as well.