Building Production-Grade LLM Apps

Sam hey everyone, my name is Diana.

Chan Morgan and I run all things community here@deeplearning AI. Today we have a very special session to talk about building production grade LLM apps. This session will be recorded and the slides notebook can be found in the video description. In this workshop we will have very special technical leaders from Pinecone and Chura that will share how to create apps that are high quality and production ready. The principles will be illustrated with Pinecone, Canopy and True Lens. In this workshop you'll be able to learn what you need to create enterprise ready AI models and apps, what you need to evaluate iterate on apps quickly so you can gain confidence that your app is ready for production and rapid response. What happens when an app generates a hallucinatory or irrelevant or unsafe response? How to monitor recognize when your app is in trouble and quickly get it back on the rails. For our event partners today we have Pinecone and Trara. This workshop is actually inspired by two of our short courses that we released with these partners that we will link in the chat below. For Pinecone, we had built out building applications with vector databases and with Truer, we had built out building and evaluating advanced RAG applications. For more information about Pinecone, Pinecone created the Vector Database which acts as a long term memory for AI models and is a core infrastructure component for AI powered applications. The managed service lets engineers build fast and scalable applications that use embeddings from AI models and get them into production sooner. For Trura, TR is the leader in AI observability helping companies to monitor, debug and test their AI apps in order to drive performance quality and trustworthiness. Powered by the Enterprise Class Artificial Intelligence Explainability technology based on over nine years of research Trailer is able to facilitate faster, more accurate ML model monitoring and analysis and debugging better than any other vendor. And now for our speakers. So I'd love to introduce Amnon Khattab. Hey Amnon.

Hey, thanks for having me here.

Absolutely super excited to dive in for some more background on Amnon. He is a machine learning engineer on the Gen AI team at Pinecone. He is a core contributor to Canopy Pinecone's open source RAG framework and focuses on the evaluation of RAG systems. Before joining Pinecone, he was an Applied Data Scientist at Microsoft. He holds an Ms. Degree in CS from Tel Aviv University where his research focused on explainability of AI. And for our second speaker we have Anuppam Dada. Hey Anupam.

Hi. Hi. Great to be Here again.

Absolutely. Glad to have you back. Anupam is the co founder, president and Chief Scientist of Truara. He was on the faculty at Carnegie Mellon University from 2007 to 2022, most recently as professor of Electrical and Computer Engineering and Computer Science at cmu. Dada's research focuses on trustworthy AI spanning evaluation, explainability, fairness and adversarial robustness of ML models and Genai applications. He obtained his PhD and Ms. Degrees from Stanford University and a B Tech from IIT Kharagpur, all in Computer science. So happy to have you guys here today. I was very excited to dive in. I know that our community was really enthusiastic about learning so many topics about building production grade LLM maps. So why don't you guys take it away?

Yeah, thanks a lot Diana and let me share my screen and we can get started. Great.

I think we're still. Oh, here we go.

Great. 2023 was the year of experimentation and prototyping. 2024 is going to be the year to move all of these cool apps and a good chunk of them into production. And so one of the big challenges facing the field today is how do you build production grade LLM maps? And that's what Amnon and I will talk about today, with a particular focus on retrieval augmented generation. So here's the structure. We'll kick off with a very quick intro, but dive in pretty quickly into what's involved in building production grade apps. And Amnon will talk about Canape and Pinecone and some of the cool work that he's been doing with his team. We will talk about how to evaluate and iterate on applications to improve their quality to get it to that point where it's ready to move into production. For that I'll talk about how True Lens challenges in building production grade rags. We also have a sample notebook for you to try hands on. You can take a look at Canape and True Lens. If you want to scan these QR codes, you can go to our githubs and the developer teams are collecting stars, so please be generous. So with that, I'm going to hand it back to Amnon to kick off the introduction.

Yeah, okay. So thanks Anupam for the great introduction. I'm really excited to be here and discuss this important topic. Let me start with a very simple example for one of the first issues you usually encounter when you start working with LLMs. For example, if you ask JGPT, what is the new name given to Twitter by Elon Musk, you're probably going to Answer something like, my knowledge cutoff is January 2022. I never heard about any name change from Twitter, which we all know it's not true. It changed to be called X. And when you build applications with LLMs, you usually want your LLM to know about the fresh data and be updated with what happens in the world. And taking a step back here, what actually happens is that models, all of their knowledge is only what they saw during training. And when you use OpenAI models or even open source models, usually they only use some web data from some point in time. They retrain the model once in a year or a few months. It's reasonable to say that the knowledge LLM have is only stale web data and they only have access to stale web data. What they don't have is access to fresh web data like in the example we just saw. And also they don't have access to propriety data. And when you create a production level application, you do want access to these two, especially for proprietary data. Because in a lot of applications, imagine you want your LLM to talk about the documents of your organization or even on your Slack channel and things like that. And obviously when OpenAI and open source teams work on training a language model, they can't use your own data. And so what we want to do, we want to connect these models to fresh web data and propriety data. And I guess it's like kind of makes sense. And the way to do that, and this is like the leading way right now to do it in the state of the art, is to do rag. I assume that most, at least most of the audience here already heard about rag, so we do a very quick high level overview of what is ragged. The main idea flag, is that instead of just giving to your model in the prompt the user question or query, you want to combine it with some relevant data from your proprietary data or fresh web data that is relevant to the query and combine it with the query, make a prompt that say this is the query, this is some pieces of relevant information and ask the model to generate a knowledgeable answer. And a lot of time when you do that, you have like a lot of data we're talking about. We have a lot of customers having like millions of documents, maybe hundreds of millions of documents. We have even few customers have billions of documents. And obviously you can't fit all of this knowledge into the prompt. So what you want to do is to use search like you search in Google and you can just always read all the Internet for every question you use search. When you do search on your own data right now, what you usually do for RAG is to use vector databases like Pinecone. The idea of I will try to give a very simple explanation of why and how you use vector databases. So to do what we call a semantic search, which is something that helps you search over unstructured data like PDFs or files, you need two things. You need vector database, you need embeddings. The role of embeddings is like a kind of model that was trained to represent your data as a vector and the way that you take your data. You present for example, your document as vectors and your question as vector. And the model was trained to make these vectors closer as much as like. As long as the query is more, the document is more relevant to the query. What you do, usually you take your data, you put it in this embedding model, you get a vector and then you store it in a vector database. Then once a query comes, you take the same embedding model or very similar embedding model, you run it through this embedding process, you get another vector and you query the vector database with it. The role of the vector database here is to store all this vast amount of embeddings. And once the query come, you need to find the closest vectors to this query so it will find the most relevant documents to your query. But if it wasn't very clear for you right now, don't worry about it. This is not the main point in this webinar and the main point is that we want to do search over the data. Once the user query comes, we just do search. We take, let's say 10 most relevant documents concatenated with the query, put it to the model and then we can get knowledgeable answers that we can provide our users and solve the issue we just discussed. Luckily, building a demo or constructing a POC for RAG is very easy. There are a lot of framework like LangChain or Llama Index and Canopy here. This is a. I provide a very simple example of how like you can use like, I don't know, five commands in PY in Python and you get a RAG system working. So I think it's fair to say that building a demo is very easy. Like in a lot of scenarios, problem starts when you go to production and we see it with a lot of customers. And like Anupam said, this is the year of production where we really encounter the actual issues with working with LLMs and building LLM production grade applications. And there are many problems. I just Gonna describe some of them and then we're gonna talk about how you can address them. So one of the most famous and common one is hallucination, meaning that sometimes your model might just invent false information. And this is not something you want in application because you don't want an application that give your users non functional information. And another problem is context overflow. Most of the models like GPT have some limits for how much tokens you can feed in the prompt. For example GPT 3.5 have a limit of 4K tokens or walls. And when you do rag a lot of time you overflow this context because doing search require like might produce a lot of tokens. Imagine you do give me the top 10 most relevant documents and all of them are very long. You might end up with 10k words or tokens. And then when you try to feed it into the model, you're just going to get an error saying you overflow the prompt or the context. So this is the context overflow problem and other problem that are kind of harder is how you to understand how you can correctly ingest the data and clean it. So search will work better how you chunk it or split it to smaller parts that you can feed into the embedding models. How you choose the right embedding model that works best for your data. And overall, doing a high quality search system is very hard. And there are a lot of things you need to tune to make it. And like Anupam said, this is something everyone, all the community is working really hard to do. And now we're going to start talking about how you can address these issues using both canopy and tool lens together. We're going to start with the canopy. What is canopy? Canopy is an open source project written in Python. It was built by Pinecone and the community and it's only focused about drag. And when we built Canopy we really focused about how we can build a framework that is suitable for a production level. Meaning that we will see it a bit. We're going to see a list of it soon. But the idea is that we thought about edge cases. We have a lot of tests there make a very solid CI that makes sure that all the code they placed together well. And we think about all the things you need to take care of in production. This is like a short list of things you get when you use Canopy. First of all you get an application that help you to do end to end drag. It's very easy in a few. This is a short list of things you get when you use Canopy first of all, you get an application that help you to do end to end rag. It's very easy in a few command lines in PyCon. In Python you can get a working end to end rag. It's built for scale using PyCon. Like for example, a few months ago I had a chance to build an application with Canopy on billion documents and it worked very well thanks to PyCon. That really excels when it comes to scale or putting a lot of data in your database. It also automatically handles the context overflow issue. What we do there is that let's say you ask for the 10 most relevant documents. We make sure that it never overflow the prompt. We know how to truncate the tokens and leave only the most relevant tokens such that you won't get this kind of error. This is something you get for free here. Also we built Canopy. We allow Canopy to be built from a config file. We're going to see it soon. It's like a no code way to build Canopy, meaning that you just provide a YAML config and you get a working canopy and you can change it without changing code. This is important for production because a lot of time, maybe there is some issue in production, you want to do a small change and you don't want to change code, so you just change the config. Maybe you have different customers that need a different version of your system. The best practice is to use configuration, not have a different code base for each customer. Another thing we provide with Canopy is that we also provide you a template of first API application within the library. And also we publish often a docker image that you can pull and just publish and have a running system. Here I want to show you a very short example of how we can run Canopy as a service. So if you just use our command line and say something like canopy, start from this configuration YAML, you get a first API application running, maybe on your computer, maybe on a docker, which can help you very quickly go to production with Canopy. And this is an example for a config YAML that actually works with Canopy. This is the default configuration. What you can see here is that you can change or see that system port. Because in a lot of time you see in the library that work with LLMs, it's very hard to find where is the prompt. It is very easy, you just go to the config file. You can change it without changing the code. For example, here in the red box, what you see is that we tell canopy to use GPT 3.5 Turbo. If you want to change your GPT 4 it's very easy. You just change the spring deploy again and that's it. You can also change to other providers like here, or even use Azure, OpenAI. And all of this you can do just from configuration, which is very easy to use. But before I go to actually show you a notebook that demonstrates how to build with Canopy, I want to talk a bit about evaluation because it's very important. What we see very often for our customers that they're really struggling with, evaluation is very important today for them. First, why evaluation is important because imagine you never evaluate your system. It means that you don't have any way to know if it's good or bad. You have no way to know how to improve it. So usually what happens is that over time you just gonna end up with a system that just don't match your users need and maybe give bad quality results for your users. But the thing is that evaluation is very hard because imagine the difference between evaluating a multi choice exam and evaluating an essay. Evaluating an essay is much harder. And this is kind of what you need to do it with LLMs because they are just talking with you. You need to evaluate this answer which is unstructured, you don't have use even if you have ground rules. It's very hard to compare it like an essay. So I think evaluation is very hard and this is where tolerance comes in and can really help you. And I will let Anupam add some color here.

Yeah, and that was a nice analogy there on evaluating an essay versus evaluating a multiple choice question. And that's the first bullet point on this slide gets into that, that it's hard to even define what evaluation metrics to use. That you can programmatically evaluate an essay or the generations coming out of a large language model or rag application and also making these evaluations reliable and comprehensive. So later we'll talk about how to address this challenge. Another piece of it is logging the app execution trace as the application executes, making sure that you have a very fine grained logging of that process so that the prompts, the responses, the intermediate results are all logged at scale that later on you can use those or parts of those execution trace as part of the evaluation and monitoring. The third piece is around scalably and cost efficiently evaluating and monitoring apps in production. As the data volume goes up, the workloads go up, that becomes another big problem. You want to start in development as the developers are building the application, you Want to be able to evaluate them, debug, improve them, but then continuing that onto production where the data volumes can be a lot larger. Doing so in a way that's cost efficient and low latency tends to be a hard challenge as well. Then finally, fast iteration to improve applications. And this is something we will get into with the notebook in a little bit. You have a first version of the app, it's not doing so well with respect to your evaluations. How do you quickly iterate on it? And that speed of iteration becomes extremely important as people are starting to roll out lots and lots of gen AI apps. These are some of the top level challenges that we will start getting into now, how to address.

Thanks Anton. I really agree with all the points. I think that if you get it right and have all these bullets solved for you and also you use a scalable system for rag, you really can get a high quality system. But now let's just switch to see like how you like. Let me do go over the agenda. Like until now we talked about like we made some introduction, we talked about the problems and a bit about Canopy. Now we're going to show you how you can build Rag with the Canopy and Pinecone on a notebook. Then we're going to continue talk about more about Tool Lens and show how you can combine all of it together. So this is the notebook. It's going to be available here in the YouTube channel for you to run. But now we're going to do a very quick walkthrough on it. And here I just going to focus on the section where we build our initial or default Canopy system. First of all we need to install our dependencies. We install Canopy, we install Tool Lens, we install some other stuff we need for this specific notebook. Here we need to provide some API keys. For example a Pinecone API key OpenAI API key specifically. Here we need also Cohere API key. It's not like always you need all of them, but here we need all of them. Here we need to provide a spec for Pycon. It's like defining which environment you run. You can configure maybe different cloud providers. It's very important in some cases. In this example I make it simple and just put the spec for a free trial for Pinecone. And here you have a link where you can understand how you can use another environment. Here we just download public data that we scrape and we explain what is this data because we're going to work within through this notebook. The data is simply scraping of Pinecone docs, which is like the documentation of Pinecone. And this is for example, a scraping of the limits page which explains you what are the limits. When you work with Pinecone, for example, you can see that the maximum vector dimensionality is 20k or that the maximum top k value we can put in queries is 10k. We're going to try to remember these numbers because we're going to work with them soon. But let's continue. This was like just the data. What we do now we're going to construct our Canopy system. First of all, we need to build a tokenizer. Tokenizer is something that just take strings and you know how to separate them to a token token, which is the units that LLMs can read. And the main reason that we need the tokenizer in Pinecone is what we talked about like in the context overflow issue. Canopy need a way to know how to count the tokens in the same way that your LLM count the tokens so it will make sure it's not overflow your prompt. So what you do here is you need to make sure that you build the tokenizer that is compatible with your LLM here. What we're going to do, we're going to take this data, we just show and upload it to Pinecone or to Canopy. For this we have the concept in Canopy called Knowledge Base. This is the thing that actually manage your data or your knowledge. And it's built on top of a Pinecone index that we will call Pinecone Docs. In this example, first of all, we build in it this instance of Knowledge base and we create a underlying Canopy index or Pinecone Index. Here we connect to this knowledge base and make sure that everything is set up. The API key and the index itself. And here we actually upload or upset these documents. What we do in the knowledge base is we orchestrate also the embedding here. So Knowledge base is talking in text. As you can see, I just built a document object that holds the text from our data. And here I just upsell this data to this knowledge base and it's stored in Pycon using Canopy. Now we have all the data in Canopy and now we want to start chatting with it or build our chatbot. First of all, we need to build something we call a context engine. The idea is that Knowledge base is only something that you can query and get Top K resource for top 10. Context Engine is like a layer above the knowledge base which talks. Once it gets a query, it's actually returning a piece of text. This is the thing that makes sure that your context is not in overflow. You can give it a budget. You say, I want for this query to get 1000 tokens and it makes sure you get up to 1000 tokens. This is the main idea of Context Engine and this is maybe the most interesting part. This is the chat engine, this is the thing that you're going to talk with, and this is the main orchestrator. And a very interesting feature about Canopy is that we made the chat engine to be fully compatible with OpenAI API, meaning that the input and the output of OpenAI, of the chat engine, of Canopy is the same as OpenAI. So, for example, here we construct a very standard history chat, like you will do in OpenAI, and we're going to ask our chatbot what is the maximum top scale for a query to Pinecone. All of you that used to work with OpenAI, you're going to observe that it's like the same output here in OpenAI. Here we're just going to call our chat engine with the chat call, provide the chat history and get an answer. And the answer is that the maximum value for top K in Pinecon is 10K. And for those of you who follow, this was the right answer. So at least for now we are happy. And now gonna take it back to Anupam who will explain how we do evaluation here and how we can connect tolens with Canopy.

Great. Yeah, thanks. Thanks, Amnon. Let me share my screen again and I'll touch on a couple of more general conceptual points about evaluation and iteration and then circle back to the same notebook to see that in action with the rag that Amnon just presented with Canape. The cool thing about Canape here is that it has this nice separation between the configuration of the app from the code. You can change a lot of hyper parameters and so on in the config file, in the YAML file without having to touch the code, which is just makes it a lot more maintainable in production. I really like that aspect. Let's get into the evaluation and iteration and this is a quick view into the overall how we see the world in terms of evaluation and iteration. As developers are building these applications like the rag that Amnon just presented, you can use our open source library called True Lens to evaluate and debug these applications to get that to that high quality that's ready for production. This fast iteration loop is very valuable and we'll see that in action shortly on the notebook. Then Once it moves into production, there's also production monitoring. As we think about a general framework for evaluation, here are some of the key pillars that I want you to take away. One is ensuring that our evaluations are comprehensive. They're quite broad and deep, that it's extensible, meaning that developers can add. There's a whole bunch that's available out of the box. But it's also easy to add your own evaluations, to extend the set of evaluations based on your use case, based on your application, and then the reliability of the evaluation themselves. Experiment.

Sorry, just to interrupt. We don't have the slides, so if we could share them.

Oh, sorry, we'll just add. Sure, one more click. Sorry, is it visible now?

Yes, absolutely. Sorry for the interruption.

Yeah, no, thank you. I was flying blind a little bit there. So this is the workflow that I was discussing around evaluation debugging and monitoring. The evaluation and debugging pieces of are available in open source through True Lens. It's a very permissive MIT license. Feel free to download and play with it. As you're using the notebook, you'll get a quick intro, hands on intro as well. And there's a lot more on our website, the GitHub and then the full platform is built for scale and also includes monitoring. And I was touching on the importance of making sure evaluations are comprehensive, extensible and reliable. Experiment tracking tends to be very helpful in order to select the best app configuration. And we'll see that in action in the notebook itself. And then on the debugging side, what we have found is it goes back to that challenge that if you. It's very important to have a fine grained logging of the trace so that you can do granular evaluations through what we call lenses on parts of it. And I'll illustrate that in a bit. Then once you get into production monitoring, scaling to large data volume, getting alerts, and then making sure that the evaluations in production are lower cost and lower latency than what you do in development. That's extremely important as well because otherwise the cost of monitoring can become 10 times the cost of just operating the app. So there we have done some additional work on using smaller models and so on. We won't get into that today. I just wanted to give you the big picture view before diving into the section around evaluation and debugging. So what kind of evaluations make sense in this context? One of the concepts that we introduced last year was this notion of a rag triad to guard against hallucinations. The Ragtiad has now become quite extensively adopted and a bit of the de facto standard in this field of how to get started with evaluations to guard against hallucinations. It very much follows the structure of the RAG that Amnon presented. The user comes in with a query. There's a set of retrieved pieces of context or chunks from the vector database and then those get summarized to produce the final response. The summarization is done by an LLM along each edge. There are failure modes that you want to guard against and that's what the evaluations help catch. There's context relevance, which checks is the retrieve context relevant to the query. The reason context relevance is important is that the retrieval step is not perfect because it's doing embedding similarity between the question and the previously put in chunks which are in the vector database. Like a pine cone. The embedding similarity can work fairly well, but it's not perfect and we'll see some examples of failure more shortly. And context relevance evaluations can help catch that. Groundedness is checking is that final response that we are getting out of the rag is that is every statement there backed up or supported by the retrieved pieces of context? Are they traceable back to those retrieved pieces of context which we put in our knowledge base and say a pine code database? This is important because this is where an LLM, even if you use a RAG with a vector database as a knowledge base, the LLM can still hallucinate in that last step of response generation. We'll see that again in a little bit concrete examples of how hallucinations creep in. Because one way to think about this is that LLMs are very eager to please, meaning that they're built to generate the next word. If you give them a set of sequence of words there, their job is to generate the completion for that sequence. Especially when the retrieved pieces of context don't contain the answer to the question that is being asked. The LLM will try to fill in the gaps with what it has learned before. In that process it can introduce hallucinations and groundedness can guard against that. Then finally there's answer relevance, which is checking is the answer relevant to the query that was asked. So with these three evaluations we can get to a fairly robust evaluation of the rag of guarding the rag against hallucinations. And these are not the only evaluations out of the box. In True Lens you have a lot of other evaluations to ensure that applications are honest and harmless and helpful. The rag triad falls in this bucket. You can see answer relevance groundedness and context relevance here. But also looking at things like toxicity, stereotyping, guarding against jailbreaks, some of these coherence concise linguistic evaluations, these are all very valuable in building up the quality of applications and get to a fairly robust evaluation of the rag of guarding the rag against hallucinations. And these are not the only evaluations out of the box. In True Lens you have a lot of other evaluations to ensure that applications are honest and harmless and helpful. The rag triad falls in this bucket. You can see answer relevance, groundedness and context relevance so that you can trust trust to move them into production. And then the custom evaluations is something we see our users make use of a lot where they can add their own evaluations or feedback functions into the, into into their code and evaluate on other dimensions that are not covered by the out of the box evaluation because it's a community effort as well. That way we see a way for the community to contribute to the open source effort, which everyone can then benefit from. If you're using it in your commercial applications, you don't have to contribute it back. It's not a GPL license, it's an MIT license so you can cure. But also looking at things like toxicity stereotyping, guarding against jailbreaks, some of these coherence concise linguistic evaluations, these are all very valuable in building up the quality of applications so that you can trust to move them into production and then the custom evaluations for your own use as well. Let me give you some concrete examples of failures of these evaluations and then we'll come back to the notebook that Amnon started to see how it plays out with Canopy. But I do want you to get some of these general learnings which are not tied to either True Lens or the Canopy, but these are general principles for how to evaluate and improve something. We see our users make use of a lot where they can add their own evaluations or feedback functions into the, into into their code and evaluate on other dimensions that are not covered by the out of the box evaluation. So as a. Because it's a community effort as well, that way we see a way to, for the community to contribute to the open source effort, which everyone can then benefit from. And if you're using it in your commercial applications, you, you don't have to contribute it back. It's not a GPL license, it's an MIT license, so you can keep it for your own use as well. So let me give you some concrete examples of failures of these evaluations and then we'll come back to the to the notebook that I'm started to see how it plays out with Canopy. But I do want you to get some of these general learnings which are not tied to either True Lens or to Canopy, but these are general principles for how to evaluate and improve your applications that are now starting to get very broad adoption. So in this rag we indexed the Truera website and we asked who is Shayak? So Shayak is my co founder. Your applications that are now starting to get very broad adoption. So in this rag we indexed the Truera website and we asked who is Shaykh? So Shaykh is my co founder and former PhD student. And the response from the rag is mostly correct except for that last sentence. That last sentence talks about him being a member of the bank of England England and former PhD student. And the response from the rag is mostly correct, except for that last sentence. That last sentence talks about him being a member of the bank of England England's AI Public Private forum. And that's not accurate. Now when we do the evaluations of context relevance with True Lens, here is what we see. The top two rows there were four pieces of retrieve context from the vector database from a pine cone. The top two were actually about Shyek, the bottom two were not. But when we do the automated evaluation of context relevance using our out of the box evaluations, the top two were directly assessed to be relevant. So they got high context relevance AI Public Private forum and that's not accurate. Now when we do the evaluations of context relevance with True lens, here is what we see. The top two rows there were four pieces of retrieve and scores on a scale of 0 to 1. It was 1, whereas the bottom two were got low context context from the vector database from a pine cone. The top two were actually about shyac, the bottom two were not. But when we do the automated evaluation of context relevance using relevance scores because they were about a different person, Shamik, who also works at Treyar. Now in fact that last sentence had come from these the third and fourth pieces of context. That's how the inaccuracy crept in. This is an example of a retrieval failure, but the automated evaluations of context relevance correctly detects it and therefore you can use a form of context filtering where you say, well, if the evaluations show a score of above 0.8, only then will I forward it to the LLM to generate the final response. If you put in a threshold based context filter or out of the box evaluations, the top two were correctly assessed to be Relevant so they got high context relevant scores on a scale of 0 to 1 it was 1, whereas the bottom two got low context ring like that. Then only the top two pieces of retrieved context goes to the LLM for some rank relevant scores because they were about a different person. Shamik, who also works at True. Now in fact that last and then you don't see that last sentence creeping because those were from these other retrieved pieces. Sentence had come from these, the third and fourth pieces of context which had these low scores. So and this can be done during development, it can also be built into guardrails Introduction. So you can actually do this in production as well. Now under the hood there are LLM based evaluations that are automatically doing this, but we also have some smaller models that can do these in a more cost efficient way. The next thing I want to show you is an example of lack of text. That's how the inaccuracy crept in. This is an example of a retrieval failure. But the automated evaluations of context relevance correctly detects it. And therefore you can use a form of context filter groundedness. Here there's a phone call person in a hotel room where you say, well, if the evaluations show a score of above 0.8, only then will I forward it to the LLM to generate the final response. So if you put in a threshold based context filter calls to room service, the details are not that important. And then there's a summarization of that call using an LLM. And what does groundedness do? It shows for each statement here in the summary, it's coming up with a score based on how backed up it is by the full call transcript. Or in the case of a rag, you can trace it back to the four pieces of retrieve context and say for each statement in the summary, can I trace it back to the call to the retrieved pieces of context? So here you can see the first and third sentences. They get a higher score because they are well backed up. But the middle one, middle sentence is not well backed up. There is a comment in the final summing like that. Then only the top two pieces of retrieve context goes to the LLM for summarization. And then you don't see that last sentence grouping because those were from these other retrieved pieces of context which are these low scores. This can be done during development about the emotional state of the person making the call, but that's not well backed up by the actual segment. It can also be built into guardrails Introduction. So you can actually do this in production as well now under the hood of the full call transcript. So this kind of groundedness evaluation can really help you do a fine grain detection of hallucinations. And then you can have a guardrail in the application that based on the use case can can take different kinds of actions. So if the answer is not well grounded at all across the board, you might just block it and say that this is I don't have enough information to answer. There are LLM based evaluations that are automatically doing this, but we also have some smaller models that can do these in a more cost efficient way. The next thing I want to show you is an example of lack of groundedness. So here there's a phone call person in a hotel room. The question if most of it is correct and some of it is not well grounded, then you could potentially mark it out and say, well, these two sentences are well grounded, this one is not. So you may want to take that with a grain of salt. So that kind of logic can be very helpful down the road. And then answer relevant calls to room service. The details are not that important. And then there's a summarization of that call. Using an element is also very similar. In the interest of time, I'll skip that. The other thing I want to quickly touch on, and there were some questions about it on the channel, is how do you select the best app configuration, what's the best chunking strategy, and so on. And the nice thing about True Lens is that it supports experiment tracking where you can explore the space of different choices like what chunk size to use, what model you're using to select embeddings, what retrieval method you're using, what LLM you're using. What does groundedness do? It shows for each statement here in the summary, it's coming up with a score based on how backed up it is by the full call transcript or in the case of a rag, you can trace it back to the four pieces of retrieved content, what hyperparameters you're using for the LLM. And as you vary these choices, you can see in the dashboard how it's doing with respect to the evaluation metrics that you care about, and cost and latency and so on. Extensive. For each statement in the summary, can I trace it back to the call to the retrieve PI sign and then you can pick the optimal configuration. In fact, this can be made even simpler with Canape because with Canopy, because all of this is in the. These kinds of things are in the config file, the YAML file that Amnon showed You earlier, you can make these changes much more easily and observe the impact of that on the metrics that matter. So with that, let me now switch back to the notebook and I want to show you a couple of contexts. So here you can see the first and third sentences. They get a higher score because they're well backed up, but the middle one, middle sentence is not well backed up. There is a comment in the final summary about the emotional state of the person making the call, but that's not well backed up by the actual substance of the full call transcript. So this kind of groundedness evaluation can really help you do a couple of ways in which we now instrument the canopy rag with True Lens. So this is our rag triad, if you remember. And when we are doing the instrumentation in the notebook, which is this section of the notebook, instrumenting static methods used by the engine with True Lens, you will see that we are instrumenting two things. There's the context engine and the chat engine, both of which Amnon talked about a few minutes ago. The context engine is covering these steps over here from the context engine. When we instrument it, the query gets logged and the retrieve pieces of context get logged. So now there's enough information available to True Lens to do the context relevance evaluation because that requires access to the query and the retrieve pieces of context. And then the other piece that we instrument is this chat engine. The chat engine has the query and the response. So it's the fine grain detection of hallucinations. And then you can have a guardrail in the application that based on the use case can take different kinds of actions. So if the answer is not well grounded at all across the board, you might just block it and say that this is I don't have enough information to answer the question. If most of it is correct and some of it is not well grounded, then you could potentially mark it out in other edge of the triad here, these two pieces. Now we have the query, the response and across with the combined instrumentation, we have all three pieces. Because we have all three pieces, we can do all three evaluations for the rag triad. You can see in the notebook that we create these feedback functions using the instrumented methods. That's over here. If I pick. You can see the input here refers to the query. The output is the final response. Say, well, these two sentences are well grounded. This one is not. So you may want to take that with a grain of salt. And the context here is the is a list of the retrieve pieces of chunks from the vector database in the that's this piece. So now in the input. So that kind of logic can be very helpful down the road and then answer. Relevance is also very similar. In the interest of time, I'll skip that. The other thing I want to quickly touch on, and there were some questions about it on the channel, is how do you select the best app configuration, what's the best chunking strategy, and so on. The input is called the query here. The context is the same as the context in this picture, and the output is this final response. So we have all the building blocks we need appropriately logged after this instrumentation, when you run the app, this will get logged into a local database, like an SQLite database, and then that can then be used for the evaluation. So over here we have a groundedness feedback function. The groundedness feedback function makes use of the combined pieces of context and the on. And the nice thing about True Lens is that it supports experiment tracking where you can explore the space of different choices like what chunk size to use output, and so it's checking for each sentence. When you later run this roundedness feedback function, what model you're using to select embeddings, what retrieval method you're using, what LLM you're using, it'll check for each statement in the output, much like I showed you in the example, what hyperparameters you're using for that LLM. And as you vary these choices, you can see in the dashboard, is it backed up? Is there evidence in the combined context to support that statement? It'll come up with a score. It'll do it for all of these sentences. Let's say there are five sentences in the summary. For each sentence in the summary you get a score, which is how well backed up is it by the retrieve pieces of context. And then this aggregation step is going to aggregate across the scores of those five pieces, how it's doing with respect to the evaluation metrics that you care about, and cost and latency and so on. And then you can pick the optimal configuration. In fact, this can be made even simpler with Canopy, because. With Canopy, because all of this is in the. These kinds of things are in the config file, the YAML file that Amnon showed you earlier of context. And it's actually a two layer aggregation for each sentence. If there's four pieces of retrieve context, then there's a score that shows the first retrieve context has, let's say, a score of zero in terms of backing it up. The second one has 10, the third one has 2, the fourth one has 1. Then you take the max and that the second one actually has the best support for this sentence. And then you get that for all the five sentences. Then you do aggregation across all five. Then similarly, you can set up the answer relevance and context relevance functions. And then once you've set up these functions, you can run the evaluation. You can create the recorded app, run it, run the evaluations, and maybe I'll. If you want to add in a bit of color here, I'm not. You can make these changes much more easily and observe the impact of that on the metrics that matter. So with that, let me now switch back to the notebook and I want to show you. Couple of ways in which we know the recorded app and what's involved in running it.

Yes.

I didn't put the screen up, but you can speak to it.

Sure. First of all, I think it's worth mentioning that I think it's really cool that we built Canopy without considering True Lens, but you still can use True Lens with Canopy, meaning that probably with any application you build, you can use True Lens and it's kind of playing around together without a lot of effort. But let's go back to the evaluation. So what we have here is like a very small set for evaluation.

I'll instrument the Canopy rag with True Lens. So this is our rag triad, if you remember. And when we are doing the instrumentation in the notebook, which is this section of the notebook, we have like three.

Questions about the Pinecone documentation and what.

We instrumenting static methods used by the engine with True Lens. You will see that we are here.

We just run all these questions.

Two things. There's the context engine and the chat engine, both of which Amnon talked about.

A few minutes error recording automatically record these answers and. And here we just print the answer for the first question, which is what's the maximum dimension for a dense vector in Pinecone? And it answers that the maximum dimension is 4 billion, which is like not true. If you remember, we saw this the the right answer when we read the documentation. The answer is.

Let's go. The context engine is covering these steps over here. So from the context engine, when we instrument it, the query gets logged and the retrieve pieces of context get logged. So now there's enough information available to True Lens to do the context relevance evaluation.

Because that 20k, I can tell you why it was wrong because I know a lot of the documentation. This is actually the maximum dimension for the sparse vector, which is a different type of vector you can put in Pinecone. This is simply put like A mistake. And the thing is that once in a while you should. And you can evaluate manually this kind of answers, but in scale you can. You can't really do this. You won't go over thousands of questions. And if you do want to do a parameter sweep, you have much more.

Requires access to the query and the retrieve pieces of context and then the other piece that we instrument question to.

Go over and you want an automatic way to evaluate your system. So this is where twera comes in and kicks in. And like here you can see the.

Results for all the metrics meant as this chat engine. The chat engine has the query and the response. So it's the other edge of the triad here, these two pieces. Now we have the query, the response and across with the combined instrumentation, we have all three pieces. Because we have all three pieces Anupam.

Described for this set of three questions for evaluation. And what you can see that groundness is very low, meaning that it seems like this metric captured the idea that the model probably got some explanation about sparse vector. And then he provided an answer about dense vector, probably because he wanted to please the user, like Anupam said, because this is what models do. But the main point is that instead of just going manually question by question here you.

We can do all three evaluations for the rag triad and you can actually.

See an automatic metric that aggregates the score and you can understand that there is a problem in the system and mainly in the groundless part. And yeah, yeah.

So maybe as we can see in the notebook that we create these feedback functions using the instrumented methods that's over here, if I pick, you can see the input here refers to the query. The output is the final response. As we ran this evaluation, these are some of the kinds of metrics that we saw and the groundedness in particular was quite low. And just returning a little bit to the comment that Amnon made, we have built rules in a very general way so that you can work with different frameworks for building rags. Canopy is a new framework that has some very significant value for building production apps. That's one of the reasons we wanted to work together. We saw that the actual abstractions that are there in True Lens and the context here is a list of the retrieved pieces of chunks from the vector database. That's this piece. So now in the input, the input is called the query. Here the context is the same as the context in the space for logging these traces in a way that. And the output is this final response. So we have all the building Blocks we need. We can. We have these lenses that can look at these spans or parts of traces to carry out this context relevance, groundedness and answer relevance kind of evaluations. And the whole set is a lot bigger than that. But the general abstractions which lets us look at these intermediate results, instrument the app, log it and then do out carry out the evaluation. We had previously done this for LangChain and Llama Index and had built out some very specific appropriately logged. After this instrumentation, when you run the app, this will get logged in take abstractions for them. That makes it easier. We work for any kind of Python app. And it was very interesting to see that for Canape the abstractions were so nice to a local database like an SQLite database. Then that can then be used for the evaluation. Over here we have a groundedness feedback functions in terms of these context engines and chat engines. That was really very little work to get this done. The groundedness feedback function makes use of the combined pieces of context and the work that canopy as well. Maybe this is the first version of the app, we know it's not doing so well with respect to groundedness. Then in the next version of the app I noticed that we are running out of time. Amnon, I'm going to speed up a little bit and say that introduce this re ranking method. Once you introduce the re ranking method, these output it's checking for each sentence. When you later run this roundedness feedback function, it'll check for each statement in the output. Much like I showed you in the example. Is it backed up? Is there evidence in the combined context to support that state? These metrics go up a lot and you can see context, relevance, groundedness and answer. Relevance really starts getting to that a plus level. The nice thing again I do want to emphasize this is this just because you can change the config file so easily. It was really fast iteration and this is something we see. It'll come up with a score. It'll do it for all of these sentences. Let's say there are a lot. And Amnon, I'm curious to hear your experiences. Pinecone has lots of users, lots of customers as well. At Trera we see this a lot. That in the development workflow it's incredibly valuable to be able to quickly iterate. Look at the space of options for different hyperparameters to quickly iterate. Identify failure modes, iterate and imperfect. Five sentences in the summary. For each sentence in the summary you get a score, which is how well backed up is it by the retrieved pieces of Context then this aggregation step is going to aggregate across the scores of those five pieces through the application. And that part of the workflow just the productivity gains have just huge impact in rolling out generative AI apps in companies. I'm curious to hear from you. I'm not also about that. I do want to point people to this course that Dan alluded to earlier which we released losses of context and it's actually a two layer aggregation for each sentence. If there's four pieces of here in November it has gotten tens of thousands of folks who have taken it. It's just a short one hour course and it covers some of the other advanced rag applications as well. And then Pinecone has a similar similar course. Some of the key takeaways here on you can build production grade rag apps significantly easier with retrieve context than for there's a score that shows the first retrieve context has a let's say a score of zero in terms of backing it up. The second one has 10, the third one has two, the fourth one has one. Then you take the max and that the second one actually has the best support for this sentence. Then with True Lens and True error you can get evaluation debugging fast iteration as you're building out the apps and continuing into production. Any final thoughts? Amnon? I think we also should take a look at one or two questions. Questions?

Yeah, I think you sum it up.

And then you get that for all the five sentences. Then you do aggregation across all five and then similarly.

Well Gany, your question. We do see that you know having the ability to iterate fast is really enable you the way to evaluate fast because usually if your system is hard to configure, you will just end up not experimenting with it because it's hard. So it's kind of the amount of valuation you can run is kind of a function of how it's easy to change your system. This is what we see. I guess we can move to questions.

You can set up the answer relevance and context relevance functions. Then once you have set up these functions you can run the evaluation. You can create the recorded app, run it, run the evaluations and maybe, maybe if you want to add in a bit of color here, I'm not let me stop sharing and yeah, perfect. Well thank you on the recorded app and what's involved in running it. I didn't put the screen up but.

You can speak so much for this workshop. I know this has been really great. I know we only have a few minutes left so just dive into some of the Most popular questions. So the first question, which models are better for rag solutions? Instruct or conversational to small models like Tiny Llama or are useful in rag implementations?

You want me to take it or.

Yeah, go for it.

Yeah. Okay, so I think for the first question, instruct or conversational? I guess first of it depends on your application. If your application is chat, you obviously need to use. Sure. So first of all, first of all, I think it's worth mentioning that like it's really a chat model, like you have no option. But in general what we see, I think it's really cool that we built canopy without considering tool lens, but you still can wrap. You can use tool lens with canopy. It meaning that probably with any application you build, you can use tool lens and it's kind of playing around together without a lot of effort. And like if you look at OpenAI, they kind of, if I'm not wrong, deprecated the instruct idea because a lot of time chat models can act as instruct models. But I guess it's again, it depends like if you're. Again, if your application is chatty, you need a chat. If the kind of model you are using is tending more to to be advanced on the chain size, you need chat. If it's invest on the instructor you use instruct, it just depends on the specific use case.

Yeah. And there's a few questions around like the scope of models and so on. So I just want to emphasize that on the true lens side, we are not really attached to OpenAI. You can use a lot of different models for evaluation. You can use the OpenAI source models.

Like, but let's go back to the evaluation. So what we have here is like a very small set for evaluation. We have like three questions about the Pinecone documentation and, and what we do here is just run all these questions to, to error Command to, to llama.

V2 and so on. There's lots of connectors to lots of models. It's just built in a very open way.

Well, error recording automatically record this answers. And, and here we just print and.

On the pinecone side as well, it's very similar.

Exactly. And we got these small models. Like obviously the downside is usually they are maybe less powerful. Upside is maybe it's easier to run them, maybe cheaper. But maybe you want to use small model even first. Like first if it's just workflow and second, if you want to fine tune because you want to fine tune your model for a specific use case, you usually want maybe to use smaller models and with fine tuning you add more data and maybe you end up with the model that is even more powerful than a larger model that is just coming out of the box. The answer for the first question, which is what's the maximum dimension for a dense vector in Pinecone?

Absolutely. Onto our next question. Do we need to feature proof are.

And it answers that the maximum dimension is 4 billion, which is like not true. If you remember we saw this the the right answer when we read the documentation. The answer is 20k. I can tell you why it was wrong because I aware I know a lot of the documentation this vector dimensionality.

As new models will be released. Do we need to upgrade it too as well?

It's a good question. So out spinecon like now is that every index know when you build an index you need in advance to specify what is the dimension. And so what happened is that like. Like imagine that if you switch model you always you also need to re index everything because your vector is going to be different. So anyway, let's imagine you switch from one model to another. Either you if you change the dimension or not. Anyway, you need to read actually the maximum dimension for the sparse vector, which is a different type of vector you can put in Pinecone. And this is simply put like a mistake. And the thing is that once in a while you should and you can evaluate manually embed everything. So on pinecons it just means you need to index everything also because it's just. Just new data. Right. In Pycon side it's only about vectors. And if you have new models, you have new vectors and this kind of answers. But in scale you can't really do this. You. You won't go over. You're just gonna need to ingest everything again. But pine cone version is the same. No need to upgrade Pinecon or something like this.

Absolutely great. How can I fix LLM inconsistency in our product?

You want to take it on?

Yeah, sure. I'm not entirely sure what you mean by LLM inconsistency, but I suspect what you might be referring to is that LLMs are stochastic. And so if you ask the same question multiple times, you may not get the same response and thousands of questions.

And if you do want to do a parameter sweep, you. You have much more questions to go over and you want an automatic way to evaluate your system. So this is where Torah comes in and kicks in. And like here you can see the results for all the metrics.

There are two pieces to it that I'LL say one is ideally you want to keep if you care about answers.

Being consistent, then describe for this set of three questions for evaluation and what.

Usually the temperature parameter you want to keep it low, close to zero. And the other piece is the on the ongoing monitoring and guardrails where you will that even when the temperature is zero it will sometimes change its output. So it's not a guarantee. So having guardrails in place to make sure that you are guarding against hallucinations, you're guarding against some of the relevance related pitfalls and toxicity and so on and having that be a part of.

Your application, you can see that groundness is very low. Meaning that it seems like this metric captured the idea that the model probably got some explanation about sparse vector. And then he provided an answer about dense vector probably because you so that.

You can block inappropriate responses or make adjustments or rerun things.

Wanted to please the user like Anupam said, because this is what models do.

That's path forward here. Yeah.

Another trick I saw people sometimes do it a bit expensive but what you can do, you can generate multiple answers and then have another layer of NLM trying to say which one is like the majority answer. In this way you kind of looking at the expectation of what the model thinks, you know, depends on your application.

Yeah, for sure.

But the main point here is that instead of just going manually question by question here you.

You absolutely. Great. Well, I think that may be all the time that we have for now. Thank you so much Anupham and Amnon for your workshop. I know our community really enjoyed it. For follow up steps you actually see.

An automatic metric that aggregates the score and you can understand that there is a problem in the system and mainly in the groundless part. Yeah.

Yeah. So maybe as we the slides will.

Be sent out and the notebook is available in the video description so you should be able to be able to access it there. And please fill out our surveys so we can continue putting on great events for everyone. Thanks everyone. We'll see you guys next time.

Bye. Thank you. Appreciate the conversation. As we ran this evaluation, these are some of the kinds of metrics that we saw and the groundedness in particular was quite low. Just returning a little bit to the comment that Amnon made. We have built rulens in a very general way so that you can work with different frameworks for building rags. Canopy is a new framework that has some very significant value for building production apps. And so that's one of the reasons we wanted to work together and we saw that the actual abstractions that are there in True lens for logging these traces in a way that we have these lenses that can look at these spans or parts of traces to carry out this context relevance groundedness and answer relevance kind of evaluations. And the whole set is a lot bigger than that. But the general abstractions which lets us look at these intermediate results, instrument the app, log it and then do out carry out the evaluation. We had previously done this for LangChain and Llama Index and had built out some very specific abstractions for them. That makes it easier. We work for any kind of Python app and it was very interesting to see that for Canopy it was the abstractions were so nice in terms of these context engines and chat engines. That was really very little work to get this to work with Canopy as well. Maybe this is the first version of the app we know it's not doing so well with respect to groundedness. Then in the next version of the app I noticed that we are running out of time Amnon so I'm going to speed up a little bit and say introduced this re ranking method. And once you introduce the re ranking method these metrics go up a lot and you can see context relevance, groundedness and answer relevance really starts getting to that a plus level. The nice thing again why I do want to emphasize this is this just because you can change the config file so easily. It was really fast iteration and this is something we see a lot. And Amnan, I'm curious to hear your experiences. Pinecone has lots of users, lots of customers as well. At Trera we see this a lot that in the development workflow it's incredibly valuable to have to be able to quickly iterate, look at the space of options for different hyper parameters to quickly iterate, identify failure modes iterate and improve the application. And that part of the workflow just the productivity gains have like just huge impact in rolling out generative AI apps and in companies. And I'm curious to hear from you. I'm not also about that, I do want to point people to this course that Dan alluded to earlier which we released last year in November has gotten tens of thousands of folks who have taken it. It's just a short one hour course and it covers some of the other advanced rag applications as well. And then Pinecone has a similar similar course. Some of the key takeaways here on you can build production grade rag apps significantly easier with Canape than with True Lens and Truera. You can get evaluation debugging fast iteration as you're building out the apps and continuing into production. Any final thoughts? Amnon? I think we also should take a look at one or two questions.

Yeah, I think you sum it up very well. Gandhi, your question. We do see that having the ability to iterate fast is really enable you the way to evaluate fast because usually if your system is hard to configure, you will just end up not expecting experiment with it because it's hard. So it's kind of the amount of valuation you can run is kind of a function of how it's easy to change your system. This is like what we see and then I guess we can move to questions. Yeah.

Let me stop sharing and yeah, perfect.

Well, thank you so much for this workshop. I know this has been really great. I know we only have a few minutes to love, so just dive into some of the most popular questions. So the first question, which models are better for rag solutions? Instruct or conversational? Do small models like Tiny Llama or are useful in rag implementations?

You want me to take it or.

Yeah, go for it.

Yeah. Okay, so I think for the first question, instruct or conversational? I guess first of it, it depends on your application. If your application is chat, you obviously need to use a chat model like you have no option. But in general what we see, like if you look at OpenAI, they kind of, if I'm not wrong, deprecated the instruct idea because a lot of time chat models can act as instruct models. But I guess it's again it depends like if you're again, if your application is is chatty, you need a chat. If the kind of model you are using is tending more to be advanced on the chances you need chat. If it's invested on the instruct, you use instruct, it just depends on the specific use case.

Yeah, and there's a few questions around like the scope of models and so on. So I just want to emphasize that on the true lens side we are not really attached to OpenAI. You can use a lot of different models for evaluation. You can use the open source models like llama V2 and so on. There's lots of connectors to lots of models. It's just built in a very open way. And on the pinecone side as well, it's very similar.

Right, exactly. And we got these small models. Like obviously the downside is usually they are maybe less powerful. Upside is maybe it's easier to run them, maybe cheaper, but maybe you want to use small model even first if it just works for You. And second, if you want to fine tune, because if you want to fine tune your model for a specific use case, you usually want maybe to use smaller models and with fine tuning you add more data and maybe you end up with the model that's even more powerful than a larger model that is just coming out of the box.

Absolutely. Onto our next question. Do we need to feature proof our vector dimensionality as new models will be released? Do we need to upgrade it too as well?

It's a good question. So I would find one like now is that every index know. Yeah, you. When you build an index, you need in advance to specify what is the dimension. And so what happened is that like. Like imagine that if you switch model you always. You also need to re index everything because your vector is going to be different. So anyway, let's imagine you switch from one model to another either if you change the dimension or not Anyway, you need to re embed everything. So in Pineconsign it just means you need to re index everything also because it's just new data, right? In Pinecone side it's only about vectors. And if you have new models, you have new vectors and you're just gonna need to ingest everything again. But Pinecone version is the same. No need to upgrade Pinecone or something like that.

Absolutely. Great. How can I fix LLM inconsistency in our product?

You want to take it?

Yeah, sure. I'm not entirely sure what you mean by LLM inconsistency, but I suspect what you might be referring to is that LLMs are stochastic.