The One Skill That Survives The AI Shift | Ofer Mendelevitch, Author of Hands-On RAG for Production

You wrote the book while people have been saying that RAG is dead, right? For the past 2 or 3 years.

There's been like RAG is dead claims every 2 months. People mostly trying to market something new. But I would say if you think about RAG, it's 3 components is retrieval, augmented generation, and retrieval is still extremely important even with agents. There was a time when people did like put 1 PDF or 2 PDFs with RAG connected to an LLM for generation and it felt magical. Look, mom, I can talk to my PDF, but that is very different than millions of documents, all kinds of formats. Doing it at scale is very, very different. Imagine a training video where I'm the instructor and I'm saying, okay, so in order to avoid catastrophe, never press this button. If you just transcribe it, you lose the meaning completely. It's called like the red button problem. You want to have visual understanding of what's in the image or in the sequence of images in the video, as well as correlate it with the audio in order to get the complete meaning. What I'm concerned about is as a society what we do with this. So I think there's a way where this is actually benefiting everyone, but in the meantime, there could be a lot of pain. We're going to end up in engineering and data science being directors as opposed to actors. The coding agents will write the code, and I think what remains a really critical challenge is to understand what we need to build and guide it in the places where the coding agents are not doing a good job. A bunch of high schoolers, they were looking at me and saying, like, did we make the The most incredibly stupidest mistake ever because they all got accepted to their computer science program. I said, no, you can graduate from college and have the capability of a senior engineer today. Use the next 4 years to learn how to be that.

I have Ofer Mendeleevitch here today. He's a person who can teach us how to actually build RAG systems that work in production, and he also recently published an O'Reilly book on exactly this topic. Welcome, Ofer.

Thanks, Angelina. Great to be here.

I don't want to give our audience a LinkedIn intro about you. How about you share with us something about yourself that's not on LinkedIn?

Oh, what's not on LinkedIn that I've seen a lot of cycles? I've started using what we now call AI, data science, machine learning is what it used to be called back in the day, a long time ago. One fun fact is when I started this, I transitioned from engineering to data science. This might sound really weird, but The best, most successful algorithm for doing any type of prediction was Naive Bayes, if anybody's heard of that. It's a very basic, very simple algorithm, but it was really hard to do in those days. CPUs, there were no GPUs at the time. So I've been in this space for a while.

Yeah, I know. I wonder if people are still using it, Naive Bayes.

I don't think so. I think it's not as strong as some of the things. So we've evolved from Naive Bayes type algorithms to the era of the XGBoost and those kinds of algorithms, decision trees. And now today we live in the world of LLMs that we all go crazy over. I got lucky enough to bump into the world of LLMs quite early on. So I used GPT-2, actually GPT-1 for maybe a month or two until it became GPT-2, was released back in 2019, I think it was something like that. And so, I got the chance to use it quite early on before it became ChatGPT that we all know and care about today.

Yeah, 2019 doesn't sound too far away, but it felt like a decade.

Yeah, think about it. It's like pre-pandemic, first of all, right? Or just before, but yeah, it's so long because the world, especially the world of technology, has transformed so much in the last I mean, since ChatGPT came out and like changed everything, is now using LLMs and agents and RAG and all that stuff. So yeah, it's changing so rapidly.

Let's talk about your book. Your book's name is Hands-On RAG for Production. I read the whole book yesterday.

Thank you.

I want to ask, why did you write this book? I think you wrote the book while people have been saying that RAG is dead, right? For the past 2 or 3 years.

Yeah, so when I started, it was about a year and a half, a little bit more than a year and a half ago when I started writing this book. And at that point in time, actually, people didn't quite say RAG is dead so much. It was pretty much the center of RAG. Everybody was doing RAG. Everybody was experimenting with RAG. And at the time, at Vectara, we were like a RAG platform. We're in the middle of all of this storm of RAG. And essentially, I've learned so much by helping our customers implement our system and also looking at the open source community. There's a lot of other options to do that, like in LangChain and LlamaIndex and a bunch of other open source projects. And I've learned so much about anywhere from ingest to querying, getting accurate results, reducing hallucinations, things like that. That I think the idea was to really try to distill all this knowledge in a way that's helpful for people, especially because a lot of the stuff was just prototypes, just initial demos. And then it is very helpful to build system with this, but how do you, when you scale them to production, a lot of other things starts to break.

Okay.

And the demo is really simple. There was a time when People did like, it was like, hey, I put 1 PDF or 2 PDFs with RAG connected to an LLM for generation and kind of, look, mom, my hands are free. I can talk to my PDF, right? That was the typical demo that people had. And it was really fun. It felt so powerful. You could have a long PDF with hundreds of pages and just ask questions and get answers grounded in that PDF. It felt magical. It's a lot of people, but that is very different than, you know, millions of documents, all kinds of formats, PowerPoint documents, different sources. Doing it at scale is very, very different. And that was the goal of the book is to really convey everything that I know and I've learned over those years in that space.

How about we do this? I have a simple use case. I actually have this use case. Maybe we can work together to build a conceptual RAG system.

I love it. Let's do it.

Let me share with you my use case. I simply want to verify, you know, LLM outputs are real or not. I want to reduce hallucination in any AI outputs. It could be a research report or a blog post, right? So what happens is I typically generate references with it. Claude does it, ChatGPT, they all have a version of it. But when I do a manual check, right, we often find that it's partially correct. There are a lot of issues. This process is very tedious, and I want to set up a system that actually automate this whole thing. And I have 3 goals. I need to get evidence from the source material. I need to know whether those evidences actually prove my claim is correct. I need to verify the numbers. So any numbers and statistics quoted in the claim, right, they need to be correct. If the source material says something like 90% of the POCs are failed in production, and in my generated result it says, let's say, 85%, That's a mismatch, but the RAG system might actually turn out to be saying this is a match, right? Because most of the words in this sentence are the same. Can you help me to think of a way to set this up correctly?

Fantastic, fantastic use case. So what I'll say is let's go through it step by step, but what I'll promise to do is sort of explain or try to explain here how to solve these challenges. Not all of them necessarily are solved by by RAG in the traditional sense, but I would say this upfront, and this is part of the book too, is we make the connection between RAG and agents, which are what's really common these days. And one thing I could sort of present is that, you mentioned earlier, RAG is dead. There's been like RAG is dead claims since the last 2 years, every 2 months, people mostly trying to market something new, which is fine, but I would say If you think about RAG, it's 3 components is retrieval, augmented generation, and the RAG. And so the R is for retrieval and retrieval is still extremely important even with agents. A lot of the book talks about how to set up proper retrieval. Generation, the difference is that generation in RAG is just a one-shot. You ask a question of the information and you get a response back and then you send it to the user, whereas agent has a— A loop in there and can ask questions again and again. And the reason I'm mentioning this is because in the use case you mentioned, right, you could, for example, look at the whole document and ask a question of RAG to evaluate all the stuff that you mentioned, the 3 requirements. You could also do it— do step 1 is give me all the things that might need review as a list, and then go through the list in the agent step by step and review each of them separately. for validation. So anyway, so we'll talk about this, but let's start with the requirements. So the input here is a document. Let's say it's a PDF for simplicity.

Okay.

And you have some source documents that constitute the source of truth for you, right? So is that 10 documents or 100 documents or a million documents, right? So they exist somewhere and then So RAG is all about grounding in some source of truth that you decide, that you set up. And so let's assume that it's there, and I'll assume it's, let's say, 100 documents for now, but we can talk later about, again, going to production. What happens when it's a million documents? It becomes much more complicated. But the first step in RAG is, of course, you take these documents and you— they're called ingest. You ingest them into the RAG system. The reason you have to do something with them is because you want them to be ready For retrieval, for accurate retrieval. And there's something called semantic search, which is— people know as the vector database kind of mechanism, which finds semantically similar information in those documents. And I'm explaining this in detail. Many people know this, just to emphasize to your point, Angelina, if you look for 85 and 95 differences, they might not show up in semantic search. It's not really a— Right. Thing. So either you find, if you say like the number of people failing production is, if you have that word of number of people failing something, if that text exists somewhere in the document and it's correct, then you'll pull that information, you'll be able to correct the number. But if it doesn't say it in those words and it just is the number mentioned somewhere where it doesn't exist, you will not be able to find it. So the ability to get the right information during retrieval is extremely, extremely important, right? And so I'll talk in a minute about 3 different ways to do that or a couple of different ways to do that that are helping the raw RAG. Let's get back to your question. So first you ingest your 100 documents and to do that, you go through the regular RAG pipeline steps. You extract the text from the documents, you do what's called chunking. Which takes the text. It could be 100 pages. So you split it into smaller sub-sentences or paragraphs, and then you vectorize them, which means you embed them, sorry, encode them with an embedding model, and you store 'em in some sort of a vector database, and then they're ready to retrieve. Of course, you store not just the vector, you store the text itself 'cause you need to retrieve it at some point. And then there's something sometimes people call hybrid search. That we talk about too in the book, which is the combination. So semantic search is the new thing with the vector database, but we all had TF-IDF and BM25, the traditional search algorithms. And so hybrid search is combining those 2. And sometimes that's important when you look for some things that don't have a semantic meaning. Like if I look for a certain product SKU, for example, doesn't mean anything. It's just a number. or the word iPhone or something, or just one word. So yeah, so first I think you do that. Does that make sense so far in terms of what you want to do for your use case?

I take the reference out and then I will retrieve the whole document as input document and I'll embed all of those. And in the meantime, we want to know what we're verifying. This one single article, I need to make sure it's correct, right? So I will look for maybe the reference point, like the where maybe 2 to 3 sentences right before the reference number. So that's the statement I'm trying to verify.

So during ingest, you actually want to— when you break it into chunks, you want to mark each chunk where is it, which page and which part of the page, so that later on in the query, we can sort of reference back to show where it was. 100%.

So we have our input data ready now.

Yes. So now when you build a RAG pipeline, essentially, and actually, let me use a— maybe I can use a diagram here.

I think it'll be— You have a diagram?

Can we see this? Yeah.

Okay.

Is it big enough? Okay. So, what you see here is kind of the flow. We just talked about taking data on the left side here and moving it through to the vector store right here, right? So, you extract text, you do the chunking. I didn't mention table processing, right? Which is sometimes your PDF might have a table or even an image, right?

Yeah.

So if you want to answer questions based on that, so for example, your numbers might be not stated in the text, but you might see a diagram that says statistics about failure of RAG implementations, and it could be like some 5 numbers from different customers. So you want to be able to understand that information as well, and this is called multimodal data in RAG, and then you encode it in the embedding model and you store it in the vector database. During query time, so now comes the time when you ask a question about the whole document. Let's start with that simple case. You give it the whole document and say, please review this whole document and tell me what things I need to fix, right? What are things and what claims are cited correctly? So this is where this becomes complicated, but you send a query in, the query gets embedded too, and it starts, it It starts, it looks to find the most similar parts of the text out of the vector store. So if your query has a lot of interesting pieces to it, including I want the citations to be accurate and I want every piece to be correct, my guess is it probably finds some of these things here, but it's only based on this one match. And that's why I said earlier, I think the better implementation of this is with agents. So let's talk about this in a minute, but at a high level, agents can then break down your query. They can read the document. The agent can read the document first, the document you want to validate, break it down into different queries and run 5 or 6 or 100 of these queries against the vector store and get the results. Just to finish up this story, right? So imagine that there's— that you just want to do one query. So you get the top, let's say, 5 or 10 results that are relevant for the query. And that can be just a vector search, or it could be, like I said, hybrid search with BM25 or TF-IDF or something. And there's sometimes a step called re-ranking, which helps get even more accurate results. And then you take a prompt and you say to the LLM, which sits here, you say, hey, I have this user query and I have this document and here's some results that might be relevant to help the user answer the user query based on this document and based on these results. And then the LLM generates the results and sends it back to you. But you can see, Angelina, here how I have a feeling that if the number of things is more than 3 and the document is pretty long, this probably doesn't work in its current form, in its one-shot form, right?

Yeah.

So what I wanted to show next is like how agents work. And there's a chapter on agents which exactly tackles this problem, which is agents are, you know, this is a sort of a rough diagram of how it works, but Agents have these things called tools, right? And so in this case, you would, and they're controlled by an LLM. So there's an LLM here, which is part of the agent, and it sort of reasons about your query, plans what to do, and then executes it by calling one or more tools. And so the tools could be calling an API, running a query in the database, but a lot of times it's actually calling The RAG pipeline that we've seen just before, right? So it calls the retrieval part of the RAG pipeline that we've seen before, right? So you can call RAG. In this case, maybe you have 10 different parts of the document that you kind of want to verify. The LLM should find all these parts on its own. You don't have to tell it what it is, and it will probably call the tool 10 times. Each time it'll ask, is this statement correct? And it'll give the results. This is a micro— and then it'll take all this information back and review it together and give you a formal answer about those that are correct, what are the citations? And again, the RAG tool will return the citations correctly, and then those that are incorrect will highlight those and ask you to sort of figure out how to fix them or suggest a fix if it exists in your document set.

Yeah? I can see why the agents might work better here because the verification process can be very nuanced.

Yeah, the whole idea of the basic RAG is this strict matching of a query to a set of documents. And in this case, the query you're going to send outside is like, I want to fix all the problems. That in itself wouldn't match any particular document part, right? So it ha— you have to have either run it as multiple queries, which is possible too, or that's what the agent will do for you essentially automatically. So that's why I kind of jumped there.

The RAG similarity kind of matching is always not performing well on like exact matches or like numbers, temporal data, like, you know, time, time dimension is also really complex, complex data, complex questions is also not going to perform very well.

Yeah, multi-hop questions, for example.

Right, exactly.

Yeah, so that's a good point. And I wanted to actually— let's talk about this a little bit because I think it's important and it's a big part of the book too, right?

Give me an example.

So first of all, so I gave you an example of— let's start from the beginning. So vector search, which is where everybody starts when they learn about RAG, is— I just want to give it enough focus, right? It is important. It's very powerful. It does this semantic search in multiple languages too, right? You can ask a question in English and get a match for a document in Korean or something, right? Or Spanish or whatever. So there's a lot of power to it, but it is how the name sounds. It's semantic search, so it looks for semantic similarity. Now, if I look for the word iPhone, I probably want to get iPhone, not necessarily Android. I want to get like— Or if I look for a certain code or something, there's no semantic meaning per se for this. So this is where you want to add hybrid search or other more traditional search. But I want to talk about 2 other things that I kind of mentioned in passing and I think are important. One is the idea of multimodal search. Okay. And again, it's relevant for RAG and it's relevant for agents, which are an extension of RAG. So When you do retrieval on multimodal data, it could be tables in a document, it could be images in a document, it can also be audio and video. And so video could be, like to give you an example, it could be a lot of companies, enterprises have huge amounts of maybe recorded meetings or training material, right? And you want to be able to search on that. And so in order to do that, and integrate it into RAG, you have to do some extra work. And again, if it's in production, you have hundreds of thousands of those, it becomes even more maybe taxing from a performance perspective. But generally, there is ways to look at— I think tables are probably the most prominent. And to give you a sense of why that can become problematic, because tables could sometimes be reversed like this. I've seen this a lot in medical journals, or there could be multiple page tables, and the traditional way that you process text is called chunking, we mentioned earlier, and essentially breaking it down into text, but you can imagine a text that has column names at the top and 100 rows, so the first chunk will have column names and 5 rows, and then the second chunk will not have column names, so now the LM doesn't know what it means. So you gotta like treat the table as a first-class citizen and have it been treated properly. So the best way I know of to do this is to identify the table as a whole table and store it separately from the chunks. So store it as a table, and then in query time, you actually retrieve the table itself and you give it to the LM as a full table, and then it can figure out whatever it wants from that. And the second thing is images. Again, images could be really, really powerful as the saying goes, right? An image can convey better than 1,000 words. In many cases it is, but you have to know what you want from that, right? So you usually want to, again, store the image as a first-class citizen, not text, and then be able to both use it in retrieval. So this image is relevant for this query and then maybe even show it as part of your result from, let's say, if it's a chatbot or something, show it as part of the result as an image, not describe what you see, right? Users can enjoy that as well. Now audio and video become even more complicated. One of the examples I like to talk about to demonstrate why video is so critical is— so audio, you can transcribe it if it's somebody speaking like you and I, but if it's music, unclear how you thing if some music is relevant for something. So usually people ignore that as far as I've seen so far. But video is like a sequence of images with audio, right? So it's like the most complicated of all.

Yep.

And it's really important to understand both. So for example, imagine a training video, okay? And I'm standing there, I'm the instructor, and I'm saying, okay, so in order to avoid catastrophe, Never press this button. Now if you look at the text, you don't know what this is. If you just transcribe it, you lose the meaning completely, right? This is called like the red button problem, right? So you want to have visual understanding of what's in the image or in the sequence of images in the video as well as correlated with the audio in order to get the complete meaning of what's going on there.

So how do we solve this issue? This is a data processing issue, like using a different type of embedding model, like a vision language type?

Yeah, well, that's a great question. It's actually quite complex, and there's a couple of ways to do it. There's— I would say there's 2 types of approaches today. One is still to try to translate it into text, right? So you take the video, you transcribe what you can, but you also, in parallel to this transcription, You have a visual model, like, I mean, a lot of the frontier models today can understand video too. And so if they can tell you, like, you take a chunk of 3 seconds, you transcribe it, you also say what is happening here, you sort of correlate them so you get a textual description of kind of what that piece of 3 seconds means and you use that. That is a high processing on the— Yeah. Sort of on the ingest side, but if you solve it correctly, then it kind of works. It makes search and retrieval really fast. There are starting to be these really good embedding models for video too. You can just give it the whole video and say like, tell me where somebody is clicking a red button saying something or something like this. They're not as good yet. So they're kind of in research. They're being Progressing really fast. So my feeling is, from what I know, is they're not necessarily good yet, and I haven't seen enough benchmarks to convince me that they are good enough.

Are companies using it though?

I've not seen anybody using it. I think people are intrigued by it because it supposedly is the right way to solve it, but you have to make sure that the models are good. And there's also— visual has another complication, which is Let me give the images an example, not video, just to make it simple, but even in images. So let's say I have this image we see in front of us in the slide. If I try to tell a model to summarize what's in the image, it'll probably notice different things based on my prompt. So normal person might say, hey, it's a diagram of a RAG workflow with a bunch of boxes including this and this and this and this, and the arrows go from here to here and do summary. But what am I interested— what if I'm interested just in the color scheme or something? You don't know what people are interested in images, right? So sometimes specifically to the dataset that you ingest and the use case in the enterprise, you might need to use different techniques depending on what the goal is and what the focus is, right? So I wouldn't say it's a solved problem either. Like it's kind of work in progress. It's really hard. It's getting better all the time. Thanks to a lot of fantastic research in the space in academia and frontier labs, but it's a very interesting topic.

In my use case, what I'm seeing is my claim that I need to verify could be very, very complex sentence. And oftentimes when I do LLM call to do verification, it can only verify like half of the sentence and it doesn't have the complete understanding of what, you know, I need this whole thing to be verified. So it's very nuanced and complex.

There's a couple of things here. One is, again, I think the agent The agent and even maybe an extension of agent is now we call this harness, harness engineering. So I think harness is like a better agent. It's agent harness. It's the same thing. It's just a better implementation of it. And so I think good harnesses of agents, good loops of agentic loops sort of really help you. They're really good at identifying the pieces of your question, breaking it down to different subtasks. And this is what you need, I think, for your use case. And then tackling each one of those use cases. If everybody in the audience here is using, of course, Claude Code or Codex, you see that they do that all the time. They even tell you, here's my 5 tasks I'm going to do and do them one by one, right? So I think that's becoming the way that these types of problems are solved. And I think that that's why I'm kind of gravitating with you here in the conversation towards this. But I want also to mention another interesting part that we didn't talk about, which is knowledge graphs. And it's not directly to answer your question. I think the agentic harnesses are where this will help a lot, but I want to talk a little bit about what are knowledge graphs and why they're important as another technique to do better retrieval. So there's a lot of work that has been done on building knowledge graphs in the last decade or so. Lots of companies in the space that you can have knowledge databases with. And for those of you maybe who's not familiar, it's essentially, think of it as a graph of nodes and edges and you build a relationship between them. So for example, the sort of common typical use case to demonstrate is movies, right? We have nodes could be movies, they could be actors, they could be directors, they could be All kinds of other participants. And there's arrows between like, oh, in this movie, this was the director and this was the actors that participated, et cetera. And you can start asking questions that are really interesting. Like for example, in the movie Inception, sorry, like let's say what other movies the director of Inception directed? So to answer this question, you gotta go from the movie to the director and then look at the other nodes that are related to that director and answer that question. In semantic search, that will probably usually not work, right? It's very hard to make that work. And so a lot of these are called the multi-hop questions where you need to go through a graph and sort of like a relationship, and it's a very powerful tool to do that. Now, the problem I've seen is that it's also So a lot of people get really excited about this, but they have never done a graph. They haven't built a graph. And so the challenge is that it's really expensive to build and maintain. So what I usually say is just really think through the ROI. Of course it's good to build a knowledge graph. Of course it could be helpful, but if it solves like 0.1% of your queries, it gives you a boost of 1% out of a You know, 10,000 queries a month, are you ready to spend, you know, the team effort and the effort on infrastructure and maintaining the graph over time? So it's really useful when you have really important use cases that are critical and a significant amount of your queries in your AI system are the type that would be, you will benefit from that.

When to use knowledge graph with your RAG system is when you have a high-stake use case. And if your data actually fits this pattern of more relationship, even with such use cases, it's still going to be expensive, right?

Yeah, it will be expensive to build, maintain, not just technically, but also just the graph itself needs constant changing, right? And you usually have to have people do that. And so I think it's, yeah, it's a mix of high impact or high importance queries with a significant amount of them that it matters, right?

Speaking of maintenance, embeddings need to be maintained. For instance, if I have a, let's say, e-commerce store, and they may have new products every week, or product could change as well, how do you maintain that system in production?

So it's a really important problem, and I'll mention 2 things about this. One is, yes, absolutely. Think about even a simple use case. A very common use case is to index all of your Google Drive. in the embedding system, in the vector database, right? Google Drive documents change every day, right? So if you have, again, 10 documents, it's not that much of a problem. You just reingest them when they change. But when you go to production, it becomes a really difficult thing because you don't want to ingest everything every time, right? So you want to keep this refresh algorithm that sort of knows what's inside the vector store and only Triggers on a change. So there's 2 ways to do that that are typical in mission-critical production systems that I've seen. One is you could store in your RAG pipeline sort of a hash of the documents. And when you reingest, let's say daily, and just go through all the documents, calculate the hash, and only do the ones that change. That's still a reindexing, but a fast one because you skip like 99% of them. A better way I think is, but it depends on whether the source system supports it, is to have a, I think it's called a kind of trigger-based, right? So let's say that Google Drive, if we take that example, or it could be S3 or whatever other blob storage, right? Could tell you every day, send you a, like an REST API call to some process that sort of It says, hey, these 5 files have changed, go reindex them, right? That is probably a much more efficient way to do it. Uh, but not all source systems support that. So sometimes you have to do the push, sometimes you have to do the pull.

Um, building this system, which component is the most expensive?

Well, there's expensive from kind of, let's say API cost. It's expensive from a systems perspective, like the machines are expensive from building it as an engineer or data scientist, right? Did you have any particular one in mind or do you want me to cover all 3?

Anything that you've seen very common in production that people aren't paying attention to, like when they're building the prototype, right?

So I would say, let's start with the retrieval pipeline here, right? So vector store has become pretty commoditized in a lot of ways. There's a lot of options. It's super easy to do right now. Again, If you have a big vector store in production, you have to remember that you have to maintain it, re-index it once in a while. But I don't usually see that as a thing that pops out in people's head as like, that's my main pain point. Now, if you want to build hybrid search or re-ranker, that becomes an engineering effort or machine learning team effort to build. There are existing re-ranking models you can use. Even the embedding models keep changing. So a lot of the The effort here is around keeping those up to date. There's more newer ones that you can utilize. And part of maintaining a system like this is being aware of the new ones and upgrading. And then you have to test everything and evaluate that nothing has changed and everything became better and not worse. So we haven't touched evaluation, but we should maybe touch on it in a minute. We should talk about this.

Yeah.

But part of this is it's not so much about cost. Even the GPUs you have to run these models are not That expensive. You can also run those on remote, like an API, like the LLMs. So depending on your setting, those are not super expensive. The ingest itself could be really expensive, right? If you want to, let's say, ingest a million documents that are all really complicated PDFs with tables and images and maybe videos, that could be really expensive. And when you think about it in production, Yeah. The first ingest is usually the most expensive one, right?

Makes sense.

If you do refreshes and it's not that much, it's 1% of the documents every day or week, it's not that much, but you have to think about this in terms of, I'll give you an example. You can think of some companies have documents that are 15,000 pages PDF, okay? Real documents that they need to ingest, right? So you need to think about, and this is the difference between a prototype and a production system. So you can build a pipeline and then page 4,000, it goes out of memory and you have to restart it and it doesn't work again. So you have to think about, and this is the cost here is a lot of engineering, smart engineering and how to build it so that it really can handle messy data, lots of data, long data. And whatever pipeline you do to extract the text, to chunk, to do all the processing that you need to do has to be very scalable. And part of it is also adding a lot of instrumentation so you can start seeing what the problems are and solve for them. It's hard to predict upfront. Every company have so much types of data and it's all messy and you always find these edge cases. Oh, this type of file with this type of image always fails and I have to fix it. So it's a continuous journey. Yeah. My recommendation is really to build with flexibility here initially. So you, and a lot of instrumentation so you can start scaling your data and see what happens. Now on cost, then you go to the generative part, right? The generative part is, well, that's, that's a cost, you know, right? If you call OpenAI or Anthropic or Gemini or something else, you know what the cost is, is per API cost. And usually the cost here could grow if you have a lot of queries every month. If you call embedding and re-ranking also in an external one, then that's the cost. You can also use the open models and host it yourself. You can take Gemma 4 or DeepSeek or QN or one of those. GLM-512 is known to be like a good one today, but then you have to host it yourself on a GPU machine or maybe a big GPU machine. So that could be also a big cost. These models are awesome, but self-hosting them is not always easy and costs you. You have to have a machine. Many people do prefer that because if you, depending on your, you can do the math and see like, if I do this for a year and I buy a machine, and the cost of the API call is this, and I have so many queries a month, what does it come down to, right?

Mm-hmm.

So that's where a lot of the costs come in. I think in terms of engineering, this part is pretty easy today. You just call to an API, or if you host it, you just have to know how to use products like VLLM and things like this. This part is more complicated from a team perspective. You have to really understand search and retrieval pretty well. I think that's what I would say about cost. We didn't talk about hallucination detection. And maybe we can touch on that just briefly in this point is I think of cost also as the cost maybe more holistically of you work so hard, you pay so much money to vendors to build a system. You want your users to be happy, right? You want the results to be accurate. You want Angelina to know that the 95 is fixed to an 85, whatever it may be, right? So I think investing in quality across all of this pipeline Including detecting hallucinations, using best-in-breed LLM, using the best models here will help your users look at this and say, yeah, it's good, as opposed to it's a new system. We have 50 engineers working on it for 6 months and launching it to the company of 10,000 people, and then everybody's like, I don't care. It's not really helping me, right? So you want to make sure that you do all this work and you really worry about quality end-to-end, including whatever—

Yeah.

And if you need a knowledge graph here, you can embed knowledge graph, whatever you need to do to make it really valuable to the end users.

So this is the part when we want to have a good, a robust eval system.

Yeah. So eval is a whole other topic. So let's jump into that because I think it's important. So, and I want to talk about that and then maybe we can talk about evaluation of agents too, a little bit because it's also interesting. Starting with RAG, what you want to do is you want to know that the result that gets sent here back to the user is a good result, right? And there's a couple aspects to that. So the first one, usually you want to check on the retrieval. Did I actually get the right chunks of information? If you didn't, it might be that your search is not good enough. Maybe the vector search is the only thing and you needed to rerank or something. It could also be another thing you may have used if you just use semantic search. When you choose embedding models, you can choose different sizes, how many, what the size of the vector. There's a lot of different things that could affect this. So you want to, you want to first check your, your output of the retrieval pipeline to see if you actually got for every query, the right set of relevant documents that you expect to have been for this type of query, right? In your case, in your use case. If I ask something about one part of the document, did I get the relevant parts that had— if I asked about the 85%, did I get the 95% or whatever it may be, right? When you think about it, this is a really hard problem in production to do. So typically how people solve it is they have a bunch of queries, let's say 1,000 queries that you have as your eval dataset, and you have to specify what are the right Chunks of data. So it's not the document, but it's the chunks of data that would return. It's really hard to sit down and write that up if you think about it. Like you said at the beginning, like, I don't want to go to all these documents and sort of figure out what the right places are. That's what's required here, right? If you think about it. Yeah. So there are good metrics for this that people can see, but in the evaluation chapter, we mention all of them. It could be precision or recall. traditional metrics like this, but the real problem is not the metric, is how do you collect the data for this? Now imagine that you also have a million documents, not 100, and then the documents always change. So how do you know, like tomorrow the documents change and the chunks that will return as a top chunk is different? So how do you keep that loop intact?

Yeah.

It's a really hard problem, and I'll mention a solution in a minute, but let me just finish the overall here. The next step is, of course, you want to look at the generative output, like what was the output response? And you want to say like, is this response correct? Does it use the ranked results properly? Because the LM could go off the rails too, right? It could get 5 results that are relevant and not use them or use them or add its own stuff, right? And you want to also ask sort of the user, like, is this answering your question? Like, it's kind of like a human feedback. Here you usually have a query and a kind of golden response, so to speak, that somebody writes. And companies I've seen, I've talked to a few companies who just sit down 2 or 3 people for a few weeks and have them write the sort of curated result. That's good. It's not bad to do that. Of course, it's not perfect, right? Because there's probably 5 ways to write the same kind of gist of the response in different words, in different ways, and but assuming that you have that, you can compare the generated result to the curated result and see how close they are and score it that way. And so one of the things we talk about in the book is something that was a bunch of papers with a fantastic group of people from University of Waterloo in collaboration with the work I've done with with Viktara, with many other people in the company, is this idea of reference-free evaluation of RAG. So this is Jimmy Lin's lab and many of his very talented students. And so the idea is to— they have a couple of metrics that actually do not require the golden answer or not require the golden chunk in the retrieval part and still have the ability to give you a really good score of whether the retrieved results or these are actually useful. So those are really, really useful techniques. They're not perfect. They have somewhat of their own downsides.

What's the design thinking behind it?

So it uses— so I'll explain one of them just for time, but one of them is called the umbrella. It's the retrieval part. And so what Umbrella does is you say you take each one of those chunks that you get out. Let's say you get 10 chunks that are most relevant and you look at the chunk and you have an LLM call to score this between 0 and 3. 0 being this chunk has nothing to do with the query at all. 1 is it's got something to do with the query. 2 is like it's quite relevant, but it doesn't answer the whole question. And 3 is like it's a perfect answer, right? Just that chunk. Now again, LLMs are not perfect, but with today's LLM, this would be a pretty good rating system. And then what you do after that is you take these ratings and that's what you use to score. Now the important thing about this research, this is nothing really transformational, right? This is called LM as a judge. You can always judge things with a prompt. The nice thing about this technique and the work that they've done at University of Waterloo is that the team have done over many years, they've done research and have asked human beings to also rank these chunks on a number of datasets. And they've shown that the correlation of that particular prompting technique and the particular way they've scored this 0 to 3. Highly correlates with the human answers of the same thing. So that's the power of it. It's not like everybody can invent a system of scoring like this, but the trick is to show that it's actually effective because otherwise you could have biases of LLMs and all kinds of different things. So that's really the power of this. And the other one, this technique used for generation is a little bit different, but it also uses LLMs to automate in a way that's been tested with human correlation. That's really the main trick there.

When do you think a company should think about building their own RAG system? For my use case, maybe I'm just verifying a few blog posts I'm generating, so it's pretty small. I can build it myself when I can tune my harness to my taste, but what if like at a scale, it's maybe a marketing agency, let's say they're maybe doing a million blog posts per month. or something like that. Do you think they should build it themselves or what do you think the solution for those people?

So, you know, so there's, there's, first of all, there's build it yourself with tools like LangChain or LlamaIndex was traditionally a year ago. And there's a couple other tools out there that are open source and you can build it yourself. And if you have expertise in, you know, data science and AI engineering, You could certainly do that. I think you have to think about whether that's your business, right? Is that something that you want to keep building, maintaining? It also depends on how complex it is, right? Do you have multimodal data? Do you need a knowledge graph? All these things compound to complexity of engineering and maintaining a team of experts in your company to do that instead of spending the same resources on your actual business, right? Your actual use case. And that's why companies like the company I work for, Vicatara, and other companies similar to that have come up with what's called RAG as a service and also agent as a service, which says we're going to build all these components for you. We're going to give you an API as a service. So you outsource in a way all of the complexity of building and maintaining all these components and all this complexity and upgrading it to the latest models. to us, and that's why you pay us essentially. And you don't have to be an expert in RAG. You just implement a RAG with the APIs on top of the platform. So that's the kind of buy versus build that happens in RAG and happens in agents in all the other areas.

What about buying an off-the-shelf RAG system solving my use case? Comparing with hiring a consulting firm, build it, Vectera build it for me. Versus like buying some domain tool already exists.

Yeah. So there's a lot, that's a great question because there are a lot of vertical domain tools like this available. And I would say this, like, so when you look, let's take legal as an example, right? There's many, many tools in the legal space that are vertical. And the question is, what do they do? Do they do what you need or not? Right. And I think that's the core question, right? A lot of them do some use cases, not the use cases you want, but I think if they do what you want, they're very easy to use. They're usually some kind of monthly subscription and you don't have to do anything and you can always stop the monthly subscription if you need something else. So I would think that's an easy decision to make when it does what you need. The problems come up with one, when they're missing key features that you need or key Yeah. Like let's say legal review could be a review of a, of an agreement, right? And redlining it. But what if you need also to suggest improvements and it doesn't do that? It just highlights that the, the changes needed or something, and they're gonna do it, but maybe in 3 years from now you need it today. So you just, you can't move forward. So sometimes you have to, to do it yourself. The other thing is sometimes the data that you need to send to it is not like they're SaaS companies and they make you send the data to them. And sometimes that's just a non-starter, right? So a lot of companies need their air-gapped environments and everything has to be in their data center. And some of these companies just won't support that yet.

Let's close out this chapter of discussing your RAG systems. You can close out the screen sharing. I have a question about DevRel for you. If I come to you as a startup, let's say I build an agentic framework. agent ecosystem company. How important is dev relationship and how should I think about strategizing this piece in the first year or maybe the first 3 years of go-to-market strategy?

Yeah, so really good question. So DevRel has become, well, it's been around for a long time, more than I've done it. So I've started as a, as I said, data scientist and machine learning expert, and I I've sort of worked on the DevRel side of the baby in the last 3, 3 and a half years only. Uh, and I love it, by the way. I think it's a wonderful career path for whoever is interested in that. But generally DevRel involves 2 things in my view, like one is, and it's, it's meant for companies who usually have some sort of API, some sort of ways for users to use a product that doesn't require like to call a salesperson and go and kind of implement a full system, right? The way we used to sell big services in the past, right? And so, when you have an API, you kind of want to get developers to use it. That's where DevRel comes in, right? So, it's literally a relationship, building relationships and trust with developers. And so, there's 2 paths to it. One is, as a DevRel, I want to convey to the developer community why the product is really helpful. What's it helpful for? And actually demonstrate it. So build demo cases, show code examples. Basically, it's being a teacher about your product. Here's what the product is and how to use it. And being a good developer helps a lot because it's your people. You can speak the same language and you can explain how to do it really well. So that's one part. The other part is bringing feedback back. to the company, right? So like you go to conferences and events, you talk, you talk to people after your public speaking things, you talk to people at different events and things like this, and you hear a lot of feedback and people say like, you know, product is great, but this thing doesn't really work really well, or this documentation part is not doing well. So I think that's where that feedback is very valuable to the company who sometimes are not aware of it or miss it or something. So bringing that back into the Product and engineering group in a company is the second kind of function of it. But to answer your question for startups, there's a lot more startups now that, you know, I see a huge growth in that in the last maybe half a year to a year in the demand for this because it is a very good growth strategy for a lot of companies where instead of investing in a really expensive sales team, that is expensive and needs to justify its existence very quickly in any startup. That's called sales-led growth. And so you can invest in what's called product-led growth, which is you take sort of the product, you make it available in a self-serve manner to a lot of developers, and then you open it up, you publicize it, you get a community growth. And so you can have 1,000 people using it, 10,000 people using it, 100,000 people using it. And so on. And then what you get in that is you get really good feedback quickly because it's a, you don't get a lot of revenue necessarily from those right away, right? You have to, it's an investment. You get a lot of feedback about the product. Sometimes you even change the product dramatically if the feedback is very overwhelming. And then the idea is that over time, a lot of developers find it really helpful. And then that is a signal of product market fit for the company. And then they call you and they say, or they call their boss and they say, because developers typically may not have the budget to buy anything, but they'll call their boss and say, we need to buy that system, go buy it for us. Or if it's a, if it's a seat thing, they might just buy 2 seats and then 4 seats. And then they tell their friends in the company, eventually you get growth that way. And so it's a really good strategy to get this growth engine going.

What's the means of doing this? Is it, are you thinking of like hackathon style or is there any other means?

Hackathon is pretty common, especially at the early stages where you want a lot of feedback from people building lots of stuff with your product and seeing where it, where it fails, where the docs are not clear and things like this. But there's a lot of other things. So, um, usually you go to events where, so let's say if you target developers, you go to events where developers for your product go to, right? There's a lot of different events. Especially the AI, there's tons of events all the time. You hopefully get to either you as a company, you can sometimes sponsor a booth and get access to all of these developers, or you can do a public, you know, do a talk in one of these events. So that's another big avenue. I think the other thing you do is you just educate. So think, how do you be a teacher, right? So you write very good blog posts and post them on a company that really matter, not And I'll say this with a little bit of kind of like, there's a lot of blog posts today that are, I think we're losing quality is how I would say this. Like there's a lot of AI slop that's going on. I think to be good at this, I'm a big user of AI and I, this is not to say I don't encourage people to use AI in content. I think it's very powerful, but make it your own voice.

Yeah.

Use AI to Help you do it faster if you like, but it's gotta be authentic. It's gotta be something that develop— developers have very short span of attention. If you bore them, they'll just leave and they'll have a bad impression of you. So create good, high-quality content that really helps people succeed in their role. Again, I always think about it as a teacher. Like if I was a teacher at a high school, I'd want my students to succeed. I think it's the same thing, same mindset.

Yeah.

So write blog posts, create demos of how to use your product with very interesting use cases that hopefully will fit what people want to build, things like that.

How do you plan a budget for a dev relationship for go-to-market, like for year 1 for a company? And how long does it take for you to see some results?

Budget really depends. It's hard to put a number on it, but I would say you should think about, you know, if you have, you have to have, first of all, a product that could be usable for DevRel. You should have, you could self-serve, you should have good documentation. If you're earlier than that, it's not going to work out. You have to have kind of be at a stage where the product is usable by thousands of people, right? If you open it up, right? Once you do that, I think you have to have some DevRel person or fractional or something to help you sort of with the strategy. But generally you need a budget for Going to events and you have to choose which ones. And I usually recommend if you're starting out, experiment, because not all events are good for you, right? So try small, like take a small booth in a couple, see how it works. By the way, the events don't only help DevRel, they also help sales. If you have— if you do founder sales or you do— if you have a first salesperson, you can do, by the way, PLG and SLG at the same time. It's very common. So you can do both of these actions at the same time. And so the events help in other ways, but experiment a little bit. Don't think that you know the strategy and go all full force. Try out different things and then you will learn really quickly what helps and what doesn't help, what converts and doesn't convert. There's also 2 modes, right? You can be in awareness building mode where I don't care about customers yet. It's too early. I want to just people to know about me and people to like Try my product and I get a lot of feedback. And a little bit later you say, now I'm really trying to get, you know, I'm trying to build a business. I'm trying to get revenue in the door and I really care more about converting some of these leads and some of these PLG motion into actual sales of enterprise customers. And that's a little bit of a different phase. Both are important, just happen in sequence. I mean, since the day you start, probably 3 to 6 months minimum. Just because, you know, you go to some events and it takes time. Like you can go to an event tomorrow. Sometimes it's—

And you go by.

Yeah. They don't have an event until November or whatever. Uh, you go to meetups, you do talks, things like this. I think also the other thing I would mention is I know there's a lot of people who think of DevRel as a, how do we measure it? How do we know if— forget about how long, but how do we measure it? And I think measurement is really important. But, and let me actually say how you measure it, right? Like you can measure by conversions, like how many people actually heard about your product and signed up to the product. You can measure by not just if they signed up, but do they use it? So you can measure a lot of stuff in your product, but I will say that I like to think about it as do as much science as you can, but don't over-engineer this. Because a lot of it is hard to attribute to, right? So the company does usually many things. There's events that are unrelated to DevRel. There are things that are happening. There's talks, there's press releases about the company and investment rounds. And it's sometimes hard to know when it was your direct influence versus somebody else. So I think of it as like, yes, try, but don't spend 3 months just designing the mechanism to measurement and making it accurate. I'd much better have you spend the time actually doing the work. And directionally, if you see I did 6 months of work and I got 5 customers, then yeah, it's not working. It's very clear. And if you get 10,000 customers, you're like, oh, something is definitely working. And then you can fine-tune and sort of dive a little bit deeper. There's a lot of, of course, mechanisms, right? Like people in traditional marketing mechanisms, how to measure Like if you give a code somewhere, you measure where the code came from and stuff like this, but don't overthink that too much is my suggestion.

I think actually 5 customer conversions and 10,000 are actually really good results because either you have PMF or you don't. The challenge is when you have 1,000, right?

That's a great question. I don't think it's easy to tell, right? So I think that's where, that's why startups are hard. I mean, Yeah. Forget about DevRel, like even as a founder of a company, when I run the company for a year and I don't get the amount of revenue that I need, it's like, are my, is the people not selling correctly? Is the product not good enough? Uh, is there too much competition? It's really hard to tell. And it's the same type of problem. So I think what I would suggest in that case is do what you do when you're a founder. It's the same thing. It's just think about it from first principles, come up with new ideas, try to At some point, founders tend to be, you kind of want to have the belief until the end and go all the way and like, I'm going to do this. I believe in this. Like you have this unreasonable belief in your product. That's what drives startups to success again and again. But at some point, but many startups actually, that belief is wrong. And so at some point after enough evidence, you also have to have a little bit of a red bulb in your head that says, Well, I gotta change something here, right? So I don't think it's a DevRel thing. I think it's a company thing. DevRel, I mean, if you have, you know, DevRel may not be necessary in a company if you move to full sales-led growth and you say DevRel is just people not interested enough to do it. And that's a fair thing to do.

How do we as engineers or data scientists adapt to this changing era of leveraging AI, right? There's, reg is dead is one topic and engineers are dead is another topic. Data scientists are significantly affected as well. We don't require that many data DS teams anymore. So what do we do?

So I do have some thoughts about this. First of all, I recognize that it's a problem and it's not just a problem. It's a pain that impacts real people with real families. And, but I do believe myself that it's a little bit of an overshoot of the companies, right? They're sort of like. They're reacting to something they hear and they don't even know what's happening. I'm talking about the big companies that are sort of firings of like 5,000 people or 10,000 people or things like this. And yes, impacting data scientists too. And people are afraid of like, are the coding agents going to replace me? So my thought on that is twofold. One is, and I guess my advice is, my feeling is that we're going to end up in engineering and data science being— it's not a word I invented, but like directors as opposed to actors, right? So the coding agents will write the code. They're becoming better and better all the time. And I think what remains a really critical challenge is to understand what we need to build and guide it in the places where the coding agents are not doing a good job, like fundamental things that are not obvious, that there's 5 ways to build something, architecture. Those are where I still think there's value, and this could also transform and change over time. But I think to me, the best engineer or data scientist of the future will be people who essentially augment themselves effectively with the coding agents, right? In a way. I, by the way, believe this is going to happen in other professions too. Like anything, think about like, I don't know, I'm going to pick a weird example of like Somebody who does digging in a site for history, right? And they find something and they say, oh, I think this is from, I don't know, Egypt in 2000 BC. And then they have to go to the library and talk to all their friends and da da da da. What if they just shown it to ChatGPT, got the response probably mostly accurate, And then did more digging at the same day. So I think everybody can augment themselves if you figure out the right way. Legal is probably the most incredible example that I think a lot of lawyers are afraid of, but generally, like a lot of legal contracts that are, I probably wouldn't use a legal LLM to protect me against, God forbid, some really high impact case that, you know, of whatever in court, but If I have a contract sent to me by a landlord that I want to review and send feedback, it's probably okay. The impact, the risk is not that high. Probably find some things. And LMs write really, really well emails back and are very articulate. So it's got a lot of use cases. And so being able to augment yourself in every profession, including in coding and data science and analytics, just to be more effective, I think that's what we're going to see. We're going to see that. Employees become that, and the ones that are more proficient at that will succeed more, in my opinion.

Do you have kids? What do you tell them?

Well, it's complicated. One of them is actually in the profession of building, you know, actually video models. So it's, so I think he's worried about other things at this point, but I do tell them Kind of the same thing to sort of like figure out if they care. Not everybody cares about this. Not everybody's afraid about it at the same level, but I tell 'em that, yeah, you have to adopt it and embrace it. And I know it's been said many times before, but I do feel it's a little bit like many other technologies that have been created in the past and we as a society adapted to it and our jobs became different. Now, that does not mean— actually, I'll say this this way. What I'm concerned about is, as a society, what we do with this. Because I think there's a way where this is actually benefiting everyone, but because of how finance works and how sort of societal differences between the poor and the rich and how governments around the world react and deal with this, I think that's the risk I'm most concerned about for me and for my kids.

Yeah.

Which is, will this result in more, you could see this result in more layoffs and companies abusing this to make more profit. They might lose talent and they might eventually not succeed, but in the meantime, there could be a lot of pain, right?

Yeah.

So I hope that doesn't happen. I was fortunate to have a bunch of high schoolers invite me to, here in the Bay Area, it was a high school and they invited me to their, they have a computer science club and it was all people in last year of high school and they all got accepted to their computer science program for the next year in college. And they were just looking at me. This is about, I want to say like 9 months ago. And they were looking at me and saying like, did we make the most incredibly stupidest mistake ever, right? What are we gonna do 4 years in college, right? So I told him something similar. I said, no, look, companies now equate a junior, like a junior engineer with somebody that just graduated from college. That's like one-to-one, but it doesn't have to be. You can graduate from college and have the capability of what would be like a mid-level or a senior engineer today. So use the next 4 years to learn how to be that. You actually have space to do that during college and learn how to use the AI tools as effectively as you can and make sure that you come out of college being this hybrid computer scientist of the future, right? Or data scientist of the future or whatever you are. I think, I hope that colleges and universities support fast enough, support their students in doing that and change the curriculum to support that.

I don't know how they're going to benchmark themselves to say I'm coming out as what used to be a mid-tier engineer. I don't know if that equates to medium-tier engineer anymore today, or maybe in the next 3 to 5 years.

Yeah, I think all those classifications are changing too. Like, who knows? Like today, I mean, I think it's still important in universities to learn the fundamentals of computer science. Those are really going to stay really important. Fundamentals of statistics, machine learning, how it works, kind of understand it. But I think both the grading in universities and also the acceptance of new employees to a company, how are they going to do the testing of write an algorithm to solve Fibonacci in 5 lines of Python? Like, that's worthless. Like, it's not really valuable. So I think the way people evaluate candidates is also going to have to change a lot. And I wouldn't be surprised if they'll say, hey, here's ChatGPT or Anthropic coding agent, and we'll give you some exercise and we see how you do with it together and they'll test how much you overcome some challenges perhaps that are sort of in that working together with the coding agent to produce something that's high quality and testable and is the right decision in the process, things like that.

I've heard of some of the founders I interview mentioning that they're incorporating AI tools in their interviews. already. Thank you so much, Ofer. Thanks for sharing so many insights about building a reliable production-grade RAG system. And, you know, giving me some thoughts about my system as well. Also, I really appreciate your insights on getting us prepared as well for this changing world. And hopefully we are all seeing a bright future of everybody, you know, being enhanced rather than, you know, being replaced.

Exactly. No, thank you, Angelina.