Good Morning, AI : Capitalizing on your private data with retrieval augmented generation (RAG)

Hello and welcome back. We are doing this wonderful series on the power of search in the world of AI. I'm so excited for it, but this is our last one, but we're gonna talk about probably my best topic, my favorite topic so far, which is really how do we build search solutions? Like one of our, uh, in the last episode we were talking about how kind of complicated it can be. And one of the biggest challenges and risks is how do we make sure that we build, you know, a search solution that doesn't grab the wrong answer or make the wrong answer. And so we are here once again with Serena Cho from Elastic. She is the Director of Product Management for search there. And we've just had such a fun time over the last couple of weeks talking about these sessions. And I would love to really understand a little bit more about, you know, we've talked kind of high level about search and, and how search is evolving. We then talked a little bit last time about the power of, of search and some of the risks and concerns and kind of responsibility of building a good search framework. But, but now let's get a little bit more specific about like, how do we actually build a search solution that doesn't lie, that doesn't accidentally pull the wrong information? Are there some specific, you know, examples or suggestions or even architectures that you might be able to provide to us to get us on the right track?

Absolutely. Thank you for having me here again and letting me, you know, just go at length about the benefits of semantic search because it's really relevant for the topic that you just posed me. A lot of customers, they want to take this leap into these systems. They want to incorporate these machine learning models, maybe for semantic search, maybe even large language models. Again, within that constraint of not exposing themselves to risk or having horrible hallucinations, the memes that come from these types of experiences have been very enjoyable to read on Reddit, but not great for anyone's like actual production use. So for us, like, if you revisit this topic around like what's the right architecture to help you prevent that type of exposure, uh, being the next meme on your chat experience there, the kind of the core solution set is, you know, again, that access and control and how you've managed your data. These are just clear foundations that continue to be something that we need to maintain as a standard, and there are clear compliance standards to adhere to there. But when you talk about using semantic search in your architecture, I think that there are viewers that might be kind of just familiar with the concept of RAG, Retrieval Augmented Generation. They might be aware of this type of architecture when they're using large language models simply for the purposes of just bringing their domain context, because you don't really want an off-the-shelf large language model that says, oh, you know, for instance, what is my— what's NASA? If you use like an off-the-shelf large language model, then you're going to probably, if you're based in Texas like I am, going to get some interesting conversational response about the space station that's based in Houston. But that's not what you want if you want that to be relevant to your organization. So, you know, you have this pillar over here of your own repository of data that you've kept safe and accessible to the right people. You're going to plug You have to weave that in, in this RAG system into your large language model, and the ways you do that has a lot of nuances that involve semantic search. So in your chosen strategy, I think you had mentioned, Noelle Russell Russell, last time around, like you have to chunk, you have to do all of these kind of iterations and embrace your experimentation here. This is where that's going to come into play, is this architecture as you tune what type of context you're willing to share with that large language model. But also keep in mind too, The way that you host or provide access to a large language model as well in this architecture can be at play. We have some customers who are, are most security conscious and want the most control, and for them, the obvious choice is to, you know, perhaps host their own large language model. Perhaps if they're in the position where they have a lot of talent or that muscle already to do the machine learning operations, they might be building their own large language model, but that's usually in the, confines of what they control and are accountable for. And then when they apply things like being able to use document-level security to say only the relevant documents for my frontline retailers are available to help them provide like easy recipes or something like that against their catalog, they have the constraints here on the retrieval side. They have the semantic search to appropriately find the semantic meaning so there are less mistakes in what's retrieved. They also have a level of control in being able to incorporate second-stage retrieving, retrieval and re-ranking models. These are models that add to your semantic search. They're not necessarily grabbing that semantic meaning, but what they are doing is, you know, they might be slower, but they are incorporated on a smaller set that you've kind of filtered down with that first layer of semantic search. To a smaller set, you can run it through at a pretty fast pace, and these re-ranker models get some very high accuracy. And then along this chain, this becomes your RAG pipeline. You can then kind of have several stages of control to guarantee that you have the right data, identify perhaps PII that's leaking through, ensure the right context that's meaningful to your large language model, and then apply that all together in an architecture that gives you that rich semantic meaning, the precision perhaps from your hybrid search if you're combining maintaining that with an existing keyword search, and then being able to get that high accuracy, high recall without a loss in performance and scale.

Yeah, I think so many of, so many of the things that you just said speak to a lot of the concerns that executives have and certainly decision makers around their data science or, or machine learning type of teams. You had mentioned last time very quickly about ML Ops and how we often kind of underestimate how much operational support we need when we get into this and start building our own solutions. But you also mentioned some of the safety controls. And I think this is another area where at the beginning, you know, I, I, I talk to lots of executives. I, I run these workshops, they're called like AI red teaming workshops. And it's a little different than the cyber red teaming because with AI there's a lot more to be worried about than just adversarial attacks, right? Like there's benign attacks, there's like, uh, just the concept of accident— not intentionally setting the right guardrails around the type of solution that you're building. And one of the reasons why I, I loved what you said about kind of this retrieval augmentation that happens with this RAG architecture, because it— most companies don't realize in the very beginning when they start this journey that they can put a box around the data that their model has access to. They can also configure some of the parameters on the model itself to change how creative it is in creating the answers that are supplied to a user when they ask a question. Most of them just start asking questions of a model, point it to some data, and they're, they kind of walk away going, well, that wasn't a terrible answer, but they don't really have the comfort or confidence that that model will not, you know, misbehave once it goes to production. I often use an analogy. I have my little baby tigers right here. You know, that a lot of these models, especially with search capability, like they're like, it's like a little baby tiger. Like it starts off being pretty cool and nifty and people wanna be around it and they wanna help contribute to it and, and support it. But no one's really asking hard questions like, wow, like what happens when we open this search up to our customers? What, what happens when we, you know, what does it eat? Where is it gonna live? How's it going to like, you know, how do we make sure it doesn't hurt anyone? Like these types of things I think become very important questions for all AI projects, but specifically specifically for those in the world of search. And I'm just curious, as we kind of think about, you know, this concept of, of RAG, I think most organizations, they may not even know it, but they might be using a model, like the services provided through Elastic that are already RAG friendly, right? They've already got this kind of architecture. Can you talk to us a little bit more about like, what are the benefits of implementing a RAG architecture? I know you talked a little bit about them, but just for those of us who are like, all right, if I'm trying to sell this to my CFO was like, hey, we want to build our generative solutions a little differently. What are some of the benefits that I can say like building it this way would achieve?

It's funny because you're— a side note earlier in this question around the, you know, just kind of configurability potential with large language models. I want to detour a little bit before I answer your question more directly. It just kind of came to mind to remind me of this customer story that I heard that I thought was really clever. You know, banking institutions by default need to be by design our conservative, most conservative offer like of search experiences, right? You need your banking institution to provide a very reliable and very accurate response. And they are some of the folks that are out there that are trying to build chat interface to accelerate their workflows and benefit from all the things that we've been talking about. And the benefits for them in employing RAG architecture is to say they already had a chat— not a chat-based experience, but they had a search experience that they stood behind where you could access transactions, you could be able to communicate with the bank employees. Like, various search experiences exist today. And I think the benefits of going to RAG is that for them, they're only one step away from responsible use of large language models with a RAG architecture. Because if they stand behind the relevance, the control that they have on that initial portion, then really it's just kind of a slight modification to choose your large language model, to use the configurability of the prompts like you indicated, and then they have a pretty robust, very secure usage that leverages their existing architecture. And that's the beauty of RAG to me, is how well it fits on top of existing search experiences today and how easy it is to just like also peel back various parts of the chain and up-level portions of it. And to go back to that configurability, I think the use case with this bank customer They talked about wanting to offer a very good summarization experience, but they also had very little appetite for very wild and crazy hallucinations, things that might talk about mud pies, you know, whatever you could imagine. They don't want any of that. So they have this strategy where they took existing prompts that they knew that were already existing in their FAQs, the common answers, the staff was trained to respond to common requests when they get called on the phone. They had a playbook. And what they did is they applied that as context along with their domain information into this large language model, and they used those kind of configurable levers to say, this is the barest amount of configuration. You can't— you can only change so many words. You can only modify certain aspects to this degree. And with that, they were able to guarantee a much higher rate of a more personal perceived experience on the responses. A more human intuitive touch, but still, you know, know that it was highly accurate. It was leveraging their many years of customer service and their knowledge bank from their playbook, but then still get the benefits here of a RAG system. And because they were already, you know, building this on top of the search experience, they were a perfect shoo-in for a RAG architecture.

You know, that's so interesting because I think one of the, you know, and you inadvertently also talked about some of the biggest benefits of the RAG architecture, right? Is that one of the things that I, I just recently uncovered, and the way I typically learn from companies leveraging AI is unfortunately on like Reddit or on, in, in the press where it might be an AI gone wrong story. And recently a customer that I work with, an airline, ended up being super overzealous maybe in providing a chat capability to their customers. Very excited. Put it out to their customers. And then their customers started asking questions and actually requesting information about refund policies. And because they weren't really aware of these controls that you talk about, like these different levels of controls that they could put on the model, gates that they could put in place, ways of kind of locking the model in, that they ended up having a model that generated refund policies on the fly and in production. So like, these are things that were like, oh, Wow, that's, that's not good for business. And it certainly makes everyone a little bit less comfortable with the series of solutions. Like they're trying to help their customers and yes, it sounds like a good idea, but at the end of the day, they kind of overhelped their customers and really had a negative impact on their business. And I think that's what RAG offers, right, is an architecture that gives you more protections and allows you to, you know, set yourself up for success. So I'm so grateful that you had a chance to talk us through that and give us a really great example, uh, to kind of learn from. Speaking of learning from, I would love for us to kind of end this episode with, you know, we've been talking about search. People are probably like, oh my gosh, I can't wait to get my hands on this. How do I test it out? What are— and often one question I get all the time is how do you actually learn and keep up with all of this, right? You are the product leader for this organization. Like, how do you keep up with all of this information? Stay up to date with your learning?

Oh, it is an interesting challenge that I embrace every day. Um, there are so many sources of information and also misinformation. There's actually quite a few people who know just enough about the space where they talk about some things and you're like, that sounds vaguely right, but not quite somehow the nuance that I'm looking for here. And so I think for any learner, any builder, whether you are choosing to peruse something like Reddit or, you know, read your— like, listen to your favorite podcast. I think the provenance end of that information should always be couched in, can you go apply that? And I think the thing that I am really excited is for listeners that have enjoyed, you know, our topics of discussion. I want to give you access to a really great set of tools where you can go and apply that and see that this is indeed the shape of the control, the access, the ways that you can incorporate this. We have a Search Labs website. It has some fantastic tutorials, GitHub repos with code examples. You could, in the next 30 minutes, go to the website, follow the tutorial, come out on the other end, and you now have a RAG application. That's not something that I promised to you, it's something that you delivered to yourself because you just took the next 30 minutes to kind of lightly read on that. The site also has several deep blog posts and thinking from the designers, not just from the product team, but from the engineers who are sitting there thinking about the latest techniques for information retrieval and distilling that in a way that we communicate to you what we're excited about, what you can leverage almost immediately.

I think it's so fascinating. I mean, that's one of my— I love doing is just going and practicing. For those of you who are in my AI Leadership Institute, we are gonna be doing live lessons and live learning labs on this very topic. I got my own access to this area. I'm super excited to start testing it out. If you are gonna test it out, I hope you will show us, demonstrate what you've ended up building through this tutorial. I would love to see it. It's been such a great series with you. Thank you for talking about enterprise search with us, sharing your personal experiences, and sharing your vision for the product, uh, that you are building and that you obviously love. Um, but so grateful to have you. Thank you, Serena.

Thank you so much.