Good Morning, AI! Deeper dive into semantic search!

Hey everybody, and welcome back. We are here with Serena Cho, the Director of Product Management for Search at Elastic. And we had a conversation earlier about the kind of art of search and how search has evolved from a few years ago when really we'd go to a company and enterprise search was not amazing. I would often just go to Google instead of trying to find something in a company website. But now we're seeing that evolve and we're seeing kind of new ways of thinking about about search, new ways of not only improving customer experience, but also improving even employee experience, right? Our ability to search for things inside our own companies. And I'm so excited about it. But one of the things that often, you know, I, I, I talk about kind of generative AI with a lot of clients and, and these clients will come back to me and say, oh my gosh, we're gonna use an LLM, you know, we're gonna use ChatGPT and we're gonna stick it in front of, you know, our huge, you know, SharePoint of, of files and it's gonna help us build a better search experience. And I often kind of think immediately like, wow, I'm not sure that's going to work. And so I wanted to invite Serena back to talk to us a little bit about maybe the more technical side of things, right? Like, how does this actually work? And before we get into building something and talking about what it takes to kind of build this solution, maybe we could take a look at the background of how search works and what's the difference between kind of the old way we search and what this new capability is with this term like semantic search. So Serena, can you talk to us a little bit about kind of what those terms mean and give us some context for the rest of the session?

Yeah, absolutely. And thanks for having me back. I had so much more to say, so I'm really excited to get into the details here. I think last time you talked a little bit about like kind of this journey between almost like a zero to one, right? You start with this, generational acceptance of keyword search. You kind of land in the middle where you're Googling things because you're dissatisfied with this, like these many different search experiences, and then you try and go to large language models as like a cure-all. Well, semantic search is a very exciting topic for me because the real answer for how this all ties together is in the concept that you just raised around semantic search and It's a really nice kind of intermediate point or even final destination for some in people who want to leverage AI, really don't want to potentially take on the risks of incorporating large language models or even the cost, but still want to gain the benefits of having more automation, more intelligence, more semantic meaning incorporated into their search. So semantic search is a really rich place for many builders of search experiences to dip their toe into the waters because you can, within a few clicks, using, for instance, our, our own proprietary model that we provide. It's a zero-shot, out-of-domain, general-purpose model that increases your relevance. When we talk about semantic search, it's particularly suited to this purpose because you will be able to leverage a machine learning model and brag to all of your C-suite, to say, yeah, we are powering our search with AI because you are using that technology. You are leveraging that innovation in choosing any embedding model or other proprietary models that are out there. The world is your oyster and the catalog of what's available in the market is growing exponentially. Just take a look at Hugging Face. If you're like our technical team, you're probably in there every day watching the leaderboards and just trying to see like who's winning because our team is very competitive and wants to know where our models stack up in relevance, how we can incorporate the best combinations of these to distill this into semantic search. For people who aren't as in the weeds as I am, really, I think the summary is just, do you want to have a search experience either voice or through your fingers as you type, to be a little more advanced than blue or black shirt? You probably do. You probably want to say, you know, where's the nearest, you know, Target that I can go and get a blue black shirt? There's so much more rich meaning in there. And then there's also a lot of collisions between colloquial terms and, you know, business-friendly terms that, you know, you as a user, when you're constrained to just like a traditional search, it's not going to be able to tap into that. But when you bring in a model like our ELSA model, then you'll be able to tap into that semantic meaning, start leveraging the connection points that are trained into that model, and then up-level your search experience almost immediately. It's a really rich place to add relevance and power to your search experience.

Yeah, I think this is something, a trend that I'm seeing kind of throughout the world of generative AI, which is kind of this concept of building built-for-purpose models.

Right?

Where, where maybe a year ago, a lot, a lot of us in this space were like, oh my gosh, do I need to go build my own semantic, like, model? Do I need to go create my own, you know, repository of embeddings? Like, and that was true back then, but over the year we've started to see a lot of maturity in, in companies that have expertise in this area. For example, what you've built at Elastic, right? These built-for-purpose models that now shortcut our journey, right? So that I now don't have to do that work myself. Could I? Maybe, but most likely the problem I'm trying to solve is not creating a model that's, you know, an expert at this kind of building this semantic layer, but instead on solving the actual business problem that I'm going after. And so I think it's really interesting for, for people to start understanding and, and looking at different ways to solve the search problem that might not just be, hey, put a large language model in front of it, right? I, I also think of sometimes com— companies, when they start thinking about search, they don't realize that right behind this, you know, large language model like a ChatGPT, that if you ask ChatGPT to go against an entire file structure, that A, there's a couple things that happen. One, the latency is pretty intense, right? It having to go and search through all these files and find meaningful information. Someone has to like chunk it up and serve it back to the model so that it knows what is the right answer, what isn't. It becomes very complicated. And one of the things that I saw with like the, the idea of semantic search, which wasn't new, like I've been doing this, you know, for now, I don't know, 5 years, 6 years, but the ability for me to now create kind of an index that shortens the trips that my LLM has to make to get the right answer and then provides me a bit of safety in making sure I know what that right answer is. You had mentioned a couple of things like, you mentioned risk and, and cost. And while cost could probably take us down a, deep rabbit hole. I do wanna talk about some of the risks associated. You know, we have a lot of people often like that are listening now that might be thinking about, you know, making a different choice about maybe even taking on search as a use case at all. Like maybe they just had dismissed it. What are some of the things that they should be worried about as they move into kind of thinking about building a semantic engine?

Wow. You just opened Pandora's box for me on that because You know, I think previously I made a statement about how we, you know, our desires here on the product team is to democratize this access, because when you mention the costs associated, you're right, people are thinking about, oh, should I just jump on this bandwagon, go train my own model? And they don't realize the intangible and very tangible costs that this ends up causing. There is a very opinionated skill set that you need. There's a whole classical set of requirements around machine learning ops that people were just not prepared for, and they can't just go hire that because the number of data scientists, the people who specialize in this, while it's growing, the demand clearly outstrips the supply, right? And so there is definitely a tangible cost in accessing talent and structuring your organization to evolve, to incorporate that into your development process and still be able to meet your outcomes. Because really, at the end of the day, you just want to be able to show people you have the best experience leveraging the technology, and that shouldn't be hard. It should be easy. And so when you talk about this maturity of models, this is what is so amazing about this time for us, is that, yeah, we took the choice to say we have the expertise, we can provide a really good out-of-the-box solution. But we by no means want to lock people into that. Our entire philosophy here is about embracing the ecosystem, really digging into our open roots, but still doing that in a responsible way. We want to enable your choice and give you all the pieces to be able to control the architecture, control the access, control the context, use that really well with the model of your choice. And that doesn't have to be with us. It should be with anyone who has the right technology for your use case. And we want to support that. We want to build a platform that doesn't involve you reinventing the wheel or having to completely restructure your entire methodology for how you're building your product outcome. That shouldn't have to change. You should have some reliable foundations to get to the same quality that you've been able to enjoy for many years, but then incorporate these new technologies in there.

Yeah, I think it's interesting because it allows a company to kind of be in different levels of the adoption curve, right? Like those that are already pretty advanced and do wanna have a lot of control can have a lot of control and choose to do that. But those that aren't as advanced, right, they can start with a model, they can evolve their experience. Uh, I always tell people like there's kind of 3 options when it comes to leveraging, you know, existing models. Like, one is that you can buy one or rent one, right, where you can use a model that's being provided to you. It's probably not the most, you know, the, the perfect solution for you. Like, it will require you to, you know, understand what the model is providing, use the features, change the configuration parameters. But at the very beginning of your journey, it's better to just use one that's offered so that you can at least start testing your hypothesis. And that's the other thing I always tell people, is that AI is research. Like, search is a research project. You are going to start by kind of thinking, how do I wanna build the best search experience? And then you're going to start testing different ways to do that. AI is not a press an easy button and poof, you've got, you know, your solution. It's kind of like, oh, let me start with maybe the ELSTER model to, to get a feeling for how this solution might play out. And maybe I'll add to it, maybe I will, you know, provide additional context to the model. Like, there's so much growth that can happen, but I think it, the most important thing, which is the last And the last thing I tell people is like, the only way to really learn how to do this is to learn by doing, to start.

Yes.

Um, and we'll talk, we'll talk more about that, you know, in, in maybe a future episode about like what people can do to get their hands on this. But, but for today, I, I think there's one big challenge that maybe not big, but it certainly has shown up on headlines. Like, you know, we're talking about search. The worst possible thing, I think, okay, maybe not the worst possible, but definitely one of the worst things that can happen is someone could search for something and you as a company company could provide them the wrong answer. Can you talk a little bit about like that risk and, and how you're mitigating, you know, that risk without getting into too much of the technical details, but just wanna educate our audience on, you know, what does it mean for a model not to, to fail or to have these errors in it?

Oh, great question. And I think that's, uh, really gonna be key for the folks that you were trying to kind of address in that earlier question who are trying to explore whether they want to even tackle a search use case. What you said a minute ago about experimentation is key. Like, if you're going to embark on a journey to uplevel your search, there's not an off-the-rack solution that fits all, right? It's not one and done. You have to embrace that, sure, any platform you choose will have different affordances for how you can incorporate technologies, different philosophies. I've talked a little bit about our flexibility mindset here. But the thing that really kind of, when it comes down to brass tacks, is you probably have this much affordance where you're like, I can experiment, I can change this one out for a different model, maybe try this one. And you have a little budget and a little appetite for that. But when it comes down to the risk of, like you said, putting that and structuring it out on the public internet and then potentially having this mind-shatteringly terrible response that either exposes you to legal consequences or potentially in some industries can actually lead to effects on a person's physical health with misdiagnoses or maybe the wrong recommendation. The risks can escalate. And so the responsibility to be able to control that data and control control the responses, needs an enterprise-grade set of toolsets. And this has been, I think, to kind of call back to when you said what has been the situation before and after. For us, this has been an unchanging constraint, even with something as simple as keyword search. Yes, it was more explainable and yes, it was more controllable and it was easier for someone to go, oh, if I boost this or if I do this, then It was pretty easy to see how you could constrain your risk. When you incorporate a model that is basically a black box that you have to trust is intelligent and built on the right sources of data, every day we're reading about new exposures of datasets that have horrible biases that's built in that now inform the relationships that are coming in for models. That seems frightening. But we shouldn't be frightened to embrace the benefits because even with that risk, you have techniques that are applicable to you that are available from the fact that if you care about compliance, there are so many standards that are coming in. And yes, large language model standards are evolving. It has to evolve. It's evolving quickly. It'll be interesting to see where the standards end up. But at the end of the day, you need control over your data. You need to know where it's hosted, who can access it. You need observability, you need auditing, you need to know that your platform has the safeguards for access to that. And something like our platform addresses all of that. It has for keyword search, and it will continue to do that, and will continue to evolve new capabilities to provide further insights and management. Management who can keep you safe.

I, you know, I think it's so important because a lot of companies think as we start telling them about responsible use and how to make sure that they build for safety, privacy, security, they think they have to do it all on their own. And I, I find it's, that's why I'm so grateful that you were able to come on and share with us some, some more interesting tidbits about, um, search and the work that you're doing at Elastic because it, it shares and demonstrates to people that they don't have to start from scratch. They don't have to start they're not alone. Like there is an opportunity for them to actually build on top. And I always say this, like build on top of the shoulders of giants, like go to organizations that do this for, you know, for their core competency as a business and leverage that in your organization as opposed to trying to start from scratch. So it's been such a fun conversation. I can't believe it. We're already at the end of this session together, but it was great. We had a chance to talk a lot more about the concept of kind of keyword search versus where we are today in, in the world of generative AI and using semantic search to build a better search experience. But we still wanna spend a little bit of time talking about how we do this. Like, what is, what is the, the architecture of a solution that does something like this? And that's what we're gonna do in the next episode. So I'm so glad you are here with me. Thank you for coming, and I'll see you next time.