The Map Is Not The Territory – Multimodal Retrieval
Brian and Ayush explain why a single search method fails for real-world data, surveying RAG variants from vector search to multimodal retrieval. They demo Semantic.art, which indexes artworks with multiple representations (CLIP, BM25, captions) and routes queries dynamically to the right search path, arguing that diverse maps beat one perfect map.
Chapters
I'm Brian. Ayesha's here as well, who will take over here in a moment. But today we're gonna wrap up this RAG series with a little bit of a sort of like, um, contrarian view on the field and on retrieval, and even a little bit contrarian to some of the things that you've heard in the rest of this series. I get to go last, which means it's easy for me to snipe all the people that came before. No big deal. But long story short, I want to try to maybe reframe some of the conversation around RAG and retrieval, because I think that we sometimes slip into some, I would say, dangerous habits with RAG and retrieval. A little bit about me. I've been doing retrieval stuff for a while. I previously wrote a book on recommendation systems. Um, Aayush has also been doing a lot of stuff related to retrieval. He and I worked on a project almost 4 years ago now related to retrieval. Recommendation system, and Ayush is also sort of like an expert in computer vision and some other areas. And so you are hearing from people that have spent a lot of time thinking about classic retrieval. And so when you feel that come through, that's the context. To kick us off, I want to talk about this start, uh, slide here, which says the map is not the territory. For those of you that may not recognize what that phrase is in reference to, It is, uh, comes back to this philosophy concept, which is our senses and language and biases create mental maps that we mistake for territory of real-world experiences. Or said more simply, when you walk around in the world, that's not the map. And when you look at a map, that's not the real world. These are two totally separate things. Maps are very useful. But you should not mistake them for the territory which they represent. And this was meant to be sort of a classic, like, philosophical concept to help us understand the disjoint between— or disjoint— uh, disjunction between our senses and what we experience and what matters. What's interesting is if you're thinking about recommendations or machine learning, through the lens of some object and this thing that you want to represent, actually this concept is very powerful, but in almost the opposite way that it was originally intended. It was originally intended as a warning to be careful, to not trust your senses too much. And in machine learning, it's actually like a tool or a weapon that you can use to kind of do a better job. And so let's talk a little bit about what that looks like. So What you've probably heard ad nauseam at this point are different definitions of RAG. And so RAG, just a reminder, is Retrieval Augmented Generation. If we break it down into the different letters, we've got R for retrieval. This means getting stuff outta some store. Augmented means we're gonna use that stuff to help the next task. And generation now means put it into a language model and sort of ask for something from that language model. And so retrieval augmented, uh, generation is purely just this concept of grabbing the context that you think might be relevant for the model and then giving it to the model. Um, this retrieval process is very deep though. Over the last year, people occasionally will toss this term naive RAG in a blog post. And naive RAG is supposed to be a sort of dunk on the people that are trying too uninformed to do any better. They're using naive in the abusive sense instead of the way that mathematicians use it, which is the simple sense. So anytime you hear the word naive, you should ask, is it abusive or is it simple? And in my claim, I think you should think of it as simple RAG, which is you have some vector store and you're going to search that vector store with another vector and things that are similar in vector space are going to come back. This is usually what people mean by naive RAG. Lately, people are starting to get very excited about another concept called agentic RAG. Um, agentic RAG is this very, uh, intimidating term, almost like the complete polar opposite of naive. They say it's a kind of like over-glorify this new concept, which is somehow the language model gets to choose how it searches. So you ask the language model, hey, where do you wanna search, or what mechanisms do you want to search? People are very excited about this because right now agents are giving people the impression that they can, uh, remove themselves from the loop of careful engineering. You may have heard of Hybrid RAG. Hybrid RAG is sort of if you take Naive RAG and you sprinkle on some old recommendation systems and search and retrieval techniques, um, this allows you to search with sort of multiple things simultaneously. The hybrid in hybrid RAG is naive RAG plus some other search vectors. If you go back to some of the earlier talks in the series, they talked a lot about hybrid RAG. You might have heard of graph RAG. So graph RAG is this concept that sort of objects that you might want to retrieve may have relationships between them. How can you use the relationships between objects to further walk down the path of retrieval? Let me give you a simple example. If you do a retrieval for, um, the other day I needed to buy some new coffee filters, and so I went to Google search, uh, Google Maps, I should say, and I typed in the exact kind of coffee filter I wanted. It not just found places that actually sold those exact coffee filters, but even the ones near me. Why you might think of this as sort of like a graph structure problem is because first you want to think about places that sell sell home goods. And then from there, you might want to link out to the additional information that they have linked to them. And so this idea of like a graph is independent things are nodes, and then information corresponding to those nodes. Graph RAG is very powerful. It's very useful for certain contexts. It's also very overhyped. Sometimes you'll hear about multimodal RAG. Here, the modes can be one of two things. Um, we live in a fun world where multimodal means multiple things. One way that we talk about multimodal RAG is you can search with images and text or audio and video. This can be really powerful because sometimes you want to kind of use multiple things simultaneously. But wait, dear reader, there's another kind of multimodal RAG, and that is when you sort of want to search in multiple loci in your latent space. I'm not going to talk a lot about that, even though it is kind of related to the thing here. But multimodal RAG is pretty, you know, pretty exciting as well. But I have to say, I'm pretty exhausted with this list of retrieval. I'm not exhausted because these concepts aren't interesting. As you could probably tell from my rambling, all of these are actually quite interesting and they all have a lot of value. Each of these has their applications and each of these unlocks new sort of sort of avenues. The reason I feel exhausted by this list is because there are all these different terms. They're all this overwhelming terminology and sort of, you know, buzzword bloat. I claim that these are all the same thing. They're just different ways of sort of like talking about it. And so what I want to talk about in this talk is not this is what Naive RAG is, this is what Agentic RAG is, this is what Hybrid RAG is, this is what Multivector. I don't really wanna do that. I'm much more interested in trying to teach you the concepts behind all of this because you could have invented literally all of these. None of these required a very big brain. And so let's talk how my small brain would've approached it. Let's talk about multimodal first. So, um, One thing that you might have had the opportunity to hear about is hypothetical document embeddings. Sometimes we'll say HIDE. I'm looking forward to hearing about Jekyll embeddings, but for now we'll focus on HIDE embeddings. So the concept here is you've got a bunch of documents. Let's call those documents tickets in your tracking system. So these are tickets for your engineers that they need to work through. And if you've ever worked on an engineering team, you know that a ticket consists of the name of the ticket. A description of what the ticket entails, lots of examples of what's important in the product spec and ways that that could go wrong. Just kidding. That's not what tickets are. Tickets are a little title with no explanation and the engineer has to figure it out on the fly. So if you're an engineer and you're looking at your ticket system, you've just got like a bunch of like weird, poor grammar explanations of shit you need to get done. And that's fine. But if you're this PM who's trying to remember, like, do we have a ticket for this? You're not writing the name of the ticket. You're writing some weird cryptic thing that makes sense in your brain because the user just hit the bug for the 19th time and you can't remember if there's even a ticket for it. So what you write is you write, oh, uh, modal doesn't load after user clicked on popover. Okay. That's not the name of the ticket. The name of the ticket is actually like, you know, sort of like a React component, name of component. Doesn't fire when other React component hasn't been ejected. That's what the name of the ticket is, because the engineer that wrote it, when you yelled at him last time, just wrote what they meant something to them. How do you marry these things? One way that you can marry these things is hypothetical document embedding, because you want that concept from the PM to map to something that is in the database. Hypothetical document embedding says, here, large language model, here is all of this vectors that you want to— or sorry, here are all the objects that you want to search for. For, rewrite them in the language that we think people will search with. And so that is one simple example of a document enrichment pipeline. Agentic RAG, on the other hand, is a query enrichment pipeline. I already told you a little bit about what Agentic RAG is. It's when someone puts something in, the agent decides how it wants to search. So when I, when I want to find when I go to Google Maps, like I did in that V60 example, if I type the name of a restaurant, I want it to search through names of restaurants. When I type V60 filters, I don't want it to search name of restaurant called V60 filter. How does Google Maps know? And you can see it in the UI, at least I can see it in my app on my phone. It'll change. It actually like uses a different color and even a different icon. It can switch between searching the name of a restaurant or maybe a thing that some random store in SF sells. How does it do that? It's a very, very primitive concept of a Gentic RAG. That is query enrichment. And so you can see it's on the other part of the query retrieval pipeline. You've got the objects underneath, they live in some little space, hypothetical document embedding like rebuilds those. It makes them make more sense in their space. Agentic RAG is query enrichment. It takes the query side and it rewrites that to make it make more sense for the space you're trying to search. Rank fusion. I hate this term, but this is a multi-stage processing technique. Rank fusion basically means you do multiple searches and you stitch them together. You can kind of think of this as sort of you, you want to get groceries for camping. And you and 4 friends are all going camping. You tell each of your friends, hey friends, pick out a couple things that you want to bring camping. And you say, neat, thanks for your lists. Okay, this object was on 2 people's lists. We'll definitely get that. This object was on this person's list, but that sounds like a pain in the ass to cook while camping. I'm nixing it. And this thing over here, I hadn't thought of it before, but it's on my friend's list and I love that. We're definitely getting that. That's all Rank Fusion is. But the concept is this thing called multi-stage processing. It's basically just taking this process of retrieval and breaking it into pieces. Okay. So I made a claim, which is that you could have invented all this shit from before. And the reason is because information retrieval has always been about these ideas. It's always been sort of like the same thing, but what is the task? What is the job to be done as the information retrieval engineer? It's asking yourself the following questions. What is the most likely representation of the user's desire? How does Google Maps understand that I want the V60 filters as an object in the store in SF and not I want to go to this hot new restaurant on the Michelin Gourmand list called V60? What representations can you generate ahead of time from the entities? So how can I take those like tickets in Linear and turn them into something searchable? How can I take that weird bespoke thing from the, like the software engineer and turn that into something that the PM could have searched for? The PM may know React, but they may not know in your particular code base what this damn component is called. How can you presage that? How can you think of it ahead of time and be prepared? And how can you correctly match these up? Because that's all retrieval really is. It's taking a query, it's taking an object, and it's saying these go together. So that's what information retrieval engineering has always been about. And so it still is. Next up, I wanna talk about sort of like a little bit of a, a contrived example. And this is to kind of give you a little bit more intuition about how these things fit together. So let's talk about an example where you want to retrieve financial documents. OK, maybe you've got thousands of financial documents of many, many different types. Maybe you work at some bank and you know you've got all these things. You got people applying to mortgages, you got people applying to homeowners loans. Sorry, those are the same thing. Like, you know, business loans. You've got people sort of like sending in their W-2. You've got documents you need to send to other people for their taxes. All these different financial documents and What are the different concepts that you can think about related to the financial documents? I call this curving space because we sometimes call the space in which you search, we call that a space, oftentimes a latent space, but we want to bend that space into the right shapes. That's that when you do your retrieval, you get the right shit back. And so what are the right things that we could do? Let's come back to this hypothetical document embeddings idea. You could write summaries. From those documents. You could pull out the tables of the data, and you could sort of make the table of data its own searchable thing. You could pull all the named entities out of the document. You could form typographies or ontologies or categories of the document types. What else could you do? I'm gonna read chat. Okay. You could make some questions about the documents. What questions do they answer? That's very much in the spirit of the original Hyde. What else could you do with these hypothetical or these documents, these financial documents? Okay. You could actually sort of like use the picture of the document to embed that. What else? Okay, you could simplify it. You could look at metadata. Super interesting, Varun. So metadata can actually be really exciting, right? Because you could sort of understand like who it's connected to. Maybe there's some relationship there. Okay, amazing. Lots of cool things that we can do here. What is the appropriate indexing strategy? So now that I've got all these different representations, how should I make it searchable? Classically, we think of like text and keyword search with like something like BM25. That's great. Some of those things that we mentioned are actually like really nice for keyword searching. For instance, named entities. That seems like a keyword search type of problem. Maybe not BM25, but maybe more just like, like string matching. Summaries could be interesting for keyword search or BM25. What about tables of data? How do you search that? How do you index tables? Some really interesting research on that one. I'll leave that to you to look up later. Okay. Sometimes you want to use vectors. I know that you've heard a lot about vectors in this RAG series, and so I won't belabor the point, but vectors or like late interaction vectors or multivectors, there's a, there's a very deep sort of like iceberg that we could go into there. I think the last talk was about that iceberg. Maybe it was the one before, I forget, but there's lots of interesting vector stuff to do here. Okay. But the reality is you need to think about those indices. So think about it. We've got one corpus of financial documents. I've now reframed that corpus of financial documents many, many different ways. And each of those ways has its own index. And now how do you fan out to those different indices? How do you match the query to the right index to make the search actually function? We think about that as a sort of staging process. We talked about agentic RAG briefly. This is where the agent would fit in depending on what they type in. How are you going to search? Are you going to search multiple indices simultaneously? Maybe you're only going to search one index. Maybe you're going to sort of like pray to the gods of multimodality and say, I don't want to think about any of this. I'm just going to use one crazy multimodal model. And so now we'll sort of appease everybody and we'll talk a little bit more about agents. I think of agents as transformers and no, no, I don't mean like, you know, like attention mechanism, blah, blah, blah. They transform information that come in and out. Some people think of agents as this like little brain and other people think of agents as like just a function call on somebody else's computer. My favorite way of thinking about agents and making them functional is by thinking of them as transformers. If I wanna convert a natural language prompt into a tool selection, that's a transformation. If I wanna convert some long paragraph of rambling in some transcript into some structured data, that, that's a transformation. If I want to let the agent decide on what code it needs to write and then call another agent, that's actually still a transformation. Is transforming a task into a selection of steps, and then each of those steps could later sort of be mutated. And so I sometimes just call them routers. And actually, my favorite agents, the ones that are most successful in my world, have been routers. And the last key thing that I want to cover is that maps aren't final. So if you've ever had a chance to look at globes from 100 years ago, they look quite different than today. You may say, well, like 100 years ago, we kind of had a pretty good sense of what the world looked like. We kind of knew where everything was and like, we didn't like discover that much new land, maybe some islands here and there, but like, why has the map changed so much in the last 100 years? Everything from sort of like, uh, geographic change to political change. Maps change quite a lot, and so too for representations. Because if you were to leave your map static or leave your embedding static, you would be missing out on a really dramatic amount of signal. One, documents change. Documents change in context to other documents. They change in context to your business. They change in context to your world. We talked about named entity recognition for those financial documents. What was a named entity recognition on a financial document from last year may need updating because your set of named entities may be updated since. You may know about entities that you didn't know a year ago that now exist, or you may want to link them to new things that didn't exist then and exist now. One example that we have in my current job is we do transcript enrichment. And one of the things that we want to do there is named entity recognition. How do you pull out names of companies and names of people and link those to the people and what they're doing? People do new things as time goes on. If I do this once and leave it, never return to it, it's going to get outdated. But that's not the only thing. The context of these financial documents may matter based on sort of like events that have taken place. A financial document that was produced in 2022 may have a very different meaning in the context of something now for the company if they've IPO'd since. And so it's actually quite important to not just update old feature engineering, but to also add new features as time goes on. You've got more and more signal and more and more opportunity to enrich the data that you have. So those, from my perspective, are the core concepts of how you build maps and how you traverse maps. And so next I wanna show you what it looks like to do this. And so I'm gonna turn it over to Ayush. He has now become the co-host. One step at a time.
Okay, cool. Yeah, I'll just take over the screen. Awesome. Let me know if you can see my screen. You can? Okay. So this is a small demo app, demo version of an application that we're working on. It's called semantic.art. And the purpose is to allow you to discover artworks in various ways, right? So you might be looking at a, you might describe an artwork using a specific mood, or you can describe it in an art artistic fashion using a poem or a prose, or you just want to find something similar to something that you've discovered that might be synthetic— synthetically generated, or something that inspires you to look. And these, these are real artworks, uh, these are not synthetic generated artwork. So this, this uses a bunch of these concepts that Brian just discussed, and we'll come to those in a minute. Just wanted to show you what this does. So just to test it out a little bit, what I did was I asked a few LLMs like Gemini, ChatGPT, and stuff to describe any artwork of their choice using different ways, right? One one could be an artistic fashion, or one could be like in a literal sense. And I got a few that I can show. So this is Persistence of Memory, which was described in a poetic fashion by, I think, Gemini or something. And, and so, you know, you, you get to see not just the original artwork, but something that is similar to that, but also derivative artwork of that original artwork. And this is the original. Or there are a few versions of it because this dataset has, it's, it's a combination of a few. And then, then you also see a bunch of others that are just inspired by this original artwork. So this is more like you, you described this in, in a poetic fashion. We can also do something in a more literal sense, uh, like this, which again is described by an LLM and, you know, again, gets you the results that, that you we're looking for and it still includes a bunch of derivative work as well. And so you can do a bunch of other things like, I mean, these are, this is just in-domain data. We can also search for some things that are not in-domain, right? For example, this is another artwork that an LLM just guessed randomly, but it's called, you know, Path Through the Fields. And you can see that it gets you the images that show path through the fields and some other like that. And again, it's, it's pretty much, it's multimodal. You can also search through images or text or combine both. For that, I'll just quickly show you an example. The idea here is that, you know, you can find an artwork that's maybe a derivative work of an original artwork, or it's a synthetically generated artwork, but you want to see if something like that really exists in the real world. So here's an artwork that's sort of a modified version of The Wave, and let's search by image. And see what we get. And we get the original ones as well as the derivative work on top of it, or something similar to that. I can also find ones that are similar to a specific artwork as well. But the basic idea here is that you can use inputs of different types, different semantic meanings, and so on. For some reason, this did not work. But yeah, let me try again.
I think that search modality is just a little slow-ish.
Oh, I did one. Cool. But yeah, the idea is that, you know, you can forget about a specific kind of query. You can address multiple types of queries at once. And I'll come to how this works and how all these concepts that Brian explained, how, how these apply here. Let me just reshare the Right now for that. Okay, let's first talk about the problem of representation. What this artwork— what this application does, it sources artworks from different sources, different datasets. Here are a few. And what we can do here is that we can extract many features, right? And the idea is that we want to be able to have multiple ways or multiple representations of the same data point. So moving forward, we have the flexibility to create different search paths. Depending on the need of changes in the future. So for example, here, each one of these features that we extract from the artworks, we can create multiple features out of them. These can, these can be a group of dense features, like, you know, vectors of the image or, you know, simple multimodal embeddings like CLIP or SigLIP, things like that, but also generate captions and then create indices out of that as well. But also we can do BM25, full-text search, things like that. And once we extract a bunch of features, these are some you can expand to even more, but that just makes, that just allows you to create more flexible search paths in the future. And here I'm using LensDB, so I can just use, it is both an object store and feature store, so I can just embed all of these here. And this is the basic ingestion pipeline. And obviously the idea here is that you're not limited to one feature or or a few features, you can just expand this dynamically to more features. And the kind of idea with, with this complex representation is that it allows you to create dynamic search paths and how to do that. It's a challenge in itself. Like, how do you use which, which search path to use for a specific type of query and to determine which is the right search path to use? That's a, that's another problem which comes for which we will look at discovery. So let me just list down the, the few basic components and the few basic search paths that we can create using those. So first, user input can be any of these. And so based on that, we have a few operations like determining query paths or generating image caption, then rewriting query or extracting the mood. And then out of all of these, we can then create pre-filters or create and then perform different kinds of searches, you know, advanced retrieval techniques like hybrid search, which, you know, is essentially combining multiple search results and then re-ranking them. It's also called— I think Brian covered this in Rank Fusion part, but this is the idea. And you can— this becomes more and more complex, right? You can also then re-rank using specific features as well, because we have a lot of features. Now, based on this, we can create multiple search paths. Now the challenge here is how to determine the search path. And to do that, I think one of the things that LLMs are good at, or at least decent at, is doing basic language tasks, right? You know, more classification, especially you see this in RAG where you just, in the G part of the RAG, which is the generators, you just simply give them a few documents and they stitch together as a sentence, right? The idea here is to use LLMs or search in a way that it acts as an agentic router. Creating its own dynamic search path, uh, you know, across different indices that we have, because this is what it exactly is. This is multiple indices and multiple features. You can use an LLM to extract the mood, or you can, if, if the user input is of a multimodal category, you can use those to get the query style or caption the image. If it contains both, you can just rewrite the query. And then you can extract the mood. And finally, once we have the mood, we can— and this is one of the search paths, right? You can do a bunch of other things with that. You can perform a basic search and then keep performing more and more searches. But in this pathway, what you can do is you can u— use the mood to pre-filter your entire dataset so that you're only looking at a specific kind of, uh, artwork. And then you can perform hybrid search because you have text features, you have image features, you have both dense and sparse features for both of them. And then you can finally rerank those results based on your, the feature that you choose, right? So this is like one dynamic pathway and this is sort of decided by agentic routing through the features. So here's the complete idea, which is have multiple ways to represent your data points. I mean, this is just what we covered is just specific to the, the Artwork search, but think of this as like, for example, when you're, when you're thinking of recommending products or like specifically clothes, you can create multiple features. Like a new user can search like based on the dress type, or they can search based on the season, they can search based on texture or color. And it's useful to have features for all of them so that later down the road you can create these agentic routers to that route the query to the right search path instead of sticking to one feature. And then you can continuously evolve it over time. So. That's the basic idea that we're presenting here. I think, yeah, that's pretty much it. Over to Brian.
Awesome. Sweet. So yeah, so that's what we want to show. And then obviously, you know, we have plenty of time for questions.
Where do you think people go most wrong with regards to like thinking that there's only one map for the territory, like Where do you think that comes from in terms of that thinking of like, hey, there's only one representation? I still find also that a lot of deployed retrieval systems for RAG, like, people only have one representation by far.
Yeah, I think because a lot of times we think that there's a natural representation. We like trick ourselves, right? We like think it's like easier than it is. We start by thinking like, What's the best way? And then when it comes time for us to realize it's not good enough, instead of saying like, what do I want to add? We start thinking like, oh, I guess I was wrong. I'll like try to fix it. And it's this, like, it's the fixing the one thing mindset. You can kind of think about this as like cars versus bicycles. And I'm going to, I'm going to really expose myself here for a second, because I think you should have one car and many bicycles. You don't really need like 4 different vehicles unless you're like really doing interesting shit. Uh, I feel like those people are maybe out of my, like, you know, experience, but you need 1 car. Maybe you need 2 cars if you have like a partner and they need to do shit too. But like, usually like you got 1 car and that's your, like, you go to the work and you also have fun in it and you like put down the top and maybe like ride down Highway 1 and you can do all these things with 1 vehicle. But those of you that have spent time cycling, you probably are like, no, no, no. I need my road bike. I need my tri bike. I need my gravel bike. I need my mountain bike. I need my backpacking bike. That's a lot of bikes. And the mentality, I think—
do you have that many bikes?
Listen, don't ask personal questions in public, Hamel Husain. That's inappropriate. So as I was saying, I think the mindset is like, we want to fix the thing that's broken. We want to, we want to like, oh, it's not good enough. I want to make it better. I'll like replace it. But actually a lot of times like augmenting it is really powerful. And also it's not always been so easy to fan out to different approaches. So let's go back to like, uh, the good old days of like Google search. You know, you've heard about sort of like these like complicated search algorithms that do sort of like boosting and like multi-index and stuff. They had to build systems around the ability to search over many different indices simultaneously and factor all that in. But if you came with a totally different way of searching, they had to build some infrastructure to be able to understand how you were searching. That's why image search and text search were totally different on Google. We live in a new world where models can help you do that. And so we do live in an exciting time where some things are getting easier. That's part of the story, but it's also, I just think, a sort of like mental, um, maybe sort of like naivete or newness of thinking about these things.
Before your talk in the series, there was two talks that I thought were highly related from Anton and Orion. You know, Orion talked about reasoning models. For RAG, but not like in the sense that you would normally think. He talked about how you can have these reasoning models think about the meta search objective and bake that in directly into the representation on the fly.
Yep.
Based on the query. And then, you know, obviously Anton talked about the ColBERT family of, you know, representations. And those two things seem like they're trying to go with the angle of make your representations flexible somehow.
Yeah.
Where do you, how good, or where do you think we are at the current point in terms of those things being flexible enough? Or do you have any thoughts?
I think they fit. Yeah, I think they fit perfectly into this model as well, because like, why would you want reasoning as part of this flow? Well, it's reasoning is query understanding. It's query rewriting. It's user put in a thing. That thing is confusing. It's not enough context. It's not enough sort of information to answer it. When I, you know, in my example about like engineering tickets, the PM puts something that like, it's not like, like illogical. It's not an unreasonable way. It's just not correct for the context of the rest of it. So what you would love that reasoning agent to do is start exposing the things that are sort of missing or exposing the additional relevant things. Let's say you have a RAG system connected to that. Well, that RAG system can go and get information about the code base that might be relevant to enrich that query. And so the reasoning step is just part of query understanding, it's just part of query rewriting. I think it's a really powerful and super exciting way to do it, and these agents can do that. One thing that we could do with Semantic.art is slap a reasoning agent in that first stage. And we candidly already have some reasoning going on. We just have some very specific reasoning, but you could add that. It could sort of like really, you know, expand the, the way that it queries. Totally reasonable. But let's talk about multivector or things like, um, ColPoly or things like that. What is that asking? That's asking sort of one model to come up with multiple representations. That's actually all late interaction is. Late interaction says during the sort of decoder process, or maybe it's, I think, yeah, the decoder process, you're basically sort of, you're forming multiple latent space representations. One thing that I went quickly through is this concept of multimodal in the latent space, i.e., one thing lives in multiple places in the same space. Let me give you a quick example of that. Pinterest came out with a paper quite a while ago, and the paper was about this concept of multimodality. And it says sometimes a user will ask for a chair and they're gonna get images of a chair, but they may want images of sort of like a nice fancy chair in a living room. They might want some images of sort of like designs of a chair, and they may want some sort of like. Architectural design sketches of like what the, like, geometry of a chair looks like. Those are all reasonable things for the user to get back. What I could do is I could say, well, based on this user and their previous sort of interactions, they want the chair in the living room. I'm just gonna give it chair in the living room. But believe it or not, users are happier if you give them mostly that a little bit of the second thing, and even a little bit of the third thing. By giving them all of these things simultaneously, they're actually the happiest. For a recommendation problem, this is really exciting. This is called multimodality because there are different modes of that one signal in your space. So they found that this is really, really powerful for recommendation systems. How's this related to things like late interaction?
Question about that. That's really fascinating. So some might say, okay, if you have an ambiguous query like chair, we need to gather user intent, kind of like the deep research.
Sure.
Are you saying that paper, that specific paper kind of found that educating the user in a way by showing them all the different pos— like these different dimensions, like made them happier?
But isn't that interesting? Even just the way you said that question, like educating the user. Believe it or not, diversity in recommendations is a type of education. How many times have you done a search thinking what you knew, what you wanted back? You're like, I know what I'm looking for. And you get a search result back and you think, well, I didn't know that. And now I want to read that too. How often has that happened? Cause it happens to me all the time. And maybe I've got like a very small brain and I don't think of all the different things that could come back. But like, regardless, I'm constantly surprised by diversity in recommendations. So education of the user, one way to do it is through diversity. Why am I interested in Semantic.art in this very multimodal approach? Because I actually think that the diversity that you can get back by different interpretations of the query is interesting. Why might you care about something like late interaction models? Because the diversity of representations that the model comes up up with when trying to fit to the training data, some of those other representations can be valuable too. I think I often hear people get really excited about late interaction as like number go up, but actually I get excited about late interaction because it's diversity as sort of like part of the model. You're building diversity into the model. Late interaction is far from the first, uh, sort of recommendation approach that we've built that builds that in. Multi-objective has been the name of the game for about a decade, candidly. From my perspective, that's why multivector is really interesting on this sort of like late interaction style. But yeah, it's just diversity is maybe like my oversimplification.
We might even say that guiding people towards looking at the data makes them happier.
Whoa, whoa, whoa, whoa. Let's not get ahead of ourselves.
Um, So there was some, there was quite a bit of discussion, questions, so on and so forth about routing. Like, hey, what is routing exactly? Is it, is it a classifier? Is it something else? Is that, how do you do that? Any, any thoughts?
Yeah.
I mean, I'll ask you used to respond as well, but I think one, uh, okay. Uh, like talk to my book here a little bit. Routing is one stage in the failure funnel. Funnel. For those of you that have heard my spiel, when you ask an LLM a question, it's a series of multiple steps, and every step can screw up the rest. And if you don't get it right at one step, you're probably downstream screwed. One stage is routing. You should evaluate routing just like you evaluate everything else. You should build evaluation datasets, and you should build priors around how things you want routed. Let me give you a quick example of routing. We, need to take transcripts from calls and run different LLM extraction pipelines depending on the call. If I have one call where I'm talking to a buddy, I want different notes from that than if I have another call where I'm talking to a collaborator about a project. And so too, if I'm talking to a potential customer, all of those are different things that I want the transcript to do. Obviously, I would like a full text transcript, but I'd love to process that information into different ways so that my RAG system and search system built on top can be more valuable. That is a routing problem. So how do I make it good? I grab some examples of all these transcripts. I label them by hand, God forbid. And then I try different routing approaches. Plenty of different ways to build a classifier in the age of our, our Lord. Uh, you choose your own. I'll hand it over to Aayush. I think he also has thought a lot about routing.
Yeah. In short, right now it is acting as a simple classifier, but there's more to it than that, right? LLMs are decent classifiers by default, but it's in a very hardcoded and guided setting, right? Because the real problem with LLMs is most of the responses are not 100% reproducible, right? So in a very guided environment, in a very hardcoded setting, yes, if you're working with a few features, it can be a decent classifier. If you want to expand this to like 100 classes or more, you might want to think of testing your own classifiers on top of that. Obviously you can think of how you want to train that classifier, like what metric you want to optimize. But I think, yeah, LLMs is a good proof of concept that semantic routing can work based on classification of the input types. But yeah, it's not foolproof as a scalable multi-label classification system.
I was going to ask. With the routing. Okay, do you want to route via kind of a traditional classifier or like a tool call? And it seems to me like at the order of 100, you probably don't want to stuff that into tool calls because models feel like at the moment they're not able to deal with that number of tools. Is that the right intuition? Would you agree, disagree? I mean, something?
Yeah, I would, I would agree to that. And also I would add that, you know, I think Mistral did release like a classifier model recently. They pretty much use their LLM backbone and put a classifier head, which is an overkill for most classification tasks, but it works pretty well. So you kind of have this situation where you can create like a reproducible classification pipeline out of an LLM with some training. So that's like. The approach that you could test when the, the labels are in, or the classes are in, in like, uh, like in hundreds or thousands, something like that.
Yeah, makes sense. There's always a BERT and it's a family of models as well.
Yep.
Okay, no, I think that's it. I think we're at the top of the hour. I think this is a really entertaining talk as usual. Uh, lots of memes are going to come out of this. So yeah, I really appreciate it.
Awesome, awesome. Well, thanks everyone, and I hope it was, uh, I hope it was fun, and I hope you add some more representations to your data.
