Back to Basics for RAG w/ Jo Bergum

Yeah, so thank you for having me. I'll talk about Back to Basics. And I'm Joe Christenburgum. And let's see if I can actually— yeah. So about me, I'm a distinguished engineer. I work at Vespa AI. And I've been at Vespa AI for 18 years, actually. So I've been working in search and recommendation space for about 20 years. And Vespa AI is basically a platform, a serving platform that was recently spun out of Yahoo. We've been open source since 2017, and in my spare time I spend some time on Twitter posting memes. Yeah, and in this talk I'll talk about stuffing text into the language model prompt. I'll talk about information retrieval. The R in RAG. And most of the talk will be about evaluation of these systems, of information retrieval systems, and how you can build your own evals to impress your CTO. And I'll talk about representational approaches for information retrieval. This includes BM25, vectors, embeddings, whatnot, and some of the baselines. So RAG, You're all familiar with RAG, but I think it's also interesting that you can use the kind of the whole RAG concept to stuff things into the prompt, not necessarily related to question answering or search. But for example, if we are building a labeler or classifier, we can also use retrieval to retrieve kind of relevant examples or examples out of our training datasets, right? right? So that's one way, but it's not that often discussed that you can also use retrieval. So let's say you have 1 billion annotated training examples, you can actually use retrieval to retrieve relevant examples and then have the large language models reason around that and predict a label. But most are thinking about RAG in the context of building this kind of question answering model that you see at Google and all these chatbots and similar where you retrieve for a question, open-ended question, and then you retrieve some hopefully relevant context and you then stuff that into the prompt and then you have hopefully the language model will generate a grounded response. It might not be hallucination-free, but some say that it improves the kind of accuracy of the generation step. So that's kind of demystifying it. And working with these, like, reference architecture, there's some orchestration component, there's some input, there's some output. Hopefully you have some evaluation of that output. Prompting, different language models, and then you have kind of state, which can be files, search engines, vector databases, regular databases, or even NumPy. So there's a lot of things going on here. And there's a lot of hype around RAG and also different methods for doing RAG. And I'm on Twitter a lot, and I see all this kind of Twitter threads, there's new model. So there's a lot of components in this machinery, lots of new tricks. Check out this. So it's a lot of kind of hype. So I like to kind of try to cut through that and, you know, what's behind this? How does this work on your data? This is actually— is this actually a model that actually have some basics or backing from research? Have you actually evaluated it on some dataset? And I think this is— it can be if you're like coming into this space and you're new to retrieval, you're new to search, and you're new to language models and you want to build something. There's a lot of confusing information going around. And I just saw this Twitter thread about RAG and people like losing faith in it. And, you know, we removed AI-powered search. And I think there's been like, RAG is only about taking a language model from, for example, OpenAI, and then you use their embeddings, and then you have a magical search experience, and that's all you need. And I think that's naive to think that you can build a great product or a great RAG solution in that way just by using vector embeddings and the language models. Because the retrieval step in this pipeline, the process of obtaining relevant information based on some query, basically has been around for, like Benjamin in his talk covered, for decades. And there are a lot of people, or the brightest minds, that have actually spent a lot of time on retrieval and search, right? Because it's so relevant across many kind of multi-billion companies like recommendation services, search like Google, Bing, and whatnot. So this is a kind of always been a very hot and interesting topic. And it's much deeper than encoding your text into one vector representation, and then that's it. But I'll talk about how we can evaluate these information retrieval systems. And in this kind of, basically, you could treat this as a more or less of a green box where you have put some data into it and you have your kind of retrieval system and you're asking that retrieval system a question and you're getting back a ranked list of documents. And then you can evaluate these documents and the quality of these documents with regards to relevance of of how relevant they are with regards to the query. And this is kind of independent if you're using what kind of retrieval method you're using or combination or hybrids of face ranking or ColBERT or SPRITE or whatnot. You can evaluate any type of system if it's using NumPy or files or whatnot. It doesn't really matter. And the basic idea of building a such system is that you take a query and you retrieve those documents, and then you can have human annotator, for example, to judge the quality of each of the documents. And there are different ways of doing this. We can do it by the binary judgment, saying that, okay, this document is relevant for the query or not. Or we can have a graded judgment where you say, okay, 0 means that the document is irrelevant for the query, And 1, it's slightly relevant, or 2 is highly relevant, right? And we can also do use this to judge the rank lists that are coming out of recommendation systems or personalization and many different systems that are producing a rank list. And in information retrieval, this is going back decades, and there are a lot of researchers working on this. And you have TREC, which is the text retrieval conference, spans multiple different topics each year. Year, news retrieval, all kinds of different retrieval tasks. MS MARCO, maybe some of you are familiar with, which is one of the largest datasets that you can publish research on. It's from Bing, actually real-world data, which is annotated, and a lot of these embedding models are trained on this dataset. Then we have BEER from Nils Reimers et al. that evaluate types of models without actually using the training data, but this is like in a zero-shot setting. So there are many different collections, and then there are metrics that can measure how well the retrieval system is actually working. So recall at K, for example, K here meaning a position in the, in the ranking list. So K could be, for example, 10 or 20 or 100 or 1000. And it's a metric that is focusing about, you know, you know that there are like 6 items that are relevant for this query, and are we actually retrieving those 6 relevant documents into the, to the top K? In most systems, you don't actually know how many relevant documents there are in the collection. In the web scale, it might be, you know, millions of documents that are relevant to the query. So it's not a— unless you have a really good control of your corpus, it's really difficult to kind of know what are the actually relevant documents in the document. But precision is much easier because we can look at those results and say, are there any irrelevant hits in the top K? So precision is one, but it's not really rank aware. So it's not bothering if the missing or relevant hit is placed at position 1 or 10. So the precision at 10 would be the same if you— if it doesn't, it doesn't necessarily depend on the position. And this is a very complicated metric, but it tries to incorporate the labels so that the graded labels and also awareness of the rank position. If you want to look up that, you can, you could basically go to Wikipedia, but it's a quite advanced metric. Reciprocal rank measures where is the first relevant hit in the position. So if you place the relevant hit, or a relevant hit at position 1, you have a reciprocal rank of 1. If you place the relevant hit at position 2, you have a reciprocal rank of 0.5. Then of course you have LGTM, which is looks good to me, maybe the most common metric used in the industry. And of course, also in industry, you have other evaluation metrics like engagement, click, if you're measuring what actually users are interacting with the search, dwell time, or e-commerce, add to cart, all these kind of signals that you can feedback. Of course, revenue, e-commerce, search, for example, it's not only about the relevancy, but also you have some objectives for your business. I also like to point out that most of the benchmarks are comparing just a flat list, and then when you're evaluating each of these queries, you get a score for each query, and then you take the average to kind of come up with an average number for the whole kind of retrieval method. But in practice, in production systems, you will see that maybe 20% of the queries actually it's contributed like 80% of the volume. So you have to think a little bit about that when you are evaluating, uh, systems. Yeah, so, and to do better than looks good to me, uh, you really have to measure, uh, how you're doing. And since you have all these benchmarks, uh, mTab and whatnot, they don't necessarily transfer to your domain or your use case. If you're building a RAG application or retrieval application over code or documentation or specific health domain or products, or because there are different domains, different use cases. So your data, your queries, and the solution to do better is to measure and building your own relevancy dataset. And it's actually not that hard. If you have actually a service in production, look at what actually users are searching for and look at the results and put in a few hours and judge the results, right? Is it actually relevant? Are you producing relevant results? And it doesn't really need to be fancy at all. And if you don't have traffic, if you haven't launched with it, you obviously have played around with the product, or you can also ask a large language model to, you know, present it some of your content, and then you can ask it, you know, okay, what's a question that will be natural for a user to retrieve this kind of passage? So you kind of bootstrap even before you have any kind of user queries. And as I said, it doesn't need to be fancy. You can log— there are some fancy tools for doing this with user interfaces and Docker and whatnot, but a simple tab-separated file will do the trick. Preferably you will have like a static collection, but maybe not everybody has the luxury that you can actually have a static collection. And the reason why you would like to have a static collection is that when you are judging the results and you're saying that for this query, for instance, query ID 3 and the document ID 5, you say that, oh, this is a relevant one. When we are judging the kind of, or computing the metric for the query, if there's a new document that is suddenly appearing which is irrelevant or relevant, it might, you know, we might actually change things in how we display things in the ranking without we able to pick it up. So that's why you preferably have these kind of static collections, but, and all the information retrieval datasets, they are usually static, right? They don't, they don't change. So we can evaluate methods and practices over time. But you can also, in this process, use language models to judge the results. And there's been interesting research coming out of Microsoft Bing team for over the last year, where they find that with some prompting techniques that they actually can have the large language models be pretty good at judging query and passages. So given a passage that is retrieved for the query, and they can ask the language model, is this relevant or not relevant? And they find that this actually correlates pretty well. And if you find like a prompt combination that actually correlates with your data or your kind of golden dataset, then you can start using this in at a more massive scale. And here's a very recent paper coming out 8 days ago where they also demonstrated that this prompt could actually work very well to assess the relevancy of queries. And this can free us from this having this kind of static golden datasets because we could start instead sampling real user queries and then ask the language model to evaluate the results. So I think this is a very interesting direction. And we have in our RAG or Vespa RAG documentation search, I built like a small golden set with about 90 query passage judgments. and I just ran them through with this prompt or a similar prompt, and I'm getting quite good correlation between what, what I'm judging the results and GPT-4 is judging them, which is good because it means that I can now much cheaper judge more results and then potentially also use this to adjust ranking. Because when you have this kind of dataset, you can also iterate and make changes, and then you can see, you know, how it's actually performing. So instead of saying, oh, we changed something, you can go to, we actually deployed this change that did increase NGCG with 30%. And this is from our documentation, Vespa documentation search, which is relevant for us. It's our domain. You see here semantic here is off-the-shelf vector embedding models, and then there are different ways in, in Vespa to use hybrids. I won't go into the details, but now I actually have numbers on it, and then when I'm making changes. So that's about evaluation. So independent of the method or technique that we're using, we can evaluate the results coming out of the retrieval system. Now I want to talk a little bit about the representational approaches and scoring functions that can be used for efficient retrieval. And the motivation for having this kind of representational approach is that you want to try to avoid scoring all the documents in the collection. So if you're using— some of you might heard about Cohere re-ranking service or this kind of ranking service services where you basically input the query and all the documents and they go and score everything. But then you have everything in memory, you already retrieve the documents. And imagine doing that at the web scale, or if you have 100 million documents, is, is not possible, right? And it's also similar to doing a grep. So instead, we would like to have some kind of technique for representing these documents so that we can index them. So that when the query comes in, that we efficiently can retrieve over this representation, and that we efficiently in sublinear time can retrieve the kind of top-ranked docs, and then we can feed that into subsequent ranking phases. And there are two primary representations, and that is the sparse representation, where we basically have the total Vocabulary is kind of the whole sparse vector representation that you potentially take, but for a given query or a given document, only the words that are actually occurring in that document or in that query have a non-zero weight. And this can be efficiently retrieved over using algorithms like WEKA or MaxScore. And inverted indexes, you're all familiar with Elasticsearch or other kind of keyword search technologies. They build on this. More recently, we also have using neural or kind of embedding or sparse embedding models, so that instead of having an unsupervised weight that is just based on your corpus statistics, you can also use transformer models to learn the weights of the words in the queries and the documents. And then you have dense representations, and this is where you have text embedding models where you take some text and you encode it into this latent embedding space, and you compare queries and documents in this latent space using some kind of distance metric. And there you can build indexes using different techniques, vector databases, different types of algorithms. And in this case also, you can accelerate search quite significantly so that you can search even billion-scale data, datasets in milliseconds, single credit. But the downside is that there are a lot of trade-offs related to that. The actual search is not exact. It's an approximate search. So you might not retrieve exactly the ones that you would do if you did a brute force search over all the vectors in the collection. And these representations are mainly supervised through transfer learning because you're using typically an off-the-shelf embedding models that's been trained on some other data, some datasets, and then you're trying to apply that to your model. You can fine-tune it if you have relevancy data and so forth, then it's no longer like a zero-shot or transfer learning, but still like a learned representation. And I think these representations and the whole ChatGPT, OpenAI, ChatGPT language model, OpenAI embeddings really opened the world of embeddings to a lot of developers. And this stuck for quite some time, and it's still stuck, I think, because people think that this will give you a magical AI-powered representation. It's not bad, and you can also use now a lot of different technologies for implementing search, vector databases, regular databases. Everybody now has a vector search report, which is great because you can now use different or more wide landscape of different technologies to kind of solve search. And, but there are some challenges with these text embedding models, especially because the way they work, most of them are based on a kind of encoder style transformer model where you take the input text, you tokenize it into a fixed vocabulary. And then you have previously in the pre-training stage and the fine-tuning stage, you have a learn representations of each of these fixed tokens. Then you feed them through the encoder network. And for each of the input tokens, you have an output vector. And then there's a pooling step, typically averaging into a single vector representation. So this is how you represent not only one word, but a full sentence, or even now with the embedding model coming out today supporting, you know, encoding several books as one vector. But the issue with this is that the representation becomes quite diluted when you kind of average everything into one vector, which is proven not to work that well for high-precision search. So you have to have some kind of chunking mechanism to, in order to have a better representation for search. And this fixed vocabulary, especially for BERT-based models, you're basing it off a vocabulary that was trained in 2018. So there are a lot of words that it doesn't know. So we had one issue here with a user that was searching for our recently announced support for running inference with GGF models in Vespa. and this has a lot of out-of-word, oh, sorry, out-of-vocabulary words. So it gets maps to different concepts, and this might produce quite weird results when you are mapping this into the latent embedding space. And then there's a final question is, is, does this actually transfer to your data, to your queries? So, and, but the framework or the kind of evaluation routines that I talked about earlier, will give you the answer to that, because then you can actually test, you know, if they're working or not. And on, and also, I think on the baselines, it's quite important to establish kind of some baselines. And in the information retrieval community, the kind of de facto baseline is BM25. So BM25 is this scoring function where you tokenize the text, linguistic processing, and so forth. It's well known, implemented in multiple mature technologies like Elastic, Vespa, Tantive, whatnot. I think there was even a library announced today, BM25, in Python. So, and it builds a model, kind of model, unsupervised from your data, looking at the words that are occurring in the collection, how many times it's occurring in the data, and how frequent the word is in the total collection. Then it's the scoring function, and it's very cheap, small index footprint, and most importantly, you don't have to invoke kind of a transformer embedding model like a 7B LLaMA model or something like that, which is quite expensive. It has limitations, but it can avoid these kind of spectacular failure cases of embedding retrieval related to out-of-vocabulary words. The huge downside is that if you want to make this work in, uh, CJK languages or Turkish or different types of languages, you need to have some kind of tokenization integrated, which you will find in engines like Elasticsearch or OpenSearch or Vespa. And long context, uh, so we did the announcement earlier this year of supporting ColBERT in a specific way. I'm just including this to show you that this is a long context document. So they are, I think they're around 3K tokens long, and the researchers evaluated these different models, and they were presenting results about M3, which is scoring 48.9 in this diagram, and they were comparing it with OpenAI embeddings, with different types of Mistral or different types of embedding models. And then we realized that, you know, this is actually quite easy to beat just using a vanilla BM25 implementation, even Lucene or Vespa or Elasticsearch or OpenSearch. So having that kind of mindset that you can evaluate and actually see what works and remember that BM25 can be a strong baseline, I think that's an important takeaway. Then there's a hybrid alternative. We saw— you see a lot of enthusiasm around that where you can combine these representations. and it can overcome this kind of fixed vocabulary issue with regular embedding models, but it's not also not a single silver bullet, reciprocal rank fusions or methods to fuse these kind of different methods. You know, it really depends on the data and the type of queries. But if you have built your own evals, then you don't have to listen to me about, you know, what you should do. Because you can actually evaluate and test things out and you can iterate on it. So I think that's, that's really critical to be able to build better RAG to improve the quality of the retrieval phase. Yeah, and of course, I talked about long context and that the long context models, we all want to get rid of chunking. We all want to get rid of all the videos about how to chunk. But the basic kind of short answer to this is that you do need to chunk in order to have meaningful representations of text for high precision search. So typically, like Nils Reimers, the de facto embedding expert, says that if you go above 250, so 256 tokens, you're starting to lose a lot of precision, right? There's other use cases that you can use these embeddings for, like classification. You know, there are a lot of different things, but for high-precision search, it becomes very diluted because of these pooling operations. And also, there's not that many great datasets that you can actually train models on and have longer text. And even if you're chunking to have meaningful representation, it doesn't mean that you have to split this into multiple rows in your database. There are technologies that allows you to kind of index multiple vectors per row. So that's possible. Finally, real-world RAG. Not sure, but if you've seen this, but there was a huge Google leak earlier in May where they revealed a lot of different signals. And in the real world, it's in the real-world search, it's more, it's more about not only about the text similarity. It's not only about BM25 or a single vector cosine similarity. There are things like freshness, authority, quality, PageRank you heard about, and also revenue. So there are a lot of different features, and GPT is still, you know, a simple, straightforward method, and it's still the kind of king of tabular features where you have specific named features and you have values for them. So combining GPT with this kind of new neural features is quite effective when you're starting to actually operate in the real world. So, quick summary, I think that information retrieval is more than just a single vector representation. And if you want to improve your retrieval stage, you should look at building your own evals. And please don't ignore the BM25 baseline and choosing some technology that has hybrid capabilities, meaning that you can have exact search for the exact tokens and still have matches and also combine the signals from text search via keywords and text search via embeddings. Can avoid some of these failure modes that I talked about. And yeah, and finally, real-world search is more than text similarity. So that's what I had, and I'm hoping for questions. If you want to check out some resources, I do a lot of writing on the blog West by AI, so you can check that out. And if you hated it, you can tweet me at Joe Bergam at Twitter. I'm quite active there, so I appreciate if you hated it, and then you can mention me there. And yeah, and you can also contact me on Twitter. I love getting questions.

So that's a, that's a bold call to action. If you hated it, it is. We definitely have a lot of questions. I'll just go through some of them. What kind of metadata is most valuable to put into a vector DB for doing RAG?

Yeah, if you look at only the text domain, if you're only concerned about text, so you have no freshness component or you don't have any authority. If you, for example, are building like a healthcare or a health, you know, users are asking health questions, you definitely want to have some kind of filtering. What's the authoritative sources? Within heart health. You don't want to drag up Reddit or things like that, right? So, and title and other metadata, of course, is— but it really depends on the use case. If you're like a text-only use case, or if it's like more like real world where you have different types of signals.

So makes sense. Yeah. Do you have any thoughts on calibration of different indices, of the different indices, not only are different document indices not aligned in terms of similarity scores, but it's also nice to have confidence scores for how likely the recommendation is to be good.

Yeah, I think it's a very tough question. So these different methods for all these different scoring function, you can call that, have a different distribution, different shape, different score ranges. So it's really hard to combine them, and they're not probabilities. As well. So it's very difficult to map them into a probability that actually this is, or filtering. People want to like, oh, I have a cosine similarity filter on 0.8, but it's different from different types of model. But combining them is also a learning task. It also kind of, you need to learn the parameters, and GPT is quite good at that because you're learning a nonlinear combination of these different features. But in order to do that, then you also have to have training data. But the way I described here for doing evaluation can also help you generate training data for training ranking models.

So does the calibration really turn into a hyperparameter tuning exercise with your eval set, or is that kind of—

Well, you could do that, right? If you don't have any data that you can train a model on to train those parameters, you could do a hyperparameter sweep and then you know, basically check if your eval is improving or not. But if you want to apply like more like an ML technique on this, right, then you would either like Google is doing like gathering search and clicks and interactions. But now we also see more that people actually using large language models to generate synthetic training data. So you can distill kind of the powers of the larger models into smaller models that you can use for for ranking purposes. But it's a very broad topic, I think. So it's very difficult to kind of deep dive, and it is very difficult to say that, oh, you should have a cutoff 0.8 on the vector similarity, and, or you can do this transformation. So there are no, like, really great tricks to do this without having some kind of training data and at least some evaluations.

Yeah. What are your observations on the efficacy of re-rankers? And do you usually recommend to use a re-ranker?

Yeah, because the re-rankers, the great thing about re-rankers is that in a phased retrieval and ranking pipelines, you're gradually throwing away hits using this kind of representational approach. And then you can have a gradual approach where you're investing more compute into fewer hits and still be within your latency budget. And the great thing about re-rankers like Cohere or cross-encoders, you can deploy them in Vespa as well, is that they offer this kind of token-level interaction because you input both the query and the document at the same time through the transformer network, and then you have token-level interactions. So you're no longer interacting between the query and the document through this vector representation, but you're actually feeding all the tokens of the query and the document into the method. So yeah, that definitely can help accuracy, but then is coming a question about cost and latency and so forth. Yeah, so a lot of trade-offs in this, but if you're only looking at accuracy and you can afford the additional cost, Yeah, definitely, they, they can help. Yeah.

Hey, William Horton is asking, do you have advice on combining usage data along with semantic similarity? Like if I have a number, then like a number of views or some, some kind of metadata like that from a document.

Yeah, it goes, it goes Yeah, it goes into more of a— if you have interaction data, it becomes more of a learning-to-rank problem. You first need to come out with labels from those interactions because there are going to be multiple interactions, and then there's going to be add to charge, and different actions will have different weights. So the standard procedure is that you convert that data into kind of a labeled dataset, similar to what I've shown here in the eval. So when you convert that to kind of a label dataset, then you can train a model, for instance, a GPT model where you can include the semantic score as well as a feature. Yeah.

Alright, someone's asking a question that you may not be familiar with, but we're just gonna give it a shot. It's a reference to someone else. What are your thoughts on Jason Liu's post about the value of generating structured summaries and reports for decision makers instead of doing RAG the way we are doing— that's commonly done today. Have you seen that? You're familiar with that?

Yeah, I mean, Jason is fantastic. I love Jason, but he's a high-volume tweeter, so I don't read everything. So I haven't caught up on that yet. No, sorry.

Okay, no worries. I don't want to try to rehash it from my memory either, so I'll just skip that one. What are some of your favorite advancements recently in text embedding models or other search technologies that people, I'll just stop the question there. Yeah, what are your—

Yeah.

What? Yeah.

Yeah, I think embedding models will become better. What I do hope is that we can have models that have a larger vocabulary. So like LLaMA vocabulary, so we have a larger vocabulary. So like BERT models, they have this old vocabulary from 2018. I think we, I would love to see a new kind of the BERTA model trained on more recent data with more recent techniques like a pre-training stage including a lot larger vocabulary for tokenization. I think, as I said, that I'm not too hyped about increasing the context length because all the information retrieval research shows that they are not that well at generalizing a long text into a good representation for high-precision search. So I'm not so excited about the direction where you're going with just larger and larger and larger and larger context windows for embedding models because I think it's the wrong direction. I would rather see larger vocabularies and better pre-trained models like the BERT. It's still a good model for embeddings. Yeah.

Someone's asking, does query expansion of out-of-vocabulary words with BM25 work better at search? And I think, like, just to add on to that, do you think people are going as far with classical search techniques as they should, like, you know, things like query expansion and all kinds of other stuff that have been around for a while before. Like, what's your feeling about the spectrum?

And like, right, yeah, I think you can get really good results by starting with BM25 and classical resorts and adding a re-ranker on top of that. You won't get the magic if you have a single word query. And there are no words in your collection, then you might fail at recall, but you don't get into these kind of really nasty failure modes of embedding vector search alone. And yeah, definitely there are techniques like query expansion, query understanding, and language models. They are also quite good at this. There's a paper from Google. They did query expansions with Gemini. Worked pretty well, not amazingly well compared to the size of the model and the additional latency. But we have much better tools for doing query expansion and all kinds of fancy techniques now involving prompting of large language models. So definitely that too is really interesting for expansion. So that's another way. But like in the diagram where I saw this machine and all these components and things like that, What I'm hoping people can take away from this is that if you're wondering about this technique, that technique, I read about this, is that if you put that into practice in a more systematic way, having your own eval, you will be able to answer those questions on your data, on your queries, without me saying that the threshold should be 0.6, right? Which is bullshit, right? Because I don't know your queries or your domain or your data. So by building these evals, then you can actually iterate and get the answers.

In one slide, you mentioned limitations in fixed vocabulary with text that is chunked poorly. How do you overcome these sort of limitations in a domain that uses a lot of jargon and that doesn't tokenize well with an out-of-the-box model?

Yeah, then you're out of luck with the regular embedding models. And that's why the hybrid capability where you actually can combine the keyword search with the embedding retrieval mechanism. But the hard thing is to understand when to completely ignore the embedding results, because embedding retrieval, no matter how far they are in the— out in the vector space, will be retrieved, right? So when you're asking for 10 nearest neighbors, they might not be so near, but you're still retrieving some junk. And then it's important to understand that this is actually junk, so that you don't use like techniques like reciprocal rank fusion, which by some vendors is sold as the full kind of blown solution to solve all this. But then you're just blending rubbish into something that could be reasonable from the keyword search. So currently, and the other alternative is as well that might do a little bit stop capping is to fine-tuning your own embedding model, but you still have the vocabulary issues. So, but if you have resources to kind of do the pre-training stage, on your data with a vocabulary that is more matching up with your domain, that might work. But then you have a training job that goes from scratch. But I hear it's a lot easier to train BERT from scratch nowadays than in 2018. So it might be a viable option for some organizations. Most of the e-commerce companies are doing this anyway. They're starting in all their semantic search papers. They basically say, here's our pipeline. We pre-trained to build a tokenizer on the whole Amazon corpus. They don't use BERT-based from 2018.

That makes sense.

Yeah. Okay.

Last question. Would you see ColBERT-based methods get around or at least improve retrieval when we're concerned with tokenizer problems?

Yeah. So ColBERT2 introduced that. It's basically another neural method. Where you, instead of learning one vector representation of the full passage or the full query, you are learning token-level vector representations. And this is a bit more expensive compute-wise at serving time than the regular single embedding models, but it has close to the accuracy of like the regular re-ranker. But it still also suffers from vocabulary because it still uses the same kind of vocabulary as other models. So, but if we can get better pre-trained models that are trained with a larger vocabulary, I hope that it's a path towards better kind of neural search with ColBERT and other embedding models as well. So yeah, there's a lot of great questions. So if you, if you want to, you know, throw them at me at Twitter and I'll try to answer as best as possible.

Thanks, Joe. Yeah, thank you.