Optimizing RAG With Reasoning Models

So today I'll be talking about new frontiers in retrieval or information retrieval, IR, about instruction following and reasoning. If you're using language models today, like you're talking to ChatGPT, there's a couple things that have been huge for defining them and helping them be really useful for users. Two of the biggest ones I think are instruction following. And if you've used a language model before, you can— what does all sorts of things, you can just tell tell it what you want and it will go out and do it. So here I asked ChatGPT to generate a haiku about information retrieval in the style of a pirate and to mention RAG, and it can follow my instructions perfectly. So here's this nice little haiku. It mentions RAG. It talks like a pirate. And this actually, I feel like, may be underwhelming to us today because we just expect them to do this, right? They're so good at instruction following. You can just tell a language model what you want and it'll give you what you want. But a few years ago, this was not the case. The other is reasoning, and this is something that's really hot in the community today. Pretty much any major player in language model space will have some reasoning model. And what sets these apart is that when you ask it a question or type something in, it's going to give these thinking tokens. So here's one from OpenAI's O-1. And here it thinks about how many Rs are in the word strawberry. It'll generate these sort of intermediate tokens. And then finally, it'll end by telling you the final output after it thinks. And so this is known as reasoning, thinking, sometimes test-time compute, all these sorts of things. And so reasoning and instruction following are two of the most major things that are making language models so useful these days. What I wanna talk about today is how can we use these in retrieval? Like, how does this relate? We have these really amazing language models. How can we bring them over? If you look at Google in 1999, it looks like this, which is really actually quite similar to how Google Search looks today in 2025, right? So we have 26 years of development, but it's still very much similar in that you're going to type words into a box it's gonna go out and do some form of matching, and then it's gonna come back and give you your list of websites. We do have nice new things like SearchGPT these days. So if you see this, it's a very nice UI, but what happens behind the scenes is it's going to send your query to a search engine. In fact, often like Bing or Google itself, just normal search. And then it's gonna pass those results back to the language model, which then creates this very nice output. So search itself really hasn't changed very much. SearchGPT is still using a normal search engine. So despite the fact that these days we're using language models, we're training models even from LLaMA to be for retrieval, they still work pretty much the same. So we're mostly just adding these wrappers around the results of search. And so what I wanna talk about today is pushing past this. Can we make retrieval use all the capabilities of the language model? How can we unlock all these capabilities that we see that are going on in language models and bring them into retrieval? Let me show some examples. Back in 1999, they mostly worked on keyword search. This was before semantic search and mostly was just keyword matching, also known as lexical matching. So you might have a query like, find websites explaining data privacy. You might also have some collection of documents. This could be things like data encryption standards from NIST, a government website, maybe a blog post called "The Wolves Outside Your Data," and then another document called "Digital Protection" from a think tank called Clearlaw. And so if you were gonna pass these to a search engine from 1999, they used exact keyword search. You're probably going to find the first two documents, but not the third. And that's because the third one here is like a synonym. It's data and digital privacy and protection. So it wouldn't match exactly. However, we have come a long way in 25 years. So now we have semantic search. And here what we do is we match it in semantic space. So you're going to go out and you're gonna look for things like paraphrases. And in the same similar space there. So a good semantic search engine should find all three of these. So the next step, I think, is where we should be going with retrieval and where I feel like this new paradigm will really unlock a lot of great benefits. Let me start off by talking about instruction-based search. So here, imagine you have the same query, but you're gonna add— you want a website that uses an extended metaphor to explain. And so here, there should only be one document that's relevant, the ones about wolves outside your data. But note here, you can't do any form of keyword matching to find this document, right? It's not gonna say the word metaphor. It probably won't even say a synonym of the word metaphor, like allegory or something like that. You really have to understand the whole document itself to be able to do this reasoning to see that, oh, this is using a metaphor. And so this is where I think instruction-based search opens up these new categories of retrieval that you couldn't get before. Any sort of meta-document reasoning. And just to show how far you could push this, I think you could move on. Oh, before I do that, it's important to note here that you can't solve this simply by using language models to re-rank. So if you have a big collection, say a million documents, and you want to find one with an extended metaphor. If you just take the top 100 or 1,000 that could fit in a language model, you may not find them, right? Because you're not gonna be able to keyword match metaphor. So I think this is really important that we still need these sort of embedding retrieval models to be able to find these, because this is not something we could just entirely throw to a language model. There's just so much out there. So then the next one I think is out there, but I think really demonstrates how far you could push this, right? And I'm gonna call this prompting-based search or reasoning-based search. Here you could add on things like, I need you to have really high recall or I'm going to lose my job. And so imagine you were just typing this into Google search, right? You're gonna type into Google search, I need you to have really high recall. It's definitely going to try to look for the word recall, right? It's gonna try to keyword match that. But again, we want the model, which is a language model, to understand the whole intent of the query and be able to go out and find it. And this time it'll think, oh, I need to have really good recall. I need to, like, be really careful here, and hopefully it does better. Some areas I think it could help in are, like, document attributes. So these days we preprocess things. You add attributes like date, length, source, all these sorts of things we have to manually add to documents. But an instruction-based retriever should just be able to look at it and know because it understands these things. And so you don't have to do this manual preprocessing. Any sort of meta-level reasoning, like natural language understanding, sentiment, style. You want it to be a positive document, you want a document in the style of a pirate, etc. I think they often include things that are multiple conditions. You want in the style of a pirate and you want a 2-page document or something like that. So you're going to have these and/or not conditions and many more. Really, I think if it's confusing to think about instructions in Google Search, I think the easiest thing to think about is that we're so used to prompting language models, let's just treat our retrievers the exact same way because they use the same underlying technology. We should be able to use all their capabilities. Okay, cool. So for this talk, I wanna talk about two different models that kind of show this in two different aspects. One is an embedding model. It's called PromptTreaver, and I think this gets at this sort of fast, quick, embeddable, understand the whole meaning of what you're saying. And the second is a more recent work called Rank-1, and this uses a thinking reasoning language model that we trained to be specific for search and to be fast, although it's still much slower than, of course, an embedding model. So yeah, let me talk about PromptTreaver. This was a work done with Samaya AI and my collaborators at Johns Hopkins. And I wanted to start by briefly talking about these two categories of models, because this is going to come up a few times. One is this sort of bi-encoder dense retriever, and this creates an embedding. So here you pass the text through the language model separately, like your query and document. They're both going to create an embedding, and then at inference time, you're going to do some sort of quick dot product or cosine similarity to get your final score. So these are very scalable, they're very fast, but they're not quite as expressive. On the other hand, we have these cross encoders. Sometimes we call them re-rankers. It's basically just a language model that you give your query and document to at the same time, and that's going to output some score. Again, very powerful, but it's much slower compared to these bi-encoders. And so for this work, what we wanted to do was, can we make these bi-encoders, these embedders, take instructions? Can we make them promptable? Can we make them reason and follow instructions? And so the key intuition that we did in this, the only thing that makes us different, there's really only one thing, is that we added training data for instruction following. And this is perhaps naive. You might think, oh, has no one really thought of that before? But it turns out that if you look at current retrieval training data, it's pretty much things like Bing search logs. So people are taking Bing search logs in a dataset called MS MARCO, and then you fine-tune on that. But Bing doesn't take instructions, so no one is typing instructions into Bing, so we never learned this sort of capability. So what we did here is we created this instruction training data so that you can take existing models that follow instructions like LLaMA and keep that instruction-following ability. I'm not going to talk really much about the data. What we ended up doing is we had to synthetically generate it. So we took this query and positive document, we passed it through a language model, and it generated this instruction. And so we had to do this synthetically because there's really not any existing data for it. But these days it's not actually terribly hard to develop this sort of data. If you're interested, there's a paper, the data's also open source, et cetera. Um, but yeah, let me show you how this does. So what To make this really simple for the community, we started from a model called Replama, which trains Llama for embeddings. And so if you're familiar with the mechanics of language models, what we actually did is— what Replama does is they take the EOS token, or the last token in the sequence, and they fine-tune the model to create a good sentence representation from that EOS token. And so we are going to use this same recipe that they did, the same model, everything for a direct comparison, we're gonna just add these instructions 'cause we wanna see how these instructions help. We're gonna evaluate on a lot of different types of data with instructions, data without instructions. And let me show you actually a few examples because I think that helps understand what the problem is. The first one's called Follower, and this is a dataset that tests if you're like a user, you type into Google, you then change your mind, make a change to Google, you type it in again, it should update the search results to be closer to what you want in that new instruction. So here you can see in this query about Teflon, they have this instruction they typed in. It used to be can be of any means, but they updated it and now it has to be related to chemicals. So if your retrieval model can follow instructions, it should have more stuff related to these chemicals and less of being in it by any means. And you can see this range from not following the instructions, doing the opposite, -100, to 100, where it follows instructions perfectly, to 0, where it just gives you the same results. And up to this point, no by-encoder embedding model scored above 0. They were pretty much all negative. They do the opposite of what you want. There's also this other really cool dataset called Instructor. They have these different personas that they apply to the query. I'm a student. My teacher wants me to look this up. Et cetera, to see if you can handle these different personas. So how does this model do? I'm not going to go too deep into results, but let me just share a few highlights here. So again, we're comparing to Replama, and Replama gets a negative score for instruction following, pretty much like every other embedding following, embedding previously. However, PromptTreaver for the first time gets this positive score. So we can see for the first time that embedding models really can follow instructions here. On the instructor benchmark, we see again, it's much better than the Replama model that couldn't, that wasn't trained with instructions. Let me show you the more interesting stuff. What if you don't have instructions? So what if you are applying your new model, you have some new customers, you want to be able to use this sort of instruction, but you don't really know what to put there. So you could just use no prompt, like any existing retrieval model. That's a great option. The other option might be you come up with these, like, generic prompts, and maybe you even optimize. If you have some small dev set, you're gonna choose the best one there and apply it to your test set. And so let me show you what we did when we came up with these 10 prompts. Here are the actual prompts that we gave to the model. Be careful when assigning relevance, as your job is on the line. Think carefully about these conditions when determining relevance. And so again, so we take any user query, We're gonna add this little prompt at the end, and if the model understands instructions, it should be like, oh, this is like really important for the user, I need to give them a good result, and performance will go up. If the model's just trying to match a keyword, performance will stay the same or get worse. And so it turns out if we do this on this benchmark called BEER, which Nandan talked about last time, if you have no prompt, both models perform about the same, which is a good sign, right? Like, you want to be able to perform well in both settings. If you add this sort of generic prompt, we see we get this nice gain. And Replama that was trained without instructions doesn't see any benefit. In fact, it gets slightly worse. All right, so you can literally just tell the model what you want. You can say you really need it to do well, and now performance can go up. And this may sound weird, but this is the easiest way to show that this aligns retrieval models with the language model community, right? Because you can prompt hack. Language models, you can tell them all sorts of things. Now you can do the same thing to retrieval models. And that means they can also understand what you mean, right? So if you took these prompts and you paraphrase them, they should be able to do equally well, right? Because it's still the same semantic meaning. And it turns out that if you take a keyword model like BM25, of course you're going to have this really large range because it's really sensitive to keywords. Replama is less sensitive to the keywords. And PromptTreaver, which is trained to follow instructions, is much less sensitive. So these sorts of models understand the whole meaning of what you're saying rather than trying to pick out keywords or trying to match to something in like a paraphrase-like way. With the right data, you can have retrievers that are prompted just like a language model. Having the sort of right data enables these sorts of capabilities because they're already there in the language model. You just have to keep them. And what's most exciting to me about this is it unlocks these new types of queries. You can ask for these meta-level reasoning type things on top of documents that you just couldn't ask before. Like, you just couldn't ask Google about metaphors because it's going to try to keyword match it. But now you have these sorts of systems that can do it, and you don't need to be picky about the keywords. So you don't need to, like, try to help your users say the right things. You can just tell it what you want and the model will understand. And so that's Prompt Treaver. It's a fast embedding model, and I think it's the way that embeddings are gonna go in the future. The next one I wanted to talk about is Rank-1. And so this is the model that's very strong, but it's slower because it's, as I've talked about before, a cross-encoder here on the right. So it takes in the text together, which means that it's gonna be slower than these embedding models. And so this is the type of thinking models, and what the language model community really loves about these is that as you increase the amount of tokens, the amount of words it does in its thinking, performance goes up. But here's O-1 on a math dataset. And as you increase the compute, performance on this math dataset goes from 20 to 80. So they've really shown that these models provide a lot of huge gains to, on these hard tasks like code or science or other sorts of math questions. And so we want to bring those same gains to retrieval. I think if you're not so familiar with these sort of thinking models, one way to think about this is it's like a long chain of thought and it's pretty much used in every model these days. Gemini-01, DeepSeek, they all use this sort of thing. Yeah, how would this even look like in retrieval? So say you have a query and a document, do snow leopards change color? Your model is going to output something like this. So it's actually quite long, right? It's long to read. Luckily, you don't actually have to read it. It can be given to a different language model. You can just take the final output, or if you're really curious, you can read it. And so here it thinks about whether, you know, leopards change colors. It even questions itself here in blue, but wait, like maybe the user says this and they mean something different. And then finally at the end, it's gonna tell you, oh, I actually don't think this document's relevant. It's false. We want this to be fast and personalized for retrieval. We have the procedure in the paper. Again, it's actually quite simple. It's all about the data. So you get some data that can teach the model to do this. You don't even need to do any fancy reinforcement learning. It's pretty basic training. But let me show you the type of evaluation data and the type of things this can do. And so this is gonna mainly focus on the BRIGHT dataset. What they did is they have all these very unique relevance definitions. So instead of just having relevance be, does this answer my question? It has all these different ones. So here at the bottom we have math. So it has a math question, and you actually want to find a different problem that uses the same theorem to solve it. So it's not trying to find the answer, it's trying to find a different question that uses the same theorem. Or maybe you have code and you want to find a different document that uses an alternate function. Or this top one, you want to find some supporting evidence. So it uses all these very cool unique ways of relevance to test if your model can do this reasoning and think through this definition of relevance. So let me— I threw an example into our model to show you. And so this was about, could you find a similar LeetCode problem? And so I gave it the input, and you can see highlighted in red here that the model says, oh, this is a— this is the max area problem, and it uses a two-pointer approach to solve it. And this other problem that you're giving me as a document also uses a two-pointer technique. So therefore, you know, they are relevant to each other. They are similar. And so these models are really able to think about what you're doing to be very instructable to do this reasoning and give you your final output. It's actually very impressive to me because I personally can't just look at some of these things and be like, oh, like, that's a two-pointer problem and tell you off the top of your head. But these models can, and so they're able to unlock that. Yeah, let me briefly show you some results here. We evaluate on a bunch of things, BRITE, the one I just showed you, a negation benchmark called Never, and a multilingual version of the Follower benchmark I showed you before. This model I'm using as a baseline was trained on 10 times more data, so about 6 million instead of 600,000. But yet, adding this thinking goes from 14 to 27, so almost double on BRITE, which is huge. On negation understanding, more than double. And then on instruction following, again, more than double. So we were blown away actually when we saw this, because it was just so much more effective. So I thought maybe like the data, like our data is just so much better. So we tested it with the same model, same data, but with or without thinking. So without this reasoning chain during training. And it turns out that you see this 10-point gain simply from having the thinking. So training it to think is like hugely effective. Let me just end by telling one fun story about this. So there's some old evaluation data that people use in retrieval. It's called DL19 from the Trek Treks. And it was created in 2019. Initially, when we evaluated, we were surprised to see some really low scores. When we dug into why that could be, we realized that the documents that we're finding had not been judged by humans. What that means is when our researchers create these new evaluation sets, they take existing models, go out and find a bunch of documents, and then have humans go and annotate those documents relevant or not until they run out of money. Because there's just too many documents for a human to go through all of them. So you take a top set and you have a human annotate it. Um, and it turned out that the documents these previous models were finding had pretty much all been seen by those humans. From the old systems. However, our model had a lot less of these judged documents. So what we did to make this fair is we personally went through and every document that had never been judged by a human, we went through and judged. So what I love about this story actually is that these thinking retrieval models are actually finding new documents that previous systems hadn't found before, which is really exciting to me because it means that it has this new fresh perspective. Um, and it also probably indicates that the community should move on from these older datasets which I think everyone agrees with. There's many more new ones. This one that we were evaluating on was done before BERT, so it's, it's very old. I think there's growing consensus that we just shouldn't use it anymore. But yeah, in summary, using this test-time compute, this thinking makes these promptable and reasoning re-rankers. You don't need any RL. It's actually really simple to create these. They are slower, but they're much, much more powerful than previous approaches. As just one example, we only trained on general web data. We didn't train on instruction data. We didn't train on multilingual data, on any specific domains. So if you do that, you're gonna see some, like, huge gains. Yeah, and so that's RANK-1. Let me just conclude by headed back to what I started with. So the goal is we want to have retrieval models that work just like language models. So we build our retrieval models these days on language models. They should have all of their capabilities. They should understand what you mean. They should be able to reason. They should be able to follow your instructions. And so what does this actually mean for you if you're a downstream user, if you have some new application, you want to have some new users on the new datasets, a new area? What it means is that these new retrievers benefit from advances in language models. So the next time you see some really cool thing coming out from language model community, it should be easily accessible to you for search, for RAG, for whatever you're doing. It should be easily brought in. And then the part I'm most excited about is all these new types of queries you can now give for RAG and retrieval. You can just type anything you want in, and it will be able to go find it, give it back to you. You don't need to try to keyword match. The language model will take care of it. And so I just want to end by saying that all of these models are open data. They're open source. You can train them yourself. They're MIT licensed, so feel free to use them for whatever. Thanks.

So just to make sure I understand, like when you were talking about PromptTrever, I believe I recall you mentioned that you applied that to the by-encoder. It's a, is that, did I recall correctly?

Yeah, exactly. Okay. It does embeddings.

And if I understand correctly, if you provide instructions on how you want to search, like you have that meta. Kind of instruction, it changes the kind of embedding that will be produced by the by-encoder. Is that the right understanding?

Exactly. So you could give the query and the instruction as like one piece of text and then it creates that single embedding.

Okay. So we have the by-encoder. First question is, how do you— so like a lot of times with RAG, you use a by-encoder and you, like, you batch process a lot of your documents and you put them through the BiEncoder, you store the vectors of your documents so that at inference time you can do that first pass with the BiEncoder before you go to the cross encoder for the re-ranking. How do you imagine, like, this would be operationalized, let's say, like, on inference time? Because I imagine, like, your queries, this meta instruction Is it the, is it, what's the idea? Is it the idea that this meta instruction would be constant when you're doing the kind of this like batch processing, store your documents in the vector database? Or is there some other trick where, you know, you would, you know, so like how do you actually operationalize this more fine-grained thing?

The way that we set it up is that the documents don't get the instructions. So you can batch process your documents just like you would any other retriever. And then at inference time, when you get a query, depending on your use case, you can either have the user type in a long query that can be its own instruction, or you can have some instruction in the backend that you append to that query. And then it just creates that embedding and you do the dot product with all the documents. Maybe the user has a specific instruction. So you want to keep your documents instruction. They're not influenced by the instruction. They're just separate in the data store. And then at inference time, the query gets the instruction and you do the dot product.

Okay. So the instruction is applied only to the query in a way. The way it's trained is like, that's really interesting. And then you mentioned the bi-encoder. Why not also make this part of the cross-encoder? Just out of curiosity. Maybe there's a reason, but just wanted to ask.

Yeah, great question. We actually do have some work doing that. I just didn't choose to highlight it in this talk, but yeah, you totally can.

Makes sense. Yeah, yeah.

I can also highlight, no, it's actually already even out with all the data. It's part of the paper. FollowIR. It also has a benchmark, which we evaluated on in this paper. But yeah, it's all out there as well. So yeah, feel free to check that out if you're interested in a re-ranker that can do that.

Next question I have is this meta search kind of instruction is extremely interesting. Like you're giving, okay, what are some nuances about the search that you should be, if you're like searching for it as like an expert, what do you think is going to happen? Do you think it's going to be? Humans will provide that full thing, or do you think LLMs will be writing it for you, the meta? Or like, what is your feeling like that might happen? What's your like, because you've been like working with this for some time.

Yeah, I think it depends on the application, but I think both are really actually exciting and interesting areas. Um, say you have a deep research system and you're trying to go through it. I think having the Deep Research system know that it can add all these sorts of really precise things will help it narrow in on exactly what it wants in sort of a RAG application. Um, on the other hand, if you're just typing into Google and you're not doing the deep research and you want to be the power user, I think that's helpful. But I actually do think there's going to be a ton of people where normal Google search works. Uh, but if you're the sort of power user where you really want to dial in and you want to find exactly what you want, and then the, the model can take that sort of thing, I think it'll really help them also.

That's interesting you mentioned deep research. Like, yeah, maybe you would have a UX where it asks you follow-up questions to help you build that meta prompt together.

Yeah, absolutely.

Okay, that— okay, so I want to ask you also about the other one, the more expensive one. What was it called again?

Yeah, Rank 1.

Rank 1, okay. And so if I understand correctly, Rank 1 basically is like a reasoning model that you've trained to basically be more powerful when it comes to dealing with context and answering questions related to context. Is that correct?

Yeah. Finding document relevance.

How does it compare to reasoning models off the shelf today? Like the frontier ones, let's say like whatever, 2.5 Pro, O3 Pro, whatever. If you're going to if you put a prompt into those reasoning models to say, hey, carefully consider if this document is relevant, blah, blah, blah, whatever. Have you, do you have any intuition on the baseline against that? How is this? I think you have trained specifically for this purpose. What is, yeah, what's the delta there?

Also, yeah, super interesting question. So we evaluated on some benchmarks that O3 has been evaluated on. And I think on those, O3 is like at 75, and our model is like at 69-ish. So there is a gap between the frontier models there, but our model is again like 7B. We have a 1.5B version. And I think that the models that we're having are most useful if you want to like control the sort of data distribution you're doing, or you want it to be faster, or something you control. If you are going for peak performance, it's going to be hard to beat OpenAI there, but ours is much faster in that way and much smaller.

That's interesting. So perhaps, are you envisioning maybe like this smaller model would be more like in the RAG pipeline itself and you would then hand it off to O3? What is it like? I'm just guessing, but you tell me.

Yeah, I think it could be done that way too. I imagine if you're dealing with a lot of queries, right? So I, and O3 is really good at this sort of tool use. So I could see this sort of thing, like Rank-1 models being used as a tool that O3 uses. Hey, go do this. So you slowly build up, you do your embedding, you do your re-ranking, then O3 puts it all together. Or if it's something like you just simply can't use O3, like you have some government application, you have some secret data you can't pass to O3, these sort of things can be done much cheaper and there you can run it yourself. So it's secure in that way.

Gotcha. What's the easiest way for people? Okay. Actually, before I get to that. Is your intuition that we can use Rank-1 kind of off the shelf for across many domains? Do you think that people may want to train it on their datasets? How easy is it to train on your dataset? Can you talk about that a little bit?

Yeah, we were actually shocked at how well it worked without the training data for specific domains. Like, it actually blew us away how good it generalized. However, I still think that means that if you give it the data you want, it's just gonna like really kill it. It was honestly so easy to train. It's just normal predict the next token loss. You just give it your data, predicts the next token, and it just goes really good. So yeah, if you have something that you want to use it for, I'd recommend just trying it off the shelf. It may just work amazing. If you want to fine-tune it for your thing, it's also easy and probably would work really well too.

Okay, so you're saying like use supervised fine-tuning basically off the shelf for against your data. Now, why is that the case? If it's a reasoning model, wouldn't you— just out of curiosity, just to make sure I understand, how is it that you're able to train it using supervised fine-tuning and why not RL something or the other?

Yeah, that's one thing that I actually really loved about that project. And what do I have in the summary? Is that you don't need that RL. So I think the Quen paper showed a bit of this in that if you have a smaller model, it can learn that distillation from the reasoning chase really easy. And that's, I think, the main reason— I'm speculating here, but it seems to me to be the main reason why OpenAI doesn't release the reasoning chains. Also, Gemini used to. I started actually gathering some from Gemini to train Rank 1, and then during that process, they turned it off. Um, I think these companies have realized that it's just so effective to use them as training that they've completely turned them off. Super well.

So where are you getting the reasoning tokens from now that They turned it off.

Yeah. Nowadays you have to choose an open source alternative, right? So it's going to have to go to like Deepseek or something similar.

Okay. And if people want to fiddle with this and try it for themselves on their applications, what's like a minimal setup that you recommend people do? Is this easy to wire up in LanceDB and something? Is there a recipe that you recommend people look at?

No specific constraints on framework. I will say VLLM is really effective for it. There's actually a demo on Hugging Face. Hugging Face right now for it. You can check it out, it has a GPU, you can test it out for different queries you have, see how well it does. Um, the otherwise it's just like a normal language model, so any sort of framework that handles language models, VLM, works great.

And how can people find that thing on Hugging Face?

Rank 1, or my name.

Okay, got it. Thanks for entertaining my questions. This is very exciting and very interesting of a presentation, so thank you so much for doing this.

Thank you so much for the opportunity, for the really great questions.