#305 RAG 2.0 and The New Era of RAG Agents with Douwe Kiela, CEO at Contextual AI, Adjunct Professor at Stanford University, Inventor of RAG

Generative AI is transforming industries at an unprecedented pace. But as AI changes how you or your team work, one thing is clear. Your skills also need to evolve. At datacamp, we offer everything you or your team need to adapt and thrive with AI. Whether it's business users looking to get the Most out of ChatGPT and Copilot, or developers and data scientists looking to fine tune models, you can learn the entire AI skills spectrum on Datacamp. Power your AI transformation today. Start learning@datacamp.com. So a lot of RAG systems like two years ago were what we called like Frankenstein's rag, where all of these parts are kind of coupled together, but they're not really designed to work well together. So our RAG 2.0 approach is about making sure that all of the components of a modern RAG pipeline. So that's not just the vector database and the language model anymore, it's much more complicated as a system. In our case, all of these components are designed to be state of the art and they're also designed to work well together.

Welcome to Data Frames. This is Ritchie Retrieval Augmented Generation, or RAG to its friends is a popular technique for retrieving supporting information to include in your prompt. This is supposed to reduce the hallucination problem that LLMs suffer. It's wildly popular at this point. It's an essential component in almost every AI application. For all its successful, I quite regularly see people proclaiming its imminent death in favor of something more sophisticated. Today I want to know whether the reports of rag's death are greatly exaggerated or not. Our guest is Dauer Keiller, who was part of the team at meta that invented RAG. These days he's CEO at Contextual AI, a platform for creating RAG agents and custom LLMs known as contextual Language Models. The Contextual blog also cheekily introduced the concept of RAG 2.0. I'd like to find out whether this is a worthy successor or marketing hype. All right, let's find out about the fate of rag. Hi Dawer, welcome to the show.

Hi, thanks for having me.

So I've been hearing a lot in the last year or so about people predicting the death of Retrieval Augmented generation. Why do you think people keep thinking this?

That usually happens with good simple ideas where people are trying to sort of rewrite history, maybe from a marketing perspective a little bit. And RAG is such a simple idea that I think it's a little bit silly to declare it dead. I guess everybody listening in here knows what RAG stands for, right? Retrieval Augmented Generation. And so the G is just any gen AI model. And then you want to make that model work on your data, which means you need to augment it with your data. And the way to do that is through some form of retrieval. So it's such a kind of generic paradigm that it doesn't really make sense to pronounce it dead, I think. But from a marketing perspective, a lot of people keep saying, oh, you don't need rag, you need fine tuning, or you don't need rag, you need long context windows. And yeah, I mean, we can go into the specifics of each of those, but I think these are mostly marketing tricks.

So I suppose it kind of feels a bit like linear regression is very simple. It's been around for a century and it's still in use because it's simple, even though there are more sophisticated models out there.

Yeah, exactly. Yeah. We even bought a domain name is ragdeadyet.com where we can point people to a blog post about whether RAG is dead or not. Yeah, these are the types of things that you cannot really declare dead, I think, but I guess I'm biased.

Are there any problems though? Like, are there any limitations to the sort of standard approach?

Oh, absolutely. So I think if you look at agents, right, agents in general, retrieval is just one of the tools in the toolbox of an agent. So I think it's very true that we should not just have RAG as the only thing that these agents can do. But yeah, that feels a little obvious in terms of the current limitations still of rag. I think part of the solution there is actually long context. So if a retrieval system is imperfect, then ideally you want to cast a wider net, so have more information that you can put into the context of the language model in the hope that there's something useful in there. And then that's where you need a longer context. That's why I always talk about these dichotomies as sort of being false. Right. It's like you need both. You're not going to put the entire Internet in the context of your language model. So that's why you are going to do some search. So rag, basically. But then you want to ideally put lots of search results in the context of the language model so that it can do its job and answer the question correctly. Right. So, yeah, we need all of those things.

So just to make sure I've understood this correctly, like, the whole point of RAG is that you're going to reduce hallucinations by just retrieving relevant information, putting that inside your prompt, and then that's going to Help the large language model give you the correct answer. And if you've got a longer context window, you can put more information in there. So that's going to assist. So the idea of long context windows and rag, they sound like they're competing, but actually they're complementary.

Is that very, very complimentary? Yes.

Okay, wonderful. Now I know you're completely contextual in your blog. You've got, you've introduced the idea of RAG 2.0. Can you tell me what's RAG 2.0 and how is it different from RAG?

Yeah, so the way we do RAG is a bit different, I guess, from what everybody else is doing. So we started the company when we saw the world get very excited after ChatGPT and then very frustrated because the technology wasn't quite ready. And that was especially true for enterprise use cases. So we knew that RAG would be part of the answer, obviously, given what we knew about rag. We also knew that RAG was just the first idea there and that even in the RAG paper we talk about sort of what the vision is and what we really want to achieve, which is that these components are designed to work together. So a lot of RAG systems like two years ago were what we called like Frankenstein's rag. All of these parts are kind of coupled together, but they're not really designed to work well together. So our RAG 2.0 approach is about making sure that all of the components of a modern RAG pipeline, so that's not just a vector database and a language model anymore, it's much more complicated as a system. In our case, all of these components are designed to be state of the art and they're also designed to work well together. And you can do that through training on the same data distribution, essentially, so that all the parts are designed to work well together. They're literally like trained to work well together. And so that combination of having very good components and then having a very good compound, that makes our system much better at RAG than anybody else.

In theory. That sounds like a very useful idea. The idea that the data in your vector database and the data in the large language model, they're kind of harmonized in some way, so all the components are working together. Do you have a sense of when this might be useful and what sort of benefits you might get from it?

They tend to be high stakes use cases with a low tolerance for mistakes and sort of high accuracy requirements where you want to have very accurate attributions. So very often this is in regulated industries and, you know, where you're also sensitive about your data. If you have those characteristics, then we are much better than anything else out there.

Yeah, I can see if you've got a high requirement that the answer you're giving is correct in some way you're going to need to put more effort into. Do you have any examples maybe from one of your customers of like things that have been built using this approach?

Yeah. So one use case I'm very proud of is the work we've been doing with Qualcomm. Their customer engineering department is using us where they're only Genai deployment at scale, as far as I know, where these are thousands of engineers who are using us on a daily basis to answer very complicated questions. So these are not the simple kind of questions where you have like your internal search and maybe it's, you know, something like glean and you would ask it who is our 401k provider? Or how many, many vacation days do I get? Right. That's not really where the ROI of AI is going to come from. You want to focus on these much more expert kind of specialist knowledge worker use cases. And so that's what we did with Qualcomm. And we can answer very complicated questions where I don't understand what the question means and definitely don't understand what the answer means. But the system does a really good job at explaining the information and giving the right answer.

So if you just got a simple like true or false or give me a number, retrieve one fact kind of situation, then rag 2.0 is maybe overkill, is that right? But if you've got a more complicated question, then that's where it's going to shine.

It's not really overkill, it's more that. So the RAG 2.0 happens during training, right? Not during inference. So during inference it's still a normal RAG system, but just a really, really good one. So it doesn't make things more complicated for you. If anything, it makes everything easier because we offer it in one platform where you can build these agents. Really? And 10 seconds, I can build a state of the art rag agent in 10 seconds, which is, I think, pretty powerful.

10 seconds, that's a pretty. Well, it's about as short of a development time as you can get. So talk me through. What do you do then to build something in 10 seconds?

So we had these two concepts of agents and data stores in our platform. And so what you would do is you would create a data store and then you would tell the agent that that is the data store that it has to work on top of, and you put some files or a database into that data store. And now you can talk to that data and do RAG on top of that data with a state of the art RAG system.

That sounds very straightforward. Just copy some files into a database and say, okay, go look at them. Tell me something about what's in there.

All right, nice.

And if you're going for one of these very sort of strict use cases that you mentioned, like the idea of regulated examples, do you need to do some tuning of that, like some kind of optimization to make sure it works properly?

Yeah. So you can do that. So our hope is that out of the box our performance is already good enough, which turns out to be the case for many use cases. But yeah, if you work in finance, for example, then you really want to get maximum performance. And so that means that you need to really specialize for the use case. And you can do that through our platform. So we allow you to tune not just the language model or just the retriever, we allow you to tune the entire RAG pipeline for your specific problem. And so often that leads to pretty substantial performance improvements on top of already state of the art out of the box performance.

All right, so it sounds like a lot of the secret to this is around the fine tuning steps and these other steps to make sure that the model quality is high.

Yeah, some of it is. So it's also just about having very good components. Right. So for once you have a grounded language model that is designed for RAG and nothing else. So it's really specialized for rag, then you can use that. If you want to use a re ranker that is state of the art and that can follow instructions, you can also do that. So it's the same for our retrieval pipeline and for our document intelligence pipeline. So all of the components that make up our state of the art RAG system, they're available on their own as well because we just like seeing what people build with them.

All right, so it seems like a lot of the secret is just like starting with a model that is designed specifically for just retrieving information, then rather than having a general purpose model.

Exactly, yeah. So a general purpose model has to be good at everything. So as a result, it is not as good at RAG as it can be. So if you really just specialize all these components just for rag, then you get something that is much better.

I'm curious as to how you measure what the benefits are, because it seems like it's quite tricky to work out what the quality of responses are in general how do you go about doing this?

Evaluation is such an important area and so underappreciated. So we do it in a variety of different ways. I mean, obviously we look at kind of standard benchmarks, right? And so like Reg QA arena is a benchmark and we are state of the art on that. And so that's what we care about too. But that doesn't even necessarily translate to real world performance. So we look much more closely at our actual customer data sets and sort of the process of UAT user acceptability testing. So do people actually consider this answer to be correct and useful? And that's ultimately what you care about, right? That's where the ROI comes from. So that's one thing. I think the other thing that's maybe worth mentioning here is that we have this framework around natural language unit testing. You're familiar with unit testing, right? It's what you do with code is like just step by step measuring small things about your code, small units, and then making sure that they are correct. And you can use the same idea for language model responses where you can delineate very precisely what the characteristics are of a good answer. So you could say, okay, like it needs to mention this thing first. It needs to be in this particular style. It can absolutely not talk about this thing. And you know, whatever the characteristics are of the answer, you can write specific unit tests for doing that gives you much, much richer signal than just looking at what everybody else is doing now with LLM as a judge models, which is just like, is this generated response equivalent to my ground truth response? Maybe that makes sense if you're generating a few sentences, but if you're generating a long answer, then that's just not going to cut it.

Okay, so you want to be able to ask specific questions about the response and just check does it match all of these different criteria and you can do pass off alpha, lots of different things then.

Exactly. But that's really great too, right? Because if you think about like a regulated industry, you will have people in like the model risk management department of the bank and they need to write out what a good answer looks like and what cannot be a part of any answer. Right. So if they can just write those unit tests in natural language and then the system can test against that for every generation, that's something that will make regulators much happier. Basically. Yeah.

I mean, certainly from my experience of creating software, it's like when you write the test, it really helps clarify what you actually want the software to do. And I'm sure it's the same situation with your large language model. You think about, well, what driven your answer to be? Yeah, yeah, absolutely. Cool. So I guess related to this is the idea of maintenance. So if your data is changing fairly rapidly, how do you go about maintaining your model? Just make sure it's continually giving the right results.

What RAG is useful for is making sure that things will keep working on your data even as the data changes. Right. That's part of the appeal of rag. So I think you kind of do that automatically there. But one of the things that we're always thinking about is how can we make sure that we have the best components and continuously keep updating the best components to make sure that they're integrated into the overall pipeline. And so that actually takes a lot of effort. I just staying at the frontier of AI, which is moving so quickly, that is a lot of work. So yeah, that's exciting too that we get to do things like that. Right. As a broader kind of research community.

Okay. So in theory you shouldn't have to do too much maintenance. As long as the system's in place and you sort of test it once, even if your data changes, it's still going to.

I was at this Gartner conference, it was data and analytics. So everybody at the Gartner conference and this is really not my crowd, to be honest. These are not like, you know, AI people, they're like real enterprise people. And it was really eye opening for me to just be there and talk to all these amazing people and get their perspectives, which were very different from what I'd been hearing everywhere. And the whole conference was just talking about making your data ready for AI and I just felt like that's a massive cop out that you should not have to make your data ready for AI. You should make your AI ready for your data. So that in an ideal world, and I think where we're headed is that you don't have to do anything to your data to make it work. You just have very good AI that works on top of that data. And we just have to accept that data is noisy. And so that's why we have like this multi stage retrieval pipeline. That's why we have this powerful re ranker that can follow instructions that. Exactly because you need to make sense of noisy data and filter out the things that you don't want and make sure you get the things that you do want into the language model. So yeah, in an ideal world, in the long run, I can't promise it now yet, but in the long run, you want to make sure that AI just works on your data and you don't have to make your data AI ready.

I think as heard, like everyone listening in who works in data governance just gasp all at once.

Just, just wait. Don't waste your time preparing for this work in the near future.

Okay, that's interesting. So do you have any advice for people who work in data governance then? Like if your company is making AI applications that involve like search or retrieval, what do data governance people need to do?

Be very careful about your metadata. So I think that's really one of the crucial parts of state of the art modern rag pipelines is making sure that you have high quality metadata sort of annotations for your documents and database schemas or things like that. So that is definitely still going to matter, having rich metadata. And so yeah, the other thing is really about I think thinking carefully through like your entitlements model, especially when you have multiple data sources that might be very disparate. We talk to a lot of companies that have like SharePoint and Confluence and like Jira and Slack and like Google Drive and all kinds of different things and then they want to work on top of all of those. But getting that to actually work with a proper sort of role based access control entitlements set up is not trivial at all. So that's one of the things that we are good at. But if you have a more centralized way of dealing with that from a data governance perspective, then you can save yourself a big headache.

Okay. Yeah. So certainly I can see how lineage is incredibly important. Do you want to go into a bit more about the entitlements? Because it sounds like if different people have got different permissions on data, it makes retrieval a bit complicated.

Yeah. So if you think about like real world companies with like a hundred thousand employees and not everybody has access to the same amount of data, some like companies actually from a regulatory perspective have sort of hard fencing between departments. Yeah. So you need to make sure that you capture that and that you don't make any mistakes. So setting things up the right way for handling that at the scale of a hundred thousand people in a large company, that's a very interesting problem, I'm sure.

Yeah. So in that case, I guess you want the model to give different responses depending on who is asking the question because they're going to see different data.

Yeah. So like I always enjoy saying everything is contextual, so you want to get that context into the model. And so for every user and every user interaction, even the Context should be different.

Okay, nice. Now I know handling permissions and things has been around forever in sort of relational databases. I don't know what the status with vector databases. Like, is it easy to manage who gets access to what data and things? Is it as sophisticated as you sort of traditional relational databases?

No, it's not. It's not nearly as sophisticated. Even with relational databases, I think it's actually not that easy. So you can do it in like one sort of single database, but doing it across databases is also not easy.

Is there a solution to this then?

Yeah. So the way we do it is we synchronize the data with our sort of data store concept, which we talked about. Right. And then when we do the retrieval step, we validate using an entitlements API that sort of calls all the upstream APIs to make sure that you still have access. And so that obviously cannot always happen in real time, depending on your latency constraints. But there are some sort of synchronization steps that happens in between there. So the model is basically like you ingest, you do the retrieval over that index, so kind of a complicated index. And then when you find the results, you validate that the results you found are actually accessible by the user using an entitlements API.

So this is one of those problems where it sounds like it's going to be really better if someone else solves it for you. Like, it doesn't sound like the sort of fun thing you want to play around with yourself.

That's what I've been trying to tell people. Stop trying to DIY complex RAG systems. It's not worth your time. Just like you wouldn't build your own database or your own language model. Like, you shouldn't build your own RAG platform. You should be building RAG agents and applications on top of that platform to solve important business problems for your business.

I mean, that seems incredibly important. It's like try and figure out where you can add the most value to solve problems to your business. It's like on where you should be spending your time. Do you have any examples?

Yeah, so. So the differentiated value for a company is what companies are all about. Like, your company wants to be better than your competition, and that's sort of the goal of every company. And so the more you can focus on that and thinking about, okay, how can we use our data and how can we automate some of our processes or improve some of the things that we do, the more time you can spend on that and not on like the optimal chunking strategy. Or like how to make sure that your vllm doesn't continuously go down. Or like basic problems that you have to deal with on the sort of red plumbing side of things. The less you have to worry about the plumbing, the more you can worry about actually solving real problems that add business value.

That makes sense. Just like really think about like how can I actually make my business better? And then try and get as close to just creating stuff on that level rather than worrying about too much about the low level infrastructure if possible. Now there's something you mentioned a few times is the idea of a re ranker. It sounds quite technical, but I feel like it's one of these sort of underloved components of AI systems that isn't talked about much. So do you want to explain what is a RE ranker and why would I need one?

The RE ranker is super important. I think every modern rag pipeline in production that I've seen has a re ranker in it. The idea is actually very simple. It's when you do retrieval because you need to do retrieval over large amounts of data, you cannot have a very big model doing that. So embeddings models are you cache the embeddings, right? It's relatively cheap compute. Maybe you do some like keyword search and then you get a bunch of results. But now because you had to do this at a very large scale, you will have made mistakes because you had to do it sort of quickly enough, right? So you couldn't have like a smarter model take a look at it. So that's what the re ranker is. It's a smarter model that does a second pass at your initial retrieval results and says actually like this, you know, I can see why it was sort of relevant, but it's really not relevant. Or this one is actually super, super important for getting the right answer. And so our re ranker is state of the art by a large margin. So it's much better than other rebrankers out there. And it's the only one that can follow instructions, which is really important because now you can tell it what your data hierarchy is basically. So again, at the level of the user or even of the individual query, you can say this is the priority of my retrieval results based on my preferences. So most recent, first, if it is a PDF, then it's more true than if it's a slack message, you know, if it comes from our internal wiki or if the boss wrote it, then it's more true than if it's an internal documents and it has draft in the title. Right. These types of rules for how to prioritize data and how to break conflicts in rag pipelines. That's what you need an instruction following reranker for. That's one of the things that we have in our platform and that's one of the things that makes us better.

Okay, yeah, so certainly I think like the standard approach for retrieving information from a vector database is just like it's simple linear algebra, it's like cosine similarity or whatever.

Exactly.

So yeah, yeah, probably I use what like 19th century math. I'm curious, you mentioned like if you want to re rank information. Well, you've got to have a set of rules on like precedents. Is that something you need to write manually, these rules or can these rules be automatically generated?

We usually have pretty good default rules in place for free ranker. And so that would be things like most recent first and kind of common sense rules so you don't have to come up with them yourself if you don't want to. You can also ask a language model based on your data what it thinks the rules should be. So you can do it yourself or you can just go with the defaults.

Again, this sounds like something where you could get really, really into the weeds and like trying to figure out what's the optimal strategy here. And it feels like again that could be.

Yeah, yeah. So if you have a very specific rag problem and there's a very specific thing that needs to happen there just because of the problem you're trying to solve, then you could try to solve it through the system prompt, but ideally you solve it by prompting your retrieval pipeline. Right. And so you obviously can't do that in your first stage retriever because it doesn't have the ability to follow any instructions. But you can do it in your second stage. So in your re ranker.

I guess the tricky part here comes when what happens if you've got information that conflicts with each other. So you've got a problem that's trying to give like a specific answer. So pull some information from your knowledge base or whatever and you've got two different results that give you a different answer. So talk me through, how do you go about resolving this?

It depends on the type of conflict. But you can tell it through instructions how to deal with that conflict or how to break the tie, basically. So you could say most recent first. So if you find two documents and one is more recent, then you rank that one much higher. Or you could say different data sources. It's like, I prefer this data source over that one. So you can put that in the instructions too. That's why it's so important that you can prompt these components is because there are different strategies for dealing with conflicts. So one is you go for most recent, the other is you report both results, but you say that one is more recent. So it really depends. And that's why it's so important that you get control over those aspects of a RAG system.

It's good that there is some way of sort of resolving these conflicts. I'm curious, is there any way to feedback from we have these conflicts to this is then going into your data governance strategy.

So one of our very special capabilities is our ability to say I don't know, which is an underrated feature. It's much better to say I don't know than to make up a wrong answer. So that's a really, really good thing. But then when you have an I don't know answer, ideally you want to have the ability to annotate that answer so that the next time you get the question, you do have relevant information in your documentation for resolving that question. So that's one feedback loop and then the other is, yeah, we collect feedback through our UI or through our APIs, and then you can export that feedback and actually train on that feedback. That's how you can specialize it for the use case over time. Just keep making it better and better by incorporating the feedback that you get.

Okay, so this sounds pretty useful. It's like it's going, okay, we've got wrong answers, we're going back to the people who actually curate the data or maintain it, and then that's gonna feed into a better system later on.

Yeah, yeah. So that's over time, at least how you can capture most of the distribution. Obviously the tail of the distribution you're never going to fully be able to capture, but you can fine tune the system maybe to generally be better also at the tail.

All right, so we talked before a bit about the idea of RAG agents. I'm curious as to whether RAG agents have a different architecture to other RAG applications.

Yes and no. I guess we should talk a bit about what an agent is. Right. Because everyone's very confused. So I think an agent is a very general concept. I mean, so it comes from reinforcement learning or even it's maybe even older than reinforcement learning itself, where it's really just about like a policy that takes actions in an environment and that policy has some sort of state and so it can manipulate the environment, but it doesn't necessarily have to manipulate the environment in order to be considered an agent. So another way to put that maybe a bit closer to home for folks listening to this podcast is if you think about an agent that can do SQL queries, some people will say, oh, but it's only an agent if it changes the environment, which would mean that it is only an agent if it generates insert queries or update queries. And if it just does select queries, then suddenly it's no longer an agent. So when you explain it like that, it sounds a bit silly, is like, obviously that's still an agent. Like Deep Research doesn't do any insert queries or update queries, but it does do a lot of useful stuff. So I think a much more useful definition of an agent is just something that actively reasons. So something that thinks about what it's doing, formulates a plan, executes on the plan, and then can revise that plan based on the information that came in. So that's active reasoning. And so the really exciting technology that has enabled all of this is just test time reasoning and the insight that shifting the compute from the training side to the test time inference side actually has very, very nice properties.

Okay, yeah, I like that. So I think your first explanation, I was just introducing all the important pieces of jargon from reinforcement learning for you understanding the idea of like policies and environments. But yeah, I like the idea of just doing reasoning at the time where you're asked stuff. And it can, can do things step by step and form a plan and go and execute it. Okay, all right. So yeah, the second part of this was if you've got some retrieval augmented generation involved in this, do you need to do things differently with an agent as opposed to just a simple chatbot?

I think like the simple chatbot is a subset of the overall concept of an agent. So you can have a chatbot agent and probably for most chatbot use cases, it doesn't have to do a lot of thinking. So the same agent can also power the chatbot use case. It is just more powerful. It depends on how much you want to budget for test time compute. And so if you want to minimize that, then you probably just have something like a standard chatbot. But the boundary is very blurry because when I have a RAG application, which maybe is not really an agent, but as a part of my rag step, I do query decomposition and I formulate sort of a plan and then I do some filtering on top, it's sort of an agent. Right. That is what Agents would do, but the agent sort of maybe determines that more dynamically on the fly, sort of what it would want to do. But yeah, the boundary is not as well defined, I think, as a lot of people like to pretend. So overall, I think this really is the year of agents. There's a lot of excitement about them. They don't really work quite yet. I think a lot of people are sort of blown away by the potential. But like, in practice, I haven't seen any real agent deployments that have like material impact on a company's business yet. I mean, I'm sure it's going to come, but it's going to take some time. But these agents, obviously they need to work on your data too, right? Just like what we were doing before, which NAI is like, yeah, we need agents, but the agents need to work on your data. So they need to be RAG agents because RAG is the way that you get things to work on top of your data. So RAG is one of the tools that these agents need to rely on for them to be useful, especially in enterprise settings.

This is interesting. I think the big difference is with the sort of standard RAG approach, you've got some software saying, okay, let's shove all these bits of information from the vector database into the prompt. And so it's kind of being pushed to the LLM. Whereas if you're doing things with a sort of inference time reasoning, then the LLM has to ask for information, just kind of pulling it in. Is that about right?

Yeah, that's right. But again, right, it's not really like one or the other. It's really a spectrum. I think most modern RAG systems, they probably have some kind of classifier that says should I retrieve or not? And then based on that you say, okay. So like if I say hello, then you don't have to retrieve in your RAG chatbot, you just say hello back. So that's already kind of active retrieval. Where in the old RAG setup, where it's really passive retrieval is you get a query, you always search for that query in your vector database. You always give the results to the language model. That's prehistoric at this point. So it's much more complicated. There's active decision making involved in these RAG pipelines. There's a lot of filtering, there's the re ranker that has a huge impact. There's this sort of active retrieval component. There's query decomposition, which is almost like formulating a retrieval plan. That's all very agentic. But now when you have an agent, you can do that much more dynamically and that's, that's why that's so exciting. Yes.

You got a lot more flexibility and like what the behavior is going to be like rather than having a more rigid pipeline.

Yeah, exactly. And it can also like recover from its mistakes, which is very important. Right. So if it retrieves something and it thinks, oh actually this is not what I wanted, let me try a different query and then it gets the result, that's very powerful and that's something you can do with abilities. And then the other thing is more around sort of multi hop questions or like multi step reasoning. It's like first I need to know this and then I need to know that and then I need to compare those things and then maybe based on the result do something else. Right. That type of multi hop problem that's I think very interesting in terms of business value as well.

So you mentioned that you think there haven't been any deployments of agents that have really wowed you so far. What do you think?

They've wowed me in terms of like seeing the potential, but they haven't wowed me in like, oh, this company has saved like $10 million this year because they had an agent doing something like a proper agent using test time reasoning and not something that was branded an agent, but that was something that we were already capable of doing before kind of reasoning models.

Okay, so what do you think the sticking points are? Like, where are we falling short?

I think generalization, actually having it work in real world settings where it's not in the toy domain. So getting things to work in a toy domain or in a nice demo. That used to be my sort of story around rag. And that's still very true. People think that rag is easy because you can build a nice rag demo on a single document very quickly now and it will be pretty nice. But getting this to actually work at scale on real world data where you have enterprise constraints, it's very different problem. So it's the same with agents where it's like, oh, I can make something like do this one particular thing when I prompt it and basically just make everything look good for exactly the one thing I wanted to do. But then when you actually have to make this work in a real world setting, then everything just breaks down very quickly still. So that's going to get better over time, obviously. So I think the hype is justified. But it's going to take some time for these systems to be enterprise grade enough for anybody to really deploy this in a critical setting.

Since A lot of companies are thinking, okay, we've got to get in on the edge and it's game. We've got to build something. What can you build that is likely to work and add value?

So you can build basic solutions for relatively boring problems. One thing you can do is go for the basic problems where you ask the basic questions like internal search, but that doesn't really get you value. It's much more like trying to find workflows that exist in your company that are a little bit boring, but where it's important that you get it right and where it requires some expertise. If you can solve those problems, then you can make your company much better. And so these could be very simple things from like checking for compliance against your set of policies or doing basic research. We have a very nice demo where we fill out Excel spreadsheets on unstructured data on the fly so that you don't have to manually go and copy and paste. You're just directly in your Excel call a macro and then fill out the spreadsheet with unstructured data from different data sources. Doing things like customer support, there's a lot of low hanging fruit in kind of the code gen side of things. A lot of it is happening across the board. It's just, yeah, doing this the right way takes time. So there's a big gap between sort of where the hype cycle is and where like reality is in enterprises. But I mean it, it's coming, it's just, yeah, it takes time.

So it sounds like the best approach then is to go for maybe slightly more narrow use cases where there's less flexibility needed, less generalization needed and I guess almost build maybe disposable agents. It's like something you can build quickly and then just solve your problem and then done.

I like that idea. You can build disposable agents on our platform. Maybe we should rebrand like that disposable agent platform. Nice.

Cool. Okay, so just to wrap up, what are you most excited about in the world of AI at the moment?

Yeah, so I'm obviously very excited about all the genetic things. I think for me personally where I see a lot of very interesting problems is at the intersection of structured and unstructured data. So you have a bunch of documents, but you also have your traditional kind of structured relational databases, your Snowflake or your Bigquery or whatever you use. And now you want to kind of cross sect that information using agentic rag. So if you can do that, which you can now start doing because of these Agentic abilities that unlocks so much interesting potential. So I think that's really exciting. The other thing is multimodality is obviously still very unexplored. I think every time there's like an image generation feature getting shipped, that kind of goes viral. But I think image understanding is actually much more valuable from a kind of enterprise perspective. So I think that's also a key unlock that is coming soon. So chart understanding, for example, and things like that. Understanding, you know, a McKinsey slide deck that has lots of different diagrams and charts and things in there. Like right now, that's not really within the capabilities of these systems, but it's coming very quickly.

Okay, I got a follow up on that. So talk me through. You want basically the ability to understand a presentation and just like throw it a PowerPoint and then explain what the output is.

That's the simple case. So the hard case is I have a hundred million PowerPoints that my company has made in the history of my company, with a hundred thousand people in my company, and now I want to answer questions based on all of that information.

Okay, yeah.

So it's not like one single PowerPoint, because that you can kind of start to do, even though it's not very accurate, but you need to do it at scale, so over lots of presentations. So if you can do that, then you can do very interesting kind of synthesis on top of it. Right. So how did our perspective on a particular type of thing change over the years? And then you can just find the relevant presentation decks that cover this particular thing, look at the charts, reason about the trends in the charts, and then combine that into a new insight. That's all kind of starting to become possible now.

That would be very interesting. Although I have to say, definitely had few colleagues in the past where it's like, even with them Talking over the PowerPoint, I'm not sure what they're all about. So, yeah.

What everybody says, garbage in, garbage out. You cannot be much better than your data, as you can try to reason about it intelligently, but there are limits there.

Absolutely, absolutely.

Wonderful.

All right, so actually, since you've been involved in RAG since like the beginning, I mean, you were there when RAG was created, part of that team. So do you think RAG has sort of lived up to your expectations as it panned out as you expected?

That's a nice question. I mean, I think the original vision was always that we would have kind of a decoupling between the knowledge and the reasoning, where the reasoning is really just like taking whatever the relevant Knowledge is and then giving the right answer on top of it without having any of the knowledge in it itself. So that didn't really pan out. And so that's part of the reason why these systems hallucinate. And that's, I mean it's a longer story but for rag, I think when the paper came out it was very focused on open domain question answering which is the domain that you evaluate these systems on. So it was well received. But at the time Gen AI like vector databases basically didn't really exist. That became a thing after the paper. Language models didn't really exist. We had like Bart and T5 but there wasn't really a concept of like an autoregressive generative model. So I think like the reason RAG became such a popular paradigm and concept and why it's called RAG is because of the G. So it's really just because Gen AI became a thing that RAG became a thing. And there are lots of papers from around that same time. There's an amazing paper from folks at Google called Realm where that didn't become the name of the paradigm because it didn't have a G in it. It was a mass language model. Right. So yeah, hindsight is sort of 2020.

It's amazing how like small change the name has a big difference on your success or not.

I mean it's not just the name. Right. It's that we were interested in trying to see if you could generate the answer without sort of. So the alternative is just predicting the answer which is what was much more normal to do at the time. So I guess we were ahead of our time in the right way there. Wonderful.

And certainly, I mean it's taken over in so many ways. It's yeah, ubiquitous now. So yeah, obviously very successful. All right, so just finally I always want new ideas for people to follow. Whose work are you most interested in at the moment?

I think there is a lot of interesting work happening in this new test time compute paradigm. So I mean I still have my part time Stanford adjunct professor gig which is great for me to kind of stay at least a little bit up to date on the latest research trends and I think there's just a lot of interesting research happening around this test time reasoning and what you can do there. I think we've only just scratched the surface and so what happened with Deep SEQ and things like that has been very encouraging. I think from that perspective where it's actually not that hard for like non frontier lab folks to do interesting things in this space and have impact. So, yeah, I follow a lot of just smart academics.

Okay. Smart academics. Always worth following, I think. Yeah. It's a good genre of people to get ideas from. Nice. All right. Thank you so much, Dawah.

Thanks for having me.