Building a Smarter AI Agent with Neural RAG - Will Bryk, Exa.ai

All right.

So I was going to give a live demo coding. But— well, I will. But I know you all are actually here to hear a cool story. So I'll tell you a story about web search built for AI, and then we do some coding at the end. This story will end with this slide— one API to get any information from the web. And you'll know what this means by the end. But the story starts in 1998. And what you're looking at is the state of the art in information retrieval in 1998. You type in a word—

Australia—

to this new search engine called Google, and it magically finds you all the documents that contain the word Australia from the web. It's crazy. And the big insight of Google was they had this PageRank algorithm, so the results are ranked by authority based on the graph structure of the web. And this was a clever algorithm, and it was really cool. I was 2 years old at the time, so if I was conscious, I would have thought this was cool. Okay, and now our story skips 23 years to 2021. By this point, I was conscious, barely. And, and I noticed that, you know, GPT-3 had recently come out and it was this magical thing that you could input a whole paragraph explaining exactly what you want and it would really understand the subtleties of your language and give you an output that exactly matched. And it's hard to remember how magical this was, but it was really magical in 2021. And at the same time, I noticed there was Google which, you know, you type in a simple query like "shirts without stripes" and it would give you shirts with stripes, which is crazy. It, like, doesn't understand the word "without" because it's doing a keyword comparison algorithm. And so I decided that for the next at least 10 years, I'm going to devote myself to building a search engine that combines the technology of GPT-3 with a search engine to make a search engine that actually understands what you're saying. At a deep level and understands all the documents on the web at a deep level and gives you exactly what you asked for. This is a very big idea and we're working on— we've worked on it for 4 years and a lot of progress, but it would change the world if you actually solve this problem. And so in 2021, we joined YC. Summer 2021, we raised a couple million dollars and we did what every YC startup should do. We spent half of it on a GPU cluster. I'm joking. You shouldn't do that. And then we also followed YC's advice where we didn't talk to any users or customers for a year and a half and we just did research. Again, you shouldn't do that. You should talk to users. But in our case it made sense because we were trying to solve a really hard problem, which is like redesign search from scratch using the same technology as GPT-3. This like next token prediction idea with transformers. What if you could apply the same thing? To search. And this is actually one of our, uh, WandB training runs. Um, the purple one I believe was a breakthrough where it like really, it really like learned. There was like a few breakthroughs along the way involving like random datasets and different transformer architectures that we were trying and this purple one like really it starts like work well. Um, and the general idea we had was like, okay, so what is, what is a search engine? You have like a trillion documents on the web and traditional search engines on a very high level, will create like a keyword index of those documents. So for each document, you are— you say— you ask what are the words in those document and you create this big inverted index where you map from like words like brown to all the documents that contain that word. And then at search time, you know, when a search without stripes comes in, you do some crazy keyword comparison algorithm and get the top results. That's obviously a simplification of what Google does, but at a fundamental level, it's doing— it's a keyword comparison. But the idea was like, what if you could actually— so with Transformers, like, the big thing is like, what if you could turn each document not into a set of keywords but into embeddings? Uh, and these embeddings can be arbitrarily powerful, right? Like, it's a list of— embedding is just a list of, of numbers, and it could represent lots of information. So an embedding, it doesn't just capture the words in the document but also the meaning, the ideas in the document, and the way people refer to that document on the web. And, you know, embedding can be arbitrarily big, and so it like, of course, in the limit, it would just destroy keywords. And so you have this like arbitrarily powerful representation. And now the fundamental idea was just like the bitter lesson, what if we could like, you know, train transformers to output embeddings for documents and if we keep getting more and more data that's high quality, we could make a search engine that actually understands you. And the way it would work at inference, at search time is like a search comes in, a query comes in like search without stripes. Traditional search engines would use the above thing where they would do a very fancy keyword comparison. And a bunch of other things. And then instead we would just embed the shirts without stripes and compare it to the embeddings of all the trillion documents. And, you know, after a year and a half we actually had a new search engine that worked in a very different way. And you search shirts without stripes on Google— I'm sorry, on XA. And you get a list of results that actually are not— do not have stripes. It's a simple example, but like you could— it could handle like way more complex queries, like paragraph-long queries. And when we launched this in November 2022, we got a lot of excitement on Twitter. This is a very new paradigm for search. You could do all sorts of interesting queries that you couldn't do before. And then 2 weeks later, this happened. It was a small tweet. And this is a visual depiction of San Francisco at the time. You guys probably all remember this. And then this is a visual depiction of the exit team at the time. Because ChatGPT completely changed the way we interact with the world's information. You know, like everyone can now use an LLM to just like talk, talk to their computer and get information. And we were thinking, wait, is there even a role for search in this world? Like these LLMs are so powerful. And then very quickly we realized, yes, there is a role because LLMs don't know everything on the web. So for example, if you ask an LLM like GPT-4, find me cool personal sites of engineers in San Francisco. It can't, like it just doesn't have that in the weights, it'll apologize, whatever. And you know, there's a very simple information theory argument here where it's like there literally isn't enough information in the weights of GPT-4 to store the whole web. GPT-4, call, like we don't know exactly how many parameters, I think someone leaked it on YouTube once, but it's like, oh, you know, a couple trillion parameters. You could call it like less than 10 terabytes in the weights of GPT-4. And then the internet is like over a million terabytes. And that's just the documents on the web. There's also images and video and that's way more. Actually, the web, if you look, I did a tweet recently about the size of the web and it's in the exabyte range. And our name is Exa, it's not a coincidence. Anyway, so like LLMs need to search the web just from this simple argument. And they're going to need to do that for a long time, which if you talk to ML researchers, they'll say the same thing. It's just like, it's too hard. Also, the web is constantly updating. That's another problem. It's not just the size of the web, it's the constant updatingness of the web. That makes it very tricky. So LLMs always will need search. That's great. And so when you combine an LLM with a search engine like Exa, you can handle these queries. So like, find me cool personal sites and engineers in SF. The LLM will search Exa, get a list of personal sites, and then like use that information to output the perfect thing for the user. You're all very familiar with this, like LLMs plus search. It's obvious now, right? Like everyone knows about it. But now let me tell you a secret about search that Most people don't know. And the secret is that traditional search engines were not built for this world of AI. Traditional search engines were built for humans. And humans are not— are very different from AI. So every search engine like Google, Bing, you name it, was built in a different era for this kind of creature. This slow flesh human that's typing keywords and wants to read a few links. And really cares about UI of the page and all these things. Like, it's a lazy human, they type simple keywords. Google is great for this creature. Google was optimized for this creature. It gives you exactly the kinds of things you would click on. But AIs are very different. This— like, an AI can gobble up information like crazy. This is a much slowed-down version of what our AIs probably feel like inside. And so AIs are very different. They want to use complex queries, not simple ones. To find not a couple links, but just tons of knowledge, as much knowledge as they could get, because they actually have the patience to just analyze it all extremely fast. And so the search algorithm that's optimal for this type of creature is not the same algorithm that's optimal for the human. Like, that would be crazy if the same algorithm that was optimal for humans was optimal for AIs. And so like all the— a lot of the tools, the search tools that we're talking about these days on Twitter and stuff like that, They're still using like the old traditional search combined with AIs. It's just not the right puzzle fit. So at XO, we're really trying to think of like what is the right search engine for this AI world. And so just a few examples we could dive deep into of how AIs are different. Well, AIs want precise, controllable information. So, oh, by the way, when I say AI, I'm usually, I'm talking about like an AI product. So imagine like in this case, like a VC that's using an AI system to find a list of companies because they want to invest. So, you know, they're looking for something. What's the next big thing? What's the next big thing that feels like Bell Labs? Well, when they tell their AI what they want, the AI will then go search a search engine, right? And if it searches a search engine like Google, it'll get a list of results that humans like to click on, but it's not very information dense and it doesn't even match what the person asked for.

The—

sorry, what the AI asked for. The AI asked for startups working on something huge that feels like Bell Labs. It should get a list of startups. It's kind of a crazy idea, but what if search engines actually returned exactly what you asked of them and not what you want to— what Google knows you'll click on? And so with AI especially, they just want a search engine that returns exactly what they asked for. Because what's— what really the world's going to look like is you're going to interact with your AI agent and you're going to ask for something, and then it's going to make tons of searches like, okay, maybe they want startups working on something like similar to Bell Labs. Maybe they want startups working only in New York City that have this quality and that quality, and they'll do all sorts of searches, and it just wants a search API that just It does what it asks. And so you need a search engine like that. So X is like that. Another difference between AIs and humans is AIs want to search with lots of context. Again, if you have an AI assistant and you talk to it all day and then you ask for restaurants or apartments or what have you, the AI has lots of context on you. So it should be able to search with this large multi-paragraph thing saying like, you know, my human is a software engineer and it likes these types of things and I like these types of things and like, Can you give me, you know, restaurants that match those preferences? And so you need a search engine that could literally handle multiple paragraphs of text. But traditional search, like search engines like Google, were not meant to do that because humans would never type in multiple paragraphs because they're too lazy. So Google was optimized for like simple keyword queries. So Google I think has like a few dozen keyword limit. Whereas Eksa can handle like multiple paragraphs of text. Another big one where AIs are different than humans is AIs want comprehensive knowledge. Like, if you give a human 10,000 links or 10,000 pages, it doesn't know what to do with that. Like, it would take 10 days of extreme patience to process all that. But AIs can do it in 3 seconds if it's parallelized, right? So if I'm a VC and I want to report on, like, all the companies in a space, I want literally all the companies. And there's a huge amount of value to getting truly all of them and not just, like, the 10 or 20 that Google is able to find. And so you need a search engine that exposes the ability to return 1,000, 10,000, whatever it is, and also has this semantic ability to like, you know, when you say like every startup funded by YC working on AI, you actually can get all of them. So like Google literally just can't do this at all. Okay, I hope that through these examples we see that the space of possible queries is actually like way larger than people realize. And until like 2022, we were kind of in this like top left blue world. So this circle is like the space of possible queries. And the blues are like, you know, specific subsets of that space. And so like we were all in that top left corner of blue for a long time where you could, you know, we could search and it could handle like, like basic keyword queries like Stripe pricing or someone's GitHub page or Taylor Swift's boyfriend or whatever it is. After 2022, everyone started to want the top right blue circle where it was like, hey, actually, I want to make queries like explain this concept to me like I'm a 5-year-old or here's my code, can you like debug it? This is a form of query. It doesn't require search, but it's another type of query that was introduced to the world in 2022. And then like, there's other types of queries like these semantic queries, like people in San Francisco who know Assembly. As far as I'm aware, Exa is like, I mean, Exa kind of like introduced this kind of query and does really well on those queries. And then there's these really complex queries like, find me every article that argues X and not Y from an author like Z. And we're starting to now have systems like X's like Websense product that can handle these things. And I think this is actually a huge space because this like turns the web into like a database you can filter however you want. And that's really what AIs want. They want this like full control database-like query system that they can just get whatever they need for their user. And then there are the queries that no one has thought of yet. Like every week we get tons of queries and like, "Oh wait, that's a really interesting type of query that no search engine could do right now." And eventually we'll try to you know, handle all the queries that are possible. But there's so many new types of queries now because we have these AI systems and the stakes, like the expectations have just gotten way higher. Okay, so now we end our story with the same slide. One API to get any information from the web. So again, like EXA is trying to, if you go back, like handle not just like the keyword queries but also the semantic queries and also the super complex queries and eventually all queries. We want one API that could like give these AI systems whatever knowledge they want. You have the AI and you have Exa providing the knowledge. Oh, I only have 4 minutes. Okay.

Okay. So that's, let's see.

Oh, how do I go to a different part of my computer? Hmm. If I change to the code editor, how do I do that?

Let's see.

What? Oh, it's there.

Oh, but I can't see it. That's so weird.

You have it down here. Oh, cool. Okay. Okay. Are you using mirror? There we go.

Okay.

Cool. Well, first of all, just very quick explanation of— this is our search dashboard. We could try different queries. I would just point out, like, in the search API endpoint, you know, we exposed lots of different toggles. So first of all, you just try out a query and get— it shows you the code and it gets you a list of results. And it exposes tons of different types of filters that you might want to do. For example, like, number of results, 10, 100, 1,000, whatever it is. You could have, like, date ranges or, you know, I only want to search over these domains. And it's a lot of toggles, but I think the point is actually you want the toggles because your AI is actually going to be calling this. You want a search engine that gives you full control. And we have like neural and keyword search, so you could try different ones. Um, okay, let me quickly jump to the code. Okay, so I prepared this like code, uh, agent.py. So we made this agent, uh, agent Mark, and Mark loves to make Markdown out of things. Anything you give it, it will make Markdown— Mark will make Markdown. Uh, and so in this case, uh, we're going to—

here.

Well, I guess in this case— Let's try this query: personal site of engineer in San Francisco who likes information retrieval. Well, this is the kind of query that Neural would be a lot better at.

Whoa.

Okay.

Save it. Oh, running the wrong agent.

Okay, so it's just— it's making a query to get, like, a list of personal sites of engineers in San Francisco who like information retrieval, and Mark, the agent, is just making a Markdown output of that. That's a very neural-type query. You also might want to do a different type of query, which is, like, a more keyword-heavy one.

Let's see, like, um— Will Bryk GitHub let me add—

My GitHub. So, OK. So here I would want to make a keyword query. So you just change the keyword. Search. So it's going to get information from my GitHub using keyword search. Because this is a very typical, like, Google-like search that would work well, right? Oh god.

I'm running this wrong one again.

OK. Cool. That's information about William Bryk's GitHub. Um, and then, okay, so when you're actually building an agent, you're going to be combining lots of different types of searches. So neural searches and keyword searches, uh, and all sorts of other searches that X exposes. So like the right agent in the future is going to be this system that decides what type of search it needs, uh, for, uh, whatever the user says. Like it'll be like, oh, okay, I'm going to make like a neural search to get a list of things, and then for each one I'm going to do a keyword search, right? You want to give the agent like just full access to the world's information in however way it wants. Not just keyword search, but also all these other things. And so here I one-shotted with O3 a GitHub agent which combines these two queries. So first it'll— 'cause I want to, you know, I want to get the GitHub of every engineer in San Francisco who likes information retrieval. So the agent will make a neural search to get a list of people, extract the names, and then search those using a keyword search to get their GitHubs. And then if you run that, here it's just getting 10 results, but we could, you know, with Exa we could do 100 or 1,000 if you're on an enterprise plan. So now it's getting all the GitHub info. Cool. So that's just an example. And yeah, I mean, there are lots of other things that you could do with Exa. Like, we actually just today launched this research endpoint where it will actually do like as much searches and LLM calls in the background to get you that perfect report or that perfect structured output for the thing you asked for. So it's kind of like a deep research API and it's state-of-the-art deep research API. Cool. That is the talk. I hope that was interesting. Thank you.