Building Brave: Private Search, One AI Layer at a Time with Josep M. Pujol

Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week we'll explore their revolutionary ideas, and together we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Thornberg. Hello and welcome back to The Cognitive Revolution. Today, I'm excited to share my conversation with Joseph Pujol, Chief of Search at Brave. As regular listeners will know, Brave has been a sponsor of The Cognitive Revolution for the last several months, and that makes this our first sponsored episode. I think it's important to mention that right at the top for the sake of transparency, though I did approach this episode in exactly the same way that I always do, with a deep dive into the company's products and an earnest search for insights that will be worth your precious time. And I can sincerely say that I believe this episode maintains the quality standard that we always strive for. As you'll hear, Josep was very open and shared a number of fascinating details about how Brave has built their search product and infrastructure over time, including the trade-offs that they accept to maintain the highest standard for user data privacy. Some of the most interesting details center around the challenges of iteratively improving an AI-powered product while operating at real scale. Brave handles over 1 million searches per hour, or roughly 300 searches per second. They do not want to take any steps backward, and so their approach is generally to add to their system, making their ensemble of models and techniques more effective, but also more complicated with each generation. I found this approach really fascinating to learn about, as it's a perspective not often found in younger startups. I also appreciated Josep's candid perspective on the challenges they face in evaluating search results. Brave has developed some scalable systems to assist in measuring quality, but Giuseppe emphasized that there still is no fully automated substitute for hands-on human evaluation, good taste, and judgment. I think this too is an important lesson for all AI builders to keep in mind. Coming away from this conversation, I genuinely believe that the Brave Search API is an excellent product. I've spent more time testing it since the recording, and for the Waymark Small Business Profile Builder, which today relies primarily on scraping small business websites, I did find it to be notably better than both Bing and Google. As such, I have recommended it to the Waymark product team. My expectation is that, like Brave, we will augment rather than replace our existing system. We'll use the Brave Search API to help us return results faster, to cover URLs that our scraper fails on, and to surface valuable bonus context from discussion and review sites that we simply aren't tapping into today at all. Overall, I think it will be a really nice product win, and given the low cost and latency, I think it's something that more AI builders should consider using. If you'd like to check it out for yourself, I've created a simple Google Colab notebook which makes Search API calls to Brave, Google, and Bing. It's nothing fancy, but this is exactly what I've used to evaluate it for Waymark. We'll put the link to that notebook in the show notes. Finally, before we get started, if your company is interested in sponsoring The Cognitive Revolution and potentially being featured in an episode, I invite you to reach out. We'll always be fully transparent about sponsorships, and we will only produce sponsored episodes with companies that I personally judge to be worthwhile on their merits. So my hope is that this will be a win-win for the show, for the sponsor, and most importantly, for you, the audience. As always, we hope you'll share the show with friends and we invite your feedback. And with that, I hope you enjoy this inside look at Brave Search and the Brave Search API with Brave's Chief of Search, Josep Pujol.

Josep Pujol, Chief of Search at Brave, welcome to the Cognitive Revolution.

Nice being here and thank you for My pleasure.

I'm excited to get deep into all the exciting developments and AI technologies that you have assembled into the Brave search experience. I guess just as a quick prelude, regular listeners to the show will know that Brave is a sponsor of The Cognitive Revolution. So we thank you for your patronage in supporting us. I've obviously listened to the ad copy a few times, and I know the audience members have as well. Do you want to start off by just giving us a little bit of a general background on Brave, now my default mobile browser, by the way? Just tell us a little bit about the company at a super high level, what the kind of motivations are, the ethos of the company, and then we'll dive after that more deeply into the technology.

Yeah, Brave, first of all, is a great company founded by Brian Bondi and Brendan Eich. And the aim, they started as a browser, especially with a focus on privacy.

And myself, I joined Brave like 3 years and a half ago out of a company that was called Cliqz, or a Tailcat spinoff. And we were basically, we were developing a search engine from scratch. And when we, like, when our previous company went belly up because of lack of funding, COVID, and whatnot, well, we had multiple potential places to go. And the one that was more appealing was Brave because we share like the same values. privacy-preserving, and also like the aim to be an alternative to big tech, right? It's not so much that what we try to achieve is that to do something totally different, right? Because at the end of the day, it's a browser, it's a search engine, it's a talk software, like video conferencing. But the point, the whole point is that, well, you can actually like have the full technology suite. Right? Without having to rely on big tech with all the compromises that thing entails. So that's very appealing. It's very appealing to me personally and to the team. That's why we joined Brave. And I think that's the best definition there is. Of course, there is a bunch of features that are pretty unique, but to me, the biggest value proposition of Brave is that why use Google when you can use Brave?

I've been impressed actually by just how big the The product line is, when I started to get into it more, the mobile browser experience is really good. And the farther I got, I was like, oh, there is a VPN that you can enable here. And there's a chatbot language model powered experience, question answering, all that stuff. It really is a pretty— and an ad network as well. There's a lot going on. How many people at the company? It's not that big of a company. I'm impressed by how many different products.

But people Brave as a whole and people working on search, like engineering and quality raters, like the core of search is about 20.

20. Wow.

On search, right? There is a lot of—

not just that search is run by 20 people, right? There is marketing, communications, HR.

There is a lot of shared resources, right?

But anyway, like the search is like 20 people Brave as a whole. So it's a very small company. And yes, we do have a lot of reach, but anyway, that's the proof that you can, I mean, you don't have to be big to provide value.

Yeah, it's impressive. We'll get into some of the head-to-head comparisons as we get a little bit deeper into this. So as head of search, we're going to spend most of our time doing a deep dive into the search technology, how it works. As I've said, our audience is very technical. interested in technical details. Many people that are listening are building stuff and they are looking for the hard-won lesson. They were looking for shortcuts to the lessons that you've learned the hard way. So maybe just for kind of starters, like, you want to give kind of a general overview of how search works? One thing that definitely stood out to me again from the ad copy is that the data is at least to some degree sourced based on pages that people are actually visiting. And that was, for starters, an interesting data point.

Yes, actually that project, the use of browsing data, if you want, that's not a start from Brave, it starts from the previous company. That's how we actually gather enough data to start developing the search engine.

But it's a very controversial topic because there is certain tendency to to have this dichotomy, either data or privacy. And this dichotomy is actually false, right?

It's as simple as that.

I wrote extensively about it. People can check it out. But basically the whole point is not so much that data is a problem, it's like how you collect that data, right? I'm not talking about opt-in, opt-outs, or like a legal issue, right? Those aside. It's more like what kind of purpose the data will have and how the data is collected, which makes it dangerous or not. For instance, like Brave, if you— people who want to contribute, they can opt in to something that's called the Web Discovery Project, right? And in there, they will send us browsing data, right? That sounds dangerous. Browsing data, what do you mean, right? It's not that we collect your browsing history or your search history.

No, right?

That would be dangerous. That would be like privacy problematic. What you will contribute is like individual data elements that are anonymous, right? So someone has visited this site and has engaged so much with it, right? Then what any of those data elements that people send us, we have absolutely no way to know whether they come from the same person or not. So we do not learn. anything about you or about me, right? What we learn is that someone is anonymous. There's like a lot of technical details here. It's not that just—

we do not use user IDs, neither implicit or explicit. It's just like the data is like basically sent using a mixing network to remove like network fingerprinting.

The messages are homogenized so that they actually have to pass a quorum. So more than X people need to send the same amount of data.

the same data in order to be actually received by us, that's like reproducible to have.

So there's like a lot of like technical merit, but at the end of the day, what is important is that the data that we receive is anonymous. It's not anonymized, right? Or it's pseudonymous. No, those are dangerous, right? Those are the things that they can later be anonymized. The data that we've got is just anonymous.

Anonymous and it serves a single purpose, right?

In this case, to know that this particular page is popular or not, right?

Or that this particular page was clicked after a particular query to build query logs. So that's the kind of data that we collect. And without that would be actually very difficult to build a search engine.

Google and Microsoft do the same thing, whether they admit it or not.

Actually, there is like a nice release of some documents that happened yesterday or like a couple of days ago where there is like some private APIs at Google. That's natural. This data is collected. And again, the problem is not the data. The same way that like the problem is not the advertisement, right?

The problem is like if by doing advertisement you actually put your privacy at risk because this tracking is dangerous, right? Knowing your browsing history or your search history is extremely problematic.

And that's something that, for example, we do not want to know and we do not collect.

One of the attack vectors that we consider is that if we actually had to release our data either to a hacker or to a government agency of some sort, we could release it knowingly that no particular person could be identified or profiled.

Yeah, that's really interesting. So if you're in a country where you have reason to be concerned about this sort of stuff, then this becomes an extremely valuable option.

Yeah. First, it's an opt-in.

Even though it's privacy-preserving by design, still we do it on opt-in because it's something that is like people have to be aware of, but they could enable it. We would not be able to—

it's something that's basically Technically, it's called record unlinkability.

So any data element that we receive, we have no way to know if those 2 elements come from the same person or not. That means that we only have individual data elements because even if the individual data elements, if we had a way to link them, we could actually create a profile.

And based on that profile, it can always be deanonymized because there will always be one of the data elements that somehow probabilistically or deterministically can link to you. And then basically the whole session is compromised. The whole point is that we have no technical means to actually build a session so that, you know, like there is like, yes, somebody visited this page, but we couldn't know what happened afterwards. But of course that data is much less useful than profile data.

Right.

Because profile data, it's not that profile data is evil Let's put names on it just for the fun of it, right? It's not that the Google engineer is evil and says, I want to collect everything on others. But basically what you want is that once you collect data, you want it to be reusable. You want the same data that you collect to be able to count how popular the page is, but also like how pages relate to each other or like how popular a particular restaurant is at what time. And you want the same dataset to be able to answer all those questions.

And that's very powerful, but that's very problematic, right? Because it can answer all these legitimate questions, but it can also answer illegitimate ones.

Like, give me the search history of whoever was here at this particular time. So what we cannot do is answer these generic questions, right?

So for every question that we have, we collect one particular data element that only answers this particular question.

So that's the difference.

Like, the how is important here.

It's not so much about data or no data.

is like the purpose of the data.

So how does this compare, or maybe it's, maybe you combine, I guess you, you have a crawler too, right? So there's, yeah, it's not that the index is entirely built from the page visits, but more that it is informed by the page visits.

Yeah. Like to give specifics on that, for example, like we may learn about a URL through web discovery project, right?

And we know this URL is popular, right? Because it has to pass a portal. So more than N people have to see that URL in order for us to receive that URL.

So that means that URL does not belong to you.

It's not your login URL, right? Because like, you know, 100 people across the world that access that URL. So that when we receive that URL, we only receive the URL, right? We don't receive the content because the content, there is like no guarantees that the content does not contain private information that belongs to you.

So then we receive the URL, which is public information, and then we actually go and fetch it. So that allows us to basically, that our index to be like small enough, but still, because it's small, because we do not crawl blindly.

We don't do like a blind crawl of everything.

What we do is like we crawl what people basically visit.

And from there and from trusted sites, we actually do some additional crawling.

But most of the crawling we do is not crawling, it's called fetching, right?

Because we actually fetch the content.

It's not that we proactively crawl the whole web because the whole web is full of noise, right? There is like at least 10 different sites that are clones of GitHub. As odd as it sounds, it is. There are like sites that are clones of GitHub, right?

So if you Blindly, you just like increase by 10 your size, but only by adding noise.

Anyway, that's, so that's like one of the, one of the things that, that we use people for, to become more efficient and to reduce the noise, which is like the ultimate goal of any machine learning or AI, right?

To reduce.

Yeah, I was just gonna say the cleaning the dataset is often where the majority of the work Absolutely.

That's like the main realization that noise reduction is the single most important thing that you have to do ever. Because noise does not only affect the quality of the output of your models, but it also affects the scalability of those models and also the costs of running those models.

Noise needs to be avoided as much as possible.

So do you have a sense for the relative size of the index compared to what either Google is doing or the whole web?

It's a 10th of what Google has.

It's a 10th of what Google has and we can achieve like 99.9% of the recall that Google has. Yeah.

Interesting. Yeah. Power laws in action. So, a little bit about the signals that you're looking at. looking at beyond the behavioral? Obviously we've got from traditional search projects, we've got like keyword-based indexing, PageRank. I assume you're still getting some value from like that sort of link hierarchy.

No, no, no.

Rank.

No, no. How polemic can I be?

As polemic as you want.

PageRank is a myth. You know that.

Yeah. Let me tell you what it means. Think about what I know and then you can correct me. I thought it worked well at the beginning, and then it became subject to a lot of adversarial attacks with SEO of a million kinds. And now I would still assume that it has some power, but that it has had to be updated a lot of times in a lot of ways to counteract all these SEO optimizations.

Okay.

I already was an adult at the university. I almost started to do a PhD when Google came out, right?

I started using Google when it was not even google.com, it was stanford.edu.

They released 2 papers.

One was the PageRank, which was like a proper scientific paper, and the other was like a technical report, like the anatomy of a large search engine, where it had the meat of why Google was so good.

Google was not good because of PageRank. PageRank was a gimmick. That was a gimmick that worked, of course, had a contribution, but was good because it was an algorithm and it had a paper on it that was published. And it was something that journalists love to stress, right? The algorithm that does— that changes everything, right? That's what people want to hear. Like, people want to hear a story that there's like an invention that does— that there's like a straight line, there's like something It's not like something happened and we went from here to here.

That's not what happened. In reality, things are like a little bit more— the reason why Google was so good from the get-go is because of anchor text. Larry and Sergey through Stanford were the first ones who, using Stanford resources, were able to crawl the whole web, crawl the whole web brute force. And that's why Perhaps you will see like there are some pictures having like hard drives on Lego structures, right? Because they had to buy so many hard drives to be able to store the whole web that they did not have like iron to put it.

So they actually had to create like Lego stands for it, right? So their project was like that where they were able to like crawl the whole web.

What does it mean to crawl the whole web?

That suddenly unlike AltaVista, Right? You had a much better representation of what the page is, right?

Because you have the backlinks.

And the backlinks, what does it say? It basically has the anchor text. It's basically like everybody else's description of your site, as opposed to what we had before, which was like your own description of what your site does. Remember, noise is everything. Noise reduction is everything. What is less noisy? The content of the web page or the summary that 1,000 different people do about your page? It's much less noisy, the anchor text. And that's why they were able to actually have much better recall with much better precision. And then, once you have the anchor text, you have the backlinks.

And the backlinks have adjacency matrix.

And with the adjacency matrix, you can calculate the first eigenvector.

tool, which is like the stationary state of random walk, yada, yada, yada, very nice paper, out. But that's like cherry on top of the cake.

The real meat was their ability to brute force via something that was not possible a couple of years before, to actually brute force and crawl the whole web in a single file system. And then the realization, very smart, hey, we have a match cleaner representation of what the page is than the page content, right? That's what made Google.

That's fascinating. Hey, we'll continue our interview in a moment after a word from our sponsors. How does that play into the Brave search today? My sense is still that it still matters if a reputable site links to you versus an unreputable site. It seems like there's some sort of authority notion.

There is authority, right?

But it's not done on PageRank. PageRank, we can actually derive it out of the authority of the pages based on popularity.

Just on traffic.

Right.

Just on traffic.

That's enough. Then yeah, you might do one or two levels deep of calculations, but in any way, that's again, that's a very small contribution to the end goal.

What the takeaway message here is not so much the history. It's a little bit more like that. Very often, like, the real innovation happens on something that is not as fashionable as an algorithm. It's more like on the methodology, right? And actually that happens on Brave Search too, right? Brave Search is as good as it can get. It's because, for example, we rely very heavily on query logs, right? With a query log is in a way is like a, is a representation of what the page is about. That is not done on the anchor text, but on the queries of another person. In a way, like Brave Search started being a recommender system engine. And that allows us to like, to keep building query logs, which aren't publicly seen. And as you grow, you have more query logs, and then you can start to do like semantic relationships with your queries.

You have never seen like, What is the age of Lady Gaga, right? But you have seen Lady Gaga's age. And then you say, like, okay, then if you've seen the other one and you know that page is good for this one, it's going to be good for the other query because semantically it's very, it's very— so you keep expanding the query semantically to do semantic search on queries, not on content, but on queries, right? And then you start to be able to use, like, learning models to generate synthetic queries. Not to try to index the content as a whole, but to create what would be queries that would be answers to this page, right?

And then you do this and then you add it and then you start, you become a little bit better. Then you start to index content of the page.

So you start to use n-grams, right?

Which is more conventional approach and more expensive than the other ones. So like the cost to benefit ratio is worse, but now It makes sense, right? Because it's the next step that you have to take. So then you keep adding and what you end up is having something that actually works. Does it make a nice history? No, because the story is boring. It's like the story is, no, you just keep building and keep iterating until there's a— you keep doing 1% increments until you have something that works. But that's like the only way that things typically work again.

Yeah.

And that's why I put so much emphasis on the example of the PageRank, because that's like the story. And, you know, like every step you keep adding and every step there are things that happen that were not possible before. For instance, back before Brave, like we, like this, the same concept that we started was not possible 5 years before we started.

Right.

And why not? Because they were like, not because doing a recommender system means that you have to have everything on. It's a big cache in a way, a big memory map. So at the time where like the servers had 16 gigs of RAM max, to host something that has 2 terabytes, it takes a lot of machines. A lot of machines means like a lot of latency. It's not feasible. However, at that time at Clix, Amazon came. Amazon released a machine that had 1 terabyte.

of RAM.

Suddenly, it was possible to create a cluster of 4 machines to actually put everything on memory and everything became very easy, right? Where before, one year before that, it would have required like 2,000 smaller servers, unfeasible from the engineering perspective. And those kind of things happen all the time, right? Another big hardware improvement that without it, Raysearch wouldn't exist. NVMe hard drives. Without NVMe hard drives, Raysearch wouldn't exist. It would not be cost effective. NVMe, I guess that you know what it is, right? It's like hard drives that are super low latency on lookups. Much faster.

Is that the solid state disk as opposed to a spinning disk, or is it even a more fine distinction than that?

No, it's actually faster. That's how it is. There is spinning disks, impossible to do what we do. Flash, still not fast enough, but NVMe is like an order of magnitude slower than RAM.

Gotcha. Interesting.

So suddenly you have what, 100 terabytes, a petabyte of NVMe. Now you can actually do what you want to do. Before it was even the, basically like the way of doing it would have to be done differently. That would actually increase costs by an order of magnitude, increase everything by an order of magnitude, and it's not doable. So again, that's the thing.

At least that's how Brave Search was built.

It's incremental, always taking advantage of what the world had to offer in a way.

So that history of the hardware enablement is really interesting. Can we do the same, an equivalent history for the progression of language models or semantically enabling models? Because going back to early days, you had literally just keywords and then n-grams, and then now obviously we've got pretty amazing language models. But I would imagine that the best language models are maybe an overkill for some of the use cases, if only because of cost and latency. So what's that history like, and where are you guys today in terms of the models that you use for semantic purposes?

Yeah. Remember the example I told you before, like Lady Gaga age versus whole world is Lady Gaga?

That's a semantic matching, right? We started to use embeddings, semantic embeddings based on a model that was called StarSpace. I don't know if it sounds familiar.

I don't know it specifically.

2019, right?

So a long time ago.

And basically this relationship of being able to go from Lady Gaga's age to how old is Lady Gaga. It was done via embeddings, 200 dimensions, 2 bytes, vector search. Back at that time, it was not called vector search. It was called nearest neighbors approximations. And that's what actually we built.

So we actually took advantage of these recent technologies. Again, there was no only semantic embeddings. Actually, there was also like N-GAMS based on queries. Remember, like we were basically very heavily based on queries.

Just to give you a number, a nice number, our query similarity system has more than 9 billion unique queries. So like the query Hotmail is only one.

Yeah.

So 9 billion. So whenever you actually get a new query, we're able able to return a set of similar queries that we have seen in the past out of these 9 billion in 20 milliseconds, right?

And that is done both using embeddings, but also using n-grams based on queries, right? Because we need both. You cannot be semantic 100%, right? And that was true back then and it's true now.

Why not?

What's the limitation there? Or where does the purely semantic approach fall short?

Because purely semantic is purely semantic. The queries that people do are not— doesn't have a semantic meaning.

It's like a particular model of a particular product, period.

And that's what they want.

Or like an error on a stack trace or like a name of someone. My name is Josep Pujol, right?

It's not Josep Pujol. Semantically, it's very similar. Spanish and Catalan will give you a very close distance in any semantic model, but it's not the same. You need both. The takeaway message, if there is any takeaway message that I could give people, is that there is like no single bullet, that there is no single anything that solves a problem that is worth solving, right?

It's always like a combination of multiple techniques and models, and then you do an ensemble on top of it.

So can you do only embeddings, semantic embeddings?

No, you need to have both. So we do have everything on the ground. So for example, on page content, right?

So we have page content. We do have embeddings based on transformers, BERT-like transformers.

Yes, we have those, right?

But at the same time, those are like what we call dense embeddings, right?

Because the dimensionality that we have is 384 dimensions with 2 bytes per dimension, right? And each page have about 1 to 5 embeddings, right, of those using different parts of the content, etc.

But that's like this on semantic matching, but we still have what we call sparse embeddings, which is 16,000 dimensions, 1 bit per dimension, which are for literal matching. And when we search, we use both, right? And it's good to have both approaches because you want to Sometimes when you search for, I don't know, Trump Tower, you just want Trump Tower, right? You don't want Trump something else. A lot of noise can come out of something else, right?

So you want to combine both approaches and that's what we have.

So this mentality, I guess it is shared by many people, including any person who comes to work for us out of university that, oh yeah, I just like doing embeddings. Out of OpenAI, put them on a vector database and do a search and that's it. That doesn't work. It only works for very narrow use cases or like for vertical searches when there is like one, only one domain, but it will not work on a general purpose because it will bring up noise. Statistically, you can think of it the following way. The embedding is not free of errors, right? So let's say that once every 10,000, the embedding has an issue, right? Doesn't provide the right embedding for some, for whatever reason, right? If you are working on a database of 100,000 elements, that will not affect you. You will see that it's working fantastically well, right? But if you have 20 billion, the chances that those errors correlate to your initial recall set are very large, right?

And then you're blind, right? Because if you have a recall set of 2,000 documents and 10% of them are noise, what else do I need?

I need other things. I need other techniques. I need other models to be able to take out the noise, right? And that's always there. So, there's no single thing that works.

It's always like a combination.

Yeah, that's probably a really good takeaway for a lot of listeners who are building all manner of RAG apps in all kinds of different contexts. I've got a project that I'm involved with that's trying to take old plumbing catalog parts manuals and make that searchable. This is for a company that has, like, a proprietary database of all these old manuals. They literally have them in paper form in a giant bookshelf and they've embarked on this.

But this actually, I can tell you another story that might actually work because in a way, the level of noise that you have on vertical search is artificially reduced. I don't know if it makes sense.

Yeah.

Because the problem is always the noise, right?

The uncertainty.

So when you're doing a vertical something, vertical search, or you're working on a particular vertical, noise is artificially reduced, right? Because in a particular domain, a particular word has only one particular meaning, or a particular token has only one particular meaning, right? So when you work only on a vertical, it's very easy to work on. And actually, one of the things that when we first started, When we started to tackle problem of search, the initial thing to try was like, hey, but search can be split into verticals, right? So if we are like 20 people, why don't we just do vertical?

You do like weather search, I do city search, you do famous people search, car search. You come up with like 100 different verticals. So what ended up happening is that all the 100 verticals work perfectly, right? there was like no way to put them together in a way that made sense, right?

Because what we actually did unintentionally is like we pushed the problem, the real problem, we pushed it under the rug, right? On the module that, the mixing module that will come in the future.

Mm-hmm.

And the mixing module is a problem, right?

Because on the movie vertical, Ted has only one meaning, right?

The movie Ted could be Ted 1, Ted 2, Ted 3, right? But now forget about that you are on a movie vertical.

Think of the whole web. But TED, what does it mean?

The movie, name of someone, the conference, a part, right? Which one is it?

That's why search is complicated because it's general purpose. And AI has the same issue, right?

The narrower the domain, the easier it is because there is less noise.

Does that assume though that you are, Working with enough data. Going back to this plumbing example, and I think trying to generalise from that to what I think a lot of people who've listened to this show are working on, I think in a lot of scenarios there's like a proprietary dataset that is too big for people to page through, right? So they do need some search to access it effectively. But then at the same time, it's not so much data that they could train a model from scratch on it. Maybe they could fine tune. And so I feel like a lot of people find themselves in that in-between zone where they're like, I need to use some sort of pretrained open source model, at least as a base. And those models, like in this plumbing scenario, everything is in a very narrow section of the broad general purpose embeddings models. And then things like parts Parts numbers, as you said, are not even really semantic at all in a lot of cases. So, we are finding that we do need a hybrid search, if only to handle these things like parts models.

That's correct.

And actually, that's why, in a way, it links back to the point, right?

And then you're like on this particular use case, and the first temptation of, if you want to be like pure AI, let's say, is now I need to fine-tune, right?

Really?

The user can use like the generic model off the shelf and then have another system that does exact matching and then you combine it. So that is the right approach. Perhaps you need to do fine-tuning, right? It depends on the use case. But at the end of the day, what is important on engineering is not so much the tool. The important thing is the problem.

And you just use the tool that is most convenient and you combine different tools. That's the most important thing, right? Do not fall in love with a particular framework or a particular methodology, right?

What you have to fall in love with is the problem.

Yeah, totally. With that in mind, how do you guys approach upgrading technology? I think this is another thing that I haven't— I'm only starting to get to the point where I'm encountering this in live apps. I have a company called Waymark, which does video generation. And there we've been through several rounds of upgrade of generative models. I haven't got to this point yet on a search or a RAG-type product where, oh, look, there's a new embedding or there's a new vector database that could support that embedding. I'm sure you've been through several rounds of this stuff over time. How do you approach that challenge and try to tame it?

It depends, right? For instance, on the summarizer, right?

We have this AI summarizer that people can check where we try to provide an answer to a query. It figures like about like for 25%, 30% of the queries. Those basically, that's of course, there is like, we do RAG against ourselves as anybody would.

We do put a lot of emphasis, we can discuss later on that, on the data selection, which snippets we pick and we run.

Narrower QA models on the snippets to like to fit, but ultimately we do have a prompt that we send to a large language model, right? This large language model has changed many times, right?

We had Summarizer V1 that was created one year ago. I think that we started with Plantify and then we upgraded to another model after a while, then we did some fine-tuned version of it.

So we did 3 changes in the course of like few months. But in Semantic AI V2, it was even more extreme because when we released, the final model was Mixtral 7B, 7-8 billion.

Llama 3 came out 2 weeks after. We tested, we saw that it's actually— we had tests, tests were like comparable quality using much less resources.

We switched it. It took us a week to switch.

So on those components, it's very easy to switch because you have a test set and you basically change one component for another one.

But that doesn't mean that it's generic because, for instance, embeddings, can we change the embeddings? No, not at all.

The embeddings is like forever.

We have embeddings that the starter space embeddings were trained in 2019 and they're still in place and they will never be replaced.

Wow.

What we can do is to actually have more embeddings.

Right.

And we can have the old system and the new system using new embeddings and yet another system using new embeddings and then do an ensemble of all. But something that's fundamental of the embedding, you cannot change it. First of all, because the embedding depends on the data and the data is massive, right? To give a concrete number, for example, the content embeddings, the dense vectors, which are based on Transformers BERT. It takes us, we can do about like 500 pages per second. It means that we pass a page and we get the embedding in 2 milliseconds. And if you work with embeddings or with transformers, you know that actually that's pretty impressive, right?

Yeah, that's fast.

That's fast.

So in 2 milliseconds, like fully optimized single GPU, 810G, I guess, That takes 1 year and a half to do it for the whole dataset.

So even being extremely fast is linked to the data.

So you cannot redo it, right? You do it once and then you build narrower models that are based on those embeddings, on the output of those embeddings, and you build heuristics that combine your part numbers of your plumbing things.

You have narrower models plus heuristics. You cannot change that, right? What you end up doing is you create like versions of it. You have embeddings V1, embeddings V2, embeddings V3, and you keep adding.

And adding is never a problem, right?

Because at the end of the day, adding more embeddings is adding more features, adding more distances for the final GBDT that will do the final ranking. So anything that is linked to the data, at least at the scale of a search engine, you don't touch it. Is stateless, like the LLM that does the summary or that does the paraphrasing, and change it then one day for the other.

So, hey, we'll continue our interview in a moment after a word from our sponsors. So I got a couple different follow-ups I wanna chase down here. When you say it can never change, do you mean that outright, literally? It seems like if nothing else, as the world changes, you would need a new embedding model to understand new developments, right?

Yes, but when I say that you never change it, it doesn't mean that you don't add. You can add new things. What I try to say is that why do you need to replace one embedding for another one? You actually want to have both of them because most likely they are uncorrelated, right?

And you want to actually build an ensemble of it. Why do you want to end up having like a query-document distance? Why do you want to have a distance when you can have 2?

And then Use both of them to actually do the final ranking.

Embedding as features is basically like an additive.

You don't replace, you just add, right?

And then the cost of adding is just a matter of resources, right?

Because as I told you before, it might take us like a month with that many GPUs and actually paying a lot of money to run one embeddings against the whole dataset. That's not something that we might actually want to do it.

Yeah.

every 3 months, right? Perhaps it will take another year until we have yet another embedding type into the mix.

Different plans have different constraints.

That's very important too.

In your case, or like the case of the audience, if they have 1,000 documents or like 10,000 or 1 million documents, that's different constraints require different solutions.

Yeah. Definitely. In the plumbing case, I can rerun the whole thing without too much trouble.

But in any case, let me reiterate, it's better to try 10 embeddings, different models, and use an ensemble than to try to find only one embedding that does it perfectly.

And your ensemble approach is heuristic, right? It's not Or is that also a learned approach?

I can learn that, right? Because at the end of the day, you have, it's like features at the end of the day.

So it's essentially a mixture of experts approach in a way.

Of course, like mixture of experts is like another way. It's like another, like it's a really bad mixture of experts is because ensemble is already like, it's already used, but the mixture of experts is just an ensemble of 8 different models with 2 are selected at a time. It's exactly the same. The same way on your case, which is the best embeddings that work for my use case? I would use like 5 of them. And if they are uncorrelated, which is like the critical factor here, because if they are correlated, they're useless. I mean, they just, they don't add anything, but if they're uncorrelated and if some of them are very semantic, others are more literal, others are like multilingual, others are English. All combined probably will give you a better ranking than using only one.

So if I'm understanding the architecture you have today accurately, you have a number of different embeddings that have been built up over time, plus a learned combination of those that either— does it choose which ones to do or does it weight them?

No, weight them.

Each one gets a scalar?

We end up having, it depends on which search engine because we have multiple search engines depending on what stage you are in, but you're kind of in a 100 features, 200 features, 300, and then you learn with the gradient descent tree and off you go. Cool.

Okay. That's definitely interesting and good food for thought for mine and probably a lot of other projects too. Going back to the topic of upgrading the language model, You said that's like easy in the sense that you can easily swap one in for another. I imagine this is true for both search and the summariser. It seems much harder to evaluate and actually determine which is the better result. So we've got obviously things like LMSys doing head-to-head Elo style. Scale AI just came out with a new private thing that they're offering. How do you guys determine Which, if you're going to make a change to search, how do you determine if it's an improvement or not? And similarly for the summarizer, how do you go about assessing which is better?

We do have a team of human assessors that their work is basically assessed manually. It's not the majority of the search team, but it's a significant chunk of it. And their job is basically to have homogeneous methodology and homogeneous criteria to assess. Because one thing that we don't want to have is heterogeneous assessments, right? So this teamwork is basically assessed based on either summary outputs or rankings or equity. So that's kind of like the— our, in a way, our ground truth, right? Of course, like the productivity is what it is. They cannot produce a lot of assessments, but they are very high quality.

And then on the other hand, we have query logs, our own query logs, or what we do collect from the discovery project. So that allows us to compare what Bing or what Google or what Yandex does, right?

So we use comparison with our competitors, right? In the sense of NDCG or like just like comparing cardinality of intersection of the result set, right? So if for a certain type of queries, we do not match what Google does, then what we do, like we check those results and then we manually assess those results, a sample of it, right? To see what's happening. Are we screwing up or we just have a different criteria for this particular set of queries? So this is like the process. It's a combination of a lot of data from ourselves and competitors, not aero type, but in a way watching what the others do. That's why also engines can resemble each other in a way because we copy each other. It's not that we copy literally, but we end up using—

You're cross-referencing at least.

Yeah. Oh yes, we are cross-referencing, right? And if we do not return the same as, for example, if we have seen a query empirically in Google on the Web Discovery Project. We also compare with ours. If we see like an intersection, that raises a flag. It's odd that this particular query has to know, right?

And then those queries are analyzed and it's added to the test set. And you keep building over time with a lot of patience and with a lot of effort. You end up like building datasets that have like thousands, tens of thousands of like high-quality human assessment.

And hundreds of thousands, millions of automatic assessments, right? And those is what you use as ground truth, like to evaluate, to see if we are going on the right direction or not. And last but not least, what we also have is like some of these queries, we kind of keep them aside from the engineering team in a way.

Only the quality team knows them to avoid overfitting because that's like a natural thing.

Even overfitting is actually a very big problem on machine learning because you can actually have overfitting even without knowing, just by fine-tuning the meta-parameters of what you have.

Yeah.

The cross-validation set and all of it.

So we do have also a reset. And at the end of the day, it's best effort.

We are not that many people and we try to do this like this gradient descent.

If the results are not worse, And performance is not worse, we just move ahead. And if the performance is worse, then we do, we have to do a judgment call on whether like we actually go for it and optimize later, or we just cancel the whole release.

How does that differ, if at all, for the summarizer type experience that comes out of the— and just to set up the context there, that the summarizer is, and this is something that's both, You can experience it in the UI of search, but then also we're going to shift in a second and talk a little bit about the Search API, and it's available via both. So this is basically, as you said, your RAG-style experience where, okay, there's all these results, but now can we deliver a synthesis of these results that kind of gets delivered in paragraph explanatory form? Another additional challenge I would imagine there is that there's probably a lot of taste elements to what would even be considered good. Do you have a sort of principle? It's interesting, we've seen the OpenAI model spec recently where they, for the first time, have really started to articulate, this is what we want the model to do. So now we can at least know if it's doing what it's supposed to be doing or if this is by design or not. But I imagine you have to make a bunch of similar choices, Do we want it to be short? Do we want it to be long? That probably varies on context. So do you have a constitution for the summarizer?

Not— yes, but it's a constitution of one. It's like the person who actually does this and he tries to do the best. Of course, there's some guidelines that are like of good taste, right? We should not be like biased. We should try to be as concise as possible. There are like certain things that are kind of like common sense, but Other than that, we do not have time enough to define guardrails or guidelines. We believe that it's more also an iterative process. The fact that we do certain fine-tuning from time to time, what helps us is that there are like certain results that we think that they are very good. Those results will be used to like to fine-tune the model and the model should automatically learn what is the style that we want, right? And who decides what is good or not? The quality assessment team.

Again, we don't have any editorial criteria.

That goes without saying, right?

We don't play politics. We do not do any censorship or anything like this. But at the end of the day, it's a matter of personal taste of whoever does the rating.

But that's, it goes for us, it goes for anyone else, right?

It goes for you, right?

Do you prefer results from Perplexity or from us or from Google? The information is actually the same. It's just a style, right? Yeah.

As a power user of all of these things, I have to say I do find myself going to different services depending on the query. It seems like I've noticed, and I would say, I think Brave is in the middle, I would say right now between like Perplexity gives me a lot of times the shortest, most to the point answer. u.com research mode is on the other extreme where it will give me the 2-page, 1,000-word. Obviously that is not suitable for every query. And I don't know how you would qualitatively describe where Brave sits in relation to those, but for me it seems like it would be certainly in the middle of those 2.

It's on the middle because Brave has an extra complication compared to Perplexity, and we can discuss about that later, but it's a little bit like It might sound very unthoughtful what I'm going to say, but it's a little bit by chance. Llama 3 is much more verbose than Mixtral. It comes out of being more verbose, period. When we switched to Llama 3, we became more verbose than Mixtral. Of course, we can tone it down a little bit, X, Y, Z, but it's certain inherent things that come out of the tools that you use. And in a way, we do not have the time or the will or know what is right in order for us to actually act upon. And then in those cases, the best thing to do is like do nothing, right? But it sounds stupid, but it's true, right? If you do not know what is the right thing to do, you just— I basically trust the model. As simple as that. Once I do have more data or I have more resources to do proper assessments and proper judgment, I will try to influence. But influencing just because someone on Twitter says something, which is very tempting. So that's one thing. And then on the— it's very interesting what you said about that you use a different system for different queries because that's not what normal people do.

Search is one-stop shop, right? And Brave Search is search.

And when people search, they don't want this.

discover.

People search because they have a problem, right? They want to go to Hotmail or they want to go to porn or they want to know like a recipe of mojito because they are doing the mojito right now. They don't want to discover the 2-page history of mojitos or like a nice AI summary for mojitos, right? They just want to know the ingredients and how to put it together quickly because you are in the middle of something, right? And that's important to realize. Realize that whenever we think of search, we always think of fancy questions or complex questions that are thoughtful, but most of the queries are not like this. And because we are the default search engine in many countries on the browser, we need to cater to the common denominator in a way. And that's actually something that is tricky, right? Because we could be much fancier if we had no traffic.

That mojito example reminds me of a certain— it's almost a meme at this point that all the recipe sites have a long story at the top of the page now. And I'm sure you're familiar with this phenomenon, right?

Yeah.

Why did that happen?

Is that something that— because I would agree, if I'm looking for a mojito recipe, I'm not really looking to read your diary entry of the time that you made these mojitos and how much fun you had. And everybody seems to agree. So where did that come from?

SEO. SEO. By random walk, that actually worked better. Once somebody learns that, the others copy. It's mostly SEO.

But why did it work for SEO? Is that a, is that sort of a misgeneralization on Google's part? There must be something where like more content of this sort is useful and then maybe They overdid it and started to return recipes based on that criteria, even though it's worse.

Now you will— how to say that? Do not, never underestimate the, that things can actually happen by accident. So if you do a page that has very good information, but you do it in a format that the parser of Google doesn't like, your content is not going to be indexed in the same way that if you do it on free text. Where the parser for sure gets it, right? So what you end up doing, again, notice the writer of things that want to be SEO optimized, what they do is an ensemble approach. They try many different things to see what works better against the algorithm. And the algorithm might not be something that really has a strong rationale behind it. It could be that certain types of structure is penalized by accident, just by accident, because that particular thing is not properly parsed, or because that other thing is like between this tag and this tag made it to the list of things that has an extra boost. And those, according to the leak, Google has 14,000 features per page. I find it a little bit too much, but we have hundreds. So I wouldn't be surprised if they have thoughts.

So how do you think this will evolve as we go into a future of just language model generated content everywhere? There's a lot of talk right now about the web is sort of— the hyperbolic take would be like the web is over, it's all going to be overrun by LLM spam. And also people don't want to necessarily be included in training data, so walls are going up. in some places. How do you expect the web to change in general and specifically due to those factors?

I mean, the web has been changing for a long time, right? The upcoming of advertisement was a change, right? The ad blocking has also changed. So everything that is like a change of incentives on the web, right? And the fact that with AI, not only the economics, which are already be like pretty, pretty, even the reputational aspect of the web might actually go away, right? Because if nobody visits your site, you have your podcast, I guess that you make money out of it, but also like for reput— there's some reputational factor to it, right? If you lose both money and reputation is what, why should I do con—

why should I do content for, right? It's worrisome to, to say the least.

I didn't know what will happen, but certainly The web will change. And I'm a little pessimistic on it, not only because I like the web, right? I like the web. I like— I'm old enough that I prefer YouTube over Netflix. Even YouTube is full of crap, but it's also full of gems, right?

And I like more YouTube than Netflix.

I like more MySpace than Facebook, even though— because I like this variety. And I think that's like— like the AI is not the one who's gonna kill it, but is yet another turn towards, towards like a version of the web that is not what I had when I was younger. That doesn't mean that we are worse off collectively, not necessarily. But I mean, AI, the AI answers, in a way, they are very convenient, and that's why Google is forced to implement them. And in a way, that's why also We are forced to implement them. Remember that for us, it's not cheap to run a summarizer, right? Because we do have about 27 million queries per day, 30 million. 10%, 30% of that to be run through an LLM, that's not— that's actually for a lot of queries, right? So we are like not forced, but there is pressure to it because people People find it very convenient, but anything that is convenient has a price. You sacrifice something for convenience.

That's given.

On the other extreme, like, I don't want to, like, I don't want to go to the river to fetch water, right?

I want water to come out of my tap.

So, but how much convenient? But that's kind of, what is the threshold? I do not know. But anyway, convenience is king. And answers with AI are here to stay. They are actually very useful and very powerful and very convenient too. But of course, we'll pay a price, not only on the ecosystem, but also the variety of answers that we get. Because I think that some of our competitors say like that answers are knowledge. No, answers Questions are knowledge. Answers are just answers, right? So the fact that you have an oracle kind of type of thing that for anything you ask, it will give you this, that basically means that everywhere, anything else does not exist. And that's dangerous.

Yeah, it does seem like it's gonna be a tough time for a lot of the content creators. I'll continue to do the podcast just so I get to talk to leading thinkers like yourself. Regardless of any, whether anyone listens, but it does seem like there's a lot of people out there who are—

Oh, and I don't know, like OpenAI had, because they have like big pockets, they tried to pay, but I do not know if the economics actually are, because the kind of the economics of the advertisement, right? Yes, Google makes a lot of money, right?

But if you have to split that money equally among all the parts, everybody gets, gets something that is not enough to live. Yeah.

We've been. By searching for endless convenience, we commoditized ourselves out of that. I don't know what will happen, honestly.

The business side of this work for Brave, I'm just doing a little back-of-the-envelope math and I'm imagining, okay, geez, if you're doing 30 million queries a day and I don't know how much, how many tokens you're putting in per query, but We can get into a little bit more specifically, like how much data is returned, but it's not an insignificant amount. It's gotta be thousands of tokens per query.

Like, it's 4,000 tokens per query usually. We usually max out our context window, which is like 4,000, but it's also, it's all self-hosted.

We do not use any third party.

Not only, not because of the independence part, which would also be because we value that we are independent, but also like We would not be able to use OpenAI or anything like this cost-wise. So everything that we do is like self-hosted and we, it's only that we have to pay machines. I think that nobody will complain if I say that, but for instance, right now I think that we have 20 A100s out of Amazon. So that's our infrastructure for the PikaLM plus our own machines. A lot of GPUs for smaller models, Q&As and classifiers. So it's a big infrastructure, but it's not as massive as many people think.

So I guess for now it's not so expensive that it forces a business model change because this is something Google's obviously going to be facing in a big way too, right? It's very unclear how, even if you're doing pay-per-click Advertising within search, it's very unclear how that fits in with these AI answers. And then more broadly, like it's just very unclear how to— do you guys think of like a subscription model in the future for something like this? Or do you have any idea for how this evolves business-wise?

It depends. If you enter into the chatbot domain where you have a back and forth that is constant, then the economics might work differently, right?

But if it's on the context of search where like the There's query, there may be query reformulation, but there is only one trip.

The cost, let's say that worst case scenario, you double the cost of infrastructure, right? People tend to over-exaggerate the new things over the old ones. Yes, running an LLM is very expensive, no question.

But it's not like a search engine is super cheap to begin with.

That's my point, right? If Google doubled the infrastructure, I'm sure that they can do any LLM.

They won without any issue and they suddenly have the margins to develop infrastructure. I think that's not like a cost problem. It's more like a, whether we want to do that problem, whether people actually want to have that. It's very unclear that majority of people want to have that.

With search, it's very, again, let me go back to the point I said before, like that once you have traffic and you like serve all use cases, right? People are not— When they search, they have a problem.

They don't want to be disturbed, right?

Because they are in the middle of something, right? And people do not usually remember good cases.

They only remember bad ones.

So you don't go to that particular shop because they treat you wrong once. It's not—

you go to the other one because on average they treat me right most of the time. So you avoid—

you don't remember good cases. reward the place, you basically punish the one that you don't like, right?

So if you start to do certain experiments, if you have a lot of users, you risk losing a lot of users because they don't like what you're offering, right? Because it's in the way, right? You're annoying them.

And that's why, again, without traffic, you can't do many things, or with a very small traffic, or with a traffic that is of a very particular segment of population, you can do many things.

But once you have the general public and your service is like running water, you open the tap and please, you cannot, you don't have the luxury of doing a lot of innovation there.

And actually Google is suffering.

I pity them honestly in the sense that it's a very difficult thing what they have to do. And we have a much easier position than Google has to react to LLMs. And you.com or Perplexity or Kaggle have a much better position than where we are, right? Because they have less commitment. So like blank slates for the win. It's always the case, right?

What does your user base look like? I know that obviously we're speaking in English, but you're From Spain and others are— the company's headquartered in Europe, correct?

No, Brave is headquartered in the US. Search is 100% Europe. There is like only these people, only people in Europe and some people in Africa, all same time zone.

But all search is EU, not EU, Europe-based. But Brave is an American company, but it's like heavily developed, it's heavily remodeled.

Maybe a more general purpose question would be, again, where are Brave users? And I wonder if you see differences cross-culturally in how people are responding to the AI moment. Like, how would you characterize how consumers are reacting to AI experiences in Europe as opposed to in the United States?

No, we have no done this analysis.

Brave Search users is actually a good sample of the world population.

About like 30% of the people are in the US.

The next big market is already in Europe. So we are like pretty well spread across all continents.

There's no odd countries or odd behaviors. It's large enough to be statistically well distributed.

My intuition would be that I'd maybe be more interested in this summary, just because less, fewer of the sites are mobile optimized and it's just more hassle to browse around.

Could be. But at the same, but at the same time, when you're on desktop, you also would expect to have, uh, more work-related difficult queries. On more, because on mobile queries are usually, they're very instrumental. They're very like phone number of restaurant.

Yeah.

Or like, uh, fun facts. Again, Age of Lady Gaga. It could be like playing a trivia of some sort with friends.

Let's maybe turn to the Search API. I mean, this is an area of the business where there is a very natural business model, and this has been like the main subject of the ad as well that we've been running. I went ahead and created a little Google Colab notebook that allows a user to easily show up and try the Brave API as compared to the Google Custom Search API and then also the Bing Search API. And I have a few observations that I would be happy to share, but maybe you should start by just telling us, like, how do you see the product as being differentiated from the other options that probably a lot of people are more familiar with?

Yeah, the reason why we had—

we released the search API to provide an alternative to big tech.

As far as we know, there is only 2 APIs out there that do a search engine, like Google and Microsoft. And Google is not public.

I think that you, you use it, the custom search from Google. That's not meant to actually be used in production. I think that they have a cap of 10,000 queries. So if you actually want to use Google API for something real, You actually have to go to talk to them and they might give you access or not. It's their own decision.

On the other hand, it's a little bit better in that respect because they have a public API. Of course you pay, but it's public. You go there, you accept the terms of service, which is a bit draconian, but that's a different topic.

But you pay and then you use, that's it.

And at Brave Search, what we try to do is like, Same as Bing API, it's a public API. You pay, you can use it, no questions asked.

We try to be less draconian on the terms of service, right? So we do not require attribution.

We allow for you to mix the results the way you want, things that like Brave would not—

sorry, that Bing would not allow.

But other than that, again, there is like nothing else other than Before you only had Microsoft, now you have Brave and Microsoft.

That's not a small thing.

And we did that like the very moment we actually could launch the API, we did, right? Because we are not like 100% independent all the way, right?

When we launched Brave Search, we were able to answer like 87% of the queries, right? For the remainder, we actually used Bing and we scaled that over the months, right? So we went like from 87, a few months after it was 90, 92, 93, 94, and a little bit more like than a year ago, we like went from the 94, no, 96 or 97, we went back to be totally independent from Bing. And at that moment we could offer the API, right? Because before it was not possible. Because of terms of service, I think.

Yeah, that's interesting. I would agree definitely with the idea that the Google Custom Search API is not really a competitor.

It's meant to do site search.

Like your own site?

Yeah, actually, I'm pretty sure that if they— you can do the trick and not pass domains so that it doesn't do site search, it does global search, but I'm pretty sure that it goes against the terms of service. But in any case, it's limited to 10,000 queries per day, which is—

The thing that stood out to me the most, and folks can go judge this for themselves with our collab notebook, which I'll put in the show notes, is really more than anything else, just more information being returned. In my case with Waymark, for example, we have this flow where A new user shows up, and I'm always amazed, like, that so few products implement something like this because it's worked really well for us. New user shows up, they want to make a video for their business. They typically don't have a ready-to-go folder of all their assets, let alone like a good profile of their business, right? So now we're going to use generative AI to put things together for them, but we need to know who are are you? Tell us about your business. What do you offer? What makes you special? Do you have any images that we can feature? And people just don't have that stuff. So most products just require you to make a profile and upload your images and whatever. We have worked pretty hard over the years to create a good experience where we go grab that content off the web for you. And as you'd imagine, the generative AI era here has really improved how well that works. We have used Google Custom Search for that. Our volume is not super high, and we're mostly just trying to get their homepage, or kind of then we'll maybe bounce a few links off of that. But I am going to look more deeply into using Brave for this use case in particular, because just more information really jumps out at me. We get a snippet from kind of the Google Custom Search, and with The Brave results, you get often a much deeper cut with multiple snippets and even some structured data for some results. I looked at this a few different ways, but in just raw characters returned for a 10-result query, the volume of data was often a multiple higher than what I was getting with Bing and Google. And that also extends down. You could fill out characters with anything, right? But a lot of that seems to come from these additional snippets, which for us are really useful. We right now first run that query and then we go crawl ourselves, and then we pull all that stuff in, and then we run our own language model to process it and turn it into a profile that we can then feed downstream into the actual generation task. But I could definitely see us simplifying a lot of things, and that's always nice to take multiple steps and condense them into one. If we just could get that multiple snippets alone, I think would be a pretty notable enhancement. That's before the summarizer too, which is something I also could see us doing because again, we're doing that ourselves today and you've implemented that whole thing. I'll have to check on the coverage for our small businesses because some of our customers are pretty long tail. They're generally not super famous businesses.

But yeah, you have to check that you get the right business at the end.

But for me, a volume of information was the biggest thing that jumped out. Would you expand on that anymore?

Yeah, but basically the main motivation to provide— we did not have the motivation to provide more volume per se, right? Because again, remember that the amount of data is meaningless— sorry, the amount of data is very important as long as there is no noise. Because if you actually just— the risk of adding noise precludes any amount of data that you can provide. So our rationale was the following. It's like we— because we do the dynamic snippet generation query depending, right? We end up having multiple candidates that we run through a Q&A model. That's a cost that we already pay.

So as long as that is above a certain threshold, which means that in principle there should not be noise, I see why we should not send it to the user so that they can have the choice to decide which snippet they want to show, or in the case that they want to do LLM inference, to have a little bit more data to to feed the prompt or to cherry-pick, right? Because remember, cherry-picking is actually very important.

And it's on the topic of summarizer. The secret sauce of our summarizer is not the model, the LLM.

Again, we change it from Mixtral to LLaMA in a heartbeat.

It's the cherry-picking of which data we feed the prompt, right?

Like, that's what we spend the time on, like, only feeding, like, non-repeated, non-contradictory information. So we do a lot of preprocessing on that. Anyway, so hopefully like on the API, what we try to offer is we give you like all the elements so that you can innovate on top of it. That's like our goal. Of course, we want to make money out of it too, right? So if it has more value than Bing, it's also good because like we will have more users. But Anyway, it's not like one or the other. You can have both reasons at the same time, right?

It is notably significantly cheaper than Bing as well. You have multiple different tiers of service for different amounts of data that get returned and somewhat different price points for different licensing as well. And there's whether you're gonna be able to store the data or not. So folks can get into the different price points, but Bing also has a lot of different price points. And at least for me, when I was looking at what I would naturally want to use, including the summarizer, that is a $9 per 1,000 queries price point with Brave. And with Bing, I would say the most comparable thing, they don't even really have that summarizer feature at all, but the most comparable price point that returns the varied results where you might get video results or different kinds of discussion type pages is a $25 per 1,000 price point. So saving more than 60%. And I would say getting more both in terms of the additional snippets, the extra information, and the ability to tap into that summarizer. Anything else that I should be conscious of as I think about the right option for my app?

No, the right option for app, it's like whatever works. And if you can afford all data combined and do an ensemble out of it.

But no, but it's like, I mean, we are very happy with the Search API and it's not just that. It's one of the—

it's actually, it's a satisfaction from us when like we talk to a customer and they say that, well, it's about time that there was an alternative to Bing.

That's actually for us is very— It's very rewarding to be able to achieve that goal. You might use Bing, you might use Google on a private deal, you might use whatever, but the fact that you have more and more to choose, that's a big deal for us.

Do you want to shout out any customers that are using the Search API that folks can go check out if they want to see a good implementation?

Oh, oui. I wouldn't be able to play favorites. So, because actually there are 2 things here. There is the vast majority of our customers do not allow us to mention them. It's a very competitive space. Many of the big players do not want to admit or let others know that they have or that they use Hey Search.

Or any other system. And then for the one who actually acknowledged, then it would be kind of unfair to single out one.

But I can tell you like some nice use cases. Of course, there's like plenty of RAG use cases that are being done by names that have been mentioned before. It has also been used by— to train significant foundational models. One use case that that I find very interesting is that it's also being used to do grounding of LLM outputs. LLM output, then you actually run it, you segment it, you run it against a search engine to try to verify whether the output is legitimate or not.

So there is like a lot of, and many other use cases that we do not know on because at the end of the day, we do not, unless they ping us, we do not ask questions, right? use it. And hopefully it's beneficial for both of us.

Are there any favorite technology complements that you would, almost like in a cookbook or demo showcase sort of way, are there things that, patterns of development that you've seen where people have combined the Brave Search API with some other framework, technology, user experience that you think is worth sharing just to inspire Yeah, there is like this, I don't know how old it is.

There was one guy who actually did like a YouTube video, and I think that these are on GitHub too, like to build a Perplexity clone using LangChain, Brave, Groq. So he did like a whole convolution of APIs and he actually built like a very nice prototype. I think it's good kind of like to get the idea of like how it works. Yeah.

Why the things sit together, and then people can decide where to to split or to go deeper. Cool.

You've been very generous with your time. Anything else that we haven't touched on that you want to make sure we cover?

No, I think that we I think that I already talked too much, so hopefully I did not bore your audience with my ramblings. But it's been it's been a pleasure, Nathan.

Very nice host. I appreciate the chat.

Cool.

The feeling is mutual. Joseph Pujol, thank you for being part of the Cognitive Revolution.

Take care. Bye-bye.

It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.