Milliseconds to Match: Criteo's AdTech AI & the Future of Commerce w/ Diarmuid Gill & Liva Ralaivola

Hello, and welcome back to The Cognitive Revolution.

Today, my guests are Dermot Gill and Leva Ralaivola, CTO and VP of Research and head of the AI lab at Criteo, the advertising technology company that powers much of the personalized advertising that we experience on the open internet. I'm also joined by my longtime friend and teammate, Alex Persky Stern, who took over for me as CEO of Waymark some 3 years ago now, and has since formed a partnership with Criteo that brings Waymark's AI-powered commercial creation product to Criteo advertisers. We begin with an explanation of how modern digital advertising works and the value that it creates for society. Personally, I tend to emphasize that without the commercial recommendation systems that allow businesses to affordably reach their target customers, a lot of the long-tail small businesses and niche products that we enjoy today simply wouldn't be viable at all. Dermot and Leiva, for their part, focus on how ad tech delivers more relevant, engaging experiences and supports free access to information, as well as emphasizing how easy it is for individuals to opt out of personalization systems. From there, we dive into how it all works. Criteo has been in business for more than 20 years, and while their AI techniques have naturally evolved with the field, most fundamentally from the earlier era of handcrafted features to the modern era of deep learning, the one constant has been their need for incredible speed. From the time your browser requests a webpage, Criteo has just milliseconds to locate your profile among the billion or so in their system and, in light of what you're doing right now, decide which one of many millions of products to recommend and also how much to bid in a real-time auction. It's a deeply challenging problem that requires lots of precomputing, but the upshot is that they've developed a highly modular system powered by multiple foundation models that supports prolific experimentation on top of cached user and product embeddings. Beyond the core tech, we also discussed Criteo's new partnership with OpenAI, which, though still in its infancy, they expect will complement ChatGPT's broad world knowledge with accurate real-time product inventory information. They tell the story of the company's European roots and share their commitment to privacy, their sense that European compliance burdens are overstated, their decision to use the same Euro-compliant tech stack globally, and their passionate belief in the European AI ecosystem and talent pool. They also explain why they're confident enough in their moats to publish a lot of their research and how this helps them attract and retain talent well enough that they're still comfortable publishing the AI lab's full 50-person roster to their website. We trade ideas regarding the role that generative AI will play in the expansion of the advertising market and evolution of personalized creative. And they share their admittedly speculative thoughts about how the fundamental value exchange of advertising might change as human time becomes more valuable and AI agents take on more product discovery and research work. Overall, I think this episode is both an informative look at how modern AI techniques are being used to make high-value commercial recommendations under extreme constraints, and a useful corrective for those who deny the ways in which cutting-edge advertising enriches modern life. With that, I hope you enjoy my conversation with Dermot Gill and Leva Ralaivola of Criteo.

Dermot Gill and Leva Ralaivola, CTO and VP of Research and head of the AI Lab at Criteo. Welcome to The Cognitive Revolution.

Thank you, Nathan. Pleasure to be here. Hi. Thank you.

Excited for this conversation. Also excited to welcome my longtime friend and teammate, Alex Persky Stern, who's the CEO of Waymark, which we had been building together for a number of years before he took over for me as CEO a few years back. Longtime listeners have heard many asides about Waymark, and Alex is the guy running the show there now. Happy to have Alex here today because Criteo is obviously in the advertising business and bringing a lot of AI to the advertising business in various ways. And Waymark, under Alex's leadership, has partnered with Criteo to provide some creative, uh, solutions as part of that whole package and go into market together. Lots to get into. I wanted to start with something that I saw recently that kind of caught my attention, and I wanna get your take on it. It was, I'm sure you've seen this, a Bernie Sanders sit-down across the table with, I believe, Claude. Having a voice conversation, and it was a bit of a strange tone. It felt a little dated to me in some ways. But the subject of the conversation was Bernie saying to the AI, what do you think Americans need to know about how their data is being collected and how companies are profiling them and how that's all being used? And it kind of had an ominous overtone to the whole thing. I think there's probably still a lot of misconceptions or misunderstandings out there about this. But this is a significant part of the business that Criteo is in. So I would love to hear from you guys, as folks who have built it and are doing it today. What do you think Americans need to know about how their data is being collected and how it's being understood and how it's being used? And what's the upside of that to— and maybe the downside as well, but What's the upside of that to businesses and consumers?

Yeah, it's a great question and it's something that we in the industry need to do probably a better job of explaining and demystifying. For me, I think it all boils down to, you know, like transparency. So, you know, explaining to users what data is collected. So for example, within Criteo, we don't collect any personal information. So it's really, you know, a random anonymous ID. And then there's some things around, you know, what products people are interested in, you know, kind of what they've seen, what they like, what they don't like, and so on. And it's all about, for me, about a value exchange, right? So relevancy. So, you know, a system that knows nothing about you is gonna show you random stuff that's irrelevant. The brain has a really great way of filtering out irrelevant stuff, whereas something that's truly interesting for you is way more engaging. It's way more resonant. And for a user, that creates a better experience. Also, I think in advertising, one of the great things is advertising is very much the lubricant that keeps the internet open and free, right? You know, that provides, that allows service providers the ability to keep their services not behind paywalls. And so there's great utility in that. And advertising is what keeps that going. It's the revenue that those content and service providers get. that allows them to provide those great services to the end users. And, you know, providing transparency so the users can actually see what's happening and the ability to opt out. That's also something that's very, very important. You know, Criteo was a pioneer with the ad choices icon, so someone can click and they can see why they saw this ad, and that gives them the ability to opt out. Once you do that, then I think it provides great value to all the participants.

We try to be like mentalists. We have very few information, very few cues, and we try to detect what is going to be the most relevant for each consumer and each end user so that in this value exchange, everyone is going to be happy down the road. So as a middle party, we have down the road the ones who have the least data and the most challenging tasks in terms of AI. And that's why actually on my part, I'm with you because the challenge in terms of machine learning, AI, is really a big one and it's the most interesting one.

Could we do kind of a double-click on a couple aspects of that? One being like, if I were to open up my file, I'm not even sure if that's quite the right way to think about it. I'd be interested to know like what's in there. And I have, you know, I've occasionally clicked the sort of ad choices thing and seen, oh, seeing this ad because you're interested in skincare. And I'm like, okay, my wife got me this one, but I don't really know. That's kind of a high-level sort of summary statement of why I'm seeing it. I don't know exactly what is under the hood. I'm also kind of confused about, like, of course we go see these, we go to websites all the time and we get this popup that says accept cookies, don't accept cookies. What's going on with those cookies? I know there was a big change to the industry, and I think it was kind of driven by Apple a few years ago. Maybe it was driven by other parties as well, where the way in which information is gathered and the nature of that information was kind of changed. I think there were some winners and losers from that. I'm not quite sure how that really shook out or if we're back to essentially the status quo ante before those changes were made effectively. But I, yeah, I'd love to hear kind of just a little bit. more concrete description of like what the data is? And then that obviously feeds into what is the machine learning layer that sits on top of that data look like to make sense of it? You know, obviously that data becomes the inputs, but I always like to get down to very brass tacks on like, what are the inputs and outputs of the models so we can really understand like what it is that the AIs are doing for us?

Sure. So there, there's a couple of different ways that I think it works. So first of all, I mentioned, you know, You know, when you arrive on, say you're on a retailer or on a brand website, so they, you know, kind of using technology like Criteos, we can create a record on the computer called a cookie with a random ID. And the ID doesn't have any personally identifiable information. And so then afterwards, as you kind of continue browsing, so if you look at a product, then afterwards when you leave that website and you go browsing the web, they can know that you've shown an interest in that product, right? And they can show you ads for that product specifically for the same one you've seen. Alternatively, what could also work is if we, for example, you took a look at a mobile phone, well then that could have you assigned to a group of people who are tech enthusiasts, right? And the type of phone that you look at could be interesting too. So iPhone users have a different profile from Android users from whatever else. So you could be part of a wider audience that could just be seen as Apple enthusiasts or tech enthusiasts and so on. And then when an opportunity, when you're browsing the web and you're looking at web.com, internet.com, whatever, an opportunity comes for advertisers to bid, to make it, to pay the website owner to show the advertisement in front of you. And based on the information that they've got about your previous browsing history, they can then decide whether they want to take this opportunity, show you the same product or show you equivalent products. And then maybe I'll hand it over to Leva to kind of say how you actually do that bid, how you decide whether to show an ad or not.

Yeah, precisely. One of the very important things is to be capable of evaluating the expectation of revenue of a placement and knowing that if you are going to place advertising in that placement, then there is a high probability for you for it to be clicked on or not. And you have to evaluate that. In order to do that, you're going to use machine learning and AI models that are precisely going to evaluate whether a placement, and given some products that we can put on, is going to be something that is going to bring revenue. And for that, we precisely do a lot of— we collect all those data that Jeremy talked about, and there is a huge machinery that we put in place in order to learn from that data. And if I had to summarize the type of problem that we're doing, even though it's a bit more complicated, should we bid or should we not bid on that place, that placement? And we learn a classifier from that. And it's, of course, there's this question that you started with, whether it's when you click and want to see more information, there's this question about utility and readability that exists. If you want to be very precise in terms of evaluating the value of a placement, then you have to use very sophisticated models. Probably have heard about deep learning models. And the more sophisticated the models are, the less easy it is to understand what they have computed. So there is this trade-off to have. Now we have all in the industry, and in particular in category video, we use those deep learning models in order to assess whether placement is good and assess whether this product is going to be relevant for you. So it means that in a way, we have what we have gained in terms of precision and relevancy, we have something to make up in terms of explainability. And just so you know, because we talked a bit before we started, that's a big topic in terms of research, scientific research, AI research, to provide explainability on those models that are doing crazy stuff. And one of the things that we are looking at as well, but it's not easy to have both high utility and high explainability.

Hey, jump in. I actually have a question I'm interested in here. The, you know, the idea of the user profile and, you know, who, what the, what this person might be interested in is obviously super, super core. One thing that I think is really interesting is you guys have this OpenAI partnership, which is super cool, and I know very new, so some of these answers might not exist yet, but one thing that people talk a lot about, of course, is that the queries are much richer in the context of AI and chat. But something that I haven't heard people talking about is whether the profiles are meaningfully different. You know, if I'm working with Claude or an agent, like, it knows a lot about me. Is that starting to change what— how we understand the user? And where do you see that going?

So many things. So starting with the first, very first thing you said, okay, that's— there's still a lot of things to unpack, to uncover. We are precisely at this stage because, of course, there are many questions about privacy of the data are defined. That's a great question. So far, there is no answer yet. But one thing that I can answer, though, is that those conversational agents, they provide with a new interface. Before, you just had essentially the websites or some apps that you could use, and now they have that. One thing that is very important is that those models, like FluentGPT, GPT, Claude, etc. They are very, very good at general reasoning, so they can do some recommendation. In some ways, you can think that they're going to be good to do. And if you ask, okay, I would like to buy shoes, they are going to propose you shoes that are relevant. But one of the things that we don't have is commerce data, which are going to tell us about what exactly people are interested in. And the big challenge that we have today is precisely to make the— to have both models, the LLM models that are behind all those conversational agents with all the models that we have in Criteo that are capable of providing very accurate commerce information. And the challenge in terms of technical challenge is precisely to merge the two and maybe some ways, but that's not on our part from the LLM. They are going to have some information that is encoded. But it was precisely not necessary to have this information, but more to know and see how we can enhance or those models can enhance our commerce models that we've built for years. And that's where we sit as of today. To you, Amit.

Yeah, I think that's exactly right. So the thing about the LLMs, and it's amazing technology, we're super, super impressed by the power of all of these. I think everyone is. But when those companies, when they train their model, it is true and accurate at that moment in time. Commerce data is actually way, way more dynamic, right? And so, for example, you know, kind of, they would not be able to know, for example, that there are flash pricing. So, you know, around Black Friday and so on, where things change very rapidly. They also wouldn't know, for example, things like ruptures in stock, right? So the way that they gather their information is by doing this massive crawling of the internet and then at that point in time when they've updated their model, very quickly it starts becoming stale, at least from the product point of view. So Criteo has this massive network of 17,000 retailers. We ingest their product data on a daily basis, sometimes multiple times a day, and it means that we always have access to fresh data. So like Leva said, we did this hybrid architecture where an LLM in partnership with technology provided by a company like Criteo can ensure that when a user asks for a product, they not only get all the richness that an LLM can provide, but they can also ensure that it's up to date and it's accurate. Because from a user point of view, it's a very bad experience when you search for a product and it comes back with something, you click through, and the product's either a different price or it's out of stock or it's not what you were thinking about. So that's why that hybrid architecture makes so much sense.

Yeah, so today, just to make sure I'm getting it, the process is ultimately pretty similar to what you'd have on the open web when you're in the chat interface. maybe with a richer query. AI is being inserted in a whole bunch of other different places in the stack, but that applies really across all surfaces.

So I would say yes and no. In fact, I think where these tools like the LLMs have an ability to elevate the whole experience is in the area of product discovery. And so if you're in the market for a new product and you're in that kind of like trying to dig out a lot more, so I think for the first time ever, we have the ability to provide the end users the experience, the same experience you get when you go into a store and you've got this really kick-ass sales assistant who only cares about giving you a good experience, who can answer your questions, who knows the full catalog, who's able to kind of tell you the good and the bad of each product. And then it leaves you with the experience of, you know, when you walk out the store, you feel like that person has really answered what you're looking for. So the LLMs in coordination with the accurate product information can do the same experience where you can actually query, you can ask extra questions, you can drill deep down, and it doesn't get tired, it doesn't get bored, and it's always giving you real-time accurate information.

And maybe there's something, maybe that's very technical, but that has been changing. It's how you're going to connect the tools and the thing that we provide Those LLMs. So you've probably heard of the agentic era, the fact that we have MCPs, those protocols that are going to help make almost transparent the use of already made-up tools. And that's something that makes it easier to combine those LLMs with what we provide. It's something that, of course, is technical. Maybe nobody cares about that, but in terms of deploying something, it's It has been a lot easier because before you had to adapt to each surface, to each website, etc. But now with those protocols that are coming up, it has been made easier. So that's just our duty to make sure that we are compliant with those protocols. And that's actually what we do. It allows us to surface all the tools that we've built over the years and make them available to—

Yeah, great point.

Hey, we'll continue our interview in a moment after a word from our sponsors. Can I dig in a little bit more on the core models that you guys are using to make predictions? And I'd love to understand the architecture of this better. I think for calibration, anybody who's listening to this feed is going to have a conversational familiarity, at least with how large language models work. So we know that they're generating a token at a time. We know that The inputs get embedded and we know the kind of mechanics of the forward pass and all that stuff, right? And we know it's autoregressive, blah, blah, blah. This strikes me as like a very different world, and I don't have nearly as much intuition for what the models are that are driving these things. I do know that they have to be a lot faster because the ad's got to show up really quickly on the page. And then I know also that there's a pretty challenging matching problem in there somewhere because I've got millions of— you've got, we've got, society collectively has got millions of these profiles of individuals. And then also, as you said, like into the tens of thousands of advertisers. And I don't know how much precomputing is done or whatever, but it has to happen pretty quick on the load of a page. So Could we kind of break down, like, how big are these models? You know, what do the inputs look like? You could imagine something very large and sort of very sparse set of inputs. But I guess it doesn't seem plausible that it's like, here's all the websites and here's which ones this user visited, right? That doesn't seem like it works. So there's got to be some sort of tokenisation or something that is kind of bringing the user profile into a manageable state size so that it can be used as an input. I'm not even sure if I'm quite asking the right questions here. So, you know, tell me what this looks like under the hood.

Yeah, so maybe I can take a quick stab at it and then Leva can take it down into more detail. So Leva actually referenced this earlier. So every single time that we want to, when we got an opportunity to show an ad, So that opportunity actually goes to multiple different ad tech providers who are all acting as kind of delegates on behalf of the actual advertiser themselves, whether they're brands or advertisers. And so the amount that we bid is based on how valuable that opportunity is to the advertiser, effectively how likely the user is to click on that ad and go back to the website and buy the product. And the way we evaluate that is we, through the mechanism we talked earlier, we see what products the users are interested in, what they've looked at, what they click through, what they've seen, what they buy, what they don't buy, and so on. And so as the display opportunity comes up, so we see the ID that we mentioned in the cookie, and then we take a look at all of the different products that that person has seen or whatever audience segments they belong to. And each one we can say, okay, based on all the different features we put into the model, so, you know, the products, the previous purchase history, the context of the website, the device they're on, a couple of other things that come in, and there's actually probably, I'm not sure, it's like 150 different features we can take in, and each of those go into this calculating as part of this massive equation which will tell us the likelihood that person is to click, the likelihood they are to click through to the website and eventually do a purchase. And all of that comes out to a value which we bid. If we win the opportunity, then we have to say, well, which products do we show and how do we do all of that kind of stuff? All that process gets done in milliseconds because we use a lot of caching. We've trained the models offline, then the inference happens at real time and really low latency.

So, one thing that is important regarding all the data that we have, like the visited websites, the products that were shown, clicks, etc., and then the model, the question, as I said before, is kind of, let's reduce it to a classification, a binary classification problem. One of the main tasks for people who have tried to do some machine learning is how you're going to encode and how you're going to represent the data. So I'm going to do that in 2 steps. The first one is going to— I'm going to talk about the legacy, all models that we used to have and where we are now, and that's where we've been for a couple of years. Before, there was this question about— so all the products, the websites that you visited, you have to encode them. And you have to encode them so that the way you encode, so the vector you're going to use to represent all those past information, still carries a meaning. If you just encode them in a silly way, you're going to lose a lot of information. So before that, there was a choice before, because of speed of computation and because you have crazy intuition about the type of model, it was like a sparse representation, like very huge vector with 2 to the 12 number of inputs with 1, 0, 1, 0, 1, 0, because you can do very fast computations on those sparse vectors. But that was a way to represent the data that we used to have. And we just learned from that vector what is called a logistic regression model. So it's a linear model. You can think of just one neuron with a lot of neurons coming in if you have the learning analogy in mind. And we used to have that and we learned the model and it was very fast, even though it was sparse. There are many libraries to do sparse matrices and sparse vectors. But then it was something that was very manual on the features before. And of course, reasons why, for instance, the CryptoEye Lab I'm the head of was created was to say, okay, maybe it's not sustainable to have to craft new features and to think about how we're going to represent data each time because the cookies can change, the information that we have. And for instance, with LLMs, it's going to change. So how can we proceed with more modern techniques? So it was the intent and the goal of the CREA lab to bring deep learning. So it was created in 2018, and it was precisely the objective to say, okay, let's go to the next level, not have handcrafted features, but rather have them computed from the data. So now, before we had like 2 to the 12th or to the 20th, depending on the encoding, sparse vectors. Now, Essentially, we have something like between 200 to 1,000 features that are automatically computed by one of the proprietary algorithms that we have, which is called DeepKNN, that computes deep learning features, on top of which we do learn some other models that are going to do those classification tasks. Maybe the essential thing to understand is we went from 2 to the 12 sparse vectors to something that is a couple hundreds. And now we are at the next level again. We're going with all the things that are available with these models that you can download. We're going to the next level, trying to even be more adaptable to the data that we're going to process. And that's something that is going on as we speak. is we refine all the models that we have. So that's the types of models that we have. And of course, the thing that's very important is, and that, Jeremy, everything happens in milliseconds. So what we have as a challenge is not only to be accurate, but also to be fast. So that's a nice challenge that we have.

To be accurate, to be fast, and then also to do it billions of times a day. at huge levels of stability and reliability. And then the other thing that's a great challenge within our system is when you have the Black Friday, Cyber Monday, the busiest period of the year. So imagine any world in where your piece of technology, you are able to run it at like 300% of what you normally run it. Imagine taking your car and running it for one week at 300 miles an hour. And then back to normal for the rest of the year. And it's the same car, same machinery, and it has to perform exactly the same way.

That sounds like a challenge.

It's fun. Yeah, the engineers behind it, my hat goes off to them. It doesn't sound like the work they do.

One thing for the people listening and for you is that one thing that is important for us is also to share how we or machine learning, you can access blogs that explain the deep KNN methodology and that explain label smart matching relevancy for retail media business. And if you want to go even deeper, we have more scientific papers because we have researchers doing AI science and publishing in conferences. And everything ties together. And if you want to know more, if you want to dig deeper, if you want to know the sizes of the models, how we train them, what are the losses that we use, you can go— we have a bunch of articles online that you can totally download and read to have more information.

Yeah, I was, of course, reading some of that ahead of this call, and it's super cool to see how you guys have made that both public but also pretty Easy to understand for— I'm not a true technical person, and it was totally easy to follow.

Hey, we'll continue our interview in a moment after a word from our sponsors. Could we talk a little bit more? And this probably is to some degree in the papers that you've put out, but the sort of architecture of models and kind of what gets precomputed. I'm thinking back to an earlier episode I did with the woman who leads AI at Stripe. And they had a pretty interesting strategy that I imagine you might have some similarities to, where they trained this foundation model for payments. And it's a huge model trained on, I don't know, like a trillion payments or something, a massive dataset that they have. It strikes me that they can be pretty open about it because they have the data, everything's flowing through them, and that's not about to change. So They can afford to be fairly open about techniques. And you guys might be in a similar spot where having the network is the moat. And so you can afford to tell more of the techniques than maybe other companies could. But one thing I thought was really interesting that they did with this foundation model for payments was instead of trying to use that model for all the different tasks that they have within the company, and there are many, they Use the embeddings of a given payment as the input to other models that may have other inputs as well. But in sort of modularising things this way, they were able to both amortise the cost of this model across a tonne of different use cases without necessarily having to even anticipate what those use cases would be, and also do something that Really just allows the developers across the company a lot of freedom to say, okay, now I know I have this really rich signal that I can kind of treat as a black box. But if I bring that into whatever machine learning task I'm trying to work on right now, they're just seeing dramatically better results across the board because that signal is so rich. I hadn't really heard that in too many other places. But I wonder if you guys have kind of a similar structure where there are models whose outputs or whose encodings, embeddings, whatever, feed into a bunch of other models.

So many things. So first, before I talk about DeepCNN, and it's precisely one way to embed the data that we process, because embedding is very, very, very important. For people listening, embedding is a way to transform the data that we a product or user timelines or website into vectors on which you're going to be able to do computation. And as you say, Nathan, the thing that is important is for those vectors capturing a lot of signal. That's very important. We want them to capture a lot of signals. And of course, that's something that the use that you mentioned, the fact that you have a way to encode all the data so that and be used for other tasks as inputs to other tasks is precisely one of the keys to being able to use the most of the data. And to answer your question of foundation models, we do have this program in Criteo working on building the foundation model. And that's very important. As you said, one of the things that we're very open about that, because first, it's not easy to build a foundation model. So that's why all the companies can talk about the fact that they are doing that because it's not easy. We have the data. And then there is a question of the roadmap, how you're going to do that, whether you want, from the get-go, you're targeting big models or you want models that are going to talk to one another. And our ways to approach that is precisely to have many foundation models, not that many, but 3 or 4 that are going to compute embeddings either on products or user timelines or et cetera. And then we're going to have a way to— we're going to make those embeddings that are going to be computed by those foundation models available to everyone in the company. And it's just starting to fruition today because people are using that as a hot start, a warm start to train models. And just so you know, one month ago, we had a hackathon And one of the— it's always a way for us to try things. And we made available those foundational model embeddings that were used by many other teams just to say, okay, now I'm going to learn something. I'm going to learn something. And I do not want to start from scratch. I know that somewhere in the company, there is gems that can be used, like those vectors that are going to contain a lot of information. And I'm not going to start from scratch. And getting back to what I said before, Regarding the manually computed features, it's totally automated. And we do have that and we do provide that. And we have— it's a huge— the CritterEyeLab, it's a huge project because the question is being able to refresh them, is being able to version them like a software, how you have a new version that is going to break up everything that is using them. And that's a big project. And we know that it's one of Cornerpiece that is going to feed all the AI models that we build.

Yeah, maybe just to build on that. So, you know, kind of as part of the technology group and, you know, Eliezer with his team in the AI lab. So one of the things that we empower is that feature and that product innovation across all of Criteo. So historically, you know, Criteo made its, you know, its first success as part of what we call retargeting, so lower funnel. But we're expanding our feature set. We're building new products in what we call the mid-funnel, so customer acquisition, product discovery, multichannel, so on open web, on social, CTV, and even in the LLMs. And so the ability to go from zero to performance, so to be able to deliver high amounts of return on ad spend, or ROAS, as the advertiser calls it, anything we can do that can accelerate that process, so the hot start, as Leva said, so those foundational models are exactly, instead of going from zero and building everything from scratch, you're already halfway there from a performance point of view. So that's why these models are so exciting for us.

And maybe I can explain one thing very quickly on how it could be used, and that's the basics. It's really the basics of how you can use it. So imagine you have products and you embed them with those foundation models because you have a lot of signals, you know that they were bought, etc. You do not know as a human how it was processed, but you learn something. You have a model like a vector, And then, for instance, you want to recommend another piece of— another type of product that is very— normally, the embeddings are well-defined, then it's very easy to have a similar product to be recommended, just looking at the similarity of the different vectors. But one thing that you can do is, for instance, you have some information about a user who has seen this product, this other product, this other product, and you have that for many users. Given one product, what you can do, given that you have this representation for one product, you also have those representations for users. You can look at all the closest users from one product, and it builds you an audience, the people that you can target. Just because you have similarities between the vectors that were computed for product and the ones that were computed for users. And conversely, you have a user and you can— it's encoded, let's say it's encoded by all the products that he saw or she saw, and you're going to have all the products that are around and you can recommend the products that are around. So that's recommendation. So it's very powerful to have those representations as long as they are semantically meaningful. And that's what we work at.

So everything, for the most part, it sounds like, is kind of precomputed. I mean, there's an interesting moment where that most recent action of the user has got to be critical. So presumably there's some marginal compute there that has to happen. But if I'm understanding correctly, you have sort of a base user profile, And base product encodings, and those are aligned such that at runtime it's an inner product of those 2 vectors.

Yeah.

So that basically everything is done except that comparison. But I guess there is probably some last-second update of the user profile as well based on where they are right now.

Yeah, yeah, totally, totally. And that's, uh, that's actually a challenge, how you recompute those embeddings, uh, like live, uh, time. And that's actually an engineering and technical problem to do that in a very precise way. And it means having other versions of the big models so that it can compute much faster, but still you do not hinder the way the similarities are going to be computed. But that's precisely where we put a lot of effort on. You just nailed it. That's the question. Or this small adjustment where you have to take into account the latest information, where you cannot rely on offline information, but doing that offline— online, sorry, is the key to what we do.

And we've done a lot of experimentation with that over the years, right, trying to find the right balance. If you have the perfect amount of time, you can come up with the perfect result, but sometimes the fast result is good enough, right, in terms of the time constraints you have, especially when You talk about the constraints of real-time bidding where you, you have to answer in milliseconds. So it's the best possible answer in the time constraints that you have.

Yeah, it's, that's really interesting. I mean, the architecture of this is, it has a, it does have a lot in common with the Stripe system. And they have a, you know, a lot of, I guess it makes sense cuz they have a lot of similar constraints in terms of they've gotta respond to, should you approve this transaction or not in an incredibly short amount of time.

Yeah, it's a great analogy.

How about on the agentic— I don't know if agentic is maybe even the right word yet, but in just like an OpenAI context, right, or in any sort of chat context where now we've got this additional signal of what is the context of the conversation that the user is having, does that also get treated essentially the same way? Do you get raw text and embed it yourselves, or they send you over an embedded form of the text, and then it becomes like another one of these already embedded inputs that goes into the decision-making process?

Okay, I can take it. So essentially, in Credo, we started to look at those conversational agents like 2 years ago or 3 years ago. We were interested in knowing what was And so I'm not going to talk about the partnerships because it depends and it's just building, etc. But in the way we think, and that's my job in the CREDI Lab, we are kind of envisioning all the different scenarios. One scenario is we have the conversation and work on that and we learn something. And like 2 years ago, there was a in Twilio who built kind of the conversational agent on Slack using the messages that were there about how to use the account strategies to try and answer some questions from clients. And he built from scratch an agent and a model trying to understand and recommend the answers to troubleshooting problems. So it was not recommending product. It was recommending a solution to a problem, but it's the same thing. And we also tried something where we actually had some kind of summary of the conversation, like just a vector, trying to see whether it was a signal that would help us to build another model and to see whether we could learn from that. It's, of course, less powerful than having conversation, but that's something that we've tried. We're trying. And that's something that actually we are, in terms of what I do in the Crio Lab, is we're trying to de-risk what we envision about what is going to come up. So, so far, in terms— and I'm going to let Jeremy talk about that— in terms of the partnerships, who we should work with, we're still investigating the right way and the right data exchange.

So it's still early days, right? So, you know, kind of, we're very, very happy with the partnership we have with OpenAI. They're a really great team to work with. We're both very much privacy-driven, right? We're both very much about user consent and so on. And so, you know, kind of, it's something that's a core principle of both companies and how, you know, can we handle that? So we're very much respectful of that. And OpenAI are exactly like that too. So, you know, kind of, we need to just have only the information we need to be able to show an advertisement at a given time. It's not going to be all the time because not every context it makes sense. So that's something that we're continuing to partner as we build out this network together.

And maybe just a last word on that. You have a team or a bunch of people working on what is called trustworthy machine learning. And I think that it has never been as important as today because precisely the type of information, the volume of information that people are ready to share is massive. And there is this question, and that's not just a research question anymore. People are like, it's very practical. And we do look at how we can— so trustworthiness is fighting against hallucination, being sure that it's privacy safe, being sure that sometimes we're not Or sometimes we do understand that the people we talk with are someone to be— that we should recommend to. So that's something that we're looking at. Maybe we are a bit like more on the upstream side of things, but we know that at some point, yeah, something that is going to be very strong either in terms of regulation, in terms of— I do not know, but we prefer to be on the same side, on the safe side, and we kind of prepare our weapons to be able to answer at the right time.

And I think, yeah, I totally agree. And I think the fact that we kind of were based in Europe, right? And where, you know, kind of there's a, definitely with things like GDPR, there's a lot of sensitivity around those. We've always had that as a core principle as we build out these products. For advertising to be truly useful, it has to be true and trustworthy, right? So the user, you know, we don't want to do anything that's creepy because if it's creepy, it won't work, right? It has to provide real utility. When the user sees it, they go, okay, that's interesting, that's engaging. If that's the case, then they're more likely to click on it. They're more likely to click on it, then it means that we're a better value provider for the advertiser.

Can you tell a little bit more about, I mean, I think probably a lot of people are surprised almost an hour into the conversation to hear the company has roots in Europe, is headquartered in Europe. You know, I tend to think of ad tech is a mostly American phenomenon. But how different is it really trying to operate in Europe versus the United States? And are there things that you actually do differently across jurisdictions based on restrictions that may exist in Europe? Or is it sort of the same approach globally? Like, I think people have a lot of sense that there's— you can't do AI in Europe is kind of what the I think the first-order summary would be from a lot of people. So if you think that's wrong, disabuse us of that notion.

So I would say that is fundamentally wrong. It's false because— so I'm from Ireland, and I moved over to France 11 and a half years ago. And the team here is just amazing. So the AI lab that Leva runs, the quality of the data scientists we have here is just off the charts. They're so, so talented, so, so dedicated to what they do. Where definitely being European-born helped us was growing up in that environment where we had to be very, very careful. It was a first principle for us in terms of how we handled user data. We've got this contract, this implied contract with the end users. In terms of how we operate, the US obviously is one of our largest markets, so we do operate many territories. Everywhere we work, we're very, very respectful of local regulations around data and what you can and can't do. And we always go the extra mile. We always are very, very careful to make sure that we're fully compliant and then some with respect to regulations because it's super, super important. And we're a very global company. We have offices in Ann Arbor, in Michigan, in Toronto, across other locations in Europe, from a development and an AI point of view. So that part can really tip in talent from all across the globe. Leva, maybe you could talk a bit about AI in Europe?

Yeah.

A French person.

I can. Yeah. One thing that I can say is that here in France, there's like a school of mathematics and computer science where people are very talented. They're really well trained into doing that. And it's not just AI, it's in general. And they are very good. They are very good people. And it's just the question of having very good people trained is kind of enough to innovate. It's not a question that— there's no question about France not innovating. And you can hear about Credo, but you heard about Mistral, you heard about many companies coming from France. And one of the things that is very important in Europe are actually— maybe I'm going to talk about France. And you know that part of my life I was a professor at university, so I know students as well. The students, they are very good. They do learn with enthusiasm. They do like And maybe that's something that is not necessarily the best thing, but in France, mathematics and computer science, the things that are very formal, is very important. And maybe people tend to forget that, but the roots of AI, the reason why you can craft and build models that are meant to answer some specific questions and you're capable of doing that, of crafting those models, relies on the ability to formalize, to model, and to find the right technical tools, either in computer science, in engineering, in more mathematical parts of things. And that's just the— that is important to innovate. And we do have that in France. And I do not know the rest of Europe. I do know France, but I do know that in France, we have that.

But also, one thing that struck me when I moved here was If you think about it, like say ancient mathematics was like a lot of it came from Greece, but modern mathematics, when you think about it, when you study it, I did engineering in university. You've got all these names, Laplace, Fourier, Fermat, all of these, Galois, which underpin the whole modern mathematics, which is the basis for machine learning and AI. And so France has really, really great done in that area, which companies like Criteo have massively benefited from.

I was quite impressed and surprised actually in looking up the Criteo AI Lab, how, first of all, just how many people are on the team. I think it's like, I think I saw like 50 faces on the website. And then specifically, I was like very surprised to see all the faces on the website because I was thinking, you know, geez, like I don't see too many AI companies in the US doing that. I think they're all afraid that if they put their names and faces on the website, that Zuckerberg's gonna come calling and the whole thing goes sideways because they all get offers they can't refuse. Not that I want to complicate your lives, but how are you thinking about that? How are you able to build and retain a team like this in the age of the Zuckerberg blank check?

So I think I'll go first, and, Leitha, you can kind of layer on top. Within Criteo, there is— obviously, I'm biased, but the culture here is amazing, right? I travel over an hour every day to come into the office because just there's so many cool people who are so good at what they do. And there's something really engaging about that. If you can give these people some really interesting problems to work on in an environment where they're surrounded by like-minded people, That's super, super engaging. And the fact that people, a lot of the people here have a long tenure, have been working there for quite a while, it creates that ecosystem that is very, very engaging. So even if we do have competition from the others, we often welcome that because it puts us on our toes and makes sure that we in the leadership have to make sure that we continue to make Credio a really, really great place to build your career. And we've been doing a great job of it for over 20 years now. So, you know, that's something we want to continue doing long into the future. Hey, Leva, maybe you can—

One of the things that is, we have research scientists, and one of the things that the research scientists do, they publish. And one of the things that is true across all research scientists, I think, and even in other companies, is that they have to to make their research reproducible. It has to be open, etc. And research and science like that. So that's something that you can see. It's kind of— you have to have a presence on the internet if you are a research scientist. That's more what I'm talking about. On the academic side of things, the people who are called scientists, usually they have their own website, they have their faces, and that's what we want them to have here in Criteo as well, to do research as other scientists from universities, from other companies. So that's the reason why. And then there is the thing that Jeremy shared is up to find challenging problems, challenging topics to work on that they can say, okay, it's good to be here to try and do science in Credo because the problems are very new, they are not easy, and it allows them to connect something that might be very upstream to something that can be also deployed, maybe not yet, not today, but in 1 year or 2 years. And that's very important.

And maybe one other thing just to build on that. Leva's team also works very close with a lot of the kind of the academic institutions in France and, you know, across Europe. And we sponsor PhD students. They'll come in, they'll work alongside the existing research team, they work on real-world projects, and so it gives them real concrete experience. And many of those PhD students actually become full-time employees. Not all do, but that's okay too. They publish their research, you know, which is good for their career. And this is a great way to keep that pipeline going. And it keeps us very well connected with the wider AI ecosystem as well.

Certainly, the ability to publish one's work in today's world is a differentiated part of the offer. I've got one episode that has been recorded 3+ months now with somebody at one of the, let's say, frontier companies that we just cannot get approved. And it's great stuff, brilliant work. We'll see if it ever sees the light of day. Maybe just one more beat on the regulatory environments. And I would separate here following the rules, which you've, you know, I think clearly stated that you're committed to doing, from advising on like what the rules should be. Do you think that There is like a— I guess, first of all, do you think there is a meaningful difference between the environments in Europe and the United States when it comes to like an individual's rights? I guess is one way to think about it, but maybe rights— what I really am trying to get at is like, who has it better? You know, do people in Europe actually have meaningfully better protection that I should envy, or do I get like meaningfully better ads that people in Europe should envy? Or is this all much ado about nothing? Who should be changing on what margins their rules to, you know, just to better serve their citizens? Do you have a point of view on that?

So we actually engage quite a lot with the authorities, you know, kind of in the EU with the different data protection offices, the DPOs in different parts of Europe, as well as in the US. And, you know, kind of we're very much advocating on behalf of the end user. to ensure that it's about transparency, it's about user consent, ensuring that users have a way to be able to opt in and opt out. And that's why we had the cookie consent message long before it was even regulatory. And we also kind of pushed through on ensuring that we live in a system where there's a fair value exchange, so that the user feels that they have proper use of their data and that they get value back. Right through free content, through free services, and so on. In a world where all of that disappears, right, then you take away all the value for the advertisers. So if about if an advertiser who's spending money to try and increase their sales, if they're not getting that value back, if their products are being shown in front of people who have no interest, then that does nothing for their business, and they're not going to spend. If they don't spend, then those Those people who provide that rich stream of services and of websites and of content, they have to monetize some other way. So then what they do is they have to put up paywalls. That's not in the interest of the end user. So as long as there's fair value exchange, as long as we're transparent about what we do, then I believe that's really important. So back to the question about US versus EU, to a large degree, a lot of that stuff's been equalized, right? A lot of those things like in California, there was a lot of changes to CCPA and CCPOR and so on. And they're very much inspired by each other. We try and build global solutions. So, you know, kind of what we've done in Europe, we use the same approach globally. So it's not like, you know, we try and be looser elsewhere. We really believe that by having that principle and by being born in Europe, it means that we have a solution that can work pretty much everywhere. Cool.

That's really interesting. Let's talk about creative a little bit. I think—

Gotta do it.

I'll invite you to help lead this part of the conversation. But it is interesting that we've made it this far and not really talked about creative. Obviously there's a lot of different formats. I don't even know today if— okay, I guess a couple ways at least I would want to come at this. One is like, I think the number I saw was Criteo has 17,000 advertisers. In our experience at Waymark, we've kind of seen often lack of creative is one of the biggest barriers to new advertisers signing up with, you know, a platform like Criteo or for that matter, Meta or Google or what have you. I wonder if that's, if you see that similarly, like is that a core barrier to market expansion? And then obviously we're in this moment of like cost of creative Of some quality is dropping precipitously. And I don't know if there is a dynamic layer to the creative. Like when we're doing these kind of matching auction prediction sort of things today, is creative an input to that decision-making? Do different— do advertisers like have multiple creatives that you're sort of scoring or embedding and kind of using to drive outcomes? Or is that still a frontier where perhaps because they don't have enough options, you can't do that in many cases? Or I don't know for what other reason, but it seems like in the future we sort of imagine like everything's gonna be highly personalised. You know, the ads are gonna be, you know, much more talking to us as individuals. It seems like you have the infrastructure to do that, but maybe just the creative is not there. What do you think is the future of creative?

Yeah, I think that's one of the most exciting areas in the whole generative AI space. Kind of the things that you can create with these next-generation models is just insane. And I think they have the possibility to create even more engaging advertisements, right? You know, you talked, Nathan, about the hyper-personalized. I think that's super, super powerful. And people will see it and they go, wow, okay, that's exactly the product I'm looking for. For, and it kind of makes it much, much more interesting and much more engaging for them. The other part as well is you touched on a point there with this technology, we've the ability to democratize that content creation. So before where, you know, kind of the mid to long tail would've been very much cut out of the picture 'cause it was just out of their reach, out of their means to be able to create that high quality content that is now becoming more and more accessible. Criteo recently launched a self-service product called Criteo Gold, and this is like really making it easier and easier for those advertisers who are very often sometimes much smaller to be able to create very engaging campaigns with really great creatives that will help drive and grow their business. That's super, super important. I mean, it's really, really enabled by platforms like Waymark, is a great partner of ours.

Thanks for the shout out. We love it.

On creative, so 2 things. First, there's something that we had been doing for years in Clio is dynamic creative with some templates, some visual assets, and we had to come up with a way to arrange them online. So that was something like you have Legos and you put them together and it's going to put you an ad, to craft you an ad. In terms of duration of those ads. And with generative AI, we're not at the level of, in terms of speed, there is no way unless you're ready to wait for 5 seconds for the page to load. We're not there. So we have to find a way again to balance things between something that is going to be done offline and online. So we're going to kind of The Legos that I talked about, the visual assets, they can be generated either by us or by our partners with generative AI. And then it's up to us to arrange that at runtime, depending— using your engine, actually, that is capable of arranging that online. But maybe at some point, one day, it's going to be very quick and maybe distribute something that is not going to be computed by us, but one day in the future, it's going to be computed by maybe your TV or your mobile phone online, and it's going to be very quick. But we're not there yet because you've tried probably to use some of those DALL-E and trying to build an image. It's not something that takes 10 years. It's far from that, but we're not there yet. At some point, maybe in 2 years, 3 years, we're going to have Uh, to be able to have that.

Yeah. Do you have a perspective? I'm interested in the level of personalization that you see the infrastructure supporting. I think, you know, there's of course the level of like the user experience that you've talked about a lot, which is super important. You don't want it to be super creepy, but when you think about the level of personalization that's at least possible infrastructurally, do you think it's gonna be literally at the individual level? We could tell you a story that is specific to you, or do you imagine it being more, audience and context level and kind of where do you see that living?

For me, I think, I think you're right. I think it'll be mostly more at the audience level, right? Because I think the point, going way too hyper-localized, I'm not sure there's huge utility for the end user, for the brand either. The brands and the advertisers will still want to retain levels of control over, you know, the look and feel and the, you know, kind of how their brand is showing up. So within those kind of parameters, I think it's mostly, you know, on an audience basis. I think that's kind of at that level, I can see it working down to hyper, you know, personalized, you know, where every single person sees a different thing. I'm not sure there's huge utility there from the point of view of driving more sales or getting more products in front of the end users.

Something that I'd That I would like to see actually at some point is to see those very efficient models to have sizes that allow them to be embedded in some devices, in glasses. And if at some point we arrive at that level, then maybe the computation is going to be shared. We're going to provide something at a level and maybe on the personal device, something that is not going to be seen by us, but just the personal device is going to tweak the things at the very end saying, actually, it was something that was proposed by, I don't know, us, but I know you, that's the device. And I have on that device something that is really personalized. And on that device, very privacy sure, privacy safe, we're going to record something, but just on the device. So maybe, so that's my dream.

Yeah, I love it. Super interesting.

Maybe there's going to be a share, something that is going to be shared in terms of computation. And maybe the personalization is not going to happen on our side, but at the end user side.

Yeah, I would agree with that. Where I could see this going is where the action is actually triggered by the end user, the consumer, right? And so the good use case is, you know, kind of, okay, so I get invited to a wedding and the south of France, say, in the summer, I wanna know what looks good. So I can ask, you know, the AI, can you recommend some outfits? And then you can see a virtual try-on, stuff like that. The usual example as well is you wanna see how something would look in your home. So that's very much, that's engaged by the user. So they're kicking it off themselves. And because they've done that, then it's not creepy. It's their action that's initiated this.

Yeah.

And that for me is actually, where hyper-personalization really comes to life and where that really makes sense. And that's great. And again, go back to what I talked about earlier, it's the utility. It's trustful, it's useful, and it's providing value, and it could help give a far better experience for all involved.

Yeah, that's super interesting, and it seems like it fits with what we were talking about earlier too, like in a chat or agent context where we're actually getting way more comfortable with what would be creepy in any other context, letting it do that personalization for you is super, super—

If you're asking for it, then I think that's perfectly cool.

Yeah. Yeah. Cool. Super interesting. One other thing I'm interested in kind of along, it's a different angle on the personalization, but contextualization. I know CTV isn't your historical bread and butter, but I know a growing concern And I think one of the best documented performance gains is when the creative of the advertising matches the creative of the context. I'm sure that's also true in some other contexts, but really true on the TV. Another place where you can imagine lots and lots of different variations that match the tone and style of the particular— not only the genre or the movie, but the very specific moment in the movie. Is that stuff that you— you know, have started to think about and bake into the way that you run your models?

Yeah, I think for me, um, it has to be, you know, what's super important about advertising is it should not be intrusive to the user experience, right? It has to kind of feel kind of seamless, and something that's getting in the way of the user experience of the content, and that also applies to websites as much as it does to connected TV. And, you know, I think we've all seen those demos of where you get advertisements inserted into the content, whether it's sports or it's your favorite sitcom and so on. I don't know if we're quite ready for something like that yet. I think you have to have something where the user feels that it's relevant, that it's not interfering with their experience of the content. With video and connected TV, there's pre-roll, there's mid-roll, there's post-roll. Which is, you know, kind of a different way of experiencing it. So as long as it's not intrusive, I think the users will be open. I think that's kind of more of the guiding principle there.

Oh, makes sense. I did have one other question that I'm interested in, which is, so I think this is a really developing moment right now, as for the first time ever, creative can really be technically powered, where it's always had a very human-in-the-loop How are you guys thinking about, as a technical organization, what your role in the creative is, not only right now, but over these next few years?

I think that, so as of today, the way we do approach those creatives so that we are sure that the quality of the thing that we provide is to build our own creative models because they're very costly, they are very expensive, et cetera. But I think that It's something that is going to be, yeah, of course it's going to be key. I don't know, it's not my own turf. So I think that it's that, yeah, we have a team that is dedicated to do that. They're programmed, they're like not a 1-year program, but like 2-year, 3-year program to integrate those creatives. But they used to be in the Crypto Lab, but now they're not anymore. So maybe I'm going to Erndit on that one.

Yeah, so for me, I think ultimately, you know, kind of there's a few different constraints. So for example, brand guidelines. So the brands themselves, the original advertiser or the retailer, they've got their own kind of look and feel and the way that they want these things to show up as well. Ultimately, we want to build advertising that works, advertising that brings value for the advertisers so they feel that their money is well spent with us. And so, you know, kind of, I think within those parameters, we try many different things. So Leva talked about, you know, kind of the way we did DCL before, which was really one of the great ingredients of our initial success. You layer on top the generative AI capabilities on top of that, and the possibilities are endless. It's really, really— the potential is just amazing.

Yeah. One, maybe just a word that you used, Alex, and it's true for everything that is going on with AI today. is human in the loop.

Yeah.

That's the key, actually, not just for creative, but for everything that we've been talking about. And the big challenge is being sure that you put the human at the point. That's very, very important. You should not strip the user away from his right to decide. If you do that, if you automate everything, it's not going to work. It's going to be a problem in terms of liability, responsibility, et cetera, et cetera. But at the same time, you're going to feel as a human being allowed to make any decision. So just so you know, the question that we have every day when we build a model is, where is the human? How do we learn so that— and how do we learn the model so that the human has still a place and we have to spot the right place? And just missing that could be very detrimental to the project that we have. And it's true for creatives, for the bidding models, it's true for the agent stuff. et cetera. So that's the big thing that is very important to actually not just us, but everyone building new tools. It's the human. And that's why those conversational agents work so well. It's still heavily relying on human interaction. So that's very important.

Well, here's a small one, but a curious one. And it is kind of relevant to our business. So it's top of mind for that reason. But also in general, right, we've been talking about like, All these different touchpoints. Where are we today on cross-device understanding of who somebody is? I could imagine that that could be very algorithmic and sort of deterministic, but it occurs to me that maybe that's another AI question in today's world. Who is one person and who's a different person when some devices are shared and some are on mobile networks and all the complications there? That was supposed to be the short one, the way. So I'm just focusing that for— and I can run a minute long, but I don't want to keep you guys long if you have something next.

Yeah. Yeah.

So there's a few different aspects to it. So again, it goes back to user consent, right? So the user needs to be okay with this because, you know, there's shared Wi-Fi at home and all this kind of stuff, and you want to make sure that you're showing the advertisement to the right household member. So there's different ways. So, you know, kind of there's the Wi-Fi No, but there's the network that you're connected to. There's also if you're logged in on multiple devices and that, you know, kind of, for example, on social or on other ways. So that can be used deterministically to say, yes, it is the same person on multiple devices. The probabilistic way can be used for certain things, but only in the kind of very, very untargeted kind of way, because if somebody has opted out of advertising, if they've gone through that, then you don't want to reshow them advertising even Uh, through a different way. So that's something that is something we always kind of keep front of mind.

Cool.

All right.

Last big one for me, zooming out kind of as, as far as possible. I see, I mean, in general with this whole AI wave, there's like dramatic uncertainty about what's going to happen, right? How powerful the AIs are going to get and how disruptive it's going to be. And, you know, do we need a whole new social Contract, et cetera. Specifically in the advertising space, I feel like I see 2 trends that kind of counter each other. One is that if all goes well, we should all be a lot richer and the value of time should go up. And that would be really good for the advertising market in the sense that fundamentally you're competing to have some little share of people's time. And the richer they are, the more valuable their time is, the more that's going to cost. So that's That's an advantage, presumably. But then the other thing that people also sort of see is, like, maybe search costs and matching costs could really drop in a lot of ways if we all have our AIs going out and vetting much broader portions of the world than we previously could. Right. Today, I'm like, I can only evaluate so many shoes, but maybe in the AI era, I could have an AI that goes out and really evaluates in a comprehensive way every possible shoe. And therefore my product decision-making is less about who was willing to bid on my time and more about how much time was I willing to have my AI invest on my behalf to go out and figure out what to do. I don't know, that's more of a prompt, I guess, than a question. But what do you think of those trends? Are there other big trends that you see as kind of being huge factors? And where do you think we are in advertising in, say, 5 years' time or the singularity, whichever comes first?

I'm not sure about the singularity. I have my doubts about that one. So for me, where I see this new generation of AI really kind of helping to, you know, kind of increase the amount of value to the end user, you mentioned time, Nathan, you know, kind of maybe there's value as well, is for that kind of product that you're looking for, right? If you're kind of a very specific thing in mind, you know, where you don't know where to get it, You're able to then query these LLMs or whatever interface, you know, whatever LLMs become, partnering with that really kind of rich commerce data. You say, here's my criteria. I'm, you know, I don't care about the price. I just want the best product possible. And, you know, you explain what it is you want. And connecting all of these systems has the possibility to give a really, really, you know, great result. When you get that great result, then you're quite happy to buy it. I think that's the kind of utility we're helping people discover, and not only just the product they're looking for, but what goes with it, how you can actually enhance it, and so on. And that kind of— when you see the way that they're working today and where they're going and how fast they're improving, I think that's incredibly exciting, right? And it can provide a really, really great consumer experience for the retailers to be able to engage in that. That allows them to get their products in front of the end users in a way that they couldn't do before, you know, because some of the advertising before, a lot of it was guesswork. Now it becomes really, really focused, and you're really kind of making sure that you're getting all the information that the end user needs to be able to buy that product, whether it's the shoes you're looking for or that new piece of tech that you kind of, you want to invest in. I think it really, really gives a very enhanced way for the end users Yeah, just first, congratulations on the prompt because it's very deep.

One of the things I would say is that there is something that is going to happen so far in the way people are exposed to advertising. You do not— you might make the choice not to be exposed. But you are exposed in a way. I think that with the advances in AI, those companions or those assistants that you're going to have with all those LLMs, platforms, etc., maybe at some point you're going to be wanting advertising. The exchange is going to change a bit because the advertising is also accessing or discovering the right products, etc. And maybe the things are going to happen behind the scenes. Like, you're going to turn a knob, say to your assistant, okay, I want to go or to see shoes, but I want you as an assistant to be exposed or to look at 10 or 12 or 100 different shoes, and I want you to select among those 12 or those 100 shoes, 6 of them, but I want to be exposed to 6 of them. And I want to be exposed to advertising. And I want to choose from that. So the— there is a new actor that there might be a new actor, a new intermediary in the, in the advertising business, and people might be wanting to be exposed so that again, to have the choice as a human to say, okay, I want to be Because having the choice of different shoes is having advertising in front of you. So maybe they will be saying, I want to see, I want a trip to Greece. I want 5 different trips. Show me 5 different trips, but I want to be shown 5 different trips, not more, not less. And then I'm going to choose. So maybe it's going to change something in the way advertising is going to be experienced. Just because the quality and the ability of the system to filter and to select them. So maybe there is a change like that. So you say in 5 years, maybe we'll be in that situation where we're going to say, okay, show me some advertising. So yeah.

Yeah, it's like the collapse of search and advertising. If the advertising is good enough, it might be better than search.

Yeah, yeah, exactly.

Interesting.

Cool.

Well, that's a, a great note to end on. Alex Persky Stern, Jeremy Gill, and Leva Ralaivola, thank you for being part of the Cognitive Revolution.

Great to be here. Thanks, Nathan.

We choose big math, we move automatic. Mentalist in the middle, reading smoke and mirrors. No name, no face, just a cookie getting clearer. 17,000 shelves, fresh catalog a river. LLM knows the world, but the price we deliver La place, la grunge, ferme en gala. Modern math on modern drums. Ooh la la, voilà, sparse 2 to the 12th. Now a dense hot start. Deepening and begin. Know the shape of your card. Billions of bits a day, milliseconds on the clock. 300 miles an hour, same car, never stop. We the mentalists in the middle. Funnel, open web, CTV, and more. Trustworthy machinery, privacy on deck. GDPR in the DNA, not a line we check. And Harvard to Toronto, Paris to the Bay, off the charts. The scientists, that's how we play. Show me 5 different trips, let the agent decide. Behind the scenes on your device, personalization.

If you're finding value in the show, we'd appreciate it if you take a moment to share with friends. Post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts which is now part of a16z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at aipodcast.ing. And thank you to everyone who listens for being part of the cognitive revolution.