Milliseconds to Match: Criteo's AdTech AI & the Future of Commerce w/ Diarmuid Gill & Liva Ralaivola

Hello, and welcome back to The Cognitive Revolution. Today, my guests are Diarmuid Gill and Liva Ralaivola, CTO and VP of Research and Head of the AI Lab at Criteo, the advertising technology company that powers much of the personalized advertising we experience on the open internet. I'm also joined by my longtime friend and teammate Alex Persky-Stern, who took over for me as CEO of Waymark some 3 years ago and has since formed a partnership with Criteo that brings Waymark's AI-powered commercial-creation product to Criteo advertisers.

We begin with an explanation of how modern digital advertising works and the value that it creates for society. Personally, I tend to emphasize that without the commercial recommendation systems that allow businesses to affordably reach their target customers, many of the long-tail small businesses and niche products that we enjoy today simply wouldn't be viable at all. Diarmuid and Liva, for their part, focus on how ad tech delivers more relevant, engaging experiences and supports free access to information, while also emphasizing how easy it is for individuals to opt out of personalization systems.

From there, we dive into how it all works. Criteo has been in business for more than 20 years, and while its AI techniques have naturally evolved with the field—most fundamentally, from the earlier era of handcrafted features to the modern era of deep learning—the one constant has been the need for incredible speed. From the time your browser requests a web page, Criteo has just milliseconds to locate your profile among the billion or so in its system and, in light of what you're doing right now, decide which one of many millions of products to recommend and how much to bid in a real-time auction.

It's a deeply challenging problem that requires lots of pre-computing, but the upshot is that they've developed a highly modular system powered by foundation models that supports prolific experimentation on top of cached user and product embeddings. Beyond the core technology, we also discuss Criteo's new partnership with OpenAI, which, though still in its infancy, they expect will complement ChatGPT's broad world knowledge with accurate, real-time product inventory information.

They tell the story of the company's European roots and share their commitment to privacy, their sense that European compliance burdens are overstated, their decision to use the same EU-compliant technology stack globally, and their passionate belief in the European AI ecosystem and talent pool. They also explain why they're confident enough in their moats to publish a lot of their research and how this helps them attract and retain talent well enough that they're still comfortable publishing the AI Lab's full 50-person roster on their website.

We trade ideas regarding the role that generative AI will play in the expansion of the advertising market and the evolution of personalized creative. They also share their admittedly speculative thoughts about how the fundamental value exchange of advertising might change as human time becomes more valuable and AI agents take on more product-discovery and research work.

Overall, I think this episode is both an informative look at how modern AI techniques are being used to make high-value commercial recommendations under extreme constraints and a useful corrective for those who deny the ways in which cutting-edge advertising enriches modern life. With that, I hope you enjoy my conversation with Diarmuid Gill and Liva Ralaivola of Criteo.

Diarmuid Gill, CTO, and Liva Ralaivola, VP of Research and Head of the AI Lab at Criteo, welcome to The Cognitive Revolution.

Thank you, Nathan. Pleasure to be here.

Hi. Excited for this conversation.

I'm also excited to welcome my longtime friend and teammate Alex Persky-Stern, who's the CEO of Waymark, which we had been building together for a number of years before he took over for me as CEO a few years back. Longtime listeners have heard many asides about Waymark, and Alex is the guy running the show there now.

I'm happy to have Alex here today because Criteo is obviously in the advertising business and bringing a lot of AI to the advertising business in various ways. Under Alex's leadership, Waymark has partnered with Criteo to provide some creative solutions as part of that whole package and go to market together. There's lots to get into.

I wanted to start with something that I saw recently that caught my attention and get your take on it. I'm sure you've seen this: a Bernie Sanders sit-down across the table from, I believe, Claude, having a voice conversation. It was a bit of a strange tone. It felt a little dated to me in some ways, but the subject of the conversation was Bernie saying to the AI, “What do you think Americans need to know about how their data is being collected, how companies are profiling them, and how that's all being used?”

It had an ominous overtone to the whole thing. I think there's probably still a lot of misconceptions or misunderstandings out there about this, but this is a significant part of the business that Criteo is in. I would love to hear from you guys, as folks who have built it and are doing it today: What do you think Americans need to know about how their data is being collected, how it's being understood, and how it's being used? What's the upside of that to businesses and consumers, and maybe the downside as well?

Yeah, it's a great question, and it's something that we in the industry probably need to do a better job of explaining and demystifying. For me, I think it all boils down to transparency: explaining to users what data is collected.

For example, within Criteo, we don't collect any personal information. It's really a random, anonymous ID. Then there are some things around what products people are interested in—what they've seen, what they like, what they don't like, and so on.

For me, it's all about a value exchange, right? Relevancy. A system that knows nothing about you is going to show you random stuff that's irrelevant, and the brain has a really great way of filtering out irrelevant stuff. Whereas something that's truly interesting for you is way more engaging and way more resonant. For a user, that creates a better experience.

Also, I think one of the great things about advertising is that it's very much the lubricant that keeps the internet open and free. It allows service providers to keep their services from being behind paywalls. There's great utility in that. Advertising is what keeps that going. It's the revenue that content and service providers get that allows them to provide those great services to end users.

Providing transparency so users can actually see what's happening and giving them the ability to opt out is also part and parcel of it. Criteo was a pioneer with the AdChoices icon. Someone can click it, see why they saw this ad, and get the ability to opt out. Once you do that, I think it provides great value to all the participants.

With Criteo, we have very little information and very few cues, and we try to detect what is going to be most relevant for each consumer and each end user. In this value exchange, everyone is going to be a happy camper.

As a middle party, we are, hands down, the ones who have the least data and the most challenging tasks in terms of AI to solve. That's why I'm here, because the challenge in terms of machine learning and AI is really a big one, and it's the most interesting one.

Could we do a double-click on a couple of aspects of that? One is that, if I were to open up my file—I'm not even sure if that's quite the right way to think about it—I'd be interested to know what's in there.

I've occasionally clicked this sort of AdChoices thing and seen, “Oh, you're seeing this ad because you're interested in skin care,” and I'm like, “Okay, my wife got me this one.” But I don't really know. That's kind of a high-level summary statement of why I'm seeing it. I don't know exactly what's under the hood.

I'm also confused about the cookies. Of course, we go to websites all the time and get this pop-up that says, “Accept cookies” or “Don't accept cookies.” What's going on with those cookies?

I know there was a big change to the industry, and I think it was driven by Apple a few years ago. Maybe it was driven by other parties as well, where the way in which information is gathered and the nature of that information were changed. I think there were some winners and losers from that. I'm not quite sure how that really shook out or if we went back to essentially the status quo ante before those changes were made.

I'd love to hear a little more concrete description of what the data is, and then what the machine-learning layer that sits on top of that data looks like to make sense of it. Obviously, that data becomes the inputs, but I always like to get down to the brass tacks. What are the inputs and outputs of the models, so we can really understand what it is that the AIs are doing for us?

Sure. There are a couple of different ways that I think it works. First of all, when you arrive on, say, a retailer or a brand website, they can use technology like tracking pixels to create a record on the computer called a cookie, with a random ID. The ID doesn't have any personally identifiable information.

Then, as you continue browsing, if you look at a product, when you leave that website and go browsing the web, they can know that you've shown an interest in that product. They can show you ads for that product specifically—the same one you've seen.

Alternatively, what could also work is if, for example, you took a look at a mobile phone. That could have you assigned to a group of people who are tech enthusiasts, right? The type of phone that you look at could be interesting, too. iPhone users have a different profile from Android users, from whatever else. You could be part of a wider audience that could just be seen as Apple enthusiasts or tech enthusiasts, and so on.

Then, when an opportunity comes—when you’re browsing the web and looking at the web, the internet, whatever—advertisers have an opportunity to bid, to pay the website owner to show an advertisement in front of you. Based on the information they have about your previous browsing history, they can decide whether they want to take this opportunity to show you the same product or equivalent products. Maybe I’ll hand over to Liva to say how you actually do that bid and how you decide whether to show an ad or not.

Yeah, precisely. One of the very important things is being capable of evaluating the expected revenue of a placement and knowing whether, if you place an advertisement in that placement, there is a high probability that it will be clicked on or not. You have to evaluate that. In order to do that, you’re going to use machine-learning and AI models that are going to evaluate whether a placement, given a product that we can put on it, is going to bring revenue. For that, we collect all the data that Diarmuid talked about, and there is a huge machinery that we put in place in order to learn from that data.

If I had to summarize the type of problem we’re solving, it’s: Should we bid or should we not bid on that placement? We learn a classifier from that. Of course, there’s this question about utility and the trade-offs that exist. If you want to be very precise in terms of evaluating the value of a placement, then you have to use very sophisticated models. You’ve probably heard about deep-learning models. The more sophisticated the models are, the less easy it is to understand what they have computed.

So, there’s this trade-off. In the industry, and here, we use those deep-learning models in order to assess whether a placement is good and whether a product is going to be relevant for you. It means that, in a way, what we have gained in terms of precision and relevancy, we have to make up for in terms of explainability. As we talked about a bit before we started, that’s a big topic in scientific AI research: providing explainability for those models that are doing crazy stuff. That’s one of the things that we’re looking at as well, but it’s not easy to have both high utility and high explainability.

Jumping in, actually, I have a question I’m interested in here. The idea of the user profile—who this person is and what this person might be interested in—is obviously super core. One thing that I think is really interesting is that you guys have this OpenAI partnership, which is super cool and very new, so some of these answers might not exist yet. People talk a lot about how the queries are much richer in the context of AI and chat, but something that I haven’t heard people talking about is whether the profiles are meaningfully different.

If I’m working with Claude or an agent, it knows a lot about me. Is that starting to change how we understand the user? Where do you see that going?

So many things. I think the very first thing you said is that there are still a lot of things to unpack and uncover. We are precisely at this stage because, of course, there are many questions about privacy and data. So far, there is no answer yet.

The one thing that I can answer, though, is that those conversational agents provide a new interface. Before, you essentially just had websites or some apps that you could use, and now you have conversational agents. One thing that is very important is that those models, like ChatGPT and so on, are very good at general reasoning. They can do some recommendation, and in some ways you can think that they’re going to be able to—if you ask, “Okay, I would like to buy shoes,” they’re going to propose shoes that are relevant.

But one of the things that we do is commerce data, which tells us what exactly people are interested in. The big challenge that we have today is precisely to make the two models work together: the LLM models behind all those conversational agents with all the models that we have in Criteo, which are capable of providing very accurate commerce information. The technical challenge is precisely to merge the two.

From the LLM side, they’re going to have some information that is going to be encoded, but what is precisely necessary is not to have this information, but rather to know and see how those models can enhance the commerce models that we’ve built for years. That’s where we sit as of today.

Yeah, I think that’s exactly right. The thing about the LLMs—and it’s amazing technology; we’re super, super impressed by the power of all of them, I think everyone is—is that when those companies train their model, it is true and accurate at that moment in time. Commerce data is actually way, way more dynamic.

For example, they would not be able to know that there are flash prices, around Black Friday and so on, where things change very rapidly. They also wouldn’t know, for example, things like stockouts. The way that they gather their information is by doing this massive crawling of the internet, and then, at the point in time when they’ve updated their model, it very quickly starts becoming stale, at least from the product point of view.

Criteo has this massive network of 17,000 retailers. We ingest their product data on a daily basis, sometimes multiple times a day, and it means that we always have access to fresh data. Like Liva said, we did this hybrid architecture where an LLM, in partnership with technology provided by Criteo, can ensure that when a user asks for a product, they not only get all the richness that an LLM can provide, but also accurate, up-to-date information.

From a user point of view, it’s a very bad experience when you search for a product, click through, and the product is at a different price, is out of stock, or isn’t what you were thinking about. That’s why that hybrid architecture makes so much sense.

So today, just to make sure I’m getting it, the process is ultimately pretty similar to what you’d have on the open web. When you’re in the chat interface, maybe with a richer query, AI is being inserted in a whole bunch of other different places in the stack, but that applies really across all surfaces.

I would say yes and no. In fact, I think where these tools, like the LLMs, have the ability to elevate the whole experience is in the area of product discovery. If you’re in the market for a new product and you’re trying to dig out a lot more information, I think for the first time ever we have the ability to provide end users with the same experience you get when you go into a store and you’ve got this really kick-ass sales assistant.

Mm-hmm.

That assistant only cares about giving you a good experience, can answer your questions, knows the full catalog, is able to tell you the good and the bad of each product, and leaves you with the experience that, when you walk out of the store, you feel like that person has really answered what you’re looking for.

The LLMs, in coordination with accurate product information, can provide the same experience. You can query, ask extra questions, and drill down deeply. It doesn’t get tired, it doesn’t get bored, and it’s always giving you real-time, accurate information.

There’s something else that has been changing: how you’re going to connect the tools and the things that we provide to those LLMs. You’ve probably heard of the agentic era and the fact that we have MCPs, those protocols that are going to make the use of already-made tools almost transparent. That makes it easier to combine those LLMs with what we provide.

It’s technical, so maybe nobody cares about that, but in terms of deploying something, it has been a lot easier. Before, you had to adapt to each surface, to each website, et cetera. But now, with those protocols that are coming up, it has been made easier. It’s our duty to make sure that we’re compliant with those protocols, and that’s actually what we do. It allows us to interface all the tools that we’ve built over the years and make them available.

Yeah, great point.

Can I dig in a little bit more on the core models that you guys are using to make predictions? I'd love to understand the architecture of this better. I think, for calibration, anybody who's listening to this feed is going to have a conversational familiarity, at least, with how large language models work. So we know that they're generating a token at a time. We know that the inputs get embedded, and we know the mechanics of the forward pass and all that stuff, right? And we know it's autoregressive.

This strikes me as a very different world. I don't have nearly as much intuition for what the models are that are driving these things. I do know that they have to be a lot faster because the ad has to show up really quickly on the page. And then I know also that there's a pretty challenging matching problem in there somewhere because I've got millions of—you've got—we as a society collectively have millions of these profiles of individuals, and then also, as you said, tens of thousands of advertisers.

I don't know how much precomputing is done or whatever, but it has to happen pretty quickly on the load of a page. So could we break down how big these models are? What do the inputs look like? You could imagine something very, very large and a sort of very sparse set of inputs, but I guess it doesn't seem plausible that it's like, "Here's all the websites and here's which ones this user visited," right? That doesn't seem like it works.

So there's got to be some sort of tokenization or something that's bringing the user profile into a manageable-size state so that it can be used as an input. I'm not even sure if I'm quite asking the right questions here, so tell me what this looks like under the hood.

Yes, maybe I can take a quick stab at this, and then Liva can take it down into more details. Liva actually referenced this earlier. Every single time that we get an opportunity to show an ad, that opportunity actually goes to multiple different ad tech providers, who are all acting as delegates on behalf of the actual advertiser themselves, whether they're brands or advertisers.

The amount that we bid is based on how valuable that opportunity is to the advertiser—effectively, how likely the user is to click on that ad and go back to the website and buy the product. The way we evaluate that is, through the mechanism we talked about earlier, we see what products the users are interested in, what they've looked at, what they click through, what they've seen, what they buy, what they don't buy, and so on.

As the display opportunity comes up, we see the ID that we mentioned in the cookie, and then we take a look at all of the different products that person has seen or whatever audience segments they belong to. Based on all the different features we put into the model—the products, the previous purchase history, the context of the website, the device they're on, and a couple of other things—there are probably, although I'm not sure, 150 different features we can take in.

Each of those goes into the calculation as part of this massive equation, which tells us the likelihood that person is to click, the likelihood they are to click through to the website, and eventually make a purchase. All of that comes out to a value, which we bid. If we win the opportunity, then we have to say, well, which products do we show and how do we do all of that kind of stuff?

All of that process gets done in milliseconds because we use a lot of caching. We've trained the models offline, and then the inference happens in real time at really low latency.

One thing that's important regarding all the data that we have—like the websites, the products that were shown, clicks, and so on—is the model. The question, as I said before, is: Let's reduce it to a classification, a binary classification problem.

One of the main things for people who have tried to do some machine learning is how you're going to encode and how you're going to represent the data. So I'm going to do, I think, 2 steps. The first one is going to be talking about the legacy, old models that we used to have, where we are now, and where we've been.

Before, there was this question about how all the products and the websites that you visited had to be encoded. You have to encode them so that the vector you're going to use to represent all that past information carries a meaning. If you just encode them in a silly way, you're going to lose a lot of information.

Before, because of the speed of computation and the type of model, it was a sparse representation—a very huge vector with 2 to the 12 inputs, with 1s and 0s—because you can do very fast computations on those sparse vectors. That was a way to represent the data that we used to have, and we just learned from that vector what is called a logistic regression model.

It's a linear model. You can think of just 1 neuron with a lot of inputs coming in, if you have the neural analogy in mind. We used to have that, and we learned the model. It was very fast, even though it was sparse. There are many libraries to do this on sparse matrices and sparse vectors.

But then there was something that was very manual in building the features. One of the reasons why, for instance, the Criteo AI Lab was created was to say, "Okay, maybe it's not sustainable to have to craft new features and to think about how we're going to represent data each time, because the cookies can change the information that we have, and, for instance, with LLMs, it's going to change. So how can we proceed with more modern techniques?"

It was the intent and the goal of the Criteo AI Lab to bring deep learning. It was created in 2018, and it was precisely with the objective of saying, "Okay, let's go to the next level—not have handcrafted features, but rather have them computed from the data."

Before, we had 2 to the 12 or 2 to the 20, depending on the encoding, in sparse vectors. Now, essentially, we have something like between 200 and 1,000 features that are automatically computed by one of the proprietary algorithms that we have, which is called DeepKNN. It computes deep learning features, on top of which we learn some other models that are going to do those classification tasks.

The essential thing to understand is that we went from 2 to the 12 sparse vectors to something that's a couple hundred features. And now we're at the next level again. With all the things that are available with these models that you can download, we're going to the next level, trying to be even more adaptable to the data that we're going to process. That's something that's going on as we speak.

So that's, in a nutshell, the models that we have. The thing that's very important—and the challenge of everything—is that everything happens in milliseconds. What we have as a challenge is not only to be accurate, but also to be fast. That's a nice challenge that we have.

To be accurate, to be fast, and then also to do it billions of times a day, right? At huge levels of stability and reliability.

And then the other thing that's a great challenge within our system is when you have Black Friday and Cyber Monday, the busiest period of the year. Imagine any world where your piece of technology is able to run at 300% of what you normally run.

Imagine taking your car and running it for 1 week at 300 miles an hour, and then going back to normal for the rest of the year. It's the same car, the same machinery, and it has to perform exactly the same way.

That sounds like a challenge. Engineers behind this—my hat goes off to them. It's the work they do.

One thing for the people listening, and for you, is that one thing that's important for us is also to share how we do our machine learning. You can access blogs that explain the DeepKNN methodology and that explain a little something relevant for the retail media business.

We want to go even deeper. We have more scientific papers because we have researchers doing AI science and publishing at conferences. Everything ties up together, and if you want to do more, if you want to dig deeper, if you want to know the sizes of the models, how we train them, and what the losses are that we use, you have a bunch of articles online that you can download and read to have more information.

Can we talk a little bit more about the architecture of the models and what gets precomputed? This probably is, to some degree, in the papers that you've put out. I'm thinking back to an earlier episode I did with the woman who leads AI at Stripe. They had a pretty interesting strategy that I imagine you might have some similarities to, where they train this foundation model for payments. It's a huge model trained on—I don't know—1 trillion payments or something, a massive data set that they have.

It strikes me that they can be pretty open about it because they have the data, everything's flowing through them, and that's not about to change. So they can afford to be fairly open about techniques, and you guys might be in a similar spot, where having the network is the moat, and so you can afford to tell more of the techniques than maybe other companies could.

One thing I thought was really interesting that they did with this foundation model for payments was, instead of trying to use that model for all the different tasks that they have within the company—and there are many—they use the embeddings of a given payment as the input to other models that may have other inputs as well. But by modularizing things this way, they were able to both amortize the cost of this model across a ton of different use cases without necessarily having to anticipate what those use cases would be, and also do something that really allows the developers across the company a lot of freedom to say, “Okay, now I know I have this really rich signal that I can kind of treat as a black box, but if I bring that into whatever machine learning task I'm trying to work on right now, they're just seeing dramatically better results across the board because that signal is so rich.”

I hadn't heard that in too many other places, but I wonder if you guys have a similar structure where there are models whose outputs, encodings, or embeddings feed into a bunch of other models.

All right, so many things. First, before I talk about DeepKNN, this is precisely one way to embed the data that we process, whether it’s a product, user timelines, or a website. Embedding is a way to transform that data into vectors on which you're going to be able to do computation. As you say, the important thing is for those vectors to capture a lot of signal. That's very important. We want them to capture a lot of signal.

Of course, the use case that you mentioned—the fact that you have a way to encode all the data so that it can be used for other tasks as input to other tasks—is precisely one of the keys to being able to make the most of the data. To answer your question about foundation models, we do have this program at Criteo working on building foundation models, and that's very important.

As you said, one thing about that is that, first, it's not easy to build a foundation model. That's why companies can talk about the fact that they're doing it. I mean, it's not easy. We have the data. Then there is a question of the roadmap: whether from the get-go you're targeting a big model, or you want models that are going to talk to one another.

Our way to approach that is precisely to have many foundation models—not that many, but 3 or 4—that are going to compute embeddings for our products, user timelines, and so on. Then we're going to have a way to make those embeddings, computed by those foundation models, available to everyone in the company.

It's just starting today, because people are using that as a hot start, a warm start, when training models. Just so you know, 1 month ago we had a hackathon. It's always a way for us to try things, and we made those foundation model embeddings available to many other teams, just to say, “Okay, now I'm going to learn something, and I do not want to start from scratch. I know that somewhere in the company there are gems that can be used, like those vectors that are going to contain a lot of information. I'm not going to start from scratch.”

Getting back to what I said before regarding the manually computed features, this is totally automated, and we do provide that. The Criteo AI Lab is a huge project, because the question is being able to refresh and version them like software. When you have a new version, it is going to break everything that is using them. That's a big project, and we know that it's the cornerstone piece that is going to feed all the AI models that we build.

Yeah, maybe just to build on that: as part of the technology group, with Criteo's team in the AI Lab, one of the things that we empower is feature and product innovation across all of Criteo. Historically, Criteo had its first success as part of what we call retargeting, so lower funnel, but we're expanding our feature set.

We're building new products in what we call the mid-funnel, so customer acquisition and product discovery; multi-channel, on the open web, on social, CTV, and even in the LLMs. The ability to go from zero to performance—to be able to deliver high amounts of return on ad spend, or ROAS, as advertisers call it—is important. Anything we can do to accelerate that process is important.

So the head start, as Liva said: those foundation models do exactly that. Instead of going from zero and building everything from scratch, you're already halfway there from a performance point of view. That's why these models are so exciting for us.

And maybe, very quickly, 1 thing about how we could use them—and this is really the basics—is how we can use them. Imagine you have a product and you embed it with those foundation models, because you have a lot of signals. You know that it was bought, and so on. You do not know how it was processed, but you learn something: you have a representation in the form of a vector.

For instance, you want to recommend another type of product. Normally, the embeddings are well-defined, so it's very easy to have a similar product recommended just by looking at the similarity of the different vectors. But one thing that you can do is, for instance, you have some information about a user who has seen this product, this other product, and this other product, and you have that for many users.

Given 1 product, what you can do—given that you have these representations for 1 product, you also have those representations for users—is look at all the users closest to that product. It builds an audience for you: the people that you can target just because you have similarities between the vectors that were computed for products and the ones that were computed for users.

Conversely, you have a user, and you can encode it—let's say you encode it based on all the products that he or she saw. You're going to have all the products that are around, and you can recommend the products that are around. So that's recommendation.

It's very powerful to have those representations, as long as they're semantically meaningful, and that's what we work on.

So everything, for the most part, it sounds like it's kind of precomputed. I mean, there's an interesting moment where that most recent action of the user has got to be critical, so presumably there's some marginal computation that has to happen. But if I'm understanding correctly, you have a base user profile and base product encodings, and those are aligned such that at runtime it's an inner product of those 2 vectors.

Yeah, basically everything is done except that comparison. But I guess there is probably some last-second update of the user profile as well, based on where they are right now.

Yeah, totally. That's actually a challenge: how you recompute those embeddings live at runtime. That's actually an engineering and technical problem, to do that precisely. It means having other versions of the big models so that you can compute much faster, but still you do not hinder the way the similarities are going to be computed. That's precisely where we put a lot of effort.

You just need it. That's the question for this small adjustment, where you have to take into account the latest information, where you cannot rely on offline information. But doing that offline—online, sorry—is the key to what we do.

And we've done a lot of experimentation with that over the years, trying to find the right balance.

If you have the perfect amount of time, you can come up with the perfect result, but sometimes the fast result is good enough, right? In terms of the time constraints you have, especially when you talk about the constraints of real-time bidding, where you have to answer in milliseconds, it's the best possible answer within the time constraints that you have.

Yeah, that's really interesting. The architecture of this does have a lot in common with the Stripe system. I guess it makes sense, because they have a lot of similar constraints in terms of having to respond to whether you should approve a transaction or not in an incredibly short amount of time. Yeah, it's a great analogy.

How about the agentic—I don't know if agentic is maybe even the right word yet—but in just an OpenAI context, right, or any sort of chat context where now we've got this additional signal of what the context of the conversation is that the user is having? Does that also get treated essentially the same way? Do you get raw text and then embed it yourselves, or do they send you an embedded form of the text? Then does it become another one of these already-embedded inputs that goes into the decision-making process?

Okay, I can take it. Essentially, in Criteo, we started to look at those conversational agents 2 or 3 years ago. We were interested in knowing what was happening. I'm not going to talk about the partnerships here, because it depends on the product we're building and so on, but in the way we think—and that's my job in the Criteo Lab—we're envisioning all the different scenarios.

One scenario is that we have the conversation, work on that, and learn something from it. About 2 years ago, there was a person at Criteo who built a kind of conversational agent on Slack, using the messages that were there about how to use the strategies to try to answer some questions from clients. He built an agent from scratch that would try to understand and recommend answers to troubleshooting problems.

It was not recommending a product; it was recommending a solution to a problem, but it's the same thing. We also tried something where we actually had some kind of summary of the conversation—just a vector—to see whether it was a signal that would help us build another model and see whether we could learn from that. It's, of course, less powerful than having the conversation, but that's something that we've been trying.

In terms of what I do in the Criteo Lab, we're trying to de-risk and envision what is going to come up. So far, in terms of the partnerships we should work with, we're still investigating the right way and the right data to exchange. I'm going to let Diarmuid talk about that.

It's still early days, right? We're very, very happy with the partnership we have with OpenAI. They're a really great team to work with. We're both very much privacy-driven, and we're both very much about user consent and so on. It's a core principle of both companies in how we handle that. We're very respectful of that, and OpenAI is exactly like that too.

We need to have only the information we need to be able to show an advertisement at a given time. It's not going to be all the time, because not every context would make sense. That's something that we're continuing to partner on as we build out this network together.

Maybe just a last word on that: you have a team, or a bunch of people, working on what is called trustworthy machine learning, and I think that it has never been as important as today. Precisely because of the type and volume of information that people are readily willing to share, there's this question—and that's not just a research question anymore. It's very practical.

We do look at how we can make trustworthiness practical. It's about fighting hallucinations, being sure that it's privacy-safe, and being sure that sometimes we're not—or sometimes we do—understand that the people we're talking with are someone to whom we should recommend something. That's something that we're looking at.

Maybe we're a bit more on the upstream side, I think, but we know that at some point something is going to be very strong, either in terms of regulation or in terms of—I don't know. We prefer to be on the safe side, and we prepare ourselves to be able to answer at the right time.

I totally agree. I think the fact that we were based in Europe, where there's definitely a lot of sensitivity around things like GDPR, has always been important. We've always had that as a core principle as we build out these products.

For advertising to be truly useful, it has to be trustworthy, right? We don't want to do anything that's creepy, because if it's creepy, it won't work. It has to provide real utility. When the user sees it, they go, “Okay, that's interesting. That's engaging.” If that's the case, then they're more likely to click on it. If they're more likely to click on it, then it means that we're a better-value provider for the advertiser.

Can you tell a little bit more about that? I think probably a lot of people are surprised, almost an hour into the conversation, to hear that the company has roots in Europe and is headquartered in Europe. I tend to think of ad tech as a mostly American phenomenon.

How different is it really trying to operate in Europe versus the United States? Are there things that you actually do differently across jurisdictions based on restrictions that may exist in Europe, or is it sort of the same approach globally? I think people have a sense that you can't do AI in Europe. That's the first-order summary from a lot of people. So, if you think that's wrong, disabuse us of that notion.

I would say that is fundamentally wrong. It's false. I'm from Ireland, and I moved over to France 11½ years ago. The team here is just amazing. The Criteo AI Lab—the quality of the data scientists we have here—is just off the charts. They're so talented and so dedicated to what they do.

Being European-born definitely helped us, because we grew up in an environment where we had to be very careful. It was a first principle for us in terms of how we handled user data. We know we've got this contract—this implicit contract—with the end users in terms of how we operate.

The United States is obviously one of our largest markets, so we operate in many territories. Everywhere we work, we're very respectful of local regulations around data and what you can and can't do. We always go the extra mile. We're always very careful to make sure that we're fully compliant, and then some, with respect to regulations, because it's super important.

We're a very global company. We have offices in Ann Arbor, Michigan, and Toronto, as well as across other locations in Europe. From a development and AI point of view, we can really tap into talent from all across the globe.

Liva, if you could talk a bit about AI in Europe. You're the French person here.

Yeah, French person. I get it.

One thing that I can say is that here in France, there is a school of mathematics and computer science where people are very talented. They're really well trained to do that, and it's not just AI; it's computer science in general. They are very good, and they are very good people. It's just a question of having very good, well-trained people to innovate.

It's not a question of France not innovating. You've heard about Criteo, but you've heard about Mistral and many companies coming from France. One of the things that is very important in Europe—actually, maybe I'm going to talk about France—is that, as you know, part of my life was spent as a professor at a university, so I know students as well.

The students are very good. They learn with enthusiasm, and they do like it. Maybe that's not necessarily the best thing, but in France, mathematics and computer science—the things that are very formal—are very important.

People tend to forget that, but the roots of AI, the reason why you can craft and build models that are meant to answer specific questions, and you're capable of crafting those models, rely on the ability to formalize things into a model and to find the right technical tools, either in computer science, engineering, or the more mathematical parts of things.

That's just important for innovation, and we do have that in France. I don't know the rest of Europe; I do know France, but I know that in France we have that.

One thing that struck me when I moved here was that, if you think about it, ancient mathematics—a lot of it would come from Greece. But modern mathematics, when you study it—I did engineering at university—you've got all these names: Laplace, Lagrange, Fermat, Galois. All of these underpin the whole of modern mathematics, which is the basis for machine learning and AI.

France has a really great standing in that area, which Criteo massively benefited from. I was quite impressed and surprised, actually, in looking up the Criteo AI Lab. First of all, just how many people are on the team? I think I saw something like 50 faces on the website.

And then, specifically, I was very surprised to see all the faces on the website, because I was thinking, jeez, I don't see too many AI companies in the US doing that. I think they're all afraid that if they put their names and faces on the website, Zuckerberg's going to come calling, and [laughter] the whole thing goes sideways because they all get offers they can't refuse.

I don't want to complicate your lives, but how are you thinking about that? How are you able to build and retain a team like this in the age of the Zuckerberg blank check?

I think I'll go first, and then you can layer on top. Within Criteo, there is obviously—I'm biased—but the culture here is amazing, right? I travel over an hour every day to come into the office because there are so many cool people who are so good at what they do, and there's something really engaging about that.

If you can give these people some really interesting problems to work on in an environment where they're surrounded by like-minded people, that's super engaging. The fact that a lot of people here have a long tenure and have been working here for quite a while creates an ecosystem that is very engaging.

Even if we do have competition from others, we can all welcome that because it puts us on our toes and means that we in leadership have to make sure that we continue to make Criteo a really great place to build your career. We've been doing a great job of it for over 20 years now, so that's what we want to continue doing long into the future.

Yeah, I'll leave that to you. One of the things is that we have research scientists, and one of the things research scientists do is publish. One thing that is true across all research scientists, I think, and in other companies, is that they have to make their research reproducible. It has to be open, et cetera.

In research and science like that, you have to have a presence on the internet if you are a research scientist. Even more so when I'm talking about the academic side of things: people who are called scientists usually have their own website and their faces on it. That's what we want them to have here in Criteo as well, to come do research just like the other scientists from universities and other companies. That's the reason why.

The other thing that Diarmuid shared is the need to find challenging problems and challenging topics to work on, so that they can say, “Okay, it's good to be here and do science in Criteo because the problems are not easy.” It allows them to connect something that might be very upstream to something that can also be deployed—maybe not yet, not today, but in 1 or 2 years—and that's very important.

Maybe one other thing, just to build on that: Liva's team also works very closely with a lot of the academic institutions in France and across Europe. We sponsor PhD students. They'll come in, they'll work alongside the existing research team, and they'll work on real-world projects, so we give them real, concrete experience.

Many of those PhD students actually become full-time employees. Not all do, but that's okay too. They publish their research, which is good for their careers, and this is a great way to keep that pipeline going. It keeps us very well connected with the wider AI ecosystem as well.

Certainly, the ability to publish one's work in today's world is a differentiated part of the offer. I've got one episode that has been recorded for 3-plus months now with somebody at one of the—what I'd say—frontier companies that we just cannot get approved, and it's great stuff, brilliant work. We'll see if it ever sees the light of day.

Maybe just one more beat on the regulatory environment. I would separate here following the rules, which you've clearly stated that you're committed to doing, from advising on what the rules should be. Do you think there's a meaningful difference between the environments in Europe and the United States when it comes to an individual's rights? I guess that's one way to think about it, but what I really am trying to get at is: who has it better?

Do people in Europe actually have meaningfully better protection that I should envy, or do I get meaningfully better ads that people in Europe should envy, or is this all much ado about nothing? Who should be changing, and on what margins, their rules to better serve their citizens? Do you have a point of view on that?

We actually engage quite a lot with the authorities in the EU, with the different data protection authorities—the DPAs—in different parts of Europe, as well as in the US. We're very much advocating on behalf of the end user to ensure that it's about transparency and user consent, and ensuring that users have a way to opt in and opt out.

That's why we had the cookie consent message long before it was even regulatory. We also push for ensuring that we live in a system where there's a fair value exchange, so that the user feels they have proper use of their data and that they get value back through free content, free services, and so on.

In a world where all of that disappears, you take away all the value for the advertisers. If an advertiser is spending money to try and increase their sales, they're not getting that value back. If their products are being shown in front of people who have no interest, then that does nothing for their business, and they're not going to spend.

If they don't spend, then those people who provide that rich stream of services, websites, and content have to monetize some other way. What they do then is put up paywalls. That's not in the interest of the end user.

As long as there's a fair value exchange, as long as we're transparent with what we do, then I believe that's really important. So, back to the question about the US versus the EU, to a large degree, that stuff is being equalized. A lot of those things, like the changes to CCPA and CPRA in California, are very much inspired by each other.

We try to build a global solution. What we've done in Europe, we use the same approach globally. It's not like we try to be looser elsewhere. We really believe that, by having that principle and by being born in Europe, we have a solution that can work pretty much everywhere.

Cool. That's really interesting. Let's talk about creative a little bit. I'll invite you to help lead this part of the conversation, but it is interesting that we've made it this far and haven't really talked about creative. Obviously, there are a lot of different formats.

I don't even know today if—okay, I guess there are a couple of ways at least I would want to come at this. One is, I think the number I saw was that Criteo has 17,000 advertisers. In our experience at Waymark, we've often seen that a lack of creative is one of the biggest barriers to new advertisers signing up with a platform like Criteo, or for that matter Meta or Google or what have you.

I wonder if you see that similarly. Is that a core barrier to market expansion? Obviously, we're in this moment where the cost of creative of some quality is dropping precipitously.

I don't know if there's a dynamic layer to the creative. When we're doing these kinds of matching, auction-prediction sort of things today, is creative an input to that decision-making? Do different advertisers have multiple creatives that you're scoring or embedding and using to drive outcomes, or is that still a frontier where, perhaps because they don't have enough options, you can't do that in many cases? Or is there some other reason?

It seems like in the future we imagine, everything is going to be highly personalized. The ads are going to be much more directly talking to us as individuals. It seems like you have the infrastructure to do that, but maybe the creative is just not there. What do you think is the future of creative?

I think that's one of the most exciting areas in the whole generative AI space. The things that you can create with these next-generation models are just insane, and I think they have the possibility to create even more engaging advertisements.

You talked about hyper-personalization. I think that's super powerful, and people will see it and go, “Wow, okay. That's exactly the product I'm looking for.” It makes it much more interesting and much more engaging for them.

The other part of it as well is—you touched on the point there—with this technology, we have the ability to democratize that content creation. Before, the mid- to long-tail would have been very much cut out of the picture because it was just out of their reach, out of their means to be able to create that high-quality content.

That is now becoming more and more accessible. Creadits recently launched a self-service product called Creadits Go, and this is really making it easier and easier for advertisers who are often much smaller to create very engaging campaigns with really great creative that will help drive and grow their business.

That's super important, and it's really enabled by platforms like Waymark, which is a great partner of ours.

Alex Persky-Stern

Thanks for the shout-out. We love it.

On creative, there are 2 things. First, something we've been doing for years in creative is dynamic creative creation with some templates and visual assets, and we have to arrange them online. It's something like Legos: you have the pieces, and it's going to put your ad together to craft your ad.

In terms of the speed of generating those ads with generative AI, we're not at that level. There is no way—unless you're ready to wait 5 seconds for the page to load—we're not there. So we have to find a way to balance things between something that's going to be done offline and online. The Legos that I talked about—the visual assets—can be generated either by us or by our partners with generative AI. Then it's up to us to arrange that at runtime, using our engine, which is capable of arranging that online.

Maybe at some point, one day, it's going to be very quick, and maybe we could generate something that's not going to be completely generated by us. One day in the future, it could be completed by your TV or your mobile phone online, and it would be very quick. But we're not there yet, because you've probably tried to use some of those tools that are trying to build an image. It's not something that anyone can do yet, but we're not there yet. At some point, maybe in 2 or 3 years, we're going to be able to have that.

Yeah. Do you have a perspective? I'm interested in the level of personalization that you see the infrastructure supporting. I think there's, of course, the level of the user experience that you talked about a lot, which is super important. You don't want to be super creepy, but when you think about the level of personalization that's at least possible infrastructurally, do you think it's going to be literally at the individual level, where we could tell you a story that's specific to you, or do you imagine it being more at the audience and context level? Where do you see that living?

For me, I think you're right. I think it will mostly be more at the audience level, right? I think the point of going way too hyperlocalized—I'm not sure there's huge utility for the end user or for the brand either. The brands and the advertisers will still want to retain levels of control over the look and feel and how their brand is showing up.

Within those parameters, I think it's mostly on an audience basis. I can see it working down to a hyperpersonalized level, where every single person sees a different thing. I'm not sure there's huge utility there from the point of view of driving more sales or getting more product in front of the end users.

Something that I would like to see at some point is for those very different models to have sizes that allow them to be embedded in devices, in glasses. If at some point we arrive at that level, then maybe the computation is going to be shared. We're going to provide something at our level, and maybe on the personal device, something that's not going to be seen by us, but the personal device is going to tweak things at the very end.

It could say, actually, this was something that was proposed by us, but I know you—that's the device—and you have on that device something that's really personalized. On that device, in a very privacy-preserving way, we're going to record something, but just on the device. So maybe that's my dream.

Yeah, I love that. Super interesting. Maybe there's going to be something shared in terms of computation, and maybe the personalization is not going to happen on our side but at the end-user side.

Yeah, I would agree with that. Where I could see this going is where the action is actually triggered by the end user, the consumer, right? If the good use case is, okay, I get invited to a wedding in the South of France in the summer, I want to know what looks good so I can ask the AI, “Can you recommend some outfits?” Then you can see a virtual try-on, stuff like that.

The usual example is that you want to see how something would look in your home. That's very much engaged by the user, so they're kicking it off themselves. Because they've done that, it's not creepy. It's their action, and you see it. For me, that's actually where hyperpersonalization really comes from, where I think it makes sense.

That's great, and again, going back to what I talked about earlier, it's the utility. It's trustful, it's useful, and it's providing value. It could help give a far better experience for all involved.

Yeah, that's super interesting, and it seems like it's what we were talking about earlier, too, in a chat or agent context, where we're actually getting way more comfortable with what would be creepy in any context. Letting it do that personalization for you is super.

Well, if there's someone who's asking for it, then I think that's perfectly cool.

Yeah, yeah. Cool, super interesting. One other thing I'm interested in—it's not a different angle on personalization, but contextualization. I know CTV isn't your historical bread and butter, but I know it's a growing concern, and I think one of the best-documented performance gains is when the creative of the advertising matches the creative of the context.

I'm sure that's also true in some other contexts, but it's really true on TV. Another place where you can imagine lots and lots of different variations that match the tone and style of the particular context—not only the movie, but the very specific moment in the movie. Is that something that you've started to think about and bake into the way that you run your models?

For me, what's super important about advertising is that it should not be intrusive to the user experience. It has to feel seamless and not get in the way of the user experience of the content. That also applies to websites as much as it does to connected TV.

We've all seen those demos where you get advertisements inserted into the content, whether it's sports or your favorite sitcom, and so on. I don't know if we're quite ready for something like that yet. I think you have to have something where the user feels that it's relevant and that it's not interfering with their experience of the content.

With video and connected TV, there's pre-roll, mid-roll, and post-roll, which is a different way of experiencing it. As long as it's not intrusive, I think users will be open. I think that's more of the guiding principle there.

Oh, makes sense. One other question that I'm interested in: I think this is a really developing moment right now, as, for the first time ever, creative can really be technically powered, where it's always had a very human-in-the-loop requirement. How are you guys thinking about, as a technical organization, what your role in the creative is, not only right now but over these next few years?

I think that, as of today, the way we approach those creatives, so that we're sure that the quality of what we provide is true, is through building our own creative models, because they're very costly and very expensive. But I think it's something that's going to be key. It's not my area, so I think we have a team dedicated to doing that.

It's not a 1-year program, but more like a 2- or 3-year program to integrate those creatives. They used to be in the creative lab, but now they're not anymore. So maybe I'm going to defer on that one.

Yeah, so for me, I think ultimately there are a few different constraints. For example, brand guidelines. The brands themselves—the original advertiser or retailer—have their own look and feel and the way they want these things to show up.

Ultimately, we want to build advertising that works, advertising that brings value for the advertiser, so they feel that their money is well spent with us. Within those parameters, we try many different things. We even talked about the way we did DCO before, which was really one of the great ingredients of our initial success.

You layer on top of that the generative AI capabilities, and the possibilities are endless. The potential is just amazing.

Maybe just a word that you used, Nathan, and it's true for everything that you're doing with AI today: human in the loop. That's the key, actually. Not just for this creative, but for everything we've been talking about. The big challenge is being sure that you put the human at the right point. That's very important.

You should not strip the user away from their right to decide. If you do that, to automate everything, it's not going to work. It's going to be a problem in terms of liability, responsibility, et cetera. At the same time, you're not going to feel, as a human being, allowed to make any decision.

The question that we have every day when we build a model is: Where is the human? How do we learn so that—and how do we train the model so that—the human still has a place? We have to spot the right place. Just missing that could be very detrimental to the project that we have, and it's true for creatives, for the bidding models, for the agent stuff, et cetera.

So, that's the big thing that's very important not just to us but to everyone building tools: it's the human. And that's why those conversational agents work so well. They're still relying heavily on human interaction. So, that's very important.

Yeah. Well, here's a small one, but a curious one, and it is kind of relevant to our business. It's top of mind for that reason, but also in general, right? We've been talking about all these different touch points. Where are we today on cross-device understanding of who somebody is?

I could imagine that being very algorithmic and sort of deterministic, but it occurs to me that maybe that's another AI question in today's world: Who is one person, and who's a different person when some devices are shared and some are on mobile networks, with all the complications there? That was supposed to be the short one, by the way, so I'm just processing that. I can make it long, but I don't want to keep you guys long if you have something next.

Yeah. So, there are a few different aspects to that. Again, it goes back to user consent, right? The user needs to be okay with this because there's shared Wi-Fi at home and all this kind of stuff, and you want to make sure that you're showing the advertisement to the right household member. So, there are different ways to do that.

There's the Wi-Fi signal, or the network that you're connected to. There's also whether you're logged in on multiple devices—for example, on social or in other ways. That can be used deterministically to say, yes, it is the same person on multiple devices. The probabilistic way can be used for certain things, but only in a very, very untargeted kind of way, because if somebody has opted out of advertising, then you don't want to reshow them advertising even through a different way. That's something that we always keep in front of mind.

Cool. All right, last big one for me. Zooming out as far as possible, in general, with this whole AI wave, there's dramatic uncertainty about what's going to happen, right? How powerful the AIs are going to get, how disruptive it's going to be, and whether we need a whole new social contract, et cetera. Specifically in the advertising space, I feel like I see 2 trends that counter each other.

One is that, if all goes well, we should all be a lot richer and the value of time should go up. That would be really good for the advertising market in the sense that, fundamentally, you're competing to have some little share of people's time, and the richer they are, the more valuable their time is, the more that's going to cost. So, that's an advantage, presumably.

But then the other thing that people also sort of see is that search costs and matching costs could really drop in a lot of ways if we all have our AIs going out and vetting much broader portions of the world than we previously could, right? Today, I can only evaluate so many shoes, but maybe in the AI era I could have an AI that goes out and really evaluates, in a comprehensive way, every possible shoe. Therefore, my product decision-making is less about who was willing to bid on my time and more about how much time I was willing to have my AI invest on my behalf to go out and figure out what to do.

I don't know. That's more of a prompt, I guess, than a question, but what do you think of those trends? Are there other big trends that you see as being huge factors? And where do you think we are in advertising in, say, 5 years' time—or the singularity, whichever comes first?

I'm not sure about the singularity. I have my doubts about that one. For me, where I see this new generation of AI really helping to increase the amount of value to the end user—you mentioned time, Nathan; maybe there's value as well—is with those kinds of products that you're looking for.

If you have a very specific thing in mind and you don't know where to get it, you're able to then query these LLMs, or whatever interface LLMs become, and, partnering that with really rich commerce data, say, “Here's my criteria. I don't care about the price. I just want the best product possible.” You explain what it is you want, and connecting all of these systems has the possibility to give a really, really great result. When you get that great result, then you're quite happy to buy it.

I think that's the kind of utility: helping people discover not only the product they're looking for, but also what goes with it, how you can actually enhance it, and so on. When you see the way that they're working today, where they're going, and how fast they're improving, I think that's incredibly exciting.

And it can provide a really, really great consumer experience. For retailers to be able to engage in that allows them to get their products in front of end users in a way that they couldn't do before, because a lot of the advertising before was guesswork. Now it becomes really, really focused, and you're making sure that you're getting all the information that the end user needs to be able to buy that product, whether it's the shoes you're looking for or that new piece of tech that you want to invest in. I think it really, really gives end users an enhanced way to discover products.

Yeah. First, I can answer the prompt because it's very deep. One of the things is that something is going to happen in the way people are exposed to advertising. You might make the choice not to be exposed, but you are exposed in a way. I think that with the advances in AI, those companions or assistants that you're going to have with all those LLM platforms, et cetera, maybe at some point you're going to want advertising.

The exchange is going to change a bit because advertising is also about accessing or discovering the right product, et cetera, and maybe things are going to happen behind the scenes. You're going to turn a knob and say to your assistant, “Okay, I want to go and see shoes. But I want you, as an assistant, to look at 10, 12, or 100 different shoes, and I want you to select among those 12 or those 100 shoes 6 of them. But I want to be exposed to 6 of them, and I want to be exposed to advertising, and I want to choose from that.”

There might be a new actor and a new intermediary in the advertising business. People might want to be exposed so that, again, they have the choice as a human to say, “Okay, I want to be exposed.” Because having the choice of different shoes is having an advertisement in front of you.

Maybe they will be saying, “I want to see—I want a trip. I want 5 different trips. Show me 5 different trips.” But I want to be shown 5 different trips, not more, not less, and then I'm going to choose. So, maybe it's going to change something in the way advertising is going to be experienced, just because of the quality and the ability of these systems to filter and to select.

Maybe there's something like that. In 5 years, maybe you're going to be in that situation where we're going to say, “Okay, show me some advertising.” So, yeah.

Yeah. That's like the collapse of search and advertising. If the advertising is good enough, it might be better than search.

Yeah, exactly.

Interesting.