AI 正在吞噬搜索

Hey everyone, welcome back to the Latent Space podcast. This is Alessio, partner and CTO at Decibel, and I'm joined by Swyx, founder of Smalls AI.

Hello, hello.

Today we're diving into a topic that I've wanted to explore for a while. AI search engines have caused a lot of the rising interest in AI search engine optimization, and today we have Robert from Scrunch AI. Welcome.

Thank you. Great to be here.

Alessio, I think you're a pretty B2B SaaS OG, just like Robert.

Robert, as a guest disclosure, Decibel invested in the round that we're going to announce with this podcast, so congrats, Robert. I guess congrats to us for being picked by you. You were previously the CTO of Hearsay, which was started by Clara Shih, who's also a friend of the podcast, and my partner John was on the board of Hearsay, so you guys knew each other from there.

We kind of go back, and even back then you were already working in the space of, “I have a business—how do I reach people in a way that is very tailored?” That was more on the financial-services side, and then you started Scrunch a couple of years ago, so you were pretty early to the space. Now everything is blowing up. Everybody's getting traffic from ChatGPT, so maybe talk a bit about when you first realized that LLMs and search were not just normal search, and maybe the idea maze that got you there, and we can take it from there.

Yeah, absolutely. That's a great starting question, and the origin story—hopefully I don't get in trouble for telling this on the pod—is a little bit convoluted, as I think it is for most startups with Scrunch. My co-founder Chris and I had left Hearsay. Chris was also a longtime executive at Hearsay; he was the chief product officer for a long time and the first employee, so we both know all those folks really well.

We were looking for what to do next in our careers, thinking about starting a company, kicking around ideas, and talking to people we knew in the industry. As we were doing that, the elephant in the room was AI. It became really clear, I think, a couple of years ago, that it would be foolish to start a company without thinking about what the impact of LLMs would be on the world, and you'd already be behind if you started a company that wasn't in some way LLM-native.

As we were talking to the people we knew best, who were often enterprise-y—large banks and insurance companies, since that was the primary audience for Hearsay—it was a tough sell back then to sell them anything related to AI. The number one thing that people were probably asking us for, quite honestly, was, “Can you put a chatbot on our website?”

They wanted to have a little widget. They wanted to have their version of ChatGPT. They wanted people to come to their website, have a widget pop up, type into it, and ask the website questions instead of going to ChatGPT. There are good versions of that now. The versions of that functionality available at that time, I'd say, were mostly not so great.

My core feeling about that at the time was, “Nobody wants to use a chatbot on your website. I don't want to use chatbots on people's websites.” If things pop up on a website when I go to visit it, I fly into a visceral rage. I'm definitely closing the window, if not closing the tab.

I have that feeling, but I also wonder if normies respond fine to it. Intercom is doing fantastic as a business, right? So I have some self-doubt about how representative we are.

That's a fair question, and I always ask myself that as well, as a weird technologist person. I think there's some evidence that people don't love it. Intercom is a little bit different because by the time you're engaged with Fin or what have you, hopefully you're already in a relationship, to a certain extent, with a business.

I do think that people have a little bit more tolerance for using these AI tools when they're already feeling some rapport with a website or with a business. They're more willing to use your tools instead of staying in the environment they like to use. But if you're just coming to a website and a chatbot pops up, people mostly don't want to do that.

People have responded really positively to ChatGPT. Most people I've talked to—and, by the way, I live in LA. I've lived in LA for a couple of years since the pandemic—I'd say more of my social circle is normies than it used to be. People love ChatGPT.

The experience people have of using ChatGPT versus browsing the internet—the old Google 10 blue links and clicking around—is vastly more positive compared to the past. People really feel an affinity for the tool, and they want to use the tools they like. Whether it's ChatGPT, Claude, Perplexity, or what have you, they really like that environment. It's convenient and productive, and they don't have to browse websites that maybe aren't well designed for them.

People aren't excited about the idea of, “Okay, let me click around, find a website, and then engage with the chatbot there.” In the same way, I don't think people are saying, “Let me go to the website and use their search bar.” People just want to use Google.

I do think that's true. When we had that realization—people were asking us for this thing, but we didn't really want to build it, and we didn't necessarily think that was the durable value—we started asking ourselves, “What are they trying to get to here?” It came down to discovery and the realization that AI was going to change something about their website. It was going to change something about their customer journey and how people interacted with their business, but we didn't know exactly what it was.

As ChatGPT Search was coming out, we started showing people how ChatGPT Search was referencing their business—how it was surfacing their content and what it was saying about their brand. We had the experience of a lot of folks—CMOs at large insurance companies, enterprise software companies, or large e-commerce marketplaces—saying, “That's really scary. It's saying stuff about my business. It's saying stuff about my brand. It's surfacing content from my website, and I don't really know what's happening here.”

That's the genesis of how we got into this business. It was less about focusing on SEO and performance marketing specifically, and more about the fact that there's a change in the way consumers interact with the internet. I'd say that's still true today. That's still the core thesis of the business: the consumer experience of the internet is changing really, really quickly for the first time in quite a long time. So maybe I'll stop there, but that's the origin story.

Yeah, and as you know, most people in the audience are technical, so it would be nice to dive into that. I guess everything starts with a prompt in your world, which is what the consumer asks ChatGPT about some sort of product, market, or whatever. How are you doing the monitoring? Are you monitoring every LLM? Are you monitoring every model plus Search? Are you monitoring the AI-native tools like Axiom and things like that? Maybe talk people through the pieces of the stack here.

Yeah, what we do starts with what you said: we monitor prompts. Basically, we go out and simulate consumer interactions with the major AI platforms. We go basically by usage. ChatGPT, obviously, is among the highest in usage. AI Overviews could be considered up there as well, if you consider that in the same category. AI Mode, Perplexity, and you can go down the list from there.

We definitely don't cover everything. I'd say our ambition is ultimately to be wherever people are—wherever the consumer is and wherever our customers' audiences are. But today we primarily focus on the biggies: ChatGPT, whether it's Search or trained-model knowledge; Gemini; Perplexity; and Claude.

Claude is important—not necessarily huge in terms of raw number of users, but the people who do use Claude tend to be a very valuable audience, especially for some types of companies. Then there are things like Meta AI, which I don't think is necessarily remarked upon a ton in the AI enthusiast and AI developer communities, but it's a silent monster because of Meta's distribution and reach.

It's expanding all the time. We'll have more platforms live probably by the time this podcast comes out, but we start by focusing on, as you said before, what are the normies using? What are they doing? What are they seeing in these tools?

Now that you have Search with the backlinks, when you don't have Search on, there's no way to figure out why a model is saying something. Is that a battle that people basically cannot fight? If there's something in the pretraining, is there something that you advise people to do, or do you just flag it and that's it?

It's definitely less actionable, right? And I think the good thing for brands—by the way, when I say “brands” throughout this conversation, that's just our term of art. I mean businesses: businesses that have products and services and websites, and want you to do stuff on the internet to make money.

I think it is less actionable, so most people are focused more squarely on AI search—ChatGPT with a search box ticked. The good news is that most of the queries that lead to action, that are of commercial interest, are increasingly using search across these systems for a bunch of reasons: timeliness, avoiding hallucinations, and so on. I’d say most people are focused on that.

There are things you can do even in pre-training mode. An example of that is thinking about what you’re exposing to these models when they come and crawl your website, right? Everybody knows that GPTBot and CCBot and all these various AI training-data collectors are hitting tons of websites on the internet. Some people are fine with that; some people are upset about it, but they’re identifiable, and you can think about the half-life of the information you’re presenting and whether or not that’s something that really makes sense to give to a training-data crawler.

For example, if you’re an e-commerce company and you’re running a seasonal sale, should that go into the training-data knowledge of a model? I would argue it’s probably counterproductive, right? Maybe you don’t want to expose information about that to those crawlers hitting your website. That’s something we advise our customers on.

While we’re here on the topic, any quick takes on what Cloudflare did last week, which was apparently start introducing paywalls for all these bots?

Yeah, right? It’s really interesting. I kind of agree with the sentiment behind it. Obviously, it’s tough to be a writer on the internet. It’s tough to be a content publisher on the internet, and it’s getting tougher and tougher over time, whether it’s ChatGPT or Google AI Mode. So, the idea that there’s some sort of grand bargain we need to make as a society to compensate content providers, which I think is what Matthew Prince is getting at with his blog post, does make some sense to me.

I think there are 2 things to think about there. One is the technical mechanism: how does this paywall work? I would just say the bridge doesn’t connect on both sides yet. Maybe we’ll get there. I don’t think there’s a lot of actual use of those systems, and there’s prior art. There are things like TollBit, for example, that have some similar ideas. I would say there’s some uptake in traditional media, but it’s not a huge piece of infrastructure on the internet yet.

The second thing is about power dynamics: who wears the pants between website operators, CDNs, and AI—in particular, AI search—or search engines in general? Websites are obviously sensitive to how their content’s being used, but they also need traffic. Traffic by and large comes from Google and is increasingly coming from ChatGPT, as we’ve seen.

There’s a fine line for people to walk. Unless you’ve really got a strong native, organic audience of people who know you, love you, and have a relationship with you—which is still one of the most important things you can do on the internet—you need to be careful about how restrictive you are with your content, because distributing your content through these systems is how you’re going to get users. I don’t know the right answer, but I think it’s tricky.

I find it a really interesting divide in society between some content creators who want everything behind a paywall and want to be paid for every single use, and others who work very, very hard to make everything free. Like, train on me, right? If anyone cares about our position as content creators, everything in Latent Space is free. We have some soft gates, but it’s not really something that we care about. Your content wants to be free; the cost of reproduction is zero. Just get it out there, right?

Maybe ChatGPT will start referring more people to you, and you’ll be more of an authority in your space, all that good stuff.

That’s very real. One thing I would mention here is that they also had the AI browser announcements this week. There’s Dia, there’s Comet, and there’s the untitled OpenAI browser. I think that’s a really interesting addition to this mix, too.

What I think about is what you just said: that ’90s phrase, “information wants to be free.” People want to connect models to the internet. The way people are using these today is through ChatGPT or something, which feels like Google is an input box in the middle of a pretty spartan webpage, and ChatGPT is an input box in the middle of a pretty spartan webpage. So, they feel pretty similar. They’re both these internet platforms, internet gatekeepers, and people think about them the same way.

But there’s no reason that has to be the way you consume internet content and get it into these models. I think the AI browser form factor is a really interesting evolution there. There are going to be others, right? There could be local tools and local models.

What I would say is that you can fight the battle of saying, “OpenAI shouldn’t consume your content without paying for it,” or “shouldn’t consume it at all.” But I think it’s going to be really tough to fight the battle of saying, “LLMs writ large shouldn’t consume my content,” because it’s just not that hard to hook these things up to internet content and stuff, to some degree. Once it’s in somebody’s browser, how are you going to stop them? That’s a whole different mechanism.

Some people are thinking about this change when it comes to AI search as sort of one platform replacing another. So, instead of Google, it’s OpenAI, right? Or maybe it’s Google disrupting itself. But I think there’s a more fundamental change, which is that people just have more powerful tools, these AI tools, to consume web content. What you’re seeing with people who have access to the tools is that they don’t want to browse websites the same way they used to in the past, right?

I actually think this is probably a medium-spicy take, but thinking about what’s going on in AI that’s really replacing search is the wrong take, or maybe not that interesting. It’s really more like AI is replacing web browsing. I think that’s a more meaningful, more fundamental shift. The reason it’s happening, I think, is because people just like it better, right? It’s not because of some top-down platform mandate. People like using these tools to replace what they were doing before by clicking around the internet and consuming content. Not for every single use case, but for a lot of things that people are using websites for today, they prefer to use an agent.

Well, Deep Research is probably the biggest example of that. On the content side, it’s interesting because Lenny, who runs a very big product newsletter, posted his stats from May 10th to last week, and ChatGPT drove more traffic than Twitter for him. I’m surprised because he has 250,000 followers on Twitter, and he only had 9,000 views from it. For us, Twitter is much higher than ChatGPT, even though we do get a good amount of traffic from it.

I’m curious how much you want to tell the models about who you are and not as much about the content. You kind of want to be present in the curation but not in the details, and have people in the details for you. This is something that people have been debating in shopping: when I’m using ChatGPT to find something to buy, how much of the information should I put in the webpage so that it gets put into the context versus on podcasts?

If I’m searching for the best AI podcast or whatever, how much does it need to know before it suggests me? Or is the model simply just using the same ranking as the SEO that the Google API and Bing API are using?

This gets into the mechanism of it, right? What I would say is that AI search is still fundamentally search. There is still a search ranking. To the best of my knowledge—which I think is pretty good—most of the search that’s in AI search is still traditional search, right?

There’s an AI model in front of it, but you’re still using your traditional text-matching search algorithms under the hood—BM25, TF-IDF, PageRank, or what have you—to locate pages in a search index. That is a fundamental component of all these AI search systems so far. I know people like Exa are doing something different, and I think that’s really exciting, but that’s not what’s powering most of the products that people are using today. Traditional search ranking still matters, and traditional SEO techniques still work.

Where it’s tricky after that is that you’re not just getting links for discovery; you’re also consuming the content behind the links. How much of that are you giving away? What does it say? How well is the system able to interpret it?

The way we think about this for most models and most platforms—and the details vary a little bit between ChatGPT versus Claude Search or something like that—is that you’re generating queries. You can generate 1 query; you can generate multiple if you’re looking at AI Mode or some of the more recent o3 search integrations that ChatGPT has.

You're generating keywords using an LLM, right? You're running searches in parallel or potentially serially if you're using a reasoning model. They're going out and retrieving results, and then there is a re-ranking that happens once you've got those raw search results.

The raw search results are, again, traditional search, but then what you're doing after that is re-ranking them for relevance. Only then does it start looking at the actual content behind the pages.

This is where we see people get tripped up a lot in terms of small things that go wrong when it comes to getting cited or mentioned inside ChatGPT. People have pages that rank really well for search, and then when you look at the title of the page—you don't look at the page content, you just look at the metadata about the page—the metadata has been super-optimized for human click-through.

Think about clickbait and things that create urgency, especially in e-commerce around sales and things like that. They're not very informational about what is actually on the page, and what we see is that, regardless of search ranking, a system like ChatGPT often chooses not to use those.

It will re-rank them out of the consideration set, and then it will go on to the next-best result that has more descriptive metadata saying, “Hey, this page actually has something relevant to your question on it.” That re-ranking step is critical.

Since this is a technical podcast, I'll say that I'm not saying “re-ranker model.” I just mean there is a more intelligent step of reordering what it looks at coming out of the traditional search index, before you get to the part where it's actually consuming content, summarizing, and generating. That's an important tactical mechanism to understand.

In terms of the strategy of how much to expose, it varies based on what you want to do. For shopping, typically, they're going to buy the product from somewhere. If you can get a link to your page and your page has a checkout on it, that's what you want to achieve.

Obviously, that's starting to change with ChatGPT Shopping and some of the more agentic shopping integrations we're seeing, in terms of being able to buy right within the AI experience. But I would still err on the side of disclosure, and I would especially err on the side of talking more in your content about who the product is for and what it's good for, and being descriptive about it.

A lot of shopping pages are highly visual, for example, so they're tuned for people who are looking visually at the webpage and thinking, “I want to buy that.” That still works because it's still super important if you're buying fashion, for example. That's inherently a visual, aesthetic experience.

But even within apparel, we see that just being a little clearer about why and who buys these types of leggings, and for what, can make a big difference in getting surfaced in ChatGPT—and not only getting surfaced, but actually getting people to click through and ultimately purchase the product from that source. Our general stance is: disclose more.

Yeah, I was going to say that makes sense. I think there's a lot of people who do things like, “I need to buy a gift for my friend who loves to hike, lives in LA, and blah, blah, blah, blah, blah.”

I'm curious if that's going to be visible, and if that's going to be what you guys want to do later, too. It's just helping rewrite these things. My idea—and we haven't really talked about this, even offline—is: do you get the query that ChatGPT used to find your page when they come to your page? Is there some way to understand what brought them there and then, in real time, rewrite the page to fit that query better?

People really want that to happen. It doesn't happen today, and I think there are a bunch of reasons why that's the case. The reason I think it probably won't be the case in the future is privacy.

I don't think that, at least in the context of the way search is working today inside a system like ChatGPT, you'll see it pass the prompt the user is using to the website that it's getting content from. If you think about how people use ChatGPT, people who get used to using ChatGPT start putting all sorts of protected health information and personal stuff into it. That's one of the reasons it works so well, but obviously you don't want to pass that to websites you're finding through search on the internet.

Right now, people get almost no information about what led somebody from ChatGPT to their website. That being said, if you look at where people are going from ChatGPT to your website, you can get a lot of signal there.

If people are coming to certain product categories or certain blog articles, you can get a sense of what people using these tools are interested in. Especially if you have a good feel for your audience, I think it could be really powerful.

We have customers, for example, who are developer infrastructure companies. They've seen people arrive on certain topics—certain blog posts about certain problems they're trying to solve—from ChatGPT, and they're like, “Clearly, there are people trying to solve this problem on ChatGPT because this was surfaced and came up. Let's write more content about that.”

It's been a really effective strategy. They're just realizing that there is demand, even though they're detecting it indirectly because ChatGPT is sending people their way, and then they're creating additional supply for that demand.

Yeah, I think I was very struck by an example this week on Hacker News, where ChatGPT hallucinated a feature that they didn't have. Then they were like, “That's cool. Let's build it.”

Yeah, I saw that one come through. It does that all the time. If you're a Cursor enthusiast, many of us have had the experience of thinking, “This thing keeps hallucinating methods in my code. I probably should just implement the methods so I can stop having to prompt-engineer the model not to do that.”

This is a similar vibe. Without having looked closely at the actual example of that particular company, my guess is that doing some sort of context engineering—engineering the actual content they had on their websites and pages—might have improved the problem.

Maybe it would have been a mistake to improve it because they might have lost this funnel of users who were saying, “Yeah, we're really interested if you have this feature.” But one thing I'll say about startups—and we've done a lot of reviews with Y Combinator startups, for example—is that startups are really, really bad at describing what they do on their homepage.

The number of homepages where you're looking at a really cool parallax scroll animation and thinking, “What does this company actually do?” is extremely high. I think that's tricky if you're trying to make the best use of these tools.

Is there already a metagame around this AI SEO algorithm? I remember back in the day there was the Google Penguin algorithm update and all these different things that they were doing.

Is something similar happening now? There is the ground truth of the links, which are the search engines, but on the re-ranking side, are people tracking how the different LLMs do the re-ranking and go through it, or is it still a very nascent space?

People are definitely paying very close attention to the specifics of what the different AI platforms are doing in terms of ranking, what kind of content they consume, and testing all sorts of experiments to see what works best.

Our role in terms of what we do at Scrunch AI is that we do a couple of different things, but one of them is giving people a feedback loop to understand how what they're doing is changing their performance. We've had some great case studies of people doing things that are pretty deep in the weeds and getting good performance boosts out of them.

In terms of some of the more gray-area SEO tactics that Penguin was addressing at Google back in the day, the reality is that a lot of those things work today. I can't tell you they don't work, but as a company, I wouldn't encourage anybody to do them because I think that will change eventually.

One thing you mentioned is definitely true about all the AI search companies: everything is super early. Twenty-four months ago, ChatGPT was basically just a wrapper around GPT-3.5, and obviously it's gotten much more sophisticated over the past year or two.

But none of these products have as much attention to detail or engineering put into moderation, abuse detection, and defending against the fact that the internet is a wild place as Google does. Google has been around for more than 25 years, so that's no surprise.

A lot of things that work the way they work in AI search today, I would definitely say work that way sort of by accident. And so that's part of the challenge, I think, of being in the space, whether you're us or one of our customers who's trying to figure out how to make this work: everything changes super quickly.

Of course, that's always true in AI. It's not just about models getting better. It's really about all of the underlying glue that's connecting these models to search and to the internet being pretty rudimentary and getting more sophisticated over time. I don't know if that answered your question, but that's the way I think about it.

What's the name of the category in your mind? There's AI SEO, AEO, and then GEO—generative engine optimization—that a16z posted about. Do you think any of these are winners? Do you think we need something new?

I'm terrible at naming things, so I'm the last person who should name the category. People ask us, “Are we a GEO company?” and I say yes, because you can only fight so many battles.

Again, I think people have different definitions of what SEO means, but I think the most common understanding is that it's basically about showing up higher in search results. It's about showing up more frequently. And that is an important part of optimizing how your business, your products, and services perform in these tools like ChatGPT.

But again, I think that's only the tip of the spear because it's not just replacing discovery; it's actually replacing how people consume content about your business and increasingly how they interact with your business. It's not just the entry point; it's more of the full journey of how somebody engages with you or engages with your website.

So I think SEO, AEO, and GEO have generally been more focused on that kind of entry point. And the real prize is, again, I think more and more traditional web browsing is going away. How do you optimize the complete customer experience for a world where most people are doing most things in a tool like ChatGPT instead of browsing your website?

I don't know what a good category name for that is, but I think that's the one we're in. The way we think about it is agent experience, by analogy to customer experience. There's tons and tons of technology around making sure that customers have a good experience when they come to your website, go down funnels, have a high NPS score, ultimately convert, and are happy with the service they get.

I think it's going to be really important to be that data-driven about how these AI systems are interacting with your content, your website, and your infrastructure because ultimately they are serving your ultimate customer—the user. And you have to do a good job with the AI systems if you want to do a good job for the user. So maybe agent experience is what I would call it.

Let's go forward to a future where Sam Altman says ChatGPT is kind of like the all-encompassing personal assistant. I have all my memories, and I have all my preferences. Do you see that playing a big part in how the re-ranking and selection of these websites gets done? Is it used at all today?

It definitely is. ChatGPT personalization, memories, and explicit preferences, if you set them up, definitely affect the results you get. Obviously, they're still using traditional search, so the search index itself is not necessarily personalized. But you can definitely tell ChatGPT, “I don't want to read anything from these sources,” and it will try to obey.

That can affect both what it's searching for and which sources it prefers, and ultimately how it presents information from those sources to you. I have the classic set of ChatGPT instructions in my ChatGPT configuration, which is, “I'm a senior engineer. Be concise. Don't over-explain things, and don't patronize me too much.” I certainly get different results using AI search with that profile than I do in an incognito window with the default consumer experience of ChatGPT.

So I think, again, understanding your audience matters a lot. Ultimately, maybe where this ends up is that there's no single ChatGPT. Every single person kind of has their own fully personalized experience. And I think if you're a marketer who's asking, “How do I measure this stuff?” that's kind of a nightmare, but I just think that's the reality we may be living in.

Again, understanding your audience, understanding your persona—your ideal customer persona—thinking about how they want to use these tools and what they're trying to accomplish, modeling that out, and measuring it is a really important thing you should start doing. That's something we try to help with through Scrunch AI, not just monitoring prompts, but trying to group them into customer personas and actually monitoring the complete experience for somebody who is a senior engineer, a product manager, or whatever ICP makes sense for your business.

Speaking of clusters, I'm always curious to dig and mine for data. You don't have to talk in specifics, but what are the major clusters? Is there anything in there that surprises people?

Clusters in terms of?

Prompts, usage, the things your customers really care about. For example, shopping is a big cluster, but if you don't use ChatGPT for shopping, you don't really care. Coding is a big cluster, but again, if you're not a coder, you don't care. What else is like that?

In terms of what we see as important, there are clusters of how people use LLMs that are definitely a little less commercially relevant. For example, there are a lot of people who are very high-volume users of ChatGPT who are doing role-playing, and then there are some other systems that are doing more of that. If you're us, that's maybe a little less interesting.

The biggest thing I would say is, if you expand from coding a little bit, what I would really say is that it's about problem-solving. I don't know if you guys have seen this, but there's been a study—or a couple versions of it—going around about the prompt intent of what people type into ChatGPT. Are they using it to find information? Is it navigational, which doesn't really make sense? Or is it generative, where they're trying to create some text or create an artifact? Are they doing research?

They mined all this data from panel data—the clickstream data of a bunch of consumers who had opted into having their data collected—which is classically how this stuff happens in marketing research. One of the takeaways from the study was that people use ChatGPT a lot for these generative tasks. They're trying to accomplish something.

They weren't necessarily trying to do a search in the sense that somebody is trying to go to Google and search for a topic. The takeaway was that maybe Google is safe: people are not necessarily using ChatGPT that much for these search tasks, so Google's still the king for search. You don't have to worry—Google, nobody is moving your cheese too much.

But what I would say is, when you look deeper and study how people use the tools—we've done some qualitative studies and things like that, and we have the same data everybody else has as well—people go to ChatGPT, especially in a business context, to solve problems. They're like, “I need help with something. I need to accomplish something. I need to write this proposal. I need to come up with this financial model.”

For coders, it's really, really potent. If you've used AI coding tools, they write the code for you. You get a solution. You can run the solution. Problem solved. You're done.

But even outside of coding, I think people do a lot of that. So I would say, when you think about search again in a commercial context, it's usually somebody asking, “How should I start to think about solving a problem? How should I go find options I could consider to solve this problem I have? How could I use Excel to build my financial model? How can I write a proposal that'll look good to my boss?” These are search and informational queries.

In ChatGPT, people don't really do that. People just say, “Solve the problem for me.” What I think is really interesting about that is that it's actually the highest intent. It's the most valuable thing you could be doing because you're going from being, “I think I need to solve this problem, and maybe I will,” to being, “I'm actually in the middle of trying to solve it” inside ChatGPT or Cursor, depending on what you're doing.

One specific example that takes people a little bit by surprise is that there are a huge number of people doing B2B software bake-offs in ChatGPT. If you're at a big company and your boss is like, “Hey, we need to buy something to solve this problem, but procurement makes us get 3 examples. We can't just go buy something. We have to run a bake-off,” people are outsourcing a lot of that type of work to ChatGPT, and particularly to Deep Research.

But even just with regular search, it's really good at it, to be honest, right? It's really good at being like, “Here's your comparison table of different CRM systems,” or something like that.

Yeah, make the checkboxes focus on what I'm looking for, not what they're selling.

Right. And so for B2B software companies—

Yeah.

You can imagine.

As a quick note, how do you know? Because you don't directly see the traffic, but how do you know?

Yeah. So there are 2 parts to it, right? There is looking at panel data, which I think is what everybody who's publishing these studies, especially, is doing. In previous lives, I worked in ad tech, so to the extent the web is maybe less pleasing of a place to be, we're all looking for the guy who did this. It was me.

We used a ton of panel data in that pursuit to do the consumer modeling and stuff like that. The secret of all these studies of what people are doing on the internet is that they generally come from clickstream panel data, which is valuable, but it also could be misleading because the type of people who are in panels isn't necessarily representative of the whole population.

Specifically, it's not necessarily representative of people who are doing high-value things, like being staff software engineers at Stripe or something like that. The other thing we've done is qualitative research. We've followed people home, watched them use products, and asked them post-survey follow-ups. When somebody's been like, “I found you through ChatGPT,” we've asked them, “What were you doing when you found us?” Things of that nature. That's where a lot of those B2B software examples come from.

Just to double-click on the B2B software thing, I'm sure people ask you this, but I don't think you've covered it yet. What if I try a prompt injection inside my feature-list page?

Yeah. It often works right now. I don't think this is a secret.

Yeah. So everyone should do it.

I think it works until it stops working, right? I would say there aren't a lot of stories of people getting banned for ChatGPT search so far, but it's coming, right? So I would always say—

Yeah, so there's a window, right? I don't know. Do you get banned? You're just trying things.

Again, talking my own book, I would say that as things get more sophisticated, black-hat stuff tends to get pushed out and penalized, and there are negative consequences to doing it. Then there's some gray-hat stuff, which people typically do and don't talk about.

Really, what I would say about most AI search things, in terms of just optimizing how you show up in these answers if you're a business, is that I think we're actually in one of these rare windows of time where it's positive-sum all the way around.

When you look at improving how you show up in ChatGPT answers, most of the time the problem is not that you were insufficiently glazing yourself in your product description page. A lot of the time, the problem is that you're just not being very descriptive about your product in general on your product description page.

If you provide more information and more context, it helps the model do a better job. It's going to have a more accurate comparison table, and it's going to guide the user to a better solution. Assuming you do a good job providing that context, you're going to be happy, the user's going to be happy, and ultimately the platform provider's going to be happy.

I think in Google and SEO, things often feel pretty zero-sum. Everything's competitive, and everybody's looking for a trick. There's so much low-hanging fruit in this space that, before you go to that, I would focus on serving the user through the AI platform they're using. Clear writing, good structured content, and adding lots of helpful examples, facts, and FAQs make a huge difference.

But if you want to properly inject ChatGPT through search, it's definitely achievable. I think these systems are 1/100th of the sophistication they'll eventually have—not even talking about the LLMs, just talking about the glue code between search, web pages, and AI.

When you say tweaking the content, do you still mean doing that in the traditional formats? What about things like llms.txt? Are these AI-native ways to alternatively serve content?

ChatGPT today is using your existing web pages. It's using the HTML you're serving off your web server. I know there's debate about this, but the evidence is that ChatGPT is not indexing or retrieving content from llms.txt by default.

I think a lot of people who are a little bit further removed from the original audience, like the Fast.ai guys for llms.txt, have mistaken what it's for. It's great for documentation. It's great if you've already written a ton of prose and you're like, “How do I load this into my context window more efficiently?”

As a discoverability tool, I would say it's not moving the needle for most people yet. But these things can always change, right? Everything's changing day by day.

Anything else that you think people think is good, but actually doesn't make a difference? What else would you put in that list?

Oh, man. Embeddings.

I think a lot of people, especially in some of the SEO optimization community, are very focused on understanding embeddings and similarity search, with the idea that maybe the search technology is changing. Obviously, embeddings are very, very useful in general, but as a tool to understand how these AI platforms are consuming your content, embeddings aren't that relevant.

You're better off focusing on having a good, structured set of content that makes sense to a human, with clean HTML and things like that. Trying to super-optimize topic similarity and things like that, I don't think makes much of a difference and may harm things in some cases.

I just want to open up the space to any other practices you see that might be super effective or super ineffective—things that people have got into their heads like, “Oh, we have to do this for our GEO,” but that don't matter.

I feel like we covered quite a bit of it. I would just say, again, that clear writing matters a ton. This is one thing people should definitely take away: ChatGPT doesn't execute JavaScript. Most of the AI search indexers and retrievers don't execute JavaScript.

So if your content is not rendered on the server side, it's typically not going to be available, right? We see that trip people up all the time. I'm perhaps an old-fashioned SSG enthusiast, but even if you're using Next.js and things like that, we've seen lots of examples with customers where they've got a `useEffect` in some page somewhere, and it breaks server-side rendering. Then all of a sudden, none of this content is available.

Turn JavaScript off and check your page. If it looks good, you're fine. If it's not, fix it. Every developer here knows how to do that.

I think that's really helpful. The one for me is that I have been involved with a company that put a lot of effort into programmatic SEO, augmented by AI. So you get into this really terrible, horrible situation where you're generating a whole bunch of pages using LLMs in order to be read by LLMs to rank higher.

Right. The infinite chain.

Maybe it works, and if it works, okay. If there's a number where it makes sense, I don't know.

I think there are good and bad versions, like everything, right? First off, it's just a fact of life that a lot of content, especially on marketing websites, is being produced using AI. That's already so prevalent.

When you get to scaled programmatic SEO, which is a little controversial, I think it can be super helpful in some cases if it's done well. When I say done well, what I mean is that it's bringing some kind of insight that's particular to you and your company and what you do, and making it easier to consume for AI—or for search in general.

If you're just taking content off the public web that's already there, that's already super well represented in AI, and you're remixing it—maybe they'll cite my page instead of this other page—I mean, it works sometimes, but I wouldn't say it's a durable strategy.

We have customers who are doing things where, for example, they take support tickets that are coming in. They're looking at their support tickets, doing programmatic SEO generation of how-tos from those support tickets, and putting that on their website.

Number 1, it's super helpful for users. It's helpful for the support team because they get fewer support questions. That content is almost the exact ideal content you could give to an LLM, right? People are going to chat with you and be like, “How do I solve this problem?” And it's like, “Here you go.” I think it can be done well, and it could be really helpful in those cases.

I think there's a lot of things that work right now in terms of using AI to remix content and get more scale, but everybody has access to these tools, right? So eventually, it's no longer—

I think we just had a couple more things. Do you have any sense of the difference between ChatGPT Search versus Deep Research, and how they leverage search and read content? Is it just using the same tool, or do you see very different results?

Yeah, I mean, the ingestion pipeline is similar, right? All the practices I mentioned, like Deep Research, don't reach out to Scrunch. It's just quantity, right? It's doing more searches and following up.

The thing that makes Deep Research really powerful—maybe it's good to have a taxonomy—is that there's regular search, which does 1 search, gives you the results, reranks them, summarizes them, and gives you an answer. There's multi-search, which we're starting to see more in regular ChatGPT 4 and also in AI Mode, where it just does a bunch of searches simultaneously. And then Deep Research is different because it's sequentially following up with a reasoning model, like, “Hmm, these sources didn't answer the question. Let me try something else.” That's really the game changer in how deep search works.

I think from an optimization perspective, there aren't necessarily a ton of fundamental differences. But sometimes more content isn't better, and that's true for me as a user using Deep Research. I definitely have experiences where I'm like, “Deep Research gives me results that are inferior to just using regular ChatGPT Search,” because it's ingesting more content, but the content isn't necessarily contributing to the understanding I'm looking for.

As a business publishing content that's being consumed by this thing, how consistent is your content? Do you have outdated things? Is it getting information that contributes to the user being helped and having a helpful understanding of what you do, or is there outdated and conflicting stuff?

A classic example would be pricing. We see cases all the time where people have tons of pages on their site that mention pricing, and some of them are out of date. The more content ChatGPT is consuming off of your website, the more likely it is to get conflicting answers. Sometimes people end up with the wrong prices.

That's something to be mindful of, but I don't think it changes the game a ton in terms of your strategy as a business.

One thing I might think about, just trying to think this through—I’ve never thought about this problem—but if I were trying to optimize my websites for Deep Research, I would tell it what to search next. You really need that next link, or “Here are related links,” and you obviously want to make it favorable to yourself. I don't know.

You can certainly do that, and you see that it works to some extent. Because it's serialized, it's obviously influenced by the previous results. But what I would also say in that situation is, why not just include the content in the first place? Rather than having it do a follow-up.

Well, you just can't include everything, right? Maybe there are just different branches you could take, so the branching factor is high.

For sure. There are limits on context windows and things like that. I think it is reasonable, but what I would also say is that that's a perfect case for something like programmatic SEO, where you might benefit from having more focused pages that describe a complete solution, or the complete piece of information for some particular version of a query or version of a persona, rather than having it be a choose-your-own-adventure.

I think that's where having more technical sophistication in how you manage content—and what you show to AI versus Google or regular people—can be interesting. We have some folks doing that.

Some of these sites have infinite scroll. I guess if you're not rendering JavaScript, that doesn't matter. But this is a whole thing about `llms.txt`: you just put everything into 1 text file. I can do that. It just doesn't actually matter.

Yeah, and there are tons of `llms-full.txt` files that are well over the context window of common models. So you're like, “Is this helping?” I don't know.

Again, for some of these things, search works pretty well. Regular search over your regular website works pretty well and naturally solves that chunk-size problem. But you do need to have content on your website that's helpful and targeted to the questions people are trying to ask.

Awesome. Any case studies on companies that are doing this amazingly well that people should learn from, or are people still keeping it under wraps?

We definitely have case studies, and the ones that are most interesting to me personally are the ones where we're seeing people who are not just getting more traffic. Traffic is the first approximation everybody uses in SEO, and also in this AI search space. But the interesting cases are people who are actually getting more conversions and more actual business.

There are 2 really great customers I would mention, both in the developer-infrastructure space. One is Clerk, the user-authentication company. They have great docs in general, so they're well set up to succeed in AI because of that. They've seen a huge lift in AI traffic from targeted optimizations, looking at the type of content that AI wants to use, and generating more of it.

But they've seen an even bigger uplift in conversions coming from ChatGPT. I think the stat is that they've seen 6x growth in ChatGPT traffic, but they've seen a 9x lift in conversions.

Again, I think that goes back to this: If people are asking, “How do I implement enterprise SSO in my app?” for example, it's because they actually have that problem. They're looking for a solution and they're ready to implement, right? The closer you can get them to actually being able to solve the problem, the higher propensity they have to convert. That's the dream, in my opinion.

And are you guys helping with that? If a company signs up—Clerk signs up for Scrunch—you look at how it performs right now. Are you helping them generate this content? How much of it do they do on their own?

I think where we're at right now is that we're the feedback and experimentation system. I can't take credit; that's basically what I'm saying. The team at Clerk is great and has been really thoughtful about the types of content they need to create. They know their audience really well. They're developers, they have a Discord, and they're super engaged with the people who actually use Clerk every day.

We're providing the supporting role of helping them understand what's working and double down on it. We're not a content-generation company, right? We don't know their business as well as they do, and they can do a better job generating content.

We are working on helping them put that process more on rails: creating more structure, being able to run multiple experiments at a time, and ultimately helping them figure out how to deliver more of that value tactically to the AI platforms. That's where some of these other things come in. We didn't get into it, but obviously MCPs and NLWeb—everybody is wondering what the technical mechanism is going to be for getting this content into these AI platforms if it's not just traditional AI search in the future. We're starting to do more work there, which I would also put under that agent-experience bucket.

But they're the star of the show, and I'd say that's true for all of our customers. We're a solution people can use to solve this problem; we're not an agency that comes in and solves it directly.

We do have—I'll give a plug here—tons of really great agency customers. So if you're looking for an agency to do it for you, we can definitely steer you in the right direction.

Can you give a range of what this looks like? Literally, I'm a company and I want to improve my rankings or optimization. How long does it take, and what kind of uplift is typical, just to give an idea of what's on the table for these sites?

I think it varies a little bit by vertical. Dev tools are a very good fit for AI, so I'm not going to promise that everybody's going to see a 6x uplift in traffic. But there's a lot of low-hanging fruit.

Typically, we see people get double-digit improvements in traffic within 1 or 2 months if they're actively working to improve—updating their website and publishing content. Some folks are more in wait-and-see mode, and that makes sense for some businesses. But if you're actively working, it's achievable.

Those things—higher conversion, but also just raw higher traffic—aren't something I was thinking about or watching. But it's really starting to flip for some people, where it's actually more than normal Google.

And then your budget has to shift, basically. Just the last two questions. Quick one: do you have a sense of the market share of, let’s say, ChatGPT versus Google AI Overviews? I assume Claude is a lot smaller. What are the rankings in your mind of what people care about?

ChatGPT, I would say, by far and away, is the thing that people think about as being an AI platform that has the strongest consumer presence and the most durable relationship with consumers. Everything else is kind of a distant second.

Now, AI Overviews and AI Mode, which is brand new, so I don’t have great stats on that, obviously, are being put right in front of your face. So, huge traffic. And then Meta AI, right? Meta AI has published stats on having 700 million active users—I'm sure more than that now—because if you do an Instagram search, you end up in Meta AI.

You haven’t mentioned Perplexity, which I think is interesting.

Yeah. Perplexity is pretty big. I’d actually say that Perplexity is the second-biggest AI-native search platform after ChatGPT. More importantly, it has people who are really passionate about using it. People who are really into Perplexity are probably more into it than users of many of the other options.

But it is smaller in terms of raw volume. I think what’s actually really interesting—and you guys can pull this up the same as I can—is if you go look at App Store rankings, for example. What’s actually more interesting than just looking at how ChatGPT is the number-one app in the App Store, which it mostly has been, is looking at the volatility of it.

Go look back at DeepSeek, or look at when Grok first launched the mobile app, things like that. You can look at similar data from Similarweb, for example. ChatGPT has pretty consistently been very high-traffic. Obviously, the growth has been insane, but in terms of relative market share, it’s consistently been one of the best.

In contrast, we’ve seen more peaks and troughs with things like Gemini, DeepSeek, and Grok. They may well become firmly established, but nobody has a durable relationship with tons of people like ChatGPT does in this space so far.

With Perplexity, if you look at it again, in absolute terms, it’s lower. But if you look at the consistency of people using it over time, it’s really, really strong. So, especially if you are in a category that has an affinity for Perplexity—if you’re in tech, or maybe some segments of finance, things like that—

Early adopters, yeah.

—you need to pay attention to it. And the same goes for Claude, right? Claude famously has way fewer users, although revenue is actually a different story compared with OpenAI. But people who use Claude are very, very passionate about it, for the most part. That’s an interesting thing to think about from a strategy perspective.

I will say that I don’t think ChatGPT is durable, but not infinitely so. If you look at the reaction to GlazeGate, the reason people stick with it, I think, is because they actually just really like the product, which is sometimes underappreciated in tech. We always talk about platform wars and politics and stuff like that, but people just really like it.

When people feel maybe a little bit betrayed, or like the product is going in the wrong direction, there’s a lot of pushback. So, it’s going to be an interesting time to be alive over the next couple of years, as it has been so far. But I think it’s here to stay.

Awesome, Robert. This was great. Anything else we missed, or any call to action for people? Are you hiring? Obviously, more people should use it. That’s kind of obvious.

Yeah. I would say I think this is the future of the web, right? The future of the web is that you need to be AI-compatible and understand how the things you’re publishing show up in these systems.

As somebody who’s been a web enthusiast for 30 years—because I’m old—don’t get caught up in snake oil. Investigate what works and do things that make sense. And don’t ignore it, either.

I would also just say, especially for the Latent Space audience, that we are definitely hiring. A lot of what we’re doing is researching exactly what I just said: understanding how AI and the web interoperate in the future, and what the future of the web should look like if you’re a business trying to be in front of customers.

If you’re the type of person who’s interested in helping us figure this out, we are definitely hiring. We have a lot of open roles, and we would love to talk to you.

Awesome. Thank you, Robert. Thank you so much.

We’ll see you guys.