用 Neural PageRank 和 500万美元 H200 击败 Google 搜索——Exa.ai 的 Will Bryk

Will, you’re now CEO and co-founder of Exa AI, formerly Metaphor Systems. What’s your background and story?

I’m the CEO of Exa. I’ve been doing it for 3 years. I guess I’ve always been interested in search, whether I knew it or not. Since I was a kid, I’ve always been interested in high-quality information. Even in high school, I wanted to improve the way we get information from news. Then, in college, I built a mini search engine.

With Exa, it’s kind of fulfilling the dream of actually being able to solve all the information needs I wanted to solve as a kid. I would say my entire life has been rotating around this problem, which is pretty cool.

What did you enter YC with?

We entered YC with, “We are better than Google at Google 2.0.”

What makes you say that? That’s so audacious to come out of the box with.

You have to remember the time. This was summer 2021, and GPT-3 had come out. Here was this magical thing that you could talk to. You could enter a whole paragraph, and it understood what you meant. It understood the subtlety of your language.

Then there was Google, which felt like it hadn’t changed in a decade because it really hadn’t. You would give it a simple query like “shirts without stripes,” and it would give you a bunch of results for shirts with stripes. Google could barely understand you; GPT-3 could.

The theory was: What if you could make a search engine that actually understood you? What if you could apply the insights from LLMs to a search engine? It’s really been the same idea ever since, and we’re actually a lot closer now to doing that.

Did you have any trouble making people believe you? Obviously, there was a Sam Altman–YC overlap. Was YC pretty AI-forward even in 2021, or was it nothing like it is today?

There were a few AI companies, but we were definitely bold. I think VCs generally like boldness. We had some AI background, and we had a working demo, so there was evidence that we could build something that was going to work.

The fundamentals were there. People were talking about how Google was failing in a lot of ways, so there was some conversation about it. But AI was not a big thing at the time.

Before we jump into Exa, any fun background stories? I know you interned at SpaceX. Any Elon stories? I know you were at Zuck’s as well, and did robotics at Harvard. Is there anything you saw early that you thought was going to get solved, but maybe still isn’t solved today?

Lots of things like that. I never really learned how to drive because I believed Elon that self-driving cars would happen. It did happen, and I take them every night to get home, but it took 10 more years than I thought.

Do you still not know how to drive?

I know how to drive now. I learned 2 years ago. It would have been great to just never need to know.

I was obsessed with Elon. I worked at SpaceX because I really just wanted to work at one of his companies. I remember they had a rule that interns couldn’t touch Elon.

Was that physically, or was it more like you couldn’t interact with him?

Physically, apparently. He’s changed a lot, but his companies are amazing.

What if you beat him in Diablo 2? Diablo 4, you know?

Maybe.

When I jump into Exa, I know there’s a lot of backstory. It used to be called Metaphor Systems, and you’ve always been a prominent company, at least in research circles. I’m curious how Metaphor got its initial aura.

You launched with very little. There was this big splash image of “This is Aurora,” or something like that. I remember thinking, “What is this thing? The vibes are good, but I don’t know what it is.” It felt much more consumer-facing than what you are today. Would you say that’s true?

No, it’s always been about building a better search algorithm. The vision has always been perfect search. If you do that, we’ll figure out the downstream use cases later.

It started with this fundamental belief that you could have perfect search over the web. We could talk about what that means, but the initial thing we released was really just our first search engine. We were trying to get it out there, kind of like when OpenAI released ChatGPT. I don’t know how much of a game plan they had; they just wanted to get something out there as a research preview.

Similarly for us, we moved from a research company to a product company. We started as a research endeavor, with a clear-eyed view that if we succeeded, it would be a massive business. That’s basically what happened. There are actually a lot of parallels between Exa and OpenAI. I often say we’re the OpenAI of search, because we’re a research startup that does fundamental research into making AGI for search, in a way, and then we have all these business products that come out of that.

I want to ask a little bit more about the Metaphor site, and then we can go full Exa. When I first met you, which was really funny because I literally stayed in your house in a very historic Hayes Valley place, you said you were building a link-prediction foundation model.

I think there’s still a lot of foundation-model work within Exa today, but what does that even mean? I can’t be the only person confused by that. There’s a limited vocabulary of tokens. Are you telling me the tokens are the links?

What we meant by link prediction is that you are literally predicting the links. Given some text, you are predicting the links that follow.

That refers to the training procedure. We find links on the web, take the text surrounding the link, and predict which link follows. Similar to Transformers, where you’re trying to predict the next token, here you’re trying to predict the next link. You hide the link from the Transformer.

Imagine some article where someone says, “Check out this really cool aerospace startup,” and then they say “spacex.com” afterward. We hide the “spacex.com” and ask the model, “What link came next?”

By doing that many, many times—billions of times—you can actually build a search engine. At query time, you type in a query like “really cool aerospace startup,” and the model will try to predict the most likely links.

There are a lot of analogies to Transformers, but to make this work, it does require a different architecture than a Transformer. It’s Transformer-inspired.

What’s the design decision between doing that versus extracting the link and its description, embedding the description, and then searching for the closest description?

You’re doing a similar thing in a way. Based on this description, what’s the closest link for it? One approach is predicting the link; the other is extracting the link and the description and then searching for the closest description.

Predicting the link here refers to a document. One confusing thing is that you’re not actually predicting the URL itself. That would require the system to have memorized URLs. You’re getting the actual document. A more accurate name could be document prediction.

This was the initial base model that Exa was trained on, but we’ve moved beyond that. Similar to how, to train a really good language model, you might start with a self-supervised objective of predicting the next token from random data on the web, you then want to add synthetic data and supervised fine-tuning to make it controllable and robust.

We just had a flow from Lindy there. Lindy started hallucinating and rolling YouTube links instead of a support guide. Interesting.

Around January, you announced your Series A and renamed the company Exa. I didn’t like the name initially, but it’s grown on me. I liked Metaphor, but apparently people can spell “metaphor.” What would you say are the major components of Exa today?

I feel like it used to be very model-heavy. Then, at the AI Engineer conference, Shas gave a really good talk on the vector database that you guys have. What are the other major moving parts of Exa?

Exa overall is a search engine, and we’re trying to make it a perfect search engine. To do that, you have to build lots of different systems.

First, you have to find the URLs to crawl. That’s connected to the crawler, but you have to find URLs, crawl those URLs, and then process them with some model. It could be an embedding model or something more complex, but you need to take the documents you gather and put them into some processed index.

Then you have to serve that index at high throughput and low latency. That’s the vector database. It’s teams of hundreds or maybe thousands of people at Google, but for us, it’s typically 1 or 2 people on each system.

It’s the crawling system, the AI-processing system, and the serving system.

Can you explain the meaning of Exa? Is it 10^16 or 10^18?

10^18.

What’s the story?

Exa means 10^18, in stark contrast to Google, which is 10^100. We have these awesome shirts that say, “10^18 is greater than 10^100.”

It’s great because it’s provocative. Every engineer in Silicon Valley says, “What? That’s not true.” Then you ask them what it actually means, and the creative ones recognize it.

10^18 is better than 10^100 when it comes to search because with search, you want the actual list of things that match what you’re asking for. You don’t want the whole web. You want to filter everything humanity has ever created down to exactly what you want.

The idea is smaller is better. You want the best 10^18, not 10^100. One way to say this is that Google often says at the top, “30 million results found.” That’s crazy, because if you’re looking for startups in San Francisco that work on hardware, there aren’t 30 million results. What you want is “325 results found,” and those are all the results.

That’s our vision. It gives you exactly what you ask for.

We’re recording this ahead of your launch. We haven’t released it, and we haven’t figured out the name of the launch yet, but what’s the product that you’re launching?

We’ve basically developed the next version of Exa: the ability to get a near-perfect list of results for whatever you want.

You could make a complex query to Exa, for example, “startups working on hardware in San Francisco,” and get a huge list of all the things that match. Our goal is that if there are 325 startups that match, we find all of them.

This is a new experience that’s never existed before. I don’t know how you would do that right now with current tools. You could apply the same technology to anything. Let’s say you want to find all the blog posts that talk about swyx’s podcast that have come out in the past year. I don’t know how you would get that full, comprehensive list.

How do you know that it’s complete? You’re saying there are only 325 results, or 30 million results, or whatever. Then how do you do the semantic understanding? “Working on hardware” might not use the word “hardware.” It might use “robotics,” “wearables,” or something else.

One aspect of this is subjective. We’ll provide parameters to the user to gauge the threshold—to say, “This is the cutoff. This is actually not what I mean.”

There needs to be a feedback loop. It might give you a few examples, and you say, “Yes, I want those.”

You’re creating a classifier on the fly, but that’s ultimately how you solve the subjectivity problem. Subjectivity and comprehensiveness are 2 different problems.

To solve comprehensiveness, you have to put more compute into the query, into the search, until you get full comprehensiveness. Not all queries are made equal. Some queries, like the blog-post query, might require scavenging throughout the whole web in a way that simply requires more compute.

At some point, there’s some amount of compute where you will be comprehensive. You could imagine running GPT-4 over the entire web and asking, “Is this a blog post about swyx’s podcast?” That would work. It would take a year and maybe cost $1 million or more, but it would work.

Given sufficient compute, you can solve the query. It’s really a question of how comprehensive you want it to be, given your compute budget.

Does that mean you have some kind of compute budget that I can specify?

Yes.

What are the upper and lower bounds?

This is something we’re still figuring out. There’s a new paradigm of variable-compute products. How do you specify the amount of compute? What happens when you run out? Can you keep going? Can you put in more credits to get more?

For some of the really large-compute queries, this can get complex. One thing we do is give you a preview of what you’re going to get, and then you can spin up a much larger job to get way more results. There is some compute limit, at least right now.

People think of searches as taking 500 milliseconds because we’ve been conditioned to have search take 500 milliseconds by search engines like Google. No matter how complex your query to Google is, it takes roughly 400 milliseconds.

But what if searches can take a minute, 10 minutes, or a whole day? What can you then do? You can do very powerful things. You can write a search, go get a cup of coffee, and come back to a perfect list. That’s okay for a lot of use cases.

The use case closest to me is venture capital. Eight years ago, I built one of the first data-driven sourcing platforms. We looked at GitHub, Twitter, Product Hunt, and all these things, and evaluated interesting companies.

If you think about some jobs people have, an analyst at a venture firm’s job is literally just to make a list of interesting companies and then reach out to them. How do you think about being infrastructure versus being a product? You could say, “This is a product to find companies,” or “This is a product to find things,” versus offering more of a blank canvas that people can build on top of.

We are a search infrastructure company. We want people to build on top of us and build amazing products on top of us.

At the same time, with this product, we’re trying to make it really easy for users to log in, put in some credits, and get amazing results right away, without having to build an API integration. We’re doing both. We want people to integrate this into all their applications, and we also want it to be easy to use.

It’s very similar to OpenAI. They have an API, but they also have a ChatGPT interface, so it’s easy to use and you can build it into your applications.

I’m still trying to wrap my head around all the implications. So many businesses run on information arbitrage: “I know this thing that you don’t,” especially in investment and financial services.

Now, all of a sudden, you have these tools where everybody can get the same information at the same time and at the same quality level as an API call. It changes a lot of things.

I think what we’re grappling with is: What is the world like if knowledge is solved? If any knowledge request you want is right there on your computer, it’s kind of different from when intelligence is solved.

I’ve written before about the difference between superintelligence and superknowledge.

The distinction between intelligence and knowledge is actually a pretty good one. They’re definitely connected and related in all sorts of ways, but there is a distinction.

You could have a world—and we are going to have this world—where you have GPT-5-level systems and beyond that can answer any complex request, unless it requires some information. You could say, “Give me a list of all the PhDs in New York City who have thought about search before,” and even this superintelligence is going to say, “I can’t find it on Google.”

That’s crazy. We’re literally going to have superintelligences that are using Google. If Google can’t find the information, there’s nothing the superintelligence can do. It can’t find it.

If you also have a superknowledge system—where you can get whatever knowledge you want—then you can pair it with a superintelligence system. The superintelligence will never be blocked by a lack of knowledge.

You told me this when we had lunch. I forget how it came up; we were talking about AGI and whatnot, and you said that even AGI is going to need search.

We’re referencing a blog post you wrote, “Superintelligence and Superknowledge.” I’ll refer people to that. This is a discussion we’ve had on the podcast a couple of times.

A lot of model weights are just memorizing facts. Some of those facts might be outdated, and some might be incomplete.

I wonder: Is there a maximum language-model size that will be the intelligence layer, and then the rest is just search? Maybe we should always do search, and the workhorse model is just a 1-billion- or 3-billion-parameter model that drives everything.

I believe this is a much more optimal system: a smaller LLM that’s really just an intelligence module, making a call to a search tool. The opposite would be that the LLM is so big that it can memorize the whole web. That would be way too big. It’s not practical, and at least right now, it’s not possible to train.

Karpathy has written about how he could see models moving more toward intelligence modules that use various tools.

For listeners, that was Karpathy on the No Priors podcast. We also talked about this on the podcast with Shinu and Harrison Chase.

I do see that as a more efficient world. You could have GPT-4-level systems calling search, but because of the cost of inference, it’s better to have a very efficient search tool and a very efficient LLM. They’re built for different things.

I’m curious because it’s still so audacious that I don’t want to undersell it: You’re building a search engine. Where do you start? How do you approach it? Are there any reference papers or implementations that really influenced your thinking?

I don’t even know where to start apart from crawling a bunch of the web, but there has to be more insight than that.

There’s more insight, but I’m always surprised by what happens when you have a group of people who are really focused on solving a problem with the tools available today. In software, there are all sorts of creative solutions that simply haven’t been thought of before, particularly in information retrieval.

A lot of the techniques are very old. I know how Google and Bing work, and they’re just not using new methods. There are reasons for that. Google has to be comprehensive over the web, return results in 400 milliseconds, and not cost too much. Those constraints limit what kinds of algorithms they can deploy at scale.

They end up using a limited, keyword-based algorithm. Google was also built in 1998, when we didn’t have LLMs or embeddings, so they never thought to build those things. Now they have this gigantic system built on old technology.

A lot of the information-retrieval field thinks in terms of that framework. We came in as newcomers, thinking, “Here’s GPT-3. It’s magical. Obviously, we’re going to build search using that technology.”

We never really thought about using keywords. We’re neural all the way. We’re building an end-to-end neural search engine. That framing makes us ask different questions and pursue different lines of work. There’s a lot of low-hanging fruit because no one else is thinking about it. We’re on the frontier of neural search at web scale.

Maybe let’s spell this out, since we’re already on the topic. The elephants in the room are Perplexity and ChatGPT Search. How would you contrast your approach with theirs, based on what we know about how they work?

There are a few of these systems now. They basically rely on traditional search engines like Google or Bing, and then combine them with LLMs at the end to output a paragraph answering your question.

Perplexity and ChatGPT Search might have their own crawlers. There’s an important distinction between having your own search system and having your own cache of the web.

Imagine you crawl 100 billion URLs and create a key-value store mapping each URL to its document. That’s technically called an index, but it’s not a search algorithm.

To search, you need to take 100 billion or more documents and, given a query, instantly return the list of results that match. That’s a much harder problem. It’s not hard to use DynamoDB to map URLs to documents; it’s very hard to build a search engine over hundreds of billions of pages.

When you make a query to ChatGPT Search, for example, it might use the Bing API to get a list of results. It can then use its cache of the content of those results. But it’s not necessarily a search engine built from scratch over hundreds of billions of pages.

Very few entities on the planet have built their own search engine from scratch. There’s Google, Bing, Yandex, and not many others. Most companies are not crazy enough to build a search engine when they can simply use traditional search APIs.

Google had PageRank. Is there an LLM equivalent, or anything you’re working on that you want to highlight?

The link-prediction objective can be seen as neural PageRank. You’re predicting the links people share. If everyone is sharing some Paul Graham essay about fundraising, our model is more likely to predict it.

Inherent in our training objective is a sense of high canonicity and quality. It’s more powerful than PageRank—strictly more powerful—because people might refer to that Paul Graham essay in 1,000 different ways. Our model learns all the different ways someone refers to the essay while also learning how important the essay is.

It’s like PageRank on steroids.

To me, that’s the most interesting thing about search today. With Google and similar systems, it’s mostly domain authority. If you search for any AI term, you get these SEO-slop websites with a bunch of things in them.

How do you think about more timeless content? Take the “Founder Mode” essay. It gets shared by a lot of people, but there might be many other essays that are also good and simply don’t get as much traction. Even if the people who share them are high quality, you don’t necessarily have that people authority. How do you solve that? How do you decide what’s worth bumping up?

You have a lot of control over the training data. You can make sure the training data contains high-quality sources.

If your training data is a bunch of garbage, your predictions will be poor. Our model will match the distribution of the training data it was trained on. There are lots of ways to tweak the training data to include the high-quality content we want.

The slop returned by traditional search engines like Google and Bing is then transferred into LLMs in systems like ChatGPT Search. If slop comes in, slop will go out.

That’s another way we’re different from traditional search engines. We want to give you the highest-quality results and give you full control. If you don’t want slop, you don’t get slop. If you put an LLM on top of that, as our customers do, you get higher-quality output.

I use Exa Search very often, and it’s very good. swyx uses it too.

The slop is everywhere, especially when it comes to AI, investment, and all of these areas where it’s valuable to be at the top. This problem is only going to get worse.

What else is in the toolkit? You have the Search API, Exa Search as the web version, and the list builder. I think you also have web scraping. People want to search and then scrape, right? Is that the use case?

A lot of our customers are building AI applications on top of Exa. They don’t just want a list of URLs; they want the full content—cleaned, parsed, in Markdown, maybe chunked, whatever they want. We give it to them.

Getting the URLs and instantly getting the content for each URL is very powerful. You can do it for 10, 100, or 1,000 URLs.

That was the first thing I asked you for when I tried using Exa. The funny story is that when I built the first version of Exa, we happened to store the content—the first 1,024 tokens—because I kept it around for debugging purposes.

When people started asking for content, it was actually pretty easy to serve it. Then we did that, and Exa took off because the content was so useful.

There are other players in this space. Jina is in this space, and Firecrawl is in this space. There are a bunch of scraper companies. Scraping is just one part of your stack, but you might as well offer it since you already do it.

It makes sense to have an all-in-one solution. We’re building the best scraper in the world.

Scraping is a hard problem. It’s easy to get a good scraper, very hard to get a great scraper, and super hard to get a perfect scraper.

Do you have a perfect scraper?

Not yet.

The web is increasingly closing itself off to bots and scrapers: Twitter, Reddit, Quora, Stack Overflow, and others. How are you dealing with that? How are you navigating those issues? OpenAI is just paying them money.

It definitely makes things harder for search engines. One response is that there’s a lot of value in the long tail of sites that are open. Even searching over those gets you most of the value.

There is content that’s increasingly unavailable, so you can get it through data partnerships. The bigger we get as a company, the easier it is to make those partnerships.

I see the future as a world where data producers and content creators make partnerships with the entities that find that data.

Are there any other fun use cases that people might not be thinking about?

There are so many. Dating is a really interesting application of search that’s completely underserved. There are a lot of profiles on the web, and a lot of people want to find love.

You could say, “Give me people within these age boundaries, with this education level, in this location.” You might want to find a partner who matches a particular education level and has written about certain topics. If you could get a list of all people like that, I think you could unblock a lot of people.

This is definitely a very Silicon Valley view of dating, and I’m well aware of that. It’s just an interesting application of better search. I’d love to meet an intellectual partner who shares a lot of my ideas. You could do that through better search.

What about Jeff? Jeff has already set me up with a few people, so Jeff is my personal dating expert. My mom’s actually a matchmaker, and she’s gotten me married.

No kidding.

Search is built into dating.

Other than dating, I know you’re having some success in colleges. I’d love to map out more use cases so listeners can use those examples to think about what they could use Exa for. It’s such a general technology that it’s hard to pin down what you should use it for and what kinds of products you can build with it.

There are many applications of Exa, and we have many companies using us for a diverse range of use cases.

One interesting example is a large customer using us to build a writing assistant for students who want to write research papers. Exa searches for a list of research papers related to what the student is writing. Then the product has an LLM that summarizes the papers.

It’s basically next-word prediction, prompted by 20 research papers that Exa returned.

It’s literally doing their homework for them.

I guess. The key point is that research is hard, and you need high-quality content as input.

We’ve had Elicit on the podcast, and it’s pretty similar. They focus almost exclusively on research papers and that research use case.

I just wanted to spell out more of the big verticals.

Finance is a big vertical. We talked about finding companies. It’s useful for VCs who want to find a list of competitors to a specific company they’re investigating, or just a list of companies in some field.

One VC told me that he and his team were using Exa for 8 hours straight, for many days on end, doing lots of different queries: “All the companies in AI for law,” “all the companies in AI for construction,” and so on. You simply can’t find that information with traditional search engines.

Finding companies is also useful for sales. If we want to find a list of writing assistants to sell to, we use Exa ourselves. That’s actually how we found a lot of our customers.

Recruiting is another great use case. We can get a list of people who have thought about search and reach out to them.

When you say “thought about search,” are you thinking about LinkedIn and Twitter, or just blogs and things people have written?

It’s pretty general. Ideally, Exa would return blogs written by people who have thought about search.

So if I don’t blog, I don’t show up in Exa? I have to blog?

An incentive for people to blog. If you’ve written about search on Twitter, we index a bunch of tweets, and we should be able to surface that.

This is something I tell people: You have to make yourself discoverable to the web. It’s called learning in public, but it’s even more imperative now. Otherwise, you don’t exist at all.

Search engines completely influence the internet itself. They have downstream effects and influence what people choose to create. Because Google is a keyword-based search engine, people are incentivized to keyword-stuff and create things that match a lot of keywords, which isn’t very high quality. Exa optimizes for high quality and actually matching what you mean, so people are incentivized to create high-quality content that they know will be found by the right person. If I’m a search researcher and want to be found by Exa, I should blog about search and all the things I’m building. The search engine will influence the downstream internet in all sorts of amazing ways.

Whatever the search engine optimizes for is what the internet looks like. Are you familiar with the term “Mahanism”?

No, what’s that?

It’s the concept that first we shape tools, and then the tools shape us. There’s a reflective connection between the things we search for and the things that get searched. Once you change the tool that searches, the things that get searched also change.

There was a clear example of that with 30 years of Google.

Exactly. Google has trained us to think of search. Google is search in people’s heads. One hard part about Exa is ripping people away from that notion of search and expanding their sense of what search could be.

I searched Exa right now for “people writing about learning in public.” It didn’t return you because it thinks about “learning in public” as public schools and focuses more on that.

How do I get to swyx? If you’re in these subcultures, I don’t think this would work in Google either. When there’s a movement, there’s confusion about what you mean. I’m using a term that I didn’t invent, but I’m trying to take over, and there’s so much associated with that term already that it’s hard to overcome.

The right solution is to specify more clearly what you mean.

The right interface to search is actually an LLM. You should be talking to an LLM about what you want. The LLM translates its knowledge of you—or its knowledge of what people usually mean—into a query that Exa uses.

You’ve called that auto-prompting, right?

Yes, but it’s a very light version of that. The real answer is that it’s the wrong interface.

Very soon, the interface to search, and really to everything, will be LLMs. The LLM will have full knowledge of you. We’re building for that world. We’re scaling to where the puck is going to be.

Since we’re moving to a world where LLMs interface with everything, you should build a search engine that can handle complex LLM queries—queries that come from LLMs.

You’re probably too lazy to write a whole paragraph explaining exactly what you mean by a word. I’m too lazy, too. But an LLM isn’t lazy. It will produce a paragraph or more explaining exactly what it wants.

You need a search engine that can handle those paragraphs. Traditional search engines like Google and Bing are designed for humans typing keywords. If you give a paragraph to Google or Bing, they completely fail. Exa can handle paragraphs, and we want to handle them better and better until it’s perfect.

What about opinions? When you think about the list product, do you think about just finding entries, or do you think about ranking entries?

On Lindy, I’ve been building a bot that gives me the top fantasy-football waiver pickups every week. Every website has different opinions. One says to pick up these 5 players, another says to pick up those 5 players.

When you’re making lists, do you also want to rank them and tell people what’s best, or are you mostly focused on surfacing information?

There’s an important distinction between filtering to things that match your query and ranking based on your preferences.

Filtering is objective: Does this document match what you asked for? Ranking is more subjective: What is the best?

It depends on what we mean by “best.” First, the table stakes are getting filtering into a perfect place, where every document matches what you asked for. No search engine can do that today.

Then ranking has all sorts of interesting possibilities. You could have the user specify more clearly what they mean by “best.” If the user doesn’t specify, you do your best based on what people typically mean.

Ideally, the user can say, “When I mean best, I mean ranked by the number of people who visited that site,” or, if you’re listing companies, “The best companies are the ones with the most employees.”

There are all sorts of ways to rank a list of results that aren’t captured by the subjective term “best.”

It’s like asking who the best NBA players in history are. Everybody has their own answer.

The search engine should definitely do as good a job as possible, even if you don’t specify.

It’s a new concept for people because we’re not used to a search engine that can handle a complex ranking system. You might type “best basketball players” rather than something more specific because that’s all Google can handle.

If Google could handle “basketball players ranked by average shots scored per game,” you would do that. But it can’t.

You haven’t used the word “agents,” but you’re kind of building a search agent. Do you believe that’s agentic in nature, or do you think that term is distracting?

I think it’s a good term. I do think everything will eventually become agentic, and then the term will lose its power.

What we’re building is agentic in the sense that it takes actions. It decides when to go deeper into something. It has a loop. It feels different from traditional search, which is an algorithm, not an agent.

Ours is a combination of an algorithm and an agent.

My reflection from seeing this in the coding space is that there’s the classic framework of self-driving levels of autonomy, from Level 1 to Level 5. The Level 5 systems all failed because we’re not at full autonomy yet. People like to be in the loop.

The Level 1 systems started as copilots, and now we have Cursor and similar tools. If it’s too agentic, it becomes too magical. I put a paragraph into a text box, it gives me an answer, and I’m disconnected from the process. I don’t trust it as much as something where I’m more intimately involved in the research process.

I think o1 is also very full self-driving. You don’t get to see the plan, and you don’t get to affect the plan. You fire off a query, it goes away for a few minutes, and it comes back. That’s effectively what you’re saying you’re going to do, too. Is there an in-between?

In building this product, we’re exploring new interfaces. What does it mean to kick off a search that takes 10 minutes? Is that a good interface? What if the search is wrong, or it’s not specified exactly enough?

That’s why you get previews. The process is iterative. Once you’ve specified exactly what you mean, you may simply want to kick off a batch job.

There’s a barrier with agents where you have to explain the full context of what you mean, and a lot of failure modes happen when you don’t.

There are failure modes from the agent not being smart enough, and there are failure modes from the agent not understanding exactly what you mean. A lot of context is shared between humans but lost between humans and this new creature.

People don’t know what’s going on. To me, the best example is system prompts. Why do you have to write, “You are a helpful assistant”? Of course you should be helpful. But people don’t yet know what they can assume the model knows.

Now people write, “You’re a very smart software engineer. You never make mistakes.” Were you going to try to make mistakes before?

With driving, people know what good driving is: Don’t crash, stay within a certain speed range, and follow the directions. With AI, models, and search, people don’t yet understand what you actually know, what your assumptions are, how you’re going to search, whether they can trust it, or whether they can influence it.

Even if you’ve crafted a great system prompt, you want to be part of the process itself. The system prompt doesn’t capture everything.

A system prompt is like choosing the person you work with. You might want a software engineer who thinks about code in a particular way. But even after choosing that person, you can’t give them a high-level command and expect them to do it perfectly. You have to be part of the process.

As a side note, my favorite system-prompt programming anecdote is now the Apple Intelligence system prompt. Someone prompt-injected it and saw that it included the words, “Please don’t hallucinate.”

Of course we don’t want it to hallucinate. It’s exactly what you’re talking about. We should train that behavior into the model, but somehow we still feel the need to inject it into the prompt.

I don’t think we’re scientific about this yet. It’s almost cargo culting. We have this magical ritual—turn around 3 times, throw salt over your shoulder before you do something—and it worked last time, so we do the same thing again. There’s no science to it.

A lot of these problems might be ironed out in future versions. They might hide the details from you. All of them might have a system prompt that says, “You are a helpful assistant,” but you don’t actually have to include it, even if that’s how they’ve implemented it on the back end.

One question I’m curious about: I’m trying to frame this in terms of the general AI search wars. You’re one player, alongside Perplexity, ChatGPT Search, and Google. But there’s also the B2B side.

We had Drew Houston from Dropbox on, and he’s competing with Glean, which we’ve also had on the show. Is there an appetite for Exa to search my company’s documents?

There is appetite, but we have to be disciplined and focused. We’re already taking on perfect search, which is a lot.

Ultimately, we want to build a perfect search engine. For many queries, that involves your personal information and your company’s information.

The grandest vision of Exa is perfect search over everything. We’re going to have an Exa satellite, because satellites can gather information that isn’t publicly available.

Can we talk about AGI? We never talk about AGI, but you had this whole tweet about o1 being the biggest step toward AGI. Why does it feel so important to you?

I know there’s criticism that it’s not smart, or that Sonnet is better, and so on. What did you see?

I’ve just been connecting the dots. This was the key thing that a bunch of labs were working on: Can you create a reward signal?

If you’re trying to learn coding or math, you could have one model act as a grading system that says, “You successfully solved this programming assessment,” and another model act as the generative system. You give it a bunch of programming assessments, and you can train on that.

Whenever you create a reward signal for a task, you can generate a bunch of tasks for yourself. The model can see that it did well on 2 of 1,000 tasks, and then train on that data.

It’s basically creating its own data and bootstrapping itself. OpenAI built the most impressive product doing that. It’s easy to see how it could scale to solving programming or mathematics, which sounds crazy, but everything about our world right now is crazy.

If you remove the assumption that it’s impossible and think clearly about what’s possible with o1, it’s easy to see how that scales.

How do you think about older GPT models? Should people still work on them? OpenAI just had the new o3. Sam talked about o2 that day, so obviously they’re spending a lot of time on it. Meanwhile, some groups are still working on making Llama better, and other labs don’t have an o1-like model yet.

That gets into what the ecosystem of models will look like in the future. Is there room for everything to become an o1-like model?

There’s definitely a question of inference speed. o1 is 2 things. First, it’s bootstrapping itself and teaching itself, so the base model is smarter. Second, it has inference-time compute, where it can spend minutes or hours thinking.

Even the base model, which is fast and doesn’t have to take minutes, is better and smarter. I believe all models will be trained with this paradigm. You want to train on the best data, but there will be many different sizes of models from many different companies.

I don’t think OpenAI is going to dominate every possible LLM for every possible use case. For many things, you want the fastest model, and that might not involve o1 methods at all.

If you were to take the idea of Exa being o1 for search literally, you would need to prioritize search trajectories. You might pay a bunch of graduate students to research things, track what they search for, and track the sequence of their searches.

It seems like that’s the gold mine here: the chain of thought, or the thinking trajectory.

When it comes to search, I’ve always been skeptical of human-labeled data.

We tried something at Exa recently where a bunch of engineers and I labeled queries. It was really hard. You have all these niche queries, and you’re looking at a bunch of results trying to identify which ones match the query. If the query is about the intricacies of some biological experiment, I have no idea what matches.

Human labelers tend to match by keyword: “This document matches a bunch of keywords, so it must be good.” But then they’re completely missing the meaning of the document.

An LLM like GPT-4 is really good at labeling. We’re using LLMs as labelers specifically for search.

It’s interesting because search and GPT-5 are different. GPT-5 might benefit from training on a lot of PhD notes because it may need to do very complex problem-solving when given an input.

Search is a different problem. You’re asking simple questions about billions of things: “Is this a startup?” “Did this person write a blog post about search?” Those are simple questions. You don’t need PhD-level training data for that.

What else do we have here? Nap pods. Exa has a very interesting company-building vibe. You have a meme-lord CTO, and you’re generally counter-consensus in a bunch of things. What’s the culture at Exa like?

Jeff and I have been best friends since the first day of college, and we have a really good vibe. It’s intense but also really fun. We laugh a lot at Exa, and you see that in every part of our culture.

We don’t really care about how the world sees things. Jeff and I are just thinking, “What should we do here? What do we need?”

With the nap pods, people get tired when they’re coding or doing anything else. Why can’t we just sleep or nap here? If we need a nap, we should get nap pods.

It’s crazy to me that there aren’t nap pods in more companies. I get tired all the time. I take a nap every other day for about 20 minutes. I’m not actually always napping; I’m often just thinking about a problem. But closing my eyes helps me come up with more creative solutions and gives me some rest.

Google was the original company that had nap pods at work.

At one point, Google was also thinking from first principles about everything.

We didn’t just get a nap pod for the office. We found something from China and asked, “Who wants to get in on this? Let’s get a container full of them.”

We try to be frugal. We were looking at different nap pods, and at some point we thought, “China has probably solved this problem.” We ordered them from China.

It was extremely heavy. When it came off the truck, it weighed about 500 pounds. The truck was having trouble putting it on the ground. The delivery guy and I tried to hold it, but we were struggling. Someone came from the street and started helping us.

Did you hurt yourself?

It was really dangerous, but we did it, and then it was awesome.

I was reading the TechCrunch article about it. There was a whole article about the nap pods, and they quoted Jeff. One paragraph said the nap pods maintain employees’ ability to stop work and sleep, rather than the idea that employees are slaves.

Jeff is a wizard with words. I’m sure that’s not what he meant.

I’m curious how people interpret things. There was a period when the discussion about startups and hustle culture went away for a bit, but now with AI, people have all these feelings about AI. Are you pro-hustle culture?

Ideally, the hustle is people having fun. People are just having fun.

From the outside, people don’t like it. People who aren’t in AI and tech are like, “These guys are at it again.” They’re the same people who gave us underpaid drivers. It was funny to see how they wanted to make it sound like Jeff was saying employees are slaves.

I can’t imagine a more exciting experience than building something from scratch that’s a huge deal with a bunch of your friends. Our team is going to look back in 10 years and think this was the most beautiful experience you could have in life.

That’s how I think about it. It’s not about whether it’s hustle. It’s about whether it satisfies your core desire to build things in the world. For us, it does.

Anything else we didn’t cover? Any parting thoughts? Are you hiring?

We’re definitely hiring. We’re growing quite fast, and we have a really smart team of engineers and researchers.

We just purchased a $5 million H200 cluster, so we have a lot more compute to play with.

Do you run all your own inference?

We use a mix of our cluster and AWS for inference. Our current cluster is made up of A100s, and we’ve updated it with the new one. We use the clusters for training and research.

What’s the training-versus-inference budget? Is it 50/50?

There will be more inference for search.

I’m sidetracking, but I’m always thinking about the economics of search. The upper limit is going to be whatever you can monetize from ads. For Google, let’s say it’s around 1 cent per thousand views. I don’t know the exact number, but it’s out there.

That means your revenue is capped by that amount, and your costs have to be lower. At some point, an LLM inference call for every page view has to cost less than the money you make from that page view.

One thing that surprised me about Perplexity and Character.AI was that they couldn’t get inference costs low enough to make the economics reasonable.

For you, it’s a mix of front-loading costs by indexing. You run the compute once a month or once a quarter when you reindex, and then there’s a smaller amount of compute when the search actually happens.

When people work out the economics of such a business, they have to think about where to put the costs.

You can’t run LLMs over the whole index at query time. You have to preprocess things, usually with LLMs, and then you can rerank a smaller set—10, 30, or 100 results—depending on the size of the Transformer you use.

One really interesting thing is that we’re building a search engine at a time when LLM costs are falling dramatically. When a useful tool becomes 200 times cheaper in a couple of years, there are new opportunities in search.

Not integrating that, and not rethinking the search algorithm from scratch given that costs are falling 200 times, would be crazy.

Thank you so much for coming on. This was fun.

Thank you. This was really fun.