20VC: Horizontal vs Vertical Agents: Who Wins | Model Routing: Commodity or Increasing in Value | The Future of the Inference Market | Why Founders Should Take Discounts for Tier 1 Firms with Harrison Chase, LangChain

I think it’s absolutely worth taking a discount for Sequoia and Benchmark. We spend roughly 30% of our engineering salaries on coding tools. That’s actually gone down over the last 2 months, while the number of tokens we’ve used has gone up. I think I really underestimated the demand for inference. Everything just works. This market is so big, and there’s so much demand for it. I don’t know if that’s going to get less intense anytime soon.

Harrison, dude, I’m so looking forward to this. I’ve been looking forward to it for a while, because I remember Miles telling me about you when he was at Benchmark and just singing your praises. I wanted to make this happen for a while, so thank you for joining me.

I’m excited to be here.

For those who don’t know, in 60 seconds, what is LangChain, and where do you sit in the stack?

We build tools to make it easy to build agents. We build developer tools primarily. We have an open-source framework that connects the model to everything else you need to build a useful agent: tools, data, computation, things like that.

We also have LangSmith. LangSmith is for observability and evals. Agents go wrong all the time. How do you know how they go wrong, and how do you measure them so you can increase their performance over time? That’s what LangSmith does.

Why is it difficult to build agents?

1. Why Agents Are Hard to Build

The hard part about building agents is that they don’t behave how you want them to behave. The models are these nondeterministic black boxes, right? They take natural language as input, and then they output natural language. Inside the black box, they’re also nondeterministic. What that means is that the input space is infinite-dimensional. It can be anything. You put a chatbot in front of someone, and they can type anything they want, right? That makes it really hard to know what they’re going to type.

Agents aren’t just one model call. They’re a series of model calls, and they interact with other tools. You have these small black boxes inside a larger system, and you get cascading errors. You have a very big input space and a very big output space, because 2 similar inputs can arrive at completely different answers.

What makes all this hard is the LLM itself. Yes, they’re very, very smart. But you don’t know exactly what they’re going to do, and they change their answers based on small changes to the input. So you build this system, but it’s really non-robust. Getting the system to behave reliably is the hardest part.

When we talk about agents 3 or 4 years ago versus agents today, they’ve gotten a lot better. The main things that have happened are, first, the models have gotten better, and second, we’ve figured out how to build systems that harness these models better. That second part, the harnessing of these models, is what we focus on.

How does your business change in a way that people don’t see as the models get better?

The main way our business changes as the models get better is that we basically have to erase big parts of our product. Cursor said this really well, too. I think one of their actual operating principles as a company was “erase the product,” or “delete the product,” I think is what they called it. Basically, these models get better, and things you had to do 6 months ago, you no longer have to do.

This is very relevant to our open source. Our open source is all about the harness and the framework for taking these models and helping them do things. The right way to take these models and help them do things changes every month, 3 months, 6 months—choose your time period—but it changes pretty rapidly.

How do you build for such an uncertain future? Do you just have to get in bed with the labs to know the roadmap, so you don’t waste 6 months building something that they then erase?

There’s no easy answer. On one hand, what are you going to do? Not build? That’s what I like to say to founders who ask this: “What do I do? Stuff is changing. The models are changing. If I build something now, it might not be relevant in 6 months.” What’s the alternative? You just don’t build anything?

I think you have to make educated guesses as to what parts are going to be around and what parts aren’t going to be around in the future. Early on, the way we did tool calling in LangChain was by having the model generate JSON DSL and then parsing it. That was something we all thought—or knew—was a bit of a hack at the time, but we had to do it because that was the only way to get it to actually steer things. You have to get it to do this type of tool calling.

A few months later, OpenAI came out with tool calling. This is a more native version of tool calling that’s directly in the model and in the APIs. I think we recognized pretty early on that was the future, so we invested a lot of effort and completely switched from JSON parsing to this tool calling.

I think you have to recognize what things are short-term hacks and what things are long-term—what will always be there. The models will never know your proprietary data. They’ll never be able to actually get the output of APIs. You have to take their API call, pass it to the API, get the output, and then continue in the loop.

I think the system around the model, connecting it to other places, will always be there. But a lot of the hacks we do around chain of thought, planning, or tool calling—things that can be much more closely integrated into the model or into the model API—will go away. I think you just have to be quick to recognize that and move on.

Why do we need neutral harnesses? We see, bluntly, very brilliant agent builders from the model providers themselves. Why doesn’t your layer just get eaten by them?

2. Neutral Harnesses Preserve Control

I think the reason we need neutral harnesses is that there’s a lot of state that you build up either before you build the agent or as part of the agent running. That state can include things like memory, whether that’s short-term or long-term. Memory also comes in different forms. It can come from interacting with the agent or from the agent learning from the environment. The state builds up as the agent interacts, goes out, learns, and chats. It learns things that make it better.

I think one of the most interesting things right now is how to make these agents do better over time and learn. Some of that learning and improvement over time will be in the model weights, but a lot of it is outside the model, in the context. When you build up this context, you really don’t want it to be linked to a single model provider. Otherwise, you can’t go to other models.

Being able to change models is both an offensive and defensive advantage. It’s offensive because you want to take advantage of the best model when it comes out, no matter who it comes from. It’s defensive because you don’t want to get locked in, so they can’t raise rates and things like that. You want to be able to switch models. The harness can be very tied to the model, or it can be neutral. Right now, the context is really intricately linked to the harness. The harness’s job is to take the model and interact with all the other things. That includes data and APIs, but also memory. If you build this memory in a way that’s linked to the harness, and the harness is linked to the model, then you’re stuck.

So the true value of the harness structure is the portability of data that it allows the user.

Yeah, you basically don’t want to be locked into models. If memory or context starts building up in a way that’s specific to how one model wants to interact with memory or context, that’s very bad for portability. It puts you at a disadvantage as you build your business going forward.

We talk with a lot of people about owning their intelligence and what that means. There are a lot of different versions of that. The version we think of includes a few different components, but one key component is having an open and controllable agent. You want to be able to adjust all the different pieces: the model and the harness, too. Sometimes these model labs are training models to be really good at the things that are in distribution for them, which is often coding. Coding is great, and there are a ton of use cases for it, but there are also a ton of use cases that aren’t coding. Maybe they’ll write code as part of it, but they’re using that to do sales, research, law, or other vertical-specific things.

When you get out of distribution from what the model labs are training on, you want control over what the harness does, and you also want control over the context. The model, the harness itself, and the context make up an open and controllable agent architecture. We think that’s a big part of what it means to own your intelligence.

I just want to back up a little bit. When we think about agent value, some people say the value is actually in the verification—whether it did well or not—because that unlocks the next step, which is automation. If value is derived through verification, who wins? The harness?

If value is derived through verification, I think the people who win are the domain experts, the people who can quantify what that value means, which is very hard. Building evals is a very hard job. A lot of the value will accrue to whoever can write those verifiers and capture that domain knowledge.

I’m massively bullish on vertical agents for different domains for this reason. The domain experts who are best suited to write those verifiers or build those evals will be able to build harnesses and models. It won’t be just the harness or just the model; it’ll be some combination of them.

Practically, if you’re getting started building a vertical agent, I’d highly suggest you start at the harness layer. Do context engineering and tweak the harness. Then, when you’ve got users, start tweaking the model with post-training. I think it’s much higher ROI to start at the context or harness layer. We’ll have all these different verticals, and they’ll be able to quantify what good looks like to them in a very domain-specific way. That’ll let them build not only vertical agents, but also vertical agent experiences. Different agents might have different experiences. I think a lot of the value will come from that. I’m massively bullish on vertical agents, and we want to enable a lot of that.

You said that if I were building one, I’d start at the harness layer. We’re seeing a lot of companies, whether it’s Harvey or Ramp, build their own models. To what extent are they actually building their own models? Mike Cannon-Brookes at Atlassian said on a show we released today that it’s largely, respectfully, BS, and they’re fine-tuning open-source Chinese models. Is that true, and how do you think about that?

I think a lot of it is that they are post-training open-weight models, absolutely. I still think that’s a valuable thing to do. I don’t think there’s anything wrong with it. There are different dimensions to owning your intelligence. The cheapest way is to control the context and use an off-the-shelf harness and model. The next level down is to start tweaking the harness, which gives you more control than just the context, but is a little more expensive. The next level down is to post-train an open-weight model. You might take an existing open-weight Chinese model and post-train it for your domain. The bottom layer is actually training a model from scratch.

I don’t think many people are training a model from scratch. I think most are post-training. If you’re training a model from scratch, you’re probably at NeoLab. I think those are among the few doing that. But the Ramp and Harvey teams have a ton of talent, a ton of data, and a lot of usage. They’re trying to optimize it, so they’ve gotten all the juice they can out of the context and the harness, and now they’ve arrived at post-training these models.

Can I ask you: the jury’s still out on the efficacy of these specialized models in the post-training realm, as we mentioned. Some reports I’ve seen say they’re massively more efficient and better, and some say less so. How do you think this will play out?

I think what we’ve seen with post-training is that a lot of the post-training that makes practical sense right now is for simpler tasks that make up a larger harness. What I mean is that when you have an agent harness doing interesting work, it’s actually very rarely just one model running the loop. That might be the core of it, but there are a bunch of other model calls that happen as guardrails, routing, sub-agents, sub-workflows, or things like that.

A lot of the places where we see post-training being efficient now are for simpler tasks like classification, extraction, or summarization. Those might be tools the agent uses, guardrails, or things like that. I think those types of post-training make a lot of sense right now and will continue to do so. The more unanswered question is about the core model that’s driving the loop, because the main way an agent works is with an LLM running in a loop and calling tools. The question is whether that model driving the loop will be a post-trained model or a frontier model.

This is where we see more back and forth. Every time a new frontier model comes out, they get better at running this core loop. We do see people like Harvey and Ramp experimenting with post-training models for this core loop. Harvey recently announced something exactly in that vein. I think that’s an open question.

If you were building your own vertical application, would you post-train your own model or not?

Not to start. Absolutely not to start. But if we’re talking five years in the future, assuming I’m getting a ton of usage, I think it could be a realistic possibility.

One of the things harnesses also need to do, I imagine, is model routing depending on the request. How much value is there in the model-routing layer?

Model routing is so interesting, and I think it’s one of the hottest and maybe most divisive topics right now.

So, a few thoughts. First, there’s a question of where the model routing should even be. Should it be at the harness level, or should it be at the gateway level? This is getting a little bit into the technical weeds, but—

What is the difference? I’m sorry.

At the harness level, it’s in the code of how it runs, and at the gateway level, it’s just an API call. The benefit of putting it at the gateway level is that, because it’s just an API call, you can drop it into any harness. You could put this routing that’s happening at the gateway level into Claude Code, into any harness, without having to change the code.

The downside, and why I think we’re more bullish on routing happening in the harness—as are, I think, Factory and Cognition, who have also written things about this—is that in the harness, you just have more knowledge of what’s going on. You have more awareness of the state. When you have an API, it’s generally assuming something like an OpenAI Chat Completions API, so it only takes in a certain amount of state. You have more state in the harness to know what’s going on with the prompt cache. You want to try to keep the prompt cache warm and available.

You can also change the prompt more easily. A lot of these models have different prompting strategies, so if you do the routing at the harness layer, you can more easily change the prompt. A lot of what we’ve seen is this debate between the gateway and the harness, and where routing belongs. I think we probably believe it belongs in the harness.

And just so we understand, OpenRouter and all the other providers, of which there are several—

There are a lot.

—they’re at the gateway.

They are at the gateway. OpenRouter is an incredible company with incredible usage.

Amazing.

A lot of why people use OpenRouter, Vercel, and other gateways is for model access. I think there’s also a difference between a router and a gateway.

Mm.

I’d probably say most of OpenRouter’s usage—and Alex obviously knows the numbers a lot better than I do—but if I had to guess, a lot of it comes from being a gateway, not a router. People specify the model they want up front, and OpenRouter just provides a single API with a single API key for accessing it.

In terms of smart routing, I think that’s happening a lot less than people think. When it is happening, I think it’s built into the harnesses by folks like Factory. Factory does a tremendous job of this. They’ve written a number of very good technical blogs about how they do routing.

I pretty strongly believe that routing has to be harness-aware. You have to know what the task is and what the harness looks like. I don’t think you’ll have routing that works for law, finance, and coding and have it all be the same. It’s going to be very domain- and harness-specific.

I’m sorry. When I listen to you, it seems so obvious that routing should be at the harness layer, given the context and memory stored there. When I have such an obvious response, I like to understand: why am I stupid? What’s the bull case for it being at the gateway layer?

When people talk about routing at the gateway layer, I think there are maybe 2 things they’re talking about. First, it makes it way easier to port. You can take Claude Code. People love Claude Code, right? If you tell people, “Hey, you can’t use Claude Code. You have to use this other harness,” there are other good harnesses out there, but let’s pretend for a second that you love Claude Code and want to let people keep using it. If you want to do some semblance of routing to maybe DeepSeek or an open-source model, then the only way you can do that is at the gateway layer, because you don’t control the harness. As a side note, this is another argument for open-source harnesses that you can control and modify.

The second thing is that when people talk about routing at the gateway layer, I think they’re often implicitly assuming there’s a domain. There are 2 general domains where you could maybe argue that you can build general routers, and I actually think OpenRouter has done this to some extent. You could argue coding. Coding is the most popular domain; that’s where all the usage is going. Let’s build a router that works really well for coding. Maybe we won’t be harness-aware, but we’ll be use-case-aware. We’ll know different models are good for different things in coding, and we can put it at the gateway.

The other thing you could say is, “Hey, we’ve got this general chatbot. We’re not trying to be best-in-class at law or finance; we’re just going to be a general chatbot across the company. We can know well enough that if you say, ‘Hi, how are you doing?’ you don’t need Opus for that. You can use a cheap model.”

I think when people talk about routing at the gateway level, they’re implicitly assuming that you’re doing either general-purpose chat or coding, and not something domain-specific. As soon as you get into a domain-specific thing, I really think it has to be at the harness layer. You’re not going to use an off-the-shelf gateway router for that.

So, in 2 to 3 years’ time, you think the majority will be within the harness, given the benefits that come from that context, but for very horizontal tasks, it will sit in the gateway?

I think that’s exactly right. If you’re building a vertical agent, I think routing needs to be in the harness. If you’re thinking about a general-purpose horizontal task, like general chat or coding, it could maybe be in the gateway.

Is it a commodity? Does everyone just build it?

I don’t think routing is a commodity. I do think people will build it, but I think building it is quite hard, even today. Part of the reason is that the models change so rapidly. You can argue that certain models are better at different things, but then a new Opus or a new OpenAI model comes out, and it’s just better at everything. Then the value of routing goes down.

Do you trust evals?

3. Evals Turn Traces Into Learning

I think evals are useful, but I wouldn’t trust them fully. A lot of the academic evals that are out there and used when models come out aren’t really that useful anymore. I think all the models that come out are benchmark-maxed against those evals, so you don’t learn a lot.

Internal evals that you have and build for your domain can absolutely still be useful. This is the main reason that building evals is such an important part of building a vertical-specific agent. You have to have some way of measuring things, and I think you can generally trust those evals.

Even then, one of the most important things to do is just look at the data. Look at the data and see what it’s doing. Don’t rely too much on a single number. This is why you want to have traces and look at what’s going on. Maybe your evals aren’t perfect. Evals are really hard. You can get some sense of things, but maybe models are doing better in a different way that you didn’t know how to capture before.

So yes, I think you can trust vertical-specific benchmarks, but you have to evolve those as well. That’s one of the main things we’ve seen, because these models just get better in different ways.

Can I ask you about traces? If traces capture the exhaustive agent work, where is the value, then? Is it the app, the observability layer, the model layer? Where is it?

If you’re building an agent, I think the value is in what makes that agent good at the domain or vertical you’re working in. Again, I think there are different ways to control what an agent does: the context, the harness, and the model. Having specific, proprietary things for any of those can be an advantage.

I think controlling the context is the easiest and most obvious thing to do when starting—

And that would be data?

—then controlling the harness. And that would be data, memory, specific APIs, and specific tools that you can use. Controlling the harness is a little bit harder, but you can still put a lot of your domain knowledge around what it should do, including routing and other things.

The model is the hardest to control, but potentially has the highest ceiling. The things that let you iterate on each of those are evals and traces. There is a ton of value in these, but only insofar as you capture insights that you bring back to the context, the harness, or the model.

That process can look different. Post-training is great for bringing insights from traces back to the model. To bring things back to the context or memory, you’re probably going to want to run another agent over those traces and have that agent output learnings for memory.

To bring the learnings in the traces back to the harness, you’re probably going to want to run an agent over those traces and update the code of the harness itself. We actually have something that does this for people.

It’s called Engine. It’s an agent in LangSmith that runs over traces and tries to update code. When you update these things, you need to know that they’re actually better, and this is where evals come in handy.

Can I ask, how does continual learning change agents and how they work?

I don’t think continual learning changes so much how agents work, but I think it changes the systems you want to have running around the agent. What I mean is, I think the core loop for a continual learning agent—the core inner loop, maybe—is still largely the same. It will be an LLM running in a loop, calling tools and interacting with a model and an environment.

I think what continual learning adds is an outer loop that looks at the exhaust from agents and does things with it. What it does could be different depending on what continual learning means for you. I think continual learning is also one of those words that doesn’t have a super precise definition.

How would you define it?

I would define it as a system that gets better over time as it runs. There are different ways it could get better: it could update the context, the code of the harness, or the weights of the model. The outer loop will look roughly the same for all of those: take traces, get learnings, fix something, measure it to make sure it improved, and then repeat.

But changing the weights of a model is very different from updating a text file, like a memory file.

If we look at the context layer, we mentioned data being a part of that. We’re seeing data companies scale to hundreds of millions in revenue in the fastest time ever—I mean, ever, ever, ever—and everyone is selling data. How do you expect this market to play out, given your neutrality on not being involved in it but being adjacent to it? Will this be a market of consolidation in the years to come? How do you see it playing out?

4. Memory Is Still Unsolved

I actually think the data market is still pretty early in the grand scheme of things, insofar as we’re talking about a context layer for agents. There are a bunch of databases. ClickHouse is a really good example of one that’s doing incredibly well recently. I think they’re still largely focused on their database, their good old-fashioned database. They’re good at storing and querying data, but still in pretty traditional ways. ClickHouse existed pre-AI.

There will be a bunch of data produced by software systems, and existing databases like ClickHouse, MongoDB, and Elastic are great for that. When we talk about a context layer for agents, I think a key part of it is, yes, there’s a database, but there’s also a system around the database. I think a context layer for agents will look more like an agent around a database than just a database.

Taking all of these learnings and ingesting them into a database requires some kind of intelligence, and I think that intelligence is really, really hard. The hardest part about memory is actually not the database at all. When we talk to companies, the way they do memory is often just storing it in a single file. The hard part is taking all the observations that happen—whether from a user interacting with traditional SaaS software or from a user interacting with an agent—and distilling them into information valuable enough to go into this context layer. That’s the hard part in memory right now.

Why is that so hard?

Because it’s tricky to get LLMs to do what you want them to do. This kind of meta-reflection is something LLMs still struggle with a lot: reflecting on prompts and improving prompts, or reflecting on agent architecture and improving agent architecture. We should build a benchmark for this internally. We do have some benchmarks for this in Engine, our agent that runs on things, but this kind of meta-reflection is still really difficult for agents.

I think it’s generally a hard thing for humans to do as well. I don’t know if we actually understand what goes on inside our brains. Our brains are complex things that we don’t fully understand. If we’re trying to replicate this learning process in a system, there are a bunch of different pieces. Models aren’t great at reflection, and I don’t think we know what this system should look like. What should the inputs be? What should the outputs be? What should the harness for this agent be? What should the different workflows be?

Will the models be meaningfully better at reflection in 12 months?

Probably. It seems like they’ll get better at everything.

Do you think the speed of model improvement will continue at the same rate of acceleration as it has so far, or do you think we’re approaching some form of asymptotic plateau?

I think the speed of model improvement is really tough to assess without being inside a lab. I’d say there are a lot of tasks for which model improvements will flatline, in the sense that the models are already good enough. I think there are a bunch of tasks where the models are just good enough.

Which would be coding?

Yeah, coding’s getting there. Summarization and routing tasks, too. There are probably more difficult tasks, or tasks we don’t even know to think about or to throw a model at. I’d say this kind of meta-learning is one of those tasks the models aren’t good at now. I imagine it’s very important to the labs in terms of cracking RSI or things like that.

I do think the models will get better at that over time, and it’s possible that progress actually looks much faster than before. It’s really tough to have an educated opinion without being inside a lab. There are some tasks where it’s saturated, and other tasks where I think it will probably just increase faster than before.

What about agents today sounds crazy to you, but you think will be incredibly common in 2 to 3 years?

I think this memory layer for agents is still massively unsolved. One thing about agents that’s rare today but will be incredibly common in the next few years is this memory layer. If you look at Instinct or Muse, I think one of the things they’ve nailed is personalization and learning, and the feeling that it’s something for you—whether that’s because it’s speaking in your tone or because it’s following up proactively.

Memory is super early. We’ve been talking with folks about memory for 2 years and trying to do things in open source. We launched an alpha and then killed a commercial solution for memory. It’s just so early, and it’s so domain-specific. What memory looks like for Instinct or Muse is pretty different from what it probably looks like for a company or org harness, and it’s pretty different from what it looks like for legal or finance.

I think memory is one thing that really makes a difference. I think Instinct and Muse have done a great job at this, and you see how it makes a difference there. But it’s still very early, and I’d expect all products in a few years to have some aspect of memory in them.

With respect, is that not quite obvious?

I think it’s quite obvious that memory will have an impact. That’s why you see so many people interested in it and why they’ve been interested in it for the past few years. I still think it’s not obvious what the answer is. It’s incredibly not obvious. I don’t think anyone knows what the answer for memory is.

Why did you kill the commercial solution for it, given the obviousness of it and the clear need for it in almost everything?

We killed it because we didn’t find product-market fit. Why didn’t we find product-market fit? I think we actually tried to be too general. Memory is so early that a lot of the solutions you want to build are very domain- or agent-specific.

And so you would build vertically specific memory solutions. What would you do differently?

Going back to what the hard part of memory is, I think it’s the process of learning from facts what to remember, and the agent that does that remembering. As I said, I’m massively bullish on vertical agents. I think having those agents be good at learning things specific to their domain is a key part of that.

I do think these learning agents, at least to start, probably have to be domain-specific. I don’t know if there’s a general solution. What does the harness for these learning agents look like? It probably looks pretty similar to a lot of the harnesses we have now.

We build Deep Agents. Deep Agents is an agent harness. It probably looks similar to that, with an LLM running the loop, but it has specific instructions and tools. Maybe there are specific sub-agents that are really good at navigating existing memory structures or distilling facts. I think you’ll have agents for remembering things. I just think they’ll be domain-specific.

Do you think you’ll have vertical agents for consumer applications? I understand and agree with you when you look at finance, accounting, sales, marketing, and legal. You mentioned Muse and Instinct. They would be horizontal agents. Do you think we’ll have vertical consumer agents?

I think my take on the consumer space is that you’ll have both. You’ll have both horizontal and vertical agents, and the horizontal agents will interact with the vertical agents.

Where does it make sense to have a vertical agent versus not?

I think it probably makes sense to have a vertical agent for most apps. Having vertical agents makes sense for two reasons, at least. First, a lot of these apps have knowledge about you and can learn things about how you interact with that app better than a general horizontal agent could. So, going back to the point about domain specificity, how you interact with music on Spotify is very different from how you might interact with Strava. Those apps are probably best at learning things about you, and I think the best way to expose those learnings is through an agent.

The other thing is that a lot of these apps have a ton of APIs. You could say, “Hey, Spotify, don’t build an agent. Just expose a bunch of APIs.” I actually think it’s a better experience if there’s an agent that wraps those APIs and exposes a natural-language interface for a horizontal agent to interact with that vertical agent. That interface could be a single one, or maybe multiple, but probably a single one.

How does the infrastructure change when agents are the fundamental interface with sites? What I mean is, I wanted tickets to a specific concert, and I said, “Check every 5 minutes and buy any seat that comes available.” For 3 days, it checked every 5 minutes. The volume of queries was 1,000 times what it would’ve been with humans. How does the infrastructure required change in a world of agent interaction, not human interaction?

When agents are interacting, the number of requests goes way up. I think you saw a story about how Resy blocked a user’s account because they were using Instinct to pull data pretty frequently. So I think there’s just way more strain on the infrastructure. How that percolates down to lower layers, I don’t know if I have an opinion on that yet. I think you have to build systems that can be much more bursty and scale much, much broader. I don’t know if I have more specifics than that.

Do you think providers will allow agent usage or not?

I think the future is providers exposing their own agents, and people either interacting with those agents or having their horizontal agents interact with them.

So my horizontal agent would interact with Rezi’s vertical agent? Sorry, just so I understand this.

I think that’s exactly right.

Why would that work? That still doesn’t solve the 2 core problems of advertising for a lot of these businesses. They lose their ads model and then upsell. They lose anything else you might like on the side, so you get a reduced basket size. Why would agent-to-agent work?

I think you could include some of those things in the agent response. You could do advertising for specific things on Rezi or Etsy or whatever, and include that in the agent’s response. It could say, “Hey, I found these 3 things. You might also want to consider this other thing.” That’s a natural-language response that goes back to the main agent.

Also, to your point about the strain on the systems, if you have a vertical agent, it’s going to be better and more efficient at interacting with the underlying systems than a third-party agent will. So I think it will make it a lot more efficient for them to do that.

When we look at the number of queries and how the volume changes with agents, it takes a lot of tokens to do all this work. How does the cost of tokens and the cost of that intelligence change over time? We always thought it would get cheaper. It hasn’t gotten much cheaper. How do we actually think about that?

Cost is one of the big things we talk with people about right now. I think there are a few different dimensions to this. First, the frontier models are getting relatively cheaper over time. Astra is fantastic in terms of price-performance compared with a lot of things that existed a year ago. Second, there’s just way more demand. When you start talking about agents talking to agents or multi-agent systems, there are so many more tokens being used, and so the costs are going up. We see this primarily in coding. Coding costs have skyrocketed, and I think a lot of organizations, including ours, are now thinking about ways to pare those costs back.

If you look at how we’re tackling coding agents, I think there are a few hints there. We think about it in 3 steps. First, visibility: showing admins how much things will cost and how much they have cost. Second is cost control, so you could say something like, “Go get me a reservation, but don’t spend more than $5 doing it.”

Third is impacting the harness itself. A lot of harness engineering has been done to make the agent better and more accurate, because that’s the biggest blocker. As cost becomes more of a concern, we’re seeing people care more about things like prompt caching, which are really good for reducing costs. We’re seeing model providers make this easier, too. This is getting a little bit in the weeds, but they’re introducing things that let you change the tools available to an agent mid-flight. Previously, that would’ve busted the cache and caused a massive increase in cost. Now you can add those tools dynamically and keep the costs the same. So I think we’re seeing more attention to both model APIs and harnesses in terms of controlling costs.

When you talk to large enterprises today, what matters most to them? Is it accuracy, cost, or security?

Another thing would be latency, by the way. We run developer surveys every year where we ask developers what they care about. For the last 2 years, the number-one response, basically double the next 2, has been accuracy. So that’s still the primary thing. The next 2 are latency and cost, and security is a little bit after that. They’re pretty close.

When we talk to large enterprises recently, I’d say cost has risen as a concern. I’d still probably say it’s below accuracy, but it’s probably higher than latency. And for large enterprises, security is a higher priority, but still probably below accuracy and cost.

What percentage of your engineering salary, give or take—not specifically—do you spend on coding tools?

I think we spend roughly 30% of our engineering salary on coding tools. It’s actually gone down in the last 2 months, while the number of tokens we’ve used has gone up. We’ve done this in 3 steps. First is visibility. We tracked all of our coding-agent usage to LangSmith so we could tell who was spending how much and what they were spending it on. We could also run analytics on that.

Second is basic cost controls. We set a budget, and if you hit it, you can talk to our VP of Engineering, who will almost always increase it. It’s just a safeguard against agents being used incorrectly or going into a bad loop.

Third is controlling the harness. One of the things we’ve done is build our own in-house cloud coding agent. It’s not an IDE or a terminal-based agent; we let people use whatever they want for that. But we built our own cloud-based agent. OpenSui is what we call it. It’s actually the only way I code anymore because I’m in Slack all the time, so I just message it from Slack and it kicks off.

In that harness, we’ve done a bunch of work to control costs, including prompt caching and routing. I actually think that’s a pretty repeatable 3-step process for getting coding-agent spend under control: first, visibility; second, basic controls; third, trying to improve the harness itself.

Totally get that. I do want to unpack that. You mentioned visibility: seeing who’s spending what. Were the people spending the most the most effective?

Sometimes the people spending the most are the most effective, and other times they’re not. What I mean by that is we have a few senior engineers who spend a lot, and I 100% trust that what they’re doing is well worth it.

No questions there. We've also had some cases that were due more to silly mistakes or improper setup than anything else. We've had massive spikes in usage.

For example, we had one or two tasks where a plugin was being used that would take a snapshot of the DOM of a webpage and send it back every turn of the agent. They were doing some front-end engineering and wanted to capture what it looked like and send it back. It would just run this massive loop. I think we spent $40K in a week on that one thing.

When it gets to that, again, I think we're pretty liberal as a company and trust people, but I know that's probably not worth it. It wasn't malicious or even intended usage, really; it was just a mistake. I think catching that is well worth it. So it depends a bit on the side of the spectrum.

30% of salary, as you mentioned.

That's a guesstimate, yeah.

A guesstimate. Totally fine. What is that in 2 years?

I think the hard part about estimating coding-agent spend is that organizations are changing so rapidly. I think there's a future where organizations are much smaller, and coding-agent spend is the same as human salaries, or more—an order of magnitude more. You have a small number of people driving and executing incredibly rapidly and performantly. I think that's an enticing vision.

I don't think many companies, including us, are probably there. I think we still need a bunch of different people with different backgrounds and areas of expertise collaborating on things, and there's a lot of communication, overhead, and review. It would be ideal to have one person driving 100 agents and driving everything that makes up a large part of the product. But I think we still see a very human component to it.

I spend a lot of time thinking about organizational design. Is it possible to have small teams—one-person teams, two-person teams—that are driving agents? I think it's very tempting. In that situation, I would say agent spend might be 3X the cost of human salaries. But the engineers who can do that are incredibly valuable. Maybe it will actually just stay the same, because there are fewer people who can drive all those agents, but you're paying them a lot more, so the spend actually stays the same. Organizational design is very interesting right now, and because of that, I think it's really hard to answer that question.

To me, that number is so central because it's the fundamental determinant of Anthropic's enterprise value. If that number goes to 5%, then the $2 trillion IPO might be a little bit overpriced. If that number actually goes to 300%, like it is in Cursor's case, then it's dramatically underpriced. That's why I focus so much on it. Do you agree that number underpins Anthropic's value?

I think it's probably right that that number underpins Anthropic's value. The tricky part is that the only place we've seen agents really have an effect so far is software engineering, though there are a ton of other places within an enterprise that I would argue have yet to be affected by agents. I think it's possible. Software engineering is very difficult to predict, but I think there are basically 2 levers here.

One is what happens in software engineering: Is it the same as today, more than today, or less than today? But I think undoubtedly, for the other domains, it will be a lot more than it is today. From that perspective, I think that's very bullish on Anthropic, OpenAI, and other model providers.

I think another question a lot of people are asking is about the proliferation of open models and how that changes the landscape. You work with some very large enterprises. One of my dear friends, Jason Lemkin from SaaStr, says that—I’m not going to do his accent because I’ll butcher it. I’ve done it before, and he said it wasn’t very good. Do you find that large enterprises are fearful of working with open-source Chinese models, or do they actually not mind?

5. Open Models and Inference Demand

We see that some enterprises are fearful of working with open-source Chinese models. We see others that don’t really care.

Do you think that fear is justified?

I don’t really think that fear is justified. I think there are different concerns. One concern is that the data it runs on will somehow go back to the Chinese labs. If you’re using Fireworks or hosting it yourself, I think that’s just ridiculous.

Other fears are that the things the models have been trained on will cause them to respond in a way that isn’t consistent with the enterprise’s values. That’s where evals come in handy, and I think that’s largely unfounded. But maybe for some domains there’s some realness there. Because of that kind of response, I could see there being some nuance. So I think it’s largely unfounded, but for some domains there may be some nuance.

Do you think in the next 12 months we’ll have comparable U.S. open-source models?

I think in the next 12 months we will absolutely have comparable U.S. open-source models. You look at everything NVIDIA is doing to work with the ecosystem; I think they care about it tremendously. Reflection had a model launch recently. I think it was still a little bit behind some of the Chinese models on some benchmarks, but there’s a tremendous amount of attention in that domain. I think those models will absolutely catch up.

When you look at the state of the landscape today in terms of open models, what does the world not see that you see very clearly, given your position?

I think there’s a lot of usage of open models, but I do think they’re used for narrower tasks than people think. We see a lot of people using more open-source models and more post-trained models than I might have expected a year or a year and a half ago. But I still think they’re primarily used for simpler tasks like routing, guardrails, extraction, or summarization. I don’t think they’re really driving the core agentic loops for a lot of the main agents.

I think open-source models are divisive, because you have some people on one end who are like, “Great, they’re good enough. Let’s use them for everything.” I don’t think they’re quite there. Then you have other people who are like, “The frontier models are just going to keep getting better and cheaper.” Yes, that’s true, but if you really know what you’re going to do, have a high-volume use case, and have a few components that are simple enough tasks, I think it’s very good ROI to start post-training models for those use cases.

How do you see the inference market playing out? You mentioned Fireworks very openly. We’re a big investor in Fireworks, and I love Lin and the team. But everyone’s trying to do it. We had Chase on the show from Cruso, which also has that as part of its business. I’m friends with the Nebius guys, and they do it as part of their business, too. How does this business play out, and how do you think about the need to do the whole stack versus inference-only providers like Fireworks or Baseten?

When it comes to how things will play out with inference, there are 2 things I really underestimated from the start of LangChain to now: the demand for inference and the demand for coding agents. I think both of those markets are tremendously bigger than I thought 3 or 4 years ago.

I think Eric Vishria at Benchmark said this recently, either in a tweet or on a podcast. I think he basically said, “Everything just works. This market is so big.” Thomas Tunks also said something about how the market for inference is just so big. I think Eric was talking about something broader than just inference, but applying it to inference, I think any inference will work. There’s just so much demand for it, and I don’t know if that’s going to change anytime soon. Fireworks will work. AWS will work. Crusoe will work. There’s just so much demand for it.

6. Investing in the AI Stack

Where would you not be investing today? I can give you some examples. I think the customer support space is wildly uninteresting. There are 20 providers, all funded with over $100 million in the last 18 months. You’ve got Salesforce and Atlassian, big incumbents, and you’ve got older players like Intercom and Talkdesk doing it, too.

Or robotics. Bold take, but there’s so much money going into robotics. We’re massively far away from having a ChatGPT moment. Humanoids, in particular, really struggle with form factor. I just think the funding and excitement exceed the capability significantly. That would be my answer.

I like to think I’m somewhat of an optimist, so I do think this technology is transformative.

And so I think there’ll be a ton of opportunity. I might answer that, and then we can back out what I wouldn’t invest in. I think inference, which we’ve already talked about. I think there’s tremendous—

Yeah, let’s do this. You’re my—

Yeah.

—newest partner—

Yeah.

—in the fund. Welcome.

Yes.

It’s so good to have you on your first day.

Thank you.

Number one, we should have inference.

We should have inference.

Okay.

There’s going to be way more software written, so we should also invest in lower-level infrastructure primitives. Modal is an incredible company. Temporal is an incredible company. Supabase—

Do they not compete with Fireworks?

—Convex. Modal does on some things, yeah.

Okay.

But they also do more. They do sandboxes.

So we’ve got inference layer one. The coding layer.

I would say inference and then infrastructure—low-level infrastructure. Temporal, Modal, Supabase, Convex. Vertical agents. I think vertical agents are going to be a good layer.

Any specific verticals?

I don’t know if there are any verticals that I would overweight. I think the important part is having a domain-specific point of view and really diving deep there. I think Patrick Collison said this on a recent podcast with Amjad from Replit, when he was talking about the next startup he would build. It probably wouldn’t be a sexy area. It would be an area where technology, or at least AI penetration, is low, but there’s a lot of domain expertise built up, and you can go in and really transform it.

The key part is having domain expertise that can be captured in data or logic that goes in the harness, and eventually maybe the model weights, but I wouldn’t start there. You need domain expertise that can be captured and put to work. By the way, there are maybe three things I would look for in all of these, and I think they’re prerequisites. If you look at all the teams that are succeeding, one is that the talent they have is tremendously strong. Modal has an incredibly strong team. Lynn and Fireworks, right? They wrote PyTorch. How cool is that?

When we met them, you know, when you’re meeting founders, the last thing they do in the brief intro is say, “I built PyTorch.”

Yeah.

Oh, well, that works. We got a tick in that box.

To build really deep infrastructure, you have to be super technically strong, and there just aren’t that many people who can do that. Two, you have to go really, really fast. Being able to iterate quickly is incredibly important. The market changes so rapidly. Speed and execution speed matter. When I think about our advantages at LangChain, I think we iterate really quickly, and that’s one advantage we have over other companies.

The last thing would be brand. A lot of companies that made it in this AI era caught lightning in a bottle, rode that wave, and captured it. Look at LangChain early on: we went very viral and executed against that. Cursor is a great example. They went very viral and executed against that. Jev, even Claude Code and ChatGPT. The space is moving so fast. Maybe that’s because there are so many competitors. There were 100 different frameworks that came out in the 2 months after LangChain came out. But I think we had a brand advantage that we built up, and it was a durable advantage that we rode. Instinct is another example of this. There are probably a bunch of personal assistants.

In terms of some of these funding rounds, if there’s a breakout company with a really strong technical roster that has proven it can execute quickly, I think you just don’t look at the price; you just invest, even if the valuation is higher than is rational.

Is being first more important than ever before?

I think so, absolutely. It’s possible to catch up. I think OpenAI has done an incredible job of catching up with Codex. But being first, like Anthropic and Claude Code—that was a huge advantage.

But to your point, LangChain comes out, and then months later more and more come out. But Instinct is the same, and the durable brand monopoly that comes from being the first is very real.

I think that is super real. I also think that you have to execute incredibly well after that. A counterexample to what I just said: just because you’re first doesn’t mean you’re guaranteed success. I don’t know if you remember the AutoGPT project, but it came out a few months after LangChain and was the fastest-growing GitHub project of all time. I haven’t heard much from them since. That’s not a knock, because it’s very hard to execute with speed. Just because you’re first in an area doesn’t mean you’ll survive.

I never like to knock people, so I find it helpful to praise them instead. If you were to write a memo joining the firm about why we should invest in the $9 billion Jev round, what would your bull case be for Jev?

The bull case for Jev is that there’s a new type of inference and a new type of model, and the model category is literally named after them. They’re going to ride that wave, and they’re a very talented team that can execute on it. I don’t know the team super well. I think that would have to be the assumption for a round at that valuation. From everything I’ve seen, that seems to be the case. They basically invented a new model category, are associated with it, and have a talented team that can execute on it.

Does taking money from Benchmark change the trajectory of a company?

100%. I think Benchmark, when Miles and really the whole Benchmark partnership invested in us, it was me and Ankush. We were 2 people. We really thought of them as a third co-founder. When I think about company-building from 2 people to where we are today, which is about 400 people, they’ve been our most consistent partner in that journey, for sure.

I’m not a VC. I’ve never done VC. There are probably other market dynamics, too. Benchmark’s involved in that Signals thing, and we were very lucky to have Sequoia join us 2 or 3 months after the Benchmark round. They’ve been a tremendous partner as well.

Who was it from Sequoia?

Sonja from Sequoia, who’s one of the most plugged-in people in the space.

How do their styles differ?

Benchmark is maybe a little bit more focused on the core nucleus of the company. They’re going to be a sounding board for that and help with it. A lot of the company-building and positioning things, I would go to them for, and they would spend time on those. Sequoia’s style, I think, is that they help with everything. They’re just a machine. They help with recruiting individual contributors and executives. They help with marketing and launches, and they’re really operational—

Does it really work?

I think so, yeah. We’ve gotten a lot of our candidates from Sequoia referrals.

Really?

Yeah. They have a talent network, and we’ve gotten a bunch of really strong candidates from there.

Would you say to founders it’s worth taking a discount for Sequoia?

I think it’s absolutely worth taking a discount for Sequoia and Benchmark, and probably other tier-one VC firms. Those are the ones we work with.

What do you know now about fundraising that you wish you could tell yourself when it was just the two of you?

I think there’s a game to fundraising, and I definitely didn’t know that going in. I’m a first-time founder. I spend all my time on product, basically, so I’m very focused on that. I do think there’s a game to fundraising where you want to fundraise without fundraising. You want to get one term sheet and then tell others you have a term sheet—

Post a revenue graph on Twitter, and then it leads to all this inbound—

Yeah. It’s—

Do you fucking like this? I fucking hate it. Maybe I’m too old for this game now. It’s just posturing. It’s like, “Harrison, do you really like me and want to build a business with me? I’ll give you the same price and the same money and whatever. Let’s just do this. Stop fucking posturing.”

I don’t like fundraising at all.

I think, kind of what you were saying: I want to build something great with people I enjoy working with. For the scale of ambitions that we have, we need to fundraise to get that done. I think that was the approach Benchmark took when we were fundraising from them as well: “Hey, let’s figure out if we like each other and want to work together.” By the way, that was very good negotiating by them, and I think they’re very good negotiators as well.

When I look at our most recent rounds, we raised from Tom at IVP for our Series B. I’d been chatting with Tom since around the Sequoia round, for over two and a half years. It was a relationship we built up over time. I got to know him, and he got to know me. So when it came time to fundraise, I felt like there were fewer games involved, and I really appreciated that.

I would love to do a quick-fire round. I’ll say a short statement, and you give me your immediate thoughts. Does that sound okay?

Let’s do it.

What was your single favorite VC meeting?

First pitch with Benchmark. So many legends in the room.

Who was the coolest to hang with?

Bill Gurley.

Were you intimidated?

I think I was too naive to be intimidated.

He’s also just a big man. It’s an intimidating figure looming over you. What is the single most important thing you believe about agents that most of the ecosystem would disagree with?

It’s still really hard to build them. Really, really hard. Maybe people would agree with that, but I think it’s just underrated how hard it is.

What have you changed your mind on most in the last 12 months?

The market for inference is massive. I remember talking with the board about whether we should do a gateway a year and a half ago. We didn’t see what value we could add over something like OpenRouter, and I think that was 100% the wrong decision.

What’s been the hardest technical thing for you and the team to build?

We built our own database for storing agent traces, SmithDB.

How do you think about competing with ClickHouse and being a customer? Kind of weird or normal?

I think there are 500 different companies that we are frenemies with, so I’ve personally gotten very used to it.

You can invest in one IPO: OpenAI at $1.5 trillion or Anthropic at $2 trillion.

OpenAI. I think they have more diverse paths to really tremendous exits and valuations.

Totally get that. What do you know now that you wish you’d known when you started in 2022?

I wish I knew what harnesses today looked like so we could have built them earlier.

Final one for you. What are you most excited for when you look forward five years?

I’m really excited for what creative things we can do with AI. A lot of AI, and a lot of what we work on, is helping enterprises be more efficient and delivering outcomes. I think creative things like Suno are tremendously interesting, and I wish we’d see more of those.

Totally agree. I remember having Mikey on the show. That was a fun one, actually. This has been so much fun to do, Harrison. Thank you so much for putting up with my waywardness and divergent schedules, but you’ve been an amazing guest.

No, thank you so much for having me.