面向智能体的上下文工程——Lance Martin，LangChain

I'm Alessio, founder of Kernel Labs, and I'm joined by swyx, founder of Small AI.

Hello. Hello. We are so happy to be in the remote studio with Lance Martin from LangChain, LangGraph, and everything else he does. Welcome.

It's great to be here. I'm a longtime listener of the pod, and it's finally great to be on.

Yeah, you've been part of our orbit for a while. You spoke at one of the AI Engineer Summits, and we're also pretty close with LangChain. Recently, though, you've also been doing a lot of tutorials. I remember you did R1 DeepResearch, which is a pretty popular project, and Async Ambient Agents.

But the thing that prompted me to reach out and say, “Okay, it's finally time for the Lance Martin pod,” is your recent work on context engineering, which is all the rage. How did you get into it?

1. Context Engineering Emerges

Well, it's funny. Buzzwords often emerge when people have a shared experience. I think lots of people started building agents early this year, midyear—quote unquote, the year of agents.

What happened is that when you put together an agent, it's just tool calling in a loop. It's relatively simple to lay out, but it's actually quite tricky to get to work well. In particular, managing context with agents is a hard problem.

Karpathy put out that tweet canonizing the term “context engineering,” and he mentioned this nice definition: “Context engineering is the challenge of feeding an LLM just the right context for the next step.” That's highly applicable to agents, and I think it really resonated with a lot of people.

I had that experience over the past year working on agents, and I wrote about it a little in my piece on building Open Deep Research. It was an interesting point: the term captured a common experience that many people were having, and it took hold because of that.

How do you define the line between prompt engineering and context engineering? Is prompt optimization context engineering in your mind? I think people are confused. Are we replacing the term? What is it?

I think prompt engineering is a subset of context engineering. When we moved from chat models and chat interactions to agents, there was a big shift.

With chat models, working with ChatGPT, the human message is really the primary input. Of course, a lot of time and effort is spent crafting the right message that's passed to the model.

With agents, the game is trickier because the agent is getting context not just from the human. Context is now flowing in from tool calls during the agent trajectory.

I think this was the key challenge that I observed, and that many people observed: when you put together an agent, you're not only managing the system instructions, system prompt, and user instructions; you also have to manage all this context that's flowing in at each step over the course of a large number of tool calls.

There have been a number of good pieces on this. Manus put out a great piece talking about context engineering, and they made the point that a typical Manus task is about 50 tool calls. Anthropic's multi-agent research is another good example. They mentioned that a typical production agent—and this is probably referring to Claude Code, but it could be other agents they've produced—is making hundreds of tool calls.

When I had my first experience with this, I think many people had the same experience. You put together an agent, you're sold the story that it's just tool calling in a loop, and that's pretty simple. You put it together, and I was building deep research. These research tool calls are pretty token-heavy, and suddenly I found that my deep researcher, with a naive tool-calling loop, was using 500,000 tokens. It was costing $1 to $2 per run.

I think this is an experience that many people had. The challenge is realizing that building agents is actually a little bit tricky because if you just naively plumb in the context from each of those tool calls, you hit the context window of the LLM. That's the obvious problem.

But Jeff from Chroma spoke about this on the recent pod. There are all these weird and idiosyncratic failure modes as context gets longer. Jeff has that nice report on context rot.

You have both these problems happening. If you build a naive agent, context is flowing in from all these tool calls—it could be dozens to hundreds—and there's degradation in performance with respect to context length. There's also the trivial problem of hitting the context window itself.

That was the motivation for this new idea that it's actually very important to engineer the context that you're feeding to an agent. That spawned a bunch of different ideas that I put together in the blog post, drawn from Anthropic, my own experience, Manus, and others.

I'm just going to put some of the relevant materials on screen because we like to have some visual aid. We did our post on GPT-5, and we called it “Thinking with Tools.” Part of the tools is to get context, and I think using tools to obtain more context lets the agent figure out what context it needs if you just tell it to.

I thought you had done a blog post on this, but apparently this was it. I will say it's funny, and I was hoping you'd bring this up. I also have a blog post, but it's all moving so quickly.

I did a meetup after the blog post and updated the story a little bit with that meetup. This is actually a better thing to show, but I do have a blog post too. Things changed between my blog post and the meetup, which were about 2 weeks apart. That's how quickly these things are moving. Exactly—that's the blog post.

Should we do this sequentially, then?

I think it's actually okay to just hit the meetup.

Because it's easier to follow one thing, and it's a superset of the blog post story. Okay. How do you define the 5 categories?

2. Offloading Agent Context

When I talked about naive agents, and the first time I built an agent, the agent made a bunch of tool calls. Those tool calls were passed back to the LLM at each turn, and you naively just plumbed all that context back. What you see, of course, is that the context window grows significantly because the tool feedback is accumulating in your message history.

A perspective that Manus shared, in particular, which I thought was really good, is that it's important and useful to offload context. Don't just naively send back the full context of each of your tool calls. You can actually offload it, and they talk about offloading it to disk.

They talked about using the file system as externalized memory. Rather than writing back the full contents of your tool calls, which could be token-heavy, write those to disk and write back a summary. It could be a URL or something else, so the agent knows it retrieved something and can fetch it on demand. But you're not just naively pushing all that raw context back to the model.

That's the offloading concept. It could be a file system, but it could also be agent state. LangGraph, for example, has this notion of state, so it could be the agent runtime state object. It could be the file system. The point is that you're not just plumbing all the context from your tool calls back into the agent's message history.

You're saving it in an externalized system and fetching it as needed. This saves token cost significantly. That's the offloading concept.

I guess the question on offloading is: What's the minimum summary, metadata, or whatever you need to keep in the context to let the model understand what's in the offloaded context? If you're doing deep research, you're offloading the full pages, maybe, but how do you generate an effective summary or blurb about what's in the file?

This is a very interesting and important point. I'll give an example from what I did with Open Deep Research. Open Deep Research is a deep-research agent that I've been working on for about a year, and it's now, according to DeepResearch Bench, the best-performing deep-research agent—at least on that particular benchmark. It's pretty good.

Listen, it's not as good as OpenAI's Deep Research, which uses end-to-end RL. It's fully open source and pretty strong.

I just do carefully prompted summarization. I try to prompt the summarization model to give an exhaustive set of bullet points covering the key things in the post, so the agent can know whether to retrieve the full context later.

I think careful prompting for recall, compressing the result, and making sure that all the key bullet points necessary for the LLM to know what's in that piece of full context are actually included is very important when you're doing this kind of summarization step.

Cognition had a really nice blog post talking about this as well, and they mentioned that you can spend a lot of time on summarization. I don't want to trivialize it, but at least my experience has been that it works quite effectively.

Prompt a model carefully to capture exactly. In this post, they talk a lot about even using a fine-tuned model for performing summarization. In this case, they're talking about agent boundaries and summarizing, for example, message history. But the same challenges apply to summarizing the full contents of token-heavy tool calls so the model knows what's in context. I basically spent a lot of time prompt-engineering to make sure my summaries capture, with high recall, what's in the document while compressing the content significantly.

I do think that compression was also part of the meetup findings from yesterday, when we were at the context engineering meetup that Chroma hosted: you do want frequent compression because you don't want to hit the context limit. Yeah, I don't think there's much else to say. Offloading is important, and you should probably do it. There was also a really interesting link—I think somebody, possibly Dex, was linking it to the concept of multi-agents—and why you do want multi-agents is because you can compress and load in different things based on the role of the agent. Probably a single agent would not have all the context.

3. Multi Agent Isolation Gets Tricky

Yeah, you know, that's exactly right. Actually, one of the other big themes I hit and talk about quite a bit is context isolation with multi-agents, and I do think this links back to the Cognition take, which is interesting. Their argument against multi-agents—

It can be hard with multi-agent systems.

Correct. And what they're arguing is a few different things. One of the main things is that it is difficult to communicate sufficient context to subagents. They talk a lot about spending time on that summarization or compression step. They even use a fine-tuned model to ensure that all the relevant information is captured. They show it a little bit further down as a kind of linear agent, but even at those agent boundaries, they talk a lot about being careful about how you compress information and pass it between agents.

Yeah, I think the biggest question for me—I mean, coding is the main use case that I have—is how much value there is in showing how the implementation was made in order to then write. If you have a subagent that writes tests or a subagent that does different things, how much do you need to explain to it about how you got to the place the codebase is in, versus not? And then does it only need to return the test to the main agent? If it has to fix some code to match the task, should it say that to the main agent? I think it's clear to me in the deep research use case, because it's atomic pieces of content that you're going through. But when you have state that depends between the subagents, I think that's the thing that's still unclear to me.

So I think that's one of the most important points about this context isolation kind of bucket. Cognition argues—which I actually think is a very reasonable argument—don't do subagents because each subagent implicitly makes decisions, and those decisions can conflict. You have subagent one doing a bunch of tasks and subagent two doing a bunch of tasks. Those decisions may conflict, and then when you try to compile the full result—in your example, with coding—there could be tricky conflicts.

I found this to be the case as well, and I think a perspective I like on this is to use multi-agents in cases where there's very clear and easy parallelization of tasks. Cognition and Walden Yan spoke on this quite a bit. He talks about this idea of read versus write tasks. For example, if each subagent is writing some component of your final solution, that's much harder. They have to communicate, as you're saying, and agent-to-agent communication is still quite early.

With deep research, it's really only reading. They're just doing context collection, and you can do a write from all that shared context after all the subagents work. I found this worked really well for deep research, and Anthropic reported on this too. Their deep researcher just uses parallelized subagents for research collation, and they do the writing in a single step at the end. So this works great. It's a very nuanced point: what you apply context isolation to in terms of the problem matters significantly.

Coding may be much harder. In particular, if you're having each subagent create one component of your system, there are many potentially implicitly conflicting decisions each of the subagents is making. When you try to compile the full system, there may be lots of conflicts. With research, you're just doing context gathering in each of those subagent steps, and you're writing in a single step.

I think this was a key tension between the Cognition take—“Don't do multi-agents”—and the Anthropic take—“Hey, multi-agents work really well.” It depends on the problem you're trying to solve with multi-agents. This was a very subtle and interesting point. What you apply multi-agents to matters tremendously, as does how you use them.

I like the take that you should apply multi-agents to problems that are easily parallelizable and read-only—for example, context gathering for deep research—and do the final “write,” in this case report writing, at the end. I think this is trickier for coding agents. I did find it interesting that Claude Code now allows for subagents. They obviously have some belief that this can be done well, or at least that it can be done. But I still think I agree with Walden's take: it can be very tricky in the case of coding if subagents are doing tasks that need to be highly coordinated.

I think that's a well-explained contrasting comparison. Not much to add there. I think it's interesting that they have different use cases and different architectures involved. I don't know if that's a permanent thing, or if that might fall to the bitter lesson, as you would put it.

Yes, we should probably talk about some of the other parts of the system that you set up. There are a lot of interesting techniques there.

4. Retrieval Without Indexing

Well, let's talk about classic old retrieval. RAG has obviously been in the air for many years, well before LLMs and this whole wave. One thing I found pretty interesting is that different code agents take very different approaches to retrieval.

Varun from Windsurf shared an interesting perspective on how they approach retrieval in the context of Windsurf. They use classic code chunking along carefully designed semantic boundaries, embedding those chunks. So, classic semantic-similarity vector search and retrieval, but they also combine that with, for example, grep. Then they do knowledge graphs. They also mention combining those results and reranking. So this is your classic, complicated, multistep RAG pipeline.

Now, what's interesting is that Boris from Anthropic and Claude Code has taken a very different approach. He's spoken about this quite a bit. Claude Code doesn't do any indexing. It's just doing “agentic retrieval,” using simple tool calls—for example, using grep to poke around your files—with no indexing whatsoever, and it obviously works extremely well.

So there are very different approaches to RAG and retrieval that different code agents are taking. This seems to be an interesting and emerging theme: when do you actually need more hardcore indexing? When can you just get away with simple agentic search using very basic file tools?

Yeah, one of the more viral moments from one of our recent podcasts was Boris—who partnered with us—and Cline also mentioning that they just don't do code indexing; they just use agentic search. That's a really good 80/20. If you really want to fine-tune it, you probably want to do a little mix, but maybe you don't have to for your needs.

I actually just saw a Cline post, I think yesterday, saying that they only use grep; they don't do indexing. So I think within the retrieval area of context engineering, there are some interesting trade-offs you can make: are you doing classic vector-store-based semantic search and retrieval with a relatively complicated pipeline, like Varun's talking about with Windsurf, or just good old agentic search with basic file tools?

I will note that I actually did a benchmark on this myself. I think there's a shared blog post somewhere. I'll bring it up right now. Yep, I actually looked at this a bit myself. This was a while ago, but I compared 3 different ways to do retrieval on all the LangGraph documentation for a set of 20 coding questions related to LangGraph.

I basically wanted to allow different code agents to write LangGraph for me by retrieving from our docs. I tested Claude Code and Cursor. I used 3 different approaches for grabbing documentation. One was that I took all of our docs—around 3 million tokens—indexed them in a vector store, and just did classical vector-store search and retrieval.

I also used an llms.txt file with a simple file-loader tool. This is more like agentic search: basically, look at this llms.txt file, which has all the URLs of our documents with some basic description, and let the LLM—or the code agent, in this case—make tool calls to fetch specific docs of interest.

I also just tried context stuffing: take all the docs, 3 million tokens, and feed them all to the code agent. These are just some results I found comparing Claude Code to Cursor, and interestingly, what I actually found was that llms.txt with good descriptions—which is just very simple.

It’s just basically a Markdown file with all the URLs of your documentation and a description of what’s in each document. Passing just that to the code agent, with a simple tool to grab files, is extremely effective. The code agent can say, “Okay, here’s the question. I need to grab this document and read it.” It’ll read it, then say, “I need to grab this document,” and read that one. This worked really well for me, and I use it all the time.

I personally don’t do vector-store indexing. I use an llms.txt file with a simple search tool, and Claude Code is kind of my go-to in this case. This was done a few months ago, and these things are always changing. At this particular point in time, Claude Code actually outperformed Cursor for my test case. That’s actually what led me to Claude Code. I did this back in April, so I’ve been kind of on Claude Code since.

That was really it. This kind of goes to the point that Boris has been making about Claude Code and Anthropic as well. You give an LLM access to simple file tools. In this case, I use an llms.txt file to help it out, so it can know what’s in each file. That’s extremely effective and much simpler and easier to maintain than building an index. That’s just my own experience as well.

The scaled-up form of llms.txt, which I really like and use quite a bit, is actually DeepWiki from Cognition. I made a little Chrome extension for myself where, for any repo, including yours, I can just hit DeepWiki. It’s an llms.txt kind of thing, but I actually read it. It’s just a better wiki.

No, no. This is a great example. I actually think that this could be a very nice approach: take a repo and compile it down to some kind of easily readable llms.txt file. What I found was that even using an LLM to write the descriptions helped a lot. I have a little package on my GitHub where it can rip through documentation and pass each page to a cheap LLM to write a high-quality summary. This works extremely well, so the llms.txt file then has LLM-generated descriptions.

Let’s see where it is. Go to my repos. No, that’s old. That’s much older. I have a million things here. Go back up to the top. I have too much here. Try repositories.

You do too much, man.

Yeah, too much open.

It won’t be at the top. It’ll be—

Yeah, this one. This is a little repo. It got almost no attention, but I found it to be very useful. Basically, it’s trivial. You just point it to some documentation, and it can rip through it, grab all the pages, send each one to an LLM, and have the LLM write a nice description and compile it into an llms.txt file.

I found that when I did this and then fed it to Claude Code, Claude Code was extremely good at saying, “Okay, based on the description, here’s the page I should load. Here’s the page I should load for the question asked.” It’s dead simple. I use this all the time when I’m trying to generate llms.txt for new documentation. I’ve done this for LangGraph and a few other libraries that I use frequently. You just give that to Claude Code, and then Claude Code can rip through and grab docs really effectively. It’s super simple.

The only catch is that I found the descriptions in your llms.txt file matter a lot, because the LLM has to use the descriptions to know what to read. Anyway, that’s just a nice little utility that I use all the time.

When we had a client who said that Context7 by Upstash, which is an MCP for project documentation and stuff like that, was one of the most-used tools, have you seen it? Have you tried it? Have you seen anything else like that that automates some of this away?

It’s funny. We have an MCP server for LangGraph documentation that basically gives Claude Code, for example, an llms.txt file and a simple file-search tool. Claude has built-in fetch tools now, but at the time we built it, it didn’t. It’s a very simple MCP server that exposes llms.txt files to Claude Code, for example. It’s called MCP Doc, so it’s a very simple utility. I use that all the time. It’s extremely useful. You can basically just point it to all the llms.txt files you want to work with.

Yeah, this one.

Well, MCP Doc has an MCP server that you can search the docs with, so it kind of rolls all the way down. But I guess my question is: should this be one server per project, or at some point are you going to have a meta-server? I think part of it is that once you move on from just doing tool calling in servers to doing things like sampling, prompts, resources, and stuff like that, you can do a lot of the extraction in the server itself as well. It goes back to your point on context engineering. Maybe you do all that work not in the context, but in the server, and then you just put the final piece that you care about in the context. It seems like it’s very early.

This is actually a very interesting point. I’ve spoken with folks from Anthropic about this quite a bit. I’ve found that storing prompts in MCP servers is actually pretty important, in particular to tell the LLM or code agent how to use the server. I end up having separate servers for different projects with specific prompts. Sometimes I’ll also have specific resources for that particular project in the server itself.

I don’t mind separating servers by project, with project-specific context and prompts necessary for that particular task. A lot of people may have missed some features of the MCP spec. You do have prompts in there, and it’s probably one of the first actual features they have that’s somewhat underrated. People tend to view MCP as just tool integration, but there’s actually a lot of stuff in there, including sampling, which is underrated too.

Yeah, that’s right. The prompting thing is pretty important, because even to use our little, simple MCP Doc server for LangGraph docs, I found it’s better if you prompt it. Initially, I had to put in the README, “Here’s how you should prompt it,” but of course that prompt can just live in the server itself. You can compartmentalize the prompt necessary for the LLM to use the server effectively within the server itself.

This was a problem I saw initially. A lot of people were using our MCP Doc server and finding, “Oh, this doesn’t work well.” It was a skill issue: you needed to prompt it better. But that’s our problem. The prompt should actually live in the server and should be available to the code agent—

Right?

—so it knows how to use the server—

Right?

So that’s maybe retrieval, and that’s a whole big theme. Retrieval obviously predates this new term, context engineering, but there’s a lot going on in the retrieval bucket. It certainly is an important subset of context engineering.

I’m wondering if there are any other trends in retrieval. Before we leave the topic, one other thing I was tracking was ColBERT and the general concept of late interaction. I don’t know if you do a ton on that, but some sort of in-between element between full agentic retrieval and full pre-indexing—sort of two-phase indexing, maybe—is what I would call it. Any comments on that?

I haven’t personally looked at ColBERT very much. I played with it only a little bit, so I don’t have much perspective there, unfortunately.

All right, happy to move on.

5. Compaction Risks Information Loss

We could talk about reducing context briefly. Everyone’s had an experience with this, because if you use Claude Code, you hit that 95% mark in the context window and Claude Code is about to perform compaction. That’s a very intuitive and obvious case in which you want to do some kind of context reduction when you’re near the context-window limit.

I think an interesting take here, though, is that there are a lot of other opportunities for using summarization. We talked about it a little bit previously with offloading, but tool-call boundaries are a pretty reasonable place to do some kind of compaction or pruning. I use that in Open Deep Research. Hugging Face actually has a very interesting Open Deep Research implementation. It uses not a coding agent, but a code-agent implementation, so instead of tool calls as JSON, the tool calls are actually code blocks that go to a coding environment that runs the code.

One argument they make there is that they perform some kind of summarization or compaction and only send limited context back to the LLM. They leave the raw tool call itself, which is often token-heavy when you’re talking about deep research, in the environment. Anthropic and their multi-agent researcher also do some summarization of findings. I think you see pruning show up all over the place. It’s pretty intuitive.

I think an interesting counter to pruning was made by Manus. They make the point—and offer the warning—that pruning comes with risk, particularly if it’s irreversible. Cognition gets at this too. They talk about how we have to be very careful with summarization. You can even fine-tune models to do it effectively. That’s actually why Manus has the perspective that you should definitely use context offloading.

So perform tool calls, offload the observations to, for example, disk so you have them. Then, sure, do some kind of pruning or summarization, like Alessio was asking before, to pass useful information back to the LLM, but you still have that raw context available to you. So you don't have lossy compression or lossy summarization. I think that's an important and useful caveat to note on the point of summarization or pruning: you have to be careful about information loss.

This is something that people do disagree on, and I'll just flag this on pruning mistakes, pruning wrong paths. Manus says, “Keep it in,” so you can learn from the mistakes.

Some other people would say that once you've made a mistake, it was going to keep going down that path. If there was a mistake, you've got to unwind, or you've got to prune it and tell it, “Do not do the thing I know to be wrong.” So then you just do the other thing. I don't know if you have an opinion, but I would call this out. There was someone who spoke yesterday who disagreed with this.

So that's actually very interesting. Drew Breunig has a really nice blog post on context failure modes.

So he has a few.

Yes. This one: context poisoning. Drew Breunig has a nice blog post that hits this point. He talks about this theme of context poisoning, and apparently Gemini reports on this in its technical report.

For example, a model can hallucinate, and that hallucination then gets stuck in the history of the agent. It can poison the context, so to speak, and steer the agent off track. I think he cited a very specific example from Gemini 2.5 playing Pokémon, which they mention in the technical report. So that's one perspective on this issue: we should be very careful about mistakes in context that can poison the context. That's perspective 1.

Perspective 2, like you were saying, is that if an agent makes a mistake—for example, calling a tool—you should leave that in so it knows how to correct. I think there is an interesting tension there. I will note that it does seem that Claude Code will leave failures in. I notice when I work with it, for example, it'll have an error, the error will get printed, and it'll use that to correct.

In my experience, when working with agents, particularly for tool-call errors, I actually like to keep them in personally. That's just been my experience. I don't try to prune them. Also, for what it's worth, it can be tricky to prune from the context, from the message history. You have to decide when to do it, so if you're introducing a bunch more code, you have to manage.

I'm not sure I love the idea of selectively trying to prune your message history when you're building an agent. It can add more logic you need to manage within your agent scaffolding or harness. That's a classic sort of precision-recall tradeoff, but reinvented for context in an agentic workflow.

Exactly. Right. While we're on the topic of Drew, Drew is obviously another really good author. He's coined a bunch of context-engineering lore. Any other commentary on stuff that you particularly like or disagree with?

I'll show you something kind of funny. If you go to his post, he and I did a meetup on this, and I like this quote from Stewart Brand. It was kind of comical: “If you want to know where the future is being made, look for where language is being invented and lawyers are congregating.”

It was talking about this idea of why buzzwords emerge. He was the one who turned me on to the idea that a term like context engineering catches fire because it captures an experience that many people are having. They don't come out of nowhere.

If you scroll down a little bit, he talks about this. He has a whole post about how to build a buzzword, but he talks a lot about the idea that successful buzzwords capture a common experience that many of us feel. I think that's the genesis of context engineering, largely because many of us build agents and realize there are lots of ways that can be quite tricky. Then, “Oh, context engineering is kind of what I've been doing,” and you hear a number of people saying it. It resonates, and you say, “Oh, okay, yes, that describes my experience.” I think that's an interesting aside on how language emerges anthropologically in different communities.

I will cosign this because that's exactly what I used to coin—or come up with—“AI engineer.”

AI engineer. No, exactly. Just because people were trying to hire software engineers who were more up to speed with AI, and engineers wanted to work at companies that would respect their work, maybe also free them from the baggage of classical ML engineering. A lot of AI engineers don't even need to use PyTorch because you can just prompt and do typical software engineering. I think that's probably the right way, at least in a world where most models—most of the frontier models—are coming from closed labs.

Alessio, do you have experience with that? I'm the worst at naming things, but you do a great job, Shawn.

Yes.

You nailed it. The few that you put on Latent Space.

That's right. Cool. Well, you know, I wanted to talk about context engineering. Okay, so—sorry, I don't know if I sidetracked you a little bit with the meta stuff—

No, that's perfect.

That hits a lot of the major themes. I can maybe just talk very briefly about one more. We could talk about the bitter lesson and some other things.

Yeah. If you go back to that table, I just wanted to give Manus a shout because I thought they had one other very interesting point.

Oh, the table that you had.

Yes, exactly. So we've talked about offloading, reducing context, retrieval, and context isolation. Those are, I think, the big ones you can see very commonly used.

I do want to highlight Manus. I thought they had a very interesting take here about caching, and it's a good argument. When people have the experience of building an agent, the fact that it runs in a loop and that all those prior tool calls are passed back through every time is quite a shock the first time you launch an agent. You have 1 token every tool call, and you incur that token cost every pass through your agent.

Manus talks about the idea of just caching your prior message history. It's a good idea. I haven't done it personally, but it seems quite reasonable. Caching reduces both latency and cost significantly.

Yeah, but don't most other APIs auto-cache for you? I mean, if you're using OpenAI, wouldn't you just automatically have a cache hit?

I'm actually not sure that's the case. For example, when you're building an agent, you're passing your message history back through every time. As far as I know, it's stateless.

I think there are different APIs for this across the different providers. Especially if you use just the Responses API, the new one, it should be that if you're never modifying the state—which is good for you if you believe that you shouldn't compress conversation history, and bad for you if you do—then everything that you passed in previously is going to be cached, which is nice. Anthropic used to require a weird header thing, and they've made it more automatic.

Okay, so that's a good callout. I had used Anthropic's caching header explicitly in the past, but it may be the case that caching is automatically done for you, which is fantastic if that's the case. I think it's a good callout for Manus.

Yeah, Gemini also introduced implicit caching. It's really hard to keep up. You basically have to follow everyone on Twitter and read everything. So that's my bullet point for it.

Yeah. It's interesting, though: APIs are now supporting caching more and more. That's fantastic. I had used Anthropic's explicit caching header in the past. I do think an important and subtle point here is that caching doesn't solve the long-context problem.

Of course, it solves the problem of latency and cost. But if you still have 100,000 tokens in context, whether it's cached or not, the LLM is utilizing that context. This came up when I asked Anton this in their context-rot meetup, or in their context-rot webinar, and they mentioned that the characterization of context rot they made would apply whether or not you're using caching.

So caching shouldn't actually help you with all the context-rot and long-context problems. It absolutely helps with latency and cost.

Yeah, I do wonder what else can be cached. I feel like there's definitely a form of lock-in because you ideally want to be able to run prompts across multiple providers and all that. Caching is a hard problem. Ultimately, you control your destiny if you can run your own open models, because then you also control the caching. Here, everything else is just a half approximation of that.

That's right. Exactly right. That's the overall, broad context engineering.

Alessio, I don't know if you have any other takes from the meetup yesterday or questions.

No, I think my main take from yesterday was the quality of compacting. I think one of the charts was showing the automated compacting of OpenCode and some of these tools, and it was basically the same as not doing it in terms of the quality of what you get from the previous instructions. I think Jeff at this talk said curated compacting is 2× better, but I'm like, how do you do curated compacting?

I think that's something that maybe we can do a future blog post on. I'm interested in how you compact coding agents, especially when things can get very, very long. For things like deep research, once I get the report, it's fine, but for coding, I would like to keep building.

I found that even when you're writing tests or doing changes, having the previous history is helpful to the model. It seems to perform better when it knows why it made certain decisions, and I think how to extract that in a way that is more token-efficient is still unclear. So I don't have an answer, but maybe this is a request for work by people listening.

Yeah, that's a great point. It actually echoes some of Walden and Dan's points from Cognition, too: the summarization and compaction step is nontrivial. You have to be very careful with it. Devin uses a fine-tuned model for doing summarization within the context of coding, so they obviously spend a lot of time and effort on that particular step.

Manus calls out that they are very careful about information loss whenever they do pruning, compaction, or summarization. They always use a file system to offload things so they can retrieve them. So it's a good callout that compaction is risky and very tricky when you're building agents.

6. Memory Learns From Feedback

I think there was previously a lot of interest in memory, and I'm always thinking about the interplay between memory and context engineering. Are they kind of the same thing? Is it just a rebrand? Are there parts of memory—and you guys recently relaunched LangMem, which is also a form of context engineering—but I don't know if there's a qualitative or philosophical difference.

Yeah, so that's a good thing to hit, actually. I maybe think about this on 2 dimensions: writing memories and reading memories, and then the degree of automation on both of those.

Take the simplest case, which I actually quite like: Claude Code. How do they do it? For reading memories, they just suck in your CLAUDE.md files every time. So every time you spin up Claude Code, it pulls in all your CLAUDE.md files. For writing memories, the user specifies, “Hey, I want to save this to memory,” and then Claude Code writes it to CLAUDE.md.

On this axis of degree of automation across read and write, it's kind of like the 0,0. It's very simple and very Boris-pilled—super simple—and I actually quite like it.

Now the other extreme is maybe ChatGPT. Behind the scenes, ChatGPT decides when to write memories and when to suck them in. Actually, I thought Simon at AI Engineer had a great talk on this. It wasn't about memory, but he hit memory in the talk, and he mentioned—I don't know if you remember this—it was a failure mode in image generation.

He wanted an image of a particular scene, and it sucked in his location and put it in the image. It sucked in Half Moon Bay or something and stuck it in the image, and it was a case of memory retrieval gone wrong. He didn't actually want that.

So even in a product like ChatGPT that spent a lot of time on memory, memory is nontrivial. I think my take is, well, the writing of memories is tricky: when actually should the system write memories? Reading memories actually kind of converges with the context engineering theme of retrieval.

Memory retrieval at large scale is just retrieval. I kind of view it as retrieval in a certain context, which is your past conversations.

That's right.

You know, in the case of sophisticated memory retrieval, it is just like a complex RAG system, in the same way we talked about with Varun and building Windsurf. It's kind of a multistep RAG pipeline.

So I kind of view memories, at least the reading part, as just retrievable. Actually, I quite like Claude's approach. It's very simple; the retrieval is trivial: just suck it in every time.

Totally. I would also highlight the semantic differences that you've established: episodic, semantic, procedural, and background memory processing. We've done an episode with the Letta folks on sleep-time compute, which I think—if you have ambient agents or very long-running agents—you're going to run into this kind of context engineering, which was previously the domain of memory.

I would say that the classic context engineering discussion doesn't have this stuff—not yet.

Yeah, so actually there's an interesting point there. I did a course on building ambient agents, and I have this little email assistant that I use to run my email. I actually think this is a bit of a sidebar in memory: memory pairs really well with human-in-the-loop.

For example, in my little email assistant, it's just an agent that runs my email. I have the opportunity to pause it before it sends off an email and correct it if I want, like, “Change the tone of this email,” or I can literally just modify the tool call to have a little UI for that.

Every time you have these ambient agents, when you edit the tool calls themselves or give the agent feedback, that feedback can be sucked into memory. And that's exactly what I do. I actually think memory pairs very nicely with human-in-the-loop.

When you're using human-in-the-loop to make corrections to a system, that should be captured in memory. That's a very nice way to use memory in a narrow way that's just capturing user preferences over time. I actually use an LLM to reflect on the changes I made, reflect on the prior instructions in memory, and just update the instructions based upon my edits.

That's a very simple and effective way to use memory when you're building ambient agents that I quite like.

There's a course which you can find on GitHub. And, yeah, you guys have done plenty of talks on agents.

That's right. But I think it's a very good point that memory is often confusing—when to use it. I think a very clear place to use it is when you're building agents that have a human-in-the-loop, because human-in-the-loop is a great place to update your agent memory with your preferences.

So it kind of gets smart over time and learns. It's exactly what I do with my little email assistant. So Harrison, I'm sure—I think he said this publicly—uses an email assistant for all his emails. He gets a lot as a CEO. I get much fewer because I'm just a lowly guy, but I still use it.

Yeah, totally. I've tried to use the email system before, but I'm still very married to my Superhuman.

Yeah, fair enough. That's right.

That's right. Okay, so cool. I think that was about the coverage that we planned on context.

That's great.

You have a little bit on the bitter lesson that we could wrap up with.

7. The Bitter Lesson For Agents

Yeah, that's a fun theme to hit on a little bit. I'd love to hear your perspective. So there's a great talk from Hyung Won Chung—

Previously at OpenAI, now at Meta Superintelligence Labs.

Yeah—

—on “The Bitter Lesson” in his approach to AI research. The take is that compute 10×s every 5 years for the same cost.

Of course, we all know that the history of machine learning has shown that capturing this scaling is the most important thing. In particular, algorithms that are more general, with fewer inductive biases and more data and compute, tend to beat algorithms with more hand-tuned features and inductive biases built in.

That is to say, just letting a machine learn how to think for itself with more compute and data, rather than trying to teach a machine how we think, tends to be better. So that's the bitter lesson, simply stated.

His argument is this subtle point: at any point in time, when you're doing research, you typically need to add some amount of structure to get the performance you want at a given level of compute. But over time, that structure can bottleneck your further progress.

That's what he's showing here: in the low-compute regime, on the left of that x-axis, adding more structure—for example, more modeling assumptions and more inductive biases—is better than less.

But as compute grows, less structure—and this is exactly the Bitter Lesson point—less structure and more generality tend to win out. So his argument was that we should add structure at a given point in time in order to get something to work with the level of compute that we have today, but remember to remove it later. A lot of his argument was that people often forget to remove that structure later.

I think this applies to AI engineering, too. If you scroll down, I have the same chart showing my little example of building Open Deep Research over the course of a year. I started with a highly structured research workflow. It didn't use tool calling. I embedded a bunch of assumptions about how research should be conducted—in particular, don't use tool calling because everyone knows tool calling is not reliable. This was back in 2024.

I decomposed the problem into a set of sections and parallelized each one, with those sections written in parallel into the final report. What I found is that you're building LLM applications on top of models that are improving exponentially. So while the workflow was more reliable than building an agent back in 2024, that flipped pretty quickly as LLMs got better and better and better.

It's exactly as was mentioned in the Stanford talk: you have to be constantly reassessing your assumptions when you're building AI applications, given the capabilities of the models. I talk a lot here about the structure—the specific structure I added: the fact that I used the workflow because we knew tool calling didn't work. This was back in 2024. The fact that I decomposed the problem because it's how I thought I should perform research—this basically bottlenecked me.

I couldn't use MCP as it got much more popular. I couldn't take advantage of the fact that tool calling was getting significantly better over time. So then I moved to an agent, started to remove structure, allowed for tool calling, and let the agent decide the research path.

A subtle mistake that I made, which links back to that point about failing to remove structure, was that I actually wrote the report sections within each sub-agent. This kind of links back to what we talked about with sub-agents in isolation. Sub-agents don't communicate effectively with one another. So if you write report sections in each sub-agent, the final report is actually pretty disjoint. This is exactly Alessio's challenge and problem about using multi-agent. I actually hit that exact problem.

So I ripped out the independent writing and did one-shot writing at the end, and this is the current version of Open Deep Research, which is quite good. At least on deep research, it's the best-performing open deep research assistant that's open source. It was kind of my own arc, although we do have some results with GPT-5 that are quite strong. The models are always getting better, and so our open-source assistant actually takes advantage and rides that wave.

But I felt like I actually experienced the Bitter Lesson myself. I started with a system that was very reliable for the current state of models back in early 2024, but I was completely bottlenecked as models got better. I had to rip out the entire system and rebuild it twice, rechecking my assumptions in order to capture the gains of the models.

So I just want to flag that this is an interesting point. It's hard to build on top of rapidly expanding models and rapidly improving model capabilities. I really enjoyed engineer Boris's talk on Claude Code. It's a very Bitter Lesson build. He talks a lot about the fact that they make Claude Code very simple and very general because of this fact. They want to give users unfettered access to the model without much scaffolding around it.

Yeah, exactly. He hits it in one of these slides. I don't know where.

Yeah, but I think it's an interesting consideration in AI engineering that we're building on top of models that are improving exponentially. One of the points he makes is that a core layer of the Bitter Lesson is that more general things around the model tend to win. When building applications, we should be thinking about this.

We should be adding the structure necessary to get things to work today, but keeping an eye on models improving rapidly and removing structure in order to unbottleneck ourselves. I think that was my takeaway. I really liked the talk from Hyung Won Chung. I think that's worth everyone listening to, and I think a lot of the lessons apply to engineering.

I think this is similar to incumbents adopting AI, putting it in existing tools because you already have the workflow, right? You already have all the structure; you just put AI in, and it becomes better.

And that's why you have Cursor and Windsurf being better than VS Code for an AI-native thing, because they didn't have to deal with removing things. That's why Cognition, again, doesn't even think about the IDE as the first thing; the IDE is a piece of the agent. I think you see this in a lot of markets: again, if you have a workflow and you put AI in it, the workflow is better.

The workflow is not the end goal. I think we're now at a place where you should just start without a lot of structure, simply because the models are so good. But I think the first 2.5 years of the market had this stance: should I just put AI into the workflow that works? Should I rewrite the workflow? But the workflow isn't that good because the models aren't that good. I think we're past that point now.

That's an amazing example, actually. If you show your chart again, there's another interesting point in your chart. An interesting point here is that in the earlier model regime, the structured approach is actually better.

Jared Kaplan, the founder of Anthropic, has a great talk at Startup School from a couple of weeks ago, and he mentions this point about how oftentimes building products that explicitly don't quite work yet can be a good approach because the model under them is improving exponentially, and it'll kind of unlock the product. We saw that with Cursor. Part of the Cursor lore is that it did not work particularly well. Claude 3.5 hits, and then boom, it kind of unlocks the product.

You hit that knee in the curve when the model capability catches up to the product needs. But in that earlier regime, the structured approach appears better. So it's kind of this interesting, subtle point: for a while, the more structured approach appears better, and then the model finally hits the capability needed to unlock your product, and suddenly your product just takes off.

There's another corollary to this: you can get tricked into thinking your structured approach is indeed better because it'll be better for a while, until the model catches up with less structured approaches.

Your chart looks very similar to the Windsurf chart. I have to bring it up because I was involved in writing this one. Isn't that similar? There's a ceiling, and then boom, you go slow. It's the Bitter Lesson, but in enterprise SaaS.

That's right. Very similar.

For me, the lines are important, but the bullet points are the main thing. If you understand the bullet points, then you can actually learn from the mistakes of others.

I spend a lot of effort on the bullet points.

Right. Cool. Generally, there is one spicy take on this: how much is LangGraph aligned with the Bitter Lesson? Obviously, you guys are aware of it, so it's not going to be a surprise. But I do think that making abstractions easy to unwind is very important if you believe in the Bitter Lesson, which you do.

No, no, this is super important, actually, and I talked about this in the post.

8. Low Level Frameworks Stay Flexible

There's an interesting subtlety when you talk about agent frameworks, and a lot of people are anti-framework. I completely understand and am sympathetic to those points. But I think when people talk about frameworks, there are 2 different things.

There can be a low-level orchestration framework. There's a great talk, for example, at Anthropic from Shopify. They built this kind of orchestration framework called Roast internally, and it's basically LangGraph. It's some kind of way to build internal orchestration workflows with LLMs.

LangGraph provides you low-level building blocks—nodes, edges, and state—which you can compose into agents and workflows. I don't hate that. I like working with low-level building blocks; they're pretty easy to tear down and rebuild. In fact, I used LangGraph to build Open Deep Research. I had a workflow, I ripped it out, and I re-agented it. The building blocks are low-level: just nodes, edges, and state.

But the thing I'm sympathetic to is that, in addition to low-level orchestration frameworks, there's also agent abstractions, like “from framework import agent.” That is actually where you can get into more trouble, because you might not know what's behind that abstraction.

I think when a lot of people are anti-framework, what they're really saying is they're also anti-abstraction. They're largely anti-abstraction, which I'm actually very sympathetic to. I don't particularly like agent abstractions for this exact reason, and I think Walden Yan made a good point: we're very early in the arc of agents; we're in the HTML era.

Agent abstractions are problematic because you don't know what's necessarily under the hood of the abstraction. You don't understand it, and if I was building, for example, Open Deep Research with an abstraction, I wouldn't necessarily know how to rip it apart and rebuild it when models got better.

So I'm actually wary of abstractions. I'm very sympathetic to that part of the critique of frameworks, but I don't hate low-level orchestration frameworks that just provide nodes and edges. You can just recombine them in any way you want. Then the question is: why use orchestration at all?

Actually, I use LangGraph because you get some nice things: checkpointing and state management. It's low-level stuff. That's the way I happen to use LangGraph, and that's why I like LangGraph. That's actually why I found a lot of customers like LangGraph. It's not necessarily for the agent abstraction, which I agree can be much trickier.

Some people like agent abstractions. That's completely fine, as long as you understand what's under the hood. But I think that's a very interesting debate about frameworks. I think the critique should be made a little bit more about abstractions, because often people don't know what's under the hood.

For those who are looking for resources, it was a bit hard to find the Shopify talk because it's unlisted.

Yeah, it's unlisted now. Exactly. I don't know why it's unlisted, but it's a nice talk. I found it through this Chinese rip-off of the talk.

Funny. There you go.

Yeah, it's hard. It's actually hard to find now.

I think there should be a browse comp where you find obscure YouTube videos, because that's something I'm very good at. That's just my bread and butter.

It's good. And you know what's funny is this talk follows exactly the arc we often see when we're talking to companies about LangGraph. People want to build agents and workflows internally. Everyone rolls their own, and it becomes hard to manage, coordinate, and review code in the context of large organizations.

It can be very helpful to have a standard library or framework that people are using, with low-level components that are easily composable. That's what they build with Roast. That's effectively what LangGraph is, and that's why a lot of people like LangGraph. I actually thought the talk on MCP—I believe it was at AI Engineer—it was John Welsh.

Yes. I think that was a super-underrated talk. I tried yelling about it, but no one listened to me. It's very good.

It's very good. So, actually, he makes a case for a lot of the reasons why people—for example, enterprises and large companies—like LangGraph. The fact is that when tool calling got good within Anthropic in mid-2024, he makes this point explicitly.

Yes, exactly. It's somewhere right around here. Actually, there's a timeline slide if you go back 1 or 2. It's very good. This is very interesting. He mentions that tool calling in Anthropic got good in mid-2024. Everyone's building their own integrations. It becomes complete chaos, and that's actually where MCP came from: let's build a standard protocol for accessing tools.

Everyone adopts it. It's much easier to have a standard and have review, and you minimize cognitive load. This is actually the argument for standardized tooling, whether it be frameworks or otherwise, within larger organizations: it's practicality. His whole talk is making that very pragmatic point, which is actually why people do tend to like frameworks, for example, in larger organizations.

Agreed. And then ship it as a gateway. This is the other big thing that they do.