Devin 的 80% 时刻：后台代理、7 倍 PR 与手把手写代码时代终结——Walden Yan & Cole Murray

When people think about the ability of an AI to run your app and test it, I think they actually overindex on the computer-use part of it. Computer use, in my mind, is the literal, “Okay, you want a button, you want to click it—can you emit the right coordinates to go click that button?”

I think testing is actually a really interesting problem-solving challenge for these AIs. If you wanted to do arbitrary testing—for example, imagine you make a change that spans the front end and the back end—to actually test that change, we have to reason through how to first run these applications and orchestrate them with each other using the right version of the code. Then, how do I trigger the feature, or how do I make the thing actually happen? That is where we spend most of our time.

Before we get into today's episode, I just have a small message for listeners. Thank you. We would not be able to bring you the AI engineering, science, and entertainment content that you so clearly want if you didn't choose to also click in and tune into our content. We've been approached by sponsors on an almost daily basis. But fortunately, enough of you actually subscribe to us to keep all this sustainable without ads, and we want to keep it that way. But I just have one favor to ask all of you. The single most powerful, completely free thing you can do is to click that subscribe button. It's the only thing I'll ever ask of you. And it means absolutely everything to me and my team that works so hard to bring the inspace to you each and every week. If you do it, I promise you we'll never stop working to make the show even better. Now, let's get into it. [music]

All right, we're in the studio with Walden Yen, co-founder and CPO.

Walden Yen

Yeah.

Which is a cool title. And you're one of the coiners of “context engineering.”

Walden Yen

Yes, yes. Although I think there were many people who used the terms in various ways beforehand. But I did find that people, both internally and externally, enjoyed the upgrade from prompt engineering—or model wrapping—into maybe a more thoughtful way to build agents.

Yeah. For those who haven't caught up on that, I have the “Don't Build Multi-Agents” post on screen, which you should read, and we might refer to it. And Cole Murray, who created OpenInspect.

Great to be here. Okay, so let's talk about it. Everyone is building their own Devins. What's going on?

Yeah, so I think the engineering world is kind of waking up to this idea of background agents, cloud agents—whatever you'd like to call them. I think we saw a shift around December 2025, where the models Claude Opus 4.5 and GPT-5.2 reached a capability where we moved away from hand-holding the model and were able to more or less autonomously drive it.

What I mean by that is that we could pretty much go from a specification to a completed pull request, assuming the spec was good enough, with very little friction. That paradigm alone changed a lot of how we interact with agents and opened this world where background agents became more practical.

Alessio Fanelli

I think for Cole, everyone experienced this in December, but I feel like there was just this increasing ramp. There was the moment—which I think was Sonnet 3.7—where you guys rewrote Devin in one night or something.

Yes. So describe 2025, or how it felt from your side.

In retrospect, we always thought it was ramping up, but even now, over the last 3–4 months, it's been ramping up even faster. It's almost funny to be talking about how big of a leap Sonnet 3.7 was. Honestly, a lot of it was stripping out parts of Devin that were no longer needed with that jump in intelligence.

I also think that a lot of the recent leaps—especially when you look at models like Opus and the latest GPT models—are reaching levels of autonomy where people are actually finding that they can't just be hands-off. People who were once debating, “Do I need to be in the weeds with my model in the IDE? Can I just completely move it off into the cloud?”—that's a more serious conversation. We've seen that in all of our growth charts.

Internally, there's this funny graph where our usage of PRs, or our merged PRs, has grown 7× since—I forget what.

I think Dave maybe tweeted that.

Yeah. It grew 7× over the last 2–3 months, something like that. Then you see our engineering headcount growth. It's gone up by about 10% or something.

We were afraid to release this. So this is Devin commit percentages on all Devin repos: It was 16% in January, and now it's 80% in March.

Yeah, it's a big shift right now. It makes sense that a lot of people are now thinking about buying Devin, but also maybe trying to build their own. I have a lot of fun building Devin, so I can see why other people would want to build their own cloud agents as well.

Well, maybe it's good to hear what initially inspired you to try to build OpenInspect.

Yeah, OpenInspect came about primarily through observing how my clients were using tools like Claude Web and OpenAI's Codex at the time, and seeing some of the friction they were having with them.

Primarily, Claude Code was being used through Slack, and a big issue they ran into was that the sessions launched were specific to whoever called them via Slack. If a PM was the one who invoked the session, they would then go to pass context to engineering, but engineering couldn't see the session. That in itself was kind of a dealbreaker, because the PM would say, “Hey, engineering, can you jump in?” but there was nothing to jump in on unless they were copy-pasting out the single response that came back.

Seeing some of these problems, I had built a similar kind of architecture internally just to experiment and test out different ideas, as this trend of moving off of localhost was starting to become more prevalent. When Ramp released their blog post, I already had a lot of the pieces for this in place and thought it would be kind of funny to see what Claude could do purely from the blog post. On my X account, there's actually a thread where I live-tweeted going through this.

Oh, wow. Comparing GPT and Claude as both of them were going through it?

Alessio Fanelli

Like on the announcement or something else?

Right after it got released.

Okay. We can put it in the show notes.

Yeah, it was helpful that I already knew how to verify the system. I knew what I was looking for. I think Ramp did a great job of really illustrating the technical aspects of how to build something. It was much more than just, “Hey, we built a great system.” It was, “Here's how you can build it, too.”

I resonated a lot with that, given the problems that I was already seeing. Looking around, I didn't really see anything in the open-source community that met this type of system. I think there are a lot that run on localhost, like Superconductor and many others, but nothing that was actually running in the cloud. So I built it, and I thought it was interesting to open-source it and allow anyone to have a foundation that they could mix and match on top of.

So literally after Devin was launched, there was OpenDevin, which became OpenHands. I don't know if you tried that.

Yeah. Well, I was going to say, one of the things that interested me a lot with OpenInspect was that you didn't try to make it something you monetize. There are a lot of open-source projects that would then really try to raise a business around them.

How did you think about that? I thought that was very interesting.

What I had seen across my clients was that having a background-agent system was going to become critical infrastructure within their company. Because of that, I wanted to open-source it so that they could fork it and put in whatever customization they wanted.

To that question, I get asked all the time, “Are you going to raise? Are you going to turn this into a service?”

I'm sure you've gotten offers. But primarily, I don't want to do that for a few reasons. One, I don't want to compete for $20 a seat. I think that's just a really difficult business. It's very easy to copy the main pieces of it. I built this fairly quickly, and because you're not owning the entire stack, it's hard to monetize.

You have money being made at the sandbox layer, with Daytona, E2B, and many other players. You have money being made at the model layer. You kind of sit in this weird in-between gray area. What are you actually selling? You're selling, I guess, the infrastructure. You're selling the integrations, maybe.

Let's ask the guy: What are you selling?

Walden Yen

Well, yeah, there's multiple layers to this in practice. It's funny you mentioned the infrastructure, because when we got started building Devin as well, we had to go figure out how to make the infrastructure.

You had to build this two years before everyone else, you know.

Yeah, exactly. Including the side that was not very polished at the start. When we built it off raw VMs from cloud providers like EC2, the boot-up time was so slow, especially when turning off the machines, saving them, and then bringing them back up again when we wanted Devin to wake up later.

It would just be out cold for like 10 minutes because that's just how long these systems took. They were not built for this repeated down-and-up usage, and so we actually had to go do all of that.

As a result, now one thing we offer when we go and sell Devin to people is, you don't have to worry about all the compute side of things. We'll make it work. We'll make it work in your cloud if you want it to.

Aside from the product—and I want to go into the agents and the tuning of the intelligence part later—I think a big part of what we do at Cognition as well is to just make sure that your company learns and uses and adopts these coding agents. For especially the largest enterprises in the world, you find that there are a lot of people who want to move over to using AI for their day-to-day workloads.

But because of the way projects are planned, and because not everyone is literate in using AI in these ways, having a team of engineers who can actually go in and onboard you, set up all the integrations you need, and set up the automations you need to really get to that level of leverage with AI is super helpful. And so—

Totally.

Walden Yen

We do that. We show up as thought partners to the customers that we work with as well.

So let's talk about architectural stuff. I think that's always been something that was the topic of conversation between the two of you. Is this sort of the mental model that you want to start with, or something else? I'll just kind of leave the floor open to you guys.

Tianyi Zhang

Yeah, I think that maybe we can start here with just a general overview: What are the pieces of a background-agent system? Then maybe we can go into some of the nuances of the decisions that you can make.

I guess also, what maybe what Walden is saying is that the agent is kind of in this open code box, I guess.

Right, this is the infrastructure, and then that's the agent. You had this discussion about whether you put the agent in here or externally. Can you sort of tease that out?

Walden Yen

Yeah. In a background-agent system, you have a decision to make about where the agent is actually going to run. This is typically described as the harness in the box or out of the box.

Tianyi Zhang

Yeah, with running the agent in the box, you're making some trade-offs by doing that. The negative trade-off you're making is primarily security, because the agent is running in that box. Unless you otherwise design it, all of your secrets need to go into that box as well.

Given the nature of AI, it can be unpredictable, and you could very easily end up accidentally exfiltrating your secrets or having other kinds of unintended behavior. The out-of-the-box approach is the idea that we are going to have the actual agent running not directly in the sandbox. We'll have the “brain” of the agent running in some type of worker or control plane.

That sandbox is going to serve as the hands, where the brain is basically operating and making tool calls into that environment to manipulate it. I guess the other trade-off that you're making between the 2 systems is that, in my opinion, running it out of the box is much more complex because you have state that has to be managed.

Whereas if you're running it in the box, all of the state of that agent is actually in the box. Yes, you could persist it elsewhere, but it's all kind of localized, and you have fewer concerns to worry about.

I think a lot of what you mentioned is why we actually built Devin from the start to what we called “separate the brain from the machine.”

The other thing that this allows you to do is reuse any existing infrastructure you have for dev boxes, perhaps. You don't have to worry as much about making a new type of dev box that has all the dependencies the brain needs, or, as you mentioned, the secrets the brain needs as well.

One thing that we've seen some customers run into is that you have a GitHub app and you want Devin, your agent, whatever, to be able to interact with GitHub through this application, but then you have different users with different actual permissions.

If they're all interacting through the same GitHub app and there's no actual separation between the system that decides what it does and the actual secrets on the machine, then you kind of run into an issue where it's hard to do that separation.

In practice, with Devin, it's much easier because we just say that whatever you put on the machine is the scope of basically what the user is free to do and what the agent is free to do. Only put the most scoped secrets on that machine, and then the brain is fully not accessible from the machine.

You don't have to worry about messing with the most secure parts of the brain if the user is free to do whatever they want with the machine.

Shawn Wang

I was going to bring up this chart from OpenAI. I don't know if “in the box” and “out of the box” is something that they use to describe it. Recently, Anthropic also did managed agents, which is their version of this.

Tianyi Zhang

I don't know. It's all variations of the same pattern, right?

Shawn Wang

Yeah. So this would be out of the box.

Tianyi Zhang

Yeah.

Shawn Wang

Which is preferable for them because it's less work.

Tianyi Zhang

I would say it's more work, but in my opinion, it is the better architecture of the 2. You're taking on a bit of complexity by doing that.

One thing I've not seen a lot of other players do well is how you manage what's actually on the box. This can be complex for many reasons. Let's say you have a big repository that's changing and updating a lot, with changing dependencies. How do you make sure that the working environment of the agent actually stays up to date and has all the credentials it needs to, let's say, run the app and test it—all the things you want in your autonomous repo setup?

Yeah, exactly. Internally at Cognition, we call this repo setup. The hardest part of it has been a perennial problem since the start of the company: How do we help people get set up? Not everyone just has working cloud environments working out of the box.

Do you find this to be a common problem with your clients?

Tianyi Zhang

Yeah, this is a very common problem, and through my consulting, this is a lot of what I help teams do. A lot of teams don't really have great developer-environment setups, if any. A lot of the time, it's “Go talk to Bob and get the secrets,” and that obviously doesn't work when the agent needs to actually set this up.

Most teams are using Docker Compose or some type of microservices.

In production?

Tianyi Zhang

Not in production. With OpenInspect, you are using this primarily to interact and make code changes. There are other use cases, but through a CLI, MCPs, or other tools, you can hook that into your production systems, primarily for SRE-type use cases. You are not necessarily trying to test your production internal microservices through the system.

Yeah, and you mentioned Docker Compose. I think one direction we saw some of our friends take early on was using Docker containers as a level of abstraction for their models.

Tianyi Zhang

There are lots of reasons why Docker containers are not great. One thing is that Docker containers are not really a true security boundary, for one. The other is that if you are running real applications, a lot of times those applications use Docker, and then you have to think about Docker-in-Docker, which is really weird.

Yes. Part of the really hard challenge of getting VMs to work—why did we do that? It was because we realized that you actually needed full VMs to be able to do these types of things.

Especially nowadays, there's actually value in running the application, clicking around, and sending you screen recordings of these things. The value just keeps adding on top of that.

It is a decision I see people run into when they try to build their own systems: In addition to this, do we put the agent in the machine or out of the machine? Do we use Docker? Do we use something else? What do you recommend people use nowadays?

Tianyi Zhang

I think Docker is a good solution for maybe not running the agent, but running your infrastructure, because that is more or less the same setup your engineers are probably already using. If they're not, then I don't know what they're using, but they're probably already using Docker Compose.

Shawn Wang

I've always had a soft spot for WebContainers. I don't know if you guys have tried them before. To me, they were supposed to be like Docker Lite.

Tianyi Zhang

No, I haven't tried them.

I think any environment that you've set up that is a good experience for your developer naturally lends itself to being easy to set up for the agent. Once you figure out that local developer story, you've more or less solved the agent-in-a-sandbox environment setup.

OpenInspect does have hooks as well where you can run a setup.sh script that will pre-install everything. You can then pre-snapshot that build so it starts instantly, and then there is a second hook to restore the state of the sandbox when it comes back.

You can already have all of those microservices running and basically get the same experience that you would on your machine within the sandbox.

Shawn Wang

Another thing that we've been thinking a lot about is different VM service offerings. Have you had customers who needed macOS-specific VMs or Windows-specific VMs?

Tianyi Zhang

Not yet.

There are many technologies in the world that only work on specific types of machines, right? If you're building an application that has to run on Windows, or, more commonly, if you want to build for iOS or macOS, platform-support choices like that—

The fundamental architecture we use, because we do the separation, does support it, but the actual work in progress is happening right now on those. Another thing that we've recently added support for, which is in beta, is Android development.

To do that, we needed to support nested virtualization within our machines, because the VM itself is a virtualized Firecracker instance, and then you have to run another Android emulator inside.

Tianyi Zhang

There are some weird performance issues, which is why it’s still in beta. We have to think through these problems, but it unlocks a lot for anyone who wants to do Android development.

Shawn Wang

I was trying to find a reference video for the testing thing. I couldn’t find it, but I think you worked on the testing capability. Why do you call it testing and not computer use? What’s the general category of problem?

Tianyi Zhang

I think that when people think about the ability of an AI to run your app and test it, they actually overindex on the computer-use part of it. Computer use, in my mind, is the literal, “Okay, you want a button, you want to click it—can you emit the right coordinates to go click that button?”

I think testing is actually a really interesting problem-solving challenge for these AIs.

Shawn Wang

Like, problem-solving. Yeah.

Tianyi Zhang

If you wanted to do arbitrary testing, imagine you make a change that spans the front end and the back end, maybe even some other, more deeply nested service. To actually test that change, we have to reason through how to first run these applications and orchestrate them with each other using the right version of the code.

Then, how do I trigger the feature? How do I make the thing actually happen? This can get arbitrarily hard. Maybe you have to be an admin, maybe a certain thing has to be feature-flagged on, or maybe you have to run 2 sessions and then send a very specific word into one of them to trigger specific behavior.

Figuring out how to do that requires a lot of codebase context and a lot of orchestration that we’ve specifically built. In some cases, we found that not one frontier model can actually do this full end-to-end task itself. We’ve seen cases where we’ve had to orchestrate different frontier models together to solve this problem.

That is where we spend most of our time when we think about this testing problem, not so much the computer-use part. Computer use, for what it’s worth, has gotten a lot better with recent models, and it’s made that part of the job certainly easier.

Yeah, especially with GPT-4.7, which they released yesterday. Apparently, it’s way better in terms of the vision stuff, which is going to encompass computer use. Having evals for all these as well is something that takes a while to build up, and having the eval be right is tricky as well.

Do you ever see clients who are building their own agents have to start standing up evals to make sure things don’t regress?

Not so much evals in the traditional sense, but specific to the testing part that has just gone in, I just added support for screenshots. In theory, you can also do video. I need to put in a plugin to do that, but they do show up natively, and it was a very heavily requested feature, especially after Cursor’s recording came out.

I think that was very enlightening for everyone: “Oh, this is a very good feature to actually have.” With Devin, you guys have had this for a while.

Yeah, first. Yeah. [laughter]

Oh, yeah. I see how screenshots work.

Yeah. I don’t know if there’s anything super non-obvious. It’s kind of like, once you know what feature to build, you can just prompt it, and it mostly works.

I think, to Walden’s point, though, computer use is kind of a subset of the larger testing problem. I think that’s very specific to the codebase that you’re working in. It’s not something that, out of the box, you could just solve. You do need the codebase context to actually know how to test it.

In the case of a background-agent system, you fortunately do have that codebase locally. You know what is changing, and you can then inspect it and use that to drive the model.

Yeah. For those who haven’t seen it before, this is an example of how it works. After the PR is done, you click “Testing approved,” and then it sends you back a video.

What I really like is that it labels what it’s testing. It’s very small here, but it actually labels what it’s testing, and then you actually see the cursor and everything. I don’t know—the engineering in this, just whatever you want to show, because this is one of those “feel the AGI” moments.

Once I look at this, I actually wish I could just merge inside Slack instead of going to GitHub, because I don’t need to see the code. I know it works. Maybe a new feature coming. [laughter]

The annotations at the bottom were also a big difference for me when I added those.

Yeah. It’s just like, what am I looking at? What are you trying to demonstrate exactly?

There’s a surprisingly long tail of small details that end up making a big difference for this kind of end metric of how fast you actually merge the code in. One experience that we spent a lot of time tuning early on was the right experience on GitHub for these tools.

Sure.

I think most tools out there, when you build the agent, you think, “Oh, it will create the PR for you.” We tried to take that a step further and say, “What if we actually made sure you could interact with Devin directly on GitHub?”

We made sure that you could comment on GitHub and Devin would actually receive those comments and address them back. But there’s quite a bit of tuning you have to do here, because you can imagine that we recently had Devin Review, for example. Devin Review will post comments on its own PR, and then Devin has to go—

He answers his own comments, which is really, really loopy. I like that it just updates here that I’ve commented, but usually it’s just me saying, “Hey, merge. Fix any merge conflicts.” [laughter]

When Devin fixes its own comments, you might be scared that maybe it will infinite-loop. We put a lot of work into making sure it doesn’t, both by making sure that the comments are high-signal and by making sure that the agent is thoughtful about which comments it tries to fix and which comments it says, “Wait a second, I think you’re wrong.”

Actually, one of my favorite moments is when Devin tells me that I’m wrong when I try to get it to do something different.

Yeah.

Tuning that behavior actually makes a big difference in terms of how useful the GitHub experience is.

Xingyao Wang

Yeah. I think, to touch on that as well, having the AI reviewer integrated into the system is a critical part of this background system. OpenInspect does have that. It has a GitHub code reviewer whose prompt you can control.

It does do comments as well. It doesn’t do them automatically yet. The capability is there, but it’s not fully—

So you have to ask for it.

You do. You can tag it on GitHub, and then whatever you named your GitHub bot, it will follow up on it. If you have merge conflicts or whatever you’ve asked it to resolve, it will resolve them, but it doesn’t do it automatically yet.

I’m curious: What’s the most common thing that people end up requesting that they still need on top of OpenInspect when you help them implement it?

I think a lot of it comes down to actually integrating it into the company. It’s one thing to have the background-agent system set up, but if it isn’t actually integrated into your larger ecosystem, it isn’t that useful.

It is useful to be able to kick off sessions, but what we really want to be able to do is hook it into all of our other systems, whether that’s the production database with read-only credentials, the logs, Confluence, or an internal knowledge-base system.

I think that is where I see the huge leap for companies, and that can be a challenge for companies as well if they’re not familiar with exactly how to approach it. Especially if they’re in environments that have more compliance-type requirements, access control can be pretty important. Deliberately thinking through these problems is one of the challenges that comes with a system like this.

Yeah. The thing we’ve found is that MCPs have obviously had this really big explosion of, “Oh, you can integrate it with all these different things.” But to actually get the integration right and get the right experience, oftentimes we’ve found that we had to go build our own ad hoc things.

Slack is a great example of this. You could give your agent the Slack MCP, and then it can post messages back to you on Slack. But we actually use Devin like a coworker in Slack, and that’s how it’s been built from the ground up.

To do that, you actually need to support webhooks that come back. Devin has to respond in a natural way, and you have to make sure it doesn’t spam your threads too much and annoy the people in your company. You have to tune that experience just right. Especially when there’s a lot of back-and-forth, we found that we actually had to go beyond the simple MCP integrations in these places.

I just pulled up the MCP marketplace. I know this is a fair amount of work. Is the answer eventually to take first-party control of all the top MCPs? Is that the—

I would love a world where you could have something more expressive than MCP that goes both ways—not just a set of tools, but a proper system that interacts back and lets it have the right experience with all these interfaces.

Xingyao Wang

So there actually is sampling in the MCP spec, but nobody uses it.

Right.

Xingyao Wang

And so I think that's the other part: We found that when the MCP spec starts to get too complicated, it starts to lose its original promise of being a simple, one-step connection. Then we have to figure out how to support all these different variations of things, and it starts to look a lot like just building first-party integrations in a lot of these cases.

Yeah. I think it matters, too, how critical it is to your company, right? If this is something that nearly every session is going through, it probably makes sense to own it so that you can make optimizations on top of it.

Versus just whatever is off the shelf.

Yeah. Awesome. Other MCPs, what else? Sorry. Well, I don't know if that's narrowing in too much on integrations, but what other elements of building OpenInspect or Devin do you guys really spend time on?

Yeah, I think a problem that comes up very frequently is this idea of memories or a knowledge base.

Oh boy. [laughter] Yes. How do you solve it?

Not solved yet is the short answer. There's an open issue for it, with someone asking about it.

Okay, DeepWiki hasn't indexed anything about memory yet.

How I'm seeing it solved across my clients is primarily through skills. I find that skills can be a good gap within that, or updating CLAUDE.md.

But I think memory as a whole is a pretty unsolved problem, and it is why I've been hesitant to add it. I think there are parts of memory that can be addressed, but as a whole, it's a very difficult retrieval problem.

Oh my God, DeepWiki didn't write anything about memory. I see zero search results. No, memory can be quite tricky to get right, because it's the retrieval but also the generation of the memories that can be really tricky. You don't want it to just be very specific.

Walk us through the Devin memory journey. [laughter] I know there's been a journey.

The first version of memory that stuck around for a while was a system we called Knowledge. The idea was that we wanted it to pick up things over time and not need the user to be proactive about teaching Devin things.

Anytime you remind Devin, “Wait, no, that's not quite the way you're supposed to use Git,” we actually want Devin to say, “Hey, do you want me to just remember this for the future?” Then you can quickly approve or reject it, and it can build up over time.

I find that 95%—I think it's some crazy statistic like that—of the memories that Devin has are all through these autogenerated things. Very few people actually want to sit down and write big docs on, “Okay, here's how you're supposed to work with the technology,” et cetera.

The generation and the retrieval have been something that we've been trying to tune a lot over the years. For generation, you don't want it to remember something like, if you asked one time, “Please open a draft PR,” you don't want it to be, “Oh, everyone forever now should get their PRs as draft PRs.” But you do want some kind of common behavior. Maybe you want to say, “Cole generally likes things to be created as draft PRs.”

The same goes for retrieval. If you have thousands of these memories, how do you actually make sure they're retrieved at the right time? That can be quite tricky to do without exploding the context with a bunch of useless information. There's a surprising amount of eval work just to make sure that memory remains a reliable system as new models come and go.

Shawn Wang

Yeah. Do you have anything that you could share around memory pruning and the temporal aspect of memory?

Cole South

Yeah, exactly. Today, the things it can do are edit memories.

Shawn Wang

I see.

Walden Yen

If your memory used to say, “Cole likes to open everything as a draft PR,” then you can imagine saying, “No, don't do that.” Then it'll say, “Do you want me to update the memory to be, ‘Cole will now want everything as open PRs’?”

I think that, at the same time, we don't know if this is going to be the final version of the system. Whatever we have here will probably translate into the new system that we'll be coming up with.

But I think one big difference between 2 years ago and today is that these agents are really good at using anything that resembles a file system natively. [laughter] Part of us is thinking, “Should we rebuild memories to feel more like a file system that we let the agent navigate on its own?” That's been an interesting exploration. There are also some ideas in the skill space.

Shawn Wang

I'm pulling up OpenClaw's memory thing right now. OpenClaw has this daily memory journal thing, right? I mean, that is a file system you can kind of grep through, and it's a source of truth. I don't know if it's the best. It's probably super noisy, but at least if you lose something, you can discover it, or you can apply some kind of forgetting algorithm to more ancient memories that don't get recalled again or something.

One thing we've been trying to do to push the boundaries of how you use agents at your company is letting an agent basically have a very similar file, like a memory.mmd or something, and just be your permanent PM for a specific set of issues, maybe.

So we have some Slack channels internally, maybe a Slack channel dedicated to a specific product, like DeepWiki. You can imagine that you want a Devin that never stops. It's always awake, but it has this memory doc that it can maintain for itself about what the number-one priorities are, what we have to fix and prioritize, and who is responsible for some upcoming work.

Maybe it'll even tag you—Devin will even tag you—on some recurring basis. It's been an interesting move to see how we can actually use Devin for more than just engineering. Can we actually move upstream, above the engineering process? Maybe it's just Devin creating tickets, which then some humans do, but then maybe other Devins do.

One of my more fun automations is: “Go research competitors and just suggest stuff to me on a weekly basis.” [laughter] That's the automation, and I can't find it right now. But basically, it just looks at competitors and suggests things.

Then there's, “Here are 3 things that you've suggested that I don't want any more of,” and you just stick that in a prompt. [laughter] I wish, actually, that when I reject a PR, it updated the memory so that I wouldn't have to go back and update the scheduled sync. Feature request. [laughter]

Cole South

We might change it soon.

Shawn Wang

With OpenInspect, in the time you've been around, has there been anything you tried to implement that you then had to undo and do a different way?

Cole South

Nothing yet, but something that's on my mind is that the initial way I built it was for each of the integrations to live as its own package. So you have the Slack bot, which handles the webhooks and basically interacts with the control plane.

As I'm seeing the system become more integrated, specifically with the GitHub bot integration, I'm considering bringing that all into the central control plane. Especially now, I want to start monitoring it, and a request that I'm getting is the ability to monitor the actual pull requests being merged, as well as tracking things like: What do I have open? How many of these are getting merged? How many comments are showing up? Just to understand the health of the system.

In the case of a GitHub app, you only have 1 webhook. So then it's a question of: Do I put that webhook in the GitHub bot package? That's kind of weird. It doesn't really make sense to live there, because that package is more for the code reviewer. Or do I centralize it? That's something that's on my mind as I'm making that decision.

I think the other one we touched on earlier is the harness in the box versus out of the box. I think long term, the architecture will eventually come back out of the box. Some of the newer tools that I've added are calling back into the control plane so that you don't have the secrets in the sandbox. I think long term I probably will pull the actual agent out of the box, but I think for now it's fine.

Shawn Wang

Just a quick question on pulling the agent out of the box. One thing I'm very bullish on this year is agents calling other agents or spawning subagents, or whatever you want to call it. Does that make it harder or easier? I can't tell. Because if the harness is in the box, you can spin up more boxes.

Cole South

Yes. If the harness is outside the box, then it's less easy, because you have a unicorn pet of a harness that's living outside the box.

I mean, in theory, it would be the same way, right? Whether one agent has launched many subsessions within it—OpenInspect, for example, can launch subsessions and actually create other environments and then monitor them. In the case where it is out of the box, that would basically just be an additional session that's running.

That session is also running outside of the box. It's running in your worker plane, wherever you're running this. Then you really just have to think about how your top-level agent interacts with it. I do think it can be more complex because, again, you now have a more difficult architecture, but I think if you figured it out once, it's probably fine.

Shawn Wang

Yeah. Walden, I'm just throwing it open to you in terms of what I call meta-Devin management.

Yeah.

Shawn Wang

Which is Devin calling Devins, Devin scheduling Devins, querying trajectories, or anything like that. What have you built or unshipped?

I think one of the surprising things we've seen is that a lot of the ways these separate agents work with each other, when you want them to parallelize their work, have still mostly followed the same manager-subagent regime. A lot of people are excited about this world where you have swarms of agents that talk with each other all over the place.

We've actually given Devin an MCP so it can arbitrarily message other Devins and create new Devins, et cetera. But it creates a really chaotic world in that sense, so we've still found that the most practical use on a day-to-day basis has been one single agent figuring out how to segregate the work and having other Devins work on it in a relatively isolated sense, each with their own boxes, not sharing machines. So there's very little room for conflict. That's the regime that you have to create today.

Shawn Wang

I'll call out the experiments from Cursor, right? This is Wilson Lin's work on single-agent to multi-agent, and you're obviously famously on the side of, "Don't build multi-agent systems," but they went through the whole thing only to arrive at this—which is exactly what Devin has. Do you think—

I think there will be a revision to that post at some point. I think multi-agent systems were very much not at all possible a year ago. You do see more multi-agent experiments today, but you can argue: are they really multi-agent systems, or are they just tool calls?

There are people who will create subagents to go look for an XYZ file or an XYZ implementation. That has really nice context-management benefits, because all of the tool calls and tokens it spends then get collapsed back to just the answer for the main agent. There are a lot of benefits to doing this.

We basically have Devin do this with DeepWiki: make a call out to DeepWiki and give you back the results. But that feels like a tool call. It's not like these 2 collaborators are actually talking back and forth with each other.

The thing that gives me the most bullishness that multi-agents might actually be possible is what I said earlier about Devin actually sometimes telling me I'm wrong and pushing back. I think that demonstrates a level of maturity and communication today that makes a multi-agent world possible. When can 2 agents who have seen different information come back to each other and actually figure out who is right and what the correct implementation is? They're not just yes-men. Claude, I guess, used to just say, "You're right" or "You're absolutely right."

Shawn Wang

You're absolutely right, yeah. Have you seen the app troll in Codex?

This is the Codex app. Inside of Settings, there's a little Easter egg. If you go to Themes or Appearance, there are all these color codes, and the top is Anthropic, and it's in Anthropic's colors, which is such a troll.

Shawn Wang

I love that Easter egg. Did you discover that yourself?

No. Someone was tweeting about it, and I was like, "Is this true?" Sometimes people just tweet stuff to get a rise out of you. But yeah, there you go—the colors.

Shawn Wang

Yeah.

We're out of this regime where it just says, "You're absolutely right," and they can have real conversations and real back-and-forths.

Shawn Wang

Yeah, you can prompt it as well to be more adversarial or whatever. To me, that is more intelligence, right? That is not just something that's a dumb tool. It's actually pushing back on you.

One you mentioned—there was one blog post they had where they fed a swarm of agents together and built a browser.

Yeah, yeah, I think that was the one.

Shawn Wang

I think it's the same one, yeah.

We found surprising success with: don't do a swarm or anything. Just have one Devin. It has its own context management. Just let it keep running for a while and give it some crazy tasks. I think we asked it to rebuild a Windows OS system.

Shawn Wang

Yes.

And it managed to do it, just going on for long enough.

Shawn Wang

Was this Andrew's thing?

Yeah, yeah.

There were lots of demos that we ended up not posting because at some point we'd just be posting way too much—a bunch of demos. But I love that because it shows that the multi-agent thing still has a bit of exciting sexiness to it, which is maybe still beyond the actual delta it adds to the capabilities of these systems, but it's absolutely the future. We're heading in that direction, and we can see the progress being made there already.

Shawn Wang

If I were to make one super minor pushback, because I don't feel that confident about it yet, I've had Ryan Lopo from OpenAI on the pod. He's a super slop cannon, right? Oh my God, that's my coding agent being done.

I downloaded this thing, PongPing. I don't know if you guys have heard of it. It takes sound packs from popular games like Command & Conquer and Warcraft, and then it plays them whenever it's done. So it's "Work work" or whatever, like "At your command" or something.

What I got from the Cursor codebase and from Ryan's thing was that there's a slop-cannon approach where you try to loosen the single-agent bottleneck, and I feel like that's probably a very important thing to try to figure out. I don't think anyone has really solved it, because then you just have more reviewer slop on top of the agent slop to try to wrangle it all.

Ryan will probably strongly object if I say he hasn't solved it. He thinks he's completely solved it. But I think it's still very important, because that is a bottleneck, right? I feel Devin is slow sometimes because I'm like, "Well, yeah, this is very readable and very sensible, but it's also slower than it could be if I just—I want a button to say, 'Just ramp this up 1,000x in parallel and see what happens.'" I don't know if that's feasible at some point in the future.

Yeah, we've also run experiments internally where we've basically tried to build entire products—true products that we knew we'd eventually ship—but for now, let's try to see if we can do it purely by vibe-coding on top of each other, auto-merging, with no code review at all.

Then there's this benchmark: how many weeks can you go on with this before you say, "We have to trash this codebase and actually rewrite it from scratch?"

Shawn Wang

Yeah, what did you find?

I think we found that the state of the art in December was that you could probably run this for about 2 weeks. By the end of those 2 weeks, you'd find that you want to change the color of a button, but it turns out this button is implemented in 10 different places, with all these different variations. You forgot one of them, and actually it's a slightly different color in one spot.

Okay, this is too much to work with. Let's actually try to do code review at the same time, make sure that we're on top of our stuff, clean it up a bit, and make sure it's done in a scalable way.

Shawn Wang

Yeah, I think building on that, the idea that you don't have to look at code is generally a bad idea. The headline is: do you think that statement will be true?

I think probably for a while it'll be true that you should continue to look at your code. A problem that I see a lot of teams run into that I work with, who are embracing AI-native, AI-first coding, is that your codebase regresses to your worst engineer.

That engineer who's very gung-ho about AI and isn't auditing their code starts cementing their patterns into the code, and now the AI is referencing those patterns. Their if-else block that's 20 if-elses back and forth—the AI is seeing that as the pattern of how things are done and starts to exponentially grow this slop.

To your point, I find a pretty good approach is having scheduled cleanup, whether by humans or through systems that are looking for duplication. They then address that. You'll end up with 12 helpers for how to format a date, and you need to address that because otherwise it will continue to sprawl.

Shawn Wang

Within bounds, I think it's fine to have some duplication, and then sometimes you have garbage collection, right?

Yeah. What I've been talking about with a lot of engineering leaders is that you want to be very strict about the boundaries between modules. It's your job as an architect, as a CTO, whatever, to say, "Okay, here's the hard contract between you guys and you guys. Whatever you do inside this black box is your business. You do whatever, but between these guys, let's be really damn clear, and any movement must be signed off by a human or by me." Then that's that.

Shawn Wang

I don't know if you have any other modifications or advice.

Well, I guess generally on the topic of where humans can be useful.

I found that some of these really deep infra problems—sometimes, just having a human who has really deep expertise can make a big difference. I've actually seen this come into play when building agents. We've had a few friends try building their own coding agents, and one problem that I recurrently heard a lot of them run into was, “Grep is really slow on our agents’ machines.”

A lot of them, I assume because they're using AI and don't themselves have super-deep infra background knowledge, say, “Okay, we're going to build our own custom grep index. It's going to be really fast,” and use that as a way around this problem. When we ran into this problem about a year and a half ago, in the early days of building Devin, we obviously didn't have AI that we could just ask, “How do you do this?” You couldn't just spin up a new GPU instance.

Shawn Wang

So what do you mean you hand-coded Devin? What?

Yeah. Can you believe we hand-wrote this code? Our infra people, who are really amazing, were looking into it, and they said, “You know what? We realized that the root cause of this problem is actually super simple, but it's a fine-grained detail.”

A lot of these virtual machines actually don't use real filesystems underneath. They use network filesystems, where things are cached over the network, actually in S3. So when you're grepping, you're actually making network calls every time you're doing these things, and that's why grep is extremely slow on these machines.

Again, it goes back to all of the crazy infra work that we had to do to get these machines working. If you try to do this yourself, there are tons of small details like this, and so we eventually had to swap out that network filesystem.

Shawn Wang

Yeah, I think there's a write-up about it, right? So I listed one about the virtual—

That was a whole other thing—the block-diff file-storage format, which is a filesystem format that we built so that the VMs could be spun up and down very quickly.

Basically, the intuition behind this is: imagine you have 1 terabyte of disk, and your agent only wrote 100 lines of code on top of that disk. How long does it take to save and bring that disk back up? Most systems aren't optimized for this case, so it's on the order of 1 terabyte of work because you have to save all of that and bring it back up.

In our system, we try to build a filesystem that incrementally builds on top of itself. So every time you save and bring the machine back up, you're only doing work that's proportional to the diff in the filesystem.

This shaves off a lot of time in the boot-up process of Devin. I think this is actually now outdated—we have a newer system inside Devin—but there are a lot of tiny details you have to get right to make the day-to-day experience of Devin good.

It's not technically agents, but it is agent infra. When you sell an agent as a company, you sell the agent plus the agent infra—at least the way we do it. The other nice thing about having the agent and agent infra being done together is that we get to deploy Devin in whatever environment we want.

Now we don't need to wait for some underlying infra provider to support VPC, on-prem, or Fed GovCloud, for instance. We can actually figure out, since we own the infrastructure, how to get that set up for you.

Shawn Wang

What's your Cloudflare dependency?

Cloudflare runs the control plane. Modal is supported for the sandbox. A contributor just added Daytona. E2B is on the roadmap, and I think there's an abstraction in place so that if any contributor wants to add a new provider, they can add it in.

Shawn Wang

How are the customers you work with? Do they generally try to set up a contract with one of these third-party providers, or do they try to do the VMs in-house?

Most of them, I see using Modal. I think Modal has a great offering. It captures all of the sandbox pieces you need, with snapshots being a pretty big piece of that. Given that they also offer GPUs, I think it's a pretty nice offering as a whole.

Shawn Wang

Yeah, no debate there. Modal is great. I think their container offering is the most natural, so especially if you're willing to forego the full VM requirements, Modal is a really fast place you can spin something up on.

Is there a point—Modal is very Python, and I feel like most workloads have really shifted to JavaScript. I don't know if you guys get the same feeling. When I started Latent Space and AI Engineer and all these things, it was roughly 50/50 Python and JavaScript. I think that's wrong now. I think JavaScript has won.

I don't know if I'm overstating it. Maybe for Cognition, you know, there's C# and Java and what have you. But for new greenfield apps, do you feel that? Do you get that sense? Does it matter?

I think that most of the libraries I see in the space are Python-native first, especially in the observability space. That said, I think there's a pretty big appeal to having your entire system in one language, especially when you have both your front end and back end communicating. You can have one central type, which is very nice.

Shawn Wang

Yeah, that's my case against Modal, because then you have to run—

Scott Wu

I mean, you can run JavaScript inside Modal. It's just one extra step that isn't native to the runtime.

Shawn Wang

Yeah, I don't use it. Do you have numbers? I don't know.

Scott Wu

The one thing I don't like about Python is that whenever AI writes Python, it always does the weirdest patterns.

Shawn Wang

Because it's mixing 2 and 3, or what?

Scott Wu

Yeah, I think it's something mixing 2 and 3. I don't know if you see this, but it always tries to use hasattr on objects.

Shawn Wang

But you shouldn't be doing that. It should error if—

Scott Wu

Because it's training on library code, I think. From what I've seen, it's more of a reward-hacking mechanism where it doesn't want the code to fail. So even when it knows it has the attribute, it'll call getattr on it.

For a lot of my clients who have moved toward more autonomous coding, we've put that in as a lint rule: if you use getattr, your pull request is going to fail.

Shawn Wang

Oh, this is a fun topic. Can you tell me more about what else is a sign of AI coding that you have to put guards in?

Scott Wu

We were talking just before this about Opus 4.7. One of the things this new model likes to do is write lots of comments. Not that it'll comment on every line, but it'll write paragraph-like PRDs on top of every function.

To its credit, these aren't slop descriptions like they were before. It's not, “Here's what this function does.” It's, “Here's the reasoning and why we chose this approach, what the alternatives were, and why we shouldn't do those alternatives.”

Shawn Wang

Still too much information. But I wonder if this might be directionally correct if you want systems that can self-maintain themselves in the long run.

Scott Wu

They write their specs in the inline context in the code as well.

Shawn Wang

So you approve?

Scott Wu

I do, but at the same time, it's a tricky problem. Maybe we'll just give our users a setting for how verbose they want it to be. I haven't loved it. I like the comment, but please get rid of it.

Shawn Wang

Yeah, yeah, yeah.

Scott Wu

But I could see a world where something of the sort becomes reality.

Shawn Wang

I don't know if you guys know about Git AI.

Scott Wu

Yes.

Shawn Wang

Yeah.

Scott Wu

We've talked about it.

Shawn Wang

Git AI—the idea behind it is that if you run an agent, the actual prompts you send to the agent should be stored alongside the code inside the Git metadata, so that future agents can reference them. Maybe code-review bots can reference them. It's an ideal world where the context for why decisions are made constantly lives beside your code.

I'm waiting for the real bull case where we just get rid of Git altogether. We're not there yet, but I'm looking for it, because that would be a big shift.

Scott Wu

Kind of on the topic of visible slop, a pattern that I see a lot across GPT models specifically is backwards compatibility at all costs. It's doing these weird import-export statements so that it doesn't have to modify the names of where the modules were. I've seen Claude 4.6 starting to do this as well.

Shawn Wang

Oh, no.

Scott Wu

Again, I think it's this reward-hacking behavior where it doesn't want failure to occur. You can address that through Semgrep or other tools, where that behavior is pretty easy to identify, but it's something that you only learn through the trade of just seeing code patterns.

Untyped tuples are a really big problem. Again, it just throws in Any—Dict[str, Any]—and you can address those through linting.

Shawn Wang

Are there any other—so, like, linting, any other tools? Devin Review, of course. It’s not so free now, but you know, we still use it.

Scott Wu

One thing that I think we try to recommend to teams as they use more AI agents goes back to this local-testing thing. At the end of the day, you want your agent to be able to do the full thing—not just write the code, but actually run it and test it. A lot of codebases were not necessarily built for this from the start.

For example, you probably do want a local DB setup, a local Docker Compose and Postgres in order to have that, so that you don’t need to give your agent any crazy production credentials to actually run and test its code. We’ve also internally done a big shift to make a lot of our core components of code testable as purely local development, without needing to actually integrate with any live services, for this reason. Obviously, the older the company, the more you have to change to shift in this direction, but you can use AI to help you perform this migration.

Shawn Wang

The older the company, the more you have to change in order to do local development? I’m misunderstanding. So you’re saying most people just build with full integration to other stuff, and there’s no code path to switch it to local?

Scott Wu

Especially when there are lots of different services and you have a microservice architecture, making that shift—the larger the codebase, the harder it is. I guess if you built it correctly from the very start, it’s possible, but a lot of companies in the world got started before Docker was a thing, and so, yeah, you’re kind of forced to make a migration at some point.

Shawn Wang

Well, Devin’s very good at making mock servers.

Scott Wu

Yes, right. You know what? One of the projects I really wanted—it’s like Little Snitch. I don’t know if you guys have heard of it. I run Little Snitch on my computer. There’s a man-in-the-middle, but it shows you all the traffic going back and forth.

Shawn Wang

But then from there, you can sort of reconstruct the server, right? And then create local mocks, so you can mock everything if you just observe traffic for a little bit.

Scott Wu

Yeah, that’s an interesting idea.

Shawn Wang

Cool. I don’t know if this will get anywhere, but I wanted to maybe talk a little bit about the Claude Code leak, because usually if I have an Anthropic person on, I can’t talk about the Claude Code leak. Did you guys learn anything from Claude Code?

Scott Wu

Our team was not that interested in that leak. We didn’t spend that much time on it.

Shawn Wang

I’m just fishing for an answer.

Scott Wu

No, I didn’t really research too much into it.

Shawn Wang

Fair enough. Okay, one last thing before we go: Windsurf 2.0. You guys shipped another thing. The sort of meta-context is that if you use background agents enough, sometimes you’re going to want to bring them to the foreground, and that little handoff from local to cloud is hard to work on. Cognition has just done it.

Scott Wu

Yeah. I think for me, the biggest gap this is trying to close is, again, how do you make the testing process as fast as possible? When it can test on its own and send you a video, it’s freaking magical. Sometimes there are just really difficult things that you do need to pull down locally to test.

We just want Windsurf to be your local command center for all your agents—your background ones and your local ones. You can imagine, “Oh, okay, this agent needs me to review something.” I’ll pull that down, move my other agents to the background, go test it. Okay, boom, done. On to the next one.

You have some issue you’ve got to fix in the background? Just click “Approve,” and start a background agent to go fix it. I’d love a world where I’d never have to leave this window. Then maybe the other window I’ve got to figure out is how to stop spending so much time in Slack, but maybe someday we’ll want to get those two as well.

Shawn Wang

Yeah. Does that require the binaries to be exactly the same for local versus cloud?

Walden Yen

The funny thing here is that the behavior between local agents and cloud agents, I think, is actually a bit different in their ideal states. I think local agents should be a bit faster and let the user make the call on things. Actually, don’t try to autonomously go test things.

The background agent mode, where you start it off, I think the agent should just assume the next message I send the user should have everything that the user needs from me. It shouldn’t run and stop; it should keep running and not stop until you have the testing done.

Shawn Wang

So that’s just a slightly different prompt.

Scott Wu

Yes, but for many reasons. Because of all the work we do to make sure that Devin works with different Git providers and that it works with different OSes and VMs, we want as much of that logic to be shared as possible. For our own practical purposes, we try to share as much of it as possible.

Shawn Wang

Yeah. I can’t imagine how much work it is to transition back and forth. So congrats on shipping this.

Scott Wu

Thank you.

Shawn Wang

Okay. Anything else that we should cover before we wrap? Just whatever you guys were talking about at lunch.

Scott Wu

Maybe use cases. What are the biggest things that your clients are trying to do with their cloud agents today?

Shawn Wang

Do you want to just ask it again so we can get a clean cut?

Scott Wu

Yeah. He was drinking his water.

Shawn Wang

Yeah. The thing I wanted to talk about was use cases. What do you think are the main things that your clients come to you about today? “Hey, this is why we want to set up cloud agents.”

I think the easiest and most common use case I see across everyone is SRE use cases. The idea is that whether we have our alerts in Slack, Datadog, or wherever they’re going, we want the agent to be the first responder on that.

That doesn’t necessarily mean that the agent is actually resolving the issue, but just being able to collect that context ahead of time is huge. Again, that agent is integrated into the production logs and the database, and it has full visibility and, over time, playbooks as well for how to address certain issues.

That’s a huge win for teams because instantly you can have a full trajectory of what is going on within the system, and oftentimes actually a pull request directly from that, which is a pretty neat flow to experience: error, pull request, done. Devin does support a trigger for that as well, so that could happen completely autonomously.

Shawn Wang

From Datadog specifically, or just—

Scott Wu

It supports Sentry. It supports a generic webhook, and if someone wants to add Datadog, they can.

The other use cases that I see are for non-builder use cases, whether that’s the PM or the marketing team. I’m seeing a lot of teams where the idea of who’s actually contributing code is starting to change. In a lot of cases, if there’s just a quick bug fix, the PM is not creating an issue anymore. The PM is just prompting through Slack, and the pull request is then being created.

I think that’s a huge win. I think that trend will continue, where we’re seeing code modifications happening outside of engineering. The last common use case that I see is customer support. They’re experiencing an issue with a customer, and they’re not entirely sure why this behavior is happening.

Previously, that world was, “Hey, there’s a bug when they tried to use this feature. We don’t know what’s going on.” Now they’re tagging that in Slack. Again, that entire full context is ready. They can then just tag in engineering and have a complete understanding of that issue, completely bypassing the previous pain points of, “Can you get more information from them?”

Walden Yen

The only things I’d add on top of that that I think I’ve seen are continual security scanning and continual security review. The SRE use case, internally, we think about it as auto-triage, because we just want every message that comes in—whether that’s an alert or a bug report—to have Devin start triaging before anything else.

We’ve leaned into this use case so much that we’ve basically tried to make it so that you don’t ever have to leave Slack to interact with this. Again, making the interactions with Devin super-fluid from the moment the report comes in, to it responding to the report, to being able to ask it questions right there with full codebase context about all the issues.

That’s very related to customer support as well. I think one thing that we found is that CLIs can sometimes be very difficult for people who aren’t technical to use. An online chat interface that anyone can ask questions in is super intuitive and doesn’t assume you have any technical knowledge, but does have access to all parts of your codebase. That’s super useful for support, salespeople, or anyone who might need to have their questions answered about the codebase.

Shawn Wang

Yeah, great callout. This might potentially be a very expensive use case. Is there a rule of thumb on how much people should spend on this? You have an unlimited budget, but other people don’t. I don’t know if this is an answerable question, because obviously it depends on a lot of factors, but—

Scott Wu

I think it depends really on how people are using it. If people are using it responsibly and they’re getting value from it, then you can kind of determine the budget.

Shawn Wang

Common numbers that I hear are anywhere from $1,000 per engineer up to $5,000 per engineer.

Scott Wu

Yeah.

Shawn Wang

I have not heard anywhere in the realm of $50,000 per engineer, for a frame of reference.

Scott Wu

We'll get there. Yeah, I've seen numbers go that high for sure.

Shawn Wang

I think this is also going to be a big theme of the coming year: we're going to see very expensive, very smart frontier models, and we're also going to see people who say, “You know what? I don't need the frontier anymore for a lot of the work I do,” because some frontier models are actually good enough for a lot of the work.

Also, shout-out: you pioneered smart friend, which is a mix. I'm really interested in a world where you basically have hybrid frontier and subfrontier systems, where you use the subfrontier part to be really fast and really efficient, and call out to the frontier part of the system so that you can still get frontier performance for the most part.

Scott Wu

Yeah, I'm trying to search, but Twitter search is completely broken. The “From” field is just completely gone. It's very sad.

Shawn Wang

No worries.

Scott Wu

I might have to make a new post at some point about the return of Smart Friend.

Shawn Wang

Yeah. Yeah. I mean, Anthropic has now officially adopted it.

Scott Wu

Yes.

Shawn Wang

Okay, cool. I think that's it. It's been a really great discussion. Great having you guys on. Background agents are a thing now, and everyone's building them. We talked a lot about the production concerns and why you would want to offer one architecture over the other.

Walden Yen

Yeah, there's a real zeitgeist in the space right now, I think, for companies to want to drive themselves into these autonomous coding factories.

Yeah. Hiring.

Walden Yen

Yes, of course.

What specifically? Give one profile that's very interesting.

Walden Yen

I think people underestimate the role of really high-taste product engineers in the space right now.

Okay.

Walden Yen

And the test is: what have you shipped end to end that is a tasteful product?

If you've shipped stuff that you think is tasteful and you're proud of, you should come talk to us.

Yeah, for me, any businesses that are looking to further their engineering org—a lot of the consulting I do is around that. Teams who are maybe starting their AI journey, whether that's with Cursor or Claude Code, but they're looking for someone to help navigate them through the state of the art and beyond just that initial deployment. As mentioned, there's a lot of lift from deploying the background agent to how we actually get this fully integrated into the company and really realize the true value of that.

Okay, well, thanks, you guys, for coming on.

Scott Wu

Cool. Thanks for having us.

Yeah. Thank you.