⚡️ 2026 年软件工厂基础设施技术栈 — Zach Lloyd, Warp.dev

Hey, it's me again. I've heard a lot of positive feedback about our episode intros, so I'm going to keep trying to do them as far as time allows. Today, we're going to dive into Warp with Zach Lloyd. Zach has been working on Warp for a while, and it's pivoted into what it is today, which is basically the definitive software factory that is also open source. He has this whole definition, I think, which has done really well. His talk at AIE has been going viral as one of the top talks of all time. I think effectively everyone is trying to set this up for themselves. They have a solution for you, but it's also kind of an open spec that is becoming table stakes for what you're going to want. I think you're going to want cloud, a control plane, and all these basic security features and team features that you take for granted. So we're releasing this episode as part of a preview for AIE Code. AIE Code is the last AIE flagship of the year. It's going to be in San Francisco, where we're gathering all the top agentic engineers and talking about software factories, among other things. System one models, Diogo is coming back, and raising your ambitions, so Tariq and Jason and Brandon from ArtCraft is doing his first conference talk ever. Zach will also be giving a great keynote on software factories, recapping the year and previewing what's next. Join us and it'll be lots of fun. See ya.

We're in the remote studio catching up with Zach Lloyd from Warp. Welcome.

It’s good to be here. Thanks for having me.

Zach, you’ve been going viral on AIE. What’s up?

You’ve hit a nerve. You’ve hit a nerve.

People are really interested in software factories, and that’s an area I’ve been focusing on, and that Warp has been focusing on. I think we have some strong, interesting takes on what a software factory actually is and what it means to be a software developer. People have been into it.

Your H1 on the landing page says, “Open infrastructure for cloud software factories.” Since you spend more time than I do talking to people about factories, what do people think a software factory is, and what do they actually need?

When I talk to companies or developers, most of the world is trying to build some sort of internal software factory or set of software factories. What I think it means—or maybe I’ll just define what it means to me—is an agentic system in the cloud where you’re defining the roles of various agents across building software for your team or your product. There’s a certain amount of autonomy in that system, so the agents can do things on their own.

That system, to me, should be very measurable and closed loop. Over time, your team can tune it, measure it, benchmark it, and ultimately gain confidence that people switching to this way of agentic development are doing it in the most scalable and efficient way.

What do people come to you with that’s a common misconception? Are they thinking, “Okay, if I buy whatever, I’m going to come out the other end with this amazing money machine”?

When people come to us trying to build their own software factory, they’re often stringing together different automations. Sometimes they think a software factory isn’t a specific thing; it’s just using agents a lot to ship software, and they call that their internal software factory.

What we try to do to make that more concrete is say, “No, you should actually think of a software factory as a very specific thing. You should define your factory in code. You should make sure that the factory has all the workflows, skills, and MCPs.” It’s a concrete thing that you should be able to run at some point and measure how it’s doing.

People have all sorts of random ideas about what these factories are. For Warp Factories, which is our platform, we say, “No, it’s actually a real thing. You can orchestrate it. You can measure it. You have one factory per product surface.” We try to take it out of the realm of a nebulous collection of arbitrary automations and interactive agents and make it an actual thing.

The way to think about a software factory—or the way we think about it, and what you’re looking at here is Warp’s software factory platform—is that a factory operates on some product surface. Imagine you’re building your mobile app or your server app. The factory itself operates on a set of repos, and it has a set of skills, a set of MCPs, and a workflow associated with it.

That workflow roughly matches what your team is probably already doing to make changes to that codebase. There could be a step where you’re doing triage, speccing, or implementation. The important thing is that there’s some defined workflow, and that workflow itself is defined in code.

For us, we basically have a YAML file that says, “These are all the repos you could be using. These are all of the MCPs, all of the integrations that are hooked up.” You basically have this whole thing defined somewhere.

The way to think about using the factory is that there are two personas. There’s someone building the product, doing product engineering and shipping features through a factory. Those people use the factory through their existing tools. They could be Slacking the factory to put work into it, using a task tracker like Linear or Jira, interacting with the factory through GitHub, or calling it via API or MCP. The metaphor is that you put a work item into the factory, and when it goes in, that starts a predefined workflow that’s defined in code. Does that make sense so far?

I love defining things in code.

Yes. You have this whole thing set up in code, and once you have that in place, the metaphor is that someone starts a task in Slack or Linear, or through an automation—for example, there’s a server error. It all goes through one system. That’s the key thing.

1. Measuring Agent Performance

Once you have that working, you get a whole bunch of benefits. Wherever these agents are running, you get a real-time view across Warp’s internal team of how we’re using our own factories. You get a full trace of what everyone is doing with coding agents across any product that has a factory.

This is cool not just because you can see what people are doing—like what Sofic[?] is doing—but because it starts centralizing all of the agent conversations, agent artifacts, and work being done by coding agents across your team. When I talk about having a closed-loop system, these agent conversations and traces are the key aspect of how the system improves.

For instance, looking at one of these agent traces, Sofic[?] is working on our mobile app. Sofic[?] was doing this in Slack, so I could pull it up there if I wanted to, but the system is actually generating a full record of the conversation. It has a full record of the computer-use screenshots that are coming out of building this thing. You can see exactly what the different pieces of the orchestration agent were doing.

You have this not just for one thing, but for every single thing. When I talk to companies, for the most part, they don’t have something like this because most of their engineers are individually using Claude Code in their terminal, the Codex app, or something like that in their own local setups. But they want to get to a world where they have this centralization.

When you have that centralization, you can get all this interesting data on how your agentic coding is working across your company. You can think of this data along different dimensions. For example, what’s our team’s output in the world of using coding agents? That includes traditional DORA metrics: how long does it take to ship something? It includes how automated you are. We have a metric for how many times a human had to do something for every PR that gets merged. There’s also throughput.

There are metrics around cost. This is something we’re constantly using, and when we talk to companies, cost is very important. You want to be able to tune your factory with the right models and skills to bring the cost per PR down over time. You want to be able to slice the cost by person, by automation, and by where in the SDLC you’re spending money.

The final thing that’s really cool, once you have this closed-loop system going, is that you can look under the hood at how well your coding agents are doing along different dimensions. You can think of this as online evals, or what we call scorers.

These can range from very general things: Are your agents being too verbose? Are they being efficient? Is the code quality good? We have work to do on this.

Do I, as a user, define this, or do you provide it as a platform?

Both. We provide a set of tools out of the box that we think are generically useful. But as we work with companies, they may have specific failure modes for their agents that they’re interested in tracking. They may ask, “Hey, is the agent making a specific type of change that’s breaking our build?” They may want something very specific to their workflow. So again, this is all defined in code and fully customizable.

When you set one of these up, what’s happening behind the scenes is that you have a set of runs—historical agent conversations that have been stored—and an agent retroactively grades them. This is a good example: it’s grading for redundant tests, which is a common failure mode with coding agents. Sometimes they write a bunch of tests they don’t need while figuring out how to implement something, and then they leave them in the code. That bloats your codebase and wastes tokens.

If you wanted to check for that, you’d create one of these evals. To show you a little more about what the eval looks like, think of it as an LLM-as-a-judge setup. You pick a judge model, then see where these run, what the prompt for judging them is, and the sampling rate, because you may not want to run these agents too often; you have to be wary of the cost.

Once you have this set up, think of it as having all these smart agents looking across your team’s coding sessions and saying, “Okay, sometimes agents are writing too many tests.” You get a record and an understanding of where you should look to improve the way your team uses coding agents. So far, so good? I want to show a couple of other things to make it more specific what you can do with this.

Overall, good. I’m a little bit wondering if there are different levels of instruction in Warp: Is what you’re showing us the CTO level? You’re running the company; you’ve got a bunch of coders.

The people who use this include engineering managers or the CTO, but also very commonly the platform team. It’s not typically your day-to-day engineer. When we work with companies, they’ll often have an internal team—the DevEx team or the platform team—that’s trying to figure out how to optimize coding agents. What I’m showing you here is the tool set they use to ask, “How do we get the most out of Claude Code at our company, or Codex or something like that?”

That part is clear.

Yep. Cool. A couple of other things I want to emphasize—not to make this completely about Warp Factories, but the general principle for anyone trying to build their internal factory, whether on Warp Factories or not, is that you want to get to a place where everything is stored, observable, and measurable. That’s when you can start applying engineering rigor to making coding agents better.

You see this on Twitter all the time. Everyone is always tweeting, “You should check out this new model. You should use this new harness. What’s better? Is it Pi or Claude Code or Cursor or whatever?” Our opinion is that the world needs to move out of the realm of vibes for understanding coding agents and into the world of measurement. We think this is the direction companies will want to go. They’ll want to put in place a system like Warp Factories, whether they use Warp Factories or build their own, that collects all this information and shows how their coding agents are doing.

2. Factories That Improve Themselves

One thing I take for granted as part of this system is that you can create all these different automations around it. People work in Slack, but they’re also setting up different trigger-based ways of putting work into the factory. Once you have the view I showed earlier, where you score how your factory is doing, you can set up another loop of agents—and everything is loops these days—that looks for failures in your scores. The agents look for things that aren’t going well and try to suggest changes to your system to make it work better.

A simple example is what we call self-improvement. Our self-improvement agents were looking at how well our triage step was doing. They found 54 failure cases of a particular form where it would have been better if the triager had been able to spawn a subagent to investigate. You can look at why it thinks this, if you want, but the cool thing is that, because you have your whole factory defined as code, coding agents can essentially make a pull request against the factory definition to try to improve the factory itself. This is a full learning loop built into the system to try to self-optimize.

We don’t have this yet, but the next thing we want to do is take any suggested change like this, replay the runs, and make sure it actually would have improved them. The possibilities for engineering this stuff get pretty cool once you have the right primitives in place.

I’ll show one last thing, and then we can talk more generally about the whole approach. Another thing I think companies should strive for, and that we find very useful internally, is the ability to do what we call benchmarking. In the factory context, benchmarking means being able to take any coding agent that’s running in your system and add it to a benchmark suite. You might curate a set of coding agent runs related to front-end coding, for example. Then you can replay those same tasks against a matrix of different configurations.

I’m a big believer in not just letting you vary the model, but also the harness. In Warp, we have our own harness, but we’ll also let you benchmark against Claude Code, Codex, or Pi. You can pick the model, and what you get looks very similar to the published results for SWE-bench or Terminal-Bench, but on your company’s own real runs.

For instance, we’re looking at a cost-versus-correctness trade-off for a certain set of tasks. You want to be in the upper-left-hand corner: lowest cost, highest correctness. These models are now slightly older because the world is changing so fast. But we could see that GPT-5.6 Sol was 95% correct on this task set and cost $6 on average, while GLM-5.3 Flash was 93% correct and cost $2. The cool thing is that you can take this information and feed it into a model-routing strategy—or harness-routing strategy, honestly—defined specifically for your own workflows and codebase.

If I zoom out, the big picture is that you’re getting out of the realm of guessing and into what we call factory engineering. You’re taking a real engineering mindset and asking, “We have this amazing new primitive of coding agents. How do we know as a team, as a company, that we’re deploying them in the right way?” We’re trying to build the infrastructure and tools for companies to do this. This approach is really resonating with a lot of companies that have been trying to build their own versions of it. That’s a quick tour of what I mean when I talk about Factory. I can stop talking for a second.

I do think the power move is to thought-lead so hard that everyone ends up trying to build Warp, even if they don’t use Warp.

Yeah.

So, you have something on your landing page that I like. You’ve done a few of these in your talks as well. Something I really like is “any surface.” It really has to be any surface. It has to be factory as code. You have to have the eval loop. What is the whole thing? People are calling this the ADLC. I don’t know if you use that term. It looks like you still use SDLC, and all that stuff.

This is how I would think about it.

3. The Factory Infrastructure Stack

Again, whether it’s Warp or not, Warp is what we’re fully focused on. We have an awesome platform for it. But in general, if you’re trying to build something in this space, I would view it as a stack—as infrastructure. At the bottom of the stack, I think it’s really foundational and important to define your factory declaratively as code.

The analogy is Terraform. You get the same benefits, but actually more. Defining it as code gives you a versioned view of your SDLC, AI-DLC, P-PDLC—whatever DLC you want. You can create branches off it and test them, and if you change something, you can revert it. You get all the obvious benefits of source control.

Then you get an added benefit that’s extremely cool: because it’s in code, coding agents can suggest improvements to it. To me, this is the foundation of a self-improving software factory setup: define your factory declaratively. The next step up in the stack is a place to gather all the information agents are producing, so you can examine it, look for failure modes, or replay past tasks with different variables to optimize your system from a measurement standpoint.

You need the data to do that: the agent conversations and artifacts, and a record of all the interactions from all the tools that led to a result. You might have a factory conversation that starts in Slack, continues in Linear, then in GitHub, where there’s a code review comment, and that whole interaction results in a piece of merged code. It might even result in something getting shipped and then having a server crash. You want to be able to go back retroactively, most likely with an agent, and diagnose how it could have been better and what failed.

You also need a way of getting data into the system, so you need some way of plugging in MCPs. As companies get bigger, they want different access for different parts of their organization: what MCPs the marketing team can access versus the sales team or the engineering team. Depending on the size of your organization, you’ll find there’s a lot of complexity if you’re trying to roll something like this out.

You really want a compute layer, and you should think of that as a separate thing. There are a lot of agent sandbox providers now, so if you’re going to use a third-party one, I would make that pluggable. We use Namespace as our primary one.

Ooh. They launched it at AI Engineer Europe.

Namespace is awesome, but it should be pluggable. I think that’s really important. If you wanted to use Daytona, Modal, E2B, or whatever, I would architect it so this is a pluggable component. Depending on your requirements, make sure you can use different platforms. By default, I think you’ll probably want a Linux-type sandbox. But for computer use, like the way we test iOS apps, you’ll want the ability to run a Mac VM, run the simulator, and click around.

We have a very open approach to inference, in line with our ethos of trying to be super flexible. A lot of companies right now have preexisting commitments with Anthropic, OpenAI, or Google that they want to burn down. We provide a wide variety of options: you can plug in your own inference provider, or you can buy inference through us. You can even plug in your own harness, which I think is a pretty big deal. You can orchestrate things like Pi, as I mentioned earlier—basically any ACP-compatible harness.

Then you want this improvement layer. This is some of the stuff I showed in the demo: the real power of this whole system is the ability to improve it. When we first started using the factory approach at Warp, it was costing us about $80 per PR because we were using Opus where we shouldn’t have. Our triage agent was using a really high-powered model, and we were doing redundant work with the shape of our agents. We had too many sub-agents instead of just having fewer agents.

We did a bunch of internal tuning and testing using these built-in features, and we were able to get the cost per PR down to about $20 or $15, something like that. We can measurably improve the throughput of our PRs. Over the last month, it’s gone up by 40 percent. We measure the revert rate, and it hasn’t gone up. I feel a lot of confidence in this general approach of observe, measure, tune.

Finally, you need some sort of orchestrator. Sometimes people talk about the orchestrator like it’s the whole platform. I think it’s a piece of it. By orchestrator, I mean you need a system that takes the factory definition code and moves work along the different stages, calling out to the right models, loading the right skills, and so on. You need something that fires the right automations.

Finally, you need a bunch of access points. Our philosophy—which I can see people having different views on—is to bring the factory into a company’s existing knowledge-work tools. The most obvious one is Slack, and about 60 percent of our work starts there. Sometimes it makes sense to start in the task tracker. Sometimes a developer who’s more comfortable using Claude Code in their terminal will do a bunch of work there and then submit a spec into the factory.

Our philosophy is to make the whole thing API-driven and MCP-driven, so the factory itself is a piece of headless infrastructure with all these different entry points. This stack-based view seems to land very well when I show it or talk to companies. They’re like, “Okay, yes, this makes sense. This is a new thing in the developer infrastructure stack. It sits one level back from where CI/CD is, but I think most companies will have something like this for software development.”

Yeah, that’s a lot of layers, actually. I’ve never actually—

Yes.

—seen it spelled out like you have.

Yeah.

I wonder, because I think most of the stories around Devin at Cognition and Cursor, whatever, try to do some bundle of things from bottom to top. There’s a little bit of bundling and unbundling with all software. There’s a critical question: I’m trying to do my job as CTO of a B2B SaaS. Do I need all these six layers?

I think the question is, who is this for? When we talk to companies, this resonates most with companies that have some amount of scale. If you’re a few people, honestly, this is overkill.

Whatever, yeah.

Don’t do this. Use the Claude Code subscription or the Codex subscription or whatever, and you’ll go faster. You probably don’t need—

Yeah, same—

You could use this. Yeah, yeah.

Same for any larger team stack, right? At some point you switch from fire, YOLO, whatever, to “We actually need a platform system here to accelerate everybody.”

Exactly. You have to be at a certain scale. We start seeing that maybe around five to ten engineers would be the floor where this makes sense. All the way up to very large companies, they want something like this. The bigger the company, the more implementation work this requires. I’m sure if you’re working with Cognition—or I don’t know how much Cursor does—to get something stood up like this within a company’s own infrastructure, it’s not going to be purely plug-and-play. The complexity of a team’s internal developer setup grows with size.

But there are a bunch of companies with maybe ten to a hundred engineers, fast-growing growth-stage startups, where it’s pretty self-serve. If they’re using the happy stack of Slack, Linear, Jira—

Hang on. Did you call it the happy stack?

I called it—yeah, that’s our… No shade to Teams or whatever, but there’s a set of integrations I find very common at startups where you can plug something like this in and get immediate value without too much setup. If you’re working with Devin or something, you can get some of the same experience. When you’re working in an environment that’s natively multiplayer, like Slack or Linear, there are all these benefits you might not realize from having a central system of record. With the Slack integration, for instance, you can get a flow of information coming in through a server crash channel and some through a user feedback channel.

You can just feed that directly into the factory and have it do the work. The factory can loop in multiple people on your team, depending on who knows the right part of the codebase. I don't think you have to be that big for this to make sense, but it definitely doesn't make sense for a small team. It really makes the most sense for a company that's like, “I want to put the right infrastructure in place so that as models and agents continue to change, we have a system we're building upon.”

There's a lot to go on here. The one follow-up question, and then I'm going to have you talk about your AI Engineer talks and generally where people can follow up with you. What is the strongest criticism of setting all this up? When you encounter criticism or pushback, what should people know about what others are running into?

4. Factories At Company Scale

I'll answer in terms of a certain class of customer where I don't think it makes sense to go in on this particular thing.

Yeah, and beyond the size thing.

The companies where we have challenges right now are those with very bespoke infrastructure. A customer who isn't a good fit for us, for instance, would be Uber or Ramp. They're building their own thing and have very large internal developer teams. But apart from the size thing, for some people it's just going to be more economical to use the subscriptions. For very small teams, though, that comes back to the size thing.

5. The Subscription Cost Advantage

Okay. Actually, I don't want to omit the subscriptions stuff, because I think a lot of people go straight to the API, paying by API, and that's between 5 and 50 times more expensive than just getting a whole bunch of subscriptions.

Yeah.

Do people use that? Is that so against the terms of service that people just shouldn't bother?

We talk to a lot of small companies that say, “Oh, this system seems amazing. We want this, but we're on this collection of $200 subscriptions that we're rotating through.” I'm like, it's probably not—

People are rotating through 17 different subscriptions, right? And then you get—

I can't imagine that's allowed, but I don't really know. The truth is, they are doing it, and honestly, if you can do that, it's going to be the cheapest way. But you lose a whole bunch of other benefits, it's not scalable, and if your company scales to a certain point, you're definitely not going to be allowed to do that. But it makes sense.

Yeah. I'm just saying, that's the extent of the subsidy. For a $200 plan, you get $8,000 worth of API usage, right?

Yeah. It's crazy.

That's not money to sneeze at.

No, no.

Yeah.

From a business perspective, it's one of the reasons I don't want to be in the pure coding agent business, because you just can't build—

Because good luck fighting—

You can't build a competitive coding agent on top of something that costs so much more money. That's just the reality of it. Cursor was brilliant. They did it, and then I think they had a perfect exit and became a model lab, because I think they fully realized the economics of it. The cool thing about our factory is that we're not trying to pitch a token-reselling product. We're trying to pitch a coding agent optimization platform and a coding agent productivity platform. We're not in that game with this product, which is a strength. I know there are still a few independent coding agents out there. I don't know how they're going to do it.

It's basically Cognition and OpenCode. That's it. Oh, and Factory, of course. Let's wrap up with a little plug for your talk. People can come see you, and we'll livestream it. Give us the general arc of the series you've been developing and where you're going to talk about next.

The talk is going to focus on what it looks like to be an engineer in this world of increasing factory engineering. What are your roles? What's your job going to be? What are the 2 types of engineering: product engineering and factory engineering? And what does this new discipline, if you want to use that word, of factory engineering actually look like? What are the tools? How do you, as an engineer, build the right skills so that you're engineering the system that builds the software, and not just building the software directly?

That is a perfect recap and overview. If people want to talk to you, where should they find you? If you're hiring, what are you hiring for? Anything that's like a call to action.

Yeah. So come check out warp.dev. Most stuff is there. We're hiring engineers, on our go-to-market team, on the data team, and a designer, so we're hiring for a whole bunch of roles. If you're interested in my musings on factories, I put out a lot of content on my Twitter now, @zachlloydtweets. Come follow that, and I'll keep sharing how I'm thinking about it.

Will do. Will do. All right. Thanks so much for the time.

Cool. Cool. Thanks for having me.

Cool. Thanks for having me.