AI 编码工厂

Hey everyone, welcome to the latest space podcast. This is Allesio, partner and CTO at Desible and I'm joined by my co-host Swixs, founder of Small AI. Hey, and today we're very blessed to have both founders of Factory AI.

Thank you for having us.

Thank you.

Matan and Eno, my favorite story about the founding of Factory is that you met at the LangChain hackathon. I’m very annoyed because I was at that hackathon, and I didn’t start a company or meet my co-founder. Maybe one of you could quickly retell that little anecdote, because I think it’s always very fun.

Both Eno and I went to Princeton for undergrad. What’s really funny is that, retrospectively, we had about 150 mutual friends but somehow never had a one-on-one conversation. If you pulled us aside and asked us about the other, we probably vaguely knew what they did and what they were up to, but we’d never had a one-on-one conversation.

At this LangChain hackathon, we were walking around and caught a glimpse of each other out of the corner of our eyes. We went up and had a conversation, and it very quickly got into code generation. This was back in 2023, when code generation was all about BabyAGI and AutoGPT. That was the big focus point there.

Both of us were speaking about it, and both were very obsessed with it. I like to say it was intellectual love at first sight because basically every day since then, we’ve been obsessively talking to each other about AI for software development.

If I recall, that LangChain hackathon wasn’t about code generation. How did you find the idea maze that led to Factory?

I was at Hugging Face, working primarily on advising CTOs and AI leaders at Hugging Face’s customers, guiding them toward how to think about research strategy and what models might pop up. In particular, we had a lot of people asking about code and code models in the context of, “We all want to build a fine-tuned version on our codebase.”

In parallel, I had started to explore building—at the time, the concept of an agent wasn’t clearly fleshed out—basically a while loop that wrote Python code and executed it for a different domain, finance. On my mind was how unhelpful it felt for finance and how incredibly interesting it felt for software. Then, when I met Matan, I believe he was exploring it as well.

At the time, I was still doing a PhD at Berkeley, technically in theoretical physics, although for a year at that point I had really switched over into AI research. I think the thing that pulled me away from string theory, which I’d been doing for about 10 years, into AI was really that string theory, physics, and mathematics make you appreciate fundamentalness, or things that are very general.

The fact that capability in code is core to performance on any LLM—and, loosely, that the better any LLM is at code, the better it is at any downstream task, even writing poetry—really nerd-sniped me. The fundamental beauty of how code is core to the way that machines develop intelligence got me to leave what I’d been pursuing for 10 years.

That, mixed with the fact that code is one of the very few things, especially at the time, that you could actually validate, made it extremely exciting to pursue. You have that agentic loop where the LLM is generating the output and you’re actually verifying the quality of that output against ground truth.

How did you guys decide that it was time to do it? If you go back, the technology was cool at a hackathon, but as you started to build a company, there were a lot of limitations. How did you flesh out the start of the company? The models weren’t great at the time, so you built a harness around them, and now the models are getting a lot better, so it’s time to go GA, as you’re doing now.

There’s a more quantitative answer and a more qualitative answer. The qualitative answer builds off what I said before: it was intellectual love at first sight. I think it was also one of those things where, if you know, you know. We met, we got along so well, and basically for the next 72 hours we didn’t sleep. We were just building together on initial versions of what would become Factory.

When something like that happens, I think it’s good to lean in and not really question or overanalyze it. At the same time, if you do actually go and analyze it, there are exactly the considerations that you’re talking about. The models at the time—GPT-3.5 was what was available—certainly weren’t enough to have a fully autonomous engineering agent. But very clearly, if you build that harness or scaffolding around them and bring in the relevant integrations and sources of information that a human engineer would have, it’s very clear how that trajectory would get to the point where more and more of the tasks a developer would do could be automated.

At the time, as you mentioned, there was BabyAGI and a couple of these other concepts that had come out, which involved putting a while loop around the LLM and feeding back some context. On the other hand, there were papers coming out on chain-of-thought and self-reflection. Of course, the scaling law papers had been somewhat established at that point.

There was a clear direction where models were going to get better at reasoning and better at having larger context windows. They were going to get cheaper, or at least the price-to-performance frontier of model capabilities was going to expand, so that good models would get cheap. The best models might stay the same price, but they would start to get really smart.

This wasn’t super obvious, but if you spent a lot of time reading through these papers and working through them, there was definitely a rumbling among most of the people in the community that this was going to continue to extend. You blend all of that with meeting somebody who had this kind of energy and clearly wanted to build, and I think it became really obvious that the opportunity was available.

I also think that we made a lot of solid progress on the initial demo—enough to convince ourselves that this was actually going to be possible. To be clear, though, it was 8 days from us first meeting to me dropping out of my PhD and Eno quitting his job.

Yeah, pretty crazy. 8 days, for sure.

My first company was a hackathon project, and I dropped out of school to actually found the company with one of my best friends, so the story resonates.

I think I’m doing hackathons wrong. Maybe I’ve had one girlfriend out of them. That was about it.

Is it still ongoing?

No.

Oh, yeah. I mean, it’s a part of the funnel.

Code generation wasn’t the topic of the hackathon back then, but today, every other event I go to has code generation as part of it. There are a lot of code-generation products. Do you guys want to talk about what Factory is and give a quick comparison of the different products that people might have heard about, so we can dive deeper?

Our focus is on building autonomous systems for the full end-to-end software development life cycle, and in particular, for enterprises. Code generation is obviously very exciting. A lot of the best engineers coming out of the popular schools want to work on reinforcement learning, do cool things with GPUs, or train models. Code is one of the most obvious areas because it’s very easy to resonate with if you’re an engineer.

That’s led a lot of the players in the space to focus on coding in particular, and on solo developers or building a quick 0-to-1 project. The use case appeals to that profile. We’re focused on the relatively underserved enterprise perspective.

There are hundreds of thousands of developers who work on codebases that are 30-plus years old. They’re ugly, hairy, and messy. If you made a demo video doing some COBOL migration, that’s not very sexy. You wouldn’t go viral or get a lot of views because it’s just not that visually appealing.

But the value you can provide, and how much you can improve those developers’ lives, is very dramatic. Seeing that underserved group is why we’ve focused our sights there.

I would add that there are a lot of really interesting constraints that people take for granted in the broader market as being fundamental to the coding-assistant and SDLC-assistant market. In particular, a lot of the players look at a platform that has been the dominant tool for software developers: the IDE.

This is a tool that was designed 20-plus years ago, or has been iterated on for 20-plus years, primarily for a human being to write every line of code. When you take a tool like that and start to introduce AI, you introduce additional constraints that exist simply because of where you’re interacting with these systems and where those systems live.

For example, latency matters a lot when you’re inside an IDE.

The cost when you are local-first and your typical consumer is on a free plan or a $20-a-month paid plan limits the amount of high-quality inference you can do, and the scale or volume of inference you can do per outcome. When you are freed of a lot of these constraints, you can start to more fundamentally reimagine what a platform needs to look like in order to shift from a very collaborative workflow—which is what I think we see with most tools today—to a more delegative workflow, where you are actually managing and delegating your tasks to AI systems. I think that the product experience of delegation is really, really immature right now, and most enterprises see that as the holy grail—not going 15% or 20% faster.

And you call them droids. Is there a story behind the naming of either Factory or droids?

We were initially incorporated as the San Francisco Droid Company.

Really?

We were just talking about that on the live podcast.

Sorry—oh, you had to bleep that out.

No, no, no, no. But our legal team advised us that Lucasfilm is particularly litigious and that we should change our name.

At the time we were thinking of renaming, I was still in my Ph.D. We incorporated 2 days after we met, which was also ridiculous. We were thinking of renaming, and I was still in an ML class at Berkeley, reading a paper on actor-critic.

In there, there was some equation that was like some function of the actor—we're just calling that y. It was f of a equals y; a is actor. Put the actor in there, and then it's, you know, Factory. That's how it originally came about. It actually works quite well—automation, that sort of thing.

Also, the Factory method, I think, was—we had that written up at some point, and I think that inspired this line of thinking. Droids kind of remained because we felt that there was a lot of hype at the time around the concept of agents. But it referred to such a specific thing that everybody saw: this endless while loop, unreliable system that just kind of went on and on and took a bunch of actions without guidance.

Our thought process was, well, that's not really what our system looks like at all. So even though underneath it is an agentic system, do we need to say that we're an agent company? It doesn't really make sense.

I kind of like that. Last year, even though you spoke at the AI Engineer World's Fair, I put you on the agents track. I almost didn't have an agents track because I thought, this is so ill-defined, and I think that instinct is good. But now the agents wave has come back the other way, and now everyone's an agent. I think defining your own term for it and just getting out of that debate is a positive.

Would you say it's closer to a workflow, which is, I guess, the more commonly industry-accepted term now?

That's a great question. I think the original version of the droids was a lot closer to what we called workflows. They were asynchronous and event-based. They would trigger, and each one had a specific kind of deterministic, semi-deterministic behavior. That was the original version.

I think what we've grown to as the models have evolved, and as our ability to build out guardrails and the system has improved, is that when you interact with droids inside the Factory platform, they are relatively unbounded in the path that they take. They are, in general, guided mainly by the concepts of planning, decision-making, and environmental grounding.

They can stay loosely goal-oriented over a long duration without needing very hard-coded guardrails, but they still tend to hit their goal according to their original plan. I think now agent is probably the proper way to describe them.

Sure.

But I think droids have a nice ring to it. Our customers really love droids as a name. I cannot tell you how many times, if we're with an enterprise customer, we'll do a POC and a day later, if they're excited and things go well, they'll share a screenshot and be like, “These are the droids we're looking for.” Honestly, every time it's so fun.

I know, and everyone thinks they're the first to make that joke, but it really is better than agent or intern or an autonomous human name.

Yeah, it's actually a pretty good insight. And actually, I think we take a bit of an objection to the idea that these things are a replacement for a human being.

As we work through harder and harder problems with agents, it's become more clear that the outer loop of software development—what a software developer does: planning, talking with other human beings, interacting around what needs to get done—is something that's going to continue to be very human-driven. Meanwhile, the inner loop, the actual execution of writing lines of code or writing down the docs, is probably going to get fully delegated to agents very soon.

You just need to put “Roger, Roger” once they ask a question in the task.

We have that emoji in our Slack and use it very frequently.

Roger, Roger. Do we want to do a quick demo?

Yeah, happy to jump in.

When you land on the platform, you're presented with the opening dashboard. We try to make it really obvious that there are different droids available for key use cases that people tend to have. Of course, you can always go and speak with a default droid that can do a lot of things pretty well, but what we've learned is that there are 3 major use cases that people keep coming back to the platform for.

The first is knowledge and technical writing. That's going in more of a deep-research-style system that will go and do some research. It will use tools available to it—search, et cetera—and come back with either a high-quality document or answers, and then you can go back and forth. The code droid is really the daily driver for a lot of folks, and this system allows you to actually delegate a task. I'll jump into that in a second; we can actually go through a full walkthrough.

And then the reliability droid. It was pretty surprising to us, the degree to which people love doing incident response, the kind of SRE-style work, inside the platform. In retrospect, it's nice because no one loves to be on call at 3:00 a.m., waking up and being like, “What's happening?” Being able to just pass an incident description or say, “Hey, something's going wrong,” and have a system really compile all the evidence, write up an RCA, and provide that for you is super high-leverage. That's actually one of the more popular droids that people use.

I can start by just going into the code droid. When you start a session with a droid, you're presented with this interface. It's a little different from typical interfaces, where we see a lot of tools really want to focus you in on the code. Our perspective is that code is important to review when it's completed, but as the agent is working, what matters most is seeing what the agent is doing and having a bit of an X-ray into its brain.

We have an activity log on the left and a context panel on the right. You'll notice as we go through this task that the context panel starts to get updated. I'm going to start by doing something that's a pretty common entry point. I'm going to paste a ticket into our platform.

We have integrations with a bunch of different stuff: Linear, Jira, Slack, GitHub, Sentry, PagerDuty—you name it. We have a bunch of these integrations that our enterprise clients have wanted over time, such that you can easily pull this info in.

If I were to say something like, “Hey, can you help me with this ticket?”—and then I'm going to use my @ command, which lets me easily reference code or codebases—“Hey, can you help me with this ticket in the Factory monorepo?” I like to be nice to them: “I'd love your help.”

You'll note that right off the bat, the droid starts working. It's doing a semantic search on part of my query in that codebase. The system has access to a bunch of different tools here: memory, project-management tools, GitHub, web search. Right now, the code droid only has search enabled by default, but you'll note that as the system starts working, it may actually want those additional tools added so that it can do its job.

Maybe an important note there is, as we deploy these droids in the enterprise, something we're pretty ideological about is that everyone expects these agentic systems to perform at the level of a human, because that's what they're always going to compare them to. But in a lot of cases, they'll have these agents just in the IDE, and that's the equivalent of onboarding a human engineer, throwing them into your codebase, and being like, “All right, go.”

But the reality is, when you onboard a human engineer, what do you actually onboard them to? Slack, Notion, Linear, Datadog, Sentry, PagerDuty—they have all of these other information sources that they need to actually be a productive engineer. And yes, in theory, if you're really, really good and you don't need contextual information, you could just work based on code, but that would be a lot harder and would probably take a lot more time.

100%.

Having those connections ends up being super important as it works through harder problems. In particular, you can see that the first thing it did after that search was reference some of the information that it found, saying, “Hey, this is what I found so far.” It gives an initial crack at a plan, presents that really clearly to you, and then goes to ask clarifying questions.

So, a lot of users, we believe, should not need to prompt-engineer agents, right? If your time is being spent hyper-optimizing every line and question that you pass to one of these systems, you're going to have a bad time. A lot of what we do is to be able to format things: if I say, “Help me with this ticket,” right, there are clearly going to be some ambiguities. The system knows that when you give a very detailed answer or request, it should follow your instructions. When you give more ambiguous requests, it should ask for clarification. This is actually a really tricky thing to get right in the model, but we spend a lot of time thinking about it.

And so I'm just going to answer some of these questions. Are there any UI mockups? No. Try to imitate the other examples. 2. Only preview when the button is clicked. 3. No, it's actually not implemented. 4. Just answering back which specific fields must be displayed. Your choice. And 5, your choice.

So now I'm basically saying to it, “You decide” for some, giving my preferences on others. This is really the balance of delegation, including delegation to non-AI systems. As a good manager, you give autonomy to people who work with you when needed, but also, if you're like, “Hey, I'm a little worried about this,” or, “I'm going to be really strict about what I expect here,” you want to extract that behavior as well. Because a lot of times, like Eno mentioned, if you give a really poor prompt and just say, “Hey, go do it,” it's going to go do it, but it'll probably make assumptions. At the end, you might not be happy, but that's just because there were some constraints in your head that you didn't actually explicitly mention when you were communicating.

So, yep, 100%. Do you guys have a template that you've seen work? When I onboarded to Devin, for example, they have the Improve Prompt button, and then it prefills their template, which is, “Give the agent instructions on how to debug, give the agent instructions on how to do this,” and asks you to fill out these things. Do you guys have a similar thing where you think, for each project, these are the questions that matter, or is that more dynamic?

No, that's a great question, and it's something that we talk about a lot internally. It's surprising how many people are building products that have reactive information requests, like, “Please fill out this form to explain how to do this thing,” or, “You need to set up this dev environment yourself manually in order for this to work.” We think about trying to be proactive with a lot of this stuff.

You'll notice in the right-hand corner there's this project overview, right? The Droid started to code after doing some search, so that's going to pop up while we do this. When I click into this project overview, what you're going to see is basically a—I'm hiding it because I'm realizing this is actually semi-sensitive.

Hide that, no worries.

It's totally fine for folks to see that it's a monorepo. If I scrolled down, that's when we'd get in a little bit of trouble. But inside that project overview, we're actually synthesizing a bunch of what we call synthetic insights on top of the codebase. That is looking at things like how to set up your dev environment, what the structure of the codebase is, and how important modules connect to each other. As we index codebases, we're actually generating these insights at a much more granular level across the entire codebase. We think that, in general, systems should be proactive in finding that information.

However, with features like memory—and we have a droid.yaml—you can set some of your guidelines. But we also feel that it's like that XKCD about standards, right? Everyone's got their own rules file, and so we ingest those automatically from all of the popular providers as well.

Wow. Okay. Is something like Cursor Rules complementary, because people might take this and then work on it in Cursor separately?

What we found is that there are sometimes extraneous pieces of advice in those, because people need to give a lot more guidance to those types of tools than they do to ours. So our system parses through and only picks the things that we don't already know.

Another thing that comes to mind related to your question—and this is something we've been thinking about a lot as well—is that, as we have more and more enterprise customers, a lot of the developers in the enterprise are not going to be as up to date on every new model and how it changes its behavior. Something that's interesting that we're thinking about is that these developers are getting familiar with Factory and how to get the most out of it. Then, let's say, when we upgraded from Sonnet 3.5 to 3.7, we suddenly had a lot of developers saying, “Hey, wait, it now does this less, or it does this more. What's happening?” Or when they go to Gemini, let's say, and they want longer context.

Something that I think is interesting is how much of the behavior difference between the models should we act as a shock absorber for, so that they can basically, as a user, use it exactly how they've been using it before and get the same sort of output? But then also, how much of that do we actually want to translate to the user? Because presumably, over the next 3 years, the way you interact with models will change, and it's not just going to be about behavior, but rather, I guess, it's alpha versus beta in the model. Some models have different personalities, and it's just the way you prompt them to get the same output. Then there are others where—I mean, for example, the reasoning models—they just work in a fundamentally different way, and so you, as the user, should know how to interact differently. So that's something that's kind of fun to wrestle with.

How do you evaluate the new models?

We listen a lot to how the model providers actually think about building out their eval suites, and in particular, we try to look at things like desired behavior versus actual behavior in a way that's sustainable for a small team. We don't have $100 million to pay data providers. A lot of the evaluation ends up being a combination of point- and task-based evals. Aider has an awesome benchmark that we built on top of internally for code editing and file generation. For the top-level agent loop, we also have our own behavioral spec, where we set a bunch of high-level principles.

We break those down into tasks. Those tasks then have grades and rubrics, and we try to run them in order to determine whether the behavior suite that we like—for example, asking questions when it's ambiguous versus not asking questions—matches up. We also use that to optimize the prompts.

Just a quick question on these types of things. I think every company should have its own internal evals, right? That is not in question, and obviously, that is your IP, so we can't know too much about it. But what is the right amount to spend on something like this?

We talked about SWE-bench before recording. SWE-bench costs like $8,000 to run. I've heard varying numbers between $8,000 and $15,000 to run. Yeah, that's high, but you should be able to spend some amount to ensure that your system as a whole works and doesn't regress. What's a rule of thumb for what is the right amount to spend on this?

Yeah, I mean, I think it's important to separate out the 2 purposes of benchmarks. One is marketing. There are so many customers that we have who came purely because they saw the charts, saw the big bar versus the little bar, and were like, “Okay, we want to go with the big bar,” which is funny, but that's just the way things go. I think that's actually a good thing, because that motivates more resources to be put into benchmarking and evaluation.

On the other hand, there definitely is a risk of going too far in that direction, or even getting to the point where you're fine-tuning just to satisfy some benchmark.

And so, like we were saying before the taping, you guys don't bother competing on SWE-bench anymore because it's not that relevant for you.

Yeah, that, and also just in the enterprise, the use cases are pretty different from those represented in something like SWE-bench. So we do have pretty rigorous internal benchmarks as well. But I think there's also a certain extent to which the vibe-based or sentiment-based evaluation internally actually matters a lot, because who has a more intimate understanding of the behaviors of these models than the people who work on them every single day, working with them and building with them? We use Factory internally every single day, and so when we switch a model, we very quickly get a sense of how things are changing.

Definitely. And I think those task-based evals tend to be the ones where it's most critical that we hill-climb continuously, versus the top-level evals. They change so much with the new model providers that we try to make sure that they have some degree of consistent behavior, that the feel is smart.

But the top-level agent is actually not that responsible for what most people call quality. That ends up being: Is it fast, accurate, and producing high-quality code edits? Does it call tools with the right parameters? Is the tool design such that the model can easily fit into it? We have noticed a lot of really interesting behaviors with the new models that have much heavier RL and post-training related to their own internal agentic tools. For example, Sonnet 3.7 smells like Claude Code, right? Same with Codex. It very much impacted the way that those models want to write and edit code, such that they seem to have a personality that wants to be in a CLI-based tool.

What's interesting is: How do we combat the preferences that RL brings into the product? For example, search with the CLI is like Grep and Glob. But what if you gave it a search tool that was way better than Grep or Glob at finding precisely what you wanted, but the model just really loves to use Grep? They're going to fight each other.

Our evals have to figure out how to make sure that, as we build tools that are better than what maybe the model providers have in their slightly more toy examples, the models use those tools to their full extent. That's actually been a very interesting, novel challenge for us that only started happening in the last 3 to 6 months as these new models have come out.

Does that make you want to do more reinforcement fine-tuning on these models? Kind of take more of that matter into your own hands?

I definitely think that it's an interesting idea, but our take in general is that freezing the model at a specific quality level and freezing the model at a specific data set just feels like it's lower leverage than continuing to iterate on all these external systems. It also feels like this is a bit of a bug. We spoke with a bunch of the research labs, and I don't think that they actually want this type of behavior. What it is ultimately is a reduction in generalization.

Cool. Anything else to see on the demo side?

Oh, yeah. I mean, it's still coding.

Yeah, yeah. So, going ham.

Yeah. So, you can see here that we're running.

Oh, because you gave it a whole bunch of things.

Yeah. So, I actually gave it quite a large project to execute live in front of us.

Got to earn it. Let's keep going.

This is why this delegation-style flow we see is really different. In general, we expect the answer or output of this to just be correct, right? It's running code, iterating on code, and making edits to a bunch of different files. It's going to have to run pre-commit hooks and test all this stuff.

I think that this is a big difference in workflow, right? We've just had a podcast conversation; meanwhile, the agent is working on my behalf. This is probably going to be mergeable at the end of this. It's ideally going to create a pull request, and we can check in on it at the end.

But I think that this difference is, like, what would I be doing right now? I think today a lot of people just open up their phone, maybe, and start browsing, or they context-switch to a different task. But the real power is unlocked when you start to realize this is the main thing that I'm going to be doing: only delegating these types of tasks. And so you start jumping to, okay, while this is happening, let me go and kick off another task and another one and another one.

Being cloud-native and being able to parallelize these tasks—I’m only sharing one tab, but if I just open another one and start right now, we support that natively. I think this feels a little bit more like how people are going to work: You maybe start the day setting off a bunch of tasks in motion, and then you spend the rest of it on harder intellectual labor, like thinking about which of these is actually highest priority to execute on.

This actually goes into something that Eno was mentioning a little bit before, but also a question that I'm sure everyone, when they see this, is going to ask, which is: Why is this browser-based? Why is this not in the IDE? I'm used to coding in the IDE.

The higher-level answer here is that, as Eno was alluding to before, over the last 20 years, the IDE was built for this world where developers are writing every single line of code. Something I think everyone can agree on is that, over the next few years, what it means to be a software developer is going to change dramatically.

Now, some people disagree, and some people say there will be no more software engineers. Some people say everyone's going to be a software engineer, and everywhere in between. But the reality is very clear: In the next few years, the number of lines of code written by humans will go down. The percentage of code written by humans will go down.

Our take is that it is very unlikely that the optimal UI or the optimal interaction pattern for this new software development, where humans spend much less time writing code, will be found by iterating from the optimal pattern when you wrote 100% of your code, which was the IDE.

Internally, we talk a lot about the Henry Ford quote, which is: If you ask people what they want, they would say, “Faster horses.” For us, the analogy here is: Can you iterate your way from a horse to a car? There's this very grotesque Ship of Theseus you can imagine of trying to turn a horse into a car. It doesn't really look pretty.

Our take is that, even though the world was built for horses at a certain point in time—there were stables everywhere throughout a city, you were used to feeding this thing and taking it with you everywhere—it is kind of a higher barrier to entry to start introducing this new means of transportation in this analogy.

We are taking that more ambitious angle: Everything is going to change about software development. In order to find that optimal way of doing it, you do need to think from scratch and think from first principles about what that new way to develop looks like.

To give some early answers that we're pretty clear about, the time developers spend writing code is going to go way down. In turn, the time that they spend understanding and planning is going to go way up. And then also, the time that they spend testing so that they can verify that these agents that they delegated to did indeed do the task correctly—that's going to go way up.

The promise of test-driven development is going to finally be delivered with this world of AI agents that are working on software development. If you do want to delegate something like this while you're doing a podcast and come back later, ideally you don't even need to check their work and you just merge the PR.

But how do you do that with confidence? You need to be really sure that the tests that you put up and said, “Hey, Droid, you're not going to be done until you pass all of these tests”—if you wrote those tests, well, then you can say, “All right, great. Pass the tests. Let's merge it. I don't even need to go in and see how it did everything.”

I mean, sometimes you do have to break the tests because you're changing functionality.

Yeah, yeah. There's a whole bunch of hard problems, but I just wanted to cap off the sort of visual component of the thing.

There's one thing you haven't shown, which is that there's a built-in browser. I have a Next.js project here that I'm running—the conference website. It spun it up itself. When I tried it out in ChatGPT Codex, it didn't work out of the box, and they didn't have a browser built in. So, it's nice that you have that kind of stuff.

No, for sure. Being able to view HTML, SVG, et cetera, on demand is super nice. I think it's pretty much wrapped up. It actually finished these changes. I think it's roughly 12 files that it edited and created, and so, right after this, because of the GitHub tool, I would just say, “Go ahead and create a pull request.”

Amazing. Yeah, good stuff. You even show a little 43% of context size used. That's actually not that much, given that this is Factory's own codebase.

Yeah. And this is actually a large monorepo. I think the big thing that I'd love for people to try out is to look at how efficient it is. It's able to really execute on precisely what it needs to edit with relatively lower token usage than other agentic tools.

Obviously, if you're just getting autocomplete, that's going to be a little bit more expensive. But compared to other agents where you get 5 credits and it takes a while to execute on anything, I think they'll see a better experience with Factory.

When you started saying things like, “Oh, we can pull in from Notion, we can pull it from Slack,” that sounded like a lot of context. You're going to have to do pretty efficient RAG to do this, right? I guess it's not even RAG; it's just retrieval.

Yeah. Yeah. I mean, there is the temptation—and I remember maybe a year ago, there was really a lot of hype around large context, because it's the dream of being super lazy and just throwing in your whole codebase, throwing in everything at it, which is what Claude Code does.

Right, exactly.

But I think the one downside of that is, okay, great, if you do have a billion-token context-window model and you throw it all in there, it's still going to be more expensive. The reason why retrieval is so important for us is because, even if there is a model that's going to have these larger context windows—and certainly, over time, we're going to get larger context windows—you still want to be very cost-efficient.

This is something that our customers care a lot about. They see a lot of the value in the effort that we put in on retrieval, because they'll see, “Wait, this was a huge monorepo and I gave it all this information, but then I see for each actual call, you're really good at figuring out what do I actually need,” as opposed to just throwing the whole repo in and praying that it works.

You mentioned the credits. What's the pricing model of the product?

We're fully usage-based. For us, it's really important to respect the users and their ability to understand what this stuff means. I think all the stuff around credits and all—it just kind of obscures what's actually happening under the hood.

I actually think that we get better users the more they understand what tokens are and how they're used in each back-and-forth.

It’s a direct bill through to what we call standard tokens, and it’s benchmarked off of the standard models that we have. Right now, when you get access to the platform, your team would pay a small fixed price just for access. Every additional user is another very small fixed price, and then the vast majority of the spend would be on usage of the system.

I think this is nicely aligned: you get a sense of how efficient it is with token usage. This is a big reason why we’ve tried really hard to make it more token-efficient, and then you can track, of course, how you’re using it in the platform.

A lot of people like to see not only raw usage, and this gets into tracking success—something that a lot of people do by, maybe, the number of tabs that you accepted or chat sessions that ended with code. For us, we try to look a little bit further and say, “Look, you used this many tokens, but here are the deliverables that you got. Here are the pull requests created. Here’s merged code.”

We help enterprise users look at things like code churn. It turns out that the more AI-generated code you have, if the platform isn’t telling you the code churn, there’s a reason for that. Code churn means the amount of code deleted versus added.

It’s basically a metric that tracks variability in a given line of code. It’s very imperfect, because some people will say that the difference between code churn and refactored code is somewhat arbitrary. It depends on the time period: if I merged some line and then changed that line, if I change it within a shorter period, it’ll churn, whereas over a longer period it’ll count as refactoring.

Generally, in enterprise codebases, if you merge a line of code and then change that code within 3 weeks, it’s because something was wrong with that code. Generally, it’s not always true, but it’s a useful metric. It averages out, because sometimes it’s like, “Wait, what if you just had an improvement or some change that wasn’t about quality?”

Is code churn up bad? Yes. In very high-quality codebases, you’ll see 3% or 4% code churn when they’re at scale. This is with millions of lines of code. In poor or poorly maintained codebases, or early-stage companies that are just changing a lot at once, you’ll see numbers like 10% or 20%.

Now, if you’re at Atlassian and you have 10% code churn, that’s a huge, huge problem, because it means that you’re wasting so much time. If you’re an early-stage startup, code churn is less important. This is why we don’t really report that to every team—just to enterprises.

Are there any other measurements that are popular? This is nice, that I’m hearing about code churn, but what else do enterprise VPs or CTOs look at?

For the enterprise, I think the biggest thing is that there are so many tricks and different dances you can do to justify ROI. The number of commits and lines-of-code metrics are usually popular, and at the end of the day, we initially went really hard on all the metrics.

What we found is that oftentimes, if they liked it, they wouldn’t care, and if they didn’t like it, they wouldn’t care. In reality, at the end of the day, no one really cares about the metrics. What people really care about is developer sentiment when you’re playing that game.

At the end of the day, if you want to use a metric, talk to developers and ask if they feel more productive. If you’re a large enterprise and you want to justify ROI, the biggest thing that we’ve seen—and what’s allowed us to deploy very quickly in enterprises—is pulling in timelines on things.

There’s one very large public company that we work with, and we pulled in a large migration task from taking 4 months to taking 3½ days. That’s the best ROI there. You don’t need to measure this or that. We had something that was going to be delivered in the next quarter, and we got it done this week with no downtime.

That is music to a VP of engineering’s ears. That’s what we tend to focus on: pulling in deliverables or increasing the scope of what you can get done in a quarter.

In order to achieve a very large refactor like you just described, do we use that same process you just saw, or is there more setup?

I think the workflow for, let’s say, a migration is probably one of the most common. I can even give a very concrete example. Let’s say you are the administrative service of a large European nation, like Germany or Italy, and you have a hospital system that runs on a 20-year-old Java codebase.

A company wants to come in—a Big Four consulting firm or something like that—and says, “We would like to transform this entire codebase to Java 21. It’s going to take X amount of time, a couple of months, and by the end, you’ll be on a relational database. You’ll be into the future, on Java 21.”

When that typically happens, you have to almost break down what that means from a human perspective first, and then map it to how it works on our platform. You’ll have a team of anywhere from 4 to 10 people come in, and you have a project manager who is going to work with engineers to analyze the codebase, figure out all the dependencies, and map that out into docs.

First, there’s an analysis and overview of the codebase. Next is a migration strategy and plan. The third is timelines, and you’re going to scope all this out. What do you do next? You go to a project management tool like Jira. You take those documents, and a human being translates that out: “We’ve got 2 epics over the next 2 months. This epic will have these tickets. That epic will have these tickets.” Then you find out the dependencies and map those out to humans.

Each of these humans is now operating such that, one after the other, they’re knocking out their work, mainly in parallel. Occasionally, pieces have to connect: one person misses something, and now the whole project gets delayed about a week.

This interplay of understanding, planning, executing on the migration incrementally, and then ultimately completing it involves a handoff period. There are docs for the new artifacts that we’ve created, and there’s all this information.

You map that over to a system like ours, and one human being can say, “Please analyze this entire codebase and generate documentation.” That’s one pass, one session in our platform: analyze each of the modules. We already do a lot of this behind the scenes, which makes it a lot easier, and actually generate an overview of what the current state is.

You can now pull those docs in with real code and then say, “What’s the migration plan?” If there’s some specific system, you can pull in docs. Our system connects with Linear and Jira, so it can create tickets, create the epic, ticket this whole process out, and figure out which are dependencies and which can be executed in parallel.

Now you just open an agent in every browser tab, and you execute all of those tasks at the same time. As a human being, you just review the code changes: “This looks good. Merge. Did it pass CI? Okay, great. On to the next one. This looks good. Merge.”

A process that typically gets bottlenecked—not by skilled humans writing lines of code, but by bureaucracy, technical complexity, and understanding—gets condensed into basically how fast a human being can delegate the tasks appropriately.

It happens outside of one session, like what we just saw, which would be one of those tasks. The planning phase is really where we see enormous condensation of time.

We just talked about your pricing. You’re just usage-based. Are you tempted to have forward-deployed engineers, just like the current meme, to execute these large things?

I think this is something we definitely do a little bit of for our larger customers because, like we said at the beginning, this is the way we think software development will look, and it’s an entirely new behavior pattern. I think it would be a little naive to just be like, “Hey, we have this new way of doing things. Go figure it out.”

We definitely go in and help show them how to do this. In the migration example that I was mentioning before, we worked with them side by side. Just us and 2 of their engineers showed them how to do it. They saw the light, if you will, and then they ended up being the internal influencers within their organization, teaching everyone else how to do it.

But if you want to change behavior, you can’t just assume that the product is going to be so good that everyone’s going to immediately get it. Developers have very efficient ways of working that they’ve built out over the last 20 years. We want to make sure that we accommodate that, earn their trust, and slowly bring them into this new way of building.

To do that, we need to extend that olive branch and meet them where they are, showing them how they can do new things.

We did an episode with Together AI maybe a year ago or so, and we were talking about what inference speed we actually needed. They always argued that we needed to get to around 5,000 tokens per second, and we were chatting about whether or not that made sense because people can’t really read it.

As you think about Factory, how much do you think you’re bound by the speed of these models? Do you know—if the models were a lot faster, would you just complete things quicker? Would you maybe fan out more in parallel? What are the limits of the models today?

I want to let you answer this, but immediately, every time this thing comes up, I always just think about the memory that Chrome tabs take. It’s never enough, and you always want more, but it also lets you be lazier.

No, for sure. I think this is kind of a funny question. It has two directions. Practically, would this make a big difference for someone who knows, loves, and uses our platform on a daily basis? I think it would probably improve their quality of life, so definitely, faster tokens would be awesome.

I think where this actually has an impact is on those who haven't yet made the jump from collaboration to delegation. If you're used to very high-latency, high-feedback experiences, then that speed difference—seeing most of that delegation happen very quickly and being able to immediately jump in—feels very nice. For the larger enterprise deployments, where they start to familiarize themselves with how this works and with the migrations, I don't think this actually makes a big difference, because most of the bottleneck ends up being, as I mentioned, almost bureaucratic in nature.

But for the average developer, I think this improves the user experience to the point where it would feel very magical. I think we could get a lot faster. It probably wouldn't change what's possible, but it would really change the ease of adoption for people who maybe aren't as in the weeds on AI tools.

And if you combine the latency with a cost reduction as well, I do think cost is one of the reasons why we haven't scaled out so greatly. Originally, we had a lot of techniques that would generate a lot of stuff in parallel, and we still know how to do that. We're very excited to bring that back, but right now we don't do it because it's cost-prohibitive, and the quality delta isn't enough to justify the cost increase.

I have a closing question, if you don't mind. This is more or less asking you about a limiting factor. It's basically 4 questions in 1: What do you see as your limiting factor right now in terms of models? What capabilities would really help you? For hiring, what skills are really hard to hire? For customers, what do you really want to unlock that's weirdly not working? You have an ICP that's more enterprise and doing well, but what's the next one? And finally, for dev tooling, what do you wish existed that you had to build for yourself, or that you just feel could be a lot better?

I'll maybe do models and dev tools, and you can take hiring and customers.

I think, right off the bat, probably the biggest thing is models that have been post-trained on more general agentic trajectories over very long time horizons. That feels like something there's an effort for right now. What I mean is an hour, 2 hours, or 3 hours of seriously working on a hard problem, such that the model knows how to keep that long-term, goal-directed behavior the whole time. That's something that I assume we'll get soon.

OpenAI has put out that Operator benchmark. They had human testers actually try for 2 hours and give up. Did you see that one?

Yeah. I think that's exactly the type of work that we want to see taken further, because I would argue that's probably one of the bigger blockers.

Would you ever do that yourself? I don't see you guys as customizing your own models a lot, but you work with the frontier labs, right? Is there a point where you would just be like, “All right, screw it. We'll do it”?

We are currently building benchmarks with a lot of the post-training techniques very much in mind. I don't know exactly, at this point in time, how much we're going to commit to that, but for sure we'll be using those benchmarks for our own internal goals and, if we need to use them later on, for post-training. I think there's a lot of compatibility.

And then, maybe for dev tools, it is still surprising to me that observability remains very challenging.

Really? There are like 80 tools out there.

No, for sure. LangSmith is actually fantastic. We use LangSmith. We don't use LangChain, but we just use LangSmith, and LangSmith is awesome.

See, the hackathon has ROI for Harrison.

They've been fantastic, and that's been cool. But I think it's really tricky to deal with enterprise customers where you can't see their code or data at all, but you're trying to build a product where you can improve the experience. A lot of it is actually subjective. It's like, “I don't like the way this code looks.”

Something that's still very unclear to us is how you build almost semantic observability into your product.

I think Amplitude and Statsig, and a lot of the feature-flag companies, are actually closer to this than existing analytics.

Actually, it's more about observing whether they're still on the platform—basically anything other than up and down thumbs, right?

Yeah. And what was the user's intent when they entered into this session?

It's the type of thing where you almost need LLMs in the observability itself.

Are you saying they've actually done it, or that Amplitude could do it?

That's where I would like to see it. As far as I understand, it hasn't really been done.

Yeah, because mostly everything is span-based. Observability products are looking at spans, but not at the semantic direction.

Exactly. Not in the semantic direction.

So for you, that's kind of solved in a way. The actual traces—you can get that information anywhere.

Our team comes from Uber and all these amazing places where they know how to do that part. I think the more tricky thing is when human beings have messy intents that are expressed in natural language. How do you really classify and understand when users are having a good time versus when they're having a bad time?

Okay, that's hard. Great. Hiring and customers.

So maybe I'll start with customers. We've been at it for just over 2 years now. I think the first year and a half was really focused on the product and on what interaction pattern works for the enterprise. Over the last 90 days, the deployments that we've had with large enterprises and Fortune 500 companies have been exploding. It's been going really well. It's very exciting.

How are they mostly finding you?

This is a good point. This is part of why we're doing more podcasts, because so far we've really just relied on word of mouth—working well with one enterprise, and then they're at some CEO dinner and mention it to someone else. That's why we have the conference: We put all the CEOs and VPs in one room.

It has worked really well, but when every one of those conversations ends up leading to a happy customer, that means you need to increase top of funnel. Accordingly, we're really putting fuel on the fire for our go-to-market for Fortune 500 large enterprises, which is obviously a very exciting thing to do. The team has been pumped.

There was a particular day in January of this year where one of those large enterprises basically had the magic moment of, “If I was the only one at my company using this, I would still tell them to have me use this instead of hiring 3 engineers for myself.” That was one of the biggest moments for us, where it was like, people in the enterprise are really getting dramatic value out of Factory.

That kicked off this period where the last 90 days have just been a whirlwind. Getting to more of these Fortune 500 companies is top of mind for us right now.

To that end, as you serve Fortune 500 customers, it becomes important to have a larger go-to-market team, both on the sales side and the customer-success side, and then also on the engineering side. So we are very much hiring.

I think everyone's hiring. What are you finding that's hard to hire? What particular roles? What's the rate limiter here?

I think one big rate limiter for us, as we go to these Fortune 500 companies, is having both the ability to talk to the CIO and VP of engineering and have that sales presence, but then also the ability to sit side by side with some of their developers and jump into the platform and their use cases.

So you need like 100 Enos, basically.

Honestly, literally, our profile—or actually, our profile when we're looking for this role—is, “Is this a junior Eno or not?” That's basically the template there.

I definitely think that if you're highly technical but you want to be a founder, and you want to move into a role where you're interfacing with CIOs and CTOs, we have maybe 3 of these roles that are probably going to be the most important roles in our go-to-market team. I think that's a huge opportunity for anyone interested in what we've talked about.

We joke that this person would basically be my best friend, because on any trip we go on to fly to a customer, they'd be there with me, talking to whoever the buyer is, as well as going in and working with the engineers. So I'm also, I guess, hiring a best friend.

I thought that's what AI was going to be for.

Yeah, I guess not.

Just to wrap, I think we're all fans of your guys' design and brand. Speaking of best friends, who does your design?

A huge privilege of Factory has been working with my older brother, Cal, who joined us. He moved from New York after being there for 5 years.

He moved out to San Francisco. Even before he moved, he was the one who designed our logo way back when, and he’s been a part of Factory from the very beginning. It’s been an absolute pleasure working with him, from the brand design and the marketing design to, of course, the product and the platform itself. I cannot recommend working with a sibling enough.

Sure. Not all of us are lucky to have that. What do you learn from working with a designer like that? I think a lot of technical people listening to us want to build a startup. They don’t have the polish that you have. They don’t have the hype.

I think a big part of this is that one of our core operating principles is embracing perspectives. Cal is not an engineer, and what’s great is that the majority of our team are engineers. Having that ability to come in with the design perspective and then also the engineering perspective, and bash those two things together until we get something perfect out of it, has been really, really important.

I think a lot of times it’s easy to fall victim to, “Oh, I’m the profile of who I’m building for, so I know what’s best.” That obviously works a lot of the time, but sometimes there are some core design tenets that you just might not think of if you’re building for yourself. So I think that’s been pretty important there.

We operate in a very AI-native company. Being able to have someone set principles that are then consumable by our own agents—design systems and consistency—I think it’s pretty surprising the degree to which even droids can actually imitate a brand voice and style that Cal created for us.

A lot of that comes not just from the droids doing that, but from our entire team of product engineers, who are all incredibly thoughtful about what they’re putting in front of users. I think they’re able to bring a lot of that into it in a way that feels safe and on-brand, and also have fun.

Factory’s semi-tongue-in-cheek slogan is “the machine that builds the machine.” It’s fun. Does it transmit exactly what it is that we do in the clearest way? No. The factory doesn’t build Factory.

Yeah, we don’t.

But to a certain extent, it’s software that writes software, right? The machine that builds the machine. It’s fun.

When you say “fun,” it’s more—actually, I see you guys hosting a lot of events at your office, and to me, that’s like, “Oh, these guys are actually social.”

Yeah, I think it’s important for us because not only is this incredibly transformational, but these are also people that we spend all of our time with, and we want to make sure that while we’re doing it—

Sweet. It’s right next to the Caltrain. You can advertise out of your window.

Yeah, no one peek in, though. A lot of secrets in there. It’s pretty sweet.

Cool. I’m very excited for your talk. We touched on a few things I’m interested in, right? We’re seeing tiny teams as a topic that I’m exploring—one person can do a lot more, so the average team size is really shrinking. The interaction of AI, design, and engineering is another thing I’m exploring. I think we’re really trying to push the frontier.

And then, obviously, there’s always the SWE agent stuff, which is ongoing. So, yeah, there’s a lot of interesting work going on.

One interesting addendum there is that there are sometimes individuals who weren’t even really developers who will use Factory and have more usage than a 100-person enterprise.

Yeah, which is crazy to see. There are some really interesting dynamics that we’ve seen play out in just how people use these tools, whether it’s for design or for that small-team use case.

Yeah, there’s an AI-native attitude that’s going to set people apart if they’re just open to it, but maybe they’re not drinking too much Kool-Aid. I think there’s a medium there.

Thank you guys for coming on. This was fun.

Thank you guys for having us. This was awesome.