Extreme Harness Engineering：100万行代码、每日10亿 tokens、0%人工编写或审查——Ryan Lopopolo，OpenAI

Shawn Wang

I do think that there is an interesting space to explore here with Codex, the harness, as part of building AI products, right? There's a ton of momentum around getting the models to be good at coding. We've seen big leaps in the task complexity with each incremental model release. If you can figure out how to collapse a product that you're trying to build, a user journey that you're trying to solve, into code, it's pretty natural to use the Codex harness to solve that problem for you. It's done all the wiring and lets you just communicate in prompts and let them all cook.

You kind of have to step back, right? You need to take a systems-thinking mindset to things and constantly be asking: Where is the agent making mistakes? Where am I spending my time? How can I not spend that time going forward? And then build confidence in the automation that I'm putting in place, so I have solved this part of the SDLC.

Before we get into today's episode, I just have a small message for listeners. Thank you. We will not be able to bring you the AI engineering science and entertainment content that you so clearly want if you didn't choose to also click in and tune into our content. We've been approached by sponsors on an almost daily basis, but fortunately enough of you actually subscribe to us to keep all this sustainable without ads, and we want to keep it that way. But I just have one favor to ask all of you. The single most powerful completely free thing you can do is to click that subscribe button. It's the only thing I'll ever ask of you, and it means absolutely everything to me and my team that works so hard to bring the AI in space to you each and every week. If you do it, I promise you will never stop working to make this show even better. Now, let's get into it.

All right, we're in the studio with Ryan Lopopolo from OpenAI. Welcome.

Hi.

Shawn Wang

Thanks for visiting San Francisco and thanks for spending some time with us.

Yeah, thank you. I'm super excited to be here.

Shawn Wang

You wrote a blockbuster article on harness engineering. It's probably going to be the defining piece of this emerging discipline.

Thank you. It's been kind of fun to feel like we've defined the discourse in some sense.

Shawn Wang

Let's contextualize a little bit. This is the first podcast you've ever done?

Yes.

Shawn Wang

Thank you for staying with us. Where is this coming from? What team are you in? All that jazz.

Sure. I work on frontier product exploration and new product development in the space of OpenAI Frontier, which is our enterprise platform for deploying agents safely at scale, with good governance, in any business. The role of me and my team has been to figure out novel ways to deploy our models into packaged end products that we can sell as solutions to enterprises.

Shawn Wang

You also have a background—I'll just squeeze it in there—in Snowflake, Brick, Stripe, and Citadel. So you've been building for exactly the kind of customer you want to serve your entire life.

I actually didn't expect that background. When I looked at your Twitter, I'm seeing the opposite, right? Stuff like this. You've got the mindset of full-send AI coding, stuff about slop, like buckling in your laptop on your Waymos. And then I look at your profile, and I'm like, "Oh, you're just cracked at the other end, too." Perfect mix.

It's quite fun to be an AI maximalist. If you're going to live that persona, OpenAI is the place to do it. It certainly helps that we have no rate limits internally, and I can go, like you said, full-send at this rate.

Shawn Wang

So, OpenAI Frontier and your special team within OpenAI Frontier?

We had been given some space to cook, which has been super, super exciting. This is kind of why I started with an out-there constraint to not write any of the code myself. I figured if we're trying to make agents that can be deployed into end enterprises, they should be able to do all the things that I do. Having worked with these coding models and these coding harnesses over 6, 7, 8 months, I do feel like the models are there enough, and the harnesses are there enough, that they're isomorphic to me in capability, in the ability to do the job. Starting with this constraint that I can't write the code meant that the only way I could do my job was to get the agent to do my job.

Shawn Wang

And just a bit of background before that: This is basically the article. What you guys did was 5 months of working on an internal tool, 0 lines of code, and over 1 million lines of code in the total code base. You say it was 10× faster than you would have been if you had done it by hand.

That's right. That's right. I started with some of the very first versions of Codex CLI with the Codex Mini model, which was obviously much less capable than the ones we have today. That was also a very good constraint, right? It was quite a visceral feeling to ask the model to build you a product feature and have it just not be able to assemble the pieces together, which kind of defined one of the mindsets we had for going into this: Whenever the model just cannot, you always pop open that task, double-click into it, and build smaller building blocks that then you can reassemble into the broader objective.

It was quite painful to do this, honestly. The first month and a half was 10 times slower than I would have been. But because we paid that cost, we ended up getting to something much more productive than any one engineer could be, because we built the tools—the assembly station—for the agent to do the whole thing.

Onward to GPT-5, 5.1, 5.2, 5.3, and 5.4. Going through all these model generations and seeing their quirks and different working styles also meant we had to adapt the codebase to change things up when the model was released. One interesting thing here is that with GPT-5.2, the Codex harness at the time did not have background shells in it, which meant we were able to rely on blocking scripts to perform long-horizon work. But with GPT-5.3 and background shells, it became less patient and less willing to block. So we had to retool the entire build system to complete in under a minute.

This is not a thing I would expect to be able to do. But because the only goal was to make the agent productive over the course of a week, we went from a bespoke Makefile build to Bazel to Turborepo to Nx, and just kind of left it there because builds were fast at that point.

Shawn Wang

Interesting. Talk about going from Turborepo to Nx. That's interesting because that's the other direction that other people have been doing. Ultimately, I don't have a lot of experience with actual front-end repo architecture. I'm going to ask you about the Nx guys. I know the Nx team, and I know Turborepo from Jared Palmer, and I'm like, "Yeah, that's an interesting competitor comparison."

The hill we were climbing was: make it fast.

Shawn Wang

Is there a bunch of micro frontends involved? Is it a web app? How complex is it?

React with Electron. Yeah, basically a single-app sort of thing. And it must be under a minute.

Shawn Wang

That's an interesting limitation. I'm actually not super familiar with the background shells stuff. It probably was talked about in the 5.3 release.

It basically means that Codex is able to spawn commands in the background and then continue to work while it waits for them to finish. So it can spawn an expensive build and then continue reviewing the code, for example. This helps it be more time-efficient for the user invoking the harness.

I guess, just to really nail this, what does a minute matter? Why not 5, you know? We want the inner loop to be as fast as possible. That 1 minute was just a nice round number, and we were able to hit it.

Shawn Wang

And if it doesn't complete, it kills it or something?

No, we just take that as a signal that we need to stop what we're doing, double-click, and decompose the build graph a bit to get CI back under a minute, so that we can enable the agents to continue to operate.

Shawn Wang

It's almost like you're—it's like a ratchet. It's like you're forcing build-time discipline because, if you don't, it'll just grow and grow and grow.

That's right. You mentioned that the current software I work on is at 12 minutes. It sucks. This has been my experience with platform teams in the past, right? You have sort of an envelope of acceptable build times, and you let it go up to breach, and then you spend 2–3 weeks to bring it back down to the lower end of the envelope, and it stops.

But because tokens are so cheap and so insanely parallel with the model, we can just constantly be gardening this thing to make sure that we maintain these invariants. That means there's way less dispersion in the code and the SDLC, which means we can kind of simplify in a way and rely on a lot more invariants as we write the software.

Shawn Wang

Ryan, you kind of mentioned in your article that humans became the bottleneck, right? You kicked off as a team of 3 people. You're putting out 1 million lines of code and something like 1,500 PRs.

Basically, what's the mindset there? As much as code is disposable, you're doing a lot of review. A lot of the article talks about how you want to rephrase everything: everything is prompting, and everything the agent can't see is kind of garbage, right? You shouldn't have it in there.

What's the high level of how you went about building it, and then how do you address the fact that humans are just kind of doing PR review? How much human-in-the-loop is there for this?

We've moved beyond even the humans reviewing the code. Most of the human review is post-merge at this point.

Shawn Wang

Post-merge. It's not even reviewed. It's just like, "Oh, let's make ourselves happy by reviewing it."

Fundamentally, the model is trivially parallelizable. As many GPUs and tokens as I am willing to spend, I can have the capacity to work on the codebase. The only fundamentally scarce thing is the synchronous human attention of my team. There are only so many hours in the day; we have to eat lunch. I would like to sleep, although it’s quite difficult to stop poking the machine because it makes me want to feed it.

You kind of have to step back, right? You need to take a systems-thinking mindset and constantly ask: Where is the agent making mistakes? Where am I spending my time? How can I avoid spending that time going forward? Then you build confidence in the automation that you’re putting in place, so you have solved this part of the software development life cycle.

Usually, what that has looked like is that we started needing to pay very close attention to the code because the agent did not have the right building blocks to produce modular software that decomposed appropriately, was reliable and observable, and actually had a working front end and these things. So, in order to not spend all of our time sitting in front of a terminal, at most doing one or two things at a time, we invested in giving the model that observability, which is that graph in the post here.

Shawn Wang

Just walk through the traces. Which existed first?

We started with just the app, and the whole rest of it, from Vector through to all these log and metrics APIs, was—I don’t know—half an afternoon of my time. We have intentionally chosen very high-level, fast developer tools. There’s a ton of great stuff out there now. We use mise a bunch, which makes it trivial to pull down all these Go-written VictoriaMetrics stack binaries in our local development. A tiny bit of Python glue spins all these up, and off you go.

One neat thing here is that we have tried to invert things as much as possible. Instead of setting up an environment to spawn the coding agent into, we spawn the coding agent—that’s the entry point, just Codex—and then give Codex, via skills and scripts, the ability to boot this stack if it chooses to. We then tell it how to set some environment variables so the app in local development points at the stack that it has chosen to spin up.

I think this is the fundamental difference between reasoning models and the GPT-4.1s and GPT-4o’s of the past. Those models could not think, so you kind of had to put them in boxes with a predefined set of state transitions. Whereas here, we have the model—the harness—as the whole box, and give it a bunch of options for how to proceed, with enough context for it to make intelligent choices.

Shawn Wang

A lot of that is around scaffolding, right?

Yes. With previous agents, you would define a scaffold, and it would operate in that loop: try again. That’s kind of pivoted since we’ve had reasoning models. They seem to perform better when you don’t have a scaffold, right? You let them go into nuances here, too, like your spec.md and having a very short AGENTS.md.

Shawn Wang

AGENTS.md?

Yes, yes. You even lay out what it is here, but I like the table of contents.

Shawn Wang

Yeah, stuff like this really helps guide people because everyone’s trying to do this. This structure also makes it super cheap to put new content into the repository to steer both the humans and the agents.

I mean, you kind of reinvented skills, right? Building agents and skills from first principles.

Skills did not exist when we started doing this, right? You have a short, one-page overall table of contents, and then you have little skills: core-beliefs.md, the tech debt tracker, and so on.

The tech debt tracker and the quality score are pretty interesting because this is basically a tiny little scaffold—a Markdown table—which is a hook for Codex to review all the business logic that we have defined in the app, assess how it matches all these documented guardrails, and propose follow-up work for itself.

Before Beads and all these ticketing systems, we were just tracking follow-up work as notes in a Markdown file, which we could spawn an agent on a cron to burn down. There’s this really neat thing: the models fundamentally crave text. A lot of what we have done here is figure out ways to inject text into the system.

When we get a page because we’re missing a timeout, for example, I can just ask Codex in Slack on that page and say, “I’m going to fix this by adding a timeout. Please update our reliability documentation to require that all network calls have timeouts.” So I have not only made a point-in-time fix, but also durably encoded this process knowledge around what good looks like.

We give that to the root coding agent as it goes and does the thing. But you can also use that to distill tests out of it, or create a code review agent that is pointed at the same things, to narrow the acceptable universe of the code that’s produced.

Shawn Wang

I think one of the concerns I have with that kind of stuff is that you think you’re making the right call by making it persist across everything for all time, but then you didn’t think about the exceptions that you need to make, right? Then you have to roll it back.

Part of it is that it sometimes can follow your instructions too well. It’s somewhat of a skill, right? It determines when it uses the tools. It’s not like it’ll run at every call; it’ll determine when it wants to check the quality score.

Shawn Wang

Yeah.

In the prompts we give these agents, we allow them to push back. When we first started adding code review agents to the PR, Codex CLI locally wrote the change, pushed up the PR, and on those PR synchronizations, a review agent fired. It posted a comment, and we instructed Codex that it had to at least acknowledge and respond to that feedback.

Initially, the Codex driving the code author was willing to be bullied by the PR reviewer, which meant you could end up in a situation where things were not converging. So we had to add more optionality to the prompts on both of these things.

The reviewer agents were instructed to bias toward merging the thing and not surface anything greater than a P2 in priority. We didn’t really define P2, but we gave it a framework within which to score its output.

Shawn Wang

Define P2.

We gave it a framework within which to score its output. Anything greater than a P0 is worse, right?

Shawn Wang

Yes. Something equal to or less than P2 is greater. P0 is, like, “You will nuke the code if you merge this thing.”

On the code-authoring agent side, we also gave it the flexibility to either defer or push back against review feedback. This happens all the time, right? I happen to notice something and leave a code review, which could blow up the scope by a factor of 2. I usually don’t mean for that to be addressed exactly in the moment. It’s more of an FYI: file it to the backlog, pick it up at the next fix-it week sort of thing. Without the context that this is permissible, the coding agents are going to bias toward what they do, which is following instructions.

Shawn Wang

I did want to check in on a couple of things. The code review agent can merge autonomously. I think that’s something that a lot of people are uncomfortable with, right? You have a list here of how much agents do: product code and tests, CI configuration, release tooling, internal developer tools, documentation, eval harnesses, review comments, scripts that manage the repository itself, production dashboard definition files—everything.

They’re all churning at the same time. Is there, like, a cord that any human on our team can pull to stop everything?

Because we are building a native application here, we’re not doing continuous deployment, right? There’s still a human in the loop for cutting the release branch.

We require a blessed, human-approved smoke test of the app before we promote it to distribution, these sorts of things.

Shawn Wang

So you’re building an app. You’re not building infrastructure where you have, like, nines of reliability and that kind of stuff.

That’s correct.

Shawn Wang

Okay.

And full recognition here that all of this activity took place in a completely greenfield repository. This applies generally—this is a production thing you’re going to ship to a customer, of course. This is real.

One of the things there is that you mentioned you started this as a repo from scratch. The onboarding for the first month or so was pretty—it was like working backwards, right? Then you had to work with the system, and now you’re at that point where you’re very autonomous.

I’m curious: How much human-in-the-loop is it? What are the bottlenecks that you wish you could still automate? Part of that is also where you see the model trajectory improving and offloading more human-in-the-loop work.

We just got GPT-5.4. It’s a really good—

Shawn Wang

Fantastic model, by the way.

Yeah, it’s a fresh one that’s emerged. It’s top-tier coding, so it’s Codex-level coding and reasoning—general reasoning, both in one model—and computer use.

Shawn Wang

Computer use. Now we have everything.

Now, with GPT-5.4, I can just have Codex write the blog post, whereas for this one I had to balance between ChatGPT and Codex.

Shawn Wang

Oh, I might be out of a job. Oh my God. You just gave me an idea for a completely AI newsletter that GPT-5.4 could do. I get it now.

This sort of thing is just one example of closing the loop. The dashboard thing you mentioned: We have Codex authoring the JSON for the Grafana dashboards and publishing them, and also responding to the pages. When it gets the page, it knows exactly which dashboards are defined and which alert was triggered by which exact log in the codebase, because all of this stuff is collated together.

It has to own everything.

Shawn Wang

Yes. Yes. And it means that if we have an outage that did not result in a page, it has the existing set of dashboards available to it, it has the existing set of metrics and logs, and it can figure out where the gaps are in the dashboard or in the underlying metrics and fix them in one go. In the same way, you would have a full-stack engineer able to drive a feature from the back end all the way to the front end.

It seems like a lot of the work you guys had to do was, as a small team, fully work in a way that the model wants the software to be written, right? It’s less human-legible for better code legibility, agent legibility. How do you think that affects broader teams?

At OpenAI, do you liaise on, “This is how software should be written”? I can imagine, say, you join a new team with this methodology, this mindset. There are ways that teams do code review, teams write code, and teams are structured, and a lot of it is for human legibility. Should we all swap? How does this play, one, more broadly into OpenAI, and then more broadly into software engineering?

Is it that teams that pick this up—it’s pretty drastic, right? You have to make a pretty big switch—should they just full-send?

The mindset is very much that I’m removed from the process, right? I can’t really have deep code-level opinions about things. It’s as if I’m group tech-leading a 500-person organization. It’s not appropriate for me to be in the weeds on every PR.

This is why that post-merge code review thing is a good analog here, right? I have some representative sample of the code as it is written. I have to use that to infer what the teams are struggling with, where they could use help, and where they’re already moving quickly, and I can pivot my focus elsewhere.

I don’t really have too many opinions about the code as it is written. I do, however, have a command-based class, which is used to house repeatable chunks of business logic that come with tracing, metrics, and observability for free, right? The thing to focus on is not how that business logic is structured, but that it uses this primitive, because I know that’s going to give leverage by default.

Shawn Wang

Yeah, back to that sort of systems thinking. You have part of that in your blog post, “Enforcing architecture and taste,” where you set boundaries for what’s used. There’s also a section on redefining engineering and stuff, but it’s interesting to hear.

As the models have gotten better, they have gotten better at proposing these abstractions to unblock themselves, which again lets me move higher and higher up the stack to look deeper into the future at what ultimately blocked the team from shipping.

Shawn Wang

You mentioned—this is primarily a 1-million-line-of-code codebase, so an Electron app, but it manages its own services as well. So it’s like a back end for front end type thing.

We do have a back end in there, but that’s hosted in the cloud. This sort of structure is actually within the separate main and renderer processes of Electron. That’s just how Electron works.

Shawn Wang

Yeah. So you’ve also treated that like an MVC-style decomposition?

Yeah, same level of rigor, which has been very fun. I have a fun pun. This is a tangent, but MVC is Model-View-Controller. Any sort of full-stack web developer knows that. My OpenAI-native version of this is Model-View-Claw. The Claw’s the harness.

Shawn Wang

That’s right.

I do think that there is an interesting space to explore here with Codex—the harness—as part of building AI products, right? There’s a ton of momentum around getting the models to be good at coding. We’ve seen big leaps in task complexity with each incremental model release.

If you can figure out how to collapse a product that you’re trying to build, or a user journey that you’re trying to solve, into code, it’s pretty natural to use the Codex harness to solve that problem for you. It’s done all the wiring and lets you just communicate in prompts to let the model cook. It’s been very fun, and it’s also a very engineering-legible way of increasing the usability.

Shawn Wang

It’s drastic, right? You just give the model all the scripts—the same scripts you would already build for yourself.

For listeners, this is Ryan saying that software engineering, or coding agents, will eat knowledge work—the non-coding parts that you would normally think, “Oh, you have to build a separate agent for it.” No, start with a coding agent and go up from there, which OpenClaw has, right? It’s pioneering the hood.

Yes. You probably define your task in code. Everything is a coding agent.

Shawn Wang

By the way, since I brought it up, this is probably the only place you’re bringing it up: is there any OpenClaw usage from you? Any?

No, no, not for AI. I don’t have any spare Mac minis rattling around my house.

Shawn Wang

Hey, you can afford it.

I’m kind of curious whether it’s changed anything in OpenAI yet, but it’s probably early days.

Shawn Wang

The other thing I want to pull on here is that you mentioned ticketing systems and you mentioned PRs, and I’m wondering if both those things have to go away or be reinvented for this kind of coding, right? Git itself is very hostile to multi-agents.

We make very heavy use of worktrees.

Shawn Wang

But even then, I just did a podcast yesterday with Cursor, and they said they’re getting rid of worktrees because they still have too many merge conflicts. It’s still too unintuitive. But, yeah, go ahead.

The models are really great at resolving merge conflicts, and to get to a state where I’m not synchronously in the loop in my terminal, I almost don’t care that there are merge conflicts.

We invoke a `$land` skill, and that coaches Codex to push the PR, wait for human and agent reviewers, wait for CI to be green, fix the flakes if there are any, merge upstream if the PR comes into conflict, wait for everything to pass, put it in the merge queue, and deal with flakes until it’s in main.

This is kind of what it means to delegate fully, right? This is a very large model repo, probably a significant tax on humans to get PRs merged, but the agent is more than capable of doing this, and I really don’t have to think about it other than keeping my laptop open.

I used to be much more of a control freak, but now I’m like, “Yeah, actually, you could do a better job of this than me.”

Shawn Wang

Yeah, with the right context. Anything else in harness engineering in general, just this piece? I just want to make sure we—

I think one thing that I maybe didn’t make super clear in the article, which I heard on Twitter as an interesting—

Shawn Wang

What was the chatter, and what was your response?

Ultimately, all the things that we have encoded in docs and tests and review agents and all these things are ways to put all the non-functional requirements of building high-scale, high-quality, reliable software into a space that prompt-injects the agent.

We either write it down as docs, or we add lints where the error messages tell it how to do the right thing. The whole meta of the thing is to basically tease out of the heads of all the engineers on my team what they think good looks like, what they would do by default, or what they would coach a new hire on the team to do to get things to merge.

That’s why we pay attention to all the mistakes that the agent makes, right? This is code being written that is misaligned with some as-yet-not-written-down non-functional requirement.

Shawn Wang

Sorry, what did the online people misunderstand?

No, what do you mean? Somebody just literally said that. I was like, “Oh, yeah, okay. This is the thing. This is what I’ve been doing.”

Shawn Wang

I see. I see. Interesting.

Another neat thing, which I totally did not expect, is that folks were just taking the link to the article and giving it to Pi or Codex and saying, “Make my repo this.”

Shawn Wang

Use the machine-learning recursion. And it was wildly effective. Really?

It was wildly effective.

Shawn Wang

No way. I actually tried something with GPT-5.4 yesterday. I didn’t have that much time; I was out speaking somewhere, and this is one of my things. I was like, “Okay, I have this article. Can we just scaffold out what it would be like to run this?”

I did it first as that, and then I was like, “Okay, let me take another little side repo and see: if I was to fully automate this, what would it look like?” I haven’t written a line of code. It’s a full stack, right? I think I’m doing it with voice TTS. I’m just slopping out whatever. It’s nothing production-ready. I’m like, “How would I make this like this?”

I feel like it’s a really good way to learn what could be changed. It’s just good analysis, right? You give it all the code, all the context, and the article, and it walks you through it very well.

That’s right.

Shawn Wang

I guess one more thing before we go to Symfony is I wanted to cover Bret Taylor’s response. We had him on the show. He is your chairman, which is wild—that he’s reading your articles as well and getting engaged in it. He said software dependencies are going away, basically. They can just be vendored.

Yes. 100% agree.

Shawn Wang

He’s still from Quip. He’s still paid by Datadog. He’s still paid by Temporal.

Thank you. Yep. The level of complexity of the dependencies that we can internalize is, I would say, low to medium right now, just based on model capability.

Shawn Wang

What is low and what is medium?

I would say a couple-thousand-line dependency is a thing that we could bring in-house with no problem in an afternoon.

One neat thing about it is that probably most of that code you don't even need, right? By in-housing an abstraction, you can strip away all the generic parts of it and focus only on what you need to enable the specific thing that you're building.

Shawn Wang

I've been calling this the end of plugins. When I publish an open-source thing, I want to accept everything and be liberal in what I accept, right? This is Postel's Law. But that means there's so much overhead.

One other neat thing about this, too, is that when we deploy Codex Security on the repo, it's able to deeply review and change the internals of dependencies in a much lower-friction way than pushing patches upstream, waiting for them to be released, pulling them down, and making sure they're compatible with all the transitive dependencies in my repo and things like that. It's also much lower friction to internalize some of these things if code is free because the tokens are cheap, sort of thing.

Shawn Wang

Yeah, yeah. I think the only argument I have against this is basically scale testing, which obviously matters for larger pieces of software like Linux and MySQL. Maybe security testing, where, classically, I think—is it Linus Torvalds who said that security in open source is the best disinfectant?

Many eyes. And if you inline your dependencies and code them up, you're going to have to relearn mistakes from other people. To internalize that dependency, you're back to zero, and you have to start reassembling all those bits and pieces to have high confidence in the code as it is written, right?

You've been part of the first intro of this. You basically mentioned that everything was written by Codex, including internal tooling, right? Internal tooling, like when you're visualizing what's going on, it's writing it for you.

Yeah, I'm building internal tools for AE now, and I just show them off and they're like, “How long did you spend?” I didn't spend any time. I just prompted it. Very funny story here.

Shawn Wang

Yeah, go ahead.

We had deployed our app to the first dozen users internally and had some performance issues, so we asked them to export a trace for us and get us a tarball. We gave it to our on-call engineer, and he did a fantastic job of working with Codex to build this beautiful local DevTools Next.js app where you drag and drop the tarball and it visualizes the entire trace. It's fantastic. It took an afternoon.

But none of this was necessary, because you could just spin up Codex, give it the tarball, and ask it the same thing and get the response immediately. In a way, optimizing for human legibility of that debugging process was wrong. It kept him in the loop unnecessarily, when instead Codex could have cooked for 5 minutes and gotten the same result.

Shawn Wang

Your instincts here were like, “This is how we used to do it,” or, “This is how I would have used to solve it.”

In this local observability stack, sure, you can deploy Jaeger to visualize the traces, but I wouldn't expect to be looking at the traces in the first place because I'm not going to write the code to fix them.

Shawn Wang

Yeah, I mean, basically there needs to be this kind of harness stack and owning the whole loop. I think that is very well established, and it sounds like you might be sharing more about that in the future, right?

Yeah, I think we're excited to do so. We're going to talk about Symphony in a little bit, but the way we distributed it as a spec—which I think folks are calling ghost libraries on Twitter; this is such a cool name—does mean it becomes much cheaper to share software with the world, right? You define a spec for how you could build your own, specifying as much as is required for a coding agent to reassemble it locally.

The flow here is very, very cool. We have taken all the scaffolding that has existed in our proprietary repo, spun up a new one, and asked Codex, with our repo as a reference, to write the spec. We tell it to spin up a tmux, spawn a detached Codex to implement the spec, wait for it to be done, then spawn another Codex in another tmux to review the spec—or review the implementation—compared to upstream and update the spec so it diverges less.

Then you just loop over and over and over, Ralph-style, until you get a spec that is, with high fidelity, able to reproduce the system as it is. It's fantastic.

Shawn Wang

You're not really adding any of your human bias in there, right? A lot of times people will write a spec and be like, “Okay, I think it should be done this way.” And you'll riff on something, and it's like, “No, the agent could have just handled it.” You're still scaffolding, in a sense, right? “I want it done this way.” It can determine the spec better.

That's right. That's right. Part of me—you know, I've been working a lot on evals recently—is wondering if an agent can produce a spec that it cannot solve. Is it always capable of things that it can imagine, or can it imagine things that are impossible to do?

I think with Symphony there's this axis, right, where you have things that are easy or hard, or established or new. Things that are hard and new are still something that the models need humans to drive.

Shawn Wang

Yeah, yeah. But I think those other quadrants are largely solved, given the right scaffold and the right thing that's going to drive the agent to completion. But it means that the humans—the ones with limited time and attention—get to work on the hardest stuff, right? The problems where it's pure white space out in front, or the deepest refactorings where you don't know what the proper shape of the interfaces are.

This is where I want to spend my time, because it lets me set up for the next level of scale. Yeah, amazing. Let's introduce Symphony. I think we've been mentioning it every now and then. Elixir, interesting option.

The Elixir implementation here is just a detail. Is it a model choice?

Uh, yeah. Yeah, and it chose that because the process supervision and the GenServers are super amenable to the type of process orchestration that we're doing here, right? You're essentially spinning up little daemons for every task that is in execution and driving it to completion, which means it gets a ton of stuff for free by using Elixir and the BEAM.

I had to go do a crash course in the BEAM and Elixir. I think most people are not operating at that scale of concurrency where you need that, but it is a good mental model for reusability and all those things, and these are some things I care about.

Shawn Wang

Tell me the origin story of Symphony. What do you use it for? How did it form? Maybe there were abandoned paths that you didn't take?

At the end of December, we were at about 3.5 PRs per engineer per day. This was before 5.2 came out. In the beginning of January, everyone got back from holiday, and with 5.2 and no other work on the repository, we were up to 5 to 10 PRs per day per engineer.

I don't know about y'all, but it's very taxing to constantly be switching like that. I was pretty tapped out at the end of the day. So, again, where are the humans spending their time? They're spending their time context-switching between all these active tmux panes to drive the agent forward.

We're like, wait, let's again build something to remove ourselves from the loop. This is what we frantically sprinted after here: finding a way to remove the need for the human to sit in front of their terminal. There was a lot of experimentation with dev boxes and automatically spinning up agents.

It seems like a fantastic end state, where my life is a beach. I open Linear twice a day and say yes or no to these things. This is again a super interesting framing for how the work is done, because I become more latency-insensitive. I have way less attachment to the code as it is written, and I've had close to zero investment in the actual authorship experience.

So if it's garbage, I can just throw it away and not care too much about it. In Symphony, there's this rework state where, once the PR is proposed and it escalates to the human for review, it should be a cheap review, right? It is either mergeable or it is not. If it's not, you move it to rework.

The Elixir service will completely trash the entire worktree and PR and start it again from scratch. This is that opportunity again to say, “Why was it trashed? What did the AI do that was bad?”

Shawn Wang

Yeah. Fix that before moving the ticket to in progress again. Why is this not in the Codex app? I guess you guys are ahead of the Codex app.

Yeah, the way the team has been working is basically to be as AI-pilled as possible and sprint ahead. A lot of the things we have worked on have fallen out into a lot of the products that we have.

We were in deep consultation with the Codex team to have the Codex app be a thing that exists, right? To have skills be a thing that Codex is able to use, so we didn't have to roll our own; to put automations into the product, so all of our automatic refactoring agents didn't have to be these hand-rolled control loops.

It has been really fantastic to be, in a way, unanchored to the product development of Frontier and Codex, and just very quickly try to figure out what works, then later find the scalable thing that can be deployed widely. It's been a very fun way to operate.

It's certainly chaotic. I have lost track very often of what the actual state of the code looks like because I'm not in the loop, right? There was one point where we had wired Playwright directly up to the Electron app with MCP.

I'm pretty bearish on MCP because the harness forcibly injects all those tokens into the context, and I don't really get a say over it. They mess with auto-compaction. The agent can't forget how to use the tool. There's probably only, like, 3 calls in Playwright that I actually ever want to use, so I pay the cost for a ton of things.

Somebody vibed a local daemon that boots Playwright and exposes a tiny little shim CLI to drive it. I had zero idea that this had occurred because, to me, I run Codex and it's able to—it's like, oh, it's slightly better. I had no knowledge of this at all. So we've had, in human space, to spend a lot of time doing synchronous knowledge sharing. We have a daily stand-up that's 45 minutes long because we almost have to fan out the understanding of the current state.

Shawn Wang

Yeah, I was going to say, this is good for a single human, multi-agent, but multi-human, multi-agent is a whole—probably an explosion of stuff.

Yeah, and this is fundamentally why we have such a rigid, 10,000-engineer-level architecture in the app, because we have to find ways to carve up the space so people are not trampling on each other.

Shawn Wang

Sorry, I don't get the 10,000 thing. Did I miss that?

The structure of the repository is, like, 500 npm packages. It's architecture to the extent of what you would consider normal for a 7-person team. But if every person is actually, like, 10 to 50 agents, then the numbers on being super, super deep into decomposition and sharding and proper interface boundaries make a lot more sense.

Shawn Wang

Yeah. To me, that's why I just talked about micro-frontends. I exited from that world, but it's cool that it's coming back. I don't know if you have other thoughts on orchestrating so much work and going through this. Is this enough? Is this any aha moment? It'll be interesting to see where—okay, so right now you pick Linear as your issue tracker, right? Or is it actually Linear?

This is actually Linear.

Shawn Wang

Oh, that's Linear. Linear. Oh, I know, I know—look at the video. The demo video I had to download to run.

I'm a Slack maxi, but, yeah, Linear is also really good. We do make good use of Slack. We fire off Codex to do all these low-velocity fix-ups, the things that sync that knowledge into the repository. It's super cheap. Just do it in Codex.

Shawn Wang

My biggest plug is that OpenAI needs to build Slack, right? You need to own Slack. Build yours to turn this into a style.

Peter Steinberger

I did read—yes, yeah. I would say that if we think that we want these agents to do economically valuable work—which is the mission, right? We want AI to be deployed widely to do economically valuable work—then we need to find ways for them to naturally collaborate with humans, which means collaboration tooling, I think, is an interesting space to explore.

Shawn Wang

Yeah, totally. GitHub, Slack, Linear. That was kind of my thing. Where do we see this going? Right now, Codex has started with Codex models and the CLI, and now there's an app. The app can let me shoot off multiple Codexes in parallel, but there's no great team collaboration for Codex, right?

It seems like your team had some say in what comes out, right? So you talked to them, and Codex kind of was a thing from there. What might you not focus on, but what do you expect other people to be building? For people that are 5×, 50×, should you build stuff that's very niche for your workflow, for your team? Should it be more general so other people can adopt this, or niche?

Part of it is just, okay, is everything just internal tooling? Do we have everything our own way? The way our team operates has its own ways that we like to communicate, or is there a broader way to do it? Is it something like an issue tracker? Just thoughts, if you want to riff on that.

Peter Steinberger

I think TBD. We have not figured this out in a general way. I do think that there is leverage to be had in making the code and the processes as much the same as possible. If you think that code is context, code is prompts, it's better from the agent behavior perspective to be able to look in a package in directory XYZ and not have to page so deeply into directory ABC because they have the same structure, use the same language, and have the same patterns internally.

That same leverage comes from aligning on a single set of skills that you're pouring every engineer's taste into to make sure that the agent is effective. In our codebase, we have, I think, 6 skills. That's it. If some part of the software development loop is not being covered, our first attempt is to encode it in one of the existing set of skills, which means that we can change the agent behavior more cheaply than changing the human driver's behavior.

Shawn Wang

Yeah. Have you ever experimented with agents changing their own behavior? Or a parent agent changing its subagents' behavior, or something like that?

Peter Steinberger

We do. Yes. We have some bits for skill distillation. For example, there's one neat thing you can do with Codex, which is just point it at its own session logs and ask it to tell you how you can use the tool better.

Shawn Wang

Better, yeah. Like introspection, or ask it to do things.

Peter Steinberger

Or ask it to organize this session better. What skills should I have?

Shawn Wang

I like the modification: you can just do things. You can just ask an agent to do things.

Peter Steinberger

Yeah, you can just Codex things. This is a silly emoji that we have, right? You can just Codex things. You can just prompt things. It's a really glorious feature we live in.

You can do that one-on-one, but we're actually slurping these up for the entire team into blob storage and running agent loops over them every day to figure out where, as a team, we can do better and how we can reflect that back into the repository so everybody learns from everybody else's behavior for free.

The same goes for PR comments. These are all feedback that means the code as written deviated from what was good. A PR comment, a failed build—these are all signals that mean, at some point, the agent was missing context. We've got to figure out how to slurp it up and put it back in the repo.

Shawn Wang

By the way, I do this exactly, right? I used to do it when I used Claude Code for my work. Claude Code is a nice product, right? And I think you would agree. I always have it tell me, "What do I do better next time?" That's the metaprogramming reflection thing.

It's almost like you have 6 reflection extraction levels in Symphony, but almost like the zero layer. The 6 levels are policy, configuration, coordination, execution, integration, and observability. We've talked about a couple of these. But the zero layer is, like, okay, are we working well? Can we improve how we work?

Peter Steinberger

Yes. Can I modify my own WORKFLOW.md or something? I don't know.

Shawn Wang

Yeah, of course you can.

Peter Steinberger

This thing is also able to cut its own tickets because we give it full access.

Shawn Wang

Yeah, make a new ticket to have it cut tickets. You can put in the ticket that you expect it to file as follow-up work. Self-modifying, yeah.

Peter Steinberger

But don't put the agent in a box. Give the agent full agency over its domain.

Shawn Wang

I had a mental reaction when you said, "Don't put the agent in a box." I think you should put it in a box. It's just that you're giving the box everything it needs.

Peter Steinberger

Yeah. Context and tools, right?

Shawn Wang

But as developers, we're used to calling out to different systems. Here, you use the open-source things like Prometheus, whatever, and you run it locally so that you can have the full loop, right? I assume.

Peter Steinberger

Yep. I think you want to minimize cloud dependencies. You also want to make sure that you think about what the agent has access to, right? What does it see? Does it go back in the loop?

At the most basic level, you let it see its own calls and traces. It can determine where it went wrong, right? But are you feeding that back in? You want to see exactly what's input and output. Does the agent have access to what is being output? It can self-improve a lot of these things.

It's all text, right? My job is to figure out ways to funnel text from one agent to the other. It's so strange. Way back at the start of this whole AI wave, Andre was like, "English is the hottest new programming language." It's here. The future is here.

A lot of software, a lot of stuff, has a GUI that's made for the human. We're seeing the evolution of CLIs for everything, right? All tools have CLIs. Your agents can use them well. But do we get good vision? Do we get good little sandboxes?

Right now, it's a really effective way. Models love to use tools. They love to bash. They love to read through text. So slap on a CLI and let it go loose. That works for everything.

We've also been adapting non-textual things to that shape in order to improve model behavior in some ways, right? We want the agent to be able to see the UI. Agents do not perceive visually in the same way that we do. They don't see a red box; they see "red box button." They see these things in the latent space.

We have a thing that goes off every time you say "latent space." Ding. Anyway, if we want to actually make it see the layout, it's almost easier to rasterize that image to ASCII art and feed it into the agent. There's no reason you can't do both, to further refine how the model perceives the object it's manipulating.

Shawn Wang

Cool. Do you want to talk about a couple more of these layers that might bear more introspection, or that you have personal passion for? I will say that the coordination layer here was a really tricky piece to get right.

Peter Steinberger

Let's do it. Yep, I'm all about that. This is the temporal core thing. This is where, when we turn the spec into Elixir, the model takes a shortcut, right? It's like, “Oh, I have all these primitives that I can make use of in this lovely runtime that has native process supervision,” which is, I think, kind of a neat way to have taken the spec and made it more real by making choices that naturally map to the domain.

In the same way that you would prefer to have a TypeScript monorepo if you're doing full-stack web development, right? Because the ability to share types across the front end and back end reduces a lot of complexity.

Shawn Wang

That's what GraphQL used to be.

Peter Steinberger

That's right. And I don't know if it's still alive, but there are no humans in the loop here, so my own personal ability to write or not write Elixir doesn't really have to bias us away from using the right tool for the job.

Shawn Wang

It is just wild. Love it. I love it. I wonder if any languages struggle more than others because of this. I feel like everyone has their own abstractions that would make sense, but maybe it might be slower, or it might be more faulty, where you have to just kick the server every now and then.

Peter Steinberger

I don't know. I think the observability layer is really well understood. The integration layer—MCP is dead. I think all these are just a really interesting hierarchy to travel up and down. It's common language for people working on the system to understand.

The policy stuff is really cool, right? You don't really have to build a bunch of code to make sure the system waits for the CI to pass.

Shawn Wang

It's your institutional knowledge.

Peter Steinberger

Yeah, you just give it the GitHub CLI with some text to say, “The CLI has to pass.” Right? It makes the maintenance of these systems a lot easier.

Shawn Wang

Do you think CLI maintainers need to do anything special for agents, or is it good as is? Because I don't think, when people made the GitHub CLI, they anticipated this happening.

Peter Steinberger

That's correct. The GitHub CLI is fantastic. It's great—super industry-standard. Everyone, go try gh repo create, gh pr checkout 153, whatever, right? And then it pulls. Basically, my only interaction with the GitHub web UI at this point is gh pr view --web, just to take a glance at the diff and be like, “Sure thing, send it.”

The CLIs are nice because they're super token-efficient, and they can be made more token-efficient really easily, right? I'm sure you've all seen this: you go to Buildkite or Jenkins and get this massive wall of build output.

Shawn Wang

Yeah, yeah, yeah, yeah. In order to unblock the humans, your developer productivity team is almost certainly going to write some code that parses the actual exception out of the build logs and sticks it in a sticky note at the top of the page.

Peter Steinberger

Similarly, in our pnpm recursive script runner, when we had one, if you do --recursive, it produces an absolute mountain of text. But all of that is for passing test suites, so we ended up wrapping all of this in another script to suppress the output.

Shawn Wang

The one channel should only output the failing parts of the test. You pipe errors versus standard output. I don't know. Okay, whatever. Too much thinking that I have to do. I used to maintain a CLI for my company, and, yeah, this is very core to my heart. But you're vibing my job.

That's right.

Shawn Wang

Cool. Any other things? I mean, this is a long spec. I appreciate that it's got a lot of strong opinions in here. Any other things that we should highlight? I was thinking we could spend the whole day going through some of these, but I do think that some of these have a lot of care behind them. Or some of this you might want to tell people, “Hey, take this, but make it your own.”

Fundamentally, software is made more flexible when it's able to adapt to the environment in which it is deployed, right? Which means that things like Linear or GitHub are specified within the spec, but they're not required pieces of it, right? There's a more platonic ideal of the thing that you could swap in, like Jira or Bitbucket, for example.

But being able to tightly specify things like the ID formats or how the Ralph loop works for the individual agents basically means you can get up and running with a fully specified system quickly that you can then evolve later on. I think we never intended for this to be a static spec that you can never change, right? It's more like a blueprint to get something worth starting with up and running for you, and then to vibe later to your heart's content.

Shawn Wang

Yeah.

You have code and scripts in here where it's like, “Oh, I think this is a really good prompt.” It's just a very, very long prompt. Fundamentally, the agents are good at following instructions. So give them instructions, right? And it will improve the reliability of the result.

Much like the way we use Symphony, we don't want folks to have to monitor the agent as it is vibing the system into existence. So being very opinionated and very strict around what these success criteria are means that our deployment success rate goes up. It means we don't have to get tickets on this thing.

Shawn Wang

Yeah, and then it all goes back to that idea that code is disposable, right? Early on, when you had the Codex CLI, you'd kick off a Codex run and it would take 2 hours. You would kind of want to monitor it, like, “Okay, I'm in the workflow of just using one. I don't want it to go down the wrong path. I'll cut it off and then, you know, just shoot out 4.” That was my favorite thing about the Codex app, right?

Yeah. Just 4 agents. It's okay. One of them will probably be right. One of them might be better. Stop overthinking it.

Shawn Wang

My first example is probably Deep Research. When you put out Deep Research, I'd ask it something about LLMs, and it thought it was something legal and spent an hour. It came back with a report completely off the rails, and I was like, “Okay, I have to monitor this thing a bit.”

No, don't monitor it. You want to build it so that it goes the right way, and you don't want to sit there and babysit, right? You don't want to babysit your agents.

With that Deep Research query that you made, looking at the bad result, you probably figured out you needed to tweak your prompt a bit, right? That's the guardrail that you fed back into the codebase for the task—your prompt—to further align the agent's execution. The same sort of concepts apply there, too.

Shawn Wang

When you talk—I mean, how are the customers feeling? For Symphony, I think we have none, right? This is a thing we have put out into the world.

Symphony's internal, right? As long as you're happy, you're the customer.

Shawn Wang

That's right. I just, you know, what's the external view? Folks are very excited about this way of distributing software and ideas in cheap ways. For us as users, it has again pushed productivity 5x, which means I think there's something here that's a durable pattern around removing the human from the loop and figuring out ways to trust the output, right?

The video that he's shared here—

Yeah.

Shawn Wang

It's the same sort of video we would expect the coding agent to attach to the PR it created.

Yeah. And that's part of building trust in the system. That's, to me, fundamentally what has been cool about building this: it more closely pushes that persona of the agent working with you to be like a teammate, right? I don't shoulder-surf you for the tickets that you work on during the week. I would never think that I would want to do that.

Shawn Wang

Yeah. I wouldn't want a screen recording of your entire session in Cursor or Claude Code. I would expect you to do what you think you need to do to convince me that the code is good and mergeable.

Yeah.

Shawn Wang

And compress that full trajectory in a way that is legible to me, the reviewer.

You can just do that because Codex will absolutely sling some effort to take around. It's great.

Shawn Wang

Oh, I mean, FFmpeg is the OG god CLI.

Yeah, Swiss Army chainsaw. I used to say there's a SaaS—micro-SaaS, let's call it—in every flag in FFmpeg.

Shawn Wang

Oh, for sure. You know what I mean? Just host it as a service and put a UI on it. People who don't know FFmpeg will pay for it.

When we were first experimenting with this, it was a wild feeling to be at the computer with windows just popping up all over the place and files appearing on my desktop. It very much felt like the future to have a thing controlling my computer for actual productive use, right?

Shawn Wang

I'm just there keeping it awake, jiggling the mouse every once in a while. That's what some office workers do, so they provide a mouse-jiggler device for—

That's right. That's right.

Shawn Wang

One thing I would ask is, as code is disposable and you can shoot off about 4 agents, are you always an xhigh-thinking guy? And where do you see GPT-5.3-Codex-Spark? There's a lot of me wanting to make quick changes.

I'm not going to open up an IDE. I'm not going to do anything, but I will say, "Okay, fix this little thing, change your line, change your color." Spark is great for that. But am I still the bottleneck? Why don't I just let that go back? Just briefly on that, is Spark such a different model compared to the xhigh-level reasoning that you get in these 5.3s?

For people, it is a different model, different architecture, different—like, it doesn't support a—

Shawn Wang

It's an incredibly fast, small model.

I have not quite figured out how to use it yet, to be honest. I was adapting it to the same sorts of tasks I would use xhigh reasoning for, and it would blow through 3 compactions before writing a line of code.

Shawn Wang

And that's another big thing with 5.4, right? A 1-million-token context.

Yes, it's fantastic.

Shawn Wang

Huge in agentic work, right? You can just run for longer before you have to compact. The more tokens you can spend on a task before compacting, the better you'll do.

That's right. That's right. I'm not sure how to deploy Spark. I think your intuition is right: it's very great for spiking out prototypes, exploring ideas quickly, and doing those documentation updates. It is fantastic for us at taking that feedback and transforming it into a lint rule, where we already have good infrastructure for ESLint in the codebase. These sorts of things it's great at, and it allows us to unblock quickly by doing those anti-fragile healing tasks in the codebase.

Shawn Wang

Yeah, that makes sense. You guys are pushing models to the freaking limit. What can Codex models not do well yet? They're definitely not there on being able to go from a new product idea to a prototype in a single shot.

This is where I find I spend a lot of time steering: translating the end state of a mock for a net-new thing—something with no existing screens—into a product that is playable. Similarly, while this has gotten better with each model release, the gnarlier refactorings are the ones that I spend most of my time with, right? The ones where I'm interrupting the most, the ones where I am now double-clicking to build tooling to help decompose monoliths and things like that.

This is a thing I only expect to get better, right? Over the course of a month, we went from the low-complexity tasks to low-complexity and big tasks in both these directions. So, this is what it means to not bet against the model, right? You should expect that it is going to push itself out into these higher and higher complexity spaces. The things we do are robust to that. It just basically means I'll be able to spend my time elsewhere and figure out what the next bottleneck is.

I do think it's also a bit of a different type of task, right? Codex is really good at codebase understanding and working with codebases, but companies like Lovable, Bolt, and Replit solve a very different problem: scaffolding from 0 to 1, right? Idea to product. Models are also pushing step-function changes there. It's just kind of different from the software engineering agents today, right?

Like I said, the model is isomorphic to myself. The only thing that's different is figuring out how to get what's in here into context for the model. And for these white-space sort of projects, I myself am just not good at it, which means that often over the agent trajectory, I realize the bits that were missing, which is why I find I need to have the synchronous interaction.

I expect, with the right harness and the right scaffold that's able to tease that out of me or refine the possible space, right—to be super opinionated around the frameworks that are deployed or to put a template in place—these are ways to give the model all those nonfunctional requirements, that extra context to anchor on, and avoid that wide dispersion of possible outcomes.

Shawn Wang

Thank you for that. I want to talk a little bit about Frontier. Overall, you guys announced it maybe a month ago, and there are a few charts in here. It's basically kind of your enterprise offering, as I view it. Is there one product, or are there many?

Peter Steinberger

I can't speak to the full product roadmap here, but what I can say is that Frontier is the platform by which we want to do AI transformation of every enterprise, from big to small. The way we want to do that is by making it easy to deploy highly observable, safe, controlled, identifiable agents into the workplace, right?

We want it to work with your company's native IAM stack. We want it to plug into the security tooling that you have. We want it to be able to plug into the workspace tools that you use.

Shawn Wang

So, you're just going to be shipping specs, right?

Peter Steinberger

We expect that there will be some harness things there. The Agents SDK is a core part of this, to enable both startup builders as well as enterprise builders to have a works-by-default harness that is able to use all the best of our models, from the Shell tool down to the Codex harness, with file attachments and containers and all these other things that we know go into building highly reliable, complex agents.

We want to make that great, and we want to make it easy to compose these things together in ways that are safe. For example, the GPT-OSS-Safeguard model—one thing that's really cool about it is that it ships the ability to interface with a safety spec. Safety specs are things that are bespoke to enterprises.

We owe it to these folks to figure out ways for them to instrument the agents in their enterprise to avoid exfiltration in the ways they specifically care about, to know about their internal company codenames, these sorts of things. Providing the right hooks to make the platform customizable, but also mostly working by default for folks, is kind of the space we're trying to explore here.

Shawn Wang

And this is like, you know, the Snowflakes of the world just need this, right? The Brexes of the world, Stripes. Makes sense. I think the demo videos that you guys had were pretty illustrative. It's kind of, to me, also an example of very large-scale agent management.

Peter Steinberger

Yes. You give people a control dashboard that, if you play any one of these multiple-agent things, you can dig down to the individual instance and see what's going on.

Shawn Wang

Yes, of course. But who's the user? Is it the CEO, the CTO, CIO, something like that?

Peter Steinberger

At least with my personal opinion here, the buyer that we're trying to build the product for here is, one, the end employees who are making productive use of these agents, right? That's going to be whatever surfaces they appear in, the connectors they have access to, things like that.

Something like this dashboard is for IT, your GRC and governance folks, your AI innovation office, your security team, right? The stakeholders in your company that are responsible for successfully deploying into the spaces where your employees work, as well as doing so in a safe way that is consistent with all the regulatory requirements that you have, customer attestations, and things like that. So, it is kind of an iceberg beneath the actual endpoints.

Shawn Wang

You jump—like, every, I guess, layer in the UI is going down a layer of abstraction in terms of the agent, right?

Peter Steinberger

Yep. Yeah, yeah, I think it's good. The ability to dive deep into the individual agent trajectory level is going to be super powerful, not only from a security perspective, but also from the perspective of someone who is accountable for developing skills.

One thing that was interesting that we also blogged about shipping was an internal data agent, which uses a lot of the Frontier technology in order to make our data ontology accessible to the agent and things like that, to understand what's actually in the data warehouse.

Shawn Wang

Yeah, semantic-layer-type things. I was briefly part of that world. Is it solved? I don't know. It's actually really hard for humans to agree on what revenue is. What is an active user?

Peter Steinberger

There are, what, 5 data scientists in the company who have defined this golden metric. They already defined it, so, yeah. And there's also internal politics as to attribution: marketing is responsible for this much, and sales is responsible for this much, and they all add up to more than 100%. I'm like, well, you guys have different definitions. And if you're a startup, everything is ARR, you know? So I think that's cool.

Shawn Wang

Oh, you guys blogged about this. Okay, I didn't see this.

Peter Steinberger

Yeah, is this the same team? I don't know. Is this what you're referring to?

Shawn Wang

Yes. Okay, well, we'll send people to read this. This is our data agent. I don't know if you have any highlights.

Peter Steinberger

No, no, no. I mean, in general, from the podcast, a lot of good things to read.

Shawn Wang

Yeah, yeah, lots of homework for people. No, but data as the feedback layer—you need to solve this first in order to have the product's feedback loop closed.

Peter Steinberger

That's right. Right. So, for the agents to understand. And this is not something that humans have solved. This is how you build—

Shawn Wang

OpenAI agents that do more than coding, right?

Peter Steinberger

Yeah, yeah. To actually understand how you operate the business, yeah. You have to understand what revenue is, what your customer segments are, right? What your product lines are, right?

One thing that's in core-beliefs.md is who's on the team, what product we're building, who our end customers are, who our pilot customers are, and what the full vision of what we want to achieve over the next 12 months is.

These are all bits of context that inform how we would go about building the software.

Shawn Wang

Oh my God. So, we have to give it to the agent, too. I’m guessing that stuff is pretty dynamic and changes over time, too, right? It’s not just a big spec. You have it as one of the things, and it iterates.

Peter Steinberger

One thing that I think is going to break your mind even more is that we have skills for how to properly generate deep-fried memes and have reacji culture in Slack. With the Slack ChatGPT app that you’re able to use and Codex, I can get the agent to shitpost on my behalf. I think humor is part of AGI. Is it funny?

Shawn Wang

It’s pretty good.

Peter Steinberger

Yeah. It’s pretty good at making deep-fried memes. I think humor is a really hard intelligence test, right? You have to get a lot of context into very few words. This is why GPT-5.4 is such a big uplift for our team culture. It’s the memeing.

Shawn Wang

Yeah, for sure. It’s very cool. So, GPT-5.4 can shitpost. That’s all we take away.

Peter Steinberger

Yeah. Maybe when you’re done here today, ask Codex to go over your coding-agent sessions and roast you.

Shawn Wang

Love it. I’ll give it a shot.

Peter Steinberger

Give it a shot.

Shawn Wang

Coming back to the final point I wanted to make, I think this is a pattern that every other company out there should adopt, regardless of whether or not they work with you. I saw this and thought, “Every company needs this.” I mean, this is multiple businesses’ worth of work to get people to actually realize the benefit and distribute it.

Peter Steinberger

Build it and distribute it.

Shawn Wang

I think it sounds boring to people. They hear, “Oh, it’s for safeguards and whatever,” but to handle agents at scale like you’re envisioning here, this is what you need. I don’t know if it’s a real screenshot or a demo, but this is my original view of what Temporal was supposed to be. You build this dashboard, and you basically have every long-running process in the company in one dashboard. And that’s it.

Peter Steinberger

That’s right. That’s right. I think it’s pretty customized toward every enterprise, right? You care about different things.

Shawn Wang

There’s a lot of customization, right? But I mean, there’ll be multiple unicorns just doing this as a service. I’m very Frontier-pilled, if you can tell. Amazing. It only clicked because, obviously, this came out first, then Harness, and then Symphony. It only clicked for me that this is actually kind of the thing you ship to do that.

Peter Steinberger

Yeah. There’s a set of building blocks here that we assembled into these agents, and the building blocks themselves are part of the product, right? The ability to steer and revoke authorization if a model becomes misaligned—all of this is accessible through Frontier. There are going to be a bunch of stakeholders in the company who have the things they need to see in the platform to enable widespread deployment. We’ll build all of those into Frontier so that we can actually do the widespread deployment. That’s the fun part.

Shawn Wang

Yeah. I’m also calling back to the levels of AGI. I don’t know if OpenAI is still talking about this, but they used to talk about 5 levels of AGI. One of them was an intern-level coding software-engineer agent. At some point, it was an AI organization. And this is it.

Peter Steinberger

That’s right.

Shawn Wang

This is level 4 or 5. I can’t remember which level, but somewhere along that path was this.

Peter Steinberger

You know how I mentioned that my team is having fun sprinting ahead here, right? We do this thing where we’re collecting all the agent trajectories from Codex to slurp them up and distill them. This is what it means to build our team-level knowledge base. We happen to reflect it back into the codebase, but it doesn’t have to be that way, right? It doesn’t have to be bound to just Codex, right?

I want ChatGPT to also learn our meme culture and how we are building it, so that when I go ask it something, it also has the full context of the way I do my work. I’m super excited for Frontier to enable this.

Shawn Wang

What do the model people say when they see you do this? You have a lot of feedback, obviously. You have a lot of usage and a lot of trajectories. I don’t imagine a lot of it is useful to them, but some of it is. And you deploy 1 billion tokens of intelligence a day. This was at the beginning of 2016. You’re cooking.

Peter Steinberger

There’s this fundamental tension, which I think you have talked about, between whether or not we invest deeper into the harness or invest deeper into the training process to get the model to do more of this by default. I think success for the way we are operating here means the model gets better taste because we can point the way there.

None of the things we have built actively degrade agent performance, because really all they’re doing is running tests. Running tests is a good part of what it means to write reliable software. If we were building an entire separate Rust scaffold around Codex to restrict its output, that would be additional harness that would be prone to being scrapped.

But if instead we can build all the guardrails in a way that’s native to the output Codex is already producing, which is code, I think we’re in no friction with how the model continues to advance. It’s also just good engineering, and that’s the whole point.

Shawn Wang

I’ve had similar discussions with research scientists where the RL equivalent is on-policy versus off-policy. You’re basically saying that you should build an on-policy harness, which is already well within distribution, and modify it from there. But if you build it off-policy, it’s not that useful. Super cool.

What are your parting thoughts? Is there anything we haven’t covered that we should get out there?

David Luan

I’ve been super excited to benefit from all the cooking that the Codex team has been doing. They absolutely ship relentlessly. This is one of our core engineering values—ship relentlessly—and the team there embodies it to an extreme degree.

Shawn Wang

Oh yeah, I have GPT-5.3, and then Spark and GPT-5.4 come out within what feels like a month. It’s phenomenally fast.

David Luan

It was exactly a month ago: GPT-5.3, and yesterday was GPT-5.4.

Shawn Wang

Do we have a release every month now? Is GPT-5.5 next?

David Luan

I can’t say that. The Polymarket bettors would be very upset, right?

Shawn Wang

I think it’s interesting that it’s also correlated with the growth. They announced it as 2 million users, but I almost don’t care about Codex anymore. This is it. This is the game, man. It’s like coding cool software and large-scale work. This is the thing to chase after.

David Luan

That’s right. This is one of the things that my team is excited to support.

Shawn Wang

Get the whole self-hosted harness thing working, which you have done, while the rest of us are trying to figure out how to catch up. But then do things with it. Do things.

David Luan

That’s right. You can just do things.

Shawn Wang

That’s the line for the episode. That’s it. You’re based in Seattle, your team, I’m guessing. New Bellevue office?

David Luan

New Bellevue office. We just had the grand opening yesterday, as of the recording date, which went fantastically. It’s a beautiful building, and I’m super excited to be part of the Bellevue community, building the future in Washington. There’s lots of work to be done in order to successfully serve enterprise customers here in Frontier. We are certainly hiring. And if you haven't tried the Codox app yet, please give it a download. We just passed 2 million weekly active users, growing at a phenomenally fast rate, 25% week over week. Peace. Come join us.

Shawn Wang

Yeah. I think that’s an interesting observation. My final observation is that OpenAI is a very San Francisco-centric company. I know people who have turned down the job or didn’t get the job because they didn’t want to move to San Francisco. Now they just don’t have a choice, right? You have to open offices in London, and you have to open offices in Seattle. I wonder if that’s going to be a shift in the culture. Obviously, you can’t say, but I was—

David Luan

I was one of the first engineering hires out of our Seattle office.

Shawn Wang

Yeah, yeah.

David Luan

Seattle was very natural. Its success has been part of what I have been building toward, and it has grown quite well, right? We have durable products and lines of business that are built out of there. A ton of zero-to-one work is happening as well, which is kind of the core essence of the way we do applied AI work at the company: sprint after it and try to figure out where we can actually successfully deploy the model. So, yes, 100%. We also have a New York office that has a ton of engineering presence.

Shawn Wang

Yeah, exactly. These are on my roadmap for AI: wherever people hire engineers, I will go.

David Luan

That’s right. It’s a cool office, too. New York is the old REI building, I believe—the REI office.

Shawn Wang

It’s just—no, you’ll never be as big, right? In New York, you can’t get the size of office that they need. The New York office has a very Mad Men sort of vibe. It’s beautiful. The Bellevue one has very green and gold fixtures, very Pacific Northwest. It’s very cool.

David Luan

It’s the vibe of the local area. For people who like New York, they want to be in New York, right?

Shawn Wang

Yeah. Yeah.

David Luan

We have a fantastic workplace team that has been building out these offices. It really is a privilege to work here.

Shawn Wang

Yeah, excellent.

Okay, well, thank you for your time. You've been very generous, and you've been cooking, so I'm going to let you get back to cooking. It's been amazing chatting with you folks. Happy Friday.

David Luan

Happy Friday.