How Anthropic ships the #1 LLM in the World — with Katelyn Lesse and Angela Jiang
- The opening’s IPO case rests on a sharp improvement in employee and capital efficiency. Citing Anthropic’s S-1, the host reports $6.6 million in trailing-12-month revenue per employee versus $1.4 million a year earlier, and a first profitable quarter at 19% on a GAAP basis versus a loss equal to 129% of revenue. The opening also puts lifetime fundraising above $100 billion and burn below $10 billion, with an IPO expected “in two weeks at a $2 trillion valuation.”
- Claude Platform gives Anthropic a business that extends beyond selling access to model intelligence. The opening says more than 50% of customers with at least $100,000 in trailing-12-month revenue use the platform, whose primitives also underpin Claude Code and Cowork. Managed Agents adds infrastructure and harness management, complementing the host’s framing of Anthropic as a cloud platform.
- Opus 5.5’s pitch combines stronger economics with a more usable model. Katelyn Lesse highlights price cuts, less convoluted writing than Opus 5, and demand for “good intelligence per dollar.” The guests qualify the host’s “top model” description with “depends who you ask”; the opening’s prediction that Fable 5.5 will probably follow within two weeks comes with an explicit “I don’t have insider information.”
- Safety engineering has become a major component of the model-release workload. Releases now require decisions about retention, online safeguards, offline review, verification programs, and authorization throughout the stack. Anti-distillation measures preserve thinking blocks while accommodating legitimate compaction. Katelyn explains the safety rationale; the host says safety, rather than commercial impact, is the main concern and notes that distillation usage pays Anthropic.
- CPU availability is becoming a material constraint alongside accelerator supply. The guests describe APIs, sandboxes, research, and growing CI activity competing for Anthropic’s allocation. A Claude-assisted port of the Messages API from Python to Rust delivered roughly 40–50% CPU savings and a 10–20% improvement in time to first token; the critical human contributions were “the plan” and “the verification.”
- Separating the agent’s brain from its hands turns security into an infrastructure property. Managed Agents keeps the harness durable, execution isolated, and credentials in a separate vault, with an egress proxy injecting secrets the agent never sees. That architecture also improves startup latency and resilience when sandboxes fail; a guest says a colleague who prioritized security was vindicated.
- Evaluation and personalization remain unresolved constraints on product quality. Different customers want conflicting levels of initiative, and the guests admit, “We don’t do a great job of it.” Their response combines broader eval suites, adjustable effort, and managed harnesses; Angela Jiang is more bullish on a consistent Claude that dynamically changes effort than on visibly switching users between models.
- The next workload opportunity is longer autonomous work with stronger verification. Angela sees some tasks approaching intelligence saturation, while difficult engineering and science problems remain open. The closing multi-agent advice is one lead, shared state, and asynchronous workers that do not block each other; throughout, humans retain responsibility for quality, because faster implementation is no excuse to ship “10X more features into prod.”
1. The IPO framing centers on efficiency and a broader platform business
The opening host says Anthropic filed its S-1 that day and frames an IPO expected “in two weeks at a $2 trillion valuation.” He contrasts reported lifetime fundraising above $100 billion with lifetime burn below $10 billion. These figures establish the episode’s opening question: how is Anthropic shipping what he regards as the strongest models while building its business unusually efficiently?
The cited operating figures describe a sharp reversal: trailing-12-month revenue per employee rises from $1.4 million to $6.6 million, almost fivefold. The host says Anthropic was “losing half a dollar for every dollar of revenue” in Q2 2025 and now generates $2 of revenue per $1 of cost. Its first profitable quarter shows 19% GAAP profit, versus a prior loss equal to 129% of revenue.
The host explains the improvement through two reinforcing loops. Better Claude models unlock use cases, adoption produces feedback, and feedback improves Claude; separately, adoption increases revenue and growth, attracting investment that funds more capability. He presents these as the S-1’s “improvement loop” and “investment loop,” linking product performance with the resources available to keep improving it.
More than 50% of customers with at least $100,000 in trailing-12-month revenue reportedly use Claude Platform. The host emphasizes that Claude Code and Cowork use the same exposed primitives. He interprets Anthropic’s positioning as moving from chatbots and copilots toward “virtual collaborators”: systems that own goals, use tools, self-correct, and work alongside people across longer tasks.
2. Opus 5.5 targets intelligence per dollar and clearer communication
Asked to celebrate the “top model in the world,” the guests immediately qualify it: “Depends who you ask.” Katelyn is particularly pleased that feedback about Opus 5’s convoluted writing translated into a cleaner successor. Price cuts address a separate demand: customers increasingly care about “good intelligence per dollar,” so the team must improve both intelligence and its cost.
The opening was recorded around the Opus 5.5 launch and describes strong reactions from the host’s testers. His additional prediction that Anthropic is sitting on Fable 5.5 and will “probably” release it within two weeks is explicitly speculation: “I don’t have insider information.” He connects that timing to the anticipated IPO rather than presenting a confirmed release schedule.
Katelyn describes product’s role as balancing intelligence, safety, cost, and commercial impact; she says engineering emphasizes integration and developer experience. Their technical product managers and product-minded engineers deliberately overlap. Asked whether software engineering being “solved” will merge the roles, Katelyn rejects that strength of language: “Ambitions will continue to elevate,” leaving substantial operating and design work for humans.
3. Anti-distillation changes protect safeguards while preserving legitimate uses
The discussion of “Stealing Reasoning Traces from Proprietary LLMs” starts with responsible disclosure, which both guests welcome. Katelyn says the paper exposed a gap Anthropic already understood: extracting chain of thought is “one of the most powerful ways that you can distill models.” The engineering question was how to close that avenue without making useful API integrations substantially worse.
Katelyn’s rationale centers on safeguards: Anthropic classifies generated tokens to reduce harmful outcomes, while a distilled model could run without those protections. The host, not Angela, says the primary concern is safety rather than commercialization and notes that “if you distill from us, you actually pay us.” His concern is a derivative of Claude being used negatively without the surrounding safety systems. Angela separately says distillation has become significantly harder in the areas Anthropic can measure.
The team distinguishes two kinds of user-experience damage: an annoying integration migration that preserves capabilities, and a change that makes previously possible work impossible. They especially want to avoid the latter. Preserving thinking blocks across turns, and tightening their treatment across accounts, required separating legitimate workflows from extraction techniques rather than simply banning every manipulation of reasoning traces.
Compaction supplied the concrete test. Some customers’ existing strategies would break under preserved-thinking requirements, so Anthropic worked with customers and open-source harness maintainers and shipped an API compaction feature compatible with them. Angela does not claim complete visibility or that extraction has been eliminated.
4. Every model now needs its own safety deployment policy
The guest discussing release safety frames it across three dimensions: architecture and infrastructure security, online safeguards operating during generation, and offline classification of stored data. Capability determines the work required. Katelyn adds that specific spikes in biology or cybersecurity can demand different treatment depending on whether capabilities appear offense-dominant or defense-dominant, rather than treating all intelligence improvements as equivalent.
Retention addresses behavior that may not be recognized while an individual request streams. Networks of accounts may collectively reveal malicious use or distillation, requiring automated or human review. Angela explicitly distinguishes models: Opus 5.5 does not carry the data-retention requirements Fable has. Those requirements are decisions made for individual releases, with engineering needed to enforce them.
Cyber verification and the newer life-science verification program let approved users access dual-use capabilities with reduced safeguards for specified purposes. Legitimate vulnerability research and biological research are the examples. Authorization must follow the request throughout the system; Angela contrasts that with casually issuing a key and flipping a feature flag. Launching these programs is now part of launching the model itself.
5. Generic trust grants make uncertain deployment scenarios manageable
Fable’s rollback becomes Angela’s example of a roadmap that cannot rely on confidently forecasting use cases. Something the team would not have predicted nevertheless happened, establishing a nonzero probability. Her response is “deeply probabilistic” product management: build “a portfolio of bets,” including controls that may never be needed but provide options if an unusual situation returns.
The access system began with granting particular users particular capabilities, but the team chose more generic primitives than conventional product scoping might favor. Those primitives proved useful for broader model access, trusted programs, and model deployments. Work toward a single-button rollback extends that same foundation. Angela says this generic approach has repeatedly paid dividends, while leaving the value of any particular hedge uncertain.
A “trust grant” is ordinary authorization expressed through Anthropic’s own data model, not a fundamentally different security concept. For Fable 5.1’s preserved-thinking rollout, existing customers received a grace period to update integrations; grants tracked those exceptions. A service on the request’s hot path checks whether the requester holds the relevant permission, allowing the same mechanism to support multiple policies.
6. Model feedback is most valuable when external harnesses expose failures
Model launches move through candidate versions, evaluation, internal dogfooding, and selected external testing. Customers such as Cognition provide a different harness from Anthropic’s own products, which matters because Claude is intended to generalize. Angela wants both formal results and subjective reactions: “Do you like it? What don’t you like?” Negative feedback is often the most useful input for further improvement.
The host’s pushback is that feedback still resembles a Slack channel saying, “Hey, this thing seems weird.” Angela agrees the process can be informal, partly because the team is “racing” to minimize the interval between a testable model and broad availability. Early-access customers therefore tend to have robust eval suites capable of producing actionable evidence quickly, alongside subjective impressions.
Laziness versus initiative illustrates why outside testing matters. One customer wants Claude to “just finish my task”; another is still planning when Claude unexpectedly writes code. Katelyn says getting that spectrum exactly right for everyone is “literally impossible.” In one case, excessive laziness led the team to put a candidate “back in the oven,” a problem internal product usage might have missed.
7. Personalization exceeds what today’s parameters can express
The host uses a code benchmark to challenge model evaluation: a benchmark can penalize excessive activity or file creation even when a user’s CLAUDE.md instructions could suppress that behavior. Is the benchmark then unfairly penalizing the model? The guests’ candid answer on evaluating personalization is, “We don’t do a great job of it.” A constitution can help, Katelyn says, but does not resolve conflicting preferences about how assistance should behave.
Effort provides a limited customization mechanism. The conversation specifically identifies Claude Code’s default as medium, while Katelyn says defaults can differ across free users, prosumers, enterprise, Cowork, and API use. The preference is for “gentle defaults” with user control. Some users want maximum intelligence; others benefit from lower effort and faster responses, making a single aggressive default difficult to justify.
Katelyn describes a “funny mishmash” of effort, budgets, and other parameters, with no obvious combination for every task. Choices include whether to expose low/medium/high or a 1-to-100 scale, and which controls to retire. As the number of customers grows, she says relying on judgment calls about conflicting preferences will not scale indefinitely; richer ways for Claude to absorb personalization will be needed, but are not solved today.
8. Managed Agents packages the harness while leaving room for experimentation
Angela distinguishes the Messages API’s “raw token access” from Claude Managed Agents, a higher abstraction that lets developers define an agent and start a session. Anthropic supplies infrastructure and a harness, including work to accommodate new model parameters and breaking changes. The value is programmatic access without requiring every customer to repeatedly rediscover how to get the best behavior from each release.
Managed Agents shares substantial infrastructure with first-party products, including the sandbox fleet, although shared infrastructure does not mean identical harnesses. Angela says the provided harness is “intentionally simple”: it incorporates useful practices and handles difficult but undifferentiated parts of the agent loop. More opinionated strategies can then sit above that foundation instead of being imposed on every developer.
Asked whether memory and dreaming have stalled, Angela acknowledges that some feature work has slowed relative to earlier in the year as safety consumes more attention. That is not an abandonment: “We love skills.” Memory and dreaming still address continuous improvement, while the team increasingly wants developers to extend existing primitives and Anthropic to concentrate on providing safe access to model capabilities.
Dreaming is itself a multi-agent construction: another managed agent examines the previous agent and writes memories for it. “It’s managed agents all the way down.” Angela also names best-of-N and verifier agents as possible strategies. An update expected “relatively soon” would expose underlying primitives and templates so developers could construct their own variants, rather than only consuming Anthropic’s particular dreaming implementation.
9. Agent engineering increasingly resembles experimental research
Angela’s eval advice is breadth: “You probably need more rather than less.” Static deterministic checks and LLM judges serve different purposes; a small supposedly perfect suite will not answer every question. She also recommends starting simply. Debating an elaborate menu of evaluation methods while having zero actual evals is worse than implementing a basic useful check and improving it.
The host proposes treating engineering like research: define the goal, formalize an eval, label results, prototype, and scale runs. Failure diagnosis must consider at least three possibilities: the eval is wrong, the environment is wrong, or the model is wrong. Katelyn agrees that “a lot of the time your eval is wrong.” This imports habits familiar to model researchers into application engineering, where satisfying a conventional product requirements document previously dominated.
Claude can help build both evals and the interface for grading them. The hosts contrast intimidating JSONL files with a usable left/right comparison UI, where people add ratings and comments. Those judgments become feedback Claude can consume to improve the system. Katelyn anticipates Managed Agents tooling that feels like working with Claude to create evaluations, rather than requiring a separate unfamiliar workflow.
A performance blog supplies an example, but the transcript gives differing figures: the host first says 9 of 12 targets were done in about three days, then later says 12 of 13 targets were hit in three days during a two-week effort. The discussion considers expanding the improvement list rather than treating faster implementation as a reason to invent unrelated products. The guests also enjoy publishing actual prompts because the method is often less exotic than imagined: sometimes it is simply, “Try harder, Claude.”
10. CPU demand now requires the planning once associated with accelerators
Katelyn says CPU scarcity is “as much of a problem today as the accelerator layer.” Capacity that teams once expected to obtain on demand, provided they had budget, now requires advance planning. VMs and sandboxes are part of the constraint. Her account is not confined to Anthropic: she says other people she talks to across the industry are encountering similar shortages.
Product services, APIs, applications, sandbox execution, research, and CI all draw on CPU. More generated code means more CI runs, adding demand beyond the agents’ immediate work. Research and product sandboxing do not commingle, but CPU allocation across the company remains “a zero-sum game”: Anthropic secures capacity from cloud providers and must divide that supply among competing internal needs.
The response combines procurement, more efficient sandbox infrastructure, and application efficiency without compromising tenant isolation or security. Scale and quality determine whether another supplier is worth integrating. Anthropic also runs across all three clouds, limiting reliance on a single provider’s specialized compute shape; the guest says the common foundation often has to be the most vanilla capabilities available across environments.
11. A Claude-assisted Rust port cuts the Messages API’s resource cost
Katelyn says that when she joined a little over a year earlier, the Messages API was a Python monolith in one GCP region. The team has since expanded across regions and confronted a CPU wall. It chose Rust despite initially having no Rust developer in the room. Claude would write implementation code; humans would design the distributed system and learn what they needed to operate it.
The crucial work was “the plan to do the thing, and then the verification of the thing.” The team built an entire Rust counterpart and sent shadow traffic through it, comparing outputs against Python before gradually enabling production traffic. The conversation contrasts that with the incremental, multi-year migration previously expected. A few more experienced Rust developers were hired along the way.
The guest describing the port says the Rust Messages API had fully rolled out the previous week. The reported ballpark savings are roughly 40–50% on CPU and approximately 10–20% improvement in time to first token, with greater improvements on longer-running workloads. These are operating results from the port, distinct from the separate blog example about making Claude.ai faster in a few days.
Scope discipline mattered: it was a port, while the rest of the organization kept adding API features. The migration team used static analysis and coordination to maintain parity, resisting redesign. Later opportunities include storing conversation state so clients need not repeatedly send the entire history. Prompt caching alone does not remove that transfer; long conversations may be few by user count but substantial by token volume.
12. Separating brain and hands improves durability before it improves security
A guest explains that Managed Agents could initially have been Claude Code placed inside a sandbox with an API attached. That was insufficient: callers wait for both sandbox and agent startup before receiving tokens, and a dead sandbox can kill the agent. The eventual design puts the harness on a durable server, keeps state separately, and spawns execution environments when needed.
Reliability and latency initially dominated the team’s explanation. A product manager kept insisting, “The most important property of this architecture is security.” The guest says they apologized to that colleague the previous day. The change of mind matters: separating orchestration from execution supplies security properties that were less obvious when the team primarily evaluated the architecture through product performance.
Credentials live in a separate vault. The agent sees a placeholder; when it calls a service, an egress proxy retrieves and injects the appropriate secret. This avoids putting raw credentials inside the agent’s execution environment and then trying to scrub or conceal them afterward. The guest calls the separation a “literal infrastructure wall” between the agent and those credentials.
Sandboxes add domain allowlists or completely disabled network access, constraining what agent-written code can contact. The guest emphasizes both architecture and configuration when customers ask about containment. Managed Agents packages those controls and subsequent improvements: customers receive evolving protections as Anthropic learns, rather than having to independently reconstruct every credential-handling, networking, and execution decision as the field changes.
13. Training breakouts and production deployments have different conditions
Challenged on whether sandbox escapes indicate defective technology or poor implementation, a guest distinguishes training incidents from deployed systems. Training may effectively tell a boxed-in model, “Solve an impossible task. I’m going to stress you out. Ready, set, go,” incentivizing escape attempts. Production can include online safeguards and separated infrastructure absent from that setup. The guest cautions against treating those environments as interchangeable.
The host also questions advocating “pacing” while releasing a frontier model 11 days later. The response is that pacing means doing the security and safeguards work needed to feel comfortable releasing; Opus 5.5 met those requirements. The guest still describes long-running agents as nascent, with teams figuring things out and racing to install controls, rather than presenting the surrounding engineering as finished.
14. Safety fallbacks expose the same unresolved problem as model routing
False positives can interrupt benign work. The host describes asking Fable to analyze its safety card: it reasoned for about five minutes, reached cybersecurity material, then fell back to Opus. The trigger appeared to involve the evolving reasoning and context rather than just the initial request. A user doing biology homework could likewise encounter a block despite a legitimate purpose.
The host’s objection is that a less capable classifier may misjudge work performed by a smarter model. A guest responds that classification and answering are different capabilities: recognizing danger need not require being able to carry it out. A smarter judge could help, but introduces cost and latency. The guests acknowledge the problem without claiming a definitive solution.
Angela explains that the API exposes several choices. Customers can opt into server-side fallbacks, use client-side SDK helpers to manage them, or re-prompt and continue with the same model. The apps’ fallback behavior therefore is not the only available integration. Angela would prefer a more unified experience that answers the benign part appropriately while reducing access to the problematic capability.
Angela connects this to ordinary cost routing: neither the task nor the model is perfectly understood upfront, and new complexity emerges during execution. Visible switches can make the experience convoluted and personalities inconsistent. She is more bullish on a consistent model that starts cheaply at low effort and scales effort continuously as needed. Across different providers, she says she does not know how to solve that discontinuity.
15. Long-running tasks remain an important source of unsaturated demand
Angela says intelligence appears increasingly saturated for some workloads, while other classes remain “very unsaturated”—more than people may anticipate. The emerging return comes from “extremely long-running agentic tasks” on problems that historically resisted brute force but might yield to more intelligence. She sees value both for businesses doing the work themselves and startups building agentic products around it.
Existing demand includes coding and enterprise products automating distinct processes. Some of those applications are starting to work; others still are not. Biology and cybersecurity introduce another category of hard science and engineering work, beyond straightforward process automation. Angela describes these as increasingly interesting workloads showing up on the platform, without claiming that capability or reliable execution has become universal.
16. Faster implementation leaves humans responsible for judgment and quality
Angela says that for documents and code, people should use Claude but ensure the result is high quality; passing along an unread generated artifact transfers the author’s work to the reader, which she calls unfair. Katelyn emphasizes that humans remain responsible for the quality. Consumers should send actual slop back, but avoid reflexively rejecting good work because an em dash or another “Claude-ism” reveals AI assistance. Quality and stylistic telltales are different questions.
In engineering, humans ensure system designs and plans are sound; Claude writes much of the code; verification establishes whether the result meets the bar. Katelyn says some changes need only a quick approval. That does not mean handing the production system to Claude with “Good luck”: operating and evolving large systems still requires human expertise.
The host cites the warning that new capabilities should not simply produce “10X more features into prod.” Katelyn argues that foundational product planning does not disappear because agents accelerate part of the process. The host admits the tension: when experimentation is cheap, trying everything feels attractive. The counterargument is to use that speed in service of an intentional goal and better understanding of users.
The same tension appears in management. Katelyn can spend an entire day unblocking decisions in Slack and feel she produced nothing; Angela protects time for thinking and writing, at the cost of being harder to reach. The host’s correction is that resolving those problems produces “a functioning team” that is unblocked. Claude assistance does not remove every difficult decision that reaches their inboxes.
17. Developer relations is shifting toward teaching people and their agents
Katelyn says the platform business can disappear from Anthropic’s public narrative between research, policy, Claude Code, and Cowork. Separate platform and Claude Code DevRel teams address different developer ecosystems. Platform still needs a manager, with individual contributors reporting directly to her. Its challenge includes finding a voice and identity, alongside the familiar responsibilities of product feedback, content, and community.
Lance’s deep technical teaching on memory, costs, and evals exemplifies the approach Katelyn wants; CJ contributes examples and community work. The host adds the “master builder” model, borrowed from Lego: demonstrate something impressive with the platform so developers understand what they could build. The work helps reveal possible applications, rather than merely announcing that another capability exists.
Cookbooks increasingly illustrate possibilities rather than serving as code people copy. Katelyn contrasts an elaborate older computer-use implementation with the Claude API skill, built by Lance and owned by DevRel, which helps coding agents create integrations. Angela adds that developers can point Claude at a reference architecture. Best practices increasingly reach users through the agents helping them build.
The host challenges whether engineers and product leaders should themselves be the public faces, as Boris is for Claude Code. His proposed separation is a public-facing expert supported by production staff: spend 30–60 minutes on a brain dump, then return to work while others help publish it. Katelyn sees DevRel enabling colleagues to communicate well; the host compares that support to HR helping everyone hire.
18. Multi-agent gains depend on shared state and workers that can progress independently
Angela’s closing recommendation is to revisit experiments previously constrained by cost or token efficiency. Opus 5.5’s economics and reduced verbosity might let developers get further by letting it run. She specifically recommends trying a multi-agent setup, while declining to supply a narrow recipe for the most promising application. She wants to see what people build with the additional capability.
The closing advice is to have one lead coordinate work that can be parallelized, with shared context and asynchronous sub-agents that do not block each other. The guests’ two architectural priorities are shared state and independent progress. Subject to the problem being suitable, they expect gains; beyond those constraints, the advice is to experiment and “let it rip” rather than overprescribe every role.
The hosts distinguish assigning agents fixed jobs from agents assigning work among themselves. A guest expects smarter Claude models to improve at splitting, cloning, and adapting their own arrangements. The guest “wouldn’t be surprised” if experimentation yielded interesting results on problems that previously seemed intractable—perhaps solving something “in maybe one shot or something.” The language remains an invitation to test, not a performance guarantee.
Full transcript
Yeah. Maybe a distinction I would make is that a lot of the incidents people are hearing about and reading about are happening to models—not just ours, but everybody’s models—that are in training. The way you run these models is you put them in a box and say, “Solve an impossible task. I’m going to stress you out. Ready, set, go.” You can only do what you can do within your environment, and the agent is basically incentivized to try to get out of its box and solve the problem.
When people are running production workloads on our API or anybody else’s API, they’re running in an environment with safeguards that aren’t necessarily running in training, where the whole point is to train the model. In production workloads, first of all, you have those online safeguards, which is really great. Second, you have more architecture and infrastructure around what’s happening. The agent isn’t just in a box trying to get out. The harness is over here, the hands are over there, and it’s just not even a problem you would have to contemplate.
Hi. Today we’re trying something new. We’re launching coverage of Anthropic for the IPO. They just filed their S-1 today, so we’re releasing a series of podcasts leading up to the IPO, which is expected in 2 weeks at a $2 trillion valuation. It’s one of the fastest-growing companies of all time, with a reported lifetime cash raise of over $100 billion and a lifetime burn of less than $10 billion.
There are a lot of efficiency stories to dive into, and I think that’s something we want to cover: how Anthropic is simultaneously shipping the strongest models in the world while doing it very efficiently and building an incredible business. Many of you know I have a former life as a financial analyst. I pulled out some statements from the S-1 that I thought would be interesting context for today’s podcast. We’re trying this as something new because I feel like some of you want that personal relationship with us talking directly to camera, rather than talking to our guests, which we’ve done for 4 years.
The first thing I want to highlight is that Anthropic is an extraordinarily efficient company by human count. That may be obvious because it’s an AI company, so obviously they’re using AI very well. But I think the degree to which they’ve changed as a company over the past year is not well understood. Trailing 12-month revenue per employee is now $6.6 million, compared with $1.4 million a year ago. They’ve grown revenue per employee almost 5 times.
I think that’s something we need to dive into more, because any business owner or engineer wants to increase revenue per employee—not just to increase profitability, but also to increase how much you can pay your people. This is obviously reflected in a significant increase in revenue per dollar of compute. It’s notable that in Q2 2025, Anthropic was basically losing $0.50 for every $1 of revenue, and now it’s making $2 for every $1 in cost. That’s an interesting analogy for how enormous the inflection in Anthropic’s revenue and business outlook has been over the past year. This year was extremely critical for Anthropic to execute well, and it has done that. We’re basically going to dive deeper into this and tell that story as much as we can from the podcast’s external point of view.
One consequence of this increase is that Anthropic has just reported its first profitable quarter—not adjusted profit, which is profit after we take out all the things that cost us money. No, this is generally accepted accounting principles, or GAAP, profit of 19%, and that is enormous, completely flipping from a year ago. More than flipping: a year ago, it was losing 129% of every revenue dollar that came in.
I think the primary attribution that explains this extraordinary efficiency is that, as you know, Claude loves loops, Boris loves loops, and Dario loves loops. There are 2 loops going on at Anthropic. The first is the improvement loop: Claude leads to better models, better models lead to more use cases, and more use cases generate more feedback, which leads to a better Claude. Then there’s the investment loop: revenue from expanding adoption leads to higher growth, higher growth leads to more investment, and more investment leads to more capabilities. That leads to a better Claude.
These 2 loops spin together and form an infinite loop that is driving the increase in efficiency until the end of time, according to Anthropic’s S-1. That’s our transition from the overall efficiency story to how they execute it. Today we’re going to focus a lot on the Claude Platform, because that is primarily one of the 3 things they call the “diffusion of Claude” in the Anthropic S-1. They call the Claude Platform “our comprehensive agentic development environment.” They provide customers with direct access to the primitives to build, deploy, and scale their own applications that leverage Claude’s capabilities within their products and workflows.
One thing that comes up a lot in this podcast is the concept of dogfooding, where a lot of Claude’s products are actually built with the same Claude Platform that’s available to you. Something else that’s interesting about the Claude Platform is that, as disclosed, over 50% of their customers with $100,000 or more in revenue on a trailing 12-month basis use it. That shows an increase in adoption and the emergence of Anthropic not only as a model lab, but also as a hyperscaler and a cloud platform. The name Claude Platform may make that inherent, but it’s going from theory to actual practice, and that’s something we’re focusing on in this podcast.
The other thing I really try to emphasize, which I’ve talked about before, is that Claude Code and Claude Cowork both use the same primitives that Claude Platform exposes. I think that’s really interesting. Ultimately, it looks like Anthropic is positioning its IPO around going from chatbots to copilots to virtual collaborators. They use this term instead of “agents”; maybe it’s friendlier, but it’s a lot of the same things you look for from agents: long-running autonomous work, judgment, self-correction, iteration, and tool access. It’s about owning a goal and working alongside you, instead of working inside one app or task.
That’s the context I want to provide for today’s podcast. This was recorded when Opus 5.5 launched, and overwhelmingly, among all the high-taste testers we talked to, Opus 5.5 is the best model in the world right now. Anthropic is even sitting on Fable 5.5 and will probably release it in the next 2 weeks. I don’t have insider information, but it makes a lot of sense for the IPO. Let’s look forward to that and learn a little bit more about how they build Opus 5.5.
All right. We’re here in the studio the day after Opus 5.5 with Katelyn and Angela. Welcome.
Thanks for having us.
Thanks for having us.
Congrats on launching the top model in the world right now.
Fair. I would say it depends who you ask.
Depends who you ask. What is the main takeaway personally for you guys? You have your official blog posts, and the pricing cuts are really cool. What do you take away when you launch a new model like this?
We’re super proud of this one. The team has put in a ton of work to make this happen, and I think we’ve really heard the feedback that Opus 5’s writing is kind of convoluted. We really tried to clean that up, and I’m really glad to see that it seems to be resonating with people. Internally, when we tried out the model, we felt the same way, so it’s great to get that validation.
To your point about the pricing cuts, there was a lot there. Increasingly, people are looking for good intelligence per dollar, so making sure we get the per-dollar part right has been really important.
There are some things on the edges that we haven’t really talked about in the blog post. We did a lot of work to make our API harder to distill from, and there have been changes to make that a little more concrete. These are things like how we do prefix-lock thinking or how you can replay thinking tokens across different types of models.
We’ve put a lot of work into adjusting things for valid use cases where you might want to do this, potentially in your compaction strategies and things like that, versus scenarios where the use case is actually from a bad actor. There’s been a lot of threading the needle on those fronts, and I’m glad to see a lot of that called out as well—in addition to shipping a model with good intelligence for your dollar and communicating a little more clearly.
Yeah. Specifically, you’re talking about the paper “Stealing Reasoning Traces from Proprietary LLMs.”
Yes, that paper. That one.
Right? They did— I mean, obviously, they did responsible disclosure with you guys.
They did super responsible disclosure, and we’re super thankful they came to us.
For those who haven’t heard about it, it’s really cool that you could do this. It’s patched, but what was it? Did it cause a freak-out at Anthropic? It probably wasn’t that big a deal. I don’t know.
I don’t know how much this paper specifically caused a freak-out, because this is something we knew was a gap.
There are 2 separate things: Chinese model distillation, and then this was a specific technique that was published and got a lot of attention.
Yeah, exactly. We’ve known for a long time that chain-of-thought extraction is one of the most powerful ways to distill models. We’ve also explored how to preserve thinking blocks across turns and make that more required.
We’ve always had a really hard trade-off between user experience—changes that might be painful for API users—and protection against distillation. We have to figure out how to walk that line. To take a step back, the reason we don’t want distillation is that Anthropic exists to help AI create positive outcomes in the world safely. We run a lot of safeguard classifiers on the tokens we produce to make sure they’re safe. If someone distills our models and runs them without safeguard classifiers, there could be bad outcomes in the world, and we don’t want that to happen. We don’t want people distilling the models to use them for malicious or bad reasons.
It’s always an interesting trade-off. We don’t want to make it harder for people who are using the models for good to get the intelligence out of them, but we have to make some trade-offs to prevent distillation. We’d been working for a while on how to prevent chain-of-thought extraction and preserve thinking blocks without making things too painful for people.
We think about user experience hits in 2 layers. One is that you have to make an integration change, which is annoying, but you can still accomplish everything you could before. The other is that you actually can’t accomplish the same things you could previously, and that’s what we’re really trying to avoid.
With the last couple of models, we made a few incremental changes to enforce preserving thinking blocks across turns. Now we’re also looking at how to do that across multiple accounts, so that you can’t take advantage of chain-of-thought extraction with the models. We shipped a couple of additional features to help with that.
We worked with a lot of open-source harness maintainers. We reached out and talked to them, and said, “You’re going to have to make these upgrades within your harnesses. We want to make sure you have a good experience.” We also worked with our big customers building harnesses on top of the API.
For some of them, their compaction strategies would have been broken by the need to preserve these thinking blocks. So we shipped a new compaction feature in the API to replace what they had been doing manually, while still working with preserved thinking. We’re walking this line, but we didn’t just say, “Now you have to preserve these thinking blocks in this exact way,” because we would have broken a lot of people. We took the time to work with customers and open-source harness maintainers and give them what they needed to make that upgrade.
While we’re on the topic, anything on the other side of distillation—the other half?
Yeah. We’ve seen that it’s harder to distill, based on the measures we’re able to look at. It’s been significantly harder. Credit to the researchers here: they disclosed it very responsibly.
They published the results publicly, and I know it trended quite a bit on Twitter, so a lot of people could use it. But in the areas we’re able to measure, it does seem to have reduced the effects we don’t want.
The bit I’d add to what Katelyn just said is that I see a lot on Twitter, or X, or whatever you want to call it, about this being a commercialization problem. If you can steal a model, you can run it cheaply, and so on. That’s true, but it’s not something we care about very much. We care much more about this from a safety point of view than anything else, so that’s the angle we tend to think about, rather than the commercial angle. Realistically, if you distill from us, you actually pay us, so commercially, we make the money. It’s not really that kind of an impact. It’s more about making sure our models—and Claude—are good. Claude could be derived, and if that ended up having a negative impact, that would be unfortunate.
And generally, also on Phi-5, since you guys are fresh off this: it’s fun to establish the dynamic between you two, because you also do a lot of appearances together. You’re head of product and head of engineering. How does that work between the two of you, especially for a model launch? Take that as an example.
1. Product And Engineering At Scale
For model launches, our product and engineering teams work together very closely. On my side, a lot of it is about the trade-offs we need to make between launching something with the right intelligence bar and the cost of preservation. Our updated cache costs are an example of us wanting to encourage integration changes that are beneficial for everyone, and to pass those savings on.
A lot of these decisions are about how we balance aspects of our safety work while making sure we have an awesome model people can take advantage of. The qualities and surrounding components of how we launch the model tend to be more on the product side. From an integration-experience standpoint, a lot comes directly from the engineering team.
If you’re talking about the API experience, that’s very tightly integrated. It’s been awesome that we have a lot of engineers who are deeply product-minded, and we love that. On my side, we also have product managers who are much more technical. We want people with a blended, interdisciplinary skill set, because they can analyze things from different angles and figure out how to do them best.
I tend to lean a little more toward commercial impact. I’d frame it as the engineers caring much more about the integration experience. They take more of a UX angle, where the user experience is about the API and developer tools.
It’s fun to be a product engineer who works on developer products and platforms.
Yeah, developer platforms.
I’d echo that. We have a lot of engineers who love thinking about the product, the user experience, and the developer experience. We have that dynamic between us, too. I have a lot of opinions and can be annoying with our product team about how we think about these things. Our product team is also really technical, and they can get into the weeds of decisions we’re making for scale or whatever it might be. It’s a fun dynamic our team has.
Katelyn and I have worked together for a long time, at Stripe, and then I guess we didn’t work together at OpenAI.
Separate labs.
Just separate labs. Then I came over. One interesting thing about both of us is that Katelyn frequently gets mistaken for a product manager, and I frequently get mistaken for someone in engineering, even though those aren’t actually our job functions. I think that pairing helps us make decisions that are consistent—
Yeah.
Across both sides.
I think I come across a lot of companies with a similar product leader, engineering leader setup. As software engineering gets solved, maybe those 2 roles merge?
I kind of think so.
Yeah, I think everyone tends to lean toward that. But if you guys are still separate, then we’re not there yet, because—
We’re definitely not there yet. So why? What’s left? I don’t know if I’d ever use language quite that strong as “software engineering getting solved.” The way I tend to think about these things is that ambitions will continue to elevate.
A good example is that a lot of what our team is spending time on right now is, of course, launching our model safely. That’s the most important thing we’re doing, because of all the work we have to do on security with agents, offline safeguards, online safeguards, and everything else. But the other half of what we’re doing is trying to scale.
When I joined Anthropic a little over a year ago, we had the Messages API. It was a Python monolith, and we were running it in a single GCP region. Everything was fine, and none of my friends and family knew what Claude was when I was going to take this job. Now we’re in an extremely different place.
The metrics are really wild in terms of how much volume we’ve seen and had to support on the API. CPU is really constrained, which has been a really interesting challenge for us. We’ve done a lot of work on scaling these systems, and that takes a lot of distributed systems expertise, along with expertise in the languages and frameworks the models know. But when it comes to operating and evolving a system at this scale, we’re nowhere near a place where it’s “Good luck, Claude. Have fun, and make all the changes you need to make to the system, roll them out safely, and operate this thing at scale.” There’s still a lot of work for humans to do, and I imagine that will continue for a while.
I’m curious how that’s changed. A while ago, when Fable first came out, we had Glasswing, and there was a whole delay and change in how models started to get released. We saw this with most frontier models: there’s safety testing. What’s the status of things now? How do you see it? How do you determine that? There’s a new Sonnet and Haiku coming, and I assume those will just come out. But for something like Opus 5.5 or a new Fable, what’s the current status?
2. Model Release Safety Policies
I’d say this starts with what our policy decisions need to be around safeguards and safety, both offline and online. There’s been a lot of discussion lately around data retention and what we’re doing in order to release these models safely. You can have many accounts across a large network of people who are choosing to spin up accounts and use the models in various ways. There’s no way that, online, as tokens are flowing through the system, we’d be able to catch that something malicious is happening with the models, or that distillation is happening, or whatever else might be going on.
So we’ve had to develop data retention policies so we can do automated review and human review, as needed, to make sure that good things are happening with the models. With each model we release, that’s one example of a policy decision we need to make: what will the policy be for this specific model? Opus 5.5, which launched yesterday, for example, doesn’t have the data retention requirements that Fable has. That’s a decision we have to make each time we launch one of these models, and if it’s going to have those policies, we have to do the work to make sure they’re enforced.
Similarly, we have programs like our cyber verification programs and, newly, our life sciences verification program. They affect how the online safeguards work for the model you’re using, so making sure those programs are in place is another part of the work. That’s a flavor of the work we do as part of each model launch. These aren’t things we used to have to think about when we launched models. Now, the majority of the work is about preventing distillation and managing online safeguards, offline safeguards, and policies.
A great example is the system we built to make sure authorization to use a model, or one of these programs that may offer relaxed safeguards, threads all the way down through our system in a way we can really trust. It’s not like, “Oh, you got access to Mythos, so we’ll give you an API key, flip a feature flag, and call it a day.” We have a robust system for authorization, to make sure that if you’re supposed to be using that model, only you are using it in the way you’re meant to. A lot of complex things are starting to happen around model launches that we never used to have to think about.
That’s good.
Yeah, and it’s scaling up with capability. As our capabilities get more advanced and start to spike in specific areas, that’s where we need to pay extra attention. A lot of that evaluation comes from those very particular capabilities that we might see as offense-dominant or defense-dominant, and that could lead us to take different strategies.
As we work all of that into the model launch, there will be components where, as Angela mentioned, we don’t exactly know how people might use those capabilities. Some are more obvious, like coding: go make awesome products and ship more things. That can be really useful. Other areas, like biology and cybersecurity, can be dual-use and have negative effects. But you could also be a valid cybersecurity researcher trying to find vulnerabilities and act in good faith. Similarly, in biology, you might be doing research. Programs like the one you have up here are meant to provide access to those dual-use capabilities. You have to specify your use case and things like that, but we do want to enable the community to take advantage of these capabilities and see what amazing things they can do with them.
We’ll put up the links, and maybe we’ll dive into them later. There’s also one for biology.
Mm-hmm.
I also think, at a high level, it’s really important that you guys actually think about this stuff. People forget that Fable was rolled back, and there’s all this distillation. People say, “Zero data retention isn’t a thing on Fable. Crazy.” But you have to remember that it got pulled back. There are distillation attacks. It’s good that you’re thinking it through and doing it the right way while releasing it.
Yeah, and a lot of that is about building the right framework in our system to be able to do that. I talked a little bit about this idea of authorization and trust grants that we’ve built throughout our system now.
Never say never, right? Not as precedent.
Yeah, but it’s also a very common thing in platform engineering: you have one incident, and then you overprepare for that incident, and the incident never happens again.
Yeah. Oh, yeah. It’s true.
It’s true. Like, now you have the button.
Well, it’s true, and it’s funny you say that, because with everything going on, we had a line item in the work our team was doing: “We’re going to put this system in place so it could be a single button to pull back for everybody.”
Yeah.
It’s funny, because it’s like, do we justify doing this work? We hope something like that never happens again. But in this case, it’s really a matter of building up those same primitives that we wanted to put in place everywhere to help us manage these verification programs and access to models like Mythos. We can put that same thing below everybody’s model access so that, in case we ever need to, we have the right controls in place.
Yeah. Decisions like this have actually proven to be—well, we’ll see about this one in particular—but examples like this have consistently been the way we’ve had to transform the way we develop our roadmap. In traditional product management, you can say, “These are the things I hypothesize will happen, and they’ll have a certain sequence, a certain ordering.” But you’re picking bets in a much more concrete way, versus the deployment of Fable, for example.
I don’t think any one of us could have guessed that would be a valid use case—that it would happen with the government, and that we’d have to go do it. But clearly, the probability is nonzero.
From this way of thinking, product management has become deeply probabilistic. Should we prioritize this? What are the chances it will ever happen again? When we consider the craziest possibilities, we realize we have very little certainty about where things will go. So we need to hedge and build a portfolio of bets.
More often than not, we’ll hedge by saying, “We don’t think this will happen again, but if it does, we want to have that capability.” We’ll prioritize things like that. And increasingly, we find that by taking this approach to product development, we end up using the things we build.
Is there another, separate example, just so I can understand?
The access program is a really good example. We developed it specifically so we could offer programs like this to customers.
Do you want to give a high-level overview of what the access program is?
Sure. If you want dual-use access—let’s take a model like Opus, for example—and you want to use its advanced cyber capabilities for research or something else, as a defensive user, you can apply to this program with your use case. If you’re approved, we’ll reduce the safeguards on your version of the model so you can use those cyber capabilities for your specific use case.
That requires understanding each use case and limiting access to a particular user. We wouldn’t do it generally; you have to apply. Even for a program like this, we designed a lot of the primitives Katelyn was talking about to handle it.
It surprised us how useful those primitives became. When we were designing the program, we asked ourselves how generic it should be. Should we focus on building a verification portal and granting very specific model capabilities? We opted for a more generic approach because we didn’t know what else we might use it for.
It ended up being massively useful for things like broader model access, managing our trusted access programs, and handling model deployments. We were very generic about how we handled each primitive. From a normal product standpoint, I might say, “That probably doesn’t make a ton of sense. We should focus on the use cases that make sense.” But by choosing a more generic approach, we find ourselves reaching for that solution more often than not, and it pays increasing dividends.
It’s something you needed to have anyway, and you just had a forcing function to build it.
On a technical basis, I’m from the AWS world, right? IAM, OIDC, and then people who get really intense about this talk about Zanzibar and all these other technical architectures. I’m not familiar with the term “trust grants.” Is there a difference between normal OIDC and trust grants?
I mean, not really. It’s a form of authorization.
Yeah, it’s IAM, but Anthropic’s IAM.
Yeah, kind of. Really, we have authorization for when you make a request to an API that accesses some data, and we make sure it’s your data that you’re accessing. That’s very vanilla authorization. We put a data model in place that’s very clean. A trust grant means you’ve been given access to a specific model or program. There are a bunch of different permutations of that, and we find ourselves reaching for it all the time.
Another good example was when we first rolled out preserved thinking for Fable 5.1. We grandfathered in some people and gave them a grace period to make the integration changes. As part of that, we had to track which existing customers needed extra time. We created a trust grant, and then we could run a service on the hot path of our requests to check whether you had a grant for the thing you were trying to do.
It’s really no different from vanilla authorization, but we had to build a system specific to our needs and where we needed to insert those checks.
Yeah, correct. These are all engineering things people take for granted and don’t really think about. What’s another example of platform work that had to happen? Maybe serving Claude Code or serving other customers. I talk a lot with Cognition, but in terms of a model launch, we’ll talk more about the safety work that’s happening. I just want to go at a high level, while it’s fresh in your memory: what goes into a model launch?
3. How Anthropic Launches Models
It’s become a bit of a machine these days. Minus the safety work, which I think is the more complicated part, there’s a lot of getting feedback from our customers. We make the model, do a lot of safety testing, and evaluate where we think it needs to be. We make the necessary trade-offs and decisions, and then we test it.
We’ll cut various versions of the model and evaluate them. We do a lot of internal dogfooding, which is what I call it, and that always really helps. We catch early issues, like when something might not be the best. We also use Claude a lot internally, so we can see early signals and figure out which variants might be better than others.
When we feel confident about a set of variants, we cut a version for our customers. Sometimes that goes through our first-party products, like Claude Code or Cowork. In other cases, it goes through our API customers, and we’ll effectively run an early access program with them. They’ll need to make changes to their harness. We’ve engaged with a lot of customers, like Cognition and Hers, and we have a process for handling that so we get the feedback we need. Does it perform well in your evals? Do you like it? What don’t you like? Some things are vibes-based, and we like to hear those things too. Other things are straightforward evals: did it do well on the things you care about? We take all of that together, and once we feel good about those pieces, we launch the model.
The state of model feedback still feels very unstructured. Basically, it’s always a Slack channel between companies: “Hey, this thing seems weird.” Then people are like, “Okay, I’ll fix it, I guess. Give me an ID or something.”
Yeah, it’s very—
Can we do better?
I mean, it’s the nature of a product that’s nondeterministic, right? But to your point, the only customers who are part of our early access programs are customers with really robust evals. A lot of this is vibes-based, and the conversations tend to be vibes-based and on the fly. Part of that is because we’re racing. We want to minimize the time between cutting a version of the model for testing and giving everyone access to it.
So we work with customers we know will kick off their whole eval suite and come back with actionable feedback as quickly as possible, alongside the vibes-based feedback. There are definitely process improvements we could make over time. But you’d be surprised—or you probably wouldn’t be surprised, but some people would—by how few products in the world are agentic and don’t have good evals to be able to—
Oh, we had—
To help test these models.
We had Boris on the pod two years ago saying, “We don’t have evals for Claude Code.”
Yeah, I mean, that’s changed. That is true; that’s changed. But a lot of things are like that: your evals are your users, and you don’t have time to formalize them, or you don’t have the experience, or whatever. And that’s fine. At some point, when you get serious, you’ll have evals.
Yeah, totally. Exactly. And I think the good thing is you guys do a lot of dogfooding, as you say. It’s commonly said that you find a lot of this by testing it yourself before it comes out, right?
Yeah. API customers in particular are very useful because they’ll put Claude into a different harness. We want Claude to be very general, so its ability to work in other harnesses is quite important to us. That style of feedback tends to be exceptionally useful because it’s novel compared with what we would have gotten from dogfooding.
What kind of feedback helps? Is it regressions?
We have regressions here; we notice good attributes here.
Mm-hmm.
The good stuff is always very validating. We’re frequently not surprised, but we like to be validated on some of the things we might hypothesize. Honestly, the most useful feedback is negative: it didn’t do well at this, it didn’t do well at that. We love that feedback, and then we’re like, “Great, how do we fix it?”
It tends to be some of these things that are really hard, because not everyone wants Claude to land at the exact same spot on every spectrum. A good example is laziness versus proactivity and taking action. Sometimes people are like, “Ugh, just finish my task. Stop being lazy. Do better.” Other times, they’re like, “Hey, we were just feeling this out and planning, and then you went and wrote a bunch of code or took a whole bunch of action that I wasn’t ready for.”
Getting that exactly right is literally impossible because everybody wants something different. But we’ve had situations where we cut an early version of the model and found too much laziness, so we put it back in the oven and tried to train that out. Those are the sorts of things we might not have caught if all we were doing was internally dogfooding within our products and API. It’s way more important for us to get it out in the API, have all our other customers run with it, and let us know what we need to change.
Yeah, it’s a core question. It’s probably one of the hardest things about PMing for a model: alignment versus personalization. There’s a default personality, a default behavior that you want for all consumers, and then you want to be able to respond quickly to someone’s CLAUDE.md saying, “No, no, no, please be more proactive than the default.”
To my knowledge, and I was involved in Frontier Code[?]—we were involved in the Fable[?] launch, which put Frontier Code on the map—and now with Opus 5.5, we have it back on the model benchmark. That’s a very big part of Frontier Code, right? The reason the model curve looks weird for Frontier Code is that it penalizes too much activity or too many files, which is good. That was a whole design purpose of Frontier Code. I could easily just say in my CLAUDE.md, “Don’t do that,” and then it won’t do it.
Am I penalizing Claude Opus 5.5 unfairly because my benchmark doesn’t have that subjective personalization element? I don’t know what the question is, but how do you deal with personalization? A model can do so many different things, and it depends on the person; everyone has their own CLAUDE.md. How do you evaluate that?
We don’t do a great job of it. No, seriously, it’s really hard. It’s super hard, and the preferences compete and contrast.
Yeah. Look at chat. It’s horrible. I thought there was the Claude Constitution—
That’s old.
Yeah, yeah, yeah. There is a constitution, but I’ll let you guys answer it.
The constitution is helpful, but personalizing to instructions and preferences—which are very different across people—is very hard for us to do. We generally try to solve this by having more parameters that allow us to tweak things per customer base. If we don’t control your customer base and you’re an API customer, then we try to give you recommendations, and you do that tweaking.
We try to provide controls around things like level of effort: how much effort you should put in, or maybe the size of the model you might care about. But that’s not quite at the level of personalization we’d need for different CLAUDE.md files, or now AGENTS.md files, or whatever. It’s a very hard problem. Beyond providing multiple parameters, we try to account for conflicting tastes across the feedback we get and make a judgment call.
As you scale up to more customers and users, that’s not a very scalable way to do things. We’ll need ways for Claude to take in your personalization, maybe through a signal you can provide or something you can tune. We’ll have to take a route that’s much more oriented in that direction. For now, we try to make judgment calls on a lot of this, and it’s pretty tough. It is what it is.
A small note there: I guess the default effort reasoning changed. It’s now medium. That’s one of those things that probably affects the average customer who doesn’t use the toggle.
Yeah, it’s like a Claude Code setting, right? It’s a Claude Code setting. They defaulted it to medium.
Yeah. You can turn it on, right?
This is a perfect example of a parameter we use to customize things for different subsegments. The default setting will adjust depending on whether you’re a free customer, a prosumer customer, using Claude Code versus Cowork, an enterprise customer, or an API customer. Those are light ways we try to customize things, knowing you can always adjust them.
One thing we’re hesitant to do is make people feel like they can’t control their customization of Claude. We could take a heavier hand, but we want to give users the ability to experience Claude’s intelligence levels as they see fit, especially if they want to go higher, or sometimes lower if that’s faster for their use case. That’s a hard product trade-off. We tend to lean toward gentle defaults but give you options to adjust. That works okay for now, but I imagine we’ll learn how to do it better over time. We’re not super happy with any of this yet.
When we launch a model, one of the big things we have to do is figure out that Claude got good at something, and sometimes that’s something we should expose as a parameter on the API for you to control. But over time, we’ve ended up with this funny mishmash: there’s effort, there are budgets, and there are all these things you can set on the API. How could anyone figure out the right combination for a given task? We’re figuring it out as we go and with customers: which parameters we deprecate, which we default better, and which should be on a 1-to-100 scale versus a low, medium, high scale.
We’ve trained Claude to get really good at understanding a whole spectrum of options for some of these things. But do we expose the entirety of those millions and billions of permutations? You wouldn’t get very far. That’s part of the value proposition of something like Claude Managed Agents: it’s still programmatic access to Claude through an API, but you get a harness out of the box. Every time we launch the model, we do a lot of the work to figure out the new parameters, the breaking changes, and how to get the best out of Claude within our harness, so people using it get that out of the box.
You don’t have to think about it. For people who are drawn to that—
Yeah, and there’s a significant and growing population. We actually did a whole CMA workshop at AI Engineer, which I thought was really well done.
Is there a separate feed people can subscribe to for updates on CMA? Like a Twitter feed or newsletter?
I don’t know.
Like a changelog?
No, I think our Claude Dev channel is where we are—
That’s it? Okay.
Yeah. We’re pretty loud about it, because it is—
Yeah, that’s it. It’s Claude Dev, an interesting option for people.
It’s Claude Dev.
Okay. I just want to let people know that if that’s your way of consuming Claude, this is the path. I think it’s also pretty underrated for people who aren’t consuming Claude this way. What is Claude Managed Agents, and why should people use it?
Oh, yeah, we should probably do a little intro.
Yeah.
I don’t know. That’s Ami’s house. Ami will talk about it, but—
A little two-sentence thing. I think a lot of people glanced over it when it came out, right? Everyone was in this phase of, “I need to do harness engineering. I need to control this.” You hear all that, and now it’s like, do you need to do all that, or can you just use Managed Agents and skip setting up a lot of infrastructure? Give us the quick three.
4. Claude Managed Agents And Memory
Yeah, it depends on your use case. Claude Managed Agents is our API today. We have the Messages API, which is essentially raw token access. Then we have Claude Managed Agents, a suite of APIs that lets you define an agent and start a session with that agent. It’s a higher-order abstraction that lets you skip setting up all the infrastructure, building your harness, and doing a lot of that work. It’s a much more out-of-the-box integration for people who want to programmatically run agents.
Does CMA share infrastructure with Claude Code? The sequencing of it seemed very much like, “Oh—”
Yeah, yes.
Claude Code.
A lot of it. One of the big things we do as the platform team is run our APIs and all of our foundational product infrastructure and services. For example, our team runs the sandboxing fleet that backs both Claude Managed Agents and products like Claude Code. But at the harness layer, those 2 products have differences in what we might want to do, so they don’t actually share a harness. They share a lot of the infrastructure that powers execution and the APIs.
I think that’s really—people should use this more. I think everyone basically had copycat clones of Managed Agents after you guys launched it. Imitation is a sincere form of flattery.
Since we’re on this topic, we’ll briefly mention all the memory stuff that’s been going on. I feel like it was a big part of Anthropic’s work in the first half of the year. Has it stalled out? Is it still ongoing? What’s the emphasis now?
Managed Agents specifically?
Memory, dreaming.
Oh, it is a big part of what we’re doing. We launched Managed Agents as a platform for people who didn’t want to do all these things themselves, or who wanted to offload the work of getting the best out of Claude to our team, which is evaluating all these things and making sure the harness is powerful. Memory and dreaming are about making sure your agent can continuously improve over time, and all of that is still an important focus for our team.
But, to the point we talked about earlier, launching models has become harder and harder from a safety perspective. I think most of our team’s energy is going less into, “Here’s a fun feature to hill-climb on this agent,” and much more into, “How do we bring safety into all these things?” The most important thing we can do is get the power of the model out there safely. A lot of our energy is going into taking these same concepts and giving you an agent system that’s powerful and safe.
I think compared to earlier this year, we’ve slowed down on skills, dreaming, and memory—these moment-in-time injections.
So you dropped your skills?
No, no, we love skills. We’re making updates. I’m not saying we’re not doing anything, but we’ve slowed down a little comparatively on some of the other efforts.
We’re very bullish on people doing some level of hill-climbing at the harness layer, as we discussed earlier. We do a lot of this ourselves, but the harness we provide is intentionally simple. We don’t want to be overly opinionated; we want to make sure best practices get merged in. There are undifferentiated pieces of the harness and agentic loop that are hard to string together, but you can also handle them as boilerplate.
Once you have that base layer, there are hill-climbing strategies we construct, and dreaming is one of them. Dreaming is a multi-agent system that we hill-climbed on top of CMA. It’s CMA on top of itself, basically.
CMA on top of itself.
It spins up another—
Yeah.
The way dreaming works is that we spin up another managed agent that can inspect the previous managed agent and then write memories to it. It’s managed agents all the way down. That’s a bit by design, because we envisioned that you could put together a lot of these modular components to do something interesting. Dreaming is just one example, one of the more interesting things we were going after.
There’s a lot of other stuff you can do. You can do best-of-N as a multi-agent strategy. You can do a verifier agent to check a goal or outcome. There’s a bunch of other stuff, and we’ve seen people do some pretty wild things. There’s been a harness that shares memory across everything, and they spin up tons of parallel agents. We thought, “We should probably just enable you to do that.”
Just as we built dreaming with managed agents all the way down, there’s an update we’ll make relatively soon where we’re updating the underlying primitive. We’ll expose it to let users construct harness strategies—or whatever you want to call them: multi-agent strategies.
Multi-agent strategies.
They’ll let you build your own version of this. We’ll give you templates, like, “This is how we started with dreaming.” Maybe there’s an update you’d want to make as a developer that would add more value for you.
Yeah, I think that’s a really good framing. It’s less that we’re slowing down on this sort of thing and more that we’ve got the primitives in place. We can let people take those and run with them, while giving them tools and examples of cool multi-agent strategies we’ve put into Claude Managed Agents to make it powerful. Now you can use Claude to take advantage of those primitives and make it even more powerful yourself—create new multi-agent strategies and whatever else you might want. The community can run with it while our team focuses on making sure you can access the models within that system safely.
One thing you brought up was that you run a lot of evals on this stuff.
Mm-hmm.
It’s a pretty hot topic right now. Any advice or best practices?
For evals specifically?
Evals on harnesses and agents.
5. Building Better Agent Evals
There are a lot of different types of evals, and not all evals are made equal for different situations and things you’re trying to test. You have static, deterministic stuff, and then LLM-as-a-judge. You have all these different strategies. What we’ve found from most people is that you probably need more rather than less.
There isn’t one perfect set of evals you can put together that will give you all the answers you need to make your system better. People end up needing a relatively large suite of these things, and you can have your agents work with you to evolve your eval suite over time. Underinvesting is probably the biggest mistake we’ve seen.
Yeah, and we start simple. I think some people go a little crazy. They look at the whole menu of options and think, “Wow, this is so crazy.”
“Let me try this and that, and what’s the best one?” While they do all those things, the number of evals they actually have is zero. So you might as well do something simple. It will help.
Yeah. I think some people are inexperienced, they’re not trained to do this, and they don’t know what good looks like.
Totally. I think it would be really helpful for you guys to publish more examples of that. Something I tell people a lot, which I’m doing with more Fable and Astro-class models, is that we’re treating engineering like research. When you treat engineering like research, you naturally do more evals, more labeling, and more scaling of runs, or prototyping and scaling of runs. You’re not training a model; you’re building the harness, the product, whatever it is. But you’re approaching it like research: defining upfront what your goal is, trying to formalize it as an eval, and then trying to identify why it’s failing. Sometimes your eval is wrong.
Totally. Actually, a lot of the time your eval is wrong.
Yes.
Sometimes your environment’s wrong, and sometimes the model’s wrong.
Yeah. And actually, I think that’s a big part of what AI engineering is becoming: doing all this stuff.
Completely agree.
We’ll continue to launch more features within Claude Managed Agents to give people more tooling to do this sort of stuff out of the box. You’ll probably find from us that you’ll get something like, “Here’s a product that will help you create evals for your managed agents,” or whatever it might be. It’ll feel very much like working with Claude to do that.
Right. That’s the best way to do it. Claude’s actually really good at creating evals to evaluate agents.
And eval UIs.
Yeah, exactly. People blank out when they look at JSONL files. But a pretty eval UI where you’re going left, right, left, right, and doing what the human graders, data people, and labelers would do anyway—
Yes, yes, yes.
—you can do that yourself. Just throw Fable at it.
Yeah.
It’ll materialize, but also write comments, because it outputs all these ratings. It’ll give you an Elo, give you whatever, and Claude can consume that and then work on itself. That’s the feedback it needs, and people just don’t know they can prompt for an eval.
I love that you said that. I think it’s a very AGI-pilled way to do it, because it also allows you to update and edit that system very systematically. As new models improve and things change, you can inspect where your system is, and it’ll evolve with you.
Swap out.
Exactly. Basically, I think it’s like how we talked about product and engineering merging. Application-layer engineering and model research are also merging, and the techniques are being shared. Model people are very used to doing evals. Application people know the only thing you care about is whether it fits the PRD.
Yeah, it’s a whole different skill set.
It’s a whole new skill set, for sure.
I think you guys put out a pretty cool blog post on this today. It’s somewhat different: it’s about making the UI a lot faster. But to your point about treating it like research, you define a goal—“We want stuff to load faster”—and then question things. What can we do instead of wall-clock timing? There are some fun quotes in there. It’s nice to see how the actual researchers or engineers prompted Claude. They had 12 things and said, “You have 2 weeks.” It did 9 out of 12 in about 3 days. It’s a fun blog post for people to read through.
I love it when we share our actual prompts. Sometimes people are like, “Oh God, they must have crazy, unique ways of prompting,” and it’s just like, “Try harder, Claude.”
Thank you. Though my only feedback is that it should link to the workbench, which is no longer a thing. I used to love the workbench.
That is good feedback.
Yeah, that’s a good idea.
Okay, good feedback. No, yeah, make everything replayable. Feedback heard. Love it.
But you were going to share more details behind the engineering story, or the Rust work.
6. Rewriting The API In Rust
I talked a little bit earlier about how scale is a problem for us. It’s a problem for everybody. Scale and safety are the two biggest things we’re spending our time on. Most people are used to talking about scale at the GPU accelerator layer, and much less so at the CPU, the old-school layer. But that’s just as much of a problem today as the accelerator layer, which is really interesting.
I think you cannot get enough VMs. You cannot get enough sandboxes.
Yeah, exactly. That’s primarily just everything being able to spin up and spin down.
With all the resources that you have, you really do have a lot of customers, too.
Yeah, a lot of users, which is crazy. Think about it: we’re using CPU to run our product services, the API, and all of our apps. We’re using CPU for sandboxing, and then for everything else. You’d be surprised how much more code is getting written, which means so many more CI runs, and therefore you need more CPU to handle that. This is true for everybody I talk to.
I’d add training. Training is probably using a lot of the sandbox.
For sure.
That’s a different bucket, right? I’m sure they don’t commingle that much.
They don’t commingle, although CPU for Anthropic as a whole is a zero-sum game. We negotiate our deals with the cloud service providers, get our CPU, and have to allocate it across everything I just mentioned, including research sandboxing versus product sandboxing. We’re allocating across absolutely everything, and everyone in the industry is short on everything.
Short on everything.
You used to be able to scale up on demand into CPU capacity as long as your budget allowed you to. Now it’s something you have to capacity-plan for in advance, which is kind of punching everyone in the face. It’s brand new, and people are starting to deal with it. One thing is obviously to try to get more CPU. We’re doing that as much as we can. The other is to run it more efficiently at the infrastructure layer: to make sure we’re running our sandboxes as efficiently as we can without compromising tenant isolation or the security properties we care about. At the application layer, we also want to run our applications and APIs more efficiently.
The thing I’m most excited about, as of last week, is that we fully rolled out the Messages API written in Rust. It’s one of those insane feats of what you’re able to do in the agentic coding era. We started talking about this earlier this year: we were going to hit a CPU wall, and there was stuff we needed to do. We’ve gone multi-region and done all these things, but we were running this Python monolith, and it was very CPU-inefficient.
Fuck Python.
So we chose Rust, which was a crazy decision at the time because our team was like, “We don’t have a single—”
Not Rust people. Got it.
Yeah, we don’t have a single Rust developer in the room.
Who cares? Just go for it.
Exactly. And not only that, but yes, Claude is going to write the code. The humans design the system. The humans are very involved in making sure that, at the end of the day, it’s a distributed-systems architecture. Claude can help write the code, but it can also help those engineers bounce around ideas and learn what they need to learn to design that system.
What’s most interesting is that the 2 things that mattered most were the plan to do the thing and the verification of the thing. The actual implementation was Claude writing code.
Yeah.
We built an entire fork of the API in Rust, then started running shadow traffic through it and verifying that the outputs we were getting from the Python system and the Rust system—
This is the same strategy. Feature flags are all you need.
Yeah, literally. And it’s just—
Load balancing, feature flags.
Yeah. It’s just, I don’t know. I never in my career would have had a team say, “Having to be in Rust? Yeah, yeah, cool. We’re just going to port the entire—” But do you still have Rust people, or no Rust people?
Well, they’d probably call themselves Rust people now. They’re Rust people now.
Yeah. I mean, they feel like they’ve stepped into Rust.
Yeah, exactly. Along the way, we’ve hired a couple of people who have more experience with Rust.
Sanity check.
But, yeah. It still is just—you never would have signed up for that. It would have taken years and years and years. You wouldn’t do it all in one shot; you would have carved out pieces of the system and said, “Migrate this, migrate that, migrate that.”
Yeah, easily a multi-year effort.
But you would never, in one swoop, port the whole thing over and then slowly turn it on. But that is what we ended up doing, and it worked. We saved a lot of CPU, and our latency has seen a nice improvement, which is pretty cool. It’s a crazy era we live in, because you just never could have done that sort of thing previously.
Order of magnitude of savings on the CPU side?
Just for people considering this?
Yeah, yeah.
Okay, order of magnitude. That’s really interesting. So—
Like, ballpark.
Yeah, yeah, ballpark.
I guess I would say we’ve saved somewhere around 40 to 50%. Something really crazy.
And latency?
I think we saw somewhere between 10 and 20% improvement on time to first token specifically. So, a lot of really cool improvements across the board. It’s higher on longer-running workloads, too.
So it’ll roll out—
Yeah.
—soon.
Yeah. But it’ll get faster. Do you fight the urge—you know, I’ve been involved in some major rewrites, and there’s always the temptation to add something new that didn’t exist before: “Since we’re rewriting anyway, right?” But then that adds scope. Do you just avoid that? Was it a pure rewrite?
It was a pure port. We did exactly that: we resisted the urge to add all these things. The little SWAT team we put together to do this was working while the rest of our team was building features into the API. They did all the static-analysis work. They’d yell at you if you built a feature over here and didn’t consider Rust over there. They dealt with all of that, but they resisted the urge to add features themselves or make the system different.
Their next goal is to do some re-architecture work, including things that I think will be more user-facing. We’re starting to roll out some things like, what if you didn’t have to pass the entire conversation state back and forth? If you didn’t have to do that on the Messages API because you were open to storing that state for some period of time, especially for long conversations where the context really builds up, you’d get a lot of great latency and efficiency wins.
Those are the sorts of things the team doing the port would say, “Ooh, what if we do this?” And we’d say, “Not yet.”
Does the new caching API have that?
We have prompt caching, yeah.
Yeah, but you still have to pass back the state of the conversation.
Okay.
Yeah. Very niche, small subsets, but I guess these are the levels of—
No, this is important.
Yeah, exactly. There’s a lot of savings to be done.
Well, from—
But it also turns them into a cloud.
And from the number-of-users point of view, it’s small.
Yes. But if you look at the actual token volume on those types of workloads—
Yeah.
—it’s going to be the majority.
Yeah.
I feel strongly about what Cloudflare did with Durable Objects. I feel like every cloud provider should clone it. Do you know what I mean? Isn’t that the right shape for a Messages API that needs continuation?
Yeah. It is really interesting, because I think we’re experimenting with what the right shape is for how we should run our compute to power all these sorts of things. But we can’t necessarily rely on one specific kind of compute, because we run across all three clouds. We don’t necessarily have one shape of compute, one type of machine we know we’re going to run on, or one way certain compute works that we can rely on across the board, other than the most vanilla versions of these things.
I think these things will continue to innovate and evolve, and we’ll be able to take more advantage of them.
I’ll just mention that if people want open-source Cloudflare Durable Objects, Rivet is the open-source equivalent.
Mm.
There are 50 people listening who will kill me if I don’t ask this question: You want CPU, and they have CPU to sell. Who do I sell to? Who do I email?
Absolutely.
And what do you—what’s a good way to help them help you? What does good look like?
Well, I guess, email me.
Are you sure?
I don’t know.
Yeah. Well, I’m going to get a lot of emails.
You’re looking at minimum fleet size, SOC 2, enterprise—give me all the privacy guarantees. For what it’s worth, we have a really awesome team that secures our accelerator compute. They’re now spending a lot more time with us on CPU and where and how we can procure it, to make sure we can continue to run our systems at the scale we need. Thankfully, it’s not my job to run around and procure compute, but we do have teams doing that, which is really great and important. There’s a quality bar that matters to us, and there’s a scale bar that matters to us. We’re not going to integrate and run on certain machines if we can’t get enough of them. All the properties you might expect of a system at this scale matter to us.
Related to the compute problem is the sandboxing problem. People have trouble containing agents in sandboxes. They keep breaking out. What’s going on there? What’s the state of the art? What’s the state of breakouts recently?
One of the things we did with Claude Managed Agents is separate the brain from the hands. We put out a blog post at launch explaining that the harness runs on a durable server—really, our server, right?—and the hands are an execution environment, like a sandbox or some sort of container.
Part of the reason we did this was latency and reliability. If the harness is running on a durable server and only spawning a sandbox or container when it needs to execute, we can get tokens back really quickly on the initial request. The sandbox can die, but the whole agent doesn’t die, because the harness is durable and the agent is storing state separately.
There was a whole product-development journey to get there. Our first iteration of an API for agents was just throwing Claude Code in a sandbox and putting an API on top. We didn’t launch that as our durable, scalable solution because you have to boot a sandbox and boot Claude Code before you can get a token back. If the sandbox dies, you lose the whole agent, so reliability wasn’t where we wanted it to be.
We architected it this way mostly for reliability and latency. But the interesting thing is that a product manager who worked on this from the very beginning said, “The most important property of this architecture is security.” Everyone was like, “Yes, yes, yes, but latency and reliability.” Now we’re vindicated. I apologized to this person yesterday. I told them, “I’m now telling everybody I meet what you’ve been saying for months.” Nobody was as excited about it as this person was.
One of the more important properties of this architecture is that you have credentials for any work the agent needs to execute. If you’re going to talk to an MCP server, use a CLI, call an API, or whatever you might want to do, you need to authenticate those things.
You need to store those credentials somewhere so the agent can use them. We architected this with a totally separate vault where the credentials are stored. Once the agent decides to call one of these services, it only ever sees a placeholder. On egress, a proxy pulls the right credential, injects it, and lets the request out.
What’s interesting about that is the agent is never in the same place as the credentials. In the old world, where you just boot the agent in a box and let it run, you had to send credentials in there. Then you had to figure out how to scrub them and make sure the agent couldn’t take them and do something it shouldn’t. This is literally an infrastructure wall that says the agents can’t access those credentials or do anything bad with them.
The other nice thing is that, with that sandboxing infrastructure, we can make sure we have really great egress controls. We have domain allowlists, or you can totally shut off network access, so untrusted code the agent writes and executes in a sandbox can’t just reach out to the internet. It can only reach the places you say it can. There are a lot of great properties in the architecture and in the way you can configure your sandboxes that make them secure by default.
One thing that’s starting to happen is that customers tell us they want to run agents for their customers or internally, but they’ve heard about incidents where agents break out of sandboxes and bad things happen. They ask how to think about it. I think the 2 most important things are the architecture, so it’s literally impossible for an agent to do something that isn’t okay, and the configuration of those sandboxes.
One really nice thing about Claude Managed Agents is that you get that out of the box. And as we learn more and continue to evolve those systems, you get those improvements for free. If you go back 9 months, nobody would have been saying all these words: brain and hands, credentials, injection, egress. These are relatively new problems we’re solving, and we want our customers to be able to take advantage of the solutions as we learn about them and solve them within our infrastructure.
Yeah. And internally, we’re just upgrading everything to be basically—
That pattern: secure by default, yes.
Yeah. But it’s better. We also unlocked something that we covered in the Tarek pod: the cloud brain, local hands—
Mm-hmm.
—which is the product. Is that the rough shape of what’s going on?
Yep. That seems right.
Yeah. I feel like it’s weird because obviously Claude Code is famous for being the super unsafe, “dangerously skip permissions” thing on your machine, but now you’re finally evolving to where you probably should’ve started in the first place. Basically, are all these sandbox leaks a fundamental issue with our understanding of sandboxing or container tech, or are we just bad at this? Are we just not doing the allowlist? It sounds like we’re just not doing the allowlist.
Yeah. Maybe a distinction I’d make is that a lot of the incidents people are hearing about and reading about are happening to models—not just ours, everybody’s models—while they’re in training. The way you run these models is you put them in a box and say, “Solve an impossible task. I’m going to stress you out. Ready, set, go.” They can only do what they can within their environment, and the agent is basically incentivized to try to get out of its box and solve the problem.
When people run production workloads on our API or anybody else’s API, they’re running in an environment with safeguards that aren’t necessarily running in training, where the whole point is to train the model. In production workloads, first, you have those online safeguards, which is great. Second, you have more of this architecture and infrastructure around what’s happening. The agent isn’t just in a box trying to get out. The harness is over here, the hands are over there, and it’s not even a problem you’d have to contemplate.
I think some of this is just people getting the right understanding of what’s happening. Some incidents in model training and elsewhere are things that wouldn’t happen in production workloads, where a lot of work has been done to put the right guardrails in place so those things never really could happen.
The second thing I’d say is that everybody’s figuring this out relatively newly, because long-running agents are a nascent thing that everybody is building and trying to deal with. Everyone’s figuring some of these things out on the fly and racing to put the right controls in place. Some of that is just happening as we go. I wouldn’t say people are bad at this. It’s more that it’s a nascent field and people are figuring it out. When we talk about pacing the frontier, we mean that we’re doing all this work for our agents internally and making sure it’s accessible to our customers, so the right guardrails are in place by default and agents don’t end up doing things you don’t want them to do.
Fair enough. Since you mentioned pacing, we’ve got to go there. We also talked about this last time. It seems superficially contradictory to advocate for pacing and then release a new frontier model 11 days later. To me, that just means it passed your safety checks, right?
Yeah. Everything we talked about earlier, like how we launch the models, means we’re continuing to evolve more and more safety properties across these dimensions. I’m starting to think about this in 3 dimensions: the architecture and infrastructure security, the online safeguards that run when a model is doing its work, and the offline classification safeguards that run on data at rest. Across all 3 dimensions, we’re doing the right work to make sure we feel good about putting a model out there safely. That’s fundamentally what we mean.
Opus 5.5 is an excellent model. It’s also really great across all the dimensions we care about in those 3 areas, and therefore we feel good about releasing it. We went for what we hoped would be the best possible user experience while we continue to hill-climb on making those false positives better. It’s still frustrating for people when they’re trying to work with a model and then fall back to a different one, so it’s something we have to keep working on.
Yeah. I think people also understand it in the safety sense. You want more false positives, right? It’s a safety issue. We clearly see the whole invasive species incident, and you’re still giving access.
Yeah.
To me, this is a close cousin of the general model-routing problem: do you want a smart router or a dumb router? Does that make sense?
Yes.
As far as I can tell, the Anthropic fallback router or detection thing is a separate model that’s online. So—
Once it hits a classifier, there’s just logic for where you need to drop to.
Exactly. So it’s not the full Fable. You’re not running full Fable at it, and that’s going to cause classification issues, because it’s clearly not as smart as Fable. So then there’s the question: when I have a classification issue, that’s an unjustified fallback. You never need to fall back. I’m just doing my bio homework, but I mentioned bio one time, and then it can fall back. So there’s the question—
It’s a hard problem. I think, in some superalignment sense, models are good at classifying stuff, but not answering it, right? Even a model that isn’t the smartest can tell this is a dangerous query, even if it can’t make any impact on it.
The simplest way to put it is: have the smarter model figure out whether the other model needs to detect it. There are trade-offs.
It’s slower, and it costs more.
There are trade-offs. I’m not saying it’s the right user experience, but it would probably be a slightly more effective route.
I’ve seen cases where it starts answering and reasoning, then gets hit by a safety filter. My first example with Fable was throwing the Fable safety card at it. I was like, “Just break down anything interesting in here.” It starts breaking it down, five minutes in—
Exactly.
—and then it gets into cybersecurity, and it’s like, “Oh, we’re cutting. We’re rolling back to Opus.” But it did a lot, and there’s a lot of context.
I see it. It got to a point where it was all about cybersecurity and safety, and then it rolled back. It wasn’t rolling back my query; it was the combination of the reasoning and the whole chain, right?
Do you have an idea of what’s better?
Yeah, let’s brainstorm. What’s better? I think the user experience and how we resolve that user experience are probably separate questions. Maybe it’s a new, super-smart model to handle it, or something like that. But I think the more appropriate user experience would be a more consistent, unified experience with Claude. Instead of falling back and then potentially falling back again, there would be logical points in our tree that do that for safety reasons and whatever.
But the user experience is really convoluted. You’re trying to do something, and I’d take the case of a good user: you’re just trying to do your bio homework or assess the cybersecurity safety card. Totally reasonable, and you get blocked and fall back. Instead, our model, classifiers, or however we do this should be smart enough to detect that this situation happens and write down the answer for you more concretely, while dialing down the potential negative use of that particular capability.
Going back to that bio homework scenario, we may have classifiers that detect it’s possibly biological. I’ll also say that on the API, we went with a spectrum of solutions for customers. If you’ve built something on the API and run into a block, you don’t have to continue the conversation on a fallback. That’s what we’ve done in our apps, and we’ve made it a feature of the API. You can choose to set, “I want to do the fallbacks,” and we’ll do that.
We also have helpers in our SDKs on the client side if you want to manage fallbacks in different ways. Or you could choose to reprompt differently and continue with the same model. But I do think there are interesting in-between solutions that we’ll continue to experiment with, to figure out how to unblock people safely and provide a good user experience.
And you brought up how it’s very similar to routers. I totally agree. I think it has a similar user-experience problem as well. With a router, let’s say you want a cost router for the complexity of your task. That makes sense, but it’s very hard to do. You don’t know everything about your task upfront, so you don’t have perfect information. Even if you could understand each model perfectly—which you can’t—you still wouldn’t have perfect information about your task.
There’s a lot of error associated with the routing logic you’d have to create, no matter how good a classifier you produce. Maybe it’s a super-big-brain-model kind of problem, but regardless, I think it’s very difficult. Finding a way to route properly in each of those scenarios is very convoluted. You could expose all that logic to customers: “Your task was to center a div, so I’m going to route you to something super cheap. This other task is detecting vulnerabilities, so I’m going to route you to something really expensive.” But I don’t think exposing all of that is a great user experience either.
For that similar class of problem, I think the more ideal user experience would be something like dynamic effort, or dynamically understanding the complexity of your task. That might evolve over time as you learn more and more about the task. But if you’re moving between a lot of different models, it becomes very disjointed.
Maybe a more unified approach would be a consistent model with a very low effort level. It’s cheap and affordable, but can scale itself up continuously, while giving you a consistent persona. I’m speaking in the context of our family of models, obviously. If you try to do that across other model families or providers, the personality and disjointedness become very complicated. I don’t know how you solve that at that layer, but within our family of models, I’m more bullish on things that are more consistent. Those might be ways to solve it more robustly for customers.
You’re not responsible for anyone else, but within Claude, yes.
Yeah.
I mean, this is your territory. It’s nice to see the way you guys think about these problems. Intent in and of itself is a hard problem. The models have gotten better, but sometimes people don’t do a good job of classifying what they want, or they slide all the way to one extreme, or the model doesn’t really understand. You ask a simple question, and it’s like, “Oh, I have a whole HTML landing page now.” I just expected a few lines of output. There’s always a mess around that.
Yeah. In this conversation, we’ve talked a lot about safety and launching models, but it’s obviously all in pursuit of helping power people’s workloads. As we discussed earlier, we’re seeing workloads change quite a bit. In some workloads, intelligence does seem to be increasingly saturated, which is a new situation. But there are still classes of tasks that are very unsaturated. I think there are more than people might anticipate, and they’re increasingly moving in the direction of getting that alpha or return by doing extremely long-running agentic tasks on things that historically you couldn’t brute-force—but maybe you can with a lot more intelligence behind it.
Those workloads are really fascinating to us, and we see a lot of value in that area, both for businesses getting value for themselves and for new companies or startups interested in building agentic products on top of it. A lot of the work today is coding, and there’s a lot of enterprise innovation and acceleration. Those tend to take the shape of automating a very distinct, specific process. In some areas, that’s really starting to work, which is awesome; in others, it still isn’t. There’s still a lot of room to evolve and push the frontier for those models.
Then there’s this new class of workload, like the biology stuff we talked about or some of the cybersecurity stuff. It’s a very complicated set of hard engineering or hard science problems. Those workloads are increasingly showing up as a really interesting area where we can deliver value to customers.
You’re at one of the few spots in the world where you’re discovering all these workloads and we’re automating them. It’s pretty crazy.
Yeah, it’s pretty crazy.
I mean, that’s very inspiring. You guys are having so much impact, and I’m really grateful for that. We’re going to go into a few closing questions, which I thought would be fun to bounce back and forth between the two of you. How do you fight slop on your teams?
7. Fighting AI Slop
This has been an interesting debate within our team. We’re no different from anybody else; I think plenty of teams are talking about this. What we’re trying to figure out is: if you’re producing a document or something you want other people to consume—code, documents, whatever—of course you should use Claude to help you. But if you use Claude to totally generate something and don’t spend any time actually reading it yourself and making sure it’s high quality, then you’re putting an extra burden on the consumer of that document or code to spend a whole bunch of time trying to parse it. That’s insane and totally unfair.
But what’s funny is we push back on our team, like, “You can’t do that,” right? If you’re producing something, use Claude, sure, whatever, but you have to make sure it’s high quality.
Yeah, you as the human are still responsible for the quality.
Right. You’re still responsible for the quality. But then you get into this really funny thing where, because of the Claude-isms—even something silly, like an em dash in the middle of a thing—people’s brains trigger on this: “Oh no, I’m reading something that was written by Claude and not a human,” and they’re annoyed because they want to read something that was fully produced by— Not necessarily fully produced by a human, but that has the quality bar a human would hold, right?
So then we get into funny situations where maybe the person actually did spend a lot of time making sure the thing was high quality. They used Claude to help them, but there are those little hints of Claude, and then the reader is turned off. And that’s unproductive as well.
What we’ve tried to do is set a bar within our team: use Claude to help you, because it would be really silly not to use this hilariously powerful agent to speed you up in your job and in your life, or just help you produce better things. But it has to meet a certain bar. And for the consumer and the reader, if it really is just slop, send it back. Just say no. You’re not going to waste your time on this. But also, don’t be too nitpicky about the little Claude-isms, because we have to find the right balance here.
It’s just really funny because we’re going back and forth on this stuff within our teams, and so is everybody else in the world right now. We’re just not any different.
But you have free tokens.
Well, sure, but that doesn’t tend to matter quite as much when it comes to—
As in, you have more than anyone else in the world. You probably see a lot—
Yeah.
You guys use a lot. You see a lot more people doing a lot more.
Mm, yeah. It’s totally fair, but it’s definitely a problem.
Heavily discouraged.
Yeah.
Engineering slop?
Yeah, I mean, it’s the same thing, right? Same thing.
Okay.
But I do think we’ve gotten to a pretty good spot where humans play a large role in making sure the designs of systems are good, and implementation tends to be Claude. We’re in a pretty good spot where, for a lot of the code that gets written, all you really need is a quick stamp and you can keep going. You don’t necessarily have someone deeply reviewing every single line of code, because you care more about the overall plan and whether the overall design is in good shape than the exact, perfect implementation. Obviously, we rely on a whole bunch of verification to make sure the thing we’re going to ship is high quality.
It’s been less of a conversation these days about code quality being slop, partly because Claude has gotten better, but partly because we have a lot of the right verification in place to make sure we’re holding a high bar.
Okay. I see. I see. Trust your tests.
Yeah.
Any product-management slop? You can ship a lot of features. You can put stuff out in a day.
So he said this to me recently. Do you remember what he said to me? Tariq tweeted this: “The right way to use model capabilities is not to ship 10X more features into prod. Spend more time understanding your users, trying experiments, building whatnot.” You could very easily just keep shipping random slop, but you should actually think about it. You should spend more time kicking off that first prompt. I know today he’s like, “Kick off plan mode. We don’t need it anymore.” But outside of plan mode, you should think about what you’re doing, right? It’s very easy to put out a lot, and you can get product slop.
Yeah.
I tend to go on this rant about how there are a lot of fundamentals about how software and products get planned and shipped that don’t need to change just because agents are helping speed up certain parts of the process. But there’s this inherent laziness that’s starting to come out from a lot of people, where it’s like, “Oh, I don’t have to think about some part of the process at all because the agents are doing it and helping me.” But those are still incredibly foundational parts of the process that you should be spending your time and energy on. I think that’s the spirit of what he’s getting at here.
As a founder or creator, it feels like because the cost of experimentation or exploration is so low now, I should be doing more of that. He’s basically saying not to do that, but to use it in service of a very intentional goal that I start with. I’m obviously coming around to it; it’s just hard. You have this toy that does everything, and you think, “Well, just throw it at the wall.”
Yeah, there’s scope, too. The blog post today was “Two Weeks to Make Claude Code Faster.” There were 13 things, and we hit 12 of the 13 targets in 3 days. You could just say, “Okay, it’s a 2-week sprint; we’ll do the 13,” but instead there are 20 things you push on. This isn’t product slop. It’s not new products; it’s refactoring. But still, you guys do a very good job of outputting a lot while keeping a high bar for quality.
Yeah.
Nice. Okay, part 2 of 3. This is going to make Angela very uncomfortable. This is Angela’s corner. I will tell people nonstop to—
Ugh.
I’m pretty sure you’ve deleted stuff, because there’s stuff in your deletions that I’m like—
That you like?
Yeah, that really altered my DNA.
Oh, I appreciate that.
High compliment. What do you think about the most? What do you refer people to the most that you want to bring up? I can pull it up for you if you want.
I actually don’t think I have a favorite, honestly.
Oh, come on.
No, I really don’t. I guess maybe I like the one that you say: “Half-assing something is an expensive way of not doing it.” Maybe that’s my favorite.
That’s very on topic for us today. Nice. Love it. I guess you’re obviously in a different job now, and you don’t do this anymore, but I think this is a power tool. This is like a Naval Ravikant thing. Naval knows how to do this.
I’ve stopped it, but yeah.
What mode of—what framework of mind are you in?
Frame of mind? That’s a good question. Six years ago. No, this was a very long time ago. I went through a period of my life where I was reading constantly. I was very bored with the stuff I was doing, and I felt like I wasn’t being intellectually challenged. So I went on a bit of a reading binge. I think I read about 200 books in a year, and I was curious about everything. I wanted to read everything.
I was missing a little bit of the intellectual muscle-building you get in school, where they give you a new topic and say, “You’ve got to learn this.” It’s tough and not super fun, actually, but you’re really stretching yourself. After working for a couple of years, I felt like I’d stopped doing that. The job was interesting and challenging, but it didn’t have that intellectual bit to it.
I went on this reading binge, and it was great. I had a really long commute, so it was a good use of my time—a very long Caltrain ride. I came out of that thinking a lot as a result. At the time, it wasn’t about building a Twitter following or anything. I was just tweeting into the abyss. I had almost no followers. These were tweets for myself, and I was practicing how to compress an idea into as few words as possible. That was the personal test I had for myself. Then people started liking it, so I kept going. Then I got too busy with work and stopped.
I’ve got to ask: after all that reading, any book recommendations, blogs, or long articles?
My favorite one is probably The Black Swan by—
Nassim Taleb.
Nassim Taleb. That was one of those life-changing philosophical books for me, where I thought, “That’s a completely different way to look at the world.”
And the concept—
You know, I was thinking of Black Swan when the Fable thing happened, right?
Oh, really?
Yeah. It’s a thing you didn’t know could happen.
Very much so.
Now that it happens, it permanently changes your framework of—
Exactly.
—how things work.
And the idea is so simple, but I think he did a great job of really hitting it home and articulating how it’s a common property of so much of the world. I really love that book, and that’s probably my favorite.
For listeners, if you’re bored with that one, there’s Fooled by Randomness, also by Taleb. Did you like Antifragile?
Less so.
That’s so much less so.
Yeah. The more famous he got, the more intolerable he became.
But also, a question for Katelyn: do you have a book you recommend to everybody?
No. That’s a good question. I need to think about it.
It is funny, because I do think we have very different styles on this front. You’re no longer in fortune-cookie-tweeting mode—
I stopped, yeah.
—but I do think a big part of how you work and how you work with other people is philosopher mode sometimes. Do you drop fortune cookies in Slack?
I do, actually. Not intentionally. I think it’s more just me trying to articulate myself.
All the time, yeah.
But I think that’s the core of succeeding at Anthropic: you have to be very good on Slack.
Oh, you do have to be very good on Slack. That’s actually true. It’s a pretty intense place. I’m a little psycho about inbox zero on both email and Slack. It’s easier on email these days than Slack, but I pretty aggressively prune the channels I’m actually in and paying attention to, versus the ones that are muted below the line. I’m always making sure I’m on top of the ones I need to be on top of.
It feels like a full-time job. There are days when I could cancel all my meetings, write no documents, do no work, and just sit on Slack all day. I’d have done what felt like an entire day’s worth of work just by engaging with everything there.
No, that’s horrible. What do you do? That’s exactly how I feel. You put in all this effort, and yes, you responded to people and got certain things over the line, but you didn’t produce the output you wanted. That’s personally a very hard thing for me to deal with. I try to dedicate time to actually thinking and producing something so I feel like I’m being productive.
And ignore stuff and drop things, right? You’re going to drop stuff.
Yes. I’ve gotten a lot of feedback that I’m a little hard to reach on Slack.
Mm.
A pretty terrible thing that happens now is I get a lot of Slacks saying, “Can you get Angela to respond to this?” And I’m like, “I’m really trying to write a doc here.”
Wait, so you guys don’t have a personal Claude assistant or something that can help you? There’s got to be a way. Do you not have a personal EA for each of you, in non-code mode?
We do have a human one, too. We also have plenty of Claude help in Slack. But I do think the types of Slack messages I’m pruning through are often about our team being blocked on something because there’s a hard decision that needs to happen. It’s a form of getting the team unblocked and executing at a high level, which is my job anyway. It just happens a lot in Slack.
I agree that it can feel like, “What did I even produce?” when you’ve spent a whole day going back and forth with people and trying to solve problems.
You produced a functioning team. You unblocked them.
Exactly.
Speaking of your team, Katelyn, I was going to go to you next: DevRel corner.
Oh, nice.
I know you’re really hiring DevRel folks. I’m going to make a callout for developer relations. You hired Adi Osmani recently. Great get. Lance—I don’t know him, but you still have a team to build out. What are you looking to hire for? This is your chance to talk about it. I’m happy to trade notes.
Props to you, because you’ve been the outsourced Anthropic recruiting team. Literally.
It’s fun.
And it’s not even just recruiting. I’m going to toot your horn a little bit. When I joined Anthropic, we were a small team. We run the platforms, the API business, and all of our product infrastructure. We had a very nascent DevRel function within Anthropic that wasn’t necessarily within the product and engineering org.
When I joined, we were like, “Okay, we need to grow up a little bit and figure out how to engage with the community and the world. We need to really invest in this DevRel function.” We ended up spinning up a DevRel team within Platform and a DevRel team within Claude Code, in part because they’re 2 pretty different developer ecosystems: the API business and agentic coding. They’re separate teams, but dotted-line to each other, and they spend a ton of time together.
I was going to be responsible for DevRel. I’d spent plenty of time with DevRel at Stripe and elsewhere, but it had never been something I was responsible for. I was like, “I have no idea what I’m doing.” And you were like, “I got you. Here are the 3 pillars of DevRel. Here’s how you think about it.” That was really great: product, content, and community. Now I can rattle that off to people, and they’re like, “Wow, you really understand DevRel.” I’m like, “No, someone taught me.”
You copied that.
Yeah. That was not original, man. Apparently, that post is the default reading at Google when they onboard new DevRel folks.
Nice.
It really helped me, because I had no idea what I was doing. I don’t have a manager for the DevRel team within Platform right now, and that’s one of the things I’m hoping to hire. Our DevRel ICs report to me directly, partly because I think it’s a really, really hard problem right now, but also because I find it really fun. I’m having a lot of fun working with the team.
What’s hard on Platform is that, sometimes in our public narrative, the Platform business is really large, but it can get lost between research and policy, all the frontier work we’re doing, and our applications like Claude Code and Cowork. Those are often what come to mind when people think about Anthropic, and less so that we’re running this massive developer platform business that so many people are building on.
A big goal for the Platform DevRel team is figuring out not only how we better engage with what people are building, and how we get feedback into the product, but also finding our voice, our brand, and our narrative. We want to figure out who we’re going to be in the community and not get lost in everything else that’s happening.
The brand we’ve started to build for our DevRel team has been interesting. Lance joined our team because you connected me with him. He’s highly technical in the way he approaches DevRel. He teaches: he writes a lot about memory and managing costs, and he goes really deep. He builds evals and puts together a lot of really technical content. I really like the idea of our DevRel team being thought leaders for people in what is a really nascent, really hard, really evolving space.
Educators, thought leaders, explainers of things. There’s another version of this I really like called the master builder: I’ll build things with the platform to show you what’s possible, and then you’ll be like, “Oh, I didn’t know that was possible—”
With the Claude platform.
Yeah, and then you just go.
Yeah. This is a thing stolen from LEGO.
Oh, nice.
Yeah, because LEGO—how do they sell Legos? They have master builders.
Yeah. They build things with Legos, and you’re like, “I want to do—”
Well, that’s the really cool—
“I want to do that.”
Yeah. I want to be like this person. And then they do that.
Nice.
And so—
That’s awesome.
Yeah, and CJ, the other person on our team, is really great as well. I think he does a lot more of that sort of writing cookbooks, putting out examples, and engaging with the community.
Cookbooks is funny, right? When APIs launched in 2023, let’s call it, for LLMs—
Yes, very important.
Now, not so much.
It’s interesting because you’re less in the spot where you’d copy and paste the code and use it, right? But I do think it shows people what’s possible, what the use case is, and how you can accomplish it. But I hear you that the code itself is maybe less…
For the longest time, we had an engineer on our team write our cookbook on computer use, and it haunts him to this day because it was massively complex, a beast of a thing. A lot of people got use out of it at a time when they were thinking, “I’m going to build a computer use implementation, and I’m going to deal with all the screenshotting and everything happening between the Messages API.”
Yeah, we were there for the hackathon you guys launched at the office.
Yeah. That was the era when you needed that sort of thing. Now we have a skill, the Claude API skill, and it’s really excellent at helping you build a high-quality integration, either within Claude Code or with other coding agents. Very few people, I think—or hope—are reaching for the big, gnarly cookbook to build a computer use integration. More often, they’re loading up the skill in a coding agent and working through that.
Or they’re just putting it into a reference architecture. It actually does much better. Increasingly, for a lot of the things where you might want different implementations—typically at the harness layer, but there could be other variants—you just point Claude at the reference architecture you want, and then you build it.
Fun fact: the Claude API skill that helps people build against the API is owned by our DevRel team. It was built by Lance.
As they should. Exactly. That’s the kind of thing. This is the most high-leverage form of DevRel: distilling all the best practices that are going to go into those agents.
One of the reasons it’s weird is that I’m regarded as this DevRel person or authority, but some part of me just wants to tell you that you don’t need DevRel.
You get so good you don’t need it.
Right. Or you turn the engineers and leaders into the public faces. All you’re really trying to do is make public faces and say, “Here’s how we do things here.” That’s kind of it. Sometimes it’s natural. You’re Anthropic; you don’t need to do that. People will hear about you.
When Boris first burst onto the scene, it seemed like, “Hey, I’m Boris. I’m the creator of Claude Code. This is how we do things.” He has a day job, but he’s also doing that function. So why doesn’t everyone just do that?
Yeah, I mean, it’s a really good question. There are teams that have done this really well. I think the Cloudflare team is a great example.
They’re all—
Yeah, they’re all—
All pumping out stuff.
Super awesome, right? Nonstop.
That’s a very visible sign of a good manager: they’re all happy with their jobs. They’re publicly identifying with the company. They’re not quitting.
Totally. It’s interesting because I think this is something we’re trying to do more of with the platform team, with our team, and with folks who can get out there, be part of the community, speak, and be on X or wherever they are. The problem for us is having the time and knowing the right way to do that stuff at a high level of quality. I see the DevRel team as the experts in doing this well. They can do it all day, every day, and help enable the rest of the team to do it well with the small amount of time they can spend on it.
I call this the unbundling of DevRel into a front office and a back office. A typical person doing this job handles the whole thing from start to finish and is autonomous. But you can build out a back office that lets product and engineering leads take the spotlight, with a simple contract: “Sit down with me for 30 to 60 minutes, and we’ll do the brain dump. Then you’re done; go back to work.” We’ll ghostwrite for you. That team of production people and writers provides the service to everyone else. I think that’s the future of this stuff.
People don’t really want to hear from DevRel. They want to hear from Lance.
Absolutely.
You have Tariq at Target. You have Lydia through Bud[?], you know? Why Chris and not a DevRel? He does the job and remains top of mind. People want to hear from people in the product, not paid shills or paid spokespersons.
Which I get a lot of, and that’s fine. That’s a job. Somebody has to do it.
You can reasonably expect people working in engineering not to be good at it. You should pay someone who specializes in it. Specialization is a skill. But I think a back-office/front-office separation can emerge.
That’s really cool.
That was my soapbox. Closing questions: You’ve been very generous with your time. Any parting thoughts or call to action? You start.
Build something cool on Opus 5.5.
What’s something people should try to push the bounds with on Opus 5.5? You guys have been using it for the longest.
It’s really cost-efficient and less verbose. The vibes are really high; it’s been fun. Yesterday was a really fun day for us because I got a whole slew of texts from friends and people who were just like, “Amazing model. Thank you.” One of my old bosses messaged me, and he was like, “Opus 5.5 just saved our butts in an incident.” It feels like we’re back.
If you wanted to experiment with something but cost, token efficiency, or those sorts of things were front of mind, that might have held you back from really letting a model run. Those are the sorts of problems you should return to with Opus 5.5, because I think you’ll be able to get a lot further than you otherwise would have with less cost-effective models.
I think you should try something with a multi-agent setup. I think you’ll find it does very well.
Any other hints, by the way? Multi-agent is very broad.
Nope. I’d be very curious to hear what people build. But yes, it should be extra good.
Including roles that aren’t strictly defined, but that the agents assign among themselves. That’s the new thing. You could always assign roles or give tools jobs, but this is like they’re giving themselves jobs.
Say more about multi-agent. Should we use subagents? How should people think about multi-agents?
Generally, you want to have one lead and split up a bunch of different things. As long as you have a task that you think is parallelizable, and they share uniform context across it, I think you’ll get a lot of gains.
Now, of course, it depends on the problem you’re pushing at. But increasingly, this is a superior way to handle a lot of these things, especially if every sub-agent—or whatever you want to call them, sub-leads—is asynchronous and they’re not blocking each other. Those are the only 2 bits I’d say you should care about in your architecture. Other than that, let it rip and see what Claude designs for you. Increasingly, I think a lot of this architecture stuff will move in this direction.
Claude will get smarter and smarter. It’s going to learn how to figure out ways to split itself, clone itself, and adjust itself to the architectures that make sense. You just need to prevent it— you just need to make sure it can parallel-process, so it needs to share state and not block its sub-agents. As long as you keep to those 2 principles, I think people should experiment and probably try some really crazy stuff. I wouldn’t be surprised to see some really interesting things come out of that. Maybe there’s some intractable problem that’s really annoying and people keep pushing on, and something like this could probably just crush it in maybe one shot or something.
Excellent. Yeah. All right, everyone go build.
Cool. [laughs]
Cool.