[BidClub_]
20VC · · 74 min

20VC: Jev Creator on Why Data Is More Important Than Compute | Will US Open-Source Models Catch Chinese | The Cost of Intelligence and Tokens: What No One Sees But Needs to Know with Diogo Almeida

Harry StebbingsDiogo Almeida

AI & SoftwareTechnicalCompany Building
Podcast ↗
TL;DR
  • Almeida argues that dependable judgment is the missing market for AI. His analogy is “Einstein like maybe 95% of the time, and Mr. Bean the other 5%”: useful for assistance, insufficient for unattended accounting. Jev’s claimed edge is reliability of intelligence—not speed or cost, which he says are not the point.
  • Task design and data outrank compute in Almeida’s account of AI progress. He ranks “doing the right task, then data, then compute, then algorithms,” and treats GPT 4.5 as evidence against compute-only scaling, conditional on pricing reflecting inference compute. Coding improves through relevant training data; general pre-trained intelligence per compute dollar is, in his understanding, barely moving.
  • Jev’s revenue thesis requires successful automation to expand customer budgets. Challenged on reaching $10 billion in revenue, Almeida predicts “mega Jevons paradox”: affordable projects must work, deliver value and justify spending beyond previous budgets. He says Jev has good margins and is profitable, while putting real automation at roughly 0% across Jev, Fable and Astra.
  • Workflow knowledge could let SaaS incumbents capture the next AI wave. Almeida favors “SaaSapalooza” and says proprietary intelligence can reside in software around shared models. He allows narrower models could be cheaper, but wants weeks of autonomous work rather than incremental copilot gains; Stebbings strongly challenges his characterization of Harvey’s vertical engineering.
  • Benchmark parity does not establish model parity, and Almeida doubts American open models will catch Chinese rivals. His guess is “no, unless China decides to stop trying,” attributing the difference to ethical and legal constraints rather than technical ability. He would not use an unmodified Chinese model in a coding agent, but argues training, guardrails and tool-call checks create other security options.
  • Almeida sees a demand-driven bubble risk despite coding-agent growth. Stebbings counters that no demand shock is visible and existing models still hold untapped value; Almeida agrees they are smart but says FOMO and exaggerated expectations can unwind. His criticism of many “Neo labs” is a business model of hiring ex-lab staff, acquiring GPUs and eventually getting acquired.
  • Typesafe raised for strategic allies while Almeida says its cash balance is already rising. Stebbings cites Jev reaching roughly a third of the Fortune 500, attributes growth from zero to $100 million in revenue in seven days to Sequoia’s Pat Grady, and gives a $7.5 billion valuation. Almeida’s priority: “What we need is allies to fight the largest companies in the world.”
Digest · the substance, structured for research

1. Reliability separates assistance from unattended automation

  • Almeida defines reliability as consistency of intelligence, rather than uptime or throughput. Models can tackle formidable math yet fail at accounting because errors have unacceptable consequences: “you get Einstein like maybe 95% of the time, and Mr. Bean the other 5%.” Companies declining to automate are responding rationally to that variance.
  • His engineering standard is upfront effort followed by software that solves a problem indefinitely in the background. That requires “nines of intelligent reliability,” potentially trading peak brilliance for consistency. ChatGPT already reliably produces recognizable, polite responses; he wants comparable reliability in the decisions inside those responses.
  • The gap is not necessarily in the weights. At 97% accuracy, Almeida wants to identify what failed: the model, the prompt or a linguistic ambiguity. Jev’s responsibility includes prompting guidance and stronger models. Being dismissed as “just a classifier” misses how much work robustness requires beyond a headline score.

2. The right task determines which data is valuable

  • Almeida’s “bitterest lesson” ranks “doing the right task, then data, then compute, then algorithms.” Stebbings challenges the scaling-plateau claim; Almeida’s criticism concerns pre-training, and he accepts compute helps with sufficient data. If pricing roughly reflects inference compute, he considers GPT 4.5 the largest model by token cost—and commercially unsuccessful.
  • Asked whether useful data is synthetic, proprietary or something else, Almeida instead works backward from the task. New tasks require data that may not exist anywhere. For RLHF, human comparisons of LLM outputs had to be defined, labeled and iterated; corporate databases contained no ready-made substitute. He says a GPT-2-sized model could beat something like a hypothetical GPT-9 when the task and data are right.
  • For RLVR, he says verified rewards were not the breakthrough: PPO attempts on unit tests already failed in 2021 without reasoning. Hidden reasoning changed the task’s structure. Good data must have the right difficulty to improve the frontier model, and what counts as good data changes as models improve. Enterprise data likewise needs shaping around the objective; merely possessing it is insufficient.
  • Stebbings argues data vendors could serve a market expanding from labs to companies owning intelligence. Asked whether Surge and Macaw are good businesses, Almeida returns the question, calls company-owned intelligence an open debate, and says the field is still in a hype wave, not yet at the reality stage.

3. Decision-model competition requires different organizational incentives

  • Asked why OpenAI cannot copy Jev, Almeida says genuine attention to reliability would create “an interesting battle.” Stebbings pushes back: labs want enterprise revenue, so how can they not care? “You have no fucking clue how little they care,” Almeida responds. His argument concerns organizational objectives, rather than an inability to build capable models.
  • He links ChatGPT’s earlier monthly-active-user focus to sycophancy and sees a similar usage objective emerging around coding agents. A dedicated decision model requires deliberate attention to judgment. He regards Anthropic’s models as best despite Chinese benchmark parity, and feels there is “no real competition right now” for Jev, while expecting that eventually to change.
  • His reversal concerns hype: launch made it look crucial, but he says customers trying benchmark-heavy alternatives returned disappointed. It matters “much less than I thought,” freeing Jev to “underpromise, overdeliver.” He welcomes serious competition as good for the world, while acknowledging it would hurt the business.

4. Workflow knowledge positions SaaS for a broader software market

  • Almeida’s complaint is that writing software has changed while software itself largely has not. His blue-LED analogy: adding blue enabled modern screens and white lighting, rather than merely expanding an indicator-light market by 50%. He hopes reliable AI becomes “a fourth fundamental logic gate,” opening possibilities far beyond proportional market growth.
  • He regards today’s AI attachments as rational. Unreliable models cannot safely make consequential company decisions, so vendors put AI where users bear the checking burden. Companies can tolerate AI making decisions with stakes to users because the user, rather than the company, bears the downside. The result is familiar SaaS with AI beside its logic, rather than deeply embedded in it.
  • Against the “SaaSpocalypse,” he adopts Martin Casado’s “SaaSapalooza.” Incumbents understand workflows and can fund engineering; distribution is not his main explanation for their advantage. His hypothetical TurboTax example asks where the interface could become smoother—a question he thinks application specialists understand better than frontier labs. Big SaaS is well positioned, without being guaranteed to win.
  • His proposed “software researcher” would build new interfaces and demonstrate where value exists. He rejects the status ladder placing ML researchers above ML engineers and software engineers. The ambition is to eliminate developers’ “can’t automate” category, leaving only a question of effort. Technical progress and helping developers exploit it are his principal bottlenecks.

5. Proprietary intelligence can reside in code around shared models

  • Stebbings brings the Fireworks/Modal case for proprietary enterprise models. Almeida calls coworker-shaped AI “the horseless carriages” of the era: automation need not resemble a person. His factory example is a machine twisting a bottle cap onto a bottle, solving one narrow operation without waiting for a general humanoid worker.
  • Company-trained coworkers could make sense if firms want that shape and possess the willingness and capability to train models for their own distribution. Almeida rejects those assumptions as the default, though he says the labs’ general-intelligence AI-coworker approach makes more sense to him. Given infinite interns costing “1,000x less,” companies should find useful work; he does not think continual learning or narrower models explains why that effect has not arrived.
  • He allows narrow models could be cheaper and admits he does not know how good particular companies’ models are. His preferred proprietary intelligence resides in software around a model: the database matters less than how it is used. Engineers, he says, hate updating dependencies unless the current version is insufficient; otherwise they would lock it away and update it mainly if it became much cheaper.
  • Stebbings strongly rejects calling Harvey a thin wrapper, citing deep vertical functionality. Almeida qualifies his view—he has no license and incomplete knowledge—but understands Harvey as a lawyer copilot after larger autonomous ambitions failed. He credits domain engineering while wanting much more: weeks of work completed in the background, rather than double-digit percentage improvements in human productivity.

6. Intelligence per token improves unevenly across tasks

  • Stebbings thinks raw intelligence gains have become minute. Almeida rejects a single IQ-like axis: models can look brilliant on one task and much weaker on another. In his understanding, general pre-trained intelligence per compute dollar is “barely moving”; coding improves substantially because more of it enters the training distribution.
  • He locates the largest coding-agent jump around “November-ish last year,” cannot remember which Opus release, and sees diminishing returns afterward, including amid enthusiasm for Fable 5.2 and Opus 5.5. Narrow cost optimizations become attractive during such plateaus. That differs from uniform improvement in the base model’s generality.
  • His countercheck is writing: “has writing improved in the last nine months?” His answer is no, immediately qualified by “Maybe, maybe I’m writing with them wrong.” More coding-agent data need not improve unrelated capabilities. Dollar-per-benchmark charts therefore do not, in his view, measure the underlying phenomenon investors think they measure.

7. Open-model rankings conceal incentives and security trade-offs

  • “All benchmarks are bad,” Almeida says; measuring intelligence is possible but difficult, with labs retaining better proxies internally. He calls OpenAI and Anthropic mostly truthful because they need not cheat to lead, correcting Stebbings’s suggestion of a duty. Second-tier labs face pressure to convince themselves they can win; he cites Llama 4, Google and Chinese labs.
  • He singles out the “rough domain of an American open-source leader” as a consolation prize rewarding optics without meaningful usage. Will American open models catch Chinese peers in true intelligence? His guess is “no, unless China decides to stop trying”—because American efforts hold themselves to higher ethical and legal standards, rather than lacking technical capability.
  • The economic importance remains conditional: if progress plateaus hard enough, a modest intelligence gap might not matter. He also rejects treating an unmodified model inside a coding agent as the only deployment architecture. “I would not use a Chinese model in my coding agent vanilla,” he says, while arguing training can remove some biases.
  • Stebbings raises prompt injections and backdoors. Almeida points to threat modeling, guardrails and Jev classifying whether tool calls are safe or “phoning home to China.” Security can involve several components, rather than depending entirely on the generating model. His frustration is with “mono model” thinking and the field’s “horse blinders.”

8. Jevons requires successful projects before it expands spending

  • Stebbings asks how extremely cheap Jev becomes a $10 billion revenue business. Almeida says good margins and efficient serving are already present; this is not the standard subsidized playbook. Lower prices make projects feasible, but Jevons paradox requires those projects to deliver enough value that customers restore previous budgets and then spend beyond them.
  • He predicts “mega Jevons paradox” and token consumption dwarfing conventional text-generating LLMs. Asked whether energy and infrastructure can support it, he answers “Yep,” while acknowledging substantial efficiency research remains necessary. The forecast depends on AI becoming useful background infrastructure that improves lives, rather than merely sending people more generated material.
  • His severe baseline: “Jev, Fable, and Astra are all tied at roughly 0%” on real automation. He allows small exceptions but sees an industry still to be created. Background AI should, he argues, be a larger opportunity than coding agents assisting engineers or making software roughly 20% more efficient to write.

9. Untapped capability does not eliminate demand-shock risk

  • Almeida has believed AI is a bubble “for a while” and plans for a burst. Stebbings sees no present demand shock; Almeida argues FOMO creates some demand among buyers overestimating capabilities. Expectations can unwind quickly when deployment disappoints, allowing demand to contract without a matching deterioration in the technology.
  • Stebbings’s counterargument is that existing models contain substantial unextracted value even if progress stops. Almeida agrees they are extremely smart but questions whether organizations can access that value through current interfaces. His counterexample is companies firing staff on automation expectations and then backtracking: the capability story may be right while the operational bet fails.
  • His “Neo lab” criticism is sharp: hire ex-lab staff, acquire GPUs, have fun, then get acquired. He sees resource waste. He likes models getting smarter through RL despite diminishing returns, but would not invest his own money; a VC’s 1% moonshot calculation might differ. His expected-value assessment is that current spending is unjustified, with willingness to be proven wrong.
  • On the GPU-export debate, he identifies Will Depue’s result but says he had heard online that the same technique suggested DeepSeek 4.1 Flash was not trained that way. He says Jev’s efficiency lets it stay on the sidelines of GPU spending. Stebbings cites Cognition as a fourth-place coding-agent vendor with $1.6 billion in revenue; Almeida does not analyze that market or forecast its three-year size, warning that FOMO has driven repeated up-and-down cycles.

10. The financing bought allies for competition with the largest labs

  • Stebbings introduces Jev as used by roughly a third of the Fortune 500, attributes growth from zero to $100 million in revenue in seven days to Sequoia’s Pat Grady, and cites a $7.5 billion valuation. Almeida says fundraising was overwhelmingly inbound, conducted without a deck while the company was too busy to prepare one.
  • Investors supplied useful intelligence by explaining how portfolio companies used Jev—information Almeida says Typesafe does not obtain by snooping on customer data. Asked about $750 million at $7.5 billion, he says terms moved around and the team initially lacked a clear fundraising objective. The eventual priority was allies, rather than maximizing money or valuation.
  • “We are profitable,” he says, with cash in the bank rising. A previously lightly funded company should not invent an immediate plan to spend a huge raise. He describes investor willingness to participate at 3x the valuation; Stebbings calls that bubble behavior. Almeida’s answer: “We don’t need the money or the valuation.”
  • He chose Andreessen for philosophical alignment with technological optimism, praising Martin Casado, and Sequoia mainly because he wanted Sonya Wang as a strategic thought partner. His operating maxim is “make great decisions slowly”: important choices deserve a clear why, philosophy and framework. That applies to product, data roadmaps, trust and hiring.

11. Task-focused data builders are the scarce hiring resource

  • The hardest hiring problem is Almeida’s data-domain team, an internal specialty producing the new shape of data Jev’s task requires. The obstacle is communicating the opportunity: brilliant people may assume they cannot advance ML without being researchers. He wants them to learn from internal experts and directly make models smarter.
  • Stebbings cites Eric at Modal competing with $20 million graduate packages. Almeida says Typesafe deliberately pays well without matching the highest lab offers, selecting for people who want its mission. He worries extraordinary pay undermines cultural selection, citing what he sees as a market for coaching applicants through Anthropic culture interviews.
  • His reluctance to hire OpenAI researchers is narrower than a blanket ban on OpenAI staff. He sees a selection bias toward algorithms and scaling, with data treated as low-status. Researchers obsessed with inventing a task may be excellent but poorly transferable to Jev’s “machine-native AI.” His universal hiring preference is “technophilia”—a shared love of pushing technology forward.

12. Software research comes before robotics in his reliability agenda

  • Robotics is his instinctive pick for the most overhyped category because reliable operation depends on many prerequisites still unsolved. The underhyped counterpart is software research: people combining AI and code to create new interfaces. He wants more exploration of what software can become as automation improves.
  • His five-year prediction concerns principal-agent problems, from electorates and representatives to shareholders and companies. Law resembles software, he argues: detailed instructions make behavior reproducible but lack a primitive for enforcing the “spirit of the law instead of the letter.” He thinks AI can let principals express intentions more directly, reducing misalignment where people agree on the spirit but disagree over implementation.
Full transcript
Diogo Almeida

Almost everyone paying attention to AI knows reliability is the absolute largest bottleneck. You get Einstein maybe 95% of the time, and Mr. Bean the other 5%. Data matters a lot more than compute. I don't want to be called a neo-lab because, in my view, most neo-labs are incredibly, incredibly wasteful. Anthropic models are just the best, even if the Chinese models match them on benchmarks.

I truly believe we're going to have a mega Jevons paradox. AI is so clearly awesome, yet software itself hasn't changed. Everyone has had these mono-model horse blinders on for so fucking long. I've believed AI has been a bubble for a while, so I would like to keep planning for the eventuality of a bubble burst. What we need is allies to fight the largest companies in the world. We don't need the money or the valuation.

Harry Stebbings

This is 20VC with me, Harry Stebbings. Now, joining me in the hot seat today is one of the best founders of our time, one of the hottest companies, Diogo Almeida. Diogo is the founder of Typesafe, the maker of Jev. Jev is used by, oh, a third of the Fortune 500. Pretty incredible. Yep, it gets better. Sequoia's Pat Grady says Jev went from zero to 100 million in revenue in just seven days. And when you're that fast and that good, it comes with some benefits. They've been valued at $7.5 billion, and they had that round led by Martin Casado at Andreessen, and Sequoia also invested. Today, we get into the future of models with Diogo. This is an incredible discussion.

You have now arrived at your destination.

Harry Stebbings

Diogo, I'm so looking forward to this. We had a brilliant exchange following Eric's show from Modal. I actually spoke to him last night about you, so I'm very excited for this. Welcome to the show.

Diogo Almeida

Thank you, thank you. It's a very small exchange, but I'm glad you liked it. It was really fun, and I didn't realize it would become this.

Harry Stebbings

Well, we said that it would be a spicy discussion. So before we get into—

Diogo Almeida

Absolutely.

Harry Stebbings

—the spice itself, I just want to lay—

Diogo Almeida

Mm-hmm.

Harry Stebbings

—some frameworks and foundations. For people who don't know what Typesafe and Jev are, can you give us the 60-second intro?

1. Why AI Still Cannot Automate

Diogo Almeida

I would argue that Jev is not a specialized model, but my pitch for people who aren't purely software developers is more about the state of AI. How can AI be so smart at so many things, like the Millennium Prize problems in math, yet not transform software at all? All the SaaS we use is basically the same as it was a decade ago, and we are not automating almost any entire job.

Drive-throughs—we keep failing. There's AI customer service, but it actually can't be trusted for the big decisions. How can that be the case? It seems so good at some things and so bad at others. To me, what we are doing with Jev is trying to bridge that gap: make AI too good to be true for automation, instead of just human-in-the-loop assistance.

Harry Stebbings

I spoke to Sonali at Sequoia, and, speaking of full end-to-end automation, she asked, “Why is reliability the biggest obstacle to automation?”

Diogo Almeida

I'll give a caveat about reliability first: I don't mean uptime. I don't mean availability or getting lots of throughput. I mean reliability of intelligence, which I'm still trying to find a good word for, but it's this non-jaggedness of intelligence. I would argue almost everyone who's paying attention to AI knows reliability is the absolute largest bottleneck.

Here's the example. How good are you at math? How good am I at math? There's not a chance we could even help with the Millennium Prize solution, I believe. I haven't read it, but I would wager that. And yet these same models can't even be used for accounting—much simpler stuff. The question is, why?

It's not that these models are extremely high variance. People love this saying that I've coined: you get Einstein maybe 95% of the time, and Mr. Bean the other 5%. The problem is that, in expected value, it just doesn't work out. Therefore, companies quite wisely learn that you can't actually automate stuff. Over the multiple AI hype cycles of the last four years, I've seen so many companies try to automate accounting. They keep failing because the math doesn't work out. It's not a diffusion problem in the slightest. Reliability is what gets you; maybe you can trade off some level of Einstein for getting it more commonly and more often, while getting the same functional intelligence. To me, that's Jev's superpower.

Harry Stebbings

So how do we get there? What's the solution to removing Mr. Bean, as much as I love Mr. Bean?

Diogo Almeida

I love Mr. Bean too.

Harry Stebbings

Uh—

Diogo Almeida

I actually love when agents are insane. You know, bad vibes forever. It's a very fun story. And that's not what I want in my software that runs in the background.

Harry Stebbings

Yeah.

Diogo Almeida

As an engineer, I want to write a thing. I want to spend a lot of effort up front. Maybe I don't realize it'll be a lot of effort, but it always is. And I want it to run in the background and solve the problem forever.

Harry Stebbings

Yeah.

2. Data Beats Compute

Diogo Almeida

You just need nines of intelligent reliability to do that. I would argue that the way to do that—man, I wish I could bring up my slides here, but I'll try to use my hands. I call it AI's Bitter Lesson. Sutton coined the Bitter Lesson, which is roughly that compute matters more than algorithms.

Harry Stebbings

Mm.

Diogo Almeida

This was the scaling era we had from 2019 through 2022 and into 2023: just scale, and the models get better.

Harry Stebbings

Mm-hmm.

Diogo Almeida

This kind of plateaued at GPT-4.5, and then we kind of stopped scaling as a field because it stopped having a payoff. I would argue that there's a bitter lesson: data matters a lot more than compute. This is very obvious to anyone who's done applied ML. But the people in the peanut gallery don't seem to realize this: data is obviously more valuable than compute.

The most important thing by far is, what is your North Star? What do you define as your right task? Before Jev, the 2 right tasks that existed in LLM land were RLHF—

Harry Stebbings

Mm-hmm.

Diogo Almeida

—which is reinforcement learning from human feedback, which means “please humans,” and RLVR, which means “score high on benchmarks.” Neither of those actually translates to software that runs in the background. I would argue that this is the root of all the problems.

It is not that AI isn’t capable, because AI is clearly super capable. You are experienced in the real world. Everyone who’s experienced the real world knows this: you get what you optimize for, right? We just happen to optimize for the human in the loop.

Harry Stebbings

Oh, thank you so much for bluntly giving me so many follow-ups in which I—

Diogo Almeida

Can I say one thing about this too? On the note of reliability, ChatGPT never fails at being ChatGPT. When was the last time you got random characters? It’s kind of odd that it—or even if you ask for a polite email, it’s not going to start saying—I probably shouldn’t curse on a podcast—but it’s not going to start saying slurs. Humans might make that error sometimes.

The problem isn’t that it’s impolite or sycophantic. The problem is that it makes wrong decisions within the text. It’s interesting that it can be reliable at some things, but not others. If you optimize for the important things to be reliable on, which are decision-making and judgment—

Harry Stebbings

Mm.

Diogo Almeida

—that is how you get real reliability.

Harry Stebbings

You said there about the plateauing of compute in terms of the benefits that accrue. First, have we got to that stage? Are you serious? I thought the whole point was that—

Diogo Almeida

Of pre-training?

Harry Stebbings

Purely on pre-training, sure. But more and more compute still leads to better and better output, no?

Diogo Almeida

Yes, but you need a lot more data than people once upon a time thought. Maybe people have warped this to make it seem like they were right the whole time. But the narrative back in the day was just, “Scale to the moon, and you will get AGI.” GPT-4.5 is what broke that narrative.

I obviously don’t know what’s going on behind the labs, but if we believe the labs were pricing roughly based on inference compute, then GPT-4.5 is still the largest model ever by token cost, and it was not commercially successful. I think this is obviously the case because you can’t just scale compute without really thinking about it.

People are throwing a lot of compute at training, but you need a lot of data with it. This is why you see so many get-rich-quick schemes around selling data—RL data to labs. You can’t really do very much without that data. I believe it might have been Dwarkesh who had a recent Twitter post showing what matters more: data or algorithms. Data was something like many, many times more effective. It was something this week.

This is something everyone has known since the beginning of ML. It’s not even specific to AI; it’s just that the compute-scaling era was a weird outlier in this.

Harry Stebbings

When we talk about the value of data, or how it’s so much more primary than compute, is that synthetic data? Is it proprietary enterprise data that everyone has in their companies? Is it AI systems generating their own experiences? Is it certain types of data, or just broad data? Can you help me understand that?

Diogo Almeida

I wish I had my visuals here. Let me work backwards again from my bitterest lesson: in order of importance, there’s doing the right task, then data, then compute, then algorithms. This is what we learned extremely, extremely well when we made RLHF. The quality of your underlying algorithm matters very little as long as you do the right task.

A GPT-2-sized model could beat something like GPT-9—which doesn’t even exist. In order to have data, you actually need to know the right task. Normally, when you create a new task, the data doesn’t even exist. When we made RLHF, human preference data comparing LLM outputs didn’t exist anywhere on the internet. It wasn’t in a company database. It wasn’t on the internet. We had to define it, understand the instructions, get human labelers, iterate, and figure out what that data should look like.

Harry Stebbings

Mm.

Diogo Almeida

Doing the right task is what unlocks the data in the first place. The problem is that bootstrapping a new task is hard because, by construction, the data doesn’t exist.

Harry Stebbings

Mm.

Diogo Almeida

It’s a similar thing for us, but it was also similar for RLVR. There’s a tricky thing about what the right task of RLVR is: reinforcement learning with verifiable rewards. Sorry if I’m going a little too advanced here.

Harry Stebbings

No, no, no. Go ahead.

Diogo Almeida

RLVR isn’t at all about the verifiable rewards. A big part of RLVR is the latent chain of thought, the hidden reasoning that lets models think in private and then answer. Even in 2021, there were verifiable rewards. People tried to do RL, like PPO without reasoning on unit tests, and it kept failing.

So it wasn’t at all about the verifiable rewards. It’s about the structure of the task, and it just turns out that once you’re doing RLVR, the models are a lot better at this. What good data means is now a function of how good the models are. This is why people who make data need access to cutting-edge models: what actually makes the data good is whether or not it improves the cutting-edge models.

They need to be at exactly the right level of difficulty to push the models forward. There’s a lot of nuance and on-policy-ness of data. Back to the original question: is the right data stored in corporate data lakes? I would argue no, because it’s probably not shaped the right way.

People are definitely trying to do this, so a lot of data vendors are trying to yoink this data and sell it. Maybe they can do some massaging to make it work, but the most important thing is focusing on how we use this thing to push our task forward. If you forget that, you are Goodharting a local objective.

Harry Stebbings

How do we use this thing to push our task forward? Are the data-providing companies—Surge, Macaw—good businesses? Are they services businesses now that have to go end to end, where they really get into implementation and the effectiveness of the model, not just data serving like they did in the early days? Are they good businesses?

Diogo Almeida

That’s more of a you question than a me question. Do you think they’re good businesses?

Harry Stebbings

I do personally, because I think the market is just so large for the amount of money these LLMs will spend on data.

Diogo Almeida

Mm-hmm.

Harry Stebbings

And if you think we’re going to move into a world where companies own their own intelligence, that data will always need supplementary data. So then the TAM actually becomes moving from LLMs to everyone.

Diogo Almeida

Interesting. That might be a fun debate: whether or not companies will own their own intelligence. That might be a fun one. I will argue that we’re still in hype-wave territory, and there’s always a trough of disillusionment, and then—

Harry Stebbings

Yeah.

Diogo Almeida

—the reality thing. I think we haven’t hit the reality part of it yet.

Harry Stebbings

You said about the hype and the trough, my friend. There is insane—

Diogo Almeida

It’s pronounced “trough.” That’s good to know.

Harry Stebbings

No, I tried to do it elegantly, okay? I’m British.

Diogo Almeida

Yeah, yeah.

Harry Stebbings

Charmed.

Diogo Almeida

Oh, yeah.

Harry Stebbings

Are you concerned by the amount of money going in, or do you think it’s justified?

Diogo Almeida

I am concerned for the world, I think. Throwing money into the furnace is a bad thing for the world. I really don’t want to be too negative, but I don’t want to be called a neo-lab because, from my sense, most neo-labs are incredibly, incredibly wasteful.

It feels like their business model is: hire a bunch of ex-lab people, get a bunch of GPUs, have fun, and then eventually get acquired.

Diogo Almeida

That seems like a waste of resources to me. On one hand, I’m not a fan of that. On the other hand, people are throwing a lot of money, for example, into RL, and I actually think this is kind of cool. There are massively diminishing returns, but the models are getting smarter. That’s kind of good for the world. I think it’s very cool.

Would I make that investment with my money? No, obviously not, because we have money to make bets, and we’re obviously not going to do that. So I’m not a fan of that in the world either, but maybe the VC math works out where there’s a 1% chance of a moonshot and it actually has positive expected value. And wait, you had a question. Am I concerned? There was a—

Harry Stebbings

Are you concerned, or is it justified?

Diogo Almeida

Oh. I would guess, based on my expected-value calculations, that it is not justified, but I’d be very happy to be proven wrong—or at least, right now.

Harry Stebbings

Did you expect it to work as quickly as it did?

Diogo Almeida

For years now, even before TypeSafe existed, I thought I must be missing something about the field. Obviously, I convinced myself I wasn’t missing something, but this direction seemed unbelievably obvious to me. It was so obvious that I gave up when I first came to the idea because I thought, “Surely Anthropic is already 80% done.” It’s the most obvious thing I’ve ever heard of.

So to me, it’s really obvious that this would be valuable in the world. I still think people don’t get it, even though the launch was obviously phenomenal. The part I didn’t expect was the ChatGPT for developers moment, because—

Harry Stebbings

Mm.

Diogo Almeida

Even non-developers were talking about it. That’s something I could not have predicted. That was—

Harry Stebbings

What do—

Diogo Almeida

—super weird.

Harry Stebbings

What do people not get? What do people get most wrong about Jev and you?

Diogo Almeida

My team hates when I talk about this, but my rule is that on a podcast, I can be let off the rails. I believe it’s a competitive disadvantage for me to talk about what Jev is really doing, because people keep missing the mark. If we just didn’t tell anyone, we would forever be king of the hill. I care too much about the field not to yell, in painful insanity, “What the fuck is going on?”

What people really don’t get is that what’s special about the model is not the cost or the speed. It’s, again, the reliability of intelligence.

Harry Stebbings

Mm.

Diogo Almeida

People are still thinking in LLM land. So many people, I think, are burning their goodwill. There should be copycat models in the future, but they should give a lot of fucks about reliability, in my opinion. The people shipping copycats right now are trying to do the same thing as in LLMs: max out benchmarks, get hype, and highlight little things like numbers that aren’t relevant, like speed and cost.

The real thing they need to do is hunker down in the bunker and make the models really, really smart. People don’t realize that’s what we’re all about. When we see benchmarks, even if we’re unbelievably in the lead, we’re still pained when our model is dumb. That’s what we focus on, and I think that’s what everyone in the field should focus on. “Hey, why is it 97% accurate? What was the missing piece?”

To be clear, it’s not always a model thing. Very often, it’s a prompting thing. But that—

Harry Stebbings

Hmm.

Diogo Almeida

—is how it is to build a developer tool, right? We need to educate people. We need to figure out how we could prompt this better. We need to figure out whether there’s some nuance in this language that could help people make more robust prompts, or whether we need to make the model smarter. This is what people really miss.

I think developers get it, and they got it way faster than I thought, because they already don’t give a shit about the benchmarks. They’re just like, “Your model is clearly the best at new stuff.” But the same people who say, “Oh, this is really boring. It’s just a classifier,” ended up making the copycats, and they don’t understand that there’s a reason you can put so much effort into making the model more and more robust.

Harry Stebbings

You said “copycats” there. Can I be blunt?

Diogo Almeida

Oh—

Harry Stebbings

OpenAI—

Diogo Almeida

Yeah, please do.

Harry Stebbings

—and others have moved very quickly in trying to create their own decision-model products after you launched. Why don’t the big model providers just copy you and compete you out?

Diogo Almeida

I don’t think that’s a blunt question at all. I think that’s a totally fair, reasonable one. If they one day started caring a lot about reliability, that would become an interesting battle. So—

Harry Stebbings

Diogo, how do they not care about reliability? They want enterprises. They want—

Diogo Almeida

You have no fucking clue how little they care.

Harry Stebbings

What? How? They want enterprises. They want revenue. How are you going to get a CIO to sign up if you don’t have reliability?

Diogo Almeida

I think people have a hard time understanding intelligence as a product, which is why people try to use benchmarks as Band-Aids. Benchmarks are kind of a proxy for understanding the intelligence of the underlying thing. But as far as I can tell, the string field has learned over the last 9 months that benchmarks don’t mean shit.

Anthropic models are just the best, even if the Chinese models match them on benchmarks. Even when you measure benchmark maxing, you still think the Anthropic models are better than the competition, even though they’re not on the Pareto curve of pretty much any benchmark.

Basically, it looks like the subfield of System 1, or decision models, learned this in—I think competition maybe started a week after. It felt like people learned it in 2 or 3 weeks. We were like, “Oh yeah, the other stuff is just benchmark max. It’s total crap.” It’s very easy to see that it’s different. I think it takes a different type of organization to really, really care about reliability.

Harry Stebbings

If they don’t care—

Diogo Almeida

Go ahead.

Harry Stebbings

—about reliability, what do they care about then, Diogo?

Diogo Almeida

It really depends on their business objectives, right? Back in the “ChatGPT is everything” days, before Claude Code, they cared about monthly active users. That’s a very different metric to optimize for than the best chatbot experience. This is how sycophancy emerged, and it kept going in a cycle of, “Oh, it’s more sycophantic. It lies sometimes, but hey, that is what people want, so we just keep doing it in some pockets of space.”

You get what you optimize for. That’s what ChatGPT optimized for. I think now people are optimizing for the MAU equivalent of coding agents. If they’re not training a dedicated model for this and putting a lot of intense thought into it, I don’t think they’ll be able to make a comparable model anytime soon.

I hope that all of them do, because I think it’d be very good for the world. It’s obviously going to be bad for the business, but right now it feels like there’s no real competition. That surely will change eventually, right? I hope we get competition and show that we’re the best because we care the most, and we have the smartest people working as hard as possible, laser-focused on the North Star. But again, this isn’t competition yet.

Harry Stebbings

When you look at the decision-model product landscape you’re in, what does that look like in 3 years? Is it an Uber/Lyft market? Is it a GCP, AWS, Azure market? What does the market composition look like in 3 years’ time?

Diogo Almeida

I’m a bit of a technological idealist, I think. I don’t really think about what the competitive landscape is going to be like. I care more about the technological landscape. Our strength is always going to be technological shock and awe, so we’re going to have to keep shipping awesome stuff. The way I see that landscape is that I’m going to ignore competition.

In terms of the number of tokens, I truly believe we’re going to have a mega Jevons paradox. It’s not going to be a subtle Jevons. It’s not going to be a fake Jevons, like LLMs. It’s going to be super-duper. There’s going to be tons and tons and tons of usage, and I think it will completely dwarf at least the total token consumption of string LLMs.

I think these things are going to be used everywhere, and we’re going to finally have AI running in the background. I think we’re going to start to see a shift in how people feel about AI: from being this thing that slops crap at them to, “Oh, it actually makes life better.” People truly like it when technology improves their lives.

Harry Stebbings

When you say “mega Jevons paradox,” do we have the energy and infrastructure to power the demand of a mega Jevons paradox that you foresee?

Diogo Almeida

Yep.

Harry Stebbings

And what are the bottlenecks to your tech utopia?

Diogo Almeida

Our models are incredibly efficient in comparison.

Diogo Almeida

We're not doing the standard subsidized playbook. We can serve these things incredibly efficiently.

Harry Stebbings

Yeah.

Diogo Almeida

I think there are many boundaries to push on that efficiency angle. Of course, we're going to have to do some sick research, but that's what I trust us to do. We'll have to push efficiency quite a bit in order to have that mega Jevons paradox occur.

I think the largest bottlenecks are going to be that it feels a little bit single-player right now. It's on us to push the boundaries of the technology from an AI point of view, and from a field point of view, we're going to need to interact a lot with developers. We're going to need to help them push the boundaries.

This is what I've been talking about: a role I dream of. I'm so sick of the status games, especially in SF, where ML researchers are a higher tier than ML engineers, who are a higher tier than software engineers. I think that's completely messed up. My hope is that we can create a new type of role that I call—my current favorite name is “software researcher”—someone who actually pushes the boundaries of software.

If you knew you were in the Cambrian explosion of the internet or mobile, you'd think there were so many boundaries to push and so much low-hanging fruit that you needed trailblazers to go out there and build these lighthouses saying, “Hey, there's value here. Let's have a gold rush there.” I'm so excited about encouraging people to push those boundaries. I wish I had the time to do it myself. There are so many crazy, exciting things to do.

I think those two are going to be the biggest bottlenecks to get to System 1 models everywhere and AI actually automating shit. If I could say one more thing: I don't know if you're a software engineer, but I guarantee every single person who's ever been a software engineer has had a conception in their head of, “I can automate this,” or, “I can't automate this.” You need to know, right? If you wanted to automate something that moves files around, you can automate this. If you want to make a voice interface for your laptop so you don't need to type with your hands, you can't automate this.

My hope is that AI will completely remove the “can't automate” part. It's only going to be a matter of effort. I have this entire soapbox about the future of software. I will cook on this undercooked idea.

Harry Stebbings

Oh.

Diogo Almeida

This is something I find to be such a beautiful analogy for AI: AI is so clearly awesome, yet software itself hasn't changed. Writing software has very much changed, but the software is the same, and that's tragic.

I love the analogy of the blue LED. When the red and green LEDs existed, they had very narrow uses. They could barely be used for anything beyond indicator lights because they're very efficient. But when you add the blue LED, it didn't just expand the market cap of LEDs by 50%. It made all sorts of things possible. We couldn't have the screens we have today, or white-light LEDs that are so much more efficient, without this missing composable piece.

My hope is that AI ends up becoming a fourth fundamental logic gate. There are three fundamental logic gates that all software is built on top of. They're so dumb, but it's the brilliance of software engineers that enables you to use these bitwise comparisons to create everything in software today. What if we could add a fourth one? I think that should expand what we can do by way more than 33%. It should expand what we can do by hundreds of thousands of times. That's why I'm excited about a Cambrian explosion specifically of software.

Harry Stebbings

Why do you think the current state of software is kind of a tragedy? If you look at enterprise software today, it's enterprise software with a sprinkling of AI at best.

Diogo Almeida

Yep.

Harry Stebbings

Can you talk about how you feel about that?

Diogo Almeida

I think it's actually extremely logical. I don't think it's a distribution problem. I think enterprises have so much incentive to adopt AI, and the people who have adopted it the hardest were the ones who were burnt the most. The argument I always make is that LLMs are unreliable, so they cannot be trusted for decisions with stakes to the company.

Harry Stebbings

Mm-hmm.

Diogo Almeida

But you can always make them make decisions with stakes to the user, because companies tend not to give a shit about their users. Tragic, but that's what happens. So what ends up happening is that you don't have AI deep in the logic of your apps. What you end up having is the same B2B SaaS with AI attached to the side. It cannot do high-stakes things, but it can shoot slop at the user, who kind of likes it because it's responsive, and the burden of checking whether it's right is on the user's plate.

This is actually extremely logical given the state of reliability. I think it's tragic that people wasted so many resources going the roundabout way to get to where we're at right now. I also think this is the right spot given how smart and reliable the models are.

Harry Stebbings

Do the biggest benefits accrue to the incumbents who have distribution? When you look at a Salesforce of the world and reflect on what you just said about the tragedy of the current state of software, do the benefits fall to those who have distribution, or is there a new generation that benefits?

Diogo Almeida

Number one, the SaaSpocalypse is obviously not happening. Martin Casado coined it as a SaaSpalooza, and I love that. I don't necessarily think it means big SaaS wins. I think the people who are best positioned to win are the people who care the most.

SaaS has a massive advantage over AI companies in that they're actually familiar with the workflows. They have the scale to make the investment upfront to do the engineering, because it takes real engineering to automate stuff and think about everything in order to actually solve people's tasks. That's the real valuable thing here.

I don't necessarily think big SaaS will always be the winner, but I think they're best positioned—not at all due to their distribution, but because they're the ones closest to the problems. They understand, “Hey, if we're at, I don't know, Intuit, if we're in TurboTax, where could we change the interface to make it smoother?” There's not a chance anyone at OpenAI or Anthropic knows how to do that, right? They're still living in their ivory tower and thinking that software just isn't going to exist as a concept. So—

Harry Stebbings

Right.

Diogo Almeida

—they could be the big winners.

Harry Stebbings

You said there about knowing the processes, knowing the workflows, and the importance of that. I've had Lin on the show from Fireworks, and I had Eric on the show from Modal. They both talked about the importance of owning your own intelligence as a company, and the benefits of having this incredible proprietary data—

Diogo Almeida

Mm-hmm.

Harry Stebbings

—and these specialized models on a per-company basis. Diogo, if the importance is in the process and the workflow, surely this is the Holy Grail, no?

Diogo Almeida

Oh, man. How do I say this elegantly and Britishly? That's an opinion, I think, that fits the current rails of how the world works and thinks about AI, where AI is being presented as a coworker. This is the direction Anthropic is really pushing toward, where it's kind of the same interface as a human being.

I believe this is going to be like the horseless carriages of the AI era. Just like in the Industrial era, people were like, “Let's just make it look like a carriage, but without the horse.” I believe AI coworkers are a form of that. It just makes sense that if you're doing automation, automation will have a different strength profile than a human, and you'd want to lean into the strength of automation.

Harry Stebbings

Mm-hmm.

Diogo Almeida

In the Industrial Revolution, you don't need to wait for a whole coworker, like a humanoid robot, to do tasks. You just find the extremely narrow thing, like twisting a bottle cap onto a bottle, and then you automate it, and it just gets solved forever. I think that's how work will generally go.

If you assume AI coworkers are the way to go—which I absolutely do not—and you assume most companies have the willingness, capabilities, and desire to train them because they need to be trained in distribution for your company, then I think the direction of owning your own intelligence can kind of make sense. I do think the labs' approach of an AI coworker with general intelligence makes a lot more sense to me.

Pretty much any company, if you gave them infinite interns that cost 1,000x less, could probably find a lot of productive uses for them. We as a field should answer the question: Why aren't we getting that effect right now? The people who say you need to own your intelligence say, “Oh, you need something like continual learning,” or, “You need something narrower,” and I just don't think that jibes with reality.

Harry Stebbings

Fireworks has helped several top-tier, brand-name firms in the Valley. I think of Harvey, Ramp, and Cognition—I might have got some of those wrong—building their own models. Are you saying it’s a mistake to pursue that route, and that horizontal or alternative models will be just as good, so it’s basically a futile exercise?

Diogo Almeida

It’s complicated. Number one, benchmarks are incredibly misleading, and the people who make them have a lot of perverse incentives to make their models look better internally and, from an external narrative perspective, to not seem like they’ll be completely boned by the large models. I don’t know how good they are. I would imagine they could get cheaper, for sure, and more specialized. I think this is a sign of maturity, when we’re plateauing at a certain capability level.

Harry Stebbings

Uh-huh.

Diogo Almeida

I think AI tends to improve in more discrete steps, and those steps occur when you start getting data in a particular distribution. I think it’d be hard to deny that the largest jump in coding agent performance happened around November-ish last year.

Harry Stebbings

Sure.

Diogo Almeida

I can’t remember which Opus that was. That was when they put it in distribution. Everything since then has been diminishing returns. We could talk about how we love Fable 5.2 or Opus 5.5, but it seems like there have been diminishing returns since then. When you’re seeing diminishing returns, you can start doing micro-optimizations, like cost per token.

My view of the world is that these are all symptoms of not reliably solving problems. Software engineers I know—and I’m a software engineer too—hate updating dependencies. We hate going to the latest and greatest stuff. The only reason we would update is because it didn’t work sufficiently before. Otherwise, you’d just lock it away in the basement and only update it when it gets way cheaper.

I believe we have a lot of insufficient automation right now, and that’s what I think the data most suggests. I’m biased, of course. My view of the world is that, because of Jevons and Jevons-like models, we’ll start locking in real automation, and people won’t want to update that. Companies won’t get their own proprietary intelligence by having their own trained models, because that’s hard to do; they’ll get it by having software around them.

The reason it’s so important for companies like Harvey and Cognition to say they own their intelligence is that otherwise they’re a very light wrapper on top of the LLMs, and it looks like they have no moat. But software and engineering, even if it wasn’t a moat, is a phenomenal thing. Investing up front to understand real problems and automate them so they never have to be solved again, and humans don’t have to do that roadwork, is so good for the world. I think that’s a much more plausible way for companies of the future to progress, and their proprietary intelligence happens at the code level. It’s kind of like how there may be some moat in which database you use, but the real meat is how you use those databases.

Harry Stebbings

Diogo, I’m fascinated. I know the Harvey legal market pretty well. Do you think they’re a thin wrapper? To me, it’s a very specific domain that requires deep product functionality that can only be done in a vertical fashion, and that’s why they exist and are very successful. I’d push back strongly on them being a thin wrapper.

Diogo Almeida

Thin is all relative. I doubt that sounds politically correct right now. It depends on how deep I’d want it to be. My understanding, last I checked with Harvey—I obviously can’t even afford a license, so I don’t fully know—is that, back when Harvey was working with OpenAI, they had this dream of automating larger and larger projects autonomously. That kind of thing didn’t end up working. They ended up having to pivot to a copilot for lawyers, which, my understanding is, they still are.

You can do a lot of domain-specific things as a copilot for lawyers, but there’s only so much engineering you can do around a copilot. Is it domain-specific MCP tools to pull up a RAG based on actual law? The reality is that people want to automate large chunks of work, not just make people double-digit percent more efficient, which is where a lot of AI tends to land.

I think it’s awesome that they’re doing a lot of engineering around it. I think there’s going to be a lot more engineering. Even system messages are a form of really ghetto engineering, where you’re programming the LLM to behave a certain way, but you have no guarantees, no confidence, and no real way to test how effective it is other than the very weird evals people do these days.

I’d like more engineering absolutely everywhere, especially things that are AI-adjacent. I hope we can one day move past copilots and just automate work that takes weeks to do in the background. Maybe that will be shaped very differently from the work humans do now, because automation tends to require adapting to what’s technologically available.

Harry Stebbings

You said that earlier about the cost of tokens, literally at the beginning of a two- to three-minute talk there—

Diogo Almeida

Oh, shit.

Harry Stebbings

—you said about how—

Diogo Almeida

Am I that bad? Oh, shit.

Harry Stebbings

No, dude.

Diogo Almeida

Okay.

Harry Stebbings

No, I love it.

Diogo Almeida

Yeah.

Harry Stebbings

It’s fantastic—

Diogo Almeida

Yeah.

Harry Stebbings

—but the trouble is, I have to remember parts of what you say. My question was going to be: when you look at raw intelligence today, is raw intelligence per token actually improving like we’re told it is?

Diogo Almeida

Can you share your take on that? I actually want to write a blog post on this.

Harry Stebbings

No, I don’t think it is anymore. I think it improved dramatically. I think raw intelligence is still improving, but minutely, to a point where you wouldn’t know. It’s kind of like meeting a 180-IQ person versus a 190-IQ person: you’re not going to tell the difference. So, yes, but not significantly.

Diogo Almeida

I actually think even the 180-to-190-IQ-person analogy is extremely generous to what LLMs are doing right now, because I wouldn’t call them 180-IQ or 190-IQ people. I’d call them occasionally 180, maybe. I don’t really know. But sometimes you get an 80, or even worse.

My take is that it’s not a single dimension. Number one, benchmarks don’t matter, so we should toss them out. If you try to look at dollars per benchmark over time, that’s complete bullshit. Number two, I think there’s a general notion of intelligence, which is pre-trained intelligence per dollar of compute, and, to my understanding, that’s barely moving right now.

As you get more data and focus on different subsets of tasks, you can make them more in-distribution, going back to the Bitter Lesson, and get way better performance within that data distribution. Are we massively moving the needle on intelligence per dollar in coding? Super-duper, but not because the base models are intrinsically more general. It’s because we’re putting coding more in-distribution, so they’re more reliable at it.

Harry Stebbings

Mm.

Diogo Almeida

But has writing improved in the last nine months? I don’t think so. Maybe I’m writing with them wrong. There are many other capabilities where there’s absolutely no improvement at all. If you keep scaling up data in coding agent land, it’s not going to improve lots of other forms of data. There are diminishing returns to it. So it’s complicated and nuanced.

Harry Stebbings

You mentioned the variance in model capabilities: sometimes 180, sometimes 80. When we look at the different models, we see open source, on many different levels, getting as close as possible to the frontier in a lot of respects. How do you see the open model landscape? What don’t many people know that they should know?

Diogo Almeida

Oh, boy.

Harry Stebbings

And what do people not know that they should know?

Diogo Almeida

That last question’s a hard one. Yeah, I think the thing people should know is that benchmarks are total bullshit. All benchmarks are bad. There are perverse incentives. Ironically, the people it hurts most are those who don’t have time to actually play with the models, because they’re the ones who get manipulated by the benchmarks themselves, with higher stakes, including for everything open-weight and open research.

There are ways of measuring intelligence. It’s very hard. Labs don’t publicly talk about it. They have some proxies internally, of course. It’s a very unfortunate public state of affairs that there’s so much misinformation out there. People aren’t incentivized to release their own best measures of intelligence, which I think is tragic.

Harry Stebbings

Diogo, who bullshits the most?

Diogo Almeida

Who bullshits the most?

Harry Stebbings

You said to add spice, so I’ll give you spice.

Diogo Almeida

Oh, okay.

Harry Stebbings

It—

Diogo Almeida

The top-tier labs are incentivized to be truthful. I believe that OpenAI and Anthropic are mostly truthful. The second-tier labs are incentivized to lie to themselves. They want to think they can become a first-tier lab, and they have a lot of internal pressure to benchmark-max. But they cannot admit that to themselves, otherwise their org chart will go ballistic.

This is what happened with Llama 4, where they were actually too on the nose about benchmark-maxing, even though there was so much internal incentive to do that. The same thing is happening with Google right now, and with a lot of the other Chinese labs. I think they’re trying to aim for the top, and therefore they’re not trying to explicitly benchmark-max. I think a lot of the people fighting for consolation prizes are doing completely insane levels of benchmark-maxing.

One domain, if I were to be a little spicy here, would be the rough domain of an American open-source leader. It’s enough of a side consolation prize that they’re not aiming for the top tier. That is pure bullshit and optics, because no one actually uses those models. So who’s going to call them out?

The decision-model landscape is so much worse, in my opinion, because no one even understands what good performance is, so they’re all benchmark-maxing to the absolute gills. It’s kind of really good for us because no one will believe those benchmarks. It gives us a lot of power over the benchmarks themselves, which I think is optimal for the field. In the ideal world, no one ever cares about benchmarks on any machine learning model, string or System 1. But there’s so much of that right now.

Harry Stebbings

Let’s unpack this, dude. Do top-tier labs have a duty to be truthful, or is it in their interest to cheat?

Diogo Almeida

No, sorry. It’s not a duty. They do not need to cheat in order to be high up there. They truly want to be top tier, and they can be. So they don’t need to cheat, is the answer.

Harry Stebbings

I think there’s this feeling in the community that we need to have an American open-source leader, and we need to propel this company forward, whether it’s Reflection, Poolside, or Thinking Machines. Do you feel that we need to propel an American open-source model?

Diogo Almeida

Who is “we,” and what does “propel” mean?

Harry Stebbings

I think there’s a consensus that Chinese open-source models are very efficient. Unless we have an alternative to them, Chinese open-source models will be used considerably and threaten some elements of American security.

Diogo Almeida

It’s actually interesting to dig into the threat model. There’s definitely a threat if you run them vanilla, as they’ve been post-trained. But if you’re actually using—I can’t remember how you described it—the private intelligence of companies, in theory—

Harry Stebbings

Mm.

Diogo Almeida

If you’re training the models, you can remove a lot of bias to different extents. In theory, this type of thing is totally doable. So it depends on whether the threat model is literally plug-and-playing these models.

Harry Stebbings

So you don’t think there are threats from Chinese open-source models in terms of dangerous prompt injections you don’t know about, or back doors?

Diogo Almeida

I would not use a Chinese model in my coding agent vanilla, just to be clear. But that, again, is one threat model. If you live in a world where a coding agent is the only shape of AI, then sure, that is very much a threat. But the alternative we need to take into account is: how good are the American open-source models, and can they catch up to the Chinese models? My understanding is that they hold themselves to a higher ethical and legal baseline, and it might be very hard to catch up to those models because of that.

If the models are actually not useful, it really depends on how much of a difference it makes to have a slightly smarter model, which is going to be hard to forecast, especially as things are plateauing. But it might be a fool’s errand if they cannot be competitive. That will have to be an open question for the field if you assume the extremely narrow view of plugging them into coding agents literally as is.

If you imagine a world where there are many other alternatives, we can measure bias, have threat models for bias, train the bias out, and put guardrails on it. This is actually a huge use case for us right now, which is Jev classifying whether the tool calls are safe or whether they’re phoning home to China, or whatever else. There are so many other ways to get security that I think everyone has been wearing these mono-model horse blinders for so long. I just reject that premise outright.

Harry Stebbings

Do you think American open-source models will catch up?

Diogo Almeida

In terms of true intelligence, my guess is no, unless China decides to stop trying. Not because of capabilities, but because of ethics. But it might not make a difference. If the field plateaus hard enough, maybe that X percent better doesn’t matter.

Harry Stebbings

Do you think we’re right to have the export ban with China?

Diogo Almeida

Oh, boy. My understanding is that there are different parts of the—I don’t know if you mean the GPU export ban specifically, or—

Harry Stebbings

Yeah, the GPU export ban specifically. Because they’re showing—

Diogo Almeida

Oh.

Harry Stebbings

—on some of the latest DeepSeek models and some of the latest Chinese models, they were in fact trained with the latest Blackwell GPUs.

Diogo Almeida

Oh, yes. That was Will Depue’s result. I’ve been talking about that with my friends. There’s some spicy stuff online, though. What I heard online was that using the same technique, you could tell DeepSeek 4.1 Flash wasn’t trained in that way.

Harry Stebbings

Hmm.

Diogo Almeida

My job is to navigate all of the insane hardware bubble and neocloud craziness in a way that we end up getting the future we want. Because we’re so efficient, we can be on the sidelines and not be a main player in the GPU stuff, and hopefully ride out whatever crazy stuff is happening right now.

Harry Stebbings

The crazy stuff that’s happening—you mentioned that, and neoclouds. Dude, it’s a wild space in the neocloud layer.

Diogo Almeida

Hmm.

Harry Stebbings

What happens in the next 12 months in neoclouds that people aren’t contemplating or seeing?

Diogo Almeida

Well, I believe AI has been a bubble for a while, so I’d like to keep planning for the eventuality of a bubble burst.

Harry Stebbings

What will happen in the bubble burst? To me, it’s not going to be a supply-side issue; it’s just going to be a demand-side issue. We’re seeing no demand-side shock. So what will cause a bubble burst?

Diogo Almeida

Oh, man. Okay, we can get into that. That’s a spicy one. Coding agents have definitely changed the game for AI, but this argument has been made for many years now. I think it’s very interesting when you combine many different exponential-variance-type things.

I believe some amount of demand is caused by FOMO. FOMO is caused by people thinking they have good results and thinking they have to go all in on this type of thing. I do believe demand can burst quite quickly when people start facing the realities of what AI can do.

My experience of a lot of the world of people spending money on AI is that they do not have a full understanding of AI’s capabilities. They’re quite miscalibrated and actually overestimate what AI can do. I think that’s the kind of thing that can cause a demand shock. I think there’s just been so much overpromise and underdelivery in the AI space.

Harry Stebbings

That goes against everything people say, which is that if model progression did not improve for a significant amount of time, there would still be so much value left for humans to extract that they haven’t extracted yet. That kind of goes against what you’re saying.

Diogo Almeida

I believe the people who say that are the people who try to extract value out of existing models. So there’s a little bit of bias in that. I don’t disagree with the spirit of what they’re saying.

Harry Stebbings

Okay.

Diogo Almeida

Which is, “Oh my God, the models are so fucking smart.” But again, we need to look at—I agree that they’re so fucking smart. I just think their interface, and how that smarts is unearthed, is suboptimal.

Again, we need to look at the flip side: the people who went way too all in on AI and got burned by it. There are a lot of companies that are like, “Let’s fire a lot of our workforce because AI is solving all of these problems,” and they need to backtrack on that. That has a lot of long-term ramifications for the future of their companies.

Those people tried not just to extract the maximum or overplay their hand; they tried to bet logically on the reality that AI can do a lot more. I think that reality—or that story—is very nuanced when it comes to whether people have the capability to extract a lot of that value, even if the capabilities pause.

Harry Stebbings

I think a lot of people look at the margin structures of a lot of these companies and say that it’s bubble-like. Do you have good margins?

Diogo Almeida

We do, yeah.

Harry Stebbings

How do you get to scaled revenue, dude? You’re so cheap, which is brilliant.

Diogo Almeida

Mm-hmm.

Harry Stebbings

And the cost is a huge benefit of using Jev. But the scale required to get to a $10 billion revenue business—

Diogo Almeida

Yep.

Harry Stebbings

—is just enormous.

Diogo Almeida

Yep, yep, yep.

Harry Stebbings

How do you help me get there?

Diogo Almeida

Jevons paradox, right? The way Jevons paradox tends to work isn’t just magic: you make it cheaper, and it gets better. That’s an insane way for the world to work. The way it should work is, “Hey, I have this budget for AI. Jev makes it that much cheaper, so now I can make this project work. And ideally, it works so reliably, because Jev’s also smarter on the axes that matter for automation, that now I can do these other projects, or I can throw intelligence at things I would never throw intelligence at.”

Even then, if you’re starting to throw intelligence at things you haven’t thrown intelligence at before, that wouldn’t make you increase your budget. Jevons paradox means increasing the budget. What you would need to see is so much value from those projects finally succeeding—which AI tends to fail at—that you’re like, “Hey, this actually works. Let’s keep betting until the budget matches what it was before, and even increases beyond that.”

This is why reliability of intelligence is so important. What I like to say is that, in the grand metrics that truly matter—which is real automation—Jev, Fable, and Astra are all tied at roughly 0%. Yes, there are a couple of little bits here and there, but we have so much automation to do. I think we’re trying to make an industry that currently doesn’t exist, which is run-in-the-background AI.

And it would clearly be—huger? Is that a word? Bigger?

Harry Stebbings

Larger.

Diogo Almeida

Than—

Harry Stebbings

Yeah.

Diogo Almeida

—larger, bigger than a coding agent assisting software engineers, a la making software 20% more efficient to write or something like that.

Harry Stebbings

How do you analyze the coding agent market today? You see Cognition in fourth place so far, a $1.6 billion revenue business. The depth of this market is so large. How do you analyze this market today, and what does that look like in 3 years’ time?

Diogo Almeida

Mm-hmm.

Harry Stebbings

Who’s third place?

Diogo Almeida

Well, you’ve got Claude Code, you’ve got Codex, you’ve got Cursor—

Harry Stebbings

Ah.

Diogo Almeida

You’ve got—

Harry Stebbings

Okay, cool, cool.

Diogo Almeida

—Cognition.

Harry Stebbings

Yeah.

Diogo Almeida

You’ve got Factory.

Harry Stebbings

If I’m totally frank—

Diogo Almeida

Yeah.

Harry Stebbings

—I don’t analyze the coding agent market. I personally want Jev to be used for coding agents, and I think there’s very cool sci-fi that could be done. I actually think that the farther away we stay from the coding agent market, the more they keep doing their little fight, and we can keep focusing on automation.

From a company perspective, it might even be better in the short term if Jev had absolutely no coding agent users, in order to make it a little bit easier. It’s a little too late for that, but I can’t tell. I don’t analyze the size of that market. I think it will be a very large market. Will the market be much larger than it is today? I don’t know, because FOMO really moves the needle on these things.

I’ve been around AI long enough, and surely you have too. You must have seen that AI looks like this exponential upward trend, but it’s actually been many ups and downs. Each FOMO of the previous up adds to the next up. But a lot of them have just crashed down. I think it would be kind of arrogant to assume this is the one thing that will just be up, up, up, up.

Diogo Almeida

Sure.

Harry Stebbings

I so remember 2015, 2016. There was real hype for about a year, and there was Bloomsbury AI, Message.ai, and a load of these companies. It was exactly that, so I completely get you there.

3. Raising For Strategic Allies

We talked about the bubble-like moment. With the greatest respect, you raised a shit ton of money.

Diogo Almeida

Sure.

Harry Stebbings

Tell me about that round. Was it just stopping off at one VC, collecting a term sheet, stopping off at another VC, collecting a term sheet? “Ah, I’ve got 28 term sheets.” How did that round go?

Diogo Almeida

It was a very, very confusing time for us. It was kind of insane. Oh my God, did we learn so much about VCs themselves, because—

Harry Stebbings

What did you learn?

Diogo Almeida

How technologically advanced the VCs’ portfolio companies were. Basically, it was all inbound. We didn’t have a deck. We had to tell people, “Hey, we don’t have time for this deck, and it doesn’t seem anyone else needs this deck. You tell us about the market signals you’re seeing, because everything’s on fire right now, and we don’t really have time to pull these things up for you.”

It was so sick to see how things evolved for different VCs over the first days and weeks: how their excitement changed, and how they told us their portfolio companies were using us. We don’t snoop on their data, so we actually don’t know what they’re using it for. It was so cool to hear them say, “Holy shit, you guys are the real deal. You’re crucial. This is the future.”

That was a really fascinating additional signal. Honestly, I missed that signal on the state of the field, and it was so fucking cool to hear about who’s using us and why and how. That is what enabled the round to work out.

Harry Stebbings

Did you start with $750 million as the ask at $7.5 billion, or did it move up?

Diogo Almeida

It moved around a lot. When we even started the conversations, we didn’t know what we were really looking for in the round. What we ended up deciding was, “Hey, we don’t actually—this is maybe kind of crazy, because we did raise a lot of money, but we realized we don’t need to valuation-max. We don’t need to do a cuck tranche or anything like that. We don’t care about that. We don’t actually need a fuck ton of money either.”

What we need is allies to fight the largest companies in the world. Let’s find the best people to be part of the team. That’s what we really optimized for. The rest, to me, was more in the micro-optimization territory, versus discovering, “Hey, we want the smartest possible money, the best allies, the people who can help us navigate the rocky waters ahead—the rough waters ahead.”

Harry Stebbings

Yeah.

Diogo Almeida

We were optimizing for very different things from other companies, because we are profitable. Money in the bank is going up right now. Quite frankly, a company that’s raised as little as we’d raised before should not have a plan to spend whatever we raised right away.

Harry Stebbings

I can’t believe “cuck tranche” is actually a word these days, but let’s roll with it.

Diogo Almeida

Wait, you know about that, surely.

Harry Stebbings

I do. It’s completely fucked, and it makes me—yeah, absolutely. I have a very big ego.

Diogo Almeida

Yeah, and that came up for us. We had so much demand for that. People were even saying, “If you’re going to be raising at 3x the valuation, count me in.” The thing is, we didn’t need the money right now—

Harry Stebbings

But that’s a bubble. That’s bullshit. I’m sorry, I really like— For someone who’s like, “YOLO, $21 billion, I don’t care.” You know what? You just need to get some self-esteem, because that is like sitting and watching.

Diogo Almeida

We don’t need the money or the valuation.

Harry Stebbings

Why did you choose Sequoia and Andreessen? I love them, but why did you choose them?

Diogo Almeida

Oh, man. It’s going to seem like I’m sucking up, but I love them a lot. I’m kind of insane about technology and a techno-optimist. I think a16z is just the embodiment of that. They’re as philosophically aligned as we can be.

I feel like most VCs are, from a strategic perspective, lovers of technology, but they’re financial institutions that care a lot about how to play their financial cards. Whereas Andreessen, deep into the firm, they care so, so, so much. That podcast I did with Martin Casado and Ben Horowitz, I had so much fun. It felt like we were old friends talking from our hearts.

Maybe they’re so good that they can convince me, and they’ve sold me on some fake-ass thing. But I don’t believe so. I think they’re just so philosophically aligned. What led us to where we’re at right now, and I hope we can keep being this way, is wearing our hearts on our sleeves about what we care about and just blazing our own path in the AI field. I’d love to act in accordance with that and have them on board. Obviously, Martin is brilliant.

Actually, to me, Sequoia—if I’m going to be totally frank, man, how do I say this? I didn’t want their money; I wanted Sonya Wang. That is why Sequoia’s around. I didn’t get to know Sequoia super well as a fund, but I love Sonya as well, and I just love that we have—

Harry Stebbings

Now help me understand. I want to be the best ambassador. What did you love so much about Sonya that made you choose her over everyone else?

Diogo Almeida

Oh, man. I would say that she’s not just brilliant, but a very unique type of brilliant that I think we would really want on our side. It’s that particular shape of brilliance that I find that, no matter how many advisors I would talk to—

Harry Stebbings

Mm.

If I have a conversation with Sonya, I get a different, very interesting perspective that should be part of our strategic cards. I care a lot about strategic thinking for myself. One of my spicy, type-safe values is “make great decisions slowly.” It’s very non-startup, but I think that’s how it should be done, and she’s an incredibly valuable thought partner there.

4. Hiring For The Mission

Harry Stebbings

“Make great decisions slowly.” What bad decision did you make fast?

Diogo Almeida

I would say I probably make decisions slower than most other people. My sense is that a lot of the business world is on autopilot. I hate working with traditional business people. They just do things, and I’m okay with doing things if it doesn’t matter how well they’re done. But for important things, I actually care a lot about the why. What’s the philosophy? What’s the framework? How is this done?

Whenever I work with traditional business people, I feel like I’m being driven crazy because they never do things so well that I don’t need to worry about them. And when I do worry about something, I need to understand the why, and I’m like, “Oh, this person’s just doing random shit.” I think having a plan actually matters quite a bit. There are so many things I would spend way more time thinking about.

I wish I could spend more time thinking about our data and our data roadmap. I want to think more about product. I want to think more about how to be more trustworthy. I want to think more about every single role we hire for, to make sure we get the best possible people for those roles. I think I should acknowledge that I don’t have time for all of these things, and it’s so fucking important to think, think, think.

Harry Stebbings

How hard is the hiring market today? You have to get the best people on your team. How hard is it, actually?

Diogo Almeida

The hardest part is hiring for our data domain team. Back to the Bitter Lesson: when you have a new task, you have a new type of data. You need a new shape of data in order to make it work. We basically have our own little subfield. Imagine the best lab in the world, but internal to a company, to make data the way we make data.

I would think this should be the easiest thing in the world to hire for because it’s so unbelievably cool. But we have such a hard time hiring for it because no one knows how to get into it. We keep getting the wrong type of people applying. That, to me, is the thing that keeps me up at night these days: how can we convince brilliant people who don’t think they can push the boundaries of machine learning because they’re not ML researchers to work in this role, where we have a subfield and they’ll get tutored by the experts in the world at it? It’s going to be so valuable because you actually make the model smarter. I don’t know how to communicate that to the world, and that, to me, sucks. Everything else is good.

Harry Stebbings

Is the comp wild? We had Eric at Modal say he has people he has to fight $20 million packages for—some grads who are utterly brilliant. Are comp packages just unprecedented?

Diogo Almeida

Not for us. I’m so happy we got clarity on this. We got real clarity that we do not want mercenaries. We want to be good, but not like the insane lab ones. I think that feels great. Anthropic, I think, is suffering a little bit from over-indexing on mercenaries lately, because they—

Harry Stebbings

Mm.

Diogo Almeida

—want missionaries, obviously, but they also pay more than anyone else. There’s a lot of incentive not just for mercenaries to join, but for people to fake their culture interviews. There’s an entire market now to prepare people for Anthropic culture interviews. I’m so glad we decided not to do that. We want to pay people good amounts, not the highest amounts ever, and we’re super happy with where we’re at comp-wise. It takes care of people and also filters out the hired guns.

Harry Stebbings

Final one for you. If you could bake—

Diogo Almeida

Oh, yeah.

Harry Stebbings

—one character trait into a potential hire, what would it be? For my company, it would be obsessive.

Diogo Almeida

Mm.

Harry Stebbings

I need someone to be obsessive about their craft, addicted to making it better. They can’t sleep at night. It actually compels them in every way. The ability to become obsessed with something.

Diogo Almeida

I don’t know if that’s the same for every role, but if I had to pick, I don’t know a better word for this, and it sounds a little sexual, but we don’t mean it in a sexual way: technophilia. Non-sexual, just to be clear. Non-sexual. Okay, relax. We truly love technology. That’s the kind of thing that’s so beautiful in a bubble, and it’s the kind of thing I want in every single role, so we can talk about technology at lunchtime or when we’re hanging out.

I want people around me that I like, and I want us all to be one big team, constantly pushing the boundaries of sci-fi, because that’s how we actually ship technological shock and awe. Again, non-sexual. I don’t know a better word. Technophile—yeah, yeah, yeah. Just to be clear, if you guys edit this out, that’ll be very embarrassing.

Harry Stebbings

Don’t worry. We’ll keep it in, and no—

Diogo Almeida

Sure.

Harry Stebbings

—no sexual. We’re going to do a quick-fire round now. I say a short statement—

Diogo Almeida

Oh.

Harry Stebbings

—and you give me your immediate thoughts.

Diogo Almeida

Oh, okay.

Harry Stebbings

First—

Diogo Almeida

I tend not to do quick-fire, as a warning. I’ll try.

Harry Stebbings

What have you changed your mind on most in the last 12 months?

Diogo Almeida

How much hype matters.

Harry Stebbings

How much hype matters?

Diogo Almeida

Yes. I think it matters much less than I thought. When Jev launched, it was like, “Holy shit, hype matters a lot.” Then other stuff launched, and they were benchmark-maxing to the gills. When we talked to real people, everyone was like, “Yeah, we tried them. They suck. No one cares. We can tell already that you are the best and you care the most.”

I was like, wow, this is so convenient. We can just do what we love most, which is underpromise and overdeliver. This is the best world ever. I would not have expected this at launch time, and this is so awesome for us.

Harry Stebbings

What is the biggest belief you hold that the smartest people at frontier labs would disagree with you on?

Diogo Almeida

How much AI can do today, and how important reliability is.

Harry Stebbings

Why do you not hire from OpenAI?

Diogo Almeida

My sense is that there’s a certain selection bias you need to get into OpenAI that’s kind of an anti-signal. I feel like you need to be in a particular Kool-Aid, number one. Number two, when I say this, I mostly mean I don’t hire researchers from OpenAI.

Harry Stebbings

Mm.

Diogo Almeida

The researchers at OpenAI, by construction, love to do algorithms and then scaling. They tend to hate data. Data is a very low-status thing at most labs. Obviously, they don’t think about the right task, because that’s quite rare. There are maybe three people at OpenAI who think about the right task. I wouldn’t hire them either, not because they aren’t fantastic, but because if you’re obsessed with the right task, why would you want to do my right task? My right task is machine-native AI.

If you’re obsessed with making text-to-image a thing, which Aditya did at OpenAI—holy shit, he invented an entire field. That is so fucking cool. He is so unbelievably good. And you can’t transplant him into another task. He should just be cooking on his sci-fi too, because that’s what you do when you’re so good at that.

So, by construction, all that’s left of the ML research that could be hireable are the people who are obsessed with algorithms. That, to me, is the kind of thing to do when you want to throw money in the furnace.

Harry Stebbings

What is the most overhyped category today?

Diogo Almeida

My instincts tell me robotics. Not because robotics isn’t cool, but I think that if you look at the grand scheme of reliability, they’re just very far away from it. If you imagine what needs to be nailed first, there are tons of prerequisites that still haven’t been nailed. I think robotics is the furthest away from that.

Harry Stebbings

What is the most underhyped?

Diogo Almeida

This direction of software research, people playing with AI plus software to make new things, especially new interfaces. I think that’s so unbelievably underhyped. It’s tragic.

Harry Stebbings

Final one for you. What do you believe today that looks crazy, but will be incredibly common in five years’ time?

Diogo Almeida

I think principal-agent problems abound in the world. When you have a principal, like an electorate, being represented by an agent that is sometimes misaligned with it, like a representative, why can’t the electorate directly represent their desires and have it behave more like a true democracy? This is just one contrived example, but it happens all over the place. People reporting to you, stockholders to companies—there are so many principal-agent problems.

My view is that we will find way better solutions to all of these, and it’s going to be crazy that we allowed the level of misalignment that we have today. That will be such a sweet world, where we can start enforcing the spirit of the law instead of the letter.

And AI is basically that. The law is kind of like writing software. It’s like writing a detailed algorithm to follow in order to make it fair and reproducible. It was just missing the primitives to specify the spirit.

I think this will really unite society, because even for our most controversial topics, a lot of people can unite on the spirit of things. The problem is that the spirit can’t be enforced, so you need to put the letter, and the letter then has a ton of disagreements.

Harry Stebbings

Do you know my problem with you, Diogo? You just don’t have much energy.

Diogo Almeida

My bad. My bad. There’s a reason they call me Cocaine Labubu on Discord.

Harry Stebbings

Dude, this has been so much fun.

Diogo Almeida

Hell yeah.

Harry Stebbings

Thank you so much for bringing the energy—

Diogo Almeida

Hell yeah.

Harry Stebbings

—and for being so willing to be so open.

Diogo Almeida

Hell yeah. It was tons and tons of fun. We should stay in touch. This definitely was very spicy, spicier than I intended.