无限代码上下文：企业级 AI 编程——对话 Blitzy CEO Brian Elliott 与 CTO Sid Pardeshi

Hello and welcome back to the Cognitive Revolution. Today my guests are Brian Elliot and Sid Pardesi, CEO and CTO of Blitzy, a company that uses AI in just about every way you can imagine to help enterprise software teams implement large-scale features and execute modernization plans with unprecedented speed. Regular listeners will know that Blitzy has recently come on as a sponsor of the Cognitive Revolution. And while this does technically make this a sponsored episode, you can rest assured that this conversation absolutely stands on its merits. In fact, I've noticed over time that my interviews with sponsors often end up being among my favorite episodes. And I think the reason is that founders who've achieved real product market fit are often unusually willing to share the nitty-gritty details of their approach. It's a uniquely effective way to convince prospective customers that they're better off buying from an AI pioneer than attempting to recreate such a sophisticated system inhouse. And it also signals that their product is still rapidly improving. So over the course of the next two full hours, we will go super deep on Blitzy's approach, what they mean when they say infinite code context, and what enterprise software development looks like when more than 80% of major projects can be done autonomously in days. Highlights include the architecture they use to generate agents dynamically just in time with prompts written and tools selected by other agents. Why they actually run enterprise apps in a parallel environment as part of their onboarding process. How they ingest 100 million line code bases and deliver value in the form of improved documentation which also improves coding co-pilot performance even before the code generation process begins. how they use detailed knowledge graphs to support sophisticated context management strategies which minimize models context anxiety and other strange behaviors. The critical role of taste in evaluating new models and framework changes on such largecale projects. Which models they find strongest for which purposes and why they always use models from different developers to check one another's work. Why they are more bullish on advances in AI memory than on fine-tuning. how they came up with their 20 cents per line of code pricing model and why they will do anything they can to deliver more value for customers even if it forces them to raise prices in the future. What it will ultimately take to achieve 99% project completion and even full autonomy in enterprise software development and finally their outlook on the software engineering labor market which favors senior engineers in the short term but junior engineers who can use AI effectively over time. Brian and Sid are both high energy guys and they were remarkably forthcoming in this conversation. I learned a ton and I expect that any enterprise software leaders who listen will come away thinking about specific projects where they'd love to put Blitzy to the test. So without further ado, I hope you enjoy this deep dive into the present and future of autonomous software engineering with Brian Elliot and Sid Pardesi of Blitzy. Brian Elliot, CEO at Blitzy. Welcome to The Cognitive Revolution.

Awesome. Let’s get into it.

One of my favorite things to do in life is talk to AI maximalists. I’ve known Blitzy by reputation for a while as the company that has figured out a way to create infinite code context, and it doesn’t get more maximalist than infinite. I’m excited to unpack what you guys are building, how it all works, and the impact that it’s having on the enterprise software industry.

We’re going to go through all the layers, but first, just to orient myself and the audience to you: How AGI are you? How AGI-pilled is Blitzy? How AGI-pilled are your customers?

We believe we can get AGI-type effects out of non-AGI LLMs, right? As folks are thinking about the impact of artificial general intelligence, they’re talking about huge swaths of work being able to be done to provide economic value autonomously across domains. That’s one amongst many definitions of AGI, and it’s a moving target for defining AGI.

The core question is: How can you achieve that output with the limitations and constraints of LLMs? We might be the most bearish on LLM capabilities as a pure, standalone, single-LLM asset, and perhaps the most bullish on the orchestration of those in long-running, complex systems.

Yeah, that really echoes a conversation I recently had with Daniel Mesler, who created this personal AI infrastructure framework. His mantra is, “Harness is more important than model.”

Obviously, one big limitation there is that the context window is finite, and even at 1 million tokens, relative to the size of an enterprise codebase, that’s not nearly enough. Are there any other limitations of LLMs as standalone creatures that you think are most important to have in mind?

Yeah, there are so many. Being so forward about the limitations is what allows you to build something really powerful and really magical.

Context is one, but there’s a difference between a context window and an effective context window. As you start to eat into, let’s say, 20%, 30%, or 40% of a context window, degradation occurs. Each model is a little bit different, and there are lots of different ways to test this with still-internal benchmarks, but you start to lose intelligence and quality as you start to fill up even the advertised context window. The degradation is a little bit different depending on the task type.

What you want to do is effectively manage the amount and type of work that you’re loading into a context window, while also pulling out anything that you don’t need. That’s a more nuanced view of the limitations of a context window.

The other limitations are how many tools an individual agent can effectively call. It used to be that they could call zero tools, then they could call 1, 2, or 3, and then they could call 8 or 10. Tool selection in the agent itself is also something that you really need to understand, steer, and give only the correct tool access to.

You can think of a tool as a calculator, a compiler, or any outside entity that one is using. Lastly, there’s maintaining the long-running intent of the human—or the intent of the machine or instruction. That’s a byproduct of context management, but it also has to do with attention in general.

If you can design a system that says, “Great, LLMs are a very cool, probabilistic type of computer. They have all these limitations, at least when leveraged as a single instance,” and then accept those limitations and build the harness, or cognitive architecture, you can really create something that can achieve AGI-type effects.

I can’t help but ask for a couple of specific tips, because right now I’m doing the work of building out the context of my own life—pulling out all my email history, my Slack history, all the transcripts of the podcast, and all this stuff into one big data soup. Now I’m trying to layer on various kinds of summaries and look at it from different angles.

In some ways, this is probably quite similar to what you guys are doing with codebases, albeit for me, it’s just my own stuff. I was thinking earlier today, “I wonder how much context I really should put into Gemini Flash, or if that’s the right model.” Maybe there’s a different model where, even though its nominal context window is shorter, I would actually get better results for a given amount of context.

How would you advise me? Are there any topline heuristics that you’d be willing to share, where you say, “This is what we see as the best, and this is where it drops off”?

Let’s put a pin in the point of not just using one family of models at all to do this, and we’ll cover that in a second. Let’s talk about how you manage this information.

Context is serial information; it’s relational. That email connects to something you said in a Slack message, and those might be on different applications. The question is: What are the core relationships that govern this domain?

We put out a paper about domain-specific context engineering, but what’s core is that context engineering is not general—it is domain-specific. There is a core set of entities that relate in certain ways inside the domain of, let’s say, personal life or work life. You have to first understand and define those relationships, and then pair that with semantic understanding.

That’s how you get closer to the context that might be important for any task, while removing the context that is not important for any task. That’s a very broad philosophy, but the idea that semantic clustering is sufficient is really inaccurate.

Yeah. I like where you’re going with this. What I’m doing right now is starting with all this raw information, and then I’m trying to build up layers of higher- and higher-order understanding.

First of all, I’m saying, “Okay, let’s create a timeline.” I’m grabbing whatever I have from all sources, sorted by date. Some might be emails, and some might be podcast transcripts. I throw all that into an LLM and say, “Give me a summary of what I was saying, doing, and thinking about at this given point in time.”

I’m building out a timeline. Then, on top of that, it’ll be, “Who are the relationships that really seem to matter over the course of all this time?” Then it’ll be, “What are the projects that I was engaged with, and which people was I working with on those projects?” I’m building up all those levels right now.

How does that play out? I'm sure it's, again, an analogous thing. How does that play out in the context of a giant enterprise codebase that you guys get your hands on?

Yeah. Well, the approach that you're taking on that personal project will be okay at first and then get worse over time, right? You're at the personal-project stage of a lower-mid-market software application, as I say. You can just shove all the stuff in there and get some approximately right results.

Gates had this quote: if you could schematize the world, you could get a computer to do anything. In your example, you're trying to schematize your life, right? In the example of code, you really are trying to schematize code and the relationships in code, agnostic of language. So, in the case of Blitzy, you could throw a 50- to 100-million-line codebase on it.

Because we have a deep relational understanding that we built first—it takes a few days of compute to build that—that deep relational understanding is the base layer that allows us to do large amounts of development work autonomously. In your example, you'd first schematize your life. That might be dates; it might be months as a group, dates as a group. It might be other activities as a group that relate to other things. But first and foremost, you need to understand the core relationships that govern the domain, and we have done that in a very unique way with code.

When an enterprise starts with us, they ingest their code. It takes a few days of compute, and we then have a deep and novel approach in the category of knowledge graphs. But that's maybe not sufficient to explain how deep the understanding is. At any line across a 100-million-line codebase, I can tell you exactly what is relationally relevant down to the line level, so that when I generate code, I am injecting and pulling out the correct context just in time.

So, obviously, dependencies are one core type of relationship within software. A lot of that has traditionally been done with static-analysis tools, right? There are all sorts of tools that can go through and say, “This file imports these other things, and they import these other things,” so we can kind of fan out that way. What's the breakdown between how much you're using those kinds of static-analysis tools versus an LLM to do this ingestion? And what's the double-click on the nature of relationships that goes beyond dependencies?

Totally. If you think about ASTs, for instance, they're version-specific, language-specific abstractions and syntax. These are a preliminary worldview for understanding the relationships and meanings between a language and a version of a programming language.

You can think of what we've invented as a super-AST—not an AST, but something that resembles the characteristics of an AST, with accuracy that is programming-language agnostic and designed for AI-agent traversal. You can think of globals, classes, variables, and functional relationships inside an application.

By having the traditional programming-language-agnostic relationships on top of actually building and running the application—which we do—as we create relationships, you're able to create a much deeper understanding. One of the powers is that you're really not able to get understanding unless you are building and running applications and putting them through the paces to understand everything from what you said on the left side, which is dependencies, to how things relate when they're run in production and have actual logs running.

You can imagine the spectrum of compile time, runtime, production, and load-time items that a software-development team might look at. Those ultimately form the base of the relationships that schematize enterprise code.

Yeah. Okay, that's really interesting. First of all, do I understand correctly that you are literally running enterprise applications in your own—

Like a parallel universe to theirs, because, of course, your clients are continuing to run their applications in production, right? So you've got to kind of mirror that—

Often in their own cloud environment.

And so they're spinning them up again in their cloud environment. To get started, one of the reasons it takes—not days, you know, a few minutes—is getting access to your environment and all the necessary keys so that you can spin up these applications and run them.

It's cool because when you get large-scale code outputs from Blitzy, you'll also see the QA that we did and screenshots of an agent clicking through and running an application in production. That happens both upon ingestion, to make sure that we can run and build the application, and then at code generation, as we go through QA.

Running the application is core to getting high-quality code because you need a recursive-correction loop—not just when something doesn't compile and build, but when it doesn't act in production the way you're expecting it to.

Yeah, just the feat of managing to actually stand up another parallel instance of the production application is, I'm sure, not trivial in many, many cases.

You need to seed a database, right? There's real implementation work in there, right? Of—

I think there are a lot of times where, because people haven't really done it, or because this thing has been running the way it's been running for a long time, they don't even have a ready plan for how you would do that. A lot of applications in insurance, for whatever reason, just really have no way to provide us these instructions.

So what we'll do is go through this iterative approach, which provides value even in the approach where we'll take the information that they think it takes to run the application. Then Blitzy will find the limit case of not being able to do it, and we'll say, “Hey, we don't have access to this package.” They're like, “Okay, well, I had no idea it depended on that package.”

You're able to go through this process of actually creating the correct build instructions for the application that's essentially been sitting somewhat dormant, which they want to activate or move over into a more modern technology stack as part of getting Blitzy to stand it up. So we've provided value just in implementation, I would say.

But it does come with, obviously, challenges. There are lots of old enterprises, for instance, where, to build the application, it's not as if you're just writing a script or a package. It requires what would have typically been a human, with dialog boxes popping up and information being entered.

But Blitzy is sophisticated enough to spin that up, put in user creds, and run in their VPC to build an application. That's how Windows applications were built back in the day. So it requires real build sophistication in the application to get this level of fidelity.

Hey, we'll continue our interview in a moment after a word from our sponsors. Want to accelerate software development by 500%. Meet Blitzy, the only autonomous code generation platform with infinite code context. Purpose-built for large, complex, enterprisecale code bases. While other AI coding tools provide snippets of code and struggle with context, Blitzy ingests millions of lines of code and orchestrates thousands of agents that reason for hours to map every linele level dependency. With a complete contextual understanding of your codebase, Blitzy is ready to be deployed at the beginning of every sprint, creating a bespoke agent plan, and then autonomously generating enterprisegrade premium quality code grounded in a deep understanding of your existing codebase, services, and standards. Blitzy's orchestration layer of cooperative agents thinks for hours to days, autonomously planning, building, improving, and validating code. It executes spec and test-driven development done at the speed of compute. The platform completes more than 80% of the work autonomously, typically weeks to months of work while providing a clear action plan for the remaining human development. Used for both large-scale feature additions and modernization work, Blitzy is the secret weapon for Fortune 500 companies globally, unlocking 5x engineering velocity and delivering months of engineering work in a matter of days. You can hear directly about Blitzy from other Fortune 500 CTOs on the modern CTO or CIO classified podcasts or meet directly with the Blitzy team by visiting blitzy.com. That's blitzy.com. Schedule a meeting with their AI solutions consultants to discuss enabling an AI native SDLC in your organization today. The worst thing about automation is how often it breaks. You build a structured workflow, carefully map every field from step to step, and it works in testing. But when real data hits or something unexpected happens, the whole thing fails. What started as a timesaver is now a fire you have to put out. Tasklet is different. It's an AI agent that runs 24/7. Just describe what you want in plain English. Send a daily briefing, triage support emails, or update your CRM. And whatever it is, Tasklip figures out how to make it happen. Tasklet connects to more than 3,000 business tools out of the box, plus any API or MCP server. It can even use a computer to handle anything that can't be done programmatically. Unlike Chat GPT, Tasklet actually [clears throat] does the work for you. And unlike traditional automation software, it just works. No flowcharts, no tedious setup, no knowledge silos where only one person understands how it works. Listen to my full interview with Tasklet founder and CEO Andrew Lee. Try Tasklet for free at taskl.ai. AI and use code cogrev to get 50% off your first month of any paid plan. That's code cogrevllet.ai.

You guys have been at this for a few years, right? One big question I had is that the capabilities of models have changed dramatically, particularly in their ability to look at a screen and understand what's going on. I think we saw that demoed for the first time with the GPT-4 launch, but it was still pretty rough around the edges and wasn't really even available after that. The computer-use benchmarks were kind of in the steep part of the S-curve.

I remember, fondly but also with frustration, the experience of early computer-use agents. Even if they could see the button, they couldn't necessarily click on it; they couldn't quite find the right place to click. That stuff has all improved dramatically. How do you think about turning Blitzy on itself? I recently did an episode with Andrew Lee from Tasklet, and he's another AI maximalist I really enjoy talking to. One of his mantras is, “Speed in the AI era—speed is the only moat,” and he takes a lot of pride in how fast they rebuild their stack from the ground up.

What would be the big unlocks that you've seen? Models couldn't do this before, so you had to do all this stuff to compensate. Now they can, so you can simplify that or aim higher in terms of what you could do. I'd be interested in what those big milestones would be as you look back, and how often you find yourself having to do major modernization work on your own stack, even if that modernization is only a few months from the last version to the new version.

It's such a good question. When we started building Blitzy in 2022, we essentially made a bet that the models were going to get faster and way better than anybody in the market expected them to get. We started building for a future universe that wasn't here when we were doing all of the design and all of the work. There's no MVP of it; it's an end-to-end platform experience, right?

The world that we built for over the last 3 years and the world in 2025 essentially intersected. These things were going to continue to get really, really good, and we were correct. When you're building systems for an ever-improving state of LLM intelligence, you want to build the systems dynamically.

When people talk about building harnesses, they're sort of hard-coding and codifying actions based on the level of LLM intelligence and capabilities. Those harnesses deprecate as LLMs get better. The level of deprecation is tied to how hard-coded your design is, let's say, and the rate of intelligence increase.

Everything that we do in Blitzy is dynamic design, meaning Blitzy's agents are generated dynamically, just in time. Prompts are written by other agents. Tool selection is assessed just in time by context injection. The whole planning process that governs all of this is chunked and revisited iteratively.

As the models get better, it's just great for us. We can more or less just do more, and it's a config file to toss in a different LLM. Because everything inside the system is dynamic, we don't feel the deprecation that one would typically feel when building harnesses in the classical way that people build harnesses today.

As a new model comes out, new prompting instructions for that model come out, right? Aren't agents just referencing the latest prompting instructions tied to their model and then self-writing a prompt for another agent that's injected? It doesn't matter that the prompt guidance changes for the next Gemini model. We'll just go reference that. The agent will just go reference that as it's dynamically writing a prompt for another agent.

That sounds awesome. It sounds like you're living the dream in many respects. One thing I do wonder about there, though, is how you evaluate that. The typical harness, as I can speak to from when I tried this sort of personal AI infrastructure at various points in time, always felt like it wasn't really there to give me tremendous value. I think now we've maybe hit that point.

As I look back at some of this old code, I'm like, “Oh my God, 8,000-token context windows when I first tried this.” That was so limiting, and I was doing so many gymnastics to try to make that work. One benefit of those gymnastics—or at least one thing that was easier—was that I could at least define an eval test set that I could wrap my head around and look at and say, “Okay, this makes sense in terms of inputs and outputs.” I could throw a new model at that and get a quick sense of whether it was better or worse.

When so much is dynamic, how do you think about evals? One thing I could imagine is that you might do some fixed evals as prep work, like characterize the effective context window of this new model and then tell itself what its effective context window is. Give it some sort of metacognitive information. But you've probably got lots of other insights into how to eval such a dynamic system, so I'd love to hear it.

I think it's important that your evals map onto the real world as closely as possible. Most evals are very easy to evaluate, right? Here's a function, and here's a different version of that function, and this other version is more accurate. But that is a local optimization on an exponential technology.

Our evals are a bunch of larger-scale applications that we've built over the years. Some of them started in open source, and we built our own versions of private applications over the years. We're testing Blitzy by executing what we ultimately want to be a 100% outcome, and we're seeing how close we get to that outcome with the new configuration of Blitzy.

We might give it a million lines of code. Maybe we'll give it Apache Spark, which is 1.3 million lines of code. We have a custom configuration of Apache Spark from previous projects that we've done personally. We'll give it those instructions, and we'll be able to see very quickly how close we got to 100% completion with this adjustment.

It requires an extreme amount of taste because if you're not 100% there, you don't get to 100% of the result. The 100% is what you did as a human in a previous life to get that to 100%. You're now saying, “Is this 85%, 88%, 90%, 95%, or 100%?”

There's always a difference between functional correctness and intent. Blitzy can guarantee functional correctness: We passed every end-to-end test, every integration test, and every feature test. But it may not be the very final version that you actually intended to put into production.

That taste is required to really improve the system and provide feedback on top of the traditional large-scale evaluation. This is why I think it's really, really hard to build these systems without the right longitudinal experience to understand what great technical design and great software implementation are like.

We always say Blitzy is the instantiation of the idea that you could have Sid—my CTO and co-founder—work at the speed of compute, because he's instantiating his technical taste into the outcome in a way that's really, really impressive for the enterprise. Of course, they can specify their own taste and their own rules, and the system will respect that. That is how we do evals: at scale, at a very large point, with a lot of taste involved.

So that final taste—if I'm looking over Sid's shoulder as he's evaluating the work of a new model thrown into the Blitzy meta-harness—what am I seeing him doing?

You are looking at the final output, but really, you're looking at the logs. We use LangSmith for logs and tracing. Shout-out to LangSmith. I'm a big fan of the LangChain guys; this is their tracing product.

If you were to type the logs out on a piece of paper and put them on a scroll, that scroll would go all the way down to the end of the block. The amount of agentic interactions that occur at runtime is absolutely massive. You have somebody injecting context, somebody writing a prompt, somebody writing code, somebody reviewing that code, somebody building the code, and somebody doing before-and-after checks—local pass-to-pass, fail-to-fail, and end-to-end.

That's happening to get a piece of functionality out in the bigger system. As you see these agents interacting, it's a lot like looking at your engineers having a technical discussion about what “correct” might look like. What you need to do is look at the final output of the meta-harness—the pull request here—and then trace back in the system.

I didn't like what happened here. What happened in the system, right? And how can I steer the system to dynamically address this kind of instance in the future? It's a completely different approach to building software because the outcome is a little bit emergent, in a way, and you have to build the system to understand how to dynamically steer and validate to get to the right outcome.

What does that steering process look like? Is it just giving the system text, like free-text feedback?

You try to be as algorithmic as possible, right? As you think about chunks of work being completed, the first step that we'll take after receiving a future-state spec from the client—which is where our system will work with you to create a future-state spec of what you want—is to send that web application portion off to do work, right?

Then you start a planning process, or Blitzy starts a planning process, and then it executes against that plan. Each one of those planning steps, and each chunk of work—planning, reading, testing, validating, QA, and doing that recursively—is driven algorithmically to get to an outcome. It's tweaking the algorithms that govern the system to get to the right outcome.

Fascinating. Hey, we'll continue our interview in a moment after a word from our sponsors. Your IT team wastes half their day on repetitive tickets. And the more your business grows, the more requests pile up. Password resets, access requests, onboarding, all pulling them away from meaningful work. With Serville, you can cut help desk tickets by more than 50%. While legacy players are bolting AI onto decades old systems, Serville was built for AI agents from the ground up. Your IT team describes what they need in plain English, and Servil AI generates production ready automations instantly. Here's the transformation. A manager onboards a new hire. The old process takes hours, pinging Slack, emailing it, waiting on approvals. New hires sit around for days. With Servil, the manager asks to onboard someone in Slack, and the AI provisions access to everything automatically in seconds with the necessary approvals. It never touches it. Many companies automate over 50% of tickets immediately after setup, and Servil guarantees 50% help desk automation by week four of your free pilot. As someone who does AI consulting for a number of different companies, I've seen firsthand how painful manual provisioning can be. It often takes a week or more before I can start actual work. If only the companies I work with were using Servil, I'd be productive from day one. Servil powers the fastest growing companies in the world like Perplexity, Vicata, Merkor, and Clay. So get your team out of the help desk and back to the work they enjoy. Book your free pilot at servil.com/cognitive. That's sv.com/cognitive.

Going back to the initial ingestion and the knowledge graph that is created, I'd love to hear your thoughts on knowledge graphs, how they relate to RAG, and whether you guys are using embeddings. There have obviously been many different approaches and schools of thought here. I've always been attracted to the idea of knowledge graphs, but for a long time, RAG was more in vogue. Then it seemed like, a lot of times, “Dump everything in the context window” started to become the prevailing approach when possible.

Obviously, that's not possible for large codebases. Are you able to get to the point where you've mapped things out so well that you don't need fuzzy semantic matching? Or do you also avail yourself of that and have something like, “This is what we were able to find structurally that's relevant, and this is also some other kind of relevant stuff that sort of fuzzy-matched that you might want to be aware of”?

You really want to use both as a hybrid source of truth, right? Then, when there are conflicts, the system wants to explore much deeper and much further. The issue with RAG as a standalone item is that sometimes people will rely on the RAG abstraction layer as a source of independent truth.

To answer your first question directly, you want to use both relational understanding and semantic understanding, and you want to pair those as agentic tools so that you can arm the agent to use these different tools to pull the right information. But you really want to use these tools as an abstraction layer to search the source of truth.

You don't want to rely on the semantic match to pull out the truth. You want to rely on the semantic match as a map or a legend against the actual source of truth: to efficiently search, traverse, and find that, and then pull the source of truth into the context window. It's really an efficiency search mechanism more than it is a storage-of-truth mechanism.

One thing I've observed that I wonder how you address is that, so often, when I have an agent searching through whatever—my Google Drive or my Gmail—one huge disadvantage it has relative to me is that I have this sense of when I have found what I was looking for. It's always clear to me. I'm always like, “I've not found it yet,” until I find it, and then I'm like, “That's what I was looking for.”

That's obviously predicated on my historical familiarity and the fact that I was involved in creating all this stuff, right? So I kind of know, yes, that's the thing. The model obviously lacks that kind of deep familiarity and historical participation, so it can't be so confident in general that that was the thing I was looking for.

How do you guide models when they're doing this kind of search to make that judgment call of when to stop the search? I find that to be a very perplexing thing in my own building.

This is all about the mechanism of the request between yourself and the model. In the instance of, “I have a fuzzy idea somewhere between some mental neurons of what I might want,” you might actually be doing the most efficient thing by just going through and searching.

But if you think about completing work in a workplace, work follows some sort of structure, right? In software development, it follows a spec. Therefore, you can express—and this is how people will do it—what you're roughly trying to achieve with Blitzy. It'll look against the source of truth and come back with a plan in the form of a future-state technical specification, like what architects deal with all day, to go do that work.

Until you can provide the system with the right structure of output, it's unlikely, from a system level, to do your bidding correctly. The question is, how do you create the right interface experience to enable humans to enter with a fuzzy input, get confirmation on a structurally strong output, and then send that task off to the system, versus the experience that you just described, where fuzzy input is sort of all you get?

Some people use chat for this. They might say, “I'm roughly thinking about this idea. I think it's this thing tied back to this date,” and then it can say, “Is it any of these possible things that you want to go explore further?” That's an intermediate abstraction layer ahead of the true deep search.

It's all about creating an intermediate experience between the system of intelligence, the system of record, and how you're expressing that ask.

As much as possible, when we actually get to the work stage in the process, you hope that, at that point, you've effectively given the agent everything it really needs to know, or at least the location of everything it really needs to know. Then it can do additional searching to read in the details of that file, that function, that service, or whatever.

But you've already had a human approve a plan and sanity-check that at that level, so it should have clarity, basically, on exactly what it needs to be reading.

That's right. What's super important is that the system is capable of doing both steps. Meaning, I can provide you what I'm trying to do inside my 30-million-line trading system, right? Then Blitzy will come back in about an hour after you give it, let's say, a page of general instructions for what you're trying to achieve on the codebase, and it'll come back with a very in-depth implementation plan.

You didn't think about the edge cases or the services that it might touch, right? It's the whole point: it's impossible for a human to grok everything that might affect the system. That is phase 1 of system interaction, being, “Hey, heads up, human with a limited human context window: here is the plan that you expressed against this enterprise codebase, and here's a bunch of things that we're going to have to do to implement this that maybe you did or didn't think of.”

By the way, if you want to do this a different way, that's cool too, but let's assess and make those trade-offs before we go off and write 100,000 or 1 million lines of code. That experience of leveraging system intelligence to generate a clear version of the work is required as phase 1 in order to do large volumes of work in phase 2.

So earlier, you mentioned that sort of, “My approach is going to work until it starts to fail.”

What’s going to cause it to fail, and what should I be mindful of as I approach? How do I know when I’m approaching failure, and how should I be prepared for those failures?

Yeah. Maybe I’ll start by saying how we recognize failure in our system, and then we can map it onto your own passion project, which I love. Inside the Blitzy system, at runtime, we’re doing as much work as we can autonomously. You can think of it as spec- and test-driven development at the time of compute, and we’ll retry, reloop, and recursively go back and self-improve between running the application and getting the desired outcome. But at a certain number of attempts, we have to say, “Okay, we can’t do this part.”

We have a separate and independent evaluation system that figures out what the desired end state was and what the system was able to do. Then it writes the documentation for, “If we could optimally get to this end state, this is the most likely path that we believe a human could take and that this system can take.”

So you need to build these mechanisms and systems—the system of work, the system of QA, and the system of evaluation—to operate somewhat independently. When you get the output, as part of the output, you also get the report on what the system failed to do. We always call that the human-completion part.

Getting these to be really accurate allows you to move with confidence. For us, that’s a project guide that says, “These functions or these parts of the application need your help. By the way, we did all of this work, passed all these tests, here’s the QA, and here are the screenshots, so you can feel good about that. Go review that code, but spend your time on this part.”

Mapping that onto yours, you would need to have: this is my intent; get some work; that work has QA involved recursively ahead of it, getting the outcome; and also a separate system to evaluate the effort of that. Both of those artifacts should come to you, and both of those systems within your application should be independent in nature.

It’s like a report card. So, how about a kind of model scouting report? You had said that you don’t want to use just one family of models. That’s clear to me, but why? Do you have rules of thumb for which families are better in which ways? How many are you using? Does Groq crack the list? Do any Chinese models crack the list? Are you fine-tuning models for particular purposes? Give us a tour of the model zoo.

Yeah. We use the 3 major families of models in Blitzy today: OpenAI, Google, and Anthropic. The other ones are great and may be incorporated in the future for different purposes. But it’s very clear that researchers’ preferences are somehow expressed in these model intelligences, and that they’re very, very smart in different ways. They’re much, much smarter when you compare different families of models and have them review each other’s work.

If you took Opus and Sonnet from Anthropic and had them compare each other’s work versus having an OpenAI model and an Anthropic model compare the work, you’re going to get demonstrably better results by having a different family of models review—or different companies review—the work, at least in all of our experience.

That is super interesting, and it changes every day. First-pass code generation: Anthropic remains really, really strong. Structured output and code review: great results from OpenAI. By the way, what I say here will probably deprecate by the time the podcast even comes out. Gemini has been better for long-horizon work, task checking, task lists, and keeping things progressing.

I don’t know—date-stamp this toward the end of January, and I’m almost certain that it’ll probably change by the end of February.

Yeah, the pace is unbelievable and relentless, for sure. Translating that back to the meta-structure of the whole thing, I’m imagining that there’s a brief given at the highest level where it’s, “For this kind of task, you’re probably going to want to use this model. For this other kind of task, you’re going to use this model.” Are you then allowing the system to dynamically select which model to use as a subagent as it unfolds itself?

Yeah. You can think of an example of a dynamic algorithmic rule: You can pick the one that you think is best for this situation, and the reviewing agent must be a different one of these other options. We’re not constraining the choice, but we’re sort of constraining the selection of choices in the review model. That’s an example of a sequence of steps used in validation that is dynamic in nature, not, “You must use Gemini, then you must use OpenAI,” for instance.

Then you asked about fine-tuning. Fine-tuning is a last-mile optimization, I would say, and not a bet on dramatically improved models. Fine-tuning is an expression of, essentially, “I can’t get enough correct context engineering within the system, and I can’t get the right results.” There’s a place for it in the ecosystem, but as soon as you fine-tune a model and the next one comes out with more raw intelligence, you’re basically out of luck.

We are much more bullish long-term on what we call memory. You see a very shallow instantiation of this in tools like ChatGPT, where it will start to remember your preferences, but there’s a lot of memory that occurs in the enterprise environment. Memory is another way to express both relational and semantic understanding, but with a lot more signal of truth.

To get to 100% autonomy within an enterprise workflow, you have to sustain memory of the actions of the best people and what they view as correct, and then store that in your instance—in the enterprise’s instance of the platform. In this situation, the enterprise instance of Blitzy. That’s how, even after the architect—the only one who knows that system—retires, the enterprise itself has that IP in its instance of its AI system.

Memory for LLMs—the missing middle, as I’ve sometimes called it—has been an obsession of mine for a long time. I was really taken by the Mamba architecture when that came out, just because here we have something that’s kind of competitive with an attention mechanism and a transformer, but it has a fixed state-space size. We can potentially run this thing indefinitely. Obviously, there are still limits to that.

There’s a spectrum in memory between pure scratchpad and deeply integrated nested-learning, continual-learning, futuristic stuff. That sounds awesome, but it also has some challenges. With a nested-learning-type approach, the model may perform better, but it doesn’t necessarily mean that you have a record of what happened or what the key lessons were, because they’re in the weights.

So what do you think—if you were going to put your own spec, let’s say, out to the frontier model companies for what you want to see memory look like—what is the shape of memory that would be the biggest difference-maker for you guys?

Long-term memory, I don’t believe, will be solved at the LLM level. LLMs have so much momentum behind them that another architecture, even if it were to solve for this, would not get the level of intelligence required to execute what these systems need. Memory is a problem to be solved at the system layer—the application layer—and that memory is domain-specific: what’s important to remember in what instance.

You can think of memory as going all the way back to the traces—a series of steps or actions. The series of steps were driven by decisions that you chose to put in context. The decision to put something in context might change in the future based on what you’ve learned from the way the enterprise expressed work.

This is tying all the way back to your context-management system. That’s where you’re storing memory and preferences based on actions, not based on model weights.

Interesting. I have some hope that there could be an integrated memory breakthrough.

That will certainly make things easier. It will make things so much easier. I hope for it; I really do. Even some expression of memory in the model layer will ease the burden on the system layer.

But when it’s—how much memory? Let me give you a specific example. If you think about memory at an enterprise-codebase layer, the things that one needs to remember are extremely locally specific. Memory on an enterprise codebase is not universal.

It’s not, “Use this payment-provider service over this payment-provider service,” even though my enterprise has 9. It’s, “Hey, when you interact with this cluster of context, you need to use this service,” even though to you they look relatively functionally equivalent.

There's some organizational or contract reason why you need to use this service, right? And so that is so local from a context-interaction perspective that impressing global memory at the model layer actually has severe limitations. And so the question is: How do you bifurcate global truths or global memories, which people instantiate with rules today to try to manipulate these models to do what they want?

How do you instantiate universally true, long-term memory in the weights and the models? Because these are more brute-force levels of intelligence, while keeping locally contextual, memory-based decisions at the system or application layer?

Yeah, I totally agree that you can't—I mean, the nature of compression is that you can't compress everything, right? Something's got to be lost.

Almost every problem, I feel, is a search-and-compression problem at the end of the day. You're trying to get rid of as little loss at compression as you can, and you're using search to try to minimize that. We think about search and compression all day.

Yeah. Yeah. It reminds me of—I'm sure you've heard this—but the old, kind of old—I don't know if it's a parable or something of the sort—where a junior developer gets a problem, gets all excited, and starts ripping off code, just typing a mile a minute, whereas the seasoned vet kind of leans back and says, “I think I've seen something like this before.”

And that's kind of the thing that I can imagine, even with a finite-size memory space. I can imagine that getting developed to the point where you could get tremendously higher reliability by going out and finding the right documentation when it's actually needed and making the right decisions. Not because the model would have memorized every last detail of it, but because it would have that sort of intuitive sense that we probably have undervalued in ourselves until we've seen how much we contrast with LLMs that lack it—a sixth sense of, “Yeah, there's something here that I kind of know I need to go, and I kind of know what I need to get.”

And if you were to look at how Blitzy spends time as the representation of the best cluster of developers at inference, we spend a huge amount of time in planning, system understanding, and impact analysis. Meaning, let me really methodically think through this, and then let me spend a lot of time figuring out everything else that this is going to affect. The code generation is relatively fast, right? And then a bunch of time on QA and validation, and recursively improving the code based on what you're trying to achieve.

But writing a million lines of code is as fast as you can stream tokens, right? Our runs are as short as 12 hours and as long as a few weeks, depending, but it's a huge, huge refactor. And so, as you break that up, it is that wise-developer motion of: Let me sit back, let me plan, let me think, let me think about everything this is going to impact across the system, and then let me implement—as opposed to the junior dev, which is just cranking out code at minute 0.

Yeah. Okay. That's a great transition to a couple of questions I had around what you might call Blitzy scaling laws. Another way to think about it would be limits to parallelization. You could just sound off on it, but I'm interested in what the curve is. Sam Altman famously tweeted that it's going to be really weird to live in a world where you can pay exponentially more for marginally better results. So you've clearly got a curve like that. I'm interested to know how you think about that curve and where you want to be on it. How do you know when to stop paying for more inference?

And then parallelization: Kimi K2 just came out. They've got their agent swarm thing, and there's another kind of logarithmic thing here where 1,000 agents does not make you go 1,000 times as fast. It can make you go 5 times as fast, maybe 10 times as fast. You could maybe characterize what that looks like and also what you think the reasons are for it. Some things, I guess, are just sequential. You've got to plan before you can execute, and so on. But yeah, that's plenty of prompt. Take it from there.

Nice. Yeah, good, structured prompt there. Let's talk about parallelism and the limits of parallelism. When you think about the work getting done at the system level, this is a core topic within the domain that we operate in: enterprise software development. What sort of work can be done in parallel versus sequentially? Trying to do everything at once is a surefire way to get really, really bad results, right?

Just like in engineering, an engineering team will look at an epic, break it down into tasks, and realize which tasks depend on what. That is a huge part of what is happening in the planning phase within Blitzy. We are deciding, based on software development fundamentals, that thing X depends on thing Y. Therefore, we have to get X to build, compile, and pass tests before even starting on thing 2 here. We must do it in that sequence. That is what is happening for us at the planning stage: parallel versus sequential tasks. That is just a software development problem set.

Now, in other domains, there are other ways to think about what can be done in parallel versus what can be done sequentially. But in engineering, it's very easy to grok what depends on what in a sequence of work, and therefore we have a system that algorithmically works through and assesses that.

That is the answer on parallelism: We want to do it as high-quality as possible, which means that in the instance where the system is not entirely sure, it will assume sequential; in the instance where it is extremely sure that it is parallel, it will do it in parallel. It's sort of a tolerance preference on quality, which answers your first question, which is: Pay more to get better results.

Our thesis is that we will pay any incremental dollar, write any incremental algorithm, and really do anything within the system to improve the quality of the code, all the way to fully autonomous, unsupervised software development as the goal for the company. We are not, in our opinion, cost-constrained, because the other side of a pull request is human labor, right?

I would much rather have that human be working on problems that are on the edge, that are truly innovative, and that are thinking about absolutely disrupting the way that they're applying technology to their business, than have them spend time on vanilla application development—just regular application development. They've already expressed their preferences vis-à-vis Blitzy on the technical design that they want implemented, and then they're handing off that work to us.

We typically do 80% to 90% of the sort of quantum of work, and then we'll call out what we need the human developer to do. But in the vision of the company, that remaining work—which is just traditional configuration and QA, “That's a bug, that's a bug”—is a bug in the system relative to the vision of the company, because software developers are problem solvers. They're engineers. They're problem solvers at day 0.

And if we can have the world's smartest people working on problems on the edge, not working—well, not worried about packaging compatibility or QA—we've done a great service to humanity.

I'm going to come back to the developer experience in a few more minutes, I suppose. Let's talk about the economics a little bit more, though, because on the website there's this $0.20-per-line component to pricing, and you can kind of complicate that. I think there's a base buy-in level and then $0.20 per line beyond a certain level, or whatever.

But that strikes me as creating a possibly interesting tension for you, where you've now said, “Okay, this is what we're going to charge you,” but then you also just said, “I'm willing to spend every incremental API call necessary to maximize value.” So is there just enough headroom under $0.20 that you don't mind bumping up and down?

Do you ever have projects where you have to go to the customer and say, “Hey, actually, I need kind of $0.25 a line, but it'll be worth it because we're going to do that much more with Claude Opus here, and it's going to make it better,” or whatever? How did you come to that $0.20? And how safe of a line in the sand has that proven to be for you?

Yeah, if we have to increase prices, we will. That's our sort of going endpoint—not in the act of contract engagement. If we have to dramatically increase compute, we can dramatically increase compute to get to 100% autonomy. Our customers will sort of coast off of that for the duration of their contract, and then we'll have to right-size it.

I'm not necessarily worried about gross day 0 versus value created day 0. It's an attractive business today, absolutely. That doesn't matter. What matters is that the amount of value left to be created is so high that if we close the gap from 80% of the work being completed autonomously to 99% of the work being completed autonomously a year from now, the net-new customers are going to be so happy—more than happy—to pay more money, because they'll be able to do so much more with the same amount of people.

And so, as you think about it, really, it's the delta of value creation that you're thinking about, and you always want to push as hard as you can on value creation, because the market size for software development is $1.2 trillion in labor, but that is an infinitely expanding market.

Software is designed to fix the productivity problems of customers. If you're telling me that we're out of problems to solve with software, that's where I don't believe you. Our market size is capped by the problems that can be solved with software.

The goal is to get to 100% autonomy and, in a way, get to 80%, 85%, 90%, or 95%, which is incredibly and deeply valuable for the enterprise that wants to move incredibly fast. They're thrilled with this level of autonomy today, and you can't let a short-term pricing decision dictate the technology decisions when the value creation is so high.

Yeah, that makes sense. You mentioned going from 80% to 99% completion. I guess, for starters, maybe even taking one step back from there, when you get a new customer, how do you know if this is going to be an easy or hard engagement, and what do you have to do?

I've seen this come up all the time in the text-to-SQL world, even in relatively small-scale things. It's one thing to look at the schema and be able to write valid queries against it, but it's another thing when there might be 3 different columns in a table that are something like variable_1, variable_2, and variable_3. Which one am I supposed to be using, and why do these exist? How do they differ in meaning?

I imagine you must come into a lot of different environments where sometimes there's great documentation and it's reasonably clear what's going on and what you need to do. Other times, probably not so much. Do you have a process for identifying what is genuinely ambiguous and potentially only exists in the heads of the employees at the company? Do you have an AI agent interview those people to extract that information? What does that human side of the onboarding look like?

Yeah, it's a great question. The typical enterprise has very little documentation and very little test coverage, right? Those are the first things that we actually look to address with Blitzy. The awesome part about this is that by addressing documentation and test coverage with Blitzy, you automatically increase the effectiveness of all of the AI code-generation tools that are in your stack. We highly recommend that you have the individual developer-productivity tools as part of that stack, so you get super-fast time to value as you're getting implemented.

As you ingest a codebase, there's an opportunity to provide whatever documentation you do have. What's super helpful here is if you have domain-specific information—say, “I'm in finance, and when we say this in our code comments, this is roughly what it means.” There's an iteration process at ingestion where we're going to provide you a spec of everything as we understand it today within your codebase.

Everything will be technically accurate to what we can surmise technically, but the product portions—where we're expressing what you're trying to achieve—are where we'll have an iteration period. We'll say, “Here's the blank slate without any information, and now let's provide the system with the information,” rather than starting at zero and asking you to tell us everything that we don't know.

Here's what the system can technically understand: all of your dependency diagrams and all of the variables are classified technically correctly. Now find out what, from a product perspective, we should have as additional context, provide us that, and then we get off and running, right? That's the process to get to truth from a spec perspective.

You also have to remember that the spec is the human-readable abstraction of the truth. Because, in context, we're always using the actual source of truth—going back to the source code and pulling that into memory just in time—the product spec can be a little bit inaccurate at the end of the day.

If you're moving from C to Rust, it doesn't necessarily matter if the thing in the spec, which is defined to be human-readable, is exactly precise, versus the fact that we can run the application and mirror the exact effects on the other end in the case of a language translation.

When you're doing product development, which we do—I would say half of our business is large-scale modernizations and refactors, and half of our business is steady-state product-development acceleration—that's when you want to be a little more prescriptive and a little more precise. The system will be using your product expression, from “We're doing this in finance,” to go and make further decisions.

So then, when you're going from zero to 80%—80%-plus of the work being done—are there moments when the system loops in a human on the Blitzy team and says, “Hey, I need help with judgment here,” or, “I think this is a question we should be able to get answered”? Or is it literally, from go time to 80%-plus, fully autonomous?

From go to pull request—just from go to pull request. It would be an impossible task to try to insert a human into this process. The way that you would do this with agents at scale just doesn't work.

The only thing that stops the process at the beginning is if we're missing an environment variable or something from a configuration perspective to actually build and run the application. You could find that out by building and running, then trying to add some net-new piece of functionality that actually calls a service that you didn't need to run the initial application.

In that instance, it will notify the customer in Blitzy and say, “Hey, we need access to this service, and this wasn't a part of implementation or setup.” But from a spec to pull request, it's all agentic, and it sort of has to be.

Yeah. Tyler Cowen rings in my ears all the time: “You are the bottleneck.” So, no doubt, you obviously have to keep that to a relative minimum, but it's interesting that you have basically zero human involvement.

The point is, if the system can't do something, it goes onto the human report for the enterprise's success. We don't have to be 100% out of the gate. We pass unit tests, integration tests, and end-to-end tests—we do all that—but whatever remaining work is part of the ultimate report goes out to be completed by the team.

That's an awesome use of Claude Code and an awesome use of Cursor. People pull that report down, go deep on whatever edge case Blitzy couldn't solve, get that ready to go into production, go to QA, go to merge, and then start their next sprint with Blitzy.

The system is designed to account for the fact that we want to accurately do as much work as we can and then say, “Great, the human pickup is on the back end of this pull request.”

So what is that last up-to-20% today? You mentioned edge cases just now. Is that the bulk of it—just unanticipated scenarios that were ambiguous or otherwise problematic that we're kicking back to the humans, not so much because of code, but because of missing judgment that wasn't supplied up front?

Yeah. It's typically items that we think were not captured in the testing strategy, right?

As a double-click, anytime Blitzy touches any file, we're doing unit tests before and after. As we do clusters of context, we're doing clusters of work. We're doing integration tests between services. When we're done, we're doing end-to-end tests at the end, right?

There will be some instances where, let's say, we're like, “Oh, we passed 73 of 75 tests,” and, for whatever reason, we change things to fix item 1 and it breaks item 2, and we change things in item 2 and it breaks item 1, right?

The system will say, “Great, we're 73 out of 75 from an end-to-end testing perspective. These are the files that we're going back and forth between, and you need to go in as a human and figure out where there is conflict between these 2 services because our system has gone back and forth so many times.”

It's funny: sometimes the task is an impossible one. You're like, “Oh, okay. You're asking for 2 contradictory things in your spec,” and this is one way to prove that you're asking for opposite things. Sometimes it's configuration stuff. Sometimes it's just QA work, right?

As part of the report, we'll break down the remaining tasks and the estimated hours to complete those tasks for the human teams, along with who would be responsible from a functional skill-set perspective. Some testing strategy that we didn't get to 100% on, which we can align on, and some plan for code review and QA are really included in what we call the final 20%.

So how did you get to 99%? Because when I hear that description, it sounds less like something that a new model is going to be able to handle and more like people just aren't that maximalist, I guess, in terms of really defining what it is they want.

We have large customers that will get a Blitzy pull request. They still go through a dual-review PR strategy.

I recommend that whatever your QA process is, you should continue to do that for the foreseeable future—for anything else, for regulatory purposes. But those customers won't touch a line of code; they'll press Merge, right? Those customers are unbelievable at expressing intent and doing spec-driven development.

For a large majority of customers that are not as far along that curve, what they'll do is express intent, get a spec, get code back, and then at the code step realize, “Oh, I didn't consider this outcome,” even though it was maybe expressed in the spec. They're moving so quickly. So we had to build into the product the ability to refine further from the Blitzy platform, right?

People aren't used to doing two or three months of work and making those decisions. You used to be able to get to month 2 and then figure out the nuance between month 2 and month 3. Once you get into the code and realize, “Oh, I didn't express this implementation the way that I would have preferred it,” and it was hard for you to really conceptualize what that would look like between the spec and the implementation, they can just go back and refine that existing pull request.

They can provide updated guidance: “Hey, actually, on the implementation of this portion, I want to use this approach.” Then it'll run for a much shorter amount of time and adjust the existing pull request to their preferences. This has to do with existing patterns or behavior today, but what we'll see is that folks will go through this flow as they're getting familiarized with Blitzy and refine that larger amount of work once or twice. Then they'll naturally start to get really good at expressing their intent or identifying it at the spec stage because they're going through the muscle of basically being a systems-level thinker, a systems-level architect, and getting all of that implemented.

What room for improvement is there on the models? It seems like what you're describing there is still that models could get better, but it's really the humans who need to get better at expressing what they want for you to drive that completion number up toward 100%.

Would model improvement then translate to even faster execution, cheaper total inference cost, or are there still things that you would highlight as, “Yeah, models are not that good at this, and it would actually be really helpful if they were better at it”?

Yeah, we ultimately want more intelligence, right? Cheaper is fine. Cheaper is fine. But if you think about the instance that I walked through with the different end-to-end tests going back and forth, failing as the code was recursively going back, running the application, and trying to fix it, today our system will just be like, “Those two things happened. Go look at it, human.” We're stuck, right?

If you had more raw intelligence, it could very prescriptively be like, “Hey, this is exactly why this is happening. Here's the trade-off decision that you need to express to us. Which one of these routes do you want to go from a trade-off decision?” Then I can go and implement that.

When the trade-offs themselves, which are complex, can be completed or understood by the model itself, it could then come back with 2 different pull requests, both of those with the end-to-end task fully passing, and say, “Hey, I took trade-off 1 here and trade-off 2 here, and those are the only 2 logical trade-offs that you could have made,” as opposed to, “I couldn't solve this problem. Over to the human.”

So we want more intelligence. It is going to allow us to go further in situations, and it'll allow you to be less precise at the spec stage or not have to be so forward-looking in your technical design.

Yeah. Okay. Interesting. It seems like we're pretty close, though. We're closer than people think.

Yeah, not that many more special requests there.

Hey, Sid, we've gone pretty deep. I was going to have you join in if there were some nuggets you wanted to drop and add in here. Just as an introduction, Sid was a prolific inventor at NVIDIA. He's been thinking about building large-scale software systems since he was a little boy, actually. He's got great stories about taking computers apart and building software when he was a little kid.

He's really the inventor of a lot of the core—really, all the core—technology here at the company: a large-scale context-engineering system that unlocks the ability for us to understand 100-million-plus-line codebases and long-running compute orchestration systems. Sid Pardeshi, welcome to The Cognitive Revolution.

Boy, yeah, we have covered a lot of ground, and Brian has done a great job of explaining a lot. I was just going to go next to strange behaviors from language models. This is a theme of my life, this feed: I am both extremely enthused about AI, love what it can do for me, and experience incredible productivity gains all the time. Then I also pay reasonably close attention to research that shows all kinds of emergent, surprising, and sometimes, in my view, scary bad behavior from language models.

One big question, of course, in the big picture is: To what degree can we successfully get AIs to monitor the work of other AIs and get to a point where we can be confident in the system overall, even if some of the models, some of the time, are doing something that we would want them not to be doing?

I'm interested in what you guys have seen there. QA is one dimension of it, just catching bugs and catching mistakes, but then there's also the famously—I think Claude 3.7 was maybe the high watermark—of writing unit tests that would just return true and always pass when obviously the core objective had not been met. How would you guys describe the trends in that? I assume it's improving, but how much have you seen that sort of thing improving?

What have you done, and how well has it worked, to get AIs to detect those kinds of problematic behaviors in one another? Obviously, at the end of the day, you want to deliver something to customers that doesn't have these fake unit tests, right?

You've really described 2 patterns there. One is strange behaviors from the LLMs and how to control them, and one is the LLM-as-a-judge philosophy. We've been super early with LLMs as a judge. I think one interesting bit you described there was getting LLMs to correct each other's work.

What we've seen is that LLMs definitely have some peculiar behaviors given the conditions, right? Assuming everything's constant—temperature, top-p, top-k, whatever parameters you're using to influence behavior—and assuming constant prompts, if you gave 2 different sets of LLMs, and let's assume they're both following the best-practice guidelines of each vendor, OpenAI and Anthropic, for example, if you gave them the same situation or condition, you may get different reactions.

For example, we've seen SWE-bench Verified, right? It's a very popular leaderboard, but we have different scores even though the problems are very similar in that sense. There are different problems that Anthropic fails on versus OpenAI. But if you go to a real-world situation where you have a lot more ambiguity, what you will see is that if you run the same situation through Claude multiple different times, you may find that it comes up with a different resolution.

For example, it's an ambiguous situation. There's only 1 way to solve it correctly. If you run it 5 times, it may be that Claude is able to solve it correctly 1 or 2 times. Maybe the approaches that it took are slightly more nuanced or different each time. That is because of how the transformer architecture works.

These are sequence-to-sequence models. They're generating the next set of tokens to answer the question, and they may end up sampling different parts of the space. That is one way you end up with a difference. Or they may just end up taking a different trajectory. They could have executed a search query. Maybe the correct answer used the tool correctly and wrote a more elegant search query to find what it was looking for.

Because these are probabilistic models, at any point in time there's a probability that the LLM lands on the right tool and uses it correctly. That's why you have these differences, and it is definitely effective. The way to make LLM-as-a-judge effective, from what we've seen, is by using 2 different models—2 dissimilar models—to evaluate each other's work.

What you're doing then is not just adjusting for these probabilities, but also for the inherent architectural differences. Not at a very deep level, but let's say GPT-5.2 is definitely built a lot differently. It has a different set of parameters and a different size than Opus 4.5. It may take a different trajectory and use tools differently. By that sense, you have now increased the chances that collectively they land at the correct answer, which solves the problem.

So that's for LLM-as-a-judge. It's an important part of landing at the correct answer. But let's talk about the strange-behavior aspect that you mentioned. That's really interesting. We've been very deep into the Claude family of models and OpenAI. For example, one interesting behavior the o-series models of OpenAI had was that they were very reluctant to use tools.

So these were reasoning models—the earliest reasoning models—but they did not like to use tools. If you ask the model to search the codebase to come up with an answer to something, you find it jumping to conclusions without doing thorough research. That was a problem with the earlier series of models.

But if you look at the latest OpenAI models, like Codex or even GPT-4, GPT-4 was by far the best model when it came to tool calling. GPT-4 was active at the time of o1 and o3. We had repeatedly provided feedback to Anthropic that GPT-4 outshone Claude 3.5 by a mile.

Even though the best thing about Claude 3.5 was that it used tools really well, it was nowhere close to being as powerful or efficient as GPT-4 and GPT-4o at tool calling. But as time went by, that changed quickly. Claude Sonnet 4, Sonnet 4.5, and even Sonnet 3.7, though not to the same extent, were really good at tool calling.

The problem with Sonnet 3.7 was that it was overeager. It made a lot of mistakes when calling tools, leading to tool schema errors. If you didn't validate that correctly, it could cause all kinds of issues in your application. But they quickly fixed that with Sonnet 4 and 4.5.

The most interesting strange behavior with these models is that they tend to give up as soon as they have real context anxiety. That's how I like to describe it. Even though Anthropic says—and this applies to OpenAI as well—that it's a much larger context-window model, for example, I think GPT-4.1 introduced 1 million tokens, if I'm not mistaken. But the documentation clearly said that if you exceed 200k tokens, you may experience different behaviors. The request will take longer, and the quality may not be that good.

For Claude, even though it says it's a 1-million-token context window, you will notice marked differences in behavior the moment you exceed about 100k or even 200k tokens. It's not just about the price. Anthropic charges you differently if you exceed that, but what you will see is that if you're working on a complex problem, the model will tend to give up.

It will say things like, “Okay, because I have these time constraints.” Now, what time constraints? I never told you that you had to finish in an hour or 10 seconds. I just gave you a problem; I expect you to solve it. But then the model brought in the concept of time and said, “Because I have these time constraints, and I have been working on this for too long”—and, by the way, “too long” was just 10 minutes—“I have to now wrap up and give a final response.” Then it gave you an incomplete response.

Then there is context pressure: “This seems too complicated. Let me take a simple approach.” That's where you have the behavior you mentioned: “Let me return true, and let's see if this solves all of the requirements. You said I should not have any bad code. Check. I should not have overly verbose code comments. Check. I'm just returning true. And the test should always pass. Check. I'm just returning true. It's always going to pass.”

What I've done is philosophically correct, justifying to itself that its decisions are correct, even though what it's doing is blatantly wrong relative to the user's original instructions. These are due to external factors that the model providers are implementing.

When we experience this, we solve it our way. There are a number of ways to prevent these issues, one of them being the obvious one, which is prompting. But we reached out to Anthropic, and Anthropic actually fixed them. Claude 4.5 Sonnet had this issue, but Claude 4.5 Opus does not. It has other kinds of issues. As an application builder, you're constantly solving for these issues in production with different labs and model providers. They all have different vectors in which they would effectively fail for any given use case.

From an overarching information-theory perspective, they call this concept entropy. The outcome of a probabilistic system has high entropy in LLMs or probabilistic systems. The goal, or the purpose, of the system and application layer is to reduce entropy and get to reliable outcomes. The techniques we're describing reduce entropy to get closer to a desired truth.

I love that you mentioned entropy, because I was just thinking about how Sid had mentioned temperature. That got me thinking back to my early LLM-based application-development days, when that was a huge lever that I would mess with depending on—

You were a high-temperature guy, I can tell.

It depends on the use, but certainly these days it seems like some of the APIs have even removed temperature. I certainly don't think about it nearly as much as I used to, so that tool to control entropy has kind of gone away. But I wonder what other strategies you guys have for perhaps progressively increasing entropy.

This is something I talked about with the AI co-scientist team at Google. They said that, in their system, searching through the scientific literature is the main source of entropy that they sometimes need to get off a local maximum or out of a local minimum, whatever you want to think of it as, and onto the next higher hill that they can then explore and climb. What do you guys do?

I would imagine maybe you want your first pass to be the most reliable; you want to take your best shot, right? I used to, in coding applications, turn the temperature to 0. I figured I would want the model's best guess first, but then, if that didn't work, maybe I'd turn the temperature up. But again, there are a lot of different ways to turn temperature up.

You could context-engineer a little bit differently, swap out to a whole other model, do a web search for some commentary on this problem, whatever. Hopefully, with different inputs, you can eventually land on the right output. Long-winded way of saying: how do you ramp up the entropy as needed when the first default answer isn't working?

Yeah, I would say the levers have changed. That's a very helpful background, so thanks for setting it. Let me add more color to it.

In the beginning, you had temperature. For code generation, or any use case where you didn't need as much creativity, you wanted to focus on getting the right answer rather than the most creative answer. So the best-practice guidance was to bring the temperature down to 0, 0.1, or 0.2, depending on the use case. Different model providers had different guidance.

But then, as you introduced tool calling with Claude 3.5 and GPT-4, having temperature with tool calling created problems. You already have the ability to land on a different response because it could take a different trajectory in tool calling, and then you have temperature influencing its behavior and creativity. That just created complications.

What changed significantly—what really changed everything—was the introduction of reasoning. When reasoning started with the o-series models and then eventually with Claude, both OpenAI and Anthropic forced you to set temperature to 1, which means you don't have any control over the temperature parameter. The lever has changed from temperature to the thinking budget.

You may have a 200k-token context window or a 1-million-token context window, and you have between 0 and however many tokens of reasoning the model supports. Typically, you've seen 32k for Opus and Sonnet, 64k for some others, or about 128k for OpenAI models. That's the reasoning budget: how much thinking the model is allowed to do before and/or in between responses.

In the beginning, you only had reasoning—1 set of batch reasoning before the model gave you a response—and then that was it. It went into its own trajectory there. There were hacks you had to use to get a model to think while it was working, while it was calling tools.

But then you had what we now call interleaved thinking. That's what Anthropic calls it, where the model thinks while making every tool call. It automatically thinks before making a call, and then there's a budget that you set for the overall amount of thinking: how much of the context window is allowed to be used for thinking.

Then there are weird metrics for prompt caching, whether or not thinking invalidates prompt caching, and how much of the thinking actually plays into the context window. All that is different between different providers, but at a high level, the reasoning budget is the lever you have.

If you allow the model to think for longer, you get higher-quality answers because, essentially, what the model is doing while it's thinking is taking a stab at creating a response. What happens is: “Okay, the user is asking me to write code to do X, Y, Z. Let me take a stab at it. Okay, this is how I would write it.” Then it writes the actual code, reviews its own code—and this is all thinking; it hasn't written a single token of output yet—and says, “Oh, but I shouldn't do this because the user asked for this.” It goes through that process.

By the time it has either exhausted its thinking budget or gotten a good enough answer to the user's request, it is now ready to write the final response. So it's essentially what you were doing earlier with setting temperature to 0 and maybe running the response 5 times, perhaps with tweaked prompts.

The model is doing that by itself by default and giving you a higher-quality response. Right? If you draw parallels to what actually makes code generation work, look at, as a base case, Claude 4.5 Opus—a really good model in terms of code generation—that gets responses right in one shot. But that is the thinking model. The moment you turn off thinking, performance drops 5 to 10 percentage points, even on SWE-bench, which is supposed to be the easiest set of problems, and the responses are no longer that high quality.

So the theory, based on the observations that we can see, is that the models are really getting better at test-time inference. They're getting more efficient at thinking. The system prompts that all of the model providers are building into the models, which encourage the models to think before responding, seem to be covering a wide spectrum of cases that allow for multiple things: first, higher-quality responses depending on the use case, and also more guardrails and ways to safeguard against things like prompt injection or getting the model to say something malicious. There are multiple layers beyond even prompting that are applicable to achieve this. But we're definitely seeing that the performance gains we're getting from models are primarily driven by test-time inference along this trajectory of model improvements.

And Sid, maybe you could comment. I think it's worth having you comment on the path from where we are today toward fully autonomous enterprise software development.

When we started, and we said that we were going to do fully autonomous software development, nobody believed us because you had context windows of tens of thousands of tokens, and models could write 200 to 300 lines of code at a time, maybe 1,000 lines. But it wasn't good. The code wouldn't compile. It wouldn't do what the user said. The context window was still too small to cater to large enterprise codebases.

We're not really seeing that change. We've had 1-million-token-context-window models for a while. We've even had 10-million-token-context-window models. But the efficient frontier for the effective context window, if you don't want to deal with issues like context pressure and if you always want code that compiles, works, runs, or eventually gets to that point, is still less than 100K, too, right?

So even though we've made a lot of progress on “intelligence”—models are more intelligent and produce higher-quality responses—you still have the problem of context. We've solved that and a series of other problems to make this work. Our perspective is that today, the folks who are getting the best results from something like Claude Code are using tons and tons of techniques to achieve that. You have CLAUDE.md, which contains the instructions; you have maybe a series of plugins that you're using, MCPs, these prompt templates, and a number of other tricks that you're doing.

You're probably using Claude Code to get one output. Then you're switching to Codex and maybe getting that reviewed, and then pasting that back in. The most elite AI users who are getting 10x gains are doing a lot of hard work to make it happen. So have you really changed or improved productivity? I would argue no, because you're still doing a lot of work to get that. You've changed what you're doing. You're not actually writing the code, but you're spending your time figuring all these tricks out, and every 3 months the models change and the prompting practices change. So you're relearning all of that. You're switching between Codex and Claude Code, and there's this constant struggle to make the model work for your codebase.

Our vision has always been that you shouldn't need to do all that. The LLMs are the models today. It matters a lot if you're using, let's say, Opus or some other open-source model, but we're seeing open source catch up. So it is our theory that LLMs will be commodities. Regardless of that, the point really is that you should be able to go to a model with your work, which is typically in your project management tool, like Jira or whatever that is. You should be able to plan the work and get a PR back that just works.

It follows all the coding practices that you outlined. It solves everything in your plan in detail. It takes into account your past, current, and future roadmap. It has the ability to fix merge conflicts if you have a very high-velocity team. It follows the specifications in your Figma, and it just works across your entire codebase. It compiles, the unit tests run, there's good code coverage, and there's evidence of testing.

This is what you'd expect from a human developer, from a really good human development team. These are the unsaid—or quite often very vocal—parameters of success, the criteria for success, that are set within the engineering organization. That is what we've set out to build with Blitzy: just PRs and high-quality code that works. We will spare no effort to make sure that we get to the highest level.

If it's LLM-as-a-judge, if it's more test-time inference, or, in the future, maybe even test-time training to learn about the specific preferences of the user, the goal and vision we have is, again, like I said, code that just works out of the box without you having to do heroics to get it to solve the success criteria.

That's a funny characterization of how work has changed. It certainly resonates with me. I don't code full-time, but I've created many more applications in recent months than I ever used to. So, in some sense, I'm definitely more productive. I made 3 AI apps for family members as Christmas presents this year, for example.

But it is definitely true that I'm always either hands-on or on Twitter looking for the latest tips and tricks. It is striking that, for all of the labor-saving nature of the technology, the people who are getting the most from it are probably working as hard or harder than anyone. Maybe that changes. Maybe it just continues this way until the singularity. I don't know.

I want to do a quick double-click on test-time training. This has been obviously highly related to continual learning, which has been a big part of the discourse recently, and there have been some really interesting advances in that space with respect to much more constrained puzzles, like ARC-AGI-type puzzles and that kind of thing.

We talked a little bit earlier, Brian and I did, about whether there's any point to using open-source models and whether there's any point to fine-tuning. It sounds like today, basically, the reality is that the frontier models are the best. You want to work with the best, but you can't really fine-tune the best, so it's usually not really worth it.

Kimi K2—or K2.5, I should say—just came out, and the community is obviously still digesting exactly where that is. It does seem like all Chinese models tend to be a little bit underrated, inasmuch as I don't think it's actually truly the best, which is what their benchmark graphics would have you believe. But I have used it a bit, and others also seem to be reporting the same thing: it does seem to be really good, and the gap is seemingly quite small between it and whatever your favorite model is for your favorite use case.

So does this change the outlook? I guess whether fine-tuning is worth it or not would seem to depend a lot on the gap between what you can fine-tune and what you can't. This gap seems to have potentially narrowed quite a bit. So I'm wondering if you're thinking, “Oh, hey, this maybe changes the trade-offs or the analysis, and maybe we do want to get into that sort of thing.”

My perspective on fine-tuning has always been very classical, in the sense that you should only fine-tune if you have a very narrow use case that you believe, by fine-tuning, you will get much better performance from, and the rate of that performance gain is much more significant than waiting another 3 months until the next series of models comes out.

You also lose things when you fine-tune. You lose the ability of the model to generalize, and it's not always a given that performance will increase when you fine-tune. You don't necessarily have access to the original dataset, and even if you did, you can't really create a map between what the influence of specific parts of the dataset was on the model's behavior.

That's why fine-tuning, especially when you don't have large amounts of data and if you don't have a very clear niche use case that hopefully has historically been successful with perhaps a previous family of models, is always like drawing from a pack of cards. It's always a risky game.

If you talk about models and their ability to get better, there's also another challenge there. For example, Gemini and, let's say, even OpenAI models are very close in terms of their score on SWE-bench. In some cases, it has been proven that there are models that beat Anthropic on code generation in very specific use cases.

Even then, in the real world, if you compare xAI, Gemini, OpenAI, and Anthropic, they are very different in terms of code generation and in terms of the use cases for code generation to which you would want to apply them.

They're very distinct, right? Even though they're creating similar-ish scores. The point really is that the current leaderboards that we have are insufficient, right? There's a lot of test-set leakage. There's a lot of broad insufficiency from the standpoint of generalizing to a typical use case.

For example, a number of leaderboards rely on the opinions of humans. They'll give you A and B, both with code to solve a specific use case, and you're supposed to select which one you feel did a better job. Depending on my mood, I could have chosen either. But if you don't define clear success metrics that would apply in an enterprise setting, you are not creating a very effective leaderboard, because the leaderboard then is only the perception.

Maybe, in someone's perception, writing a lot of comments is very helpful because I read the comments and I understand the code. In someone else's perspective, this is overwhelming: I cannot read that many comments when I'm trying to understand the code; it's just distracting. So leaderboard design is actually a complicated problem.

You even mentioned ARC-AGI. The fun part is that François Chollet, the creator of that leaderboard, talks about how, when LLMs got to 70% plus on that leaderboard, everyone said, “Oh, I guess AGI is here.” But then he brought in ARC-AGI-2, which didn't really change the difficulty of the problems; it just had different problems of the same kind.

So if you were to give ARC-AGI-1 to a 5-year-old and ARC-AGI-2 to a 5-year-old, they would perform relatively the same on both leaderboards. But an LLM that scores 76% on ARC-AGI-1 would not even score 20% on ARC-AGI-2 when it just came out, even though you have massive gains in intelligence and gains, relatively on paper, on the leaderboard. From a real-world scenario, just because of how LLMs work, you don't really have a change in the LLM's ability to learn something that it has seen for the first time.

The highlight of ARC-AGI-2 is that these are problems that are different from what an LLM would have seen in its training set. They're not harder; they're just different.

There are 2 broad definitions of AGI. If you focus on the academic definition that François is alluding to, he says it's the ability of the model to learn patterns and adapt to them on the fly—patterns that it has not seen before—apply its intelligence to a new problem, and be able to solve it. The other definition of AGI that's more popular, that I've seen floating around much more often, is just human-level performance on a broad range of tasks.

By definition and by real-world results, these are fundamentally different constructs. The problem that I see is that we've gravitated far more toward the latter but ignored the former, and that is why I'm bullish on test-time training. What test-time training promises is that, if we detect a pattern that the LLM is not familiar with, where it's not going to perform well, we can give it more context about solving that particular problem such that it does better and produces better results.

Now, in any problem in general, it's very hard to know whether or not you're going to get the correct answer because you don't have a metric. For example, with code, you can compile the code and know whether or not you're working on the correct answer, or you can define unit tests that you can execute to learn if you're on the right track. That doesn't apply to general scenarios. Specifically in the case of code, I'm bullish that you can implement test-time training in such a way that you improve the odds of getting to the correct answer.

But even then, many of the techniques I've read papers on regarding test-time training—we're not at a point where, at the moment, it's practical to implement that. But I definitely see that becoming a real thing in the next 1 to 2 years.

Yeah, that's something I'm watching very closely to see how that develops as well. I think the 2 last things I want to talk about are just security briefly, because I know that's obviously a huge concern for enterprise customers broadly. They don't want to be importing a bunch of insecure code into their environment, and of course LLMs have a reputation for writing insecure code.

The other thing I want to talk about, maybe in closing, is the labor market in light of all these changes. That could also include who you are looking to hire and as much information as you would be willing to share about your hiring practices. But on the security side, where are we today in terms of security? What have you found to work, and do you think this problem is going away?

I've seen some research suggesting that formal methods can be used both to validate code that LLMs write and also as a reward signal that should get them to write far more secure code far more often. My sense is that, like many other things—“LLMs can't reason,” or “they can't do this, they can't do that”—this is probably going to be something we'll leave behind.

But I know you guys have also had to solve, or at least do the best solution you can, before the models themselves have been properly trained. So I guess all that is to say: what's your view on the security of LLM-generated code?

Yeah, I think it's a shared responsibility. That's the first thing I would like to say. One is that there are many behaviors of the LLM that can be influenced and prevented at the training step itself. If you look at the reports that Anthropic, OpenAI, and Google all put out when they launch a new model, they test against these behaviors.

These behaviors could be getting the model to do something it should not be doing. For example, let's say I need a recipe to create a weapon. If I put that as a prompt, hopefully the model does not respond with the correct answer. But what people have typically done is fool the model by framing it as an emergency situation, such that if the model provided the recipe, it would save someone's life or make a positive change.

So they try to game the reward function that may have been defined for the model and get a response. Prompt injection is one of the ways where they've been able to do that, and there are several other ways to jailbreak what the LLM can do. But ultimately, it comes down to system design.

Security considerations would be different for something like Claude Code, where you interact directly with the model, as opposed to Blitzy, where you have a plan, then you execute that plan, and then Blitzy decides whether or not to follow the instructions and in what way to deliver the code. When you're not interacting directly with the model, the attack vectors change. That's one aspect.

Specifically for the code-generation use case, one aspect of security is causing harm or using content that is not considered clean for that use case. Like I said, the models typically refuse to send you a response, or, if they do, you have to set different kinds of guardrails depending on the system.

In terms of the software itself, it could just be having an outdated knowledge reference. Most models right now have, I believe, January 2025 as the knowledge cutoff, and a number of libraries got updated with security fixes after that date. So if your model did not look up the web when using an open-source library, or it did not realize that this was a bad practice in code because it was newly discovered knowledge, it is likely that your code generated using the LLM has these security flaws.

But thankfully, as with all things in software, you have a number of ways to detect and prevent that as far as the software itself is concerned. One is having defensive tests within the code. If you know some of the attack vectors that your application or product is vulnerable to, you can define tests, and you can use AI to create these tests, have them in the code, and make sure that your code does not have those flaws.

Every time you run a job, you make sure the test is passed, so you add more tests as needed. Two is having tools that check against known vulnerabilities. There are a number of such tools. Sentry is one that comes to mind; there are a number of others that report vulnerabilities and CVEs in the code.

Then you can use AI to address those vulnerabilities. In Blitzy, we run a pre-check to detect security flaws and address them before creating the pull request, so that you don't have to go through that process. At a high level, because you have access to such tools, and different languages and frameworks have different sets of tools, you can provide Blitzy the ability to check for them, and you can also do that with other tools.

Code is significantly easier to protect from security gaps, and I definitely believe that, from the standpoint of coding, we will have tools—or you will have the ability to configure tools—that prevent security issues.

This has been outstanding, and I really appreciate how much you guys have been willing to share. I'm going to take the transcript of this episode and turn it into a to-do list for my own personal AI infrastructure project, and we'll start implementing.

The last thing I want to talk about for just a couple of minutes in closing is the effects that all this is having on people. There was a paper you guys probably saw.

It ended up being fake, but I think it was interesting that it resonated, which was maybe the most interesting thing about it. It was supposedly about materials scientists at some big company, and supposedly they had introduced AI and become more productive, but job satisfaction had dropped. Again, this turned out to be fake, but I think it was shared so much because people felt like it satisfied their expectations, if nothing else.

I'm interested in how you see the role of the software engineer changing. Do software engineers like the way it's changing? And then there's also this big question around junior developers. Is the death of the junior developer much exaggerated? Are you guys hiring junior developers? What are you looking for in your hiring? If you want to tell us a little bit about what your comp looks like, that would be very interesting, but I understand if that's not something you want to talk about on a podcast. What do you think of the impacts? What's underhyped and overhyped when it comes to the impacts on the roles people have and the labor market more broadly?

I think if you think of it from the standpoint of short term, medium term, and long term, then in the short and immediate term, what happens is code is now a commodity. In the olden days, if someone had written a script to do something that was a very complicated or boring task, that script was like gold. You would pursue that developer and befriend them in the hopes that they would maybe share that script with you, the one they had gotten after scouring hundreds of pages of documentation and drawing on the raw experience of having done that numerous times. Now I can just go to Claude, prompt it, get a script back, and do something.

But if I'm a junior developer, I won't be able to look at the script and know whether it would destroy my production database, whether it would do what I'm expecting it to do, or whether it would produce unintended effects, A, B, or C. That is the danger, and that is the difference between using AI and not using AI, to me. If you can't tell that difference, that's the danger.

In the short term, the market is unfavorably weighted toward senior developers. When you give a senior developer access to AI for writing code, they don't have to go through the boring mechanical process of writing a lot of code or even copy-pasting a lot of code. They just feed it to AI, get code back, review it, and get done with it.

But then, as AI gets better—as the chatbots get better, as the models get better, and as the tools get better at preventing unexpected, unintended outcomes, understanding intent, and writing code that satisfies the intent—what's going to happen is that junior developers, and this is already happening with mid-level developers, are performing at the level of senior engineers just because code is a commodity. Mid-level developers have spent some time with the code. They know what a bad action looks like, they know how to make corrective measures, and they still produce velocity gains.

The 2 advantages that senior developers had were depth of knowledge—maybe including the ability to understand how the system works—and speed. You can now get all of that from AI. You can connect Claude Code or Blitzy, or any other tool, to your codebase and have an accurate understanding of what the code is like. There may be hallucinations along the way, but that's changing quickly.

You cannot beat AI in speed. You connect Cerebras to some model, and you're going to get very fast tokens. Even the labs' baseline models are very fast; Claude 4.5 is really fast. You cannot beat the models on speed. As for the knowledge piece, if the model is intelligent enough, as I said, to understand the intent, you're going to solve that problem as well.

Because of that, I believe that in the medium to longer term, you will have junior developers who are far more valuable in that they are cheap to hire. There are a ton of them who are now doing computer science degrees and are not going to be employed, just because the rate at which enterprises are hiring has gone down. In the short term, enterprises are favoring more senior talent. But these developers, assuming they upskill on AI and continue to remain in the industry using the tools, are now going to be much better at getting work done. As the talent ages out, it's going to be replaced by more junior developers. That's a theory I have.

Now, in terms of hiring, we've hired senior, junior, and mid-level developers, and we have a mix of them. They're obviously doing different things. The challenge we have is that, as a startup, we need to produce a lot of code—quantity—and it has to have quality. Time is a very critical factor. So for us, we've obviously shared the bias toward initially hiring a lot of senior developers.

But what we quickly realized is that, for tasks that don't really require senior developer input—it's not a large codebase, it's not really cutting-edge technology, and it's something that is well known—for example, running Blitzy on a leaderboard and writing scripts that automate that process, we hired high schoolers as interns last summer to do this. We have junior developers who are research engineers using Blitzy to run this. They're using AI tools to run all these operations, and we can hire them at very favorable compensation. That's going to be an asset.

Because the market is really flipping on its head, expectations in terms of salaries for software developers are unfortunately going to go down. Junior developers who know AI don't have to unlearn. The biggest challenge with some of the more senior folks is that they have to learn to trust AI. The biggest hesitation for any senior developer who's been around long enough is, “I can't trust anything else other than myself. If I don't write the code, I can't trust it.”

That's a psychological hurdle that senior developers have to adapt to with AI, I would say. The ones who do adapt are going to be immensely successful. But then there's going to be that challenge, and I believe that's a gap that mid-level developers, once they know enough, and junior developers will fill, especially because of the favorable cost equation.

Then you asked about the salary ranges. We've had a number of open positions, and the salary range is anywhere between $100,000 and $300,000 from a cash standpoint. Equity is separate in that discussion, and there's always room for us to pay more for the right talent.

It's interesting how the definition of right talent has changed. Typically, you paid more for someone who had many years of experience and had built many systems. But now, if you were to run a hackathon, you'd be very surprised by who is actually winning that hackathon. You have high schoolers who are extremely adept at using tools and prompting. Often, a good prompt and a good tool can beat out what a senior engineer can do in the same span of time, especially if you're talking about greenfield development.

Hands down, someone with a few years of experience can do a lot better just because of the psychological gaps. But if you're talking about legacy enterprise software where you have to check a lot of boxes, you need a lot of experience. You think something is right, but you realize only after being bitten by doing something wrong. That's a space where senior engineers will continue to thrive.

I love it. That was a great answer, and again, I appreciate how much you have been willing to share. Outstanding conversation. I'm looking forward to getting under the hood with Blitzy, and this is certainly a space that we will continue to watch closely. For now, Brian Elliott and Sid Pardeshi, CEO and CTO at Blitzy, thank you both for being part of The Cognitive Revolution.

Thank you.

If you're finding value in the show, we'd appreciate it if you'd take a moment to share with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitive revolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at aipodcast.ing. And thank you to everyone who listens for being part of the cognitive revolution.